Share parameters between flows

Overview

Etlworks offers multiple options for setting parameters (variables) that are accessible to the current flow and all flows within the nested flow. You can learn more about variables in Etlworks in the relevant documentation.

However, these parameters are not accessible to other flows that are not part of the nested flow. In some scenarios, sharing parameters across different flows is necessary. Here are a few examples: 

• Multiple flows are running in parallel and need to share information.
• The flow logic depends on parameters set by a previously executed flow. 
• You need to collect statistics and parameters associated with the flow in real-time using a webhook and listener.

Etlworks provides a library that allows sharing parameters between flows using a small program written in any of the supported scripting languages.

Implementation

Use JavaScript or Python to set and get parameters. All examples in this article use JavaScript.

Set and Get multiple parameters in a single call

var params = new java.util.HashMap(); 
params.put("key1", "value1"); 
params.put("key2", "value2");

/* 
Sets the parameters as a map of keys and values. 
anyid is a any string, for example flowId or aiditId, 
expireInSec is positive integer which sets the number of seconds after 
which the parameters will expire and will be evicted. 
The max expireInSec is 86400 which is 24 hours.
*/
com.toolsverse.io.RedisSequenceGenerator.instance().setParameters("anyid", params, expireInSec);

/*
Returns the parameters as a map of key and values. 
If there are no matching keys returns empty map.
anyid is a any string, for example flowId or aiditId, 
key is a previously stored key, the key can be null. 
If the key is null the function will use a wildcard key *.
*/
var params = com.toolsverse.io.RedisSequenceGenerator.instance().getParameters("anyid", key);

var value1 = params.get("key1");
var value2 = params.get("key2");
// if the key does not exist returns null
var nullValue = params.get("non existing key");

Set and Get a single parameter

/* 
Sets the single parameter. anyid is a any string, for example flowId or aiditId, 
key is a key (string) and value is a value (also string).
expireInSec is positive integer which sets the number of seconds after 
which the parameters will expire and will be evicted. 
The max expireInSec is 86400 which is 24 hours.
*/
com.toolsverse.io.RedisSequenceGenerator.instance().setParameter("anyid", key, value, expireInSec);

/*
Returns the value of the parameter. 
If there is no matching key returns null.
anyid is a any string, for example flowId or aiditId, 
key is a previously stored key, the key can be null. 
If key is null the function will use a wildcard key *.
*/
var value = com.toolsverse.io.RedisSequenceGenerator.instance().getParameter("anyid", "key1");
var nullValue = com.toolsverse.io.RedisSequenceGenerator.instance().getParameter("anyid", "non existing key");

What to use as a key in set methods

We recommend either auditId (the unique id of the specific flow execution) or the flowid (or any combination of both).

To get auditId of the currently running flow from anywhere in the code:

var auditId= com.toolsverse.config.SystemConfig.instance().getEtlThreadContext().getRequestId();

To get flowId of the currently running flow from anywhere in the code:

var flowId = com.toolsverse.config.SystemConfig.instance().getEtlThreadContext().getFlowId();

What to use as a key in get methods when collecting parameters associated with the flow using a webhook+listener

// the message is a payload sent by webhook

var audit = JSON.parse(message).entity;

var auditId = audit.auditId;

Set parameters when running flow on Integration Agent

The Integration Agent operates on a different network, which prevents it from accessing the same resources as the host Etlworks instance.

To address this limitation, you can implement a workaround by creating a flow which sets the shared parameters by executing /parameters API endpoint on a host machine running Etlworks.

This is a step-by-step instruction for setting shared parameters for all global variables. Modify if you need to set the custom parameters.

Step 1. Create Etlworks API connection. Enter the following parameters:

Endpoint path: POST /etl/rest/v1/parameters.

Base URL: host Etlworks instance URL.

Content Type Header: application/json.

User: user name.

Password: password.

Tenant: tenant id if your are running this under the tenant, otherwise empty. 

Step 2. Create Call HTTP endpoint flow. Select Etlworks API connection created in step 1. Enter the following parameters:

Payload: {parameters_payload}

Script: 

importPackage(com.toolsverse.util);
importPackage(com.toolsverse.config);

var props = SystemConfig.instance().getProperties();

var payload = '{"id":"{flowid}","parameters":{{params}},"expire":3600}';

var flowId = SystemConfig.instance().getEtlThreadContext().getFlowId();

payload = Utils.findAndReplace(payload, '{flowid}', flowId, false);

var params = "";

var keys = props.keySet().toArray();
for (var i = 0; i < keys.length; i++) {
   var key = keys[i];
   var value = props.get(key);
   if (Utils.isNothing(value)) {
      continue;
    }
    params += '"' + key + '":"' + value + '"';
    if (i < keys.length - 1) {
       params += ',';
    }
}

payload = Utils.findAndReplace(payload, '{params}', params, false);

props.put('parameters_payload', payload);

This script will produce a payload which looks like below:

{"id":"flowid","parameters":{"var1":"value1","var2":"value2"},"expire":3600}

You can now add this flow anywhere within the flow executed by the agent and it will record the parameters which you typically record using RedisSequenceGenerator.instance().setParameters(...).