Parameterization of the Connections

Parameterization of the Connections

In Etlworks, nearly all connection parameters can be dynamically configured using tokens. These tokens allow you to replace static values with dynamic ones by simply entering {variable_name} into any field of the connection configuration. During execution, the system automatically replaces the token with the value of a global variable that matches the token name.

Parameterization of the Database Connections

By default, Etlworks opens all database connections referenced in a flow before the flow begins execution. Once the flow completes, either successfully or with an error, the system coordinates any distributed transactions and closes the connections.

This default behavior works well for most cases but introduces challenges when the database connection uses tokens. Since tokens (global variables) are not set until the flow starts running, the flow may fail during the initial connection attempt, before the tokens are available.

To avoid this issue, you need to configure database connections in a way that supports tokenization.

Dynamic URL 

You can parameterize the URL of the of the connection (for example HTTP connection) with dynamic parameters ({tokens}).  Any part of the URL can be configured using {tokens}.

Process

Step 1: Create a connection to a web service using a URL that includes tokens:

Example URL: https://my.api.com/metrics/?apiKey=111-222-333&periodStart={period_start}&periodEnd={period_end}

Notice the tokens {period_start} and {period_end}. 

Step 2. Create a flow that retrieves data from the web service, using the connection created in Step 1 as the source (FROM) connection.

Step 3. Set the values of the global variables (tokens) before the flow executes. You can do this using any of the following methods:

• Set global variables in a Script : Define and assign values to global variables using a script.
• Database Loop Parameters : Set variables based on the results of a database query.
• File Loop Parameters : Loop through the files matching the wildcard name and set variables to file name.
• Flow variables set at the flow level : Set specific variables within the flow configuration.
• Variables set when executing flow manually : When running a flow manually, variables can be entered directly before the flow starts.
• Variables set when configuring the schedule : When setting up a scheduled flow, variables can be defined as part of the schedule configuration.
• Variables set when configuring the flow to be executed by Integration Agent : If the flow is executed by an Integration Agent, variables can be set in the agent’s flow configuration.
• URL parameters in call-flow-by-name API :Pass variables through URL parameters when triggering a flow by name.
• Payload in call-flow-by-ID API : Provide variables via the payload when triggering a flow by ID.
• User-defined API URL parameters : Use parameters defined in your custom API endpoints.
• HTTP Preprocessor : Set variables dynamically using an HTTP preprocessor script.