Overview
In Etlworks, the FROM (source) and TO (destination) fields in a source-to-destination transformation can be dynamically configured using variables. This allows for flexible and reusable transformations by replacing static values with dynamic ones during runtime.
How it works
Simply enter the {variable name} in any FROM or TO field. During execution, the system will automatically replace the {variable_name} with the value of the corresponding global variable, making your transformations adaptable to different contexts or datasets.
Use one of the following methods to set global variables:
- Set global variables in a Script : Define and assign values to global variables using a script.
- Database Loop Parameters : Set variables based on the results of a database query.
- File Loop Parameters : Loop through the files matching the wildcard name and set variables to file name.
- Flow variables set at the flow level : Set specific variables within the flow configuration.
- Variables set when executing flow manually : When running a flow manually, variables can be entered directly before the flow starts.
- Variables set when configuring the schedule : When setting up a scheduled flow, variables can be defined as part of the schedule configuration.
- Variables set when configuring the flow to be executed by Integration Agent : If the flow is executed by an Integration Agent, variables can be set in the agent’s flow configuration.
- URL parameters in call-flow-by-name API :Pass variables through URL parameters when triggering a flow by name.
- Payload in call-flow-by-ID API : Provide variables via the payload when triggering a flow by ID.
- User-defined API URL parameters : Use parameters defined in your custom API endpoints.
- HTTP Preprocessor : Set variables dynamically using an HTTP preprocessor script.
Example
A real-life example would be transforming a response from the web service to the file with a new name calculated as constant_{current_timestamp}.json. The {current_timestamp} is a global variable set using a JavaScript.