Articles in this section

See more

Workflow Editor: Advanced Workflow Techniques

  • Updated
Follow

Overview

The Workflow Editor supports advanced configuration patterns that allow workflows to be more dynamic and reusable.

These patterns allow workflows to:

  • use runtime variables dynamically 
  • reference connections 
  • execute flows dynamically 
  • control workflow execution programmatically

This article describes several advanced techniques that are commonly used when building flexible or metadata-driven workflows.

Flow Variables

Flow Variables are key-value pairs defined at the workflow level.

These variables can be referenced in multiple places inside the workflow and in child flows.

Flow variables are referenced using the syntax:

{VARIABLE_NAME}

Flow variables can be used in:

  • SQL queries 
  • transformations 
  • scripts

Using Flow Variables in SQL Queries

Flow variables can be used inside SQL queries.

select * from connection
where lower(name) like '%{NAME}%'

using-flow-vars.png

Using Flow Variables in Transformations

Flow variables can be referenced inside transformations.

{SOURCE_FILE_NAME}
{DESTINATION_FILE_NAME}

They can be used in:

  • source  
  • destination

This allows transformations to adapt dynamically based on workflow parameters.

Using Flow Variables in JavaScript

Flow variables can also be accessed from scripts executed within the workflow.

Examples:

// get value
var value = scenario.getVariable("unique key").getValue();

// set value
scenario.getVariable("unique key").setValue('value');

This allows runtime logic to read or modify parameters during workflow execution.

For more details about variables in Etlworks, see the dedicated documentation on variables.

Auto-Population for Parameter Keys and Values

To improve usability and maintain consistency, Etlworks supports auto-population for parameter keys and values.

This feature provides dropdown suggestions when entering parameters.

Suggestions are based on:

  • predefined parameters 
  • previously entered parameters 
  • global settings configured by a super administrator

This helps standardize parameter usage across workflows.

Learn more about configuring auto-suggestions for flow parameters

Named Connections

Nested workflows normally inherit connections from their inner flows.

However, additional Named Connections can be defined at the workflow level.

Named connections are configured in the Connections tab of the workflow.

Using Named Connections

Named connections can be used in scripts executed within child flows.

Example:

value = com.toolsverse.etl.core.task.common.FileManagerTask.delete(etlConfig, 'to_delete', '*.csv');

You can find more examples here: Using Java Packages and Classes in JavaScript

Other use cases include:

  • dynamically selecting database connections 
  • executing SQL against different systems 
  • performing workflow-level operations

Dynamic Flow

The Dynamic Flow feature allows you to execute any Flow by its name, determined at runtime. This is particularly useful for creating flexible and reusable workflows.

Create Dynamic flow

Step 1. In Flows window, clicking Add flow, and type in dynamic flow into the Select Flow Type box:

mceclip0.png

Step 2. Enter an actual flow name or a {VARIABLE_NAME} in the Flow Name field.

mceclip1.png

Step 3. Add Dynamic Flow as a step to any nested flow.

mceclip2.png

Use Dynamic Flow

Once the Dynamic Flow is created and added as a step to the nested flow it can be used by dynamically passing the value of the {VARIABLE_NAME}.

Below are some of the examples:

When executing the Flow manually

Step 1. Click Run flow button

mceclip3.png

Step 2. Click Add parameters link

mceclip4.png

Step 3. Add {VARIABLE_NAME} in KEY and Flow Name in VALUE

mceclip6.png

When scheduling the Flow

Open the schedule, add {VARIABLE_NAME} in KEY and Flow Name in VALUE in Flow patameters:

mceclip7.png

When using Remote Integration Agent

Step 1. Click Edit flow patameters

mceclip8.png

Step 2. Add {VARIABLE_NAME} in KEY and Flow Name in VALUE in Flow patameters:

mceclip9.png

Stop the flow programmatically without raising an exception

Problem

There are situations when you may want to stop the nested flow early based on conditions. For example, if there are no files to process. At this point, there could be multiple steps that have not been executed yet. You can add a condition to each step, but it requires a lot of manual labor.

Solution

Anywhere in the JavaScript or Python code, executed within the loop add the following line of code. This call is thread-safe.

etlConfig.setRequestStopFlow(true)

Example

if (dataSet.getRecordCount() == 0) {
   etlConfig.log("Empty dataset, hence exiting the flow");

   etlConfig.setRequestStopFlow(true);
}

Summary

Advanced workflow patterns allow workflows to become more dynamic and reusable.

Key techniques include:

  • Flow variables for parameterized workflows 
  • Named connections for dynamic connection usage 
  • Dynamic Flow for runtime flow selection 
  • Programmatic workflow termination

These patterns are commonly used in metadata-driven workflows and complex automation pipelines.

Related articles