Schedule and Run Flows in the Integration Agent

• Startup
• Business
• Enterprise
• On-Premise
• Add-on

Overview

The Integration Agent can run Flows created in the Etlworks Integrator. The Flows can be executed manually, scheduled to run automatically, and stopped.  

Process

Flows can be added, removed, and scheduled when the specific Integration Agent is running. The Agent will reconfigure itself on the next communication session between the Agent and the Etlworks Integrator.  The communication session is always initiated by the Agent as we assume that the Etlworks Integrator cannot access the Agent which is running behind the firewall. You can change the frequency at which the Agent and the Integrator communicate to each other by changing the configuration parameter Agent Ping Delay.  

Here's how you can add Flows to run in the remote Integration Agent:

Step 1. In Etlworks Integrator, open the Agents window and select the already created Integration Agent. 

Step 2. Add previously created Flow(s) to the Agent.

Step 3. Select one of the available schedule types.

Step 4. Configure the schedule if the Schedule type is not Manual . 

Execute flow every n seconds

Execute flow periodically using a cron-based schedule

Step 5. Optionally provide a description for the flow.

The description is an additional attribute available when adding flows to the Agent. It can be used as an informational field only but it is required if you wish to schedule the same flow multiple times with different flow variables. Read about parametrization. 

Step 6. Optionally add flow variables.

Click the Editor icon next to the flow status link.

then add flow variables as key-value pairs.

Read about parametrization.

Pass secure parameters such as passwords in theSecure parametersbox:

Filter flows by name, description, and tags

It is possible to filter flows, associated with the agent by

1. Tags.

2. Name and Description

Run the Flow manually

You can request to run any flow added to the Integrator Agent. If the Flow is not already running it will be executed on the next communication session between the Agent and the Etlworks Integrator. You can change the frequency at which the Agent and the Integrator communicate to each other by changing the configuration parameter Agent Ping Delay.

The Run button is disabled if the Agent is offline. 

Stop the running Flow

You can request to stop any running flow added to the Integrator Agent. If the Flow is running it will be stopped on the next communication session between the Agent and the Etlworks Integrator. You can change the frequency at which the Agent and the Integrator communicate to each other by changing the configuration parameter Agent Ping Delay.

The Stop button is disabled if the Agent is offline. 

Parametrization

It is possible to add flow variables to the flow executed by the Integration Agent. Flow variables can be referenced as {VARNAME} anywhere in the code.

Examples:

• Parameterization of the source and destination queries. 
• Parametrization of the command-line (shell) scripts.

Flow variables can be converted to global variables.

Read how to use global variables.

• Parameterization of the Connections 
• Parameterization of transformations 
• Parameterization of file operations

Read how to add flow variables to the flow executed by Integration Agent.

Monitor the Flow

Flow Statistics dashboard

Each time the Flow is executed by the Integration Agent (manually or automatically by the scheduler) it reports stats, metrics, and optionally console log for that specific run back to the Etlworks Integrator.

Unlike flows running in the cloud Etlworks Integrator, the tasks and the log for the flows executed by the Agent are not updated in real-time. 

To access the history of runs as well as starts, metrics, and log for each run click the status link next to the flow added to the Integration Agent.

The standard Etlworks Flow Statistics dashboard will be displayed.

Flow Execution Log

By default, the Agent does not upload the Flow execution log into the cloud Etlworks Integrator instance. Read how to enable log upload.

When the log is available it can be viewed and downloaded from the Flow Statistics dashboard.