- Startup
- Business
- Enterprise
- On-Premise
- Add-on
Overview
The Integration Agent can run Flows created in Etlworks. The Flows can be executed manually, scheduled to run automatically, and stopped.
Process
Flows can be added, removed, and scheduled when the specific Integration Agent is running. The communication session is always initiated by the Agent as we assume that Etlworks cannot access the Agent which is running behind the firewall.
Here's how you can add Flows to run in the remote Integration Agent:
Step 1. In Etlworks, open the Agents
window and select the already created Integration Agent.
Step 2. Add previously created Flow(s) to the Agent.
Step 3. Select one of the available schedule types.
Step 4. Configure the schedule if the Schedule type is not Manual
Execute flow every n seconds
Execute flow periodically using a cron-based schedule
Optionally configure automatic activation or deactivation of the schedule
If you want to auto activate flow schedule, make sure that flow is deactivated before saving (deselect checkbox on the left side of the flow). When setting "Deactivate on date" make sure that flow is active before saving.
Auto activation and deactivation of agent flow schedule happens on the server side, not the individual agent.
- The activation and deactivation occurs at midnight in the server’s timezone on the specified date via
Auto enable/disable schedules
maintenance task execution. - The server timezone is determined by the host operating system running Etlworks. On Linux systems, it is typically UTC unless manually changed.
- Maintenance task schedule can be adjusted in the maintenance task management UI, however date matching during the task execution will be calculated in the server timezone, make sure to adjust with time difference in mind.
Step 5. Optionally provide a description for the flow.
The description is an additional attribute available when adding flows to the Agent. It can be used as an informational field only but it is required if you wish to schedule the same flow multiple times with different flow variables. Read about parametrization.
Step 6. Optionally add flow parameters.
Click the Editor
icon next to the flow status
link.
then add the Flow parameters that will be passed to the Flow at run time. The parameters will be added as global and Flow variables.
Read about parametrization.
Entering Empty String Values
When entering parameters for a flow in the format key=value, the UI does not accept an empty string ("") as a valid value. To pass an empty string as the value of a specific parameter, you must enter it using the Unicode representation for an empty character: \u200B. This allows the system to recognize and process the parameter with an empty string as its value.
Entering Secure Parameters
Pass secure parameters such as passwords in theSecure parameters
box:
Step 7. Optionally configure email notifications.
When creating or modifying a Schedule, it is possible to configure the email notifications.
Click Email button
Configure Notifications
The following notifications are available:
Send email on success
This notification is sent when the Flow is successfully executed.
Step 1. Enable On success
.
Step 2. Use Emails
to configure the email address(es) that will be receiving a notification.
Send email on failure
This notification is sent when the Flow is either failed or canceled.
Step 1. Enable On failure
.
Step 2. Use Emails
to configure the email address(es) that will be receiving a notification.
Send an email if Flow execution is taking longer than expected
This notification is sent when the Flow execution is taking longer than expected. The notification will be sent one time when the actual run time exceeds the configured threshold.
Step 1. Use Expected flow execution time
to specify the expected Flow execution time in minutes.
Step 2. Use Emails
to configure the email address(es) that will be receiving a notification.
Send an email and stop the flow if Flow execution is taking longer than maximum allowed time in minutes
This notification is sent and the flow is stopped when the Flow execution is taking longer than maximum allowed time in minutes. The notification will be sent one time when the actual run time exceeds the configured threshold.
Step 1. Use Stop flow after time
to specify the maximum allowed Flow execution time in minutes.
Step 2. Use Emails
to configure the email address(es) that will be receiving a notification.
The stop-flow signal will be send to the agent but it is not guaranteed that the flow will be actually stopped. There are cases when the flow is stuck when executing not cancellable IO operation (for example, opening the database connection with no configured timeout). In this cases the flow will continue running until it is finally able to stop or the agent is rebooted.
Filter flows by name, description, and tags
It is possible to filter flows, associated with the agent by
1. Tags.
2. Name and Description
Run the Flow manually
You can request to run any flow added to the Etlworks Agent. If the Flow is not already running it will be started by the Agent.
The Run
button is disabled if the Agent is offline.
Stop the running Flow
You can request to stop any running flow added to the Etlworks Agent. If the Flow is running it will be stopped by the Agent.
The Stop
button is disabled if the Agent is offline.
The stop-flow signal will be send to the agent but it is not guaranteed that the flow will be actually stopped. There are cases when the flow is stuck when executing not cancellable IO operation (for example, opening the database connection with no configured timeout). In this cases the flow will continue running until it is finally able to stop or the agent is rebooted.
Parametrization
It is possible to add flow variables to the flow executed by the Integration Agent. Flow variables can be referenced as {VARNAME}
anywhere in the code.
Examples:
- Parameterization of the source and destination queries.
- Parametrization of the command-line (shell) scripts.
Flow variables can be converted to global variables.
Read how to use global variables.
- Parameterization of the Connections
- Parameterization of transformations
- Parameterization of file operations
Read how to add flow variables to the flow executed by Integration Agent.
Duplicating Flows Attached to Agents
The platform provides an easy way to duplicate existing flows that are attached to agents, allowing you to quickly replicate and reuse configurations without manually recreating them. As shown in the screenshot, the Duplicate button (represented by two overlapping squares) is available next to each flow.
Duplicate Flow: When clicked, the system creates a copy of the selected flow with all its parameters, schedules, and settings intact. This is particularly useful when you need to set up similar ETL flows or modify an existing flow for a new use case without starting from scratch. The duplicated flow can then be edited and customized independently of the original.
Monitor the Flow
Flow Statistics dashboard
Each time the Flow is executed by the Integration Agent (manually or automatically by the scheduler) it reports stats, metrics, and optionally console log for that specific run back to Etlworks.
Unlike flows running in the cloud Etlworks, the tasks and the log for the flows executed by the Agent are not updated in real-time.
To access the history of runs as well as starts, metrics, and log for each run click the status
link next to the flow added to the Integration Agent.
The standard Etlworks Flow Statistics
dashboard will be displayed.
Flow Execution Log
Read how to enable and configure log upload.
When the log is available it can be viewed and downloaded from the Flow Statistics
dashboard.
Comments
0 comments
Please sign in to leave a comment.