Schedule Flows

Basics of Scheduling

In the Etlworks Integrator, you can schedule Flows for automatic execution.

To schedule a Flow, open the Scheduler window.

Click Add schedule:

In this window, you can also:

• Filter scheduled Flows (events) by name and type.
• Select events for group operations.
• Disable events (the scheduler will not automatically execute disabled events).
• See Flow statistics.
• Manually execute selected Flows.
• Edit events (change scheduling parameters).

Schedule Flows from the Flows window

In the Flows window, click View schedules.

Click Add schedule.

If the Flow has been already scheduled, the icon changes from the grayed-out calendar to the calendar with a checkmark.

Schedule periodically executed Flows

Flows can be scheduled to be executed automatically at configurable time intervals. A scheduled Flow is called an event.

When scheduling events, the following parameters are available:

• Name: the name of the event.
• Flow: the Flow associated with the event.
• Schedule: a cron-based or seconds-based time when the Flow should be executed.
• Active: inactive Flows will not be executed automatically.
• Send email on success: if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the Flow.
• Send email on failure: if enabled, the Scheduler will notify an Emails subscription list about each failed execution of the Flow.
• Expected Flow execution time: receive an email notification if the Flow execution takes longer than the specified number of minutes. 
• Emails: a space-separated list of email addresses that will be notified about success, failure, or delays in Flow execution.
• Retry on failure: if enabled and execution has failed, the Scheduler will retry the Flow for the configured number of retries.
• Retries: the number of retries for failed execution attempts.
• Delay between retries: the delay between retries, in minutes.
• Record Console Log: if enabled, the debug level log will be recorded for each Flow execution. This option is enabled by default for the scheduled periodically executed Flows.
• Extra Log Step: an extra checkpoint that is added to the Flow log every time the specified number of data records has been processed.  Use this if you want to monitor a Flow that is processing a large number of data records, typically > 1M(one million). Otherwise, keep it empty. 
• Keep Empty Statistics: keep Flow execution statistics even in successful execution with 0 records or files processed. By default, this option is enabled. 

Examples of scheduling using the cron pattern

Every day, every four hours

1. Select time period.
2. Select Every day.
3. Select 0, 8, 8, 12, 16, 20, 23 for hours and 0 for minutes.

Every Monday, Wednesday, and Friday at 23:59

1. Select time period.
2. Select Every week.
3. Select Monday, Wednesday, and Friday for days and 23:59 for hours and minutes.

Every day at 1:00 am

1. Select time period.
2. Select Month.
3. Select Every day of the month.
4. Select 1:00 for hours and minutes.

Every 20 minutes

1. Select time period.
2. Select Every hour.
3. Select 0,20,40,59.

Schedule Flow to run continuously

In the Etlworks Integrator, it is possible to schedule a Flow to run continuously until there is nothing to do, then stop for a configurable number of seconds and restart.

To configure Flow to run continuously:

Step 1. Set Schedule to seconds.

Step 2. Enter a number of seconds to delay the restart. The smallest wait time until a restart is one second.

Step 3. Enable Use as delay. If this is disabled (it is enabled by default), the Flow will attempt to restart every N seconds, where N is a number of seconds set in step two. The smallest time between attempts is 2 seconds.  

Step 4. Optionally enable Disable on failure. If this option is enabled and the Flow execution fails, the Schedule will be automatically deactivated. You will need to reactivate the Schedule manually.

Schedule real-time Flows

• Starter
• Business
• Enterprise
• On-Premise
• Add-on

Flows can be scheduled to be executed in real-time.

Real-time scheduling is only available for Enterprise and on-premise plans and must be manually enabled. Please contact support@etlworks.com for more information.

If real-time Flow fails, the schedule will be automatically deactivated. You will need to reactivate the schedule manually.

To pause/resume the real-time Flow, simply select it and toggle Activate/Deactivate.

When scheduling real-time events, the following parameters are available:

• Name: the name of the event.
• Flow: the Flow associated with the event.
• Schedule: select Real-time.
• Active: inactive Flows will not be executed automatically.
• Send email on success: if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the Flow.
• Send email on failure: if enabled, the Scheduler will notify an Emails subscription list about each failed execution of the Flow.
• Expected Flow execution time: receive an email notification if the Flow execution is taking longer than the specified number of minutes. 
• Emails: a space-separated list of email addresses that will be notified about success, failure, or if the flow execution is taking longer than expected.
• Keep Empty Statistics: keep the Flow execution statistics even in successful execution with 0 records or files. By default, this option is disabled.

Schedule event-driven Flows

When scheduled, event-driven Flows are executed when a certain event happens, for example, when a third-party application calls the Etlworks Integrator's HTTP endpoint, which is linked to an HTTP Listener.

The event-driven Flows are clearly marked in the Schedules grid:

When scheduling event-driven Flows, the following parameters are available:

• Name: the name of the event.
• Flow: the Flow associated with the event.
• Active: inactive Flows will not be executed automatically.
• Send email on success: if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the Flow.
• Send email on failure: if enabled, the Scheduler will notify an Emails subscription list about each failed execution of the Flow.
• Expected flow execution time: receive an email notification if the Flow execution is taking longer than the specified number of minutes. 
• Emails: a space-separated list of email addresses that will be notified about success, failure, or if the Flow execution is taking longer than expected.
• Retry on failure: if enabled and execution has failed, the Scheduler will retry the Flow for the configured number of retries.
• Retries: the number of retries for failed execution attempts.
• Delay between retries: the delay between retries, in minutes.
• Record Console Log: if enabled, the debug level log will be recorded for each Flow execution. This option is disabled by default for the scheduled event-driven Flows.
• Extra Log Step: an extra checkpoint that will be added to the Flow log every time the specified number of data records is processed. Use this if you want to monitor a Flow that is processing a large number of the data records, typically > 1M(one million). Otherwise, keep it empty.
• Keep Message: if enabled, the system will store the bodies of the Post/Put requests in the database. The stored messages can be viewed in the messages app. The option is disabled by default.
• Keep Empty Statistics: keep Flow execution statistics even in successful execution with 0 records or files. By default, this option is enabled.

After an event-driven Flow has been scheduled, you will need to wait one minute before starting to use it. The reason: Listeners need to be activated, which usually takes 1 minute.

Configure Notifications

When creating or modifying a Schedule, it is possible to configure the notifications. The following notifications are available:

Send email on success

This notification is sent when the Flow is successfully executed.

Step 1. Enable Send email on success.

Step 2. Use Email to configure the email address(es) that will be receiving a notification.

Send email on failure

This notification is sent when the Flow is either failed or canceled.

Step 1. Enable Send email on failure.

Step 2. Use Email to configure the email address(es) that will be receiving a notification.

Send an email if Flow execution is taking longer than expected

This notification is sent when the Flow execution is taking longer than expected. The notification will be sent one time when the actual run time exceeds the configured threshold.

Step 1. Use Expected flow execution time to specify the expected Flow execution time in minutes. 

Step 2. Use Email to configure the email address(es) that will be receiving a notification.

Send an email if the schedule was automatically disabled due to the error

If the Flow is scheduled to run continuously or in real-time, and the schedule is configured to Disable on failure and Email address(es) are configured the notification will be sent one time when the scheduler will automatically disable the schedule. 

Configure Auto-Retry on Failure

When creating or modifying a Schedule, it is possible to configure auto-retries. When the auto-retry is enabled, and the Flow execution failed, the system will attempt to execute the Flow again until the Flow is successfully executed or the configurable number of times, whichever comes first.

Step 1. Enable Retry of failure. 

Step 2. Select the number of retries (5 is the maximum).

Step 3. Select Delay between retries (1-10 minutes).

Configure a Flow to stop automatically after a period of time

If the Flow is running longer than expected, it is possible to stop it after a configurable time automatically.  

Enter the number of minutes. After which, the Flow should be stopped in the field Stop flow after time.

The stop signal will be send to the flow but it is not guaranteed that the flow will be actually stopped. There are cases when the flow is stuck when executing not cancellable IO operation (for example, opening the  database connection with no configured timeout). In this cases the flow will continue running until it is finally able to stop or the Etlworks instance is rebooted.

Read about force execution.

Configure the Schedule to stop running if other Flows are running

It is possible to configure the schedule to stop running if other Flows are running.

Simply add Flows to the No running flows list.

Configure Flow parameters

It is possible to configure the Flow parameters that will be passed to the Flow at run time. The parameters will be added as global and Flow variables.

It is recommended to add parameters as uppercased variables without whitespaces.

Pass secure parameters such as passwords in theSecure parametersbox:

Force execution

In Etlworks the flow can be manually stopped by user or cancelled by scheduler when it is running longer than expected. It is not 100% guaranteed that the flow will be actually completely stopped. There could be long running blocking IO operations, such as executing the SQL query, which could be still running hence preventing the flow from completely stopping.

When the flow is stopped by user or cancelled it is marked as canceled by the system:

When it happens the ETL engine sets a flag which indicates that the flow might still be running. The flag is cleared when the flow actually stops or after 24 hours, whatever comes first.

This flag prevents the flow from being executed manually or by scheduler. It is possible to ignore the flag by enabling Force execution. Note that by default Force execution is disabled for new schedules.