Schedule Flows

Overview

In Etlworks, you can schedule Flows for automatic execution.

To schedule a Flow, open the Scheduler window.

Click Add schedule:

You can also:

• Filter scheduled Flows (events) by name and type.
• Select events for group operations.
• Disable events (the scheduler will not automatically execute disabled events).
• See Flow statistics.
• Manually execute selected Flows.
• Edit events (change scheduling parameters).

Schedule Flows from the Flows window

In the Flows window, click View schedules.

Click Add schedule.

If the Flow has already been scheduled, the icon changes from the grayed-out calendar to the calendar with a checkmark.

Schedule periodically executed Flows using cron expressions

Flows can be scheduled to be executed automatically at configurable time intervals using cron expressions. A scheduled Flow is called an event.

When scheduling events, the following parameters are available:

• Name: the name of the event.
• Flow: the Flow associated with the event.
• Schedule: a cron-based or seconds-based time when the Flow should be executed.
• Active: inactive Flows will not be executed automatically.
• Send email on success: if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the Flow.
• Send email on failure: if enabled, the Scheduler will notify an Emails subscription list about each failed execution of the Flow.
• Expected Flow execution time: receive an email notification if the Flow execution takes longer than the specified number of minutes. 
• Emails: a space-separated list of email addresses that will be notified about success, failure, or delays in Flow execution.
• Retry on failure: if enabled and execution has failed, the Scheduler will retry the Flow for the configured number of retries.
• Retries: the number of retries for failed execution attempts.
• Delay between retries: the delay between retries, in minutes.
• Record Console Log: if enabled, the debug level log will be recorded for each Flow execution. This option is enabled by default for the scheduled periodically executed Flows.
• Extra Log Step: an extra checkpoint that is added to the Flow log every time the specified number of data records has been processed.  Use this if you want to monitor a Flow that is processing a large number of data records, typically > 1M(one million). Otherwise, keep it empty. 
• Keep Empty Statistics: keep Flow execution statistics even in successful execution with 0 records or files processed. By default, this option is enabled. 

Examples of scheduling using the cron pattern

Every day, every four hours

1. Select time period.
2. Select Every day.
3. Select 0, 8, 8, 12, 16, 20, 23 for hours and 0 for minutes.

Every Monday, Wednesday, and Friday at 23:59

1. Select time period.
2. Select Every week.
3. Select Monday, Wednesday, and Friday for days and 23:59 for hours and minutes.

Every day at 1:00 am

1. Select time period.
2. Select Month.
3. Select Every day of the month.
4. Select 1:00 for hours and minutes.

Every 20 minutes

1. Select time period.
2. Select Every hour.
3. Select 0,20,40,59.

Schedule Flow to run continuously

In Etlworks, it is possible to schedule a Flow to run, then when it stops, wait a configurable number of seconds and restart. We recommend this type of schedule for streaming flows, such as "stream CDC events" or "stream messages from the queue."

To configure Flow to run continuously:

Step 1. Set Schedule to seconds.

Step 2. Enter a number of seconds to delay the restart. The smallest wait time until a restart is one second.

Step 3. Enable Use as delay. If this is disabled (it is enabled by default), the Flow will attempt to restart every N seconds, where N is a number of seconds set in step two. The smallest time between attempts is 2 seconds.  

Step 4. Optionally enable Disable on failure. If this option is enabled and the Flow execution fails, the Schedule will be automatically deactivated. You will need to reactivate the Schedule manually.

Schedule real-time Flows

• Starter
• Business
• Enterprise
• On-Premise
• Add-on

Flows can be scheduled to be executed in real-time.

Real-time scheduling is only available for Enterprise and on-premise plans and must be manually enabled. Please contact support@etlworks.com for more information.

If real-time Flow fails, the schedule will be automatically deactivated. You will need to reactivate the schedule manually.

To pause/resume the real-time Flow, simply select it and toggle Activate/Deactivate.

When scheduling real-time events, the following parameters are available:

• Name: the name of the event.
• Flow: the Flow associated with the event.
• Schedule: select Real-time.
• Active: inactive Flows will not be executed automatically.
• Send email on success: if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the Flow.
• Send email on failure: if enabled, the Scheduler will notify an Emails subscription list about each failed execution of the Flow.
• Expected Flow execution time: receive an email notification if the Flow execution is taking longer than the specified number of minutes. 
• Emails: a space-separated list of email addresses that will be notified about success, failure, or if the flow execution is taking longer than expected.
• Keep Empty Statistics: keep the Flow execution statistics even in successful execution with 0 records or files. By default, this option is disabled.

Schedule event-driven Flows

When scheduled, event-driven Flows are executed when a certain event happens, for example, when a third-party application calls Etlworks HTTP endpoint linked to an HTTP Listener.

The event-driven Flows are clearly marked in the Schedules grid:

When scheduling event-driven Flows, the following parameters are available:

• Name: the name of the event.
• Flow: the Flow associated with the event.
• Active: inactive Flows will not be executed automatically.
• Send email on success: if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the Flow.
• Send email on failure: if enabled, the Scheduler will notify an Emails subscription list about each failed execution of the Flow.
• Expected flow execution time: receive an email notification if the Flow execution is taking longer than the specified number of minutes. 
• Emails: a space-separated list of email addresses that will be notified about success, failure, or if the Flow execution is taking longer than expected.
• Retry on failure: if enabled and execution has failed, the Scheduler will retry the Flow for the configured number of retries.
• Retries: the number of retries for failed execution attempts.
• Delay between retries: the delay between retries, in minutes.
• Record Console Log: if enabled, the debug level log will be recorded for each Flow execution. This option is disabled by default for the scheduled event-driven Flows.
• Extra Log Step: an extra checkpoint that will be added to the Flow log every time the specified number of data records is processed. Use this if you want to monitor a Flow that is processing a large number of the data records, typically > 1M(one million). Otherwise, keep it empty.
• Keep Message: if enabled, the system will store the bodies of the Post/Put requests in the database. The stored messages can be viewed in the messages app. The option is disabled by default.
• Keep Empty Statistics: keep Flow execution statistics even in successful execution with 0 records or files. By default, this option is enabled.

After an event-driven Flow has been scheduled, you will need to wait one minute before starting to use it. The reason: Listeners need to be activated, which usually takes 1 minute.

Configure Exclusions

Cron-based schedules and schedules for continuously running flows can be configured to exclude specific weeks of the month and days of the weeks. 

If scheduled flow execution falls on any of the selected exclusion time slots, it will skip that execution. Note that next execution time shown in UI will not take exclusions into consideration and may show next execution that in reality will be skipped.

Navigate to Schedule Exclude and select exclusions from the list. It is possible to select multiple exclusions. To remove the exclusion click the [x] button.  

Configure Notifications

When creating or modifying a Schedule, it is possible to configure the notifications. The following notifications are available:

Send email on success

This notification is sent when the Flow is successfully executed.

Step 1. Enable Send email on success.

Step 2. Use Email to configure the email address(es) that will be receiving a notification.

Send email on failure

This notification is sent when the Flow is either failed or canceled.

Step 1. Enable Send email on failure.

Step 2. Use Email to configure the email address(es) that will be receiving a notification.

Send an email if Flow execution is taking longer than expected

This notification is sent when the Flow execution is taking longer than expected. The notification will be sent one time when the actual run time exceeds the configured threshold.

Step 1. Use Expected flow execution time to specify the expected Flow execution time in minutes. 

Step 2. Use Email to configure the email address(es) that will be receiving a notification.

Send an email if the schedule was automatically disabled due to the error

If the Flow is scheduled to run continuously or in real-time, and the schedule is configured to Disable on failure and Email address(es) are configured, the notification will be sent one time when the scheduler will automatically disable the schedule. 

Configure Auto-Retry on Failure

When creating or modifying a Schedule, it is possible to configure auto-retries. When the auto-retry is enabled and the Flow execution fails, the scheduler will attempt to execute the Flow again until the Flow is successfully executed or the configurable number of times, whichever comes first.

Step 1. Enable Retry of failure. 

Step 2. Select the number of retries (5 is the maximum).

Step 3. Select Delay between retries (1-10 minutes).

Configure a Flow to stop automatically after a period of time

If the Flow is running longer than expected, it is possible to stop it after a configurable time automatically.  

Enter the number of minutes. After which, the Flow should be stopped in the field Stop flow after time.

The stop signal will be send to the flow but it is not guaranteed that the flow will be actually stopped. There are cases when the flow is stuck when executing not cancellable IO operation (for example, opening the  database connection with no configured timeout). In this case, the flow will continue running until it is finally able to stop or the Etlworks instance is rebooted.

Read about force execution.

Configure the Schedule to stop running if other Flows are running

It is possible to configure the schedule to stop running if other Flows are running.

Simply add Flows to the No running flows list.

Configure the Schedule to automatically activate/deactivate on a specific date

In Etlworks, you can set a fixed future date for a schedule to automatically activate or deactivate. Once deactivated, the schedule will be disabled and will not trigger the flow execution until it is reactivated manually, via an API call or "Activate on date" setting.

Steps to Configure Activation:

1. Select the schedule you want to configure.

2. Set the Activate on date to any future date.

3. Make sure schedule is not Active before saving. Setting "Activate on date" on already active schedule will have no effect.

Steps to Configure Deactivation:

1. Select the schedule you want to configure.

2. Set the Deactivate on date to any future date.

3. Make sure schedule is Active before saving. Setting "Deactivate on date" on already deactivated schedule will have no effect.

Key Points:

• The activation and deactivation occurs at midnight in the server’s timezone on the specified date via Auto enable/disable schedules maintenance task execution.

• The server timezone is determined by the host operating system running Etlworks. On Linux systems, it is typically UTC unless manually changed.

• Maintenance task schedule can be adjusted in the maintenance task management UI, however date matching during the task execution will be calculated in the server timezone, make sure to adjust with time difference in mind.

Configure Flow parameters

It is possible to configure the Flow parameters that will be passed to the Flow at run time. The parameters will be added as global and flow variables.

It is recommended to add parameters as uppercased variables without whitespaces.

Entering Empty String Values

When entering parameters for a flow in the format key=value, the UI does not accept an empty string ("") as a valid value. To pass an empty string as the value of a specific parameter, you must enter it using the Unicode representation for an empty character: \u200B. This allows the system to recognize and process the parameter with an empty string as its value.

Entering Secure Parameters

Pass secure parameters such as passwords in theSecure parametersbox:

Force execution

In Etlworks the flow can be manually stopped by the user or canceled by scheduler when it is running longer than expected.

It is not 100% guaranteed that the flow will be actually completely stopped. There could be long-running blocking IO operations, such as executing the SQL query, which could still be running, hence preventing the flow from completely stopping.

When the flow is stopped by the user or canceled it is marked as canceled by the system:

When it happens, the ETL engine sets a flag that indicates that the flow might still be running. The flag is cleared when the flow actually stops or after 24 hours, whatever comes first.

This flag prevents the flow from being executed manually or by the scheduler. It is possible to ignore the flag by enabling Force execution. Note that by default, Force execution is disabled for new schedules. 

Scheduler Timezone 

• Schedules are always set in server timezone. The server timezone is the timezone set for the host OS running Etlworks. On Linux it is UTC in most cases (unless manually changed). It is recommend to use the default server timezone.
• The message in UI tells user what time that schedule will run in their local timezone (browser timezone) for convenience purposes, so that they don't have to calculate it in their head. They can adjust schedule so that it matches desired time in their timezone.

• Settings -> Timezone -> Administration Timezone sets Super Admin level timezone that is used when calculating and aggregating statistics for the Statistics screen. It is used the same way as Timezone set in the tenant creation/edit screen.

Duplicate schedule

To duplicate the schedule (create a copy of the schedule) navigate the to schedule that you want to duplicate

and click the Duplicate button at the bottom of the screen. 

It will create a copy of the schedule, with all the settings and parameters. To save the copy click the Save button.