Scheduling flows

Scheduling basics

In Integrator, you can schedule flows for automatic execution.

To schedule a flow, open the Scheduler window

...and click the Add Schedule button:

In this window you can also:

• Filter scheduled flows (events) by name and type.
• Select events for group operations.
• Disable events. Disabled events will not be automatically executed by the scheduler.
• See flow statistics.
• Manually execute selected flows.
• Edit events (change scheduling parameters).

Scheduling flows from Flows window

In the Flows window click View Schedules button.

Click Add Schedule button

If the flow has been already scheduled the icon changes from the grayed-out calendar to the calendar with a checkmark.

Scheduling periodically executed flows

Flows can be scheduled to be executed automatically at configurable time intervals. A scheduled flow is called an Event.

When scheduling events, the following parameters are available:

• Name - the name of the event.
• Flow - the flow associated with the event.
• Schedule - a cron-based or seconds-based time when the flow should be executed.
• Active - inactive flows will not be executed automatically.
• Send email on success - if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the flow.
• Send email on failure - if enabled the Scheduler will notify an Emails subscription list about each failed execution of the flow.
• Expected flow execution time - receive an email notification if the flow execution is taking longer than the specified number of minutes. 
• Emails - a space-separated list of email addresses that will be notified about success, failure, or if the flow execution is taking longer than expected.
• Retry on failure - if enabled and execution has failed, the Scheduler will retry the flow for the configured number of retries.
• Retries - the number of retries for failed execution attempts.
• Delay between retries - the delay between retries, in minutes.
• Record Console Log - if enabled, the DEBUG level log will be recorded for each flow execution. This option is enabled by default for the scheduled periodically executed flows.
• Extra Log Step - an extra checkpoint that will be added to the flow log every time the specified number of data records has been processed.  Use Extra Log Step if you want to monitor a flow that is processing a large number of data records, typically > 1M(one million). Otherwise, keep it empty. 
• Keep Empty Statistics - keep flow execution statistics even in case of successful execution with 0 records, or files processed. By default, this option is enabled. 

 

Examples of scheduling, using the cron pattern

Every day, every 4 hours

1. Select time period
2. Select Every day
3. Select 0, 8, 8, 12, 16, 20, 23 for hours and 0 for minutes

Every Monday, Wednesday, and Friday at 23:59

1. Select time period
2. Select Every week
3. Select Monday, Wednesday, and Friday for days and 23:59 for hours and minutes

Every day at 1:00 am

1. Select time period
2. Select Month
3. Select Every day of the month
4. Select 1:00 for hours and minutes

Every 20 minutes

1. Select time period
2. Select Every hour
3. Select 0,20,40,59

Scheduling flow to run continuously

In Etlworks Integrator, it is possible to schedule a flow to run continuously until there is nothing to do, then stop for a configurable number of seconds and restart.

To configure flow to run continuously:

Step 1. Set Schedule to seconds.

Step 2. Enter a number of seconds to delay the restart. The smallest wait time until a restart is 1 second.

Step 3. Enable Use as delay. If 'Use as delay' is disabled (it is enabled by default) the flow will attempt to restart every N seconds, where N is a number of seconds set in step 2.   The smallest time between attempts is 30 seconds.  

Step 4. Optionally enable Disable on failure. If this option is enabled and the flow execution fails the schedule will be automatically deactivated. The user will need to reactivate the schedule manually.

Scheduling real-time flows

Flows can be scheduled to be executed in real-time.

Real-time scheduling is only available for Enterprise and on-premise plans and must be manually enabled. Please contact support@etlworks.com for more information.

If real-time flow fails the schedule will be automatically deactivated. The user will need to reactivate the schedule manually.

To pause/resume the real-time flow, simply select it and toggle the Activate/Deactivate button.

When scheduling real-time events, the following parameters are available:

• Name - the name of the event.
• Flow - the flow associated with the event.
• Schedule - select Real-time.
• Active - inactive flows will not be executed automatically.
• Send email on success - if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the flow.
• Send email on failure - if enabled the Scheduler will notify an Emails subscription list about each failed execution of the flow.
• Expected flow execution time - receive an email notification if the flow execution is taking longer than the specified number of minutes. 
• Emails - a space-separated list of email addresses that will be notified about success, failure or if the flow execution is taking longer than expected.
• Keep Empty Statistics - keep flow execution statistics even in case of successful execution with 0 records, or files. By default, this option is disabled.

Scheduling event-driven flows

When scheduled, event-driven flows are executed when a certain event happens, for example, when a third-party application calls Integrator's HTTP endpoint, which is linked to an HTTP listener.

The event-driven flows are clearly marked in the Schedules grid:

When scheduling event-driven flows, the following parameters are available:

• Name - the name of the event.
• Flow - the flow associated with the event.
• Active - inactive flows will not be executed automatically.
• Send email on success - if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the flow.
• Send email on failure - if enabled, the Scheduler will notify an Emails subscription list about each failed execution of the flow.
• Expected flow execution time - receive an email notification if the flow execution is taking longer than the specified number of minutes. 
• Emails - a space-separated list of email addresses that will be notified about success, failure or if the flow execution is taking longer than expected.
• Retry on failure - if enabled and execution has failed, the Scheduler will retry the flow for the configured number of retries.
• Retries - the number of retries for failed execution attempts.
• Delay between retries - the delay between retries, in minutes.
• Record Console Log - if enabled, the DEBUG level log will be recorded for each flow execution. This option is disabled by default for the scheduled event-driven flows.
• Extra Log Step - an extra checkpoint that will be added to the flow log every time the specified number of data records is processed. Use Extra Log Step if you want to monitor a flow that is processing a large number of the data records, typically > 1M(one million). Otherwise, keep it empty.
• Keep Message - if enabled, the system will store bodies of the POST/PUT requests in the database. The stored messages can be viewed in the messages app. The option is disabled by default.
• Keep Empty Statistics - keep flow execution statistics even in case of successful execution with 0 records, or files. By default, this option is enabled.

After an event-driven flow has been scheduled, you will need to wait one minute before you can start using it. The reason: Listeners need to be activated, which usually takes ~ 1 minute.

Configuring Notifications

When creating or modifying a schedule, it is possible to configure notifications. The following notifications are available:

Send email on success

This notification is sent when the flow is successfully executed.

Step 1. Enable "Send email on success".

Step 2. Use the field "Email" to configure the email address(es) that will be receiving a notification.

Send email on failure

This notification is sent when the flow is either failed or canceled.

Step 1. Enable "Send email on failure".

Step 2. Use the field "Email" to configure the email address(es) that will be receiving a notification.

Send an email if flow execution is taking longer than expected

This notification is sent when the flow execution is taking longer than expected. The notification will be sent one time when the actual run time exceeds the configured threshold.

Step 1. Use the field "Expected flow execution time" to specify the expected flow execution time in minutes. 

Step 2. Use the field "Email" to configure the email address(es) that will be receiving a notification.

Configuring Auto-Retry on Failure

When creating or modifying a schedule, it is possible to configure auto-retries. When the auto-retry is enabled and the flow execution failed, the system will attempt to execute the flow again until the flow successfully executed or the configurable number of times, whichever comes first.

Step 1. Enable "Retry of failure". 

Step 2. Select the number of "Retries" (5 max).

Step 3. Select "Delay between retries" (1-10 minutes)

Configuring flow to stop automatically after a period of time

If the flow is running longer than expected it is possible to stop it automatically after a configurable period of time.  

Enter the number of minutes after which the flow should be stopped in the field Stop flow after time.

It is not always possible to stop the flow gracefully, specifically if it is stuck in performing the IO operation, such as connecting to the database.

Configuring the schedule to not run if other flows are running

It is possible to configure the schedule to not run if other flows are running.

Simply add flows to the No running flows list.

Configuring flow parameters

It is possible to configure the flow parameters that will be passed to the flow at run time. The parameters will be added as global and flow variables.

It is recommended to add parameters as UPPERCASED variables without whitespaces.