Scheduling flows

Scheduling basics

In Integrator, you can schedule flows for automatic execution.

To schedule a flow, open the Scheduler window

...and click the Add Schedule button:

In this window you can also:

Filter scheduled flows (events) by name and type.
Select events for group operations.
Disable events. Disabled events will not be automatically executed by the scheduler.
See flow statistics.
Manually execute selected flows.
Edit events (change scheduling parameters).

Scheduling flows from Flows window

In the Flows window click View Schedules button.

Click Add Schedule button

If the flow has been already scheduled the icon changes from the grayed-out calendar to the calendar with a check mark.

Scheduling periodically executed flows

Flows can be scheduled to be executed automatically at configurable time intervals. A scheduled flow is called an Event.

When scheduling events, the following parameters are available:

Name - the name of the event.
Flow - the flow associated with the event.
Schedule - a cron-based or seconds-based time when the flow should be executed.
Active - inactive flows will not be executed automatically.
Send email on success - if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the flow.
Send email on failure - if enabled the Scheduler will notify an Emails subscription list about each failed execution of the flow.
Expected flow execution time - receive an email notification if the flow execution is taking longer than the specified number of minutes.
Emails - a space-separated list of email addresses that will be notified about success, failure or if the flow execution is taking longer than expected.
Retry on failure - if enabled and execution has failed, the Scheduler will retry the flow for the configured number of retries.
Retries - the number of retries for failed execution attempts.
Delay between retries - the delay between retries, in minutes.
Record Console Log - if enabled, the DEBUG level log will be recorded for each flow execution. This option is enabled by default for the scheduled periodically executed flows.
Extra Log Step - an extra checkpoint which will be added to the flow log every time the specified number of data records has been processed. While expanding the verbosity of the log, it can also negatively impact performance. Use Extra Log Step if you want to monitor a flow which is processing a large number of data records, typically > 1M(one million). Otherwise, keep it empty.
Keep Empty Statistics - keep flow execution statistics even in case of successful execution with 0 records, or files. By default, this option is enabled.

Examples of scheduling, using the cron pattern

Every day, every 4 hours

Select time period
Select Every day
Select 0, 8, 8, 12, 16, 20, 23 for hours and 0 for minutes

Every Monday, Wednesday and Friday at 23:59

Select time period
Select Every week
Select Monday, Wednesday and Friday for days and 23:59 for hours and minutes

Every day at 1:00 am

Select time period
Select Month
Select Every day of the month
Select 1:00 for hours and minutes

Every 20 minutes

Select time period
Select Every hour
Select 0,20,40,59

Scheduling flow to run continuously

In Etlworks Integrator, it is possible to schedule a flow to run continuously until there is nothing to do, then stop for a configurable number of seconds and restarts.

It is most useful when extracting data from the streaming data sources.

Examples:

To configure flow to run continuously with intervals:

Step 1. Set Schedule to seconds.

Step 2. Enter number of seconds to delay the restart.

Step 3. Enable Use as delay

Step 4. Optionally enable Disable on failure. If this option is enabled the flow execution fails it will be automatically deactivated. The user will need to reactivate the flow manually.

Scheduling real-time flows

Flows can be scheduled to be executed in real-time.

Real-time scheduling is only available for Enterprise and on-premise plans.

If real-time flow fails it will be automatically deactivated. The user will need to reactivate the flow manually.

To pause/resume the real-time flow, simply select it and toggle the Activate/Deactivate button.

When scheduling real-time events, the following parameters are available:

Name - the name of the event.
Flow - the flow associated with the event.
Schedule - select Real-time.
Active - inactive flows will not be executed automatically.
Send email on success - if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the flow.
Send email on failure - if enabled the Scheduler will notify an Emails subscription list about each failed execution of the flow.
Expected flow execution time - receive an email notification if the flow execution is taking longer than the specified number of minutes.
Emails - a space-separated list of email addresses that will be notified about success, failure or if the flow execution is taking longer than expected.
Keep Empty Statistics - keep flow execution statistics even in case of successful execution with 0 records, or files. By default, this option is disabled.

Scheduling event-driven flows

When scheduled, event-driven flows are executed "on event," which typically means when the payload arrives. For example, when a third-party application calls Integrator's HTTP endpoint, which is linked to an HTTP listener.

The event-driven flows are clearly marked in the Schedules grid:

When scheduling event-driven flows, the following parameters are available:

Name - the name of the event.
Flow - the flow associated with the event.
Active - inactive flows will not be executed automatically.
Send email on success - if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the flow.
Send email on failure - if enabled, the Scheduler will notify an Emails subscription list about each failed execution of the flow.
Expected flow execution time - receive an email notification if the flow execution is taking longer than the specified number of minutes.
Emails - a space-separated list of email addresses that will be notified about success, failure or if the flow execution is taking longer than expected.
Retry on failure - if enabled and execution has failed, the Scheduler will retry the flow for the configured number of retries.
Retries - the number of retries for failed execution attempts.
Delay between retries - the delay between retries, in minutes.
Record Console Log - if enabled, the DEBUG level log will be recorded for each flow execution. This options is disabled by default for the scheduled event-driven flows.
Extra Log Step - an extra checkpoint which will be added to the flow log every time the specified number of data records is processed. While expanding the verbosity of the log, it can also negatively impact performance. Use Extra Log Step if you want to monitor a flow which is processing a large number of the data records, typically > 1M(one million). Otherwise, keep it empty.
Keep Message - if enabled, system will store bodies of the POST/PUT requests in the database. The stored messages can be viewed in the messages app. The option is disabled by default.
Keep Empty Statistics - keep flow execution statistics even in case of successful execution with 0 records, or files. By default, this option is enabled.

After an event-driven flow has been scheduled, you will need to wait one minute before you can start using it. The reason: Listeners need to be activated, which usually takes ~ 1 minute.

Configuring Notifications

When creating or modifying a schedule, it is possible to configure notifications. The following notifications are available:

Send email on success

This notification is sent when the flow is successfully executed.

Step 1. Enable "Send email on success".

Step 2. Use the field "Email" to configure email address(es) which will be receiving a notification.

Send email on failure

This notification is sent when the flow is either failed or canceled.

Step 1. Enable "Send email on failure".

Step 2. Use the field "Email" to configure email address(es) which will be receiving a notification.

Send email if flow execution is taking longer than expected

This notification is sent when the flow execution is taking longer than expected. The notification will be sent one time when the actual run time exceeds the configured threshold.

Step 1. Use the field "Expected flow execution time" to specify the expected flow execution time in minutes.

Step 2. Use the field "Email" to configure email address(es) which will be receiving a notification.

Configuring Auto-Retry on Failure

When creating or modifying a schedule, it is possible to configure auto-retries. When the auto-retry is enabled and the flow execution failed, the system will attempt to execute the flow again until flow successfully executed or the configurable number of times, whichever comes first.

Step 1. Enable "Retry of failure".

Step 2. Select the number of "Retries" (5 max).

Step 3. Select "Delay between retries" (1-10 minutes)

Configuring flow to stop automatically after a period of time

If the flow is running longer than expected it is possible to stop it automatically after a configurable period of time.

Enter the number of minutes after which the flow should be stopped in the field Stop flow after time.

It is not always possible to stop the flow gracefully, specifically if it is stuck in performing the IO operation, such as connecting to the database.

Configuring the schedule to not run if other flows are running

It is possible to configure the schedule to not run if other flows are running.

Simply add flows to the No running flows list.

Configuring flow parameters

It is possible to configure the flow parameters that will be passed to the flow at run time. The parameters will be added as global and flow variables.

It is recommended to add parameters as UPPERCASED variables without whitespaces.

Articles in this section

Scheduling basics

Scheduling flows from Flows window

Scheduling periodically executed flows