Scheduling flows

Scheduling basics

In Integrator, you can schedule flows for automatic execution.

To schedule a flow, open the Scheduler window

...and click the Schedule button:

In this window you can also:

• Filter scheduled flows (events) by name and type.
• Select events for group operations.
• Disable events. Disabled events will not be automatically executed by the scheduler.
• See flow statistics.
• Manually execute selected flows.
• Edit events (change scheduling parameters).

Scheduling periodically executed flows

Flows can be scheduled to be executed automatically at configurable time intervals. A scheduled flow is called an Event.

When scheduling events, the following parameters are available:

• Name - the name of the event.
• Flow - the flow associated with the event.
• Schedule - a cron-based or seconds-based time when the flow should be executed.
• Active - inactive flows will not be executed automatically.
• Send email on success - if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the flow.
• Send email on failure - if enabled the Scheduler will notify an Emails subscription list about each failed execution of the flow.
• Expected flow execution time - receive an email notification if the flow execution is taking longer than the specified number of minutes. 
• Emails - a space-separated list of email addresses that will be notified about success, failure or if the flow execution is taking longer than expected.
• Retry on failure - if enabled and execution has failed, the Scheduler will retry the flow for the configured number of retries.
• Retries - the number of retries for failed execution attempts.
• Delay between retries - the delay between retries, in minutes.
• Record Console Log - if enabled, the DEBUG level log will be recorded for each flow execution. This option is enabled by default for the scheduled periodically executed flows.
• Extra Log Step - an extra checkpoint which will be added to the flow log every time the specified number of data records has been processed. While expanding the verbosity of the log, it can also negatively impact performance. Use Extra Log Step if you want to monitor a flow which is processing a large number of data records, typically > 1M(one million). Otherwise, keep it empty.
• Keep Empty Statistics - keep flow execution statistics even in case of successful execution with 0 records, or files. By default, this option is enabled.

Scheduling flow to run continuously

In Etlworks Integrator, it is possible to schedule a flow to run continuously until there is nothing to do, then stop for a configurable number of seconds and restarts.

It is most useful when extracting data from the streaming data sources.

Examples:

• Extracting data from CDC-enabled databases
• Real time data replication using Kafka and Debezium

To configure flow to run continuously with intervals:

Step 1. Set Schedule to seconds.

Step 2. Enter number of seconds to delay the restart.

Step 3. Enable Use as delay

Step 4. Optionally enable Disable on failure. If this option is enabled the flow execution fails it will be automatically deactivated. The user will need to reactivate the flow manually.

Examples of scheduling, using the cron pattern

Every day, every 4 hours

1. Select time period
2. Select Every day
3. Select 0, 8, 8, 12, 16, 20, 23 for hours and 0 for minutes

Every Monday, Wednesday and Friday at 23:59

1. Select time period
2. Select Every week
3. Select Monday, Wednesday and Friday for days and 23:59 for hours and minutes

Every day at 1:00 am

1. Select time period
2. Select Month
3. Select Every day of the month
4. Select 1:00 for hours and minutes

Every 20 minutes

1. Select time period
2. Select Every hour
3. Select 0,20,40,59

Scheduling real-time flows

Flows can be scheduled to be executed in real-time.

Real-time scheduling is only available for Enterprise and on-premise plans.

If real-time flow fails it will be automatically deactivated. The user will need to reactivate the flow manually.

To pause/resume the real-time flow, simply select it and toggle the Activate/Deactivate button.

When scheduling real-time events, the following parameters are available:

• Name - the name of the event.
• Flow - the flow associated with the event.
• Schedule - select Real-time.
• Active - inactive flows will not be executed automatically.
• Send email on success - if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the flow.
• Send email on failure - if enabled the Scheduler will notify an Emails subscription list about each failed execution of the flow.
• Expected flow execution time - receive an email notification if the flow execution is taking longer than the specified number of minutes. 
• Emails - a space-separated list of email addresses that will be notified about success, failure or if the flow execution is taking longer than expected.
• Keep Empty Statistics - keep flow execution statistics even in case of successful execution with 0 records, or files. By default, this option is disabled.

Scheduling event-driven flows

When scheduled, event-driven flows are executed "on event," which typically means when the payload arrives. For example, when a third-party application calls Integrator's HTTP endpoint, which is linked to an HTTP listener.

When scheduling event-driven flows, the following parameters are available:

• Name - the name of the event.
• Flow - the flow associated with the event.
• Active - inactive flows will not be executed automatically.
• Send email on success - if enabled, the Scheduler will notify an Emails subscription list about each successful execution of the flow.
• Send email on failure - if enabled, the Scheduler will notify an Emails subscription list about each failed execution of the flow.
• Expected flow execution time - receive an email notification if the flow execution is taking longer than the specified number of minutes. 
• Emails - a space-separated list of email addresses that will be notified about success, failure or if the flow execution is taking longer than expected.
• Retry on failure - if enabled and execution has failed, the Scheduler will retry the flow for the configured number of retries.
• Retries - the number of retries for failed execution attempts.
• Delay between retries - the delay between retries, in minutes.
• Record Console Log - if enabled, the DEBUG level log will be recorded for each flow execution. This options is disabled by default for the scheduled event-driven flows.
• Extra Log Step - an extra checkpoint which will be added to the flow log every time the specified number of data records is processed. While expanding the verbosity of the log, it can also negatively impact performance. Use Extra Log Step if you want to monitor a flow which is processing a large number of the data records, typically > 1M(one million). Otherwise, keep it empty.
• Keep Message - if enabled, system will store bodies of the POST/PUT requests in the database. The stored messages can be viewed in the messages app. The option is disabled by default.
• Keep Empty Statistics - keep flow execution statistics even in case of successful execution with 0 records, or files. By default, this option is enabled.

After an event-driven flow has been scheduled, you will need to wait one minute before you can start using it. The reason: Listeners need to be activated, which usually takes ~ 1 minute.

Configuring Notifications

When creating or modifying a schedule, it is possible to configure notifications. The following notifications are available:

Send email on success

This notification is sent when the flow is successfully executed.

Step 1. Enable "Send email on success".

Step 2. Use the field "Email" to configure email address(es) which will be receiving a notification.

Send email on failure

This notification is sent when the flow is either failed or canceled.

Step 1. Enable "Send email on failure".

Step 2. Use the field "Email" to configure email address(es) which will be receiving a notification.

Send email if flow execution is taking longer than expected

This notification is sent when the flow execution is taking longer than expected. The notification will be sent one time when the actual run time exceeds the configured threshold.

Step 1. Use the field "Expected flow execution time" to specify the expected flow execution time in minutes. 

Step 2. Use the field "Email" to configure email address(es) which will be receiving a notification.

Configuring Auto-Retry on Failure

When creating or modifying a schedule, it is possible to configure auto-retries. When the auto-retry is enabled and the flow execution failed, the system will attempt to execute the flow again until flow successfully executed or the configurable number of times, whichever comes first.

Step 1. Enable "Retry of failure". 

Step 2. Select the number of "Retries" (5 max).

Step 3. Select "Delay between retries" (1-10 minutes)