Create Flows to run in the Integration Agent

• Starter
• Business
• Enterprise
• On-Premise
• Add-on

Overview

The Etlworks Integrator instance running in a public cloud cannot access the data behind the company's firewall, and therefore you need the remote Integration Agent to run the flows. You can still create Connections, Formats, and Flows using the Etlworks Integrator web UI but unfortunately, you cannot run Flows directly in the Etlworks Integrator cloud instance. This process outlines how to create Flows to run in the remote agent.

Create Connections and Formats

Create all required Connections and Formats using the Etlworks Integrator Web UI. If you are connecting to the database or app running on-premise behind the firewall, use Connection parameters, such as URLs that point to the local resources.

The Remote Integration Agent does not support Listeners and you will not be able to test the local Connections.

Create Flows

Similar to Connections and Formats, create Flows using the Etlworks Integrator Web UI.

Mapping

If the Etlworks Integrator is able to access the database, API, or file, it can pull the metadata, such as table's names, columns, data types, etc., from the data source. This directly helps with the Mapping since you can map source objects to destination objects, and source columns to destination columns by selecting them from the drop-down. 

When the data source is not available (for example, when the Connection points to the local database), you can still use the Mapping interface. However, you won't be able to access the actual tables and columns and you need to enter them manually in the Mapping screen.

We suggest the following workarounds:

Mapping if the local resource is a database 

The only available workaround is to create a "dummy" database with the same tables and columns as in the actual on-premise database but available on the Internet. Once you are done with the Mapping, you can switch the Connection back to the local database.

Mapping if the local resource is a file or a web service

If a file or a web service is not accessible from the cloud instance, you can always create a template when configuring a Format.

Schedule and Run Flows in the Integration Agent

• Add and schedule flows.
• Run Flows manually.
• Stop the running Flows.

For obvious reasons, you cannot test the transformation that requires access to the local resource directly in the Etlworks Integrator cloud instance.