Articles in this section

About the Integration Agent

  • Updated
Follow
  • Startup
  • Business
  • Enterprise
  • On-Premise
  • Add-on

Overview

An Integration Agent is a zero-maintenance, easy-to-configure, fully autonomous ETL engine which runs as a standalone app or a background service behind the company’s firewall. It can be installed on Windows and Linux and on any OS which supports Docker.

It allows you to run Flows that can access on-premise applications and databases. Outbound communication between Integration Agent and Etlworks cloud instance is fully secured and data is never staged in the cloud. 

How it works

A few things you should know regarding the Integration Agent:

1. Same ETL Engine as Etlworks:

The Integration Agent utilizes the same ETL engine as Etlworks, allowing it to execute the same Flows locally without relying on the Etlworks cloud instance.

2. Full Integration with Etlworks Cloud:

It is fully integrated with the Etlworks cloud instance. This means that Flows created in the Etlworks Web UI can be manually triggered or scheduled to run periodically on the remote Integration Agent.

3. Agent-Initiated Communication:

The communication session between the Agent and the Etlworks host is always initiated by the Agent. This approach is taken under the assumption that Etlworks cannot directly access the Agent running behind a firewall.

4. Secure Communication:

The Integration Agent communicates with the Etlworks host by calling API endpoints on the host over a secure HTTPS connection.

5. Action Queue and Execution:

When a user triggers any action, such as running a Flow, stopping a Flow, or stopping the Agent, the action is added to a queue. These queued actions are executed in order during the next communication session between the Agent and the Etlworks host.

6. Agent requires a separate subscription:

The Integration Agent requires a separate subscription with a small monthly fee. Please contact Etlwork's support for more details.

For further reading, check out the articles below:

Supported scenarios

On-premise to cloud

Example: load data from the on-premise SQL Server database to the Snowflake data warehouse.

In this scenario, the Integration Agent runs in your on-premise infrastructure and has access to the needed local resources (local file system, databases, and so on). The data extracted from the on-premise database ends up in the destination cloud application (like Snowflake) without going through the Etlworks cloud instance.

Cloud to on-premise

Example: load data from cloud application (for example, Marketo) to the on-premise database.

The Integration Agent runs in your on-premise infrastructure. It pulls the data from the cloud application without going through the Etlworks cloud instance.

On-premise to on-premise

Example: upload a local file to an on-premise database.

All traffic and data movements stay within your on-premise infrastructure.

Cloud to cloud 

Example: load data from the cloud application (for example, Marketo) to the cloud database (for example, Snowflake).

You would typically run this type of scenario directly in Etlworks cloud instance. Still, there are some advantages of offloading this to the remote Integration Agent installed on-premise:

  • Load balancing.
  • Access to the private cloud resources, not otherwise available from the Internet. 
  • Records processed by the Integration Agents will not count toward the record limits set for the account.

Related articles

Comments

0 comments

Please sign in to leave a comment.