Architecture and Deployment

Overview

The Etlworks Integrator is a modern, horizontally scalable web application that can be deployed to a single node (any VM, for example, EC2 instance or physical box) or to multiple nodes in a cluster behind a load balancer. The infrastructure can be configured to have a fixed number of nodes or to scale up and down depending on the load.

Deployment

We are extremely flexible in providing a choice to our customers to run the Etlworks Integrator in any platform and operating system, cloud, and on-premise.

The following deployment options are available:

• Shared cloud instances, owned, managed, and operated by Etlworks. Example: https://app.etlworks.com. Our shared instances are running in AWS us-east-1 and eu-west-2 regions.
• Dedicated cloud instances, owned, managed, and operated by Etlworks. Dedicated instances are available for the customers on Enterprise plans. We support all major cloud providers: AWS, Azure, Google Cloud, Oracle Cloud, and IBM cloud, as well as all available regions, including GovCloud. 
• Dedicated cloud instances, owned, managed, and operated by the customer but provisioned and upgraded by Etlworks. We push updates from our centralized build server. Etlworks must have SSH access to the instance.
• Dedicated cloud or on-premise instances, owned, managed, and operated by the customer when Etlworks doesn't have any access to the instance. Etlworks provides a fully automated installation script for Ubuntu 18.4 and up, Amazon Linux (1 and 2), and CentOS. The same script can be used to automatically update the instance to the latest version of the Etlworks Integrator.

Architecture

In a multi-node setup, the ETL jobs are distributed between all active nodes in a symmetrical cluster. The load balancer chooses a node to run the job in by using one of the configurable load-balancing algorithms. The default is round-robin.

The scheduler always runs on a single node and automatically migrates to the next available node.

All nodes in a cluster must share the same server storage (for example EFS on AWS), Redis, and Postgres. 

AWS multi-node deployment diagram