Architecture and Deployment

Overview

The Etlworks Integrator is a modern, horizontally scalable web application that can be deployed to a single node (any VM, for example, EC2 instance or physical box) or to multiple nodes in a cluster behind a load balancer. The infrastructure can be configured to have a fixed number of nodes or to scale up and down depending on the load.

Deployment

We are very flexible in providing a choice to our customers to run the Etlworks Integrator in any platform and operating system, cloud and on-premise.

The following deployment options are available:

• Shared cloud instances, owned, managed, and operated by Etlworks. Our shared instances operate in two AWS regions:
  • us-east-1 (Ohio)
  • eu-west-1 (Ireland)
• Dedicated cloud instances owned, managed, and operated by Etlworks. Dedicated instances are available for customers on Enterprise plans. We support all major cloud providers: AWS, Azure, Google Cloud, Oracle Cloud, and IBM Cloud, as well as all available regions, including GovCloud. 
• Dedicated cloud instances which are owned, managed, and operated by the customer but provisioned and upgraded by Etlworks. We push updates from our centralized build server. Etlworks must have SSH access to the instance.
• Dedicated cloud or on-premise instances which are owned, managed, and operated by the customer when Etlworks has no access to the instance. Etlworks provides a fully automated installer for Ubuntu 20.04, Amazon Linux 2, CentOS, Red Hat 7 and 9,  Windows Server (2012-2022), all editions of Windows 10 and Windows 11, and Docker. The same installer can be used to automatically update the instance to the latest version of the Etlworks Integrator.

Architecture

In a multi-node setup, the ETL jobs are distributed between all active nodes in a symmetrical cluster. The load balancer chooses a node to run the job in by using one of the configurable load-balancing algorithms. The default is round-robin.

The scheduler always runs on a single node and automatically migrates to the next available node.

All nodes in a cluster must share the same server storage (for example, EFS on AWS), Redis (for example, Elasticache Redis on AWS), and Postgres (for example, RDS Postgres on AWS). 

AWS multi-node deployment diagram

AWS multi-node setup details can be found here.