Overview
Etlworks is a horizontally and vertically scalable web application. The same software runs everywhere — from a single VM to a multi-node cluster to a managed Kubernetes cluster — so you can start small and change the deployment model without changing your flows.
This article explains the components, the deployment options, and the supported topologies for reaching your data.
Architecture and components
Every Etlworks node runs the same pair of Java services in a Tomcat container:
- App — the UI backend, the REST API, the scheduler, Explorer, and Simba, the AI assistant. Users work in the browser; systems integrate through the open API, webhooks and the CLI.
- ETL Engine — executes flows: ETL and ELT, change data capture (CDC), streaming, file operations and API integrations. All connectors, including premium connectors (installed and licensed from Settings), are part of the engine.
The nodes share three backing services:
- PostgreSQL — the repository: metadata, configuration, revision history and execution history.
- Redis — caching, messaging and cluster coordination.
- Shared storage — server storage for application data: staging files, CDC files, uploaded keys.
Integration Agents are optional satellite runners installed inside your network. They execute flows next to the data and talk to Etlworks over outbound HTTPS only (topologies).
Deployment options
Etlworks runs on any platform and operating system, in the cloud and on-premises:
| Option | Who operates it | Best for |
|---|---|---|
| Etlworks Cloud (shared) | Owned, managed and operated by Etlworks | Getting started quickly; most workloads. Your tenant is fully isolated from other tenants. |
| Etlworks Cloud (dedicated instance) | Owned, managed and operated by Etlworks; single-tenant | Enterprise plans that need a dedicated instance in a specific cloud and region — AWS, Azure or Google Cloud. |
| Your VM or Docker (single node or unmanaged cluster) | You | Full control on your own infrastructure: native installer for Windows and Windows Server, bash installer for Linux (RHEL, Rocky, Ubuntu, Debian and others), or Docker Compose. The same installer updates the instance to the latest release. |
| Managed Kubernetes cluster | You own the cluster; the Etlworks installer manages the deployment | Kubernetes-native operation with automated delivery, in-app scaling, pod sizing, upgrades, backup and recovery — on a self-managed cluster, Amazon EKS, Azure AKS or Google GKE. |
Not sure which fits? See Choosing the Right Etlworks Deployment.
Single node and the unmanaged multi-node cluster
A single node — a VM, a physical server or a Docker host — runs the complete platform and scales vertically with CPU and memory.
For horizontal scale and high availability, Etlworks runs as a symmetrical multi-node cluster behind a load balancer:
- Every node runs the same App and Engine; any node can run any flow.
- The load balancer picks the node for each request using a configurable algorithm — round-robin by default — and scheduled jobs are distributed across the active nodes.
- The scheduler is elected: it runs on one node at a time and automatically migrates to another node if that node fails.
- All nodes share the same PostgreSQL, Redis and server storage — on AWS, typically Amazon RDS, ElastiCache and EFS.
- The cluster can run a fixed number of nodes, or nodes can be added and removed with the load.
Setup guides: Multi-node Etlworks deployment on an unmanaged cluster and AWS multi-node deployment.
Managed Kubernetes cluster
Since release 10.0.14, Etlworks runs as a managed cluster on your own Kubernetes — the installer delivers, configures and operates the whole deployment, and day-two operations move into the Etlworks UI:
- Platforms: self-managed Kubernetes clusters, Amazon EKS, Azure AKS and Google GKE, on ordinary Linux node pools (AMD64 and ARM64).
- Inside the cluster: identical application pods (App + Engine) behind an HAProxy ingress, with PostgreSQL and Redis either provisioned by the installer or brought by you as external services, and shared NFS storage with retained disks that survive reinstalls.
- Scaling: the pod range, load targets and capacity schedules are set on the Settings → Elastic cluster page; the cluster adds and removes application pods automatically. Pod CPU and RAM can be changed from the UI (10.0.16 and newer).
- Lifecycle: application upgrades (rolling or maintenance) from the UI or CLI, offline backup with verified restore, failed-upgrade recovery, and migration between regular and managed deployments.
- Trial: a 14-day Kubernetes trial is available from the downloads page.
Full guides: Install Etlworks on Kubernetes and Install Etlworks on cloud Kubernetes: Amazon EKS, Azure AKS and Google GKE.
Supported topologies
Wherever Etlworks itself runs, there are three ways to reach your data:
Direct connections
Cloud databases, SaaS applications, APIs and file services are connected directly. For an on-premises database that can't be exposed, Etlworks can connect through an SSH tunnel.
Hybrid: Integration Agents
An Integration Agent is installed inside your network (Windows, Linux or Docker) and runs flows next to the data — internal databases, file shares, ERPs and legacy applications. The agent connects out to Etlworks over HTTPS; no inbound ports are opened, and the data it moves between internal systems never has to leave your network.
Fully on-premises
Etlworks itself runs in your network — on a VM, in Docker, on an unmanaged cluster, or as a managed Kubernetes cluster — next to the systems it integrates. Native installs support air-gapped deployments; on Kubernetes, load the container images into a registry your cluster can reach.