Overview
This guide provisions the AWS infrastructure for a symmetrical, horizontally scalable Etlworks cluster — an unmanaged cluster built from AWS managed services: EC2 application nodes behind HAProxy, with Amazon RDS PostgreSQL, Amazon ElastiCache Redis, and Amazon EFS shared storage.
Related guides:
- Multi-node Etlworks deployment on an unmanaged cluster — installing and upgrading the node software (including Windows nodes), upgrade rules, and the AWS TCP-timeout tuning for long-running flows.
- Architecture and Deployment — the components and all deployment options, including the managed Kubernetes cluster, where an installer provisions and operates everything for you.
- Prefer self-hosted shared components over AWS managed services? The host CLI can provision HAProxy, PostgreSQL and Redis itself on any Linux cluster — see Running Etlworks on-premise.
Deployment steps
1. Application EC2 instances
1.1. Create two to N EC2 instances from a current general-purpose family (for example m6i or m7i) with Amazon Linux 2023 or a current Ubuntu LTS.
1.2. The security group must allow port 8080 from the HAProxy EC2 instance (step 5) and from your incoming traffic load balancer (step 7).
2. EFS shared storage
2.1. Create an EFS file system shared by all application EC2s.
2.2. Its security group must allow the NFS port 2049 from the application EC2s' security group.
2.3. Mount the EFS on every application EC2 at /opt/app-data, replacing {efs_url} with the actual EFS URL. On Amazon Linux 2023:
sudo dnf install -y amazon-efs-utils
sudo mount -t efs -o tls {efs_url}:/ /opt/app-data
On Ubuntu, mount over standard NFSv4.1 instead (package nfs-common):
sudo mount -t nfs4 -o nfsvers=4.1 {efs_url}:/ /opt/app-data
3. RDS PostgreSQL
3.1. Create an RDS PostgreSQL instance on a recent PostgreSQL major version (db.m6i.large is sufficient for most installations).
3.2. Enable password authentication and note the password for the application configuration.
3.3. Set Initial database name to integrator.
3.4. Allow port 5432 from the application EC2s' security group.
4. ElastiCache Redis
4.1. Create an ElastiCache Redis instance (cache.t4g.small is sufficient for most installations).
4.2. Allow port 6379 from the application EC2s' security group.
5. HAProxy EC2
5.1. Create a small EC2 instance (for example t3.small) with Amazon Linux 2023.
5.2. Its security group must allow port 80 from the application EC2s' security group.
5.3. Install HAProxy:
sudo dnf install -y haproxy
5.4. Configure HAProxy by replacing /etc/haproxy/haproxy.cfg with:
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 10000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 10
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 5m
timeout server 0
timeout http-keep-alive 10s
timeout check 10s
maxconn 10000
frontend http-in
bind *:80
default_backend servers
backend servers
balance roundrobin
option httpchk GET /etl/rest/v1/health
http-check expect status 200
server tomcat1 {node-1-ip}:8080 check
server tomcatN {node-N-ip}:8080 check
5.5. Replace {node-1-ip}, {node-N-ip} with the actual application EC2 IPs (step 1), adding a server line per node.
Note: the recommended setting for timeout server is 0 (infinite). It allows continuous running of streaming flows, such as CDC. Any other value (for example 43200m, which is 30 days) can time out the HTTP connection between the scheduler and the ETL process — and when that connection times out, the ETL process is killed by the server.
Note: on AWS, idle TCP connections are also dropped by the network itself after about 350 seconds. Configure TCP keepalives on the HAProxy node and the EC2 network interfaces as described in AWS: long-running flows and idle TCP timeouts.
6. Etlworks installation
Install Etlworks on each application EC2 node:
6.1. Download the installer for each node, or request it from the support team. The installers are identical, but each contains a unique license.
6.2. Create the /opt/etlworks directory and copy the installer archive into it.
6.3. Unarchive the installer:
sudo tar -zxf etlworks-installer.tar.gz
6.4. Run the installation:
sudo ./etlworks-cli.sh install -s -u <user> --external-db \
--conf-postgres-url jdbc:postgresql://<rds_postgres_host>:5432/integrator \
--conf-postgres-username <rds_postgres_username> \
--conf-postgres-password <rds_postgres_password> \
--conf-redis-host <elasti_cache_redis_host> \
--conf-redis-port 6379 \
--conf-app-data /opt/app-data \
--conf-jwt-secret <jwt_16_character_alphanumeric_secret>
Replace:
- <user> — ec2-user on Amazon Linux 2023, ubuntu on Ubuntu.
- <rds_postgres_host> — the RDS PostgreSQL endpoint (step 3).
- <rds_postgres_username> — the RDS PostgreSQL username (step 3); postgres by default if not changed during initial setup.
- <rds_postgres_password> — the RDS PostgreSQL password (step 3).
- <elasti_cache_redis_host> — the ElastiCache Redis endpoint (step 4). If a password is set for Redis, also add --conf-redis-password <password>; if encryption in transit is enabled, also add --conf-redis-ssl true.
- <jwt_16_character_alphanumeric_secret> — a random 16-character alphanumeric string. It must be the same on every node.
If a configuration mistake was made during installation, fix it afterwards with sudo ./etlworks-cli.sh update-conf.
Note: for installing and upgrading shared-resource nodes — including the --shared-resources mode, Windows nodes, and the coordinated-backup upgrade rules — see Multi-node Etlworks deployment on an unmanaged cluster.
7. Incoming traffic load balancer
7.1. Configure your incoming traffic load balancer to proxy requests to port 8080 of the application EC2s. Terminate HTTPS here (or in front of it), so users and API clients always connect over TLS.
8. Global application settings
After the installation completes, log in with the super admin account created by the installation — and change its password immediately — then open Settings:
8.1. Under General, set Home URL to the load balancer URL, including the protocol.
8.2. Under Email, set the email configuration the system uses to send notifications — required for adding new users, resetting passwords, and email notifications.
8.3. Under Network, set ETL Engine Proxy URL to the HAProxy URL (step 5).