Articles in this section

Multi-node Etlworks deployment on an unmanaged cluster

  • Updated
Follow

Overview

Etlworks can run as a symmetrical, horizontally scalable multi-node cluster on your own infrastructure — an unmanaged cluster that you install and operate with the host CLI on each node, as opposed to the managed Kubernetes deployment where an installer and lifecycle controller manage the application pods for you.

All nodes share PostgreSQL, Redis (same prefix), the application data directory, and the JWT secret. The installer supports automatic provisioning of the Etlworks servers and the shared components (HAProxy, PostgreSQL, and Redis) in any cluster running on Linux. Read how to provision a multi-node Etlworks cluster on AWS.

Simplified deployment diagram:

Etlworks can run as a symmetrical, horizontally scalable cluster sharing PostgreSQL, Redis (same prefix), the application data directory, and the JWT secret. Read how to provision a multi-node Etlworks cluster on AWS.

Installing and upgrading shared-resource nodes

Use --shared-resources --external-db for every node of a cluster. In this mode the CLI installs or updates Java, Tomcat, WARs, drivers, and native files only — it does not install PostgreSQL or Redis, does not create the shared application data directory, does not copy it into the node recovery workspace, and does not change its ownership or ACLs. The shared data directory must already exist and be writable by the service identity.

Linux node:

sudo ./etlworks-cli.sh install --shared-resources --external-db \
  --conf-postgres-url jdbc:postgresql://db-host:5432/integrator \
  --conf-postgres-username postgresql --conf-postgres-password PASSWORD \
  --conf-redis-host redis-host \
  --conf-app-data /mnt/etlworks/app-data \
  --conf-jwt-secret JWT_SECRET

Windows node:

etlworks-cli.bat install --shared-resources --external-db --pg-bin "C:\PostgreSQL-tools\bin" --conf-postgres-url jdbc:postgresql://db-host:5432/integrator --conf-postgres-username postgresql --conf-postgres-password PASSWORD --conf-redis-host redis-host --conf-app-data "\\fileserver\etlworks\integrator-data" --conf-jwt-secret JWT_SECRET

On Windows, shared-resource Tomcat runs as NT AUTHORITY\NetworkService. For a domain SMB share, grant the node's machine account (DOMAIN\COMPUTER$) the required share and NTFS permissions before setup — the installer verifies reachability but deliberately does not take ownership or rewrite shared ACLs.

Upgrading a node:

etlworks-cli.bat upgrade --shared-resources --external-db --pg-bin "C:\PostgreSQL-tools\bin"

Interactive upgrade requires confirmation that PostgreSQL, Redis, and the shared app data have a coordinated external backup; unattended upgrade adds --shared-resources-backed-up. The node recovery workspace contains only the local Tomcat/service backup — it is not a rollback backup for shared data. Automatic Windows Java 8 → 25 migration is rejected in shared-resource mode: database migration, writer quiescence, compatibility, and rollback must be coordinated for the whole cluster (plan it with support). Coordinate downtime and release compatibility across all nodes — mixed-version rolling operation is not promised to be safe. Each node needs the license installed and validates independently; all nodes share the same JWT secret.

AWS: long-running flows and idle TCP timeouts

AWS Nitro v6 instances, including M8a, use a default idle established-TCP connection-tracking timeout of 350 seconds when the ENI has no custom value. A long-running Etlworks flow can execute for a long time while producing no HTTP traffic, so AWS may expire the app-to-engine connection even though the engine continues running. The two configurations below keep that connection alive.

Configure EC2 network interfaces

Apply this to every ENI participating in the app → HAProxy → engine communication path, particularly ENIs attached to Nitro v6 Etlworks nodes.

1. Find an instance's ENIs.

aws ec2 describe-instances \
  --instance-ids i-xxxxxxxxxxxxxxxxx \
  --query 'Reservations[0].Instances[0].NetworkInterfaces[].NetworkInterfaceId' \
  --output text

2. Inspect the current setting.

aws ec2 describe-network-interfaces \
  --network-interface-ids eni-0123456789abcdef0 \
  --query 'NetworkInterfaces[0].ConnectionTrackingConfiguration'

How to read the result: null means no custom value is configured; on M8a / Nitro v6, null means the 350-second default applies; a displayed TcpEstablishedTimeout value is measured in seconds.

3. Set the maximum five-day timeout.

aws ec2 modify-network-interface-attribute \
  --network-interface-id eni-0123456789abcdef0 \
  --connection-tracking-specification TcpEstablishedTimeout=432000

4. Verify. Run the inspect command from step 2 again. Expect:

{
  "TcpEstablishedTimeout": 432000
}

AWS Console path. EC2 → Network Interfaces → select the ENI → Actions → Modify idle connection tracking timeout → set TCP established timeout to 432000 → Save.

Notes. This change does not require an EC2 reboot. It does not recover connections whose tracking state has already expired. Launch templates, Auto Scaling configuration, or Infrastructure as Code must also be updated so replacement nodes receive the same setting.

References: EC2 security group connection tracking — timeouts, Modify network interface attributes.

Configure HAProxy TCP keepalives

Increasing the ENI timeout alone is insufficient for flows that may run indefinitely. HAProxy should send TCP keepalive probes on both the app-facing and engine-facing connections.

Add the following to the applicable HAProxy defaults section:

option clitcpka
option srvtcpka

clitcpka-idle  240s
clitcpka-intvl 10s
clitcpka-cnt   3

srvtcpka-idle  240s
srvtcpka-intvl 10s
srvtcpka-cnt   3

The 240-second idle interval is below AWS's 350-second default. Keepalive probes must be enabled independently on both sides because HAProxy does not forward TCP keepalive probes between its client and server connections.

1. Validate the complete configuration.

sudo haproxy -c \
  -f /etc/haproxy/haproxy.cfg \
  -f /etc/haproxy/conf.d

2. Reload HAProxy.

sudo systemctl reload haproxy

Notes. A server reboot is not required. The keepalive settings apply to newly created connections. Existing stuck executions must be terminated and rerun. After configuration, verify with a flow that runs longer than 350 seconds.

Reference: HAProxy configuration — option clitcpka.

Related articles