• Starter
• Business
• Enterprise
• On-Premise
• Add-on

Overview

This article is the administrator reference for installing, upgrading, and operating the full Etlworks application on your own infrastructure: fresh Linux and Windows installations, routine upgrades, the Java 25 / Tomcat 11 runtime migration introduced in Etlworks 10.0.0, PostgreSQL upgrades, backup and restore, recovery and rollback, multi-node deployments, connectors, and health monitoring.

It covers three different command-line tools. Keep them distinct:

• The external application host CLI — etlworks-cli.sh on Linux, etlworks-cli.bat / etlworks-cli.ps1 on Windows. It installs, upgrades, backs up, restores, and controls the Etlworks application from the host shell and works even while the application is stopped. This is the tool most of this article is about.
• The built-in Etlworks CLI — runs through the application itself and requires a functioning, signed-in application. See Using the Etlworks CLI.
• The Integration Agent CLI — etlworks-agent, a separate product with its own installation and upgrade documentation. Agent installation and updates are not covered here; see Install, update, configure and monitor the Integration Agent.

9.x and 10.x: two runtime stacks

Etlworks 10.0.0 moves the application from Java 8 / Tomcat 8.5 to Java 25 / Tomcat 11:

A 10.x upgrade is a runtime migration, not just a WAR replacement: Java, Tomcat, the application artifacts, the database prerequisites, JVM and application configuration, and the recovery plan must be handled together. The migration workflows in this article do that in one coordinated operation per platform.

Legacy 9.x procedures remain in this article and are clearly labeled. Use them only for maintaining an existing 9.x installation; new installations should start on 10.x.

Supported operating systems and platforms

For Etlworks 10.x (Java 25 / Tomcat 11):

• Ubuntu 22.04
• Amazon Linux 2023
• Red Hat Enterprise Linux 9
• Windows x64 — Windows Server and Windows 10/11. Windows ARM64 is not a supported target.
• Docker — see Run Etlworks in Docker; also the supported path on macOS.

The Linux migration and 10.x installation commands are available only on the three Linux distributions above. Other distributions remain usable for existing 9.x installations:

• Amazon Linux 2, Ubuntu 20.04, CentOS 7, Red Hat 7 and 8 — legacy installation and 9.x maintenance paths only. To move to 10.x, migrate to a supported distribution.
• Windows Server 2012–2022 and Windows 10/11 remain supported for existing 9.x installations through the legacy Etlworks.exe installer.

Hardware requirements

Read more.

License Validation and Firewall

Etlworks on-premise deployments require outbound communication from the on-premise server(s) to our license server (host: lic.etlworks.com, dedicated IP address: 18.217.236.168) over HTTPS (port 443). This communication is strictly outbound and does not require opening inbound firewall rules.

Key Requirements

• All servers running Etlworks in the environment must be able to access: lic.etlworks.com:443 (outbound traffic only).
• License validation occurs on startup and then every hour to ensure the instance remains authorized.
• License validation on startup is required.
• Each node in multi-node deployments independently reports license validation data.

Firewall

Inbound Firewall

No inbound firewall changes are required for Etlworks to function in standard on-premise deployments.

However, if the Etlworks instance is used to serve API requests, either from built-in APIs or user-defined APIs, and these requests originate from outside the firewall, the following inbound ports should be opened:

• Port 80 (HTTP) – If APIs are served without SSL.
• Port 443 (HTTPS) – If APIs are served over a secure connection.

Outbound Firewall

• Etlworks requires outbound access to lic.etlworks.com:443 (dedicated IP address: 18.217.236.168) for license validation.
• If an outbound firewall is in place, port 443 (HTTPS) must be open for communication with our license server.
• If the on-premise Etlworks server needs to access external resources — such as databases, message brokers, cloud APIs, or third-party services — the relevant outbound ports must also be opened to allow connectivity.

Data Transmitted During License Validation

During each license validation session, the Etlworks server(s) securely transmit the following information to our license server:

• Server identification details, including IP address, Etlworks version, and digital fingerprints unique to the instance.
• Digital fingerprints include non-sensitive hardware metadata (physical or virtual).
• Instance usage metrics: number of flows, connections, formats, listeners, schedules, and agents.
• System health status, as returned by the /health API.

Security Measures

• All transmitted data is encrypted using a pair of environment-specific encryption keys, ensuring that only Etlworks and the authorized instance can read the payload.
• No sensitive or customer data is ever transmitted.
• Neither our license server nor other elements of Etlworks infrastructure ever initiate inbound connections to on-premise servers.

Failure Scenarios

• If a license is found to be invalid or has been remotely deactivated by Etlworks, the server(s) will stop functioning during the next validation check.
• If an on-premise server fails to connect to the license server for 24 hours, it will automatically shut down until connectivity is restored.
• License validation on startup is required.

Refresh the license without restarting

License properties can be refreshed on a running installation — for example after a renewal or plan change. A Super Admin can either click Refresh license under Profile → About, or run the built-in CLI command refresh-license. The refresh is applied immediately on all live application nodes; no restart is required.

Downloads, licensing, and tooling

Download installers and CLI tools

Installers and CLI tools are included with every Etlworks installation. They can also be downloaded separately.

Installers, the Linux CLI (etlworks-cli.sh), and the Windows CLI (etlworks-cli.bat / etlworks-cli.ps1) are available from a single self-service page on the marketing site:

etlworks.com/downloads/tools.html

The page is also reachable from etlworks.com → Resources → Installer & CLI tools.

You don't need a license to download the CLI tooling from this page. Refresh the CLI tools before a 10.x upgrade, a migration, or a restore: the 10.x migration commands exist only in current CLI packages.

IMPORTANT — the current CLI package is the 10.x installer/upgrader. It includes a new download client, and once you replace your tooling with it, that installation's tooling can no longer install or upgrade 9.x releases. If you intend to stay on 9.x for now, keep your existing 9.x tooling (the 1.x Linux CLI, or Etlworks.exe on Windows) and download the new CLI only when you are ready to move to 10.x.

Keep the difference between four things clear:

• Downloading installer/CLI tooling (no license) — the tools page above.
• Obtaining a license or trial installer — the download request form; the archive includes your unique .lic license file.
• Downloading licensed application artifacts — done by the CLI itself during install/upgrade using your license.
• Updating the CLI — replaces the tooling only; it does not change the installed application. On Linux, upgrade-cli updates the CLI in place and cli-version shows the installed CLI version. On Windows, check the version with cli-version and update by downloading the current package and extracting all of its files together (there is no Windows upgrade-cli command).

License placement

• Linux: the CLI looks for a .lic file next to itself in /opt/etlworks, then in the Tomcat conf directory. The installed license lives in TOMCAT_HOME/conf.
• Windows: put the .lic file next to the CLI scripts, or pass --license. The installed license is preserved across upgrades and migrations.
• Multi-node: every node needs the license installed; each node reports license validation independently.

Version selection and verification

• list shows the application versions visible to your license. Installations already on Tomcat 11 are offered 10.x releases; a 9.x installation is offered 9.x releases until it is migrated.
• install/upgrade without --version target the latest visible release — the normal case. To upgrade to a specific version, run list first to get the version, then pass an exact --version. On Windows, the CLI reads the exact version from the downloaded application WAR and pins the related artifact downloads to that same version; downgrade attempts are rejected.
• Downloaded artifact bundles are integrity-checked (SHA-256 manifests, validated archive extraction). Checksums detect corruption and mixed artifacts — they are not signatures and do not make an arbitrary download source trustworthy.
• On Windows, --artifact-source PATH_OR_HTTPS_URL is a supported alternative that installs from an approved local or HTTPS release directory (manifest + payload ZIPs) instead of licensed downloads — used for offline transfer and testing. Side-loading files does not remove runtime license verification, and HTTP sources are rejected.

Installation

Outbound Internet Access During Installation

During installation, the Etlworks installer retrieves required components over secure HTTPS connections. Outbound internet access is required for a successful installation.

Linux Installations

When installing Etlworks on Linux, the installer installs Java and Tomcat from the distribution's package repositories and pulls the PostgreSQL and Redis container images from public container registries (Docker Hub) over HTTPS.

Windows Installation

When installing Etlworks on Windows, the installer downloads pinned, checksum-verified packages for the required components:

• Java 25 (Azul Zulu, installed privately under Program Files\Etlworks)
• Tomcat 11
• PostgreSQL 17
• Redis

These packages are retrieved from their official distribution sources over HTTPS, and their versions are pinned in the CLI package — the installer never installs an unverified "latest" build.

Important Network Recommendation

With the exception of the Etlworks license server, the IP addresses of third-party distribution endpoints are not static. Over time IP addresses may change, components may be added or removed, and distribution sources may be updated.

Because of this, we strongly recommend not restricting outbound HTTPS access during installation by static IP address. If outbound traffic must be restricted, allow HTTPS access by DNS hostname rather than IP where possible. Restricting outbound internet access during installation may cause incomplete setup or failed component downloads.

1. Request a link to download an installer or license

To request a download link, open https://etlworks.com/downloads/request.html in any browser. Fill in the form and click Email me the installer.

Important: Enter a valid email address. The Etlworks license server will send a short-lived download link to the email provided during registration.

If you already have an account with Etlworks, please enter the same Company, Name, and Email used when the existing account was created. If you're unsure about these details, contact Etlworks support at support@etlworks.com for assistance.

2. Open the download link

After submitting the registration form, the Etlworks license server will generate a personalized download link and send it to the registered email address. This link will remain active for approximately 10 minutes. Make sure to access it within this time frame to download the required files.

3. Select the installer to download

Select either the Linux or Windows installer. Read how to run Etlworks in Docker.

4. Install Etlworks

Install Etlworks 10.x on Linux

Applies to: Ubuntu 22.04, Amazon Linux 2023, Red Hat 9, with the current (2.x) Linux CLI. Root privileges are required for every CLI command.

The Linux installer is provided as a tar.gz archive containing required libraries, the etlworks-cli.sh command-line script, and your unique license file.

Step 1. Copy etlworks-installer.tar.gz into the /opt/etlworks directory and unarchive it:

cd /opt/etlworks
sudo tar -zxvf etlworks-installer.tar.gz

Step 2. Review the available commands and the CLI version:

sudo ./etlworks-cli.sh help
sudo ./etlworks-cli.sh cli-version

Step 3. Install:

sudo ./etlworks-cli.sh install

This is all most installations need: it installs the latest release visible to your license. To install a specific version instead, run sudo ./etlworks-cli.sh list to see the available versions and pass --version X.Y.Z.

What gets installed. The default Linux stack uses native Tomcat 11 with the distribution's Java 25 package, plus locally managed PostgreSQL 17 and Redis Docker containers defined in /opt/etlworks/docker-compose.yml. This is different from running the entire application in Docker — only the databases are containers. Application data defaults to /opt/etlworks/app-data; Tomcat and its configuration live under /usr/share/tomcat; the service listens on HTTP port 8080.

JVM defaults. A managed options setter configures the Tomcat JVM on every start: initial heap 2 GB, maximum heap 75% of physical RAM, string deduplication, compact object headers, and the Java 25 module opening required by the Snowflake and Databricks Arrow readers (see JVM options).

Options:

• --version VERSION (alias -v) — application version to install (default: latest).
• --skip-confirm (alias -s) — proceed without confirmation prompts.
• --user USER_NAME (alias -u, default ec2-user) — the sudoer service user.
• --no-app (alias -n) — provision the system only; do not install application artifacts.
• --external-db — do not install, configure, or start the local PostgreSQL/Redis containers. All of the following become required: --conf-postgres-url (format jdbc:postgresql://<host>:<port>/<database>), --conf-postgres-username, --conf-postgres-password, --conf-redis-host, --conf-app-data, and --conf-jwt-secret. Optional: --conf-redis-port (default 6379), --conf-redis-password, --conf-redis-ssl (default false).
• --shared-resources — multi-node mode; requires --external-db. The application data directory must already exist and be readable/writable by Tomcat; its ownership and permissions are not changed. See multi-node deployments.

JWT secret format: 10.x accepts a 64+ character secret or an existing 16-character alphanumeric legacy secret; pre-10 releases require the legacy format. When migrating or adding nodes, reuse the existing JWT secret — do not rotate it as part of a migration.

With --external-db, the databases must already exist and be reachable: the CLI skips all local database provisioning and only installs Java, Tomcat, and the application. Supply a PostgreSQL database already created for Etlworks.

Install Etlworks 10.x on Windows

Applies to: Windows x64 with the current Windows CLI (etlworks-cli.bat / etlworks-cli.ps1). Run every command from an elevated Command Prompt or PowerShell (Administrator). The CLI package contains the BAT wrapper plus several PowerShell helper scripts — always extract all files together; the wrapper sets execution policy only for its own child PowerShell process and changes nothing machine-wide.

If Windows blocks the CLI scripts

Even from an elevated prompt, two things can stop the CLI scripts (.bat/.cmd and .ps1) from running: built-in Windows script policies and third-party protection software. Resolve this before installing or upgrading.

Built-in Windows policies (PowerShell execution policy and blocked downloaded files)

1. Always start the CLI through etlworks-cli.bat — the wrapper launches its child PowerShell with the required execution policy without changing machine-wide settings. If you need to run the PowerShell script directly, use the equivalent one-shot form:

   powershell.exe -NoProfile -ExecutionPolicy Bypass -File etlworks-cli.ps1 help

2. Files downloaded from the internet carry the Mark of the Web, and Windows can refuse to run them with errors such as "running scripts is disabled on this system" or "... is not digitally signed". Extracting a downloaded ZIP with Windows Explorer propagates the mark to every extracted file. After extracting, unblock the CLI files — right-click each file → Properties → Unblock, or for the whole folder:

   powershell.exe -NoProfile -Command "Get-ChildItem -Path C:\etlworks -Recurse | Unblock-File"

3. If the execution policy is enforced by Group Policy, the Bypass flag cannot override it. Check with:

   powershell.exe -NoProfile -Command "Get-ExecutionPolicy -List"

   A value set under MachinePolicy or UserPolicy comes from Group Policy — ask your Windows administrator to allow the Etlworks CLI scripts (for example, permit local scripts or exempt the CLI folder) before running the installer.

Protection software (antivirus / EDR)

Security products can block or quarantine command-line scripts even for Administrators. Typical symptoms: the window closes immediately, Access is denied, or the CLI reports that a required helper script is missing (a sign one of the .ps1 files was quarantined).

1. Work with your security team to allow the CLI folder (for example C:\etlworks) and its scripts before installation or upgrade.
2. Restore anything that was quarantined, then re-extract the complete CLI package — the CLI requires every file from the package, and a silently removed helper script causes later failures.
3. Re-run the command after the exclusion is in place. If the security product logged the block, that log confirms which file and rule were involved.

Step 1. Extract the CLI package, place your .lic file next to the scripts (or use --license), and review the commands:

cd /d C:\etlworks
etlworks-cli.bat help
etlworks-cli.bat help install

Step 2. Install:

etlworks-cli.bat install

This is all most installations need: it installs the latest release visible to your license. To install a specific version instead, run etlworks-cli.bat list to see the available versions and pass --version X.Y.Z. The CLI prompts for the application data directory and database credentials, lists the components it will install, and asks for one installation approval. It downloads and verifies all payloads before changing anything, and checks administrator rights, ports, target paths, the license, disk space, and existing endpoints first.

Interactive prompts:

Application data directory [C:\etlworks\integrator-data] (Enter to accept):
PostgreSQL password (Enter to use an autogenerated password, or type your own):

Pressing Enter at the password prompt generates a cryptographically random 32-character password for the new local PostgreSQL. Generated passwords are not printed: before dependency installation they are saved in postgres-credentials.json inside the ACL-protected recovery workspace whose path is printed. On failure, keep that file so the installed database's password can be recovered for a retry; on success the temporary workspace is removed and the credentials remain in the Tomcat conf/context.xml datasource. Treat both files as secrets. An existing or external database requires its actual password — the CLI never changes existing credentials.

Fresh-install defaults:

• Private Java 25 under Program Files\Etlworks — the CLI never replaces machine-wide Java, JAVA_HOME, or PATH.
• Tomcat 11 at Program Files\Etlworks\Tomcat11, Windows service Tomcat11, running as LocalService.
• Dedicated PostgreSQL 17 at Program Files\PostgreSQL\17, service etlworks-postgresql-17 (when locally provisioned).
• Local Redis in its normal MSI location (Program Files\Redis, service Redis), bound to loopback; no firewall rule is requested.
• HTTP port 8080; application data at C:\etlworks\integrator-data.
• Initial JVM heap 2048 MiB; maximum heap 75% of physical RAM — override with --heap-min-mb / --heap-max-mb. A warning is shown when little non-JVM headroom remains.
• Managed Java 25 JVM options, including string deduplication, compact object headers, and the Arrow module opening (see JVM options).

Dependency patch versions are pinned per CLI package and change only through qualified package updates — treat them as package-specific, not permanent product promises.

Externally managed databases:

etlworks-cli.bat install --external-db --pg-bin "C:\PostgreSQL-tools\bin" --conf-postgres-url jdbc:postgresql://db-host:5432/integrator --conf-postgres-username postgresql --conf-redis-host redis-host --conf-app-data C:\etlworks\integrator-data

Supply a database already created for Etlworks. Modern releases require PostgreSQL server major 17 or newer. Missing PostgreSQL client tools can be extracted from the approved installer without installing or replacing the server. An unreachable existing database is a connection problem to fix — not permission to replace it with an empty database.

Preview with --check. install --check (and upgrade --check) stages and validates the packages, configuration, and prerequisites without changing application files or services. It can download payloads and create an inspection workspace on disk — it is non-mutating for your installation and services, but it is not "no filesystem changes". A fresh noninteractive check requires --conf-postgres-password.

Unattended installation: --silent (aliases --skip-confirm, -s) skips prompts but never guesses missing answers: a fresh silent install requires --install-missing, all required settings, and --conf-postgres-password. --silent alone never authorizes migration, archiving an existing directory, switching install to upgrade, uninstall, recovery, or data deletion — each has its own approval flag.

Legacy: install Etlworks 9.x on Windows (Etlworks.exe)

This section applies to 9.x installations only. The legacy Windows installer Etlworks.exe remains the 9.x-to-9.x installation and maintenance path. It installs Java 8 (Azul Zulu OpenJDK 8), Tomcat 8.5, PostgreSQL, and Redis.

Download the 9.x archive, extract Etlworks.exe with its license file into a directory such as c:\etlworks, run it, and follow the wizard.

Note: Some older versions of antivirus software, specifically Windows Defender before version 1.381.3081.0, report that Etlworks.exe contains a trojan. THIS IS A FALSE POSITIVE. Update your antivirus to the latest version before downloading and installing.

Advanced Etlworks.exe switches (/skipprereqs=1, /skipjava=1, /skiptomcat=1, /skipredis=1, /skippostgresql=1, download-URL and install-command overrides, /VERYSILENT) apply only to this legacy installer. They are not options of the modern etlworks-cli — do not combine the two tool's switches. After a successful migration to 10.x, the CLI retires the old Etlworks.exe (archived with a receipt, not deleted blindly); future installations and updates use etlworks-cli.bat.

5. Finish installation and configuration

IMPORTANT: Keep the downloaded installer package and the provided license file. The same tooling is used to upgrade Etlworks. If you lose them, request them again at the download page providing the same Company, Name and Email used the first time.

After the installation is over, the service should be available on port 8080. Test it by opening the Etlworks URL in the browser, for example http://localhost:8080. The default Super Admin username and password are:

• user: admin
• password: admin1

Change the default password immediately after the first sign-in. Then navigate to Settings.

Set Home URL

Under General, set Home URL to your instance public URL, for example http://myhost:8080.

Configure Email

Important: This is required for adding new users, resetting passwords, and sending email notifications.

Under Email, provide the email configuration that will be used by the system to send notifications.

Read more.

Upgrade

Three upgrade cases exist. Identify yours first:

For every case: suspend flow executions persistently and take a backup first (see Before an upgrade or migration). The application is stopped during the upgrade. A successful command — or a service that shows Running — does not by itself prove the release works: verify version, health, sign-in, and a representative flow afterward.

Upgrade 10.x on Linux (artifacts only)

cd /opt/etlworks
sudo ./etlworks-cli.sh upgrade

This is all most installations need: it upgrades to the latest release visible to your license. Optional parameters: --version X.Y.Z upgrades to a specific version instead — run sudo ./etlworks-cli.sh list first to see the versions available to your license; --skip-confirm (alias -s) runs unattended. The CLI downloads and verifies the artifacts, stops Tomcat, replaces the artifacts, and starts Tomcat. It takes up to a few minutes for the service to restart.

Important — what is replaced on Linux. The ordinary Linux upgrade replaces the entire contents of the Tomcat drivers directory and the webapps directory with the downloaded bundles. Files you added to drivers yourself, and any unrelated web applications deployed into the same Tomcat, are removed. Keep customer additions in TOMCAT_HOME/lib (which is not touched), keep copies of custom driver files outside Tomcat, and re-add them after the upgrade. This differs from Windows, which preserves existing drivers.

Upgrade 10.x on Windows (artifacts only)

cd /d C:\etlworks
etlworks-cli.bat upgrade

This is all most installations need: the CLI finds the registered installation and upgrades it to the latest release visible to your license.

Installations migrated from 9.x: always pass --service Tomcat11. The Java 8 migration intentionally leaves the old Tomcat8 service and its Tomcat 8.5 installation in place, so the CLI now sees two Etlworks installations. A bare upgrade on such a machine refuses to guess and fails with Multiple Etlworks installations found: ... Select --service or --catalina-home; nothing changed. On every subsequent 10.x upgrade, select the modern installation explicitly:

etlworks-cli.bat upgrade --service Tomcat11

Optional parameters:

• --version X.Y.Z — upgrade to a specific version instead of the latest. Run etlworks-cli.bat list first to see the versions available to your license.

• --catalina-home, --service, --pg-bin — needed only with multiple or non-default installations. After a successful install or upgrade the CLI remembers the installation in %ProgramData%\Etlworks\cli\installation.json, so these are normally unnecessary; explicit options always win. Example:

  etlworks-cli.bat upgrade --catalina-home "C:\Program Files\Etlworks\Tomcat11" --service Tomcat11 --pg-bin "C:\Program Files\PostgreSQL\17\bin"

The CLI stops Tomcat, takes verified file/data/database backups, deploys both WARs, the shared driver bundle, and native DLLs, and restarts the same service. It does not download or replace an already compatible Java/Tomcat runtime, and it re-applies the managed Java 25 JVM flags without resetting heap limits or unrelated service options.

What is preserved on Windows:

• Everything in the existing Tomcat drivers directory is retained — nested folders, JARs, license files, other files. New bundles only add missing filenames; when a same-name bundled file differs, the CLI warns and keeps your existing file. No older driver is automatically replaced — review driver versions deliberately.
• Engine JDBC folders absent from the incoming WAR are preserved (for example a customer jdbc/salesforce folder).
• Configuration, secrets, memory/service settings, licenses, and data are reused.

If the existing PostgreSQL server is below major 17, the upgrade plans the required database migration as part of the same operation — see PostgreSQL upgrades. If the installation is a Tomcat 11 runtime still registered under a legacy service name (Tomcat8/Tomcat9/Tomcat10, left by an earlier CLI), a normal upgrade also repairs it to the clean Tomcat11 service and tray monitor — the service name alone does not identify the runtime version.

Unattended: upgrade --version ... --silent; adding missing dependencies additionally requires --install-missing, and a pending database migration requires --dedicated-postgres.

Legacy: upgrade 9.x

IMPORTANT: Use your existing 9.x tooling for these procedures. The current 10.x CLI package cannot install or upgrade 9.x releases — do not replace your 9.x tooling until you are ready to migrate.

Linux 9.x → 9.x. With the legacy (1.x) CLI in /opt/etlworks:

sudo ./etlworks-cli.sh upgrade

Optional --version (alias -v) and --skip-confirm (alias -s). A 9.x installation is offered 9.x releases until it is migrated to the 10.x stack.

Windows 9.x → 9.x. Use the legacy installer:

1. Stop Tomcat (optional but recommended).
2. Navigate to the directory with the Etlworks Windows installer, for example c:\etlworks. The directory must contain a valid license file.
3. Execute Etlworks.exe.

Note: After installation it takes up to a few minutes for the service to restart. The Etlworks.exe /skip... switches belong to this legacy installer only; they are not modern CLI options.

Troubleshoot Windows 9.x upgrade

If, after installing the upgrade, the service is not coming online:

Or the ETL engine version is not available in About:

Then the upgrade wasn't successful. To fix:

Step 1. Stop the Tomcat service. If the scripts do not run at all, see If Windows blocks the CLI scripts.

Step 2. Navigate to TOMCAT_HOME/webapps and manually delete the files etl.war and ROOT.war and the folders etl and ROOT.

Step 3. Run Etlworks.exe again.

Migrate from Java 8 / Tomcat 8.5 to Java 25 / Tomcat 11

This is the 9.x → 10.x path. It upgrades the runtime (Java, Tomcat), the database where required (PostgreSQL 17), the application artifacts, and the JVM/application configuration in one coordinated operation per platform.

Before you begin, on either platform:

1. Update the CLI tooling from etlworks.com/downloads/tools.html. On Windows, use CLI 1.3.7 or newer — the migration backup then records the metadata required by the failed-migration rollback command.
2. Suspend flow executions persistently and drain or stop running flows.
3. Take a backup (in-app task or host CLI) and keep it somewhere independent of the host. The migration makes its own recovery backup, but temporary migration backups are deleted after a verified success — an independent backup is your only rollback option after that cleanup.
4. Plan downtime: the application is stopped for the duration. The migration cannot undo writes your flows have already sent to external systems.
5. External or shared PostgreSQL/Redis require coordination beyond the local workflow — see the per-platform notes below.

Linux migration

Applies to: Ubuntu 22.04, Amazon Linux 2023, Red Hat 9, with the current (2.x) CLI. On other distributions the migration commands are unavailable; move the installation to a supported distribution first (install 10.x there, then restore your backup).

The coordinated command upgrades Tomcat to 11, Java to 25, the managed local PostgreSQL container to 17 — each only when actually needed — and then upgrades Etlworks itself to the latest 10.x release:

cd /opt/etlworks
sudo ./etlworks-cli.sh upgrade-environment-to-v10

The command warns that Java 25 and Tomcat 11 are not compatible with Etlworks releases before 10.0.0 and asks for one confirmation (--skip-confirm for unattended use). It takes no --version option — it always targets the latest post-10.0.0 release.

Individual component commands exist for staged maintenance:

sudo ./etlworks-cli.sh upgrade-tomcat-11
sudo ./etlworks-cli.sh upgrade-java-25
sudo ./etlworks-cli.sh upgrade-postgres-17

Use them only as part of completing the full migration. Do not leave the system between states — old application binaries cannot run on the new stack, and the CLI offers 10.x artifacts only once Tomcat 11 is in place. If you run components individually, finish with upgrade to the 10.x application before starting the service. Availability of each command depends on the CLI version, the distribution, and what is detected as already upgraded.

What the Linux migration does:

• Tomcat: the existing Tomcat is stopped and moved to /tmp/old_tomcat; Tomcat 11 is installed in its place. The connection configuration (context.xml), licenses, application.properties, and cdata.* connector files from the old Tomcat lib are transferred. Configuration keys are renamed for the modern runtime (logging.path → logging.file.path, spring.redis.* → spring.data.redis.*). Other old lib additions are not carried over — re-install regular connectors afterward (see Installing connectors).
• Java: the system Java 8 package is removed and the distribution's Java 25 package is installed. If other applications on the host share the system Java, plan for that before migrating.
• PostgreSQL: the managed local container is upgraded to 17 with an automatic dump/restore; the old container volume is moved to /tmp/old_postgres_data. An external PostgreSQL is not touched — upgrade it to major 17 with your DBA tooling before migrating.
• JVM options: a fresh managed options setter is generated with the Java 25 flags (see JVM options).

Important: material retained under /tmp (/tmp/old_tomcat, /tmp/old_postgres_data) is working space, not a durable backup — many systems clear /tmp on reboot. Your pre-migration backup is the durable copy. There is no rollback command on Linux; recovery from a failed migration uses your backup and, where intact, the retained /tmp material. The Windows rollback command does not exist on Linux.

After the migration completes, start and verify (see verification).

Windows migration

From an elevated Command Prompt, in the directory with the extracted CLI tools:

cd /d C:\etlworks
etlworks-cli.bat upgrade

The CLI detects the installed legacy runtime and plans the complete migration. Optional parameters — needed only when detection cannot identify the installation, multiple installations exist, or you want a specific version (run etlworks-cli.bat list first). Substitute your actual paths and service name, which may differ:

etlworks-cli.bat upgrade --version X.Y.Z --catalina-home "C:\Program Files\Apache Software Foundation\Tomcat 8.5" --service Tomcat8 --pg-bin "C:\Program Files\PostgreSQL\9.5\bin"

How it works:

1. Discovery. The CLI reads the runtime from the Windows service's configured JVM (not from PATH), checks the running PostgreSQL server version, and inventories what must change.
2. One approval. The CLI lists the planned runtime, required components, preservation and backup steps, and asks a single question: Proceed with this migration and all changes listed above? [y/N]. Answering anything but y cancels before anything is installed, archived, or stopped. If the local PostgreSQL is below 17, the plan includes the database migration and separately asks you to attest that the cluster belongs exclusively to this installation (see PostgreSQL upgrades).
3. Staging. Tomcat 11, Java 25, and (when needed) PostgreSQL 17 are staged alongside the existing installation. A full recovery backup (files, data, database dump, Redis snapshot, service/registry exports) is taken first.
4. Cutover. The legacy Tomcat service is stopped and set to Disabled — it is not deleted, and its Tomcat 8.5 directory and Java runtime are retained. Because of that retained installation, every later 10.x upgrade on this machine must select the modern installation explicitly with --service Tomcat11 (see Upgrade 10.x on Windows). A separate automatic Tomcat11 service is created, and the matching Tomcat 11 tray monitor replaces the old one.
5. Verification. The new application starts only after database checks, and success requires three consecutive healthy checks of /health2 plus the authenticated engine endpoint — a service that merely shows Running is not treated as success.

What is preserved: application configuration (application.properties), JWT secrets, ports, data paths, the license, certificates, logs, everything in the old Tomcat drivers directory, and cdata.jdbc* files (with their license/config companions) from the old Tomcat lib. Redis properties are written in both the legacy and Spring Boot 3 families so the app and engine cannot diverge. Other old Tomcat lib files are not carried forward — do not copy all Java 8 shared libraries into Tomcat 11; re-install regular connectors instead.

Requires review before migrating: custom APR/TLS connectors, extra web applications in the same Tomcat, external context overrides, custom service JVM agents, and unusual filesystem layouts. The CLI checks supported configuration before stopping anything, but it is not an automatic migrator of arbitrary Tomcat sites.

Earlier-CLI leftover state: if a previous migration left the Tomcat 11 runtime registered under the legacy service name, a normal modern upgrade repairs it — see the 10.x upgrade.

Unattended migration uses distinct approvals — suppressing prompts is not blanket authorization:

etlworks-cli.bat upgrade --silent --migrate --install-missing --dedicated-postgres

• --silent / --skip-confirm — run without prompts; missing approvals are errors, never guesses.
• --migrate — authorizes the Java/Tomcat migration itself.
• --install-missing — authorizes installing the listed missing components.
• --dedicated-postgres — the exclusive-ownership attestation, required only when the database migration applies.
• --archive-existing — required when an inactive target directory must be archived.
• External/shared dependencies require their own backup attestations (for example --external-redis-backed-up; shared-resource nodes use --shared-resources-backed-up).

If the migration fails after mutation begins, Tomcat remains stopped and the recovery workspace is retained — do not retry over an incomplete migration; use recovery and rollback.

Verify the migration

1. Version: sudo ./etlworks-cli.sh version / etlworks-cli.bat version shows the installed application version, date, commit, and license; About in the UI shows the application and engine versions.
2. Health: /health returns status UP (see health monitoring).
3. Sign in and confirm connections, schedules, and a representative flow while executions are still suspended.
4. Resume flow executions only after validation passes.

PostgreSQL upgrades

Etlworks 10.x requires PostgreSQL server major 17 or newer. The upgrade path depends on who manages the database:

• Managed local database (installed by the Etlworks installer) — the CLI can migrate it, as described below.
• External or shared database (RDS, Azure Database for PostgreSQL, a DBA-managed server, a database shared by multiple nodes) — the CLI never replaces or upgrades it. Upgrade it to major 17 with your own tooling before migrating the application.

Linux: managed container upgrade

The Linux installer runs PostgreSQL as a managed local Docker container. To upgrade it:

sudo ./etlworks-cli.sh upgrade-postgres --version 17

The version is an official PostgreSQL Docker image version. The command stops Tomcat, runs a backup automatically, stops the containers, moves the existing data volume to /tmp/old_postgres_data, recreates the container on the new image, and restores the dump with the application's roles, grants, and schema ownership. Start and validate the application explicitly afterward. The old volume under /tmp is working space — remove it only after acceptance, and rely on the automatic backup under /opt/etlworks/backup as the recovery copy.

This command manages only the local container arrangement. It is not an external-database upgrader.

Windows: service-based migration

Windows supports the database migration two ways: coordinated (as part of the normal upgrade, described in the migration section — no separate command needed) and standalone for a database-only maintenance window:

cd /d C:\etlworks
etlworks-cli.bat upgrade-postgres --version 17 --check
etlworks-cli.bat upgrade-postgres --version 17 --dedicated-postgres
etlworks-cli.bat start

How it works and what it requires:

• The CLI checks the running server version. An existing compatible major-17-or-newer server is reused, never downgraded; a matching major is a no-op. This command does not patch within a major.
• Automatic migration is limited to eligible dedicated local primary clusters. You must confirm the cluster belongs exclusively to this installation and all other writers are stopped — discovery cannot prove the absence of dormant external clients. Extra databases, custom extensions other than plpgsql, user tablespaces, replication, prepared transactions, preload libraries, custom TLS, or another detected application make the preflight fail: those layouts need a DBA-managed migration.
• Source PostgreSQL tools are required (--pg-bin selects them). Preserving roles, password hashes, and ownership requires a superuser: if the application user is not one, also supply --pg-admin-user and --pg-admin-password.
• The pinned PostgreSQL 17 installer from the CLI package supplies the target binaries (hash and signature checked). The new server is staged alongside the source on an unused loopback port, the database and cluster globals are dumped with target-version tools, and the restore preserves encoding, roles, password hashes, owners, grants, schemas, sequences, and data — failing on SQL errors rather than skipping them.
• Only after restore validation is the original PostgreSQL service stopped and disabled and the new server moved onto the original port. Application connection settings and credentials remain unchanged. The old cluster and the recovery workspace are retained even after success — retire them only after acceptance.
• Standalone success leaves Tomcat stopped. Start it explicitly and verify the application and flows.
• Failure blocks startup: while a PostgreSQL migration is incomplete, CLI start/restart/recover/upgrade refuse to run. Recover a failed standalone attempt with the retained workspace:

etlworks-cli.bat upgrade-postgres --recover --workspace "C:\ProgramData\Etlworks\cli\postgres-<id>" --check
etlworks-cli.bat upgrade-postgres --recover --workspace "C:\ProgramData\Etlworks\cli\postgres-<id>"

Recovery validates identities and backup hashes, disables the target service, restarts the untouched source cluster, and leaves Tomcat stopped with both clusters retained. Silent recovery requires --confirm-rollback. Recovery cannot reverse a completed migration or reconcile application writes made after it. A failed combined application + database migration uses the full rollback command instead — never database-only recovery against a partly migrated application.

Unattended standalone migration: --silent --confirm-postgres-upgrade --dedicated-postgres. --check stages and verifies the installer and runs preflight without stopping services. Future majors are supported through pinned, qualified package manifests (--prerequisites-file) — the CLI never scrapes a "latest" installer.

JVM options and configuration migration

Required Java 25 option for Snowflake and Databricks

On Java 25, the Snowflake and Databricks JDBC drivers' Arrow result readers require this JVM option:

--add-opens=java.base/java.nio=ALL-UNNAMED

Without it, Snowflake/Databricks reads fail during Arrow initialization (the first errors in the log show java.nio.Buffer.address access denied).

How the installers handle it:

• Windows: current installers add the option exactly once during fresh installation, Java 8 → 25 migration, and ordinary Java 25 artifact upgrades, while preserving unrelated service options and heap settings. Java 8 installations never receive it.
• Linux: fresh 10.x installation and the runtime migration generate a managed options setter that includes it. An ordinary artifact upgrade can retain an older setter — after upgrading the CLI or application on an existing 10.x Linux host, verify the effective options include this flag and restart if you add it.

A new WAR or CLI download alone does not give a running JVM the option — a full JVM (Tomcat) restart is required after the option is added. Keep Java 25-only options out of any Java 8 configuration, including rollback configurations.

Managed Java 25 flags and heap

On both platforms the managed Java 25 configuration enables -XX:+UseStringDeduplication and -XX:+UseCompactObjectHeaders and removes obsolete flags (-XX:+UseParallelGC, -server) and duplicates. Heap defaults are initial 2 GB and maximum 75% of physical RAM. Custom heap limits and unrelated service options are preserved across upgrades; the existing service/JVM registration is backed up before changes on Windows.

Configuration that migration changes — and preserves

• Spring Boot configuration keys: the modern runtime uses logging.file.path and the spring.data.redis.* family. Migration renames the keys in application.properties. On Windows both Redis property families (spring.data.redis.* and legacy spring.redis.*) are written and updated together, so the modern app, the engine, and a restored legacy runtime cannot diverge; reads prefer the Boot 3 key and fall back to the legacy one.
• JWT secrets are preserved. 10.x accepts your existing 16-character legacy secret as well as 64+ character secrets. Do not rotate the secret merely to migrate — all nodes and the engine must share it.
• Upload limits: the application's multipart limits (multipart.max-file-size / multipart.max-request-size, default 50MB, mapped to Spring's spring.servlet.multipart.* keys) are preserved from your existing configuration. Remember these are independent of any reverse-proxy body-size limit — raising one does not raise the other, and changed properties require an application restart. See reverse proxy and networking.
• Site-specific configuration is preserved, not replaced — ports, data paths, certificates, service accounts, and your tuned settings carry over. The migration never resets your configuration to demo defaults.
• Tomcat configuration parity: fresh installs and migrations apply the established server defaults (resource cache size, HTTP connector thread limit and compression, JAR-scanner scope, and datasource validation/abandoned-connection settings). Existing explicit site values for these settings are preserved; artifact upgrades only add missing entries.

Service control and monitoring with the host CLI

Service control targets Tomcat (the Etlworks application) — not PostgreSQL or Redis, which are managed independently. Keep the different questions distinct: service status says whether Tomcat runs; version says what is installed; /health says whether the application and its dependencies are actually healthy; and flow-execution state is visible in the UI — none of these implies the others.

Service control on Linux

The Linux CLI is installed as /opt/etlworks/etlworks-cli.sh and is invoked with sudo from the install directory:

cd /opt/etlworks
sudo ./etlworks-cli.sh status
sudo ./etlworks-cli.sh version
sudo ./etlworks-cli.sh cli-version
sudo ./etlworks-cli.sh start
sudo ./etlworks-cli.sh stop
sudo ./etlworks-cli.sh restart
sudo ./etlworks-cli.sh log --lines 200
sudo ./etlworks-cli.sh log --engine --follow

stop and restart accept --skip-confirm (alias -s).

Logs. log prints the application log (catalina.out); --engine (alias -e) prints the ETL engine log (etlprocess.log) instead. --follow (alias -f) tails live — stop following with Ctrl+C. --lines N (alias -n) limits output. Log files live under TOMCAT_HOME/logs; on legacy Amazon Linux 2 installations the application log is read from journald instead.

Service control on Windows

Run from an Administrator PowerShell or Command Prompt — required because Tomcat is a Windows service:

cd /d C:\etlworks
etlworks-cli.bat status
etlworks-cli.bat version
etlworks-cli.bat cli-version
etlworks-cli.bat start
etlworks-cli.bat stop
etlworks-cli.bat restart
etlworks-cli.bat log --lines 200
etlworks-cli.bat log --engine --follow

The service start/stop/restart/status commands operate on the Tomcat Windows service only and do not read or connect to Redis. The Tomcat 11 tray monitor (Tomcat11w.exe) remains available for interactive control; logs are in the Tomcat logs directory, by default C:\Program Files\Etlworks\Tomcat11\logs on 10.x (legacy 9.x: C:\Program Files\Apache Software Foundation\Tomcat 8.5\logs).

Automation. All commands return a nonzero exit status on failure and support unattended use with the documented approval flags. Pass credentials only through the documented options or configuration — do not invent environment-variable or prompt-free credential paths. There is no generic "monitor" command; use status, log --follow, and the /health endpoint together.

Maintenance

Maintenance for an on-premise Etlworks instance covers service control (above), suspension, and backup and restore — protecting PostgreSQL, Redis, and the app.data folder against host failure, data corruption, ransomware, and bad upgrades. The instructions assume you're an admin on the host. Linux commands assume the /opt/etlworks layout; Windows commands assume the standard installer layout (10.x: C:\Program Files\Etlworks\Tomcat11; legacy 9.x: C:\Program Files\Apache Software Foundation\Tomcat 8.5) with PostgreSQL and Redis as Windows services.

Which services must be running for each maintenance task

Etlworks depends on three services on the host: Tomcat (the Etlworks application), PostgreSQL (metadata database), and Redis (shared runtime state, queues, cache, and the persistent flow-execution suspend flag). Different maintenance tasks need different combinations up while they execute.

The most common source of confusion is whether Tomcat needs to be up for backup, restore, and suspend/resume. The short answer: only when the task is invoked through the UI or the built-in Etlworks CLI. Tasks invoked through the host CLI talk to PostgreSQL and Redis directly and do not need Tomcat — which is what makes them the right choice during restore windows, planned outages, and multi-node maintenance. The one exception is transient suspension, which messages the running application and therefore requires it.

Rule of thumb. For any task that touches the contents of PostgreSQL or Redis — backup, restore, or the persistent suspend flag — PostgreSQL and Redis must be up. Tomcat is only required for tasks that run through the UI, the built-in CLI, or transient suspension.

Suspend and resume flow executions during maintenance

Before using this workflow, download or update the latest CLI tools from etlworks.com/downloads/tools.html — an out-of-date CLI package will not have these commands.

Suspending flow executions blocks new flow runs from starting, without stopping the application. Operators use this before upgrades, migrations, restore testing, and emergency maintenance windows — and especially after restoring from backup, because schedules can otherwise wake up and start firing the moment Tomcat starts. Persistent suspend lets you bring the application online, validate the restored system, and only then re-enable executions.

Behavior at a glance

• Suspend blocks new flow executions. Currently running flows are not stopped automatically. To also request stopping running flows, use the transient --stop-flows variant from the host CLI or the force variant in the app — and verify in the UI that they actually stopped: publishing a stop request is not confirmation that flows have stopped.
• Persistent suspension is stored in Redis and survives application restart. It must be cleared explicitly. The host CLI commands set and clear this flag by default and work while Tomcat is stopped.
• Transient suspension lives in the running application. It clears automatically on restart. Transient commands publish to the running application and require it to be up and subscribed to Redis.
• Persistent and transient state are separate. The host CLI's resume-flows clears the persistent flag; resume-flows --transient clears the transient state. flow-suspend-status reports only the persistent flag and ignores transient suspension. In the application UI, the Resume action clears the suspension it shows.
• Multi-node deployments share persistent state through Redis: one persistent suspend affects every node using the same Redis and prefix.
• The UI reflects CLI-initiated changes. When a persistent suspend is set from the CLI while Etlworks is running, the suspended badge appears in the top bar and Settings → Flows shows Resume. Nodes may take a few seconds to reflect changes because the app refreshes the Redis-backed state periodically.

Persistent vs. transient suspend

Suspend and resume from the UI

1. Go to Settings → Flows.
2. Click Suspend flow executions.
3. If prompted, choose whether to stop or drain running flows in addition to blocking new ones.
4. To make the suspension survive a restart, enable Keep suspended after restart until resumed manually.
5. Click Suspend. The suspended badge appears in the top bar of the UI.
6. When you are ready to re-enable executions, return to Settings → Flows and click Resume flow executions.

Built-in Etlworks CLI commands

These commands are available from the built-in Etlworks CLI while the application is running. They operate on the same suspend state as the UI, so a CLI change is visible in the UI and vice versa.

Host CLI (Linux)

The host CLI operates on the persistent Redis-backed flag directly, so it works even when Tomcat is stopped, as long as Redis is running and reachable:

cd /opt/etlworks
sudo ./etlworks-cli.sh suspend-flows
sudo ./etlworks-cli.sh flow-suspend-status
sudo ./etlworks-cli.sh resume-flows

Transient variants (require the application to be running and subscribed to Redis):

sudo ./etlworks-cli.sh suspend-flows --transient
sudo ./etlworks-cli.sh suspend-flows --transient --stop-flows
sudo ./etlworks-cli.sh resume-flows --transient

--stop-flows is valid only together with suspend-flows --transient. It sends a stop signal to currently running flows; verify in the UI that they stopped.

Redis connection options. By default the CLI reads the connection from application.properties, preferring the Spring Boot 3 keys and falling back to the legacy keys. Override any value on the command line:

• --redis-host HOST
• --redis-port PORT
• --redis-password PASSWORD
• --redis-ssl true|false
• --redis-prefix PREFIX

Host CLI (Windows)

Run from an Administrator PowerShell or Administrator Command Prompt:

cd /d C:\etlworks
etlworks-cli.bat suspend-flows
etlworks-cli.bat flow-suspend-status
etlworks-cli.bat resume-flows
etlworks-cli.bat suspend-flows --transient
etlworks-cli.bat suspend-flows --transient --stop-flows
etlworks-cli.bat resume-flows --transient

The Redis override options, property precedence, and persistent/transient semantics are the same as on Linux. Windows transient suspension uses the same Redis message protocol as Linux, so mixed-platform tooling behaves identically.

Recommended workflow: upgrade or migration

1. Update the CLI tools from etlworks.com/downloads/tools.html.
2. Suspend new executions persistently: UI — Settings → Flows → Suspend flow executions with Keep suspended after restart until resumed manually; Linux — sudo ./etlworks-cli.sh suspend-flows; Windows — etlworks-cli.bat suspend-flows.
3. Drain or stop currently running flows, and verify in the UI that nothing is still running.
4. Run Prepare state for backup if applicable.
5. Run the backup and verify the archive exists and its size is plausible.
6. Perform the upgrade or migration.
7. Start Etlworks.
8. Validate the instance while it is still suspended — sign in, check schedules, open a representative flow, confirm connections respond. Do not run a test flow while suspension still blocks execution; validation that requires executing a flow happens after the deliberate resume.
9. Resume executions only after validation passes: UI — Settings → Flows → Resume flow executions; Linux — sudo ./etlworks-cli.sh resume-flows; Windows — etlworks-cli.bat resume-flows.

Recommended workflow: restore

1. Restore PostgreSQL, Redis, and app.data using the documented restore command — see Restore with Linux CLI or Restore with Windows CLI. The host CLI stops Tomcat and leaves it stopped.

2. Before starting Tomcat, set or verify the persistent suspend from the host CLI so no schedules fire immediately after startup. This matters after a Redis restore in particular: the restored Redis contents overwrite whatever suspension flag was set earlier, so re-check it after the restore:

   sudo ./etlworks-cli.sh flow-suspend-status   # Linux
   sudo ./etlworks-cli.sh suspend-flows
   etlworks-cli.bat flow-suspend-status           :: Windows
   etlworks-cli.bat suspend-flows

3. Start Tomcat.
4. Sign in and validate the instance while it is suspended: schedules, flows, Redis-backed state, representative connections.
5. Resume executions only after validation: UI — Settings → Flows → Resume flow executions; Linux — sudo ./etlworks-cli.sh resume-flows; Windows — etlworks-cli.bat resume-flows.

Multi-node deployments

• Persistent suspend is stored in Redis, so one suspend command affects every application node that shares the same Redis and prefix.
• Run suspend-flows once per shared Redis environment, not once per node.
• Each node may take a few seconds to display the suspended badge and to actually block new executions, because nodes refresh the Redis-backed state on a short interval.
• Resume from any node (UI or CLI) clears the shared Redis flag; other nodes see the change on their next refresh.

Where the persistent flag lives (advanced)

Persistent suspend is stored in Redis under the configured Redis prefix. The default key is:

integrator.conf.system.flow.executions.suspended

If your deployment uses a non-default Redis prefix, substitute it for integrator.

Do not edit this key manually except as a last-resort support-guided step. Prefer the UI or the CLI — both apply the change consistently, refresh the application state, and update the UI. Manual Redis edits are not synchronized to nodes any faster than the CLI, they do not update the UI, and they bypass audit logging.

Backup and restore overview

Etlworks supports three independent ways to create a backup.

Each backup contains up to three data sections: PostgreSQL (instance database), Redis (orchestration state), and app.data (filesystem state). Host CLI backups additionally capture the application WARs and the shared drivers. Restore picks up whatever sections the chosen restore path consumes — an archive containing WARs or drivers does not mean every restore path restores them (details per platform below).

Restore is destructive. It replaces the live PostgreSQL database, Redis state, and (when included) the app.data directory. app.data is not merged. See app.data backup/restore notes before restoring.

Host restore leaves Tomcat stopped on both platforms. Validate the restored state, set or verify persistent suspension, then start the service explicitly.

Back up from the Etlworks UI

The in-app Backup instance data task creates a portable .zip archive that contains PostgreSQL, Redis, and app.data sections, plus a manifest. The archive can be restored on the same host or moved to another host and restored by either host CLI.

Configure the in-app backup task

The task is not added automatically. You must add it explicitly:

1. Sign in as a super-admin.
2. Open Settings → Maintenance.
3. Click Add task and pick Backup instance data.
4. Configure the parameters described below.
5. Save. The task can be left without a schedule and run manually, or scheduled to run automatically.

Recommended schedule: once daily during a low-activity window — for example 0 2 * * * (2:00 AM). This is a recommendation, not a requirement.

Key parameters

The archive is named:

etlworks-backup-YYYYMMDD-HHMMSS-<id>.zip

What is included in app.data backup

Standard app.data folders (recommended). Captures the operationally important files without blindly copying high-volume transient event data. The selectable standard folders are keys, logs, rag, debezium-data, debezium-events, metrics, metadata, and edi. By default all are included except debezium-events, which when enabled includes only csv, zip, gzip/gz, and json event files. The Additional app data includes field accepts extra relative folder or file glob patterns, one per line — for example logs/**, **/logs/**, **/staging.dat — applied after the standard folder toggles.

Back up entire app.data. Includes every regular file under app.data. Use with care — app.data can be hundreds of gigabytes in production. Preview the file count and total size in the UI before enabling.

Locked / unreadable files (Windows in particular). Some files may be held by running processes when the backup runs. By default the task skips unreadable files and records a warning rather than failing the entire backup. Review the task result for the skipped list and decide whether to re-run during a quieter window.

Backup retention and backup health notifications

Retention, the no-recent-backup warning, and notification emails are configured on the task as described above. Webhook events are configured separately under the regular Etlworks webhook settings (super-admin only): Backup success, Backup failure, and Backup warning (archive created but some files skipped).

Backup progress and long-running backups

The in-app task reports progress while it runs, at the stage level: PostgreSQL dump, Redis backup, app.data archive, cleanup/retention, notifications. Progress within the app.data stage is approximate — operations stream straight into the archive. Full app.data backups on a multi-hundred-GB installation can run for tens of minutes to hours; schedule them during a quiet window and ensure the location has space for the new archive plus older archives still within retention.

Back up with Linux CLI

Note: the Linux host backup works against the managed local PostgreSQL/Redis container layout created by the Linux installer. For external or shared databases, use the in-app task with a suitable profile, or your own database tooling.

Run from /opt/etlworks with sudo. The backup captures the PostgreSQL dump, the Redis RDB snapshot, the application WARs, the shared drivers, and (optionally) app.data. The Redis section copies the container's existing dump.rdb snapshot — it does not force a fresh Redis save; for a maximally current Redis state, run the in-app Prepare state for backup first.

Basic backup (database, Redis, WARs, drivers).

cd /opt/etlworks
sudo ./etlworks-cli.sh backup --skip-confirm

Include the full app.data tree.

sudo ./etlworks-cli.sh backup --app-data --skip-confirm

Include only the standard app.data folders (same scope as the in-app task).

sudo ./etlworks-cli.sh backup --app-data-standard --skip-confirm

Fail the entire backup if any app.data file can't be read (strict mode).

sudo ./etlworks-cli.sh backup --app-data --fail-on-locked-files --skip-confirm

Create a portable CLI archive (zip).

sudo ./etlworks-cli.sh backup --app-data-standard --cli-archive /backups/etlworks-cli-backup.zip --skip-confirm

Without --cli-archive, the CLI leaves a folder-based backup at /opt/etlworks/backup (subfolders postgres/, redis/, webapps/, drivers/, app-data/), overwriting the previous folder backup. --backup-path DIR selects a different folder. With --cli-archive and no explicit --backup-path, only the ZIP is persisted and the temporary staging folder is removed; supplying an explicit --backup-path keeps the folder backup alongside the ZIP.

Restore with Linux CLI

Warning — restore is destructive. It replaces the live PostgreSQL database, Redis state, and (when included) app.data. The CLI stops Tomcat first and leaves it stopped when the restore completes — validate, set or verify persistent suspension, then start explicitly.

Restore from an in-app backup archive.

cd /opt/etlworks
sudo ./etlworks-cli.sh restore --archive /backups/etlworks-backup-YYYYMMDD-HHMMSS-id.zip --skip-confirm

Restore from a CLI zip archive.

sudo ./etlworks-cli.sh restore --cli-archive /backups/etlworks-cli-backup.zip --skip-confirm

Restore from a folder-based CLI backup (explicit per-section paths).

sudo ./etlworks-cli.sh restore \
    --postgres /path/to/integrator_dump.sql \
    --redis /path/to/dump.rdb \
    --app-data /path/to/app-data \
    --skip-confirm

The Linux restore consumes the PostgreSQL, Redis, and app.data sections. WARs and drivers present in a CLI backup are not redeployed by the Linux restore — they are reference copies; redeploy application artifacts with upgrade if needed. Restoring a backup made on a Windows installation requires PostgreSQL tools compatible with that dump on the Linux host.

Preserve the existing app.data tree before overwriting (requires at least 2x app.data free space):

sudo ./etlworks-cli.sh restore --archive /backups/etlworks-backup.zip --skip-confirm --keep-pre-restore-app-data

With --keep-pre-restore-app-data, the CLI renames the existing app.data to <app-data-path>.pre-restore.<timestamp> before extracting the backup; without it the existing app.data is deleted. After validating the restore, delete the .pre-restore. directory manually.

Back up with Windows CLI

Run from an elevated PowerShell or Administrator Command Prompt — the CLI starts and stops Windows services and reads data under protected paths.

Standalone backup/restore discover the installed PostgreSQL executables automatically; with multiple installations select them with --pg-bin (or per-tool --pg-dump/--pg-psql/--pg-restore). Connection defaults come from the selected Tomcat's jdbc/integrator datasource in conf/context.xml; overrides: --pg-host, --pg-port, --pg-database, --pg-user, --pg-password, --pg-sslmode. Passwords are passed through a scoped process environment, never as subprocess arguments.

Windows CLI backups and restores include the application WARs (ROOT.war and etl.war) in addition to the database, Redis, drivers, and optional app.data.

cd /d C:\etlworks
etlworks-cli.bat backup --app-data-standard --cli-archive "D:\Backups\etlworks-cli-backup.zip"
etlworks-cli.bat backup --app-data --cli-archive "D:\Backups\etlworks-cli-backup.zip"
etlworks-cli.bat backup --app-data --fail-on-locked-files --cli-archive "D:\Backups\etlworks-cli-backup.zip"

As on Linux, without an explicit backup folder the CLI stages, zips, and removes the staging; an explicit backup folder is retained alongside the ZIP. Locked or unreadable app.data files are skipped and reported by default; add --fail-on-locked-files when you need every file or none.

Restore with Windows CLI

Warning — restore is destructive. Run from an Administrator shell. The CLI stops Tomcat first (with --skip-stop-tomcat it instead verifies Tomcat is already stopped before touching any data) and leaves Tomcat stopped afterwards — validate the restore and start the service explicitly.

cd /d C:\etlworks
etlworks-cli.bat restore --cli-archive "D:\Backups\etlworks-cli-backup.zip" --dry-run
etlworks-cli.bat restore --cli-archive "D:\Backups\etlworks-cli-backup.zip"
etlworks-cli.bat restore --archive "D:\Backups\etlworks-backup-YYYYMMDD-HHMMSS-id.zip"
etlworks-cli.bat start

What the Windows restore does and does not do:

• WARs present in the backup are validated and restored; each restored WAR's matching expanded application is retired, and unrelated webapps are retained. The prior WARs and expanded folders are kept under %ProgramData%\Etlworks\cli\war-restore-<id>. Backups without WARs leave webapps unchanged.
• Restore does not migrate Java or Tomcat and does not restore shared drivers, even though drivers are included in CLI backups. It is not a complete machine backup.
• If a file replacement fails, the CLI attempts to put the prior files back, but it does not roll back the database, Redis, or app.data restoration already performed. Keep Tomcat stopped on failure and review the reported recovery path.
• Verify restored WAR/database/runtime compatibility before starting — do not start a restored 9.x WAR set on a Tomcat 11 runtime or vice versa.

Redis restore on Windows. Logical restore requires Redis running and reachable. RDB restore stops and restarts the Redis service around the dump.rdb replacement; if the service can't be stopped or the file can't be replaced (access denied), rerun from an Administrator shell.

Restore from an in-app backup archive

In-app archives (etlworks-backup-YYYYMMDD-HHMMSS-<id>.zip) restore from either host CLI through --archive:

sudo ./etlworks-cli.sh restore --archive /path/to/etlworks-backup-*.zip --skip-confirm
etlworks-cli.bat restore --archive C:\path\to\etlworks-backup-*.zip

Whatever data sections the archive contains (PostgreSQL, Redis, app.data) are restored; sections missing from the archive are left untouched on the target. In-app archives normally contain only data; if a webapps/ payload is present, the Windows restore also restores it.

Restore from a CLI backup archive

CLI archives (the --cli-archive output) restore through --cli-archive; folder-based CLI backups restore through explicit per-section paths (Linux: --postgres / --redis / --app-data). The CLI archive format differs from the in-app archive format, but the restore command accepts either kind through its matching flag.

PostgreSQL backup/restore notes

Backup uses pg_dump. The in-app task exposes a configurable pg_dump command with built-in profiles for common deployment shapes; for non-standard installs (RDS, Azure Database for PostgreSQL, Cloud SQL, custom Docker, non-PATH installations) use the custom profile and override the host, port, database, user, password, container, and command as needed. The instance database is integrator by default.

Command tokens (when the pg_dump command uses them, it is treated as a complete command): {host}, {port}, {database}, {username}, {container}, {dumpFile}, {extraArgs}. Docker example:

docker exec -e PGPASSWORD {container} pg_dump --username {username} {database}

Restore uses psql / pg_restore as appropriate and applies the required grants and ownership after the database is loaded. Use client tools compatible with the server and the dump format.

Choosing a PostgreSQL backup profile

Explicit fields always override profile defaults. If the backup fails with "No such container", pick the profile whose container name matches (docker ps) or switch to custom; with "pg_dump not found", supply the full pg_dump path.

Redis backup/restore notes

Cross-version warning. Logical restore requires payload compatibility between the source and target Redis versions, and RDB files are also version-sensitive — switching to rdb mode does not automatically resolve every cross-version incompatibility. If you change Redis versions during a migration, test the restore in staging first. For managed Redis (AWS ElastiCache, Azure Cache for Redis, …) check the supported version against your backup source.

app.data backup/restore notes

app.data can be very large. Hundreds of GB is not unusual on busy CDC installations. Treat it accordingly:

• Default restore deletes the existing app.data directory and extracts the backup directly into the target. This is intentional — it avoids requiring 2x app.data disk space.
• Archives are streamed directly into the target, not first extracted into a temporary folder.
• To keep a copy of the old app.data for rollback safety, use --keep-pre-restore-app-data — the existing app.data is renamed to <app-data-path>.pre-restore.<timestamp>. Only use this when the host has enough free disk for both copies.
• You must delete the .pre-restore. directory manually after validating the restore. The CLI never deletes it.

For day-to-day backups, prefer Standard folders over Entire app.data.

Cross-platform compatibility matrix

A portable archive format does not guarantee compatibility across arbitrary Redis versions, paths, runtimes, or database versions — validate cross-platform and cross-version restores in staging.

Recommended production procedures

Recommended backup plan

• Schedule the in-app backup task daily during a quiet window.
• Store backups outside the app.data directory, ideally on a different disk than the live instance.
• Set retention based on disk size and recovery-point objective, and set Warn if no backup for days.
• Enable Email on failure and the backup-failure webhook event; enable Email on success only when needed for compliance.
• Test restore on a staging instance at least once per quarter.
• For very large app.data installations, prefer Standard folders over Entire app.data.

Before an upgrade or migration

1. Suspend flow executions persistently (see Suspend and resume).
2. Drain or stop running flows.
3. If applicable, run the in-app Prepare state for backup to flush Redis state.
4. Run a backup — in-app task or host CLI.
5. Verify the archive exists and the size is plausible, and keep a copy independent of the host.
6. Proceed with the upgrade or migration.
7. If a restore is needed, follow the restore procedure.

Restore procedure

1. Run the restore command from an Administrator / elevated shell (the CLI stops Tomcat for you).
2. Wait for the restore to complete — it covers the data sections available in the archive.
3. Before starting Etlworks, set or verify persistent suspension from the host CLI — a restored Redis can overwrite the earlier suspension flag.
4. Start Tomcat / Etlworks explicitly (host restore leaves it stopped).
5. Validate while suspended: sign in, check schedules, flows, Redis-backed state, representative connections.
6. Resume executions only after validation passes, then run a representative flow as the final check.
7. Only after validation, delete any .pre-restore.<timestamp> app.data directory left behind.

Recovery, rollback, and uninstall (Windows)

These are distinct operations — pick the one that matches the situation:

Shared rules: recovery workspaces and journals live under %ProgramData%\Etlworks\cli\ (setup-<id>, postgres-<id>, uninstall-*, clean-<id>). After a failure that changed application files, Tomcat remains stopped for controlled recovery — startup may already have modified the database schema, so the CLI never claims that restoring old binaries alone is safe. External and shared dependencies are protected: the CLI never drops a remote database, flushes a remote Redis, or restores an external Redis itself. No operation can undo writes already sent to external systems.

recover

cd /d C:\etlworks
etlworks-cli.bat recover --check
etlworks-cli.bat recover --service Tomcat11 --catalina-home "C:\Program Files\Etlworks\Tomcat11"

Recovery starts the service if stopped, verifies readiness (three consecutive healthy checks), and records completion without reinstalling or redeploying artifacts. Unattended: --silent --confirm-recovery. An incomplete deployment journal routes to recovery validation before another deployment is allowed.

rollback: failed Java 8 → 25 migration

rollback restores the pre-migration Java 8 installation. It applies only to a failed or interrupted migration whose recovery workspace was created by CLI 1.3.7 or newer — that CLI records the machine-bound rollback manifest, file inventory, service identities, original Java home, Redis identity, and permission snapshots the command requires. Use CLI 1.3.7+ when performing the migration, not just when rolling back; the CLI will not guess missing metadata from older workspaces (use the manual recovery guidance from support in that case).

From an Administrator prompt, with the exact workspace path printed by the failed migration:

cd /d C:\etlworks
etlworks-cli.bat rollback --workspace "C:\ProgramData\Etlworks\cli\setup-<id>" --check
etlworks-cli.bat rollback --workspace "C:\ProgramData\Etlworks\cli\setup-<id>"

The preview validates the backup inventory, original Java 8 and service identity, PostgreSQL dump/tools/connectivity, and Redis configuration without changing anything. It refuses stale workspaces superseded by another deployment, backups from another machine, changed service accounts, missing snapshots, and already successful or recovered migrations. After confirmation, rollback:

1. Stops and disables the target Tomcat11 service, stops the original service, and backs up the current database into a new attempt directory inside the workspace.
2. Restores the entire original runtime (both WARs, expanded apps, configuration, licenses, drivers, native files, logs), the app data, and their permissions. Current trees are moved aside, not deleted.
3. Restores the matching PostgreSQL dump and the dedicated local Redis RDB snapshot — this replaces data with its pre-migration snapshot; stop other writers and coordinate external database users first.
4. Restores the original service registration (JVM parameters with Java 25-only options removed, executable path, account, display name, startup mode).
5. Verifies the service points at the original Java 8 / Tomcat 8 runtime, starts it, and runs app/engine health checks. Only then is the migration recorded as rolled back.

Requirements and behavior to know:

• The original Windows service must still exist — rollback preserves its account; it does not recreate deleted services.
• Local Redis must be dedicated to this installation, running, and configured for RDB persistence. External/shared Redis is never restored by the CLI: stop Tomcat, have its owner restore the matching snapshot, and confirm that step explicitly (silent mode: --external-redis-restored instead of --dedicated-redis).
• Unattended execution: --silent --confirm-rollback --dedicated-redis.
• --no-start completes the restoration but leaves Tomcat stopped for inspection; start manually after checking compatibility.
• On failure, everything is retained and start/restart/recover/upgrade stay blocked; fix the reported cause and retry with the same workspace. A completed rollback is idempotent.
• PostgreSQL server software is not downgraded or uninstalled; the inactive Java 25 runtime remains on disk for inspection.

Rollback is not a downgrade command. A successful migration deletes its temporary backup after health checks pass — after that cleanup there is no rollback window. Keep an independent backup if you need the option of returning to 9.x after a successful migration, and remember that restoring old binaries after the new application has run can be unsafe because of schema changes.

uninstall and uninstall --clean

Data-preserving uninstall — retire the application service and runtime while keeping databases, data, licenses, logs, and recovery backups:

cd /d C:\etlworks
etlworks-cli.bat uninstall --check
etlworks-cli.bat uninstall

The service registration is exported, only the selected Etlworks Tomcat service is stopped and removed, and its runtime directory is archived beside the original as <home>.retired-<timestamp>-<id>. This is not a database or data reset — a later install reuses the existing databases. Unattended: --silent --confirm-uninstall.

Clean uninstall (destructive, opt-in) — remove the dedicated installation including live data:

cd /d C:\etlworks
etlworks-cli.bat uninstall --clean --service Tomcat11 --check
etlworks-cli.bat uninstall --clean --service Tomcat11

Read every DELETE and RETAIN line of the preview. Interactive execution requires typing CLEAN Tomcat11 exactly — this also attests that the listed PostgreSQL/Redis instances are dedicated to this installation. Sharing cannot be fully inferred from local configuration: never authorize dependency removal when another application uses them. For the default dedicated installation, cleanup removes the Tomcat service and runtime, the local etlworks-postgresql-17 service and PostgreSQL through its vendor uninstaller, the default Redis service, the private Java 25 runtime, and the configured application-data directory. Retained always: license files, anything matching backup naming (backup paths, .bak/.dump/.zip files — copied with hash verification into %ProgramData%\Etlworks\cli\clean-<id>\retained), existing recovery workspaces, retired runtime archives, and the CLI itself. External, shared, or ambiguous dependencies are refused, not guessed. The separate Integration Agent is never a cleanup target.

Unattended cleanup requires --silent --confirm-clean --dedicated-dependencies. If interrupted, keep the printed receipt, do not reinstall yet, resolve the reported error, and resume with --clean-resume "C:\ProgramData\Etlworks\cli\clean-<id>".

Recovery on Linux

The Linux CLI has no rollback or uninstall commands. Recovery from a failed Linux migration or upgrade uses your pre-migration backup (restore) together with the retained material under /tmp (/tmp/old_tomcat, /tmp/old_postgres_data) while it still exists — remember /tmp is not durable. Contact Etlworks support before attempting a manual Linux rollback: running old binaries against a newer database schema can be unsafe.

Multi-node deployments

Etlworks can run as a symmetrical, horizontally scalable cluster sharing PostgreSQL, Redis (same prefix), the application data directory, and the JWT secret. Read how to provision a multi-node Etlworks cluster on AWS.

Installing and upgrading shared-resource nodes

Use --shared-resources --external-db for every node of a cluster. In this mode the CLI installs or updates Java, Tomcat, WARs, drivers, and native files only — it does not install PostgreSQL or Redis, does not create the shared application data directory, does not copy it into the node recovery workspace, and does not change its ownership or ACLs. The shared data directory must already exist and be writable by the service identity.

Linux node:

sudo ./etlworks-cli.sh install --shared-resources --external-db \
  --conf-postgres-url jdbc:postgresql://db-host:5432/integrator \
  --conf-postgres-username postgresql --conf-postgres-password PASSWORD \
  --conf-redis-host redis-host \
  --conf-app-data /mnt/etlworks/app-data \
  --conf-jwt-secret JWT_SECRET

Windows node:

etlworks-cli.bat install --shared-resources --external-db --pg-bin "C:\PostgreSQL-tools\bin" --conf-postgres-url jdbc:postgresql://db-host:5432/integrator --conf-postgres-username postgresql --conf-postgres-password PASSWORD --conf-redis-host redis-host --conf-app-data "\\fileserver\etlworks\integrator-data" --conf-jwt-secret JWT_SECRET

On Windows, shared-resource Tomcat runs as NT AUTHORITY\NetworkService. For a domain SMB share, grant the node's machine account (DOMAIN\COMPUTER$) the required share and NTFS permissions before setup — the installer verifies reachability but deliberately does not take ownership or rewrite shared ACLs.

Upgrading a node:

etlworks-cli.bat upgrade --shared-resources --external-db --pg-bin "C:\PostgreSQL-tools\bin"

Interactive upgrade requires confirmation that PostgreSQL, Redis, and the shared app data have a coordinated external backup; unattended upgrade adds --shared-resources-backed-up. The node recovery workspace contains only the local Tomcat/service backup — it is not a rollback backup for shared data. Automatic Windows Java 8 → 25 migration is rejected in shared-resource mode: database migration, writer quiescence, compatibility, and rollback must be coordinated for the whole cluster (plan it with support). Coordinate downtime and release compatibility across all nodes — mixed-version rolling operation is not promised to be safe. Each node needs the license installed and validates independently; all nodes share the same JWT secret.

AWS: long-running flows and idle TCP timeouts

AWS Nitro v6 instances, including M8a, use a default idle established-TCP connection-tracking timeout of 350 seconds when the ENI has no custom value. A long-running Etlworks flow can execute for a long time while producing no HTTP traffic, so AWS may expire the app-to-engine connection even though the engine continues running. The two configurations below keep that connection alive.

Configure EC2 network interfaces

Apply this to every ENI participating in the app → HAProxy → engine communication path, particularly ENIs attached to Nitro v6 Etlworks nodes.

1. Find an instance's ENIs.

aws ec2 describe-instances \
  --instance-ids i-xxxxxxxxxxxxxxxxx \
  --query 'Reservations[0].Instances[0].NetworkInterfaces[].NetworkInterfaceId' \
  --output text

2. Inspect the current setting.

aws ec2 describe-network-interfaces \
  --network-interface-ids eni-0123456789abcdef0 \
  --query 'NetworkInterfaces[0].ConnectionTrackingConfiguration'

How to read the result: null means no custom value is configured; on M8a / Nitro v6, null means the 350-second default applies; a displayed TcpEstablishedTimeout value is measured in seconds.

3. Set the maximum five-day timeout.

aws ec2 modify-network-interface-attribute \
  --network-interface-id eni-0123456789abcdef0 \
  --connection-tracking-specification TcpEstablishedTimeout=432000

4. Verify. Run the inspect command from step 2 again. Expect:

{
  "TcpEstablishedTimeout": 432000
}

AWS Console path. EC2 → Network Interfaces → select the ENI → Actions → Modify idle connection tracking timeout → set TCP established timeout to 432000 → Save.

Notes. This change does not require an EC2 reboot. It does not recover connections whose tracking state has already expired. Launch templates, Auto Scaling configuration, or Infrastructure as Code must also be updated so replacement nodes receive the same setting.

References: EC2 security group connection tracking — timeouts, Modify network interface attributes.

Configure HAProxy TCP keepalives

Increasing the ENI timeout alone is insufficient for flows that may run indefinitely. HAProxy should send TCP keepalive probes on both the app-facing and engine-facing connections.

Add the following to the applicable HAProxy defaults section:

option clitcpka
option srvtcpka

clitcpka-idle  240s
clitcpka-intvl 10s
clitcpka-cnt   3

srvtcpka-idle  240s
srvtcpka-intvl 10s
srvtcpka-cnt   3

The 240-second idle interval is below AWS's 350-second default. Keepalive probes must be enabled independently on both sides because HAProxy does not forward TCP keepalive probes between its client and server connections.

1. Validate the complete configuration.

sudo haproxy -c \
  -f /etc/haproxy/haproxy.cfg \
  -f /etc/haproxy/conf.d

2. Reload HAProxy.

sudo systemctl reload haproxy

Notes. A server reboot is not required. The keepalive settings apply to newly created connections. Existing stuck executions must be terminated and rerun. After configuration, verify with a flow that runs longer than 350 seconds.

Reference: HAProxy configuration — option clitcpka.

Reverse proxy and networking

When Etlworks runs behind nginx, HAProxy, a load balancer, or another reverse proxy:

• DNS and Home URL. Set Home URL to the public URL users and Integration Agents actually reach. The hostname must resolve and be reachable from every client — including Agent containers; a proxy cannot repair a URL that resolves to an unreachable endpoint.
• TLS and forwarding. Terminate TLS at the proxy or at Tomcat, and forward the original Host header to the application.
• Licensing connectivity is outbound from the Etlworks hosts to lic.etlworks.com:443 and does not pass through your inbound proxy.
• Inactivity timeouts are not flow failures. Proxy read/send timeouts measure gaps between upstream I/O operations, not the total lifetime of a flow. An upstream timeout can close an idle HTTP request, but it does not itself change the engine's flow state. For long-running and CDC flows, raise the proxy's read/send timeouts (some deployments use values as high as 24 hours) and configure TCP keepalives — these are deployment-specific choices, not universal defaults. Evaluate every proxy/load-balancer hop in the path.
• Upload limits exist at two layers. The proxy body-size limit (for example nginx client_max_body_size) and the application multipart limits (multipart.max-file-size / multipart.max-request-size, default 50MB) are independent — configure both for the largest intended upload, then restart the application after changing its properties.

Run Etlworks in Docker

To run the full Etlworks application in Docker, use our official image. This is also the supported path on macOS.

Do not confuse the two ways Docker appears in this article: a full-app Docker deployment runs the entire application as a container from the official image, while the standard Linux native installation runs Tomcat natively and uses Docker only for its local PostgreSQL and Redis containers. The host CLI commands in this article manage the native installation; a full-app Docker deployment is upgraded by replacing the image, with persistent state on mounted volumes.

Installing connectors

Etlworks comes with hundreds of built-in connectors. It is also possible to add new regular or premium connectors as needed. Connectors are installed into TOMCAT_HOME/lib (10.x default: C:\Program Files\Etlworks\Tomcat11\lib on Windows, /usr/share/tomcat/lib on Linux).

Note: In a multi-node setup the connector and its license must be installed on each node. A restart is required after installation on each node.

Preservation during upgrades and migration:

• Windows: artifact upgrades leave lib untouched and retain everything in drivers. The Java 8 → 25 migration carries cdata.jdbc* connector files (with their license/config companions) from the old Tomcat lib into the new one; other custom lib JARs must be re-installed into the new Tomcat 11 lib. Do not bulk-copy all old Java 8 shared libraries into Tomcat 11 — old library versions can be incompatible with the modern runtime.
• Linux: upgrades replace drivers and webapps but leave lib untouched. The migration to Tomcat 11 carries over cdata.* files from the old lib; re-install other regular connectors afterward.
• Keep the vendor license files (.lic) together with their connector JARs; they are preserved and carried the same way.

Installing Regular Connectors

Step 1. Obtain the connector: Contact Etlworks support to obtain the connector. It will be provided as a .jar file and may include a license file and other required files.

Step 2. Copy the files: Copy all files obtained from Etlworks support to TOMCAT_HOME/lib.

Step 3. Restart Etlworks: Restart Etlworks to apply the changes.

Installing Premium Connectors

Step 1. Obtain the connector: Contact Etlworks support to obtain the connector. It will be provided as a .jar file along with a license key. If a license key is not provided, you can install a trial license.

Step 2. Copy the files: Copy the .jar file obtained from Etlworks support to TOMCAT_HOME/lib.

Step 3. Navigate to the folder: Open a terminal and navigate (cd) to TOMCAT_HOME/lib.

Step 4. Run the license activation command:

java -jar connector.jar -license

Replace connector.jar with the actual name of the file you received from Etlworks.

Step 5. Provide the required information: When prompted, enter the following:

• Name: support
• Email: support@etlworks.com
• License key: Enter the license key provided by Etlworks or type TRIAL for a trial license.

Step 6. Verify that the license file was successfully generated: The license file name should match the connector file name, but with the .jar extension replaced by .lic. For example, if the connector file is cdata.jdbc.salesforce.jar, the license file should be named cdata.jdbc.salesforce.lic.

Step 7. Restart the service.

Handle cases where the license file is not generated

Occasionally, the installer fails to generate the license file (including trial licenses). In this case, the output of the installer will provide a link to download the license file:

Step 1. Copy the link and open it in any browser. Follow the on-screen instructions to download the license file.

Step 2. After downloading the file, rename it to match the connector file name, replacing the .jar extension with .lic.

Step 3. Copy the .lic file to TOMCAT_HOME/lib.

Step 4. Restart the service.

Monitoring Etlworks instance health

To monitor the health of an Etlworks instance, use the /health API executed from an external orchestration tool.

Interpret the JSON, not just the HTTP code. The public /health endpoint returns HTTP 200 with a JSON body whose status field is UP, WARNING, or DOWN — an HTTP 200 alone does not establish health. By default the endpoint returns only the overall status; when detail exposure is enabled in the deployment configuration, the response also includes per-indicator details (RAM, CPU, disk, data disk, database, Redis). Because /health is public, enable details only when your network placement allows it.

Multi-node aggregation. With a load balancer configured, /health?allNodes=true aggregates health from all nodes through Redis: the response lists each node and an overall status that is DOWN when any node is DOWN.

Related endpoints are different things: the installer's readiness checks use the /health2 endpoint plus a JWT-authenticated engine probe; those are installer contracts, not your monitoring endpoint. Use public /health for external monitoring.

Possible responses from the /health API endpoint

Everything is Good

HTTP Response Code: 200 — Status: UP

{
   "status": "UP",
   ...
}

High Resource Utilization but Service is Operational

HTTP Response Code: 200 — Status: WARNING

{
   "status": "WARNING",
   ...
}

Details: The response includes information about resources (RAM/CPU/Disk/Postgres/Redis) in a warning state. This is a temporary condition, but it should be monitored closely as it could lead to more severe issues.

Infrastructure Issues (Service Not Operational)

HTTP Response Code: 200 — Status: DOWN

{
   "status": "DOWN",
   ...
}

Details: The response includes information about resources (Disk/Postgres/Redis) that are down. This state requires action, typically a cleanup (e.g., disk cleanup) and a service restart.

API and UI Not Responsive

HTTP Response Code: 504 (Timeout)

Details: The service is running but very slow to respond. This might be a temporary condition; repeat the check several times with delays before deciding to restart the service.

Tomcat is Down but Reverse Proxy is UP

HTTP Response Code: 502 (Bad Gateway)

Details: The application behind the proxy is not reachable. A 502 does not by itself prove a specific crash cause — Tomcat may be restarting, out of memory, or stopped for maintenance. Check status and the logs before assuming a crash, then restart if the service is actually down.

Service is Down or Network Issues

HTTP Response Code: Any 5xx code or a refused connection

Details: The service is not running, or network issues prevent the client from reaching the endpoint. Cloud VM migrations can cause temporary unreachability (up to a few minutes). A refused connection does not by itself identify the cause — repeat the check several times with delays before restarting; in rare cases the VM itself must be restarted.

Implementation Suggestions

• External monitoring: Use an external orchestration tool like Airflow to perform health checks, with a bounded request timeout.
• Data collection: To integrate with a third-party collection tool (e.g., Grafana), use a non-production Etlworks instance to push the data collected from the primary instance's /health endpoint to that tool.
• Retry logic: Retry with delays for slow responses and refused connections, allowing temporary conditions to resolve before acting.
• Alerts and remediation: Automate alerts and remediation (disk cleanup, service restart) based on the status and resource information, with escalation when remediation does not clear the condition.
• Separate the questions: availability (can I reach it), dependency health (DB/Redis/disk), resource pressure (WARNING), and actual flow failures are different signals — monitor flow failures through Etlworks notifications and the flow dashboard, not through /health.

On-premise multi-node deployment

Our installer supports the automatic provisioning of the Etlworks servers and shared components (HAProxy, PostgreSQL and Redis) in any cluster running on Linux OS.

Here is a simplified deployment diagram for an on-premise multi-node deployment.