• Starter
• Business
• Enterprise
• On-Premise
• Add-on

Overview

Etlworks can be installed on Linux, Windows, and MacOS. 

Supported operating systems and platforms:

• Amazon Linux 2
• Amazon Linux 2023
• Ubuntu 20.04
• Ubuntu 22.04
• CentOS 7
• Red Hat 7
• Red Hat 8
• Red Hat 9
• Windows Server (2012-2022)
• All editions of Windows 10 and Windows 11 
• MacOS via Docker 
• Docker

Hardware requirements

Read more.

License Validation and Firewall

Etlworks on-premise deployments require outbound communication from the on-premise server(s) to our license server (host: lic.etlworks.com, dedicated IP address: 18.217.236.168) over HTTPS (port 443). This communication is strictly outbound and does not require opening inbound firewall rules.

Key Requirements

• All servers running Etlworks in the environment must be able to access: lic.etlworks.com:443 (outbound traffic only).
• License validation occurs on startup and then every hour to ensure the instance remains authorized.
• License validation on startup is required.
• Each node in multi-node deployments independently reports license validation data.

Firewall

Inbound Firewall

No inbound firewall changes are required for Etlworks to function in standard on-premise deployments.

However, if the Etlworks instance is used to serve API requests, either from built-in APIs or user-defined APIs, and these requests originate from outside the firewall, the following inbound ports should be opened:

• Port 80 (HTTP) – If APIs are served without SSL.
• Port 443 (HTTPS) – If APIs are served over a secure connection.

Outbound Firewall

• Etlworks requires outbound access to lic.etlworks.com:443 (dedicated IP address: 18.217.236.168) for license validation.
• If an outbound firewall is in place, port 443 (HTTPS) must be open for communication with our license server.
• If the on-premise Etlworks server needs to access external resources—such as databases, message brokers, cloud APIs, or third-party services—the relevant outbound ports must also be opened to allow connectivity.

Data Transmitted During License Validation

During each license validation session, the Etlworks server(s) securely transmit the following information to our license server:

• Server identification details, including IP address, Etlworks version, and digital fingerprints unique to the instance.
• Digital fingerprints include non-sensitive hardware metadata (physical or virtual).
• Instance usage metrics: number of flows, connections, formats, listeners, schedules, and agents.
• System health status, as returned by the /health API.

Security Measures

• All transmitted data is encrypted using a pair of environment-specific encryption keys, ensuring that only Etlworks and the authorized instance can read the payload.
• No sensitive or customer data is ever transmitted.
• Neither our license server nor other elements of Etlworks infrastructure ever initiate inbound connections to on-premise servers.

Failure Scenarios

• If a license is found to be invalid or has been remotely deactivated by Etlworks, the server(s) will stop functioning during the next validation check.
• If an on-premise server fails to connect to the license server for 24 hours, it will automatically shut down until connectivity is restored.
• License validation on startup is required.

Installation

Outbound Internet Access During Installation

During installation, the Etlworks installer retrieves required components over secure HTTPS connections. Outbound internet access is required for a successful installation.

Linux and Docker Installations

When installing Etlworks on Linux, the installer pulls required artifacts from public container registries (Docker Hub) over HTTPS.

These images include the necessary runtime components used by the platform.

Windows Installation

When installing Etlworks on Windows, the installer downloads MSI installers for required components, including:

• PostgreSQL
• Redis
• Java
• Tomcat

These installers are retrieved directly from their respective official distribution sources over HTTPS.

License Validation

Both Linux and Windows installers communicate with the Etlworks license server over HTTPS during installation to validate and activate the license.

The Etlworks license server uses a static IP address.

Important Network Recommendation

With the exception of the Etlworks license server, the IP addresses of third-party distribution endpoints are not static.

Over time:

• IP addresses may change
• Components may be added or removed
• Distribution sources may be updated

Because of this, we strongly recommend not restricting outbound HTTPS access during installation by static IP address.

If outbound traffic must be restricted, allow HTTPS access by DNS hostname rather than IP where possible.

Restricting outbound internet access during installation may cause incomplete setup or failed component downloads.

1. Request a link to download an installer or license

To request a download link, open https://etlworks.com/downloads/request.html in any browser. Fill in the form and click Email me the installer.

Important: Enter a valid email address. The Etlworks license server will send a short-lived download link to the email provided during registration.

If you already have an account with Etlworks, please enter the same Company, Name, and Email used when the existing account was created. If you’re unsure about these details, feel free to contact Etlworks support at support@etlworks.com for assistance.

2. Open the download link

After submitting the registration form, the Etlworks license server will generate a personalized download link and send it to the registered email address. This link will remain active for approximately 10 minutes. Make sure to access it within this time frame to download the required files.

3. Select the installer to download

Select either Linux or Windows installer. Read how to run Etlworks in Docker. 

4. Install Etlworks

Install Etlworks on Linux

Linux installer is provided as a tar zip archive. The archive contains required libraries, a command-line script, and a unique license file.

• Download the archive with the installer and copyetlworks-installer.tar.gzfile into/opt/etlworksdirectory and then unarchive it.

  sudo tar -zxvf etlworks-installer.tar.gz

• Start the installation by executing:

  sudo ./etlworks-cli.sh install

• Optional parameters/flags:
  • --version APPLICATION_VERSION (Default: latest)
    • alias: -v APPLICATION_VERSION
    • application version to install. If not specified by default latest version will be installed
  • --skip-confirm
    • alias: -s
    • proceed with installation without confirmation
  • --user USER_NAME (Default: ec2-user)
    • alias: -u USER_NAME
    • sudoer user name
  • --no-app
    • alias: -n
    • do not install application artifacts, only provision the system

To view all available versions use list command:./etlworks-cli.sh list

Install Etlworks on Windows

Antivirus Warning (False Positive)

Windows installer is provided as a zip archive. The archive contains required libraries, a Windows installer (Etlworks.exe), and a unique license file.

Note: Some older versions of antivirus software, specifically Windows Defender before version 1.381.3081.0, report that Etlworks.exe contains a trojan. THIS IS A FALSE POSITIVETo avoid warnings, we recommend updating your antivirus to the latest version before downloading and installing Etlworks on Windows.

Step 1. Download and Extract the Installer

• Download the archive with the installer and copyetlworks-installer.zipfile into any directory, for examplec:\etlworksand unarchive it.

Step 2. Run the Installer

• Start the installation by executing Etlworks.exe. 
• Continue with the setup

Java Installation Notes for Windows Installer (Version 9.x and Later)

Starting with Etlworks 9.x, the Windows installer has been updated.

The installer no longer downloads Oracle Java. Instead, it automatically downloads and installs Zulu OpenJDK (Java 8 LTS).

This eliminates Oracle licensing concerns and no manual Java replacement is required.

No additional action is needed after installation when using Etlworks 9.x or later.

Java Installation Notes for Windows Installer prior to 9.x

IMPORTANT: This section applies only to Etlworks Windows installers for versions prior to 9.x.

During installation, the Windows installer for versions prior to 9.x automatically downloads and installs Java 8 (JRE 8u351). This is required for the installation process to complete successfully.

However, Oracle has changed the Java licensing terms. Newer Oracle JDK 8 updates (for example, 8u441) require a commercial license for production use. Because of this, we do not recommend upgrading to a newer Oracle JDK.

For this reason, we recommend replacing the bundled Oracle JRE with OpenJDK 8 LTS after the Etlworks installation completes.

Step 1. Install OpenJDK 8 LTS

• Download OpenJDK 8 LTS for Windows.

  Recommended distribution (Zulu OpenJDK): https://www.azul.com/downloads/?version=java-8-lts&os=windows&architecture=x86-64-bit&package=jre

• Install OpenJDK using the downloaded installer.

Step 2. Update Java Used by the Tomcat Service

• Open the Tomcat Configuration Utility:
  • Press Windows Start
  • Search for Configure Tomcat
  • If search cannot find Configure Tomcat app run: tomcat8w.exe from C:\Program Files\Apache Software Foundation\Tomcat 8.5\bin
• In the configuration window, open the Java tab.
• In the Java Virtual Machine field, update the path to point to the OpenJDK installation. Example: C:\Program Files\Zulu\zulu8-jre\bin\server\jvm.dll. Click Apply->OK.

Step 3. Restart Tomcat

Restart the Tomcat service to apply the new Java configuration.

Once restarted, Etlworks will run using OpenJDK instead of the bundled Oracle JRE.

Etlworks Windows Installer v9 Overview

Note: In typical installations, no command-line flags or overrides are required. These options are intended for advanced or edge-case scenarios such as custom environments, preinstalled components, or restricted networks.

Java version and JRE

• Java runtime switched from Oracle JRE to Azul Zulu OpenJDK 8
• Installer now downloads and installs: https://cdn.azul.com/zulu/bin/zulu8.92.0.21-ca-jre8.0.482-win_x64.msi
• Java install type changed from EXE to MSI (msiexec silent mode).
• Default Java target directory changed to: {commonpf64}\Zulu\zulu-8

Skip Components

You can skip all extra components or skip each component independently.

Skip all prerequisites

• /skipprereqs=1
  /skipcomponents=1 (alias)

Skip individual components

• /skipjava=1
  /skiptomcat=1
  /skipredis=1
  /skippostgresql=1

When a component is skipped, verification and installation for that component are bypassed.

Command-line overrides for download URLs

Defaults can be overridden at runtime:

• /tomcaturl=...
  /redisurl=...
  /postgresqlurl=...

Command-line overrides for install commands
Install command templates can now be overridden at runtime:

• /tomcatinstallcmd=...
  /redisinstallcmd=...
  /postgresqlinstallcmd=...

Supported tokens inside command templates:

• {installer}: full downloaded installer path in %TEMP%
• {targetdir}: default install target path for that component
• {superaccount}: resolved PostgreSQL superuser
• {superpassword}: resolved PostgreSQL superuser password

Default templates used if not overridden:

Tomcat:
/S /D={targetdir}

Redis (passed to msiexec):
/i "{installer}" INSTALLDIR="{targetdir}" REBOOT=ReallySuppress /qn
PostgreSQL:
--mode unattended --unattendedmodeui none --prefix "{targetdir}" --superaccount "{superaccount}" --superpassword "{superpassword}"

Tomcat context.xml JarScanner settings updated

Generated context.xml includes:

<JarScanner scanManifest="false"> <JarScanFilter defaultPluggabilityScan="false" /> 
</JarScanner>

This setting improves startup time after fresh installation or upgrade.

Example command lines

Skip all prerequisites
Etlworks.exe /skipprereqs=1

Skip only Java and Redis
Etlworks.exe /skipjava=1 /skipredis=1

Override Tomcat/Redis/PostgreSQL download URLs
Etlworks.exe /tomcaturl="https://example.org/apache-tomcat.exe" /redisurl="https://example.org/redis.msi" /postgresqlurl="https://example.org/postgresql.exe"

Override install commands (using tokenized templates)
Etlworks.exe /tomcatinstallcmd="/S /D={targetdir}" /redisinstallcmd="/i ""{installer}"" INSTALLDIR=""{targetdir}"" REBOOT=ReallySuppress /qn" /postgresqlinstallcmd="--mode unattended --unattendedmodeui none --prefix ""{targetdir}"" --superaccount ""{superaccount}"" --superpassword ""{superpassword}"""

Silent install when PostgreSQL already exists
Etlworks.exe /VERYSILENT /superaccount="postgres" /superpassword="your_password"

Java Compatibility Roadmap

Etlworks currently runs on Java 8.

We are actively working on making Etlworks compatible with Java 24 LTS. This will allow on-premise installations to run on a modern, long-term supported Java runtime in future releases.

5. Finish installation and configuration

IMPORTANT: Keep the downloaded installer, libraries required for installation, and a provided license file. The same installer can be used to upgrade Etlworks to the latest version. If for any reason you lost the installer and required libraries you can re download it by and providing the same Company, Name and Email that you used when you downloaded the installer first time.

After the installation process is over, the service should be available on port8080. You can test it by opening the Etlworks URL in the browser, for example: http://localhost:8080. The default Super Admin username and password are:

• user: admin
• password: admin1

After you first log in, navigate toSettings.

Set Home URL

UnderGeneral, set Home URLto your instance public URL, for example http://myhost:8080.

Configure Email

Important: This is required for adding new users, resetting passwords, and sending email notifications.

UnderEmail, provide the email configuration that will be used by the system to send notifications. 

Read more.

AWS multi-node deployment

Read how to provision multi-node Etlworks cluster on AWS. 

Upgrade

Upgrade Etlworks on Linux

In order to upgrade service to the latest version:

• Navigate to/opt/etlworksdirectory.

  sudo ./etlworks-cli.sh upgrade

• Optional parameters/flags:
  • --version APPLICATION_VERSION (Default: latest)
    • alias: -v APPLICATION_VERSION
    • application version to upgrade to
  • --skip-confirm
    • alias: -s
    • proceed with upgrade without confirmation

Note: After installation is over it takes up to a few minutes for Etlworks service to restart.

Upgrade Etlworks on Windows

In order to upgrade service to the latest version:

1. Stop Tomcat (optional but recommended) 
2. Navigate to the directory with Etlworks Windows installer, for examplec:\etlworks. Note: The directory must contain a valid license file.
3. Execute Etlworks.exe.

Note: After installation is over it takes up to a few minutes for Etlworks service to restart.

Troubleshoot Windows upgrade

If, after installing the upgrade, the service is not coming online:

Or ETL engine version is not available in About:

Then, the upgrade wasn't successful.

Here are the steps to fix:

Step 1. Stop the Tomcat service.

Step 2. Navigate to TOMCAT_HOME/webapps and manually delete files etl.war, ROOT.war and folders etl and ROOT.

Step 3. Run Etlworks.exe again.

Maintenance

Maintenance for an on-premise (self-hosted) Etlworks instance covers two distinct areas:

1. Service control — stop, start, view status, view logs.
2. Backup and restore — protect PostgreSQL, Redis, and the app.data folder against host failure, data corruption, ransomware, and bad upgrades.

This section is the operator reference for both. The instructions assume you're an admin on the host running Etlworks. Linux commands assume a typical /opt/etlworks install layout; Windows commands assume the standard installer layout under C:\Program Files\Apache Software Foundation\Tomcat 8.5 with PostgreSQL and Redis installed as Windows services.

Download installers and CLI tools

Installers and CLI tools are included with every Etlworks installation. They can also be downloaded separately.

Installers, the Linux CLI (etlworks-cli), and the new Windows CLI (etlworks-cli.ps1 / etlworks-cli.bat) are available from a single self-service page on the marketing site:

etlworks.com/downloads/tools.html

The page is also reachable from etlworks.com → Resources → Installer & CLI tools.

You don't need a license to download installers or CLI tools from this page — use it to refresh tooling on your hosts whenever new versions ship. Always pull the latest tooling before running a major upgrade or restore.

Service control

Service control on Linux

The Linux CLI is installed as /opt/etlworks/etlworks-cli.sh and is invoked with sudo. Operate from the install directory.

Stop the service

cd /opt/etlworks
sudo ./etlworks-cli.sh stop

Optional --skip-confirm (alias -s) skips the confirmation prompt.

Start the service

sudo ./etlworks-cli.sh start

View status

sudo ./etlworks-cli.sh status

View installed version

sudo ./etlworks-cli.sh version

View logs

sudo ./etlworks-cli.sh log

Optional flags:

• --engine (alias -e) — show the ETL engine log instead of the application log.
• --follow (alias -f) — live tail.
• --lines N (alias -n N) — show the last N lines.

Service control on Windows

Use the Apache Tomcat tray icon (or run tomcat8w.exe from C:\Program Files\Apache Software Foundation\Tomcat 8.5\bin) to stop, start, and view status of the Tomcat service. Logs are written to the Tomcat logs directory — by default C:\Program Files\Apache Software Foundation\Tomcat 8.5\logs.

The new Windows CLI (etlworks-cli.ps1 / etlworks-cli.bat) also exposes start / stop commands. Run from an Administrator PowerShell or Command Prompt — required because Tomcat is a Windows service.

.\etlworks-cli stop
.\etlworks-cli start
.\etlworks-cli version

Which services must be running for each maintenance task

Etlworks depends on three services on the host: Tomcat (the Etlworks application), PostgreSQL (metadata database), and Redis (shared runtime state, queues, cache, and the persistent flow-execution suspend flag). Different maintenance tasks need different combinations of these to be up while they execute.

The most common source of confusion is whether the Etlworks application (Tomcat) needs to be up for backup, restore, and suspend / resume. The short answer: only when the task is invoked through the UI or the built-in Etlworks CLI. Tasks invoked through the host CLI (Linux etlworks-cli.sh, Windows etlworks-cli.ps1 / etlworks-cli.bat) talk to PostgreSQL and Redis directly and do not need Tomcat to be running — which is what makes them the right choice during restore windows, planned outages, and multi-node maintenance.

The table below maps every maintenance task to what must be running while it executes. Requirements are:

• Must be running — the task fails or produces incomplete results without it.
• May be either — the task works whether the service is running or stopped.
• Stopped during the task — the host CLI stops the service itself, performs the task, and starts the service again.

Rule of thumb. For any task that touches the contents of PostgreSQL or Redis — backup, restore, or the persistent flow-execution suspend flag — PostgreSQL and Redis must be up. Tomcat is only required when you invoke the task through the UI or the built-in Etlworks CLI. Every task in the host CLI works with Tomcat stopped, and the host CLI's restore path stops it for you.

Suspend and resume flow executions during maintenance

Requires Etlworks 9.6.10 or newer. Before using this workflow, download or update the latest CLI tools from etlworks.com/downloads/tools.html (also reachable from etlworks.com → Resources → Installer & CLI tools). The host CLI commands documented below ship with those tools, so an out-of-date CLI package will not have them.

Suspending flow executions blocks new flow runs from starting, without stopping the application. Operators use this before upgrades, migrations, restore testing, and emergency maintenance windows — and especially after restoring from backup, because schedules can otherwise wake up and start firing the moment Tomcat starts. Persistent suspend lets you bring the application online, validate the restored system, and only then re-enable executions.

Behavior at a glance

• Suspend blocks new flow executions. Currently running flows are not stopped automatically — use the force variant if you want to stop them as well.
• Resume re-enables executions. A single Resume action clears both the temporary and persistent flags.
• Temporary suspend keeps the pre-9.6.10 behavior: it clears automatically when the application restarts.
• Persistent suspend is stored in Redis and survives Tomcat and application restart. It must be cleared explicitly through the UI or CLI.
• Multi-node deployments share the persistent state through Redis. One suspend command affects every node using the same Redis and prefix.
• The UI reflects CLI-initiated changes. If a persistent suspend is set from the CLI while Etlworks is running, the suspended badge appears in the top bar and Settings → Flows shows Resume. Nodes may take a few seconds to reflect the change because the app refreshes the Redis-backed state periodically.

Temporary vs. persistent suspend

Use persistent suspend for anything that involves stopping and starting Etlworks. Use temporary suspend only for very short maintenance windows where you know you will resume before restarting the application.

Suspend and resume from the UI

1. Go to Settings → Flows.
2. Click Suspend flow executions.
3. If prompted, choose whether to stop or drain running flows in addition to blocking new ones.
4. To make the suspension survive a restart, enable Keep suspended after restart until resumed manually.
5. Click Suspend. The suspended badge appears in the top bar of the UI.
6. When you are ready to re-enable executions, return to Settings → Flows and click Resume flow executions. Resume clears both temporary and persistent suspend states.

Built-in Etlworks CLI commands

These commands are available from the built-in Etlworks CLI while the application is running. They operate on the same suspend state as the UI, so a CLI change is visible in the UI and vice versa.

Host CLI (Linux)

The host CLI operates on the persistent Redis-backed suspend flag directly, so it works even when Tomcat is stopped, as long as Redis is running and reachable. This is the tool you use during backup, restore, and upgrade windows — when the application is offline and the built-in CLI is not available.

cd /opt/etlworks
sudo ./etlworks-cli.sh suspend-flows
sudo ./etlworks-cli.sh flow-suspend-status
sudo ./etlworks-cli.sh resume-flows

Redis override options. By default the CLI reads its Redis connection settings from application.properties. Pass any of the following flags to override them:

• --redis-host HOST
• --redis-port PORT
• --redis-password PASSWORD
• --redis-ssl true|false
• --redis-prefix PREFIX

Defaults (used when a flag is not passed and application.properties does not set the corresponding key):

Host CLI (Windows)

Run from an Administrator PowerShell or Administrator Command Prompt. The commands are provided by etlworks-cli.ps1 and etlworks-cli.bat in the updated CLI tools package (see Download installers and CLI tools if you need a fresh copy).

cd C:\etlworks
.\etlworks-cli suspend-flows
.\etlworks-cli flow-suspend-status
.\etlworks-cli resume-flows

Redis override options are the same as on Linux:

• --redis-host HOST
• --redis-port PORT
• --redis-password PASSWORD
• --redis-ssl true|false
• --redis-prefix PREFIX

Defaults follow the same rules as on Linux — read from application.properties, with the fallbacks listed in the Linux section above.

Recommended workflow: upgrade or migration

1. Upgrade Etlworks to 9.6.10 or newer, then update the CLI tools from etlworks.com/downloads/tools.html.
2. Suspend new executions persistently:
   • UI: Settings → Flows → Suspend flow executions, enable Keep suspended after restart until resumed manually.
   • Linux CLI: sudo ./etlworks-cli.sh suspend-flows
   • Windows CLI: .\etlworks-cli suspend-flows
3. Drain or stop currently running flows.
4. Run Prepare state for backup if applicable.
5. Run the backup.
6. Perform the upgrade or migration.
7. Start Etlworks.
8. Validate the instance while it is still suspended — sign in, check schedules, open a representative flow, confirm connections respond.
9. Resume executions only after validation passes:
   • UI: Settings → Flows → Resume flow executions.
   • Linux CLI: sudo ./etlworks-cli.sh resume-flows
   • Windows CLI: .\etlworks-cli resume-flows

Recommended workflow: restore

1. Restore PostgreSQL, Redis, and app.data using the documented restore command — see Restore with Linux CLI or Restore with Windows CLI.
2. Before starting Tomcat, optionally set a persistent suspend from the host CLI so no schedules fire immediately after startup:

   sudo ./etlworks-cli.sh suspend-flows        # Linux
   .\etlworks-cli suspend-flows                 # Windows

   This step is optional but strongly recommended for production restores, especially when the source instance had frequently-firing schedules.
3. Start Tomcat.
4. Sign in and validate the instance while it is suspended.
5. Check schedules, flows, Redis-backed state, and representative connections.
6. Resume executions only after validation:
   • UI: Settings → Flows → Resume flow executions.
   • Linux CLI: sudo ./etlworks-cli.sh resume-flows
   • Windows CLI: .\etlworks-cli resume-flows

Multi-node deployments

• Persistent suspend is stored in Redis, so one suspend command affects every application node that shares the same Redis and prefix.
• Run suspend-flows once per shared Redis environment, not once per node.
• Each node may take a few seconds to display the suspended badge and to actually block new executions, because nodes refresh the Redis-backed state on a short interval.
• Resume from any node (UI or CLI) clears the shared Redis flag; other nodes see the change on their next refresh.

Where the persistent flag lives (advanced)

Persistent suspend is stored in Redis under the configured Redis prefix. The default key is:

integrator.conf.system.flow.executions.suspended

If your deployment uses a non-default Redis prefix, substitute it for integrator.

Do not edit this key manually except as a last-resort support-guided step. Prefer the UI or the CLI — both apply the change consistently, refresh the application state, and update the UI. Manual Redis edits are not synchronized to nodes any faster than the CLI, they do not update the UI, and they bypass audit logging.

Backup and restore overview

Etlworks supports three independent ways to create a backup. All three produce archives that the Linux CLI and Windows CLI can restore from.

Each backup contains up to three sections: PostgreSQL (instance database), Redis (orchestration state), and app.data (filesystem state). Any combination can be backed up; restore picks up whatever sections are present in the archive.

Restore is destructive. It replaces the live PostgreSQL database, Redis state, and (when included) the app.data directory. app.data is not merged. See app.data backup/restore notes before restoring.

Back up from the Etlworks UI

The in-app Backup instance data task creates a portable .zip archive that contains PostgreSQL, Redis, and app.data sections, plus a manifest. The archive can be restored on the same host or moved to another host and restored by either the Linux CLI or the Windows CLI.

Configure the in-app backup task

The task is not added automatically. You must add it explicitly:

1. Sign in as a super-admin.
2. Open Settings → Maintenance.
3. Click Add task and pick Backup instance data.
4. Configure the parameters described below.
5. Save. The task can be left without a schedule and run manually, or scheduled to run automatically.

Recommended schedule: run once daily during a low-activity window — for example, 1:00 AM local instance time. This is a recommendation, not a requirement; adjust to your operational profile.

Required parameters

The archive is named:

etlworks-backup-YYYYMMDD-HHMMSS-<id>.zip

What is included in app.data backup

The task supports two app.data backup modes plus the option to skip app.data entirely.

Standard folders (recommended). Captures the operationally important files without blindly copying high-volume transient event data. By default the standard backup includes:

• keys and all subfolders / files.
• debezium-data root and selected subfolders.
• debezium-data/events is excluded by default except for selected event file types: csv, zip, gzip/gz, json.
• metrics and all subfolders / files.
• metadata and all subfolders / files.
• edi and all subfolders / files.
• rag and all subfolders / files.
• logs.

A multi-select control lets you include or exclude individual standard folders, and you can add custom folders or file-name patterns through the additional-folders / file-masks fields.

Entire app.data. Backs up every regular file under the app.data directory. Use with care — app.data can be hundreds of gigabytes in production. Click Estimate in the UI to preview the number of files and the total size before enabling this mode.

Locked / unreadable files (Windows in particular). Some files may be held by running processes when the backup runs. By default the in-app task skips unreadable files and records a warning rather than failing the entire backup. A warning means the archive was created but not every selected app.data file could be read — review the task result for the list of skipped files and decide whether to re-run during a quieter window.

Backup retention and backup health notifications

Keep backups for (days). The task removes archives older than this many days from the configured backup location. Minimum 1, default 7.

Warn if no backup for (days). Set this when the task is scheduled, to surface broken schedules, disabled jobs, repeated failures, or misconfigured backup locations. If no successful backup has been created within the configured window, the task triggers the same notification channels as a regular failure. The check only applies to scheduled tasks.

Notification emails. Recipients to notify on backup events:

• Notification emails — comma-separated list.
• Email on failure — send when a backup fails.
• Email on success — send when a backup succeeds. Enable for compliance auditing; disable to reduce inbox noise.

Webhook events. Webhooks are configured separately under the regular Etlworks webhook settings (super-admin only). Relevant events:

• Backup success.
• Backup failure.
• Backup warning (where exposed) — raised when the archive was created but some files were skipped.

Backup progress and long-running backups

The in-app task reports progress while it runs — pushed from the backend, not polled. Progress is reported at the stage level: PostgreSQL dump, Redis backup, app.data archive / copy, cleanup / retention, notifications.

For app.data backups, progress within the app.data stage is approximate — file counts and sizes can be very large and operations stream straight into the archive. Don't expect exact byte-level progress.

Full app.data backups on a multi-hundred-GB installation can run for tens of minutes to hours. Schedule them during a quiet window and make sure the backup location has enough free space for both the new archive and any older archives still within the retention window.

Back up with Linux CLI

The Linux CLI (etlworks-cli) supports backup of PostgreSQL, Redis, and app.data. Run from the install directory with sudo.

Basic backup (database and Redis only).

sudo etlworks-cli backup --skip-confirm

Include the full app.data tree.

sudo etlworks-cli backup --app-data --skip-confirm

Include only the standard app.data folders.

sudo etlworks-cli backup --app-data-standard --skip-confirm

Fail the entire backup if any app.data file can't be read (strict mode).

sudo etlworks-cli backup --app-data --fail-on-locked-files --skip-confirm

Create a portable CLI archive (zip). The CLI first creates a working backup folder, then zips it into the destination, then deletes the working folder.

sudo etlworks-cli backup --app-data --cli-archive /opt/etlworks/backup/etlworks-cli-backup.zip --skip-confirm

Standard app.data + CLI archive.

sudo etlworks-cli backup --app-data-standard --cli-archive /opt/etlworks/backup/etlworks-cli-backup.zip --skip-confirm

Without --cli-archive, the CLI leaves a folder-based backup at the configured backup location (postgres/, redis/, app-data/ subfolders). With --cli-archive, you get a single zip you can move to another host. The CLI archive format is different from the in-app archive format, but the restore command accepts either.

Restore with Linux CLI

Warning — restore is destructive. It replaces the live PostgreSQL database, Redis state, and (when included) app.data. Stop Etlworks before running it. By default the CLI stops Tomcat for you and aborts before modifying anything if the stop fails.

Restore from an in-app backup archive.

sudo etlworks-cli restore --archive /opt/etlworks/backup/etlworks-backup-YYYYMMDD-HHMMSS-id.zip --skip-confirm

Restore from a CLI zip archive.

sudo etlworks-cli restore --cli-archive /opt/etlworks/backup/etlworks-cli-backup.zip --skip-confirm

Restore from a folder-based CLI backup (explicit per-section paths).

sudo etlworks-cli restore \
    --postgres /path/to/integrator_dump.sql \
    --redis /path/to/dump.rdb \
    --app-data /path/to/app-data \
    --skip-confirm

Preserve the existing app.data tree before overwriting (requires at least 2x app.data free space):

sudo etlworks-cli restore --archive /opt/etlworks/backup/etlworks-backup.zip --skip-confirm --keep-pre-restore-app-data

When --keep-pre-restore-app-data is set, the CLI renames the existing app.data to <app-data-path>.pre-restore.<timestamp> before extracting the backup. After you validate the restore, delete the .pre-restore. directory manually.

Back up with Windows CLI

The Windows CLI (etlworks-cli.ps1, with a etlworks-cli.bat wrapper) ships in the Windows installer package and lives in the installer's extracted root folder, not in Tomcat\bin. Download / refresh from etlworks.com/downloads/tools.html as needed.

Run from an elevated PowerShell or Administrator Command Prompt. Administrator rights are required because:

• The CLI starts and stops Windows services (Tomcat, Redis).
• PostgreSQL and Redis data live under C:\Program Files / C:\ProgramData.
• Replacing dump.rdb fails without elevation.

Help / version.

.\etlworks-cli backup --help
.\etlworks-cli restore --help
.\etlworks-cli version
.\etlworks-cli cli-version

Basic backup (database and Redis only).

.\etlworks-cli backup --skip-confirm

Full app.data backup.

.\etlworks-cli backup --app-data --app-data-mode full --skip-confirm

Standard app.data backup.

.\etlworks-cli backup --app-data --app-data-mode standard --skip-confirm

Strict mode — fail on locked files.

.\etlworks-cli backup --app-data --app-data-mode full --fail-on-locked-files --skip-confirm

Create a portable CLI archive (zip).

.\etlworks-cli backup --app-data --app-data-mode full --cli-archive C:\etlworks\backup\backup_test.zip --skip-confirm

As on Linux, the Windows CLI creates the working backup folder first, then the zip, then removes the working folder after the archive is written.

Notes. By default, locked or unreadable app.data files are skipped and reported. Add --fail-on-locked-files when you need every file or none.

Restore with Windows CLI

Warning — restore is destructive. Run from an Administrator PowerShell or Command Prompt. By default the CLI stops Tomcat first and verifies it is stopped; if Tomcat cannot be stopped, restore fails before modifying data. Restore does not start Tomcat afterwards — validate the restore and start the service manually.

Restore from an in-app backup archive.

.\etlworks-cli restore --archive C:\etlworks\backup\etlworks-backup-YYYYMMDD-HHMMSS-id.zip --skip-confirm

Restore from a CLI zip archive.

.\etlworks-cli restore --cli-archive C:\etlworks\backup\backup_test.zip --skip-confirm

Restore and preserve the old app.data tree.

.\etlworks-cli restore --archive C:\etlworks\backup\etlworks-backup.zip --skip-confirm --keep-pre-restore-app-data

Restore when Tomcat is already stopped. Skip the verification step.

.\etlworks-cli restore --archive C:\etlworks\backup\etlworks-backup.zip --skip-confirm --skip-stop-tomcat

Start Etlworks after restore.

.\etlworks-cli start

Redis restore on Windows. Redis logical restore requires Redis to be running and reachable. Redis RDB restore stops and restarts the Redis service around the dump.rdb replacement; if the service can't be stopped or the file can't be replaced (access denied), rerun from Administrator PowerShell or Command Prompt.

Restore from an in-app backup archive

In-app archives (etlworks-backup-YYYYMMDD-HHMMSS-<id>.zip) restore from either CLI through the --archive flag:

sudo etlworks-cli restore --archive /path/to/etlworks-backup-*.zip --skip-confirm

.\etlworks-cli restore --archive C:\path\to\etlworks-backup-*.zip --skip-confirm

Whatever sections the archive contains (PostgreSQL, Redis, app.data) are restored. Sections missing from the archive are left untouched on the target.

Restore from a CLI backup archive

CLI archives (the --cli-archive output) restore from either CLI through the --cli-archive flag. Folder-based CLI backups (no zip) restore through explicit per-section paths.

Linux CLI archive (zip)

sudo etlworks-cli restore --cli-archive /path/to/etlworks-cli-backup.zip --skip-confirm

Linux folder backup

sudo etlworks-cli restore --postgres /path/to/integrator_dump.sql --redis /path/to/dump.rdb --app-data /path/to/app-data --skip-confirm

Windows CLI archive (zip)

.\etlworks-cli restore --cli-archive C:\path\to\etlworks-cli-backup.zip --skip-confirm

Windows folder backup

Move backup files into Windows-friendly paths and supply --postgres / --redis / --app-data.

PostgreSQL backup/restore notes

Backup uses pg_dump. The in-app task exposes a configurable pg_dump command with built-in profiles for the common deployment shapes; for non-standard installs (RDS, Azure Database for PostgreSQL, Cloud SQL, custom Docker, non-PATH installations) use the custom profile and override the host, port, database, user, password, and command as needed.

The instance database is integrator by default. Override only if your installation was customized to use a different database name.

Tokens supported in the pg_dump command (when using the custom profile): {username}, {database}, and (depending on the UI you see) {host}, {port}, {password}, {container}.

Linux Docker example (built-in profile equivalent):

docker exec -e PGPASSWORD postgres pg_dump --username {username} {database}

Windows service example (built-in profile equivalent):

C:\Program Files\PostgreSQL\9.5\bin\pg_dump.exe

Restore uses psql / pg_restore as appropriate and applies the required grants and ownership after the database is loaded.

Choosing a PostgreSQL backup profile

The in-app task exposes a set of pre-built profiles plus a custom option. Pick the one that matches your installation:

Selection process

1. Start with the profile that matches your installation type.
2. If the backup fails with "No such container", pick the profile whose container name matches, or switch to custom.
3. If the backup fails with "pg_dump not found", use windows_service on Windows or custom with the full pg_dump path on Linux.
4. For managed / external PostgreSQL, use custom and configure host, port, database, user, password, and any extra pg_dump arguments.
5. The database name is normally integrator; only override if your installation was customized.

The PostgreSQL host, port, database, user, password, Docker container, and additional pg_dump arguments overrides are all exposed in the UI — leave them blank unless the installation differs from the selected profile.

Redis backup/restore notes

The in-app task offers three Redis backup modes; pick by what's available on the host and how Redis is deployed.

Cross-version warning. Logical Redis restore (DUMP / RESTORE) requires payload compatibility between the source Redis version and the target Redis version. If you change Redis major versions during a migration, test restore in staging before going to production. For managed Redis (AWS ElastiCache, Azure Cache for Redis, …) check the supported Redis version against your backup source. When in doubt, use rdb mode.

app.data backup/restore notes

app.data can be very large. Hundreds of GB is not unusual on busy CDC installations. Treat it accordingly:

• Default restore deletes the existing app.data directory and extracts the backup directly into the target. This is intentional — it avoids requiring 2x app.data disk space.
• In-app archives and CLI archives are streamed directly into the target. They do not first extract app.data into a temporary folder — that would double disk usage during restore.
• To keep a copy of the old app.data for rollback safety, use --keep-pre-restore-app-data. The CLI renames the existing app.data to <app-data-path>.pre-restore.<timestamp> before restore. Only use this flag if the host has enough free disk for both copies.
• You must delete the .pre-restore. directory manually after validating the restore. The CLI never deletes it.

For day-to-day backups, prefer Standard folders over Entire app.data — the standard set captures the operationally important state without including high-volume transient event data.

Cross-platform compatibility matrix

Recommended production procedures

Recommended backup plan

• Schedule the in-app backup task to run daily during a quiet window (e.g., 1:00 AM).
• Store backups outside the app.data directory, ideally on a different disk than the live instance.
• Set retention based on disk size and recovery-point objective.
• Enable Email on failure and the backup-failure webhook event.
• Enable Email on success only when needed for compliance.
• Test restore on a staging instance at least once per quarter.
• For very large app.data installations, prefer Standard folders over Entire app.data. Use entire only when business requirements demand it, and monitor disk and runtime when you do.

Before an upgrade or migration

1. Suspend flow executions persistently so the block survives the restart: UI — Settings → Flows → Suspend flow executions with Keep suspended after restart until resumed manually enabled; Linux CLI — sudo ./etlworks-cli.sh suspend-flows; Windows CLI — .\etlworks-cli suspend-flows. See Suspend and resume flow executions during maintenance. (Alternatively, wait for a natural quiet window before pausing.)
2. Drain or stop running flows.
3. If your deployment supports it, run the in-app Prepare state for backup control (or its CLI equivalent) to flush Redis state.
4. Run a backup — in-app task or CLI.
5. Verify the archive file exists and the size is plausible.
6. Proceed with the upgrade or migration.
7. If a restore is needed, follow Restore procedure below.

Restore procedure

1. Stop Etlworks / Tomcat. (The CLI does this for you with --archive / --cli-archive.)
2. Confirm no flows are running.
3. Run the restore command from an Administrator / elevated shell.
4. Wait for the restore to complete — it covers PostgreSQL, Redis, and app.data as available in the archive.
5. Optional but recommended for production: before starting Etlworks, set a persistent suspend from the host CLI so no schedules fire the moment Tomcat comes up. Linux — sudo ./etlworks-cli.sh suspend-flows; Windows — .\etlworks-cli suspend-flows. See Suspend and resume flow executions during maintenance.
6. Start Tomcat / Etlworks.
7. Validate: sign in, check schedules, open a representative flow, run a test execution, confirm Redis-backed state is intact. (If you set a persistent suspend before startup, validate while it is still in effect.)
8. If a persistent suspend was set, resume executions only after validation passes. UI — Settings → Flows → Resume flow executions; Linux — sudo ./etlworks-cli.sh resume-flows; Windows — .\etlworks-cli resume-flows.
9. Only after validation, delete any .pre-restore.<timestamp> app.data directory left behind.

Troubleshooting

"Cannot run program pg_dump" / "pg_dump not found"

The selected PostgreSQL profile points to a pg_dump command that isn't on PATH. Either pick a built-in profile that matches your installation (e.g., windows_service with the full pg_dump.exe path, or the Docker profile that matches your container name), or switch to custom and supply the full pg_dump command.

"No such container"

The Docker-based PostgreSQL profile's container name doesn't match the actual container name on the host. Run docker ps, find the right name, pick the matching profile, or switch to custom and supply your container name.

Windows: "Access denied" on Redis dump.rdb

Redis data is usually under C:\Program Files or C:\ProgramData, which require elevation. Rerun the CLI from an Administrator PowerShell or Administrator Command Prompt.

Backup completes with warnings — locked files

One or more app.data files were held by a running process at backup time. The archive was created but those files are missing. Read the task result for the file list; if any are critical, schedule the backup during a maintenance window, or rerun with --fail-on-locked-files to make the locking explicit and visible (the backup will fail rather than complete with a warning).

Warning vs. failure. A warning means the archive was created but some optional app.data files were skipped. A failure means the backup did not complete.

A .pre-restore.<timestamp> folder is left behind

You used --keep-pre-restore-app-data (or an older CLI version that defaulted to that behavior). Validate the restore, then delete the .pre-restore. directory manually to reclaim disk.

Restore fails on disk usage

The default restore avoids 2x app.data disk usage by extracting straight into the target. If you used --keep-pre-restore-app-data, the host needs space for the old app.data plus the new one. Either free space, or rerun without the flag and accept that there's no automatic rollback.

Redis logical restore fails (checksum, version, payload)

The source and target Redis versions are incompatible, or the payload was corrupted in transit. Confirm the Redis versions match, or use the rdb backup mode (which copies the on-disk dump file) and re-test the restore in staging.

"Cannot stop Tomcat"

The CLI aborts the restore before modifying any data when it can't confirm Tomcat is stopped. Stop Tomcat manually, confirm with ./etlworks-cli.sh status (or the Windows tray icon), then rerun the restore — or use --skip-stop-tomcat only after independently confirming Tomcat is down.

Instance starts suspended after restore

Symptom. After a restore, Etlworks starts up but all flow executions are blocked and the suspended badge appears in the top bar.

Cause. The persistent Redis suspend flag was included in the restored Redis state, or an operator set it before startup and did not clear it during validation. In both cases, the persistent behavior is working as designed — the flag survived the restart.

Fix. Check status and resume:

# Linux
sudo ./etlworks-cli.sh flow-suspend-status
sudo ./etlworks-cli.sh resume-flows

# Windows
.\etlworks-cli flow-suspend-status
.\etlworks-cli resume-flows

Or use Settings → Flows → Resume flow executions from the UI.

Suspend CLI cannot connect to Redis

Symptom. suspend-flows, flow-suspend-status, or resume-flows fails with a Redis connection error.

Fix. The host CLI reads its Redis connection settings from application.properties by default. Verify that the host, port, password, SSL flag, and prefix are correct for your deployment, then override any of them on the command line:

sudo ./etlworks-cli.sh suspend-flows \
  --redis-host redis.internal \
  --redis-port 6380 \
  --redis-password secret \
  --redis-ssl true \
  --redis-prefix myprefix

See Host CLI (Linux) or Host CLI (Windows) for the full flag list and default resolution rules.

UI still shows suspended after resume

Symptom. The Resume action was executed successfully, but the suspended badge is still visible in the top bar.

Fix.

• Give it a few seconds. Nodes refresh their view of the Redis-backed suspend state on a short interval; there is a brief lag between resuming and the badge clearing.
• Confirm the persistent flag is actually cleared:

  sudo ./etlworks-cli.sh flow-suspend-status
  .\etlworks-cli flow-suspend-status

• In a multi-node deployment, confirm that all nodes point at the same Redis and prefix. If a node is looking at a different Redis, resuming on one Redis will not clear the state seen by nodes reading a different one.

Schedules do not start after restore

Symptom. After a restore and validation, scheduled flows are not firing when they should.

Fix.

• Check whether persistent suspend is still enabled: flow-suspend-status. Resume if it is.
• Persistent suspend is a common cause of "schedules didn't come back after a restore." When it fires, it is doing exactly what it was designed to do — blocking new executions until the operator explicitly resumes.
• If flow-suspend-status reports the instance is not suspended and schedules are still not firing, the cause is elsewhere — check the flow's schedule configuration and the application log.

Installing connectors

Etlworks comes with hundreds of built-in connectors to meet a wide range of integration needs. However, it is also possible to add new regular or premium connectors as needed. Multiple connectors can be installed into TOMCAT_HOME/lib. This section provides step-by-step instructions for installing both types of connectors on the machine running the Etlworks.

Note: In multi-node setup the connector and license must be installed on each node.

Installing Regular Connectors

Step1. Obtain the connector: Contact Etlworks support to obtain the connector. It will be provided as a.jar file and may include a license file and other required files.

Step 2. Copy the files: Copy all files obtained from Etlworks support to TOMCAT_HOME/lib.

Step 3. Restart Etlworks: Restart Etlworks to apply the changes.

Installing Premium Connectors

Step 1. Obtain the connector: Contact Etlworks support to obtain the connector. It will be provided as a.jar file along with a license key. If a license key is not provided, you can install a trial license.

Step 2. Copy the files: Copy the.jar file obtained from Etlworks support to TOMCAT_HOME/lib

Step 3. Navigate to the folder: Open a terminal and navigate (cd) to TOMCAT_HOME/lib.

Step 4. Run the license activation command:

java -jar connector.jar -license

Replace connector.jar with the actual name of the file you received from Etlworks.

Step 5. Provide the required information:

When prompted, enter the following:

• Name: support
• Email: support@etlworks.com
• License key: Enter the license key provided by Etlworks or type TRIAL for a trial license.

Step 6. Verify that license file successfully generated:
The license file name should match the connector file name, but with the.jar extension replaced by.lic.

For example, if the connector file is cdata.jdbc.salesforce.jar, the license file should be named cdata.jdbc.salesforce.lic.

Step 7. Restart the service.

Handle cases where the license file is not generated

Occasionally, the installer fails to generate the license file (including trial licenses). In this case, the output of the installer will provide a link to download the license file. Here is the step-by-step instruction for installing the license manually.

Step 1. Copy the link and open it in any browser. Follow the on-screen instructions to download the license file.

Step 2. After downloading the file, rename it to match the connector file name, replacing the.jar extension with.lic.

For example, if the connector file is cdata.jdbc.salesforce.jar, the license file should be named cdata.jdbc.salesforce.lic.

Step 3. Copy lic file to TOMCAT_HOME/lib.

Step 4. Restart the service.

Monitoring Etlworks instance health

To monitor the health of an Etlworks instance, we recommend using the /health API executed from an external orchestration tool.

Possible responses from the /health API endpoint

Everything is Good

• HTTP Response Code: 200

• Status: UP

• Response JSON:

{
   "status": "UP",
   ...
}

High Resource Utilization but Service is Operational

• HTTP Response Code: 200

• Status: WARNING

• Response JSON:

{
   "status": "WARRNING",
   ...
}

Details: The response JSON includes information about resources (RAM/CPU/Disk/Postgres/Redis) in a warning state. This is a temporary condition, but it should be monitored closely as it could lead to more severe issues.

Infrastructure Issues (Service Not Operational)

• HTTP Response Code: 200

• Status: DOWN

• Response JSON:

{
   "status": "DOWN",
   ...
}

Details: The response JSON includes information about resources (Disk/Postgres/Redis) that are down. This state requires action, typically a cleanup (e.g., disk cleanup) and a service restart.

API and UI Not Responsive

• HTTP Response Code: 504 (Timeout)

• Status: Timeout

Details: The service is running but very slow to respond. This might be a temporary condition; it is recommended to repeat the check several times with delays before deciding to restart the service.

Tomcat is Down but Reverse Proxy is UP

• HTTP Response Code: 502 (Bad Gateway)

• Status: DOWN

Details: The service is not running, likely because the Tomcat server exited with a core dump due to 100% RAM utilization. This requires a service restart.

Service is Down or Network Issues

• HTTP Response Code: Any 5xx code or ERR_CONNECTION_REFUSED

• Status: DOWN or temporary network issues

Details: The service is not running, or there are network issues preventing the client from reaching the API endpoint. Cloud VM migrations to a different host might also cause temporary unreachability (up to a few minutes). This might be a temporary condition; it is recommended to repeat the check several times with delays before deciding to restart the service. In rare cases, this might require restarting the VM.

Implementation Suggestions

• External Monitoring: Use an external orchestration tool like Airflow to perform health checks. 

• Data collection: If you need to integrate with a third-party data collection tool (e.g., Grafana), use a non production Etlworks instance (e.g., Dev) to push the data collected from the /health endpoint of the primary instance to that tool.

• Retry Logic: Implement retry logic for scenarios where the service is slow to respond, allowing for temporary conditions to resolve before taking action.

• Alerts and Remediation: Automate alerts and remediation steps (e.g., disk cleanup, service restart) based on the status and resource information provided by the health check.

Run Etlworks in Docker

To run Etlworks in Docker please use our official image.

AWS multi-node deployment

Etlworks can be deployed to symmetrical, horizontally scalable cluster in AWS.

Read more. 

On-premise multi-node deployment

Our installer supports the automatic provisioning of the Etlworks servers and shared components (HaProxy, PostgreSQL and Redis) in any cluster running on Linux OS.

Here is a simplified deployment diagram for an on-premise multi-node deployment.