• Starter
• Business
• Enterprise
• On-Premise
• Add-on

Overview

Etlworks can be installed on Linux, Windows, and MacOS. 

Supported operating systems and platforms:

• Amazon Linux 2
• Amazon Linux 2023
• Ubuntu 20.04
• Ubuntu 22.04
• CentOS 7
• Red Hat 7
• Red Hat 8
• Red Hat 9
• Windows Server (2012-2022)
• All editions of Windows 10 and Windows 11 
• MacOS via Docker 
• Docker

Hardware requirements

	CPU	RAM	Hard drive
Minimum	2 cores	8GB	100GB SSD
Better	4 cores	16GB	250GB SSD
Optimal	8-16 cores	32GB-64GB	500GB-1TB SSD

Read more.

License Validation and Firewall

Etlworks on-premise deployments require outbound communication from the on-premise server(s) to our license server (host: lic.etlworks.com, dedicated IP address: 18.217.236.168) over HTTPS (port 443). This communication is strictly outbound and does not require opening inbound firewall rules.

Key Requirements

• All servers running Etlworks in the environment must be able to access: lic.etlworks.com:443 (outbound traffic only).
• License validation occurs every hour to ensure the instance remains authorized.
• Each node in multi-node deployments independently reports license validation data.

Firewall

Inbound Firewall

No inbound firewall changes are required for Etlworks to function in standard on-premise deployments.

However, if the Etlworks instance is used to serve API requests, either from built-in APIs or user-defined APIs, and these requests originate from outside the firewall, the following inbound ports should be opened:

• Port 80 (HTTP) – If APIs are served without SSL.
• Port 443 (HTTPS) – If APIs are served over a secure connection.

Outbound Firewall

• Etlworks requires outbound access to lic.etlworks.com:443 (dedicated IP address: 18.217.236.168) for license validation.
• If an outbound firewall is in place, port 443 (HTTPS) must be open for communication with our license server.
• If the on-premise Etlworks server needs to access external resources—such as databases, message brokers, cloud APIs, or third-party services—the relevant outbound ports must also be opened to allow connectivity.

Data Transmitted During License Validation

During each license validation session, the Etlworks server(s) securely transmit the following information to our license server:

• Server identification details, including IP address, Etlworks version, and digital fingerprints unique to the instance.
• Digital fingerprints include non-sensitive hardware metadata (physical or virtual).
• Instance usage metrics: number of flows, connections, formats, listeners, schedules, and agents.
• System health status, as returned by the /health API.

Security Measures

• All transmitted data is encrypted using a pair of environment-specific encryption keys, ensuring that only Etlworks and the authorized instance can read the payload.
• No sensitive or customer data is ever transmitted.
• Neither our license server nor other elements of Etlworks infrastructure ever initiate inbound connections to on-premise servers.

Failure Scenarios

• If a license is found to be invalid or has been remotely deactivated by Etlworks, the server(s) will stop functioning during the next validation check.
• If an on-premise server fails to connect to the license server for 24 hours, it will automatically shut down until connectivity is restored.

Installation

1. Request a link to download an installer or license

To request a download link, open https://etlworks.com/downloads/request.html in any browser. Fill in the form and follow the on-screen instructions.

Enter a valid email address. The Etlworks license server will send a short-lived download link to the email provided during registration

If you already have an account with Etlworks, please enter the same Company, Name, and Email used when the existing account was created. If you’re unsure about these details, feel free to contact Etlworks support at support@etlworks.com for assistance.

2. Open the download link

After submitting the registration form, the Etlworks license server will generate a personalized download link and send it to the registered email address. This link will remain active for approximately 10 minutes. Make sure to access it within this time frame to download the required files.

3. Select the installer to download

Select either Linux or Windows installer. Read how to run Etlworks in Docker. 

 

4. Install Etlworks

Install Etlworks on Linux

Linux installer is provided as a tar zip archive. The archive contains required libraries, a command-line script, and a unique license file.

• Download the archive with the installer and copyetlworks-installer.tar.gzfile into/opt/etlworksdirectory and then unarchive it.

  sudo tar -zxvf etlworks-installer.tar.gz

• Start the installation by executing:
  

  sudo ./etlworks-cli.sh install

• Optional parameters/flags:
  
  • --version APPLICATION_VERSION (Default: latest)
    • alias: -v APPLICATION_VERSION
    • application version to install. If not specified by default latest version will be installed
  • --skip-confirm
    • alias: -s
    • proceed with installation without confirmation
  • --user USER_NAME (Default: ec2-user)
    • alias: -u USER_NAME
    • sudoer user name
  • --no-app
    • alias: -n
    • do not install application artifacts, only provision the system

To view all available versions use list command: sudo ./etlworks-cli.sh list

Install Etlworks on Windows

Windows installer is provided as a zip archive. The archive contains required libraries, a Windows installer (Etlworks.exe), and a unique license file.

Some older versions of antivirus software, specifically Windows Defender before version 1.381.3081.0, report that Etlworks.exe contains a trojan. THIS IS A FALSE POSITIVE. To avoid warnings, we recommend updating your antivirus to the latest version before downloading and installing Etlworks on Windows.

• Download the archive with the installer and copyetlworks-installer.zipfile into any directory, for examplec:\etlworksand unarchive it.
• Start the installation by executing Etlworks.exe. 
• Continue with the setup

 

5. Finish installation and configuration

Keep the downloaded installer, libraries required for installation, and a provided license file. The same installer can be used to upgrade Etlworks to the latest version. If for any reason you lost the installer and required libraries you can re download it by requesting a new download link and providing the same Company, Name and Email that you used when you downloaded the installer first time. 

After the installation process is over, the service should be available on port8080. You can test it by opening the Etlworks URL in the browser, for example: http://localhost:8080. The default Super Admin username and password are:

• user: admin
• password: admin1

After you first log in, navigate toSettings.

Set Home URL

UnderGeneral, set Home URLto your instance public URL, for example http://myhost:8080.

Configure Email

This is required for adding new users, resetting passwords, and sending email notifications.

UnderEmail, provide the email configuration that will be used by the system to send notifications. 

Read more.

AWS multi-node deployment

Read how to provision multi-node Etlworks cluster on AWS. 

Upgrade

Upgrade Etlworks on Linux

In order to upgrade service to the latest version:

• Navigate to/opt/etlworksdirectory.
  

  sudo ./etlworks-cli.sh upgrade

• Optional parameters/flags:
  
  • --version APPLICATION_VERSION (Default: latest)
    • alias: -v APPLICATION_VERSION
    • application version to upgrade to
  • --skip-confirm
    • alias: -s
    • proceed with upgrade without confirmation

To view all available versions use list command: sudo ./etlworks-cli.sh list

After installation is over it takes up to a few minutes for Etlworks service to restart.

Upgrade Etlworks on Windows

In order to upgrade service to the latest version:

• Navigate to the directory with Etlworks Windows installer, for examplec:\etlworks. Note: The directory must contain a valid license file.
• Execute Etlworks.exe.

After installation is over it takes up to a few minutes for Etlworks service to restart.

Troubleshoot upgrade

If, after installing the upgrade, the service is not coming online:

Or ETL engine version is not available in About:

Then, the upgrade wasn't successful.

Here are the steps to fix:

Step 1. Stop the Tomcat service.

Step 2. Navigate to TOMCAT_HOME/webapps and manually delete files etl.war, ROOT.war and folders etl and ROOT.

Step 3. Run Etlworks.exe again.

Maintenance

Maintenance on Linux

Stop service on Linux

• Navigate to/opt/etlworksdirectory.
• Executestopcommand.
  

  sudo ./etlworks-cli.sh stop

• Optional parameters/flags
  • --skip-confirm
    • alias: -s
    • skip confirmation dialog

Start service on Linux

• Navigate to/opt/etlworksdirectory.
• Executestartcommand.
  

  sudo ./etlworks-cli.sh start

View service status on Linux

• Navigate to/opt/etlworksdirectory.
• Executestatuscommand.
  

  sudo ./etlworks-cli.sh status

View installed version on Linux

• Navigate to/opt/etlworksdirectory.
• Executeversioncommand.
  

  sudo ./etlworks-cli.sh version

View service logs on Linux

• Navigate to/opt/etlworksdirectory.
• Executelogcommand. 
  

  sudo ./etlworks-cli.sh log

• Optional parameters/flags
  • --engine
    • alias: -e
    • show ETL engine log instead of the application log
  • --follow
    • alias: -f
    • live log output
  • --lines NUMBER_OF_LINES
    • alias: -n NUMBER_OF_LINES
    • number of lines to show

Backup service on Linux

• Navigate to/opt/etlworksdirectory.
• Executebackupcommand. 
  

  sudo ./etlworks-cli.sh backup

• On your Linux system, backups are automatically created in:
  
  • Postges: /opt/etlworks/backup/postgres/integrator_dump_yyyyMMddhhmmss.sql
  • Redis: /opt/etlworks/backup/redis/dump_yyyyMMddhhmmss.rdb

Restore service from backup on Linux

Restoring PostgreSQL and Redis from Backup on Linux.

Restoring from backup will permanently delete current state.

• Identify the Latest Backup files. List available PostgreSQL and Redis backups: 
  

  ls -lt /opt/etlworks/backup/postgres/integrator_dump_*.sql
  ls -lt /opt/etlworks/backup/postgres/dump_*.rdb

• Navigate to/opt/etlworksdirectory.
• Executerestorecommand. Make sure to replace example paths to backup files with actual paths obtained in the first step
  

  sudo ./etlworks-cli.sh restore --postgres /opt/etlworks/backup/postgres/integrator_dump_YYYYMMddhhmmss.sql --redis /opt/etlworks/backup/redis/dump_YYYYMMddhhmmss.rdb

Maintenance on Windows

Stop service on Windows

• Click on the Apache Tomcat icon in the Windows system tray.

• Click the Stop button. If the button is grayed out, the service is already stopped.

Start service on Windows

• Click on the Apache Tomcat icon in the Windows system tray.

• Click the Start button. If the button is grayed out, the service is already started.

View service logs on Windows

On Windows, you can find service logs under /TOMCAT_HOME/logs directory, which by default is C:\Program Files\Apache Software Foundation\Tomcat 8.5\logs.

Backup service on Windows

To manually back up PostgreSQL and Redis on Windows, you need to create dumps of their data and store them in the c:\etlworks\backup folder. Since Windows does not have an automated CLI backup script like Linux, you will perform these steps manually.

Backup PostgreSQL on Windows

Step 1. Open Command Prompt as Administrator.

Step 2. Navigate to the PostgreSQL bin directory (adjust the path as needed based on your installation location). Example:

cd "C:\Program Files\PostgreSQL\9.5\bin"

Step 3. Run the following command to create a backup of the database (replace your_password as needed). If prompted, enter the PostgreSQL password.

pg_dump -U postgres -F p -b -v -f "C:\etlworks\backup\postgres\integrator_dump.sql" integrator

(replace postges with an actual postges user)

Backup Redis on Windows

For Redis, back up the dump.rdb file from the Redis data directory. The default location is C:\ProgramData\Redis\dump.rdb, so copy it to the backup folder (powershell):

Copy-Item "C:\ProgramData\Redis\dump.rdb" -Destination "C:\etlworks\backup\redis\dump.rdb" -Force

Restore service from backup on Windows

Your PostgreSQL and Redis backups are stored in c:\etlworks\backup.

Restoring PostgreSQL from backup on Windows

To restore PostgreSQL, open Command Prompt (cmd) as Administrator, navigate to the PostgreSQL bin directory, and run:

psql -U postgres -d integrator -f "C:\etlworks\backup\postgres\integrator_dump.sql"

(replace postges with an actual postges user)

If restoring to a new database, create it first:

createdb -U postgres integrator_new
psql -U postgres -d integrator_new -f "C:\etlworks\backup\postgres\integrator_dump.sql"

Restoring Redis from backup on Windows

Step 1. To restore Redis, first stop the Redis service by running PowerShell as Administrator and executing:

Stop-Service redis

Step 2. Then, replace the existing dump.rdb file with the backup:

Copy-Item "C:\etlworks\backup\redis\dump.rdb" -Destination "C:\ProgramData\Redis\dump.rdb" -Force

Step 3. Finally, restart Redis:

Start-Service redis

Installing connectors

Etlworks comes with hundreds of built-in connectors to meet a wide range of integration needs. However, it is also possible to add new regular or premium connectors as needed. Multiple connectors can be installed into TOMCAT_HOME/lib. This section provides step-by-step instructions for installing both types of connectors on the machine running the Etlworks.

In multi-node setup the connector and license must be installed on each node.

Installing Regular Connectors

Step1. Obtain the connector: Contact Etlworks support to obtain the connector. It will be provided as a .jar file and may include a license file and other required files.

Step 2. Copy the files: Copy all files obtained from Etlworks support to TOMCAT_HOME/lib.

Step 3. Restart Etlworks: Restart Etlworks to apply the changes.

Installing Premium Connectors

Step 1. Obtain the connector: Contact Etlworks support to obtain the connector. It will be provided as a .jar file along with a license key. If a license key is not provided, you can install a trial license.

Step 2. Copy the files: Copy the .jar file obtained from Etlworks support to TOMCAT_HOME/lib

Step 3. Navigate to the folder: Open a terminal and navigate (cd) to TOMCAT_HOME/lib.

Step 4. Run the license activation command:

java -jar connector.jar -license

Replace connector.jar with the actual name of the file you received from Etlworks.

Step 5. Provide the required information:

When prompted, enter the following:

• Name: support
• Email: support@etlworks.com
• License key: Enter the license key provided by Etlworks or type TRIAL for a trial license.

Step 6. Verify that license file successfully generated:
The license file name should match the connector file name, but with the .jar extension replaced by .lic.

For example, if the connector file is cdata.jdbc.salesforce.jar, the license file should be named cdata.jdbc.salesforce.lic.

Step 7. Restart the service.

Handle cases where the license file is not generated

Occasionally, the installer fails to generate the license file (including trial licenses). In this case, the output of the installer will provide a link to download the license file. Here is the step-by-step instruction for installing the license manually.

Step 1. Copy the link and open it in any browser. Follow the on-screen instructions to download the license file.

Step 2. After downloading the file, rename it to match the connector file name, replacing the .jar extension with .lic.

For example, if the connector file is cdata.jdbc.salesforce.jar, the license file should be named cdata.jdbc.salesforce.lic.

Step 3. Copy lic file to TOMCAT_HOME/lib.

Step 4. Restart the service.

Monitoring Etlworks instance health

To monitor the health of an Etlworks instance, we recommend using the /health API executed from an external orchestration tool.

Possible responses from the /health API endpoint

Everything is Good

• HTTP Response Code: 200

• Status: UP

• Response JSON:

{
   "status": "UP",
   ...
}

High Resource Utilization but Service is Operational

• HTTP Response Code: 200

• Status: WARNING

• Response JSON:

{
   "status": "WARRNING",
   ...
}

Details: The response JSON includes information about resources (RAM/CPU/Disk/Postgres/Redis) in a warning state. This is a temporary condition, but it should be monitored closely as it could lead to more severe issues.

Infrastructure Issues (Service Not Operational)

• HTTP Response Code: 200

• Status: DOWN

• Response JSON:

{
   "status": "DOWN",
   ...
}

Details: The response JSON includes information about resources (Disk/Postgres/Redis) that are down. This state requires action, typically a cleanup (e.g., disk cleanup) and a service restart.

API and UI Not Responsive

• HTTP Response Code: 504 (Timeout)

• Status: Timeout

Details: The service is running but very slow to respond. This might be a temporary condition; it is recommended to repeat the check several times with delays before deciding to restart the service.

Tomcat is Down but Reverse Proxy is UP

• HTTP Response Code: 502 (Bad Gateway)

• Status: DOWN

Details: The service is not running, likely because the Tomcat server exited with a core dump due to 100% RAM utilization. This requires a service restart.

Service is Down or Network Issues

• HTTP Response Code: Any 5xx code or ERR_CONNECTION_REFUSED

• Status: DOWN or temporary network issues

Details: The service is not running, or there are network issues preventing the client from reaching the API endpoint. Cloud VM migrations to a different host might also cause temporary unreachability (up to a few minutes). This might be a temporary condition; it is recommended to repeat the check several times with delays before deciding to restart the service. In rare cases, this might require restarting the VM.

Implementation Suggestions

• External Monitoring: Use an external orchestration tool like Airflow to perform health checks. 

• Data collection: If you need to integrate with a third-party data collection tool (e.g., Grafana), use a non production Etlworks instance (e.g., Dev) to push the data collected from the /health endpoint of the primary instance to that tool.

• Retry Logic: Implement retry logic for scenarios where the service is slow to respond, allowing for temporary conditions to resolve before taking action.

• Alerts and Remediation: Automate alerts and remediation steps (e.g., disk cleanup, service restart) based on the status and resource information provided by the health check.

Run Etlworks in Docker

To run Etlworks in Docker please use our official image.

AWS multi-node deployment

Etlworks can be deployed to symmetrical, horizontally scalable cluster in AWS.

Read more. 

On-premise multi-node deployment

Our installer supports the automatic provisioning of the Etlworks servers and shared components (HaProxy, PostgreSQL and Redis) in any cluster running on Linux OS.

Here is a simplified deployment diagram for an on-premise multi-node deployment.