- Starter
- Business
- Enterprise
- On-Premise
- Add-on
Overview
Etlworks can be installed on Linux, Windows, and MacOS.
Supported operating systems and platforms:
- Amazon Linux 2
- Amazon Linux 2023
- Ubuntu 20.04
- Ubuntu 22.04
- CentOS 7
- Red Hat 7
- Red Hat 8
- Red Hat 9
- Windows Server (2012-2022)
- All editions of Windows 10 and Windows 11
- MacOS via Docker
- Docker
Hardware requirements
CPU | RAM | Hard drive | |
Minimum | 2 cores | 8GB | 100GB SSD |
Better | 4 cores | 16GB | 250GB SSD |
Optimal | 8-16 cores | 32GB-64GB | 500GB-1TB SSD |
License Validation and Firewall
Etlworks on-premise deployments require outbound communication from the on-premise server(s) to our license server (host: lic.etlworks.com, dedicated IP address: 18.217.236.168) over HTTPS (port 443). This communication is strictly outbound and does not require opening inbound firewall rules.
Key Requirements
- All servers running Etlworks in the environment must be able to access: lic.etlworks.com:443 (outbound traffic only).
- License validation occurs every hour to ensure the instance remains authorized.
- Each node in multi-node deployments independently reports license validation data.
Firewall
Inbound Firewall
No inbound firewall changes are required for Etlworks to function in standard on-premise deployments.
However, if the Etlworks instance is used to serve API requests, either from built-in APIs or user-defined APIs, and these requests originate from outside the firewall, the following inbound ports should be opened:
- Port 80 (HTTP) – If APIs are served without SSL.
- Port 443 (HTTPS) – If APIs are served over a secure connection.
Outbound Firewall
- Etlworks requires outbound access to lic.etlworks.com:443 (dedicated IP address: 18.217.236.168) for license validation.
- If an outbound firewall is in place, port 443 (HTTPS) must be open for communication with our license server.
- If the on-premise Etlworks server needs to access external resources—such as databases, message brokers, cloud APIs, or third-party services—the relevant outbound ports must also be opened to allow connectivity.
Data Transmitted During License Validation
During each license validation session, the Etlworks server(s) securely transmit the following information to our license server:
- Server identification details, including IP address, Etlworks version, and digital fingerprints unique to the instance.
- Digital fingerprints include non-sensitive hardware metadata (physical or virtual).
- Instance usage metrics: number of flows, connections, formats, listeners, schedules, and agents.
- System health status, as returned by the /health API.
Security Measures
- All transmitted data is encrypted using a pair of environment-specific encryption keys, ensuring that only Etlworks and the authorized instance can read the payload.
- No sensitive or customer data is ever transmitted.
- Neither our license server nor other elements of Etlworks infrastructure ever initiate inbound connections to on-premise servers.
Failure Scenarios
- If a license is found to be invalid or has been remotely deactivated by Etlworks, the server(s) will stop functioning during the next validation check.
- If an on-premise server fails to connect to the license server for 24 hours, it will automatically shut down until connectivity is restored.
Installation
1. Request a link to download an installer or license
To request a download link, open https://etlworks.com/downloads/request.html in any browser. Fill in the form and follow the on-screen instructions.
Enter a valid email address. The Etlworks license server will send a short-lived download link to the email provided during registration
If you already have an account with Etlworks, please enter the same Company, Name, and Email used when the existing account was created. If you’re unsure about these details, feel free to contact Etlworks support at support@etlworks.com for assistance.
2. Open the download link
After submitting the registration form, the Etlworks license server will generate a personalized download link and send it to the registered email address. This link will remain active for approximately 10 minutes. Make sure to access it within this time frame to download the required files.
3. Select the installer to download
Select either Linux or Windows installer. Read how to run Etlworks in Docker.
4. Install Etlworks
Install Etlworks on Linux
Linux installer is provided as a tar zip archive. The archive contains required libraries, a command-line script, and a unique license file.
- Download the archive with the installer and copy
etlworks-installer.tar.gz
file into/opt/etlworks
directory and then unarchive it.sudo tar -zxvf etlworks-installer.tar.gz
- Start the installation by executing:
sudo ./etlworks-cli.sh install
- Optional parameters/flags:
-
--version APPLICATION_VERSION (Default: latest)
- alias: -v APPLICATION_VERSION
- application version to install. If not specified by default latest version will be installed
-
--skip-confirm
- alias: -s
- proceed with installation without confirmation
-
--user USER_NAME (Default: ec2-user)
- alias: -u USER_NAME
- sudoer user name
-
--no-app
- alias: -n
- do not install application artifacts, only provision the system
-
--version APPLICATION_VERSION (Default: latest)
To view all available versions use list
command: sudo ./etlworks-cli.sh list
Install Etlworks on Windows
Windows installer is provided as a zip archive. The archive contains required libraries, a Windows installer (Etlworks.exe), and a unique license file.
Some older versions of antivirus software, specifically Windows Defender before version 1.381.3081.0, report that Etlworks.exe contains a trojan. THIS IS A FALSE POSITIVE. To avoid warnings, we recommend updating your antivirus to the latest version before downloading and installing Etlworks on Windows.
- Download the archive with the installer and copy
etlworks-installer.zip
file into any directory, for examplec:\etlworks
and unarchive it. - Start the installation by executing
Etlworks.exe
. - Continue with the setup
5. Finish installation and configuration
Keep the downloaded installer, libraries required for installation, and a provided license file. The same installer can be used to upgrade Etlworks to the latest version. If for any reason you lost the installer and required libraries you can re download it by requesting a new download link and providing the same Company, Name and Email that you used when you downloaded the installer first time.
After the installation process is over, the service should be available on port8080
. You can test it by opening the Etlworks URL in the browser, for example: http://localhost:8080
. The default Super Admin username and password are:
- user: admin
- password: admin1
After you first log in, navigate toSettings
.
Set Home URL
UnderGeneral
, set Home URL
to your instance public URL, for example http://myhost:8080
.
Configure Email
This is required for adding new users, resetting passwords, and sending email notifications.
UnderEmail
, provide the email configuration that will be used by the system to send notifications.
AWS multi-node deployment
Read how to provision multi-node Etlworks cluster on AWS.
Upgrade
Upgrade Etlworks on Linux
In order to upgrade service to the latest version:
- Navigate to
/opt/etlworks
directory.
sudo ./etlworks-cli.sh upgrade
- Optional parameters/flags:
-
--version APPLICATION_VERSION (Default: latest)
- alias: -v APPLICATION_VERSION
- application version to upgrade to
-
--skip-confirm
- alias: -s
- proceed with upgrade without confirmation
-
--version APPLICATION_VERSION (Default: latest)
To view all available versions use list
command: sudo ./etlworks-cli.sh list
After installation is over it takes up to a few minutes for Etlworks service to restart.
Upgrade Etlworks on Windows
In order to upgrade service to the latest version:
- Navigate to the directory with Etlworks Windows installer, for example
c:\etlworks
. Note: The directory must contain a valid license file. - Execute
Etlworks.exe
.
After installation is over it takes up to a few minutes for Etlworks service to restart.
Troubleshoot upgrade
If, after installing the upgrade, the service is not coming online:
Or ETL engine version is not available in About:
Then, the upgrade wasn't successful.
Here are the steps to fix:
Step 1. Stop the Tomcat service.
Step 2. Navigate to TOMCAT_HOME/webapps
and manually delete files etl.war
, ROOT.war
and folders etl
and ROOT
.
Step 3. Run Etlworks.exe again.
Maintenance
Maintenance on Linux
Stop service on Linux
- Navigate to
/opt/etlworks
directory. - Execute
stop
command.
sudo ./etlworks-cli.sh stop
- Optional parameters/flags
-
--skip-confirm
- alias: -s
- skip confirmation dialog
-
Start service on Linux
- Navigate to
/opt/etlworks
directory. - Execute
start
command.
sudo ./etlworks-cli.sh start
View service status on Linux
- Navigate to
/opt/etlworks
directory. - Execute
status
command.
sudo ./etlworks-cli.sh status
View installed version on Linux
- Navigate to
/opt/etlworks
directory. - Execute
version
command.
sudo ./etlworks-cli.sh version
View service logs on Linux
- Navigate to
/opt/etlworks
directory. - Execute
log
command.
sudo ./etlworks-cli.sh log
- Optional parameters/flags
-
--engine
- alias: -e
- show ETL engine log instead of the application log
-
--follow
- alias: -f
- live log output
-
--lines NUMBER_OF_LINES
- alias: -n NUMBER_OF_LINES
- number of lines to show
-
--engine
Backup service on Linux
- Navigate to
/opt/etlworks
directory. - Execute
backup
command.
sudo ./etlworks-cli.sh backup
- On your Linux system, backups are automatically created in:
- Postges:
/opt/etlworks/backup/postgres/integrator_dump_yyyyMMddhhmmss.sql
- Redis:
/opt/etlworks/backup/redis/dump_yyyyMMddhhmmss.rdb
- Postges:
Restore service from backup on Linux
Restoring PostgreSQL and Redis from Backup on Linux.
Restoring from backup will permanently delete current state.
- Identify the Latest Backup files. List available PostgreSQL and Redis backups:
ls -lt /opt/etlworks/backup/postgres/integrator_dump_*.sql
ls -lt /opt/etlworks/backup/postgres/dump_*.rdb
- Navigate to
/opt/etlworks
directory. - Execute
restore
command. Make sure to replace example paths to backup files with actual paths obtained in the first step
sudo ./etlworks-cli.sh restore --postgres /opt/etlworks/backup/postgres/integrator_dump_YYYYMMddhhmmss.sql --redis /opt/etlworks/backup/redis/dump_YYYYMMddhhmmss.rdb
Maintenance on Windows
Stop service on Windows
- Click on the Apache Tomcat icon in the Windows system tray.
- Click the Stop button. If the button is grayed out, the service is already stopped.
Start service on Windows
- Click on the Apache Tomcat icon in the Windows system tray.
- Click the Start button. If the button is grayed out, the service is already started.
View service logs on Windows
On Windows, you can find service logs under /TOMCAT_HOME/logs
directory, which by default is C:\Program Files\Apache Software Foundation\Tomcat 8.5\logs
.
Backup service on Windows
To manually back up PostgreSQL and Redis on Windows, you need to create dumps of their data and store them in the c:\etlworks\backup folder. Since Windows does not have an automated CLI backup script like Linux, you will perform these steps manually.
Backup PostgreSQL on Windows
Step 1. Open Command Prompt as Administrator.
Step 2. Navigate to the PostgreSQL bin directory (adjust the path as needed based on your installation location). Example:
cd "C:\Program Files\PostgreSQL\9.5\bin"
Step 3. Run the following command to create a backup of the database (replace your_password as needed). If prompted, enter the PostgreSQL password.
pg_dump -U postgres -F p -b -v -f "C:\etlworks\backup\postgres\integrator_dump.sql" integrator
(replace postges with an actual postges user)
Backup Redis on Windows
For Redis, back up the dump.rdb
file from the Redis data directory. The default location is C:\ProgramData\Redis\dump.rdb
, so copy it to the backup folder (powershell):
Copy-Item "C:\ProgramData\Redis\dump.rdb" -Destination "C:\etlworks\backup\redis\dump.rdb" -Force
Restore service from backup on Windows
Your PostgreSQL and Redis backups are stored in c:\etlworks\backup.
Restoring PostgreSQL from backup on Windows
To restore PostgreSQL, open Command Prompt (cmd) as Administrator, navigate to the PostgreSQL bin
directory, and run:
psql -U postgres -d integrator -f "C:\etlworks\backup\postgres\integrator_dump.sql"
(replace postges with an actual postges user)
If restoring to a new database, create it first:
createdb -U postgres integrator_new
psql -U postgres -d integrator_new -f "C:\etlworks\backup\postgres\integrator_dump.sql"
Restoring Redis from backup on Windows
Step 1. To restore Redis, first stop the Redis service by running PowerShell as Administrator and executing:
Stop-Service redis
Step 2. Then, replace the existing dump.rdb
file with the backup:
Copy-Item "C:\etlworks\backup\redis\dump.rdb" -Destination "C:\ProgramData\Redis\dump.rdb" -Force
Step 3. Finally, restart Redis:
Start-Service redis
Installing connectors
Etlworks comes with hundreds of built-in connectors to meet a wide range of integration needs. However, it is also possible to add new regular or premium connectors as needed. Multiple connectors can be installed into TOMCAT_HOME/lib
. This section provides step-by-step instructions for installing both types of connectors on the machine running the Etlworks.
In multi-node setup the connector and license must be installed on each node.
Installing Regular Connectors
Step1. Obtain the connector: Contact Etlworks support to obtain the connector. It will be provided as a .jar
file and may include a license file and other required files.
Step 2. Copy the files: Copy all files obtained from Etlworks support to TOMCAT_HOME/lib
.
Step 3. Restart Etlworks: Restart Etlworks to apply the changes.
Installing Premium Connectors
Step 1. Obtain the connector: Contact Etlworks support to obtain the connector. It will be provided as a .jar
file along with a license key. If a license key is not provided, you can install a trial license.
Step 2. Copy the files: Copy the .jar
file obtained from Etlworks support to TOMCAT_HOME/lib
Step 3. Navigate to the folder: Open a terminal and navigate (cd
) to TOMCAT_HOME/lib
.
Step 4. Run the license activation command:
java -jar connector.jar -license
Replace connector.jar
with the actual name of the file you received from Etlworks.
Step 5. Provide the required information:
When prompted, enter the following:
-
Name:
support
-
Email:
support@etlworks.com
-
License key: Enter the license key provided by Etlworks or type
TRIAL
for a trial license.
Step 6. Verify that license file successfully generated:
The license file name should match the connector file name, but with the .jar
extension replaced by .lic
.
For example, if the connector file is cdata.jdbc.salesforce.jar
, the license file should be named cdata.jdbc.salesforce.lic
.
Step 7. Restart the service.
Handle cases where the license file is not generated
Occasionally, the installer fails to generate the license file (including trial licenses). In this case, the output of the installer will provide a link to download the license file. Here is the step-by-step instruction for installing the license manually.
Step 1. Copy the link and open it in any browser. Follow the on-screen instructions to download the license file.
Step 2. After downloading the file, rename it to match the connector file name, replacing the .jar
extension with .lic
.
For example, if the connector file is cdata.jdbc.salesforce.jar
, the license file should be named cdata.jdbc.salesforce.lic
.
Step 3. Copy lic file to TOMCAT_HOME/lib.
Step 4. Restart the service.
Monitoring Etlworks instance health
To monitor the health of an Etlworks instance, we recommend using the /health API executed from an external orchestration tool.
Possible responses from the /health API endpoint
Everything is Good
• HTTP Response Code: 200
• Status: UP
• Response JSON:
{
"status": "UP",
...
}
High Resource Utilization but Service is Operational
• HTTP Response Code: 200
• Status: WARNING
• Response JSON:
{
"status": "WARRNING",
...
}
Details: The response JSON includes information about resources (RAM/CPU/Disk/Postgres/Redis) in a warning state. This is a temporary condition, but it should be monitored closely as it could lead to more severe issues.
Infrastructure Issues (Service Not Operational)
• HTTP Response Code: 200
• Status: DOWN
• Response JSON:
{
"status": "DOWN",
...
}
Details: The response JSON includes information about resources (Disk/Postgres/Redis) that are down. This state requires action, typically a cleanup (e.g., disk cleanup) and a service restart.
API and UI Not Responsive
• HTTP Response Code: 504 (Timeout)
• Status: Timeout
Details: The service is running but very slow to respond. This might be a temporary condition; it is recommended to repeat the check several times with delays before deciding to restart the service.
Tomcat is Down but Reverse Proxy is UP
• HTTP Response Code: 502 (Bad Gateway)
• Status: DOWN
Details: The service is not running, likely because the Tomcat server exited with a core dump due to 100% RAM utilization. This requires a service restart.
Service is Down or Network Issues
• HTTP Response Code: Any 5xx code or ERR_CONNECTION_REFUSED
• Status: DOWN or temporary network issues
Details: The service is not running, or there are network issues preventing the client from reaching the API endpoint. Cloud VM migrations to a different host might also cause temporary unreachability (up to a few minutes). This might be a temporary condition; it is recommended to repeat the check several times with delays before deciding to restart the service. In rare cases, this might require restarting the VM.
Implementation Suggestions
• External Monitoring: Use an external orchestration tool like Airflow to perform health checks.
• Data collection: If you need to integrate with a third-party data collection tool (e.g., Grafana), use a non production Etlworks instance (e.g., Dev) to push the data collected from the /health endpoint of the primary instance to that tool.
• Retry Logic: Implement retry logic for scenarios where the service is slow to respond, allowing for temporary conditions to resolve before taking action.
• Alerts and Remediation: Automate alerts and remediation steps (e.g., disk cleanup, service restart) based on the status and resource information provided by the health check.
Run Etlworks in Docker
To run Etlworks in Docker please use our official image.
AWS multi-node deployment
Etlworks can be deployed to symmetrical, horizontally scalable cluster in AWS.
On-premise multi-node deployment
Our installer supports the automatic provisioning of the Etlworks servers and shared components (HaProxy, PostgreSQL and Redis) in any cluster running on Linux OS.
Here is a simplified deployment diagram for an on-premise multi-node deployment.
Comments
0 comments
Please sign in to leave a comment.