• Starter
• Business
• Enterprise
• On-Premise
• Add-on

Overview

Etlworks can be installed on Linux, Windows, and MacOS. 

Supported operating systems and platforms:

• Amazon Linux 2
• Amazon Linux 2023
• Ubuntu 20.04
• Ubuntu 22.04
• CentOS 7
• Red Hat 7
• Red Hat 8
• Red Hat 9
• Windows Server (2012-2022)
• All editions of Windows 10 and Windows 11 
• MacOS via Docker 
• Docker

Hardware requirements

	CPU	RAM	Hard drive
Minimum	2 cores	8GB	100GB SSD
Better	4 cores	16GB	250GB SSD
Optimal	8-16 cores	32GB-64GB	500GB-1TB SSD

Read more.

Installation

1. Request a link to download an installer or license

To request a download link, open https://etlworks.com/downloads/request.html in any browser. Fill in the form and follow the on-screen instructions.

Select either Linux or Windows installer. Read how to run Etlworks in Docker. 

 

2. Install Etlworks

Install Etlworks on Linux

Linux installer is provided as a tar zip archive. The archive contains required libraries, a command-line script, and a unique license file.

• Download the archive with the installer and copyetlworks-installer.tar.gzfile into/opt/etlworksdirectory and then unarchive it.

  sudo tar -zxvf etlworks-installer.tar.gz

• Start the installation by executing:
  

  sudo ./etlworks-cli.sh install

• Optional parameters/flags:
  
  • --version APPLICATION_VERSION (Default: latest)
    • alias: -v APPLICATION_VERSION
    • application version to install. If not specified by default latest version will be installed
  • --skip-confirm
    • alias: -s
    • proceed with installation without confirmation
  • --user USER_NAME (Default: ec2-user)
    • alias: -u USER_NAME
    • sudoer user name
  • --no-app
    • alias: -n
    • do not install application artifacts, only provision the system

To view all available versions use list command: sudo ./etlworks-cli.sh list

Install Etlworks on Windows

Windows installer is provided as a zip archive. The archive contains required libraries, a Windows installer (Etlworks.exe), and a unique license file.

Some older versions of antivirus software, specifically Windows Defender before version 1.381.3081.0, report that Etlworks.exe contains a trojan. THIS IS A FALSE POSITIVE. To avoid warnings, we recommend updating your antivirus to the latest version before downloading and installing Etlworks on Windows.

• Download the archive with the installer and copyetlworks-installer.zipfile into any directory, for examplec:\etlworksand unarchive it.
• Start the installation by executing Etlworks.exe. 
• Continue with the setup

 

3. Finish installation and configuration

Keep the downloaded installer, libraries required for installation, and a provided license file. The same installer can be used to upgrade Etlworks to the latest version. If for any reason you lost the installer and required libraries you can re download it by requesting a new download link and providing the same Company, Name and Email that you used when you downloaded the installer first time. 

After the installation process is over, the service should be available on port8080. You can test it by opening the Etlworks URL in the browser, for example: http://localhost:8080. The default Super Admin username and password are:

• user: admin
• password: admin1

After you first log in, navigate toSettings.

Set Home URL

UnderGeneral, set Home URLto your instance public URL, for example http://myhost:8080.

Configure Email

This is required for adding new users, resetting passwords, and sending email notifications.

UnderEmail, provide the email configuration that will be used by the system to send notifications. 

Read more.

AWS multi-node deployment

Read how to provision multi-node Etlworks cluster on AWS. 

Upgrade

Upgrade Etlworks on Linux

In order to upgrade service to the latest version:

• Navigate to/opt/etlworksdirectory.
  

  sudo ./etlworks-cli.sh upgrade

• Optional parameters/flags:
  
  • --version APPLICATION_VERSION (Default: latest)
    • alias: -v APPLICATION_VERSION
    • application version to upgrade to
  • --skip-confirm
    • alias: -s
    • proceed with upgrade without confirmation

To view all available versions use list command: sudo ./etlworks-cli.sh list

After installation is over it takes up to a few minutes for Etlworks service to restart.

Upgrade Etlworks on Windows

In order to upgrade service to the latest version:

• Navigate to the directory with Etlworks Windows installer, for examplec:\etlworks. Note: The directory must contain a valid license file.
• Execute Etlworks.exe.

After installation is over it takes up to a few minutes for Etlworks service to restart.

Troubleshoot upgrade

If, after installing the upgrade, the service is not coming online:

Or ETL engine version is not available in About:

Then, the upgrade wasn't successful.

Here are the steps to fix:

Step 1. Stop the Tomcat service.

Step 2. Navigate to TOMCAT_HOME/webapps and manually delete files etl.war, ROOT.war and folders etl and ROOT.

Step 3. Run Etlworks.exe again.

Maintenance

Maintenance on Linux

Stop service on Linux

• Navigate to/opt/etlworksdirectory.
• Executestopcommand.
  

  sudo ./etlworks-cli.sh stop

• Optional parameters/flags
  • --skip-confirm
    • alias: -s
    • skip confirmation dialog

Start service on Linux

• Navigate to/opt/etlworksdirectory.
• Executestartcommand.
  

  sudo ./etlworks-cli.sh start

View service status on Linux

• Navigate to/opt/etlworksdirectory.
• Executestatuscommand.
  

  sudo ./etlworks-cli.sh status

View installed version on Linux

• Navigate to/opt/etlworksdirectory.
• Executeversioncommand.
  

  sudo ./etlworks-cli.sh version

View service logs on Linux

• Navigate to/opt/etlworksdirectory.
• Executelogcommand. 
  

  sudo ./etlworks-cli.sh log

• Optional parameters/flags
  • --engine
    • alias: -e
    • show ETL engine log instead of the application log
  • --follow
    • alias: -f
    • live log output
  • --lines NUMBER_OF_LINES
    • alias: -n NUMBER_OF_LINES
    • number of lines to show

Backup service on Linux

• Navigate to/opt/etlworksdirectory.
• Executebackupcommand. 
  

  sudo ./etlworks-cli.sh backup

• All backup files will be created under/opt/etlworks/backupdirectory.

Maintenance on Windows

Stop service on Windows

• Click on the Apache Tomcat icon in the Windows system tray.

• Click the Stop button. If the button is grayed out, the service is already stopped.

Start service on Windows

• Click on the Apache Tomcat icon in the Windows system tray.

• Click the Start button. If the button is grayed out, the service is already started.

View service logs on Windows

On Windows, you can find service logs under /TOMCAT_HOME/logs directory, which by default is C:\Program Files\Apache Software Foundation\Tomcat 8.5\logs.

Monitoring Etlworks instance health

To monitor the health of an Etlworks instance, we recommend using the /health API executed from an external orchestration tool.

Possible responses from the /health API endpoint

Everything is Good

• HTTP Response Code: 200

• Status: UP

• Response JSON:

{
   "status": "UP",
   ...
}

High Resource Utilization but Service is Operational

• HTTP Response Code: 200

• Status: WARNING

• Response JSON:

{
   "status": "WARRNING",
   ...
}

Details: The response JSON includes information about resources (RAM/CPU/Disk/Postgres/Redis) in a warning state. This is a temporary condition, but it should be monitored closely as it could lead to more severe issues.

Infrastructure Issues (Service Not Operational)

• HTTP Response Code: 200

• Status: DOWN

• Response JSON:

{
   "status": "DOWN",
   ...
}

Details: The response JSON includes information about resources (Disk/Postgres/Redis) that are down. This state requires action, typically a cleanup (e.g., disk cleanup) and a service restart.

API and UI Not Responsive

• HTTP Response Code: 504 (Timeout)

• Status: Timeout

Details: The service is running but very slow to respond. This might be a temporary condition; it is recommended to repeat the check several times with delays before deciding to restart the service.

Tomcat is Down but Reverse Proxy is UP

• HTTP Response Code: 502 (Bad Gateway)

• Status: DOWN

Details: The service is not running, likely because the Tomcat server exited with a core dump due to 100% RAM utilization. This requires a service restart.

Service is Down or Network Issues

• HTTP Response Code: Any 5xx code or ERR_CONNECTION_REFUSED

• Status: DOWN or temporary network issues

Details: The service is not running, or there are network issues preventing the client from reaching the API endpoint. Cloud VM migrations to a different host might also cause temporary unreachability (up to a few minutes). This might be a temporary condition; it is recommended to repeat the check several times with delays before deciding to restart the service. In rare cases, this might require restarting the VM.

Implementation Suggestions

• External Monitoring: Use an external orchestration tool like Airflow to perform health checks. 

• Data collection: If you need to integrate with a third-party data collection tool (e.g., Grafana), use a non production Etlworks instance (e.g., Dev) to push the data collected from the /health endpoint of the primary instance to that tool.

• Retry Logic: Implement retry logic for scenarios where the service is slow to respond, allowing for temporary conditions to resolve before taking action.

• Alerts and Remediation: Automate alerts and remediation steps (e.g., disk cleanup, service restart) based on the status and resource information provided by the health check.

Run Etlworks in Docker

To run Etlworks in Docker please use our official image.

AWS multi-node deployment

Etlworks can be deployed to symmetrical, horizontally scalable cluster in AWS.

Read more. 

On-premise multi-node deployment

Our installer supports the automatic provisioning of the Etlworks servers and shared components (HaProxy, PostgreSQL and Redis) in any cluster running on Linux OS.

Here is a simplified deployment diagram for an on-premise multi-node deployment.