- Starter
- Business
- Enterprise
- On-Premise
- Add-on
Overview
This article outlines how to install, configure and monitor the remote Integration Agent.
Add new Integration Agent
Step 1. Open Agents
window. Click Add
button.
Step 2. Enter the Agent name, click Save
.
Managing Integration Agents: Duplicate and Delete
When working with integration agents in the platform, you have the option to easily manage them using two key actions: Duplicate and Delete, both accessible via buttons next to each agent’s listing.
Duplicate: The button (shown as two overlapping squares) allows you to create a copy of the existing agent. This is particularly useful when you need to replicate an agent’s configuration and settings for a new integration without having to manually re-enter all parameters. The duplicate agent will inherit all settings from the original, including version, OS, and any specific configurations.
Delete: The trash can icon enables you to remove an agent from the system. Once deleted, the agent and its configurations are permanently removed. Be cautious when using this option, especially for critical agents, as this action cannot be undone.
Download and install the Integration Agent
Install the Integration Agent on Windows
Step 1. In Agents
window select previously created Integration Agent.
Step 2. Click Download
at the bottom of the screen.
Step 3. Select Windows
OS. Click Agent
button. It will download the Agent installer into your local machine.
Windows installer is provided as integrationagent.exe
.
Step 4. Run the downloaded installer.
Step 5. Enter your Etlworks User ID (or email) and Etlworks instance URL.
After installing the Agent you must install a license.
Step 6. Continue to Install a license.
Install the Integration Agent on Linux
Step 1. In Agents
window select previously created Integration Agent.
Step 2. Click Download
at the bottom of the screen.
Step 3. Select Linux
OS. Click Agent
button. It will download the Agent installer into your local machine.
Linux installer is provided as a tar zip archive integrationagent.tar.gz
. The archive contains required libraries and a command-line scripts.
Step 4. Create a folder for the agent in the hard drive, for example, /opt/agent
, copy integrationagent.tar.gz file into that folder and then unarchive it.
sudo tar -zxvf integrationagent.tar.gz
Step 5. cd to the folder, start the installation by executing:
sudo ./install.sh
After installing the Agent you must install a license.
Step 6. Continue to Install a license.
Install the Integration Agent as a Docker container
Read how to run Integration Agent as a docker container.
Install a license and start the Agent
Integration Agent requires a license.
To obtain and install the license:
Step 1. Open Agents
window. Select the Agent to download the license for.
Step 2. Click Download
at the bottom of the screen.
Step 3. Click License
. The licenses for Windows and Linux are the same.
Step 4. Check if Agent is running:
Step 5. If Integration Agent is running stop Agent service.
Step 6. If exist delete files AGENT_HOME/license/*.alic
and AGENT_HOME/license/*.chk
. DO NOT DELETE AGENT_HOME/license/integration-agent-online.lic
.
Step 7. Copy the new license file into the AGENT_HOME/license
.
Step 8. Start Agent service.
Configure the Integration Agent to run as a Windows Service
By default, the Windows installer already installs the Integration Agent as a Windows Service so this step is optional.
If you will ever need to re-install Integration Agent as a Windows Service here are the steps:
- Open elevated Command Prompt (Admin).
-
cd
into the directory where you have installed the agent, for example,cd c:\etlworks\integration-agent
. - Execute
install.cmd
. - Verify that service is installed by opening the Services desktop app. There must be a service with the name
IntegrationAgent
and descriptionEtlworks LLC. Integration Agent
.
Configure the Integration Agent to run as a Linux Service
By default, the Linux installer already installs the Integration Agent as a Linux Service so this step is optional.
If you will ever need to re-install Integration Agent as a Linix Service here are the steps:
- Open terminal.
-
cd
into the directory where you have installed the agent, for example,cd \opt\agent
. - Execute
sudo .\install.sh
.
Update Service
When Windows or Linux installer installs the Agent it also installs the UpdateService. The UpdateService can be used to remotely start, stop, restart, update and reconfigure the Agent from Etlworks UI.
If you are on version of Agent above or equal to 5.9.0-SNAPSHOT
but below 6.3.4-SNAPSHOT
you must download and install UpdateService. Agent installer version 6.3.4-SNAPSHOT
and above comes bundled with the UpdateService.
Install Update Service
Here is how to download and install the UpdateService:
Step 1. Click Download
button.
Step 2. Select OS and click Update service
.
Step 3 (Windows). Windows installer is provided as updateservice.exe
.
Run the downloaded installer.
Make sure it install into the same folder as Agent.
The installer will install all file and start the UpdateService.
Step 3 (Linux). Linux installer is provided as updateservice.tar.gz
.
Unarchive the file into the same folder as Agent.
sudo tar -zxvf integrationagent.tar.gz
Step 4. cd to the folder, start the installation by executing:
sudo ./update-service-install-and-start.sh
This will install all file and start the UpdateService.
Step 5. Verify that UpdateService is running
1. Click Status
button
2. Check the Updater status.
Manually start, stop, install and remove Update Service
Once installed, the Update Service will persist through machine restarts, as it is configured to automatically start on reboots. You can manually start, stop and remove he service using provided scripts.
Manage Update Service on Windows
Stop
- Open elevated Command Prompt (Admin).
-
cd
into the directory where you installed the agent, for examplecd c:\integration-agent
. - Execute
update-service-stop.cmd
.
Start
- Open elevated Command Prompt (Admin).
-
cd
into the directory where you installed the agent, for examplecd c:\integration-agent
. - Execute
update-service-start.cmd
.
Remove service
- Open elevated Command Prompt (Admin).
-
cd
into the directory where you installed the agent, for examplecd c:\integration-agent
. - Execute
update-service-remove.cmd
.
Install service
- Open elevated Command Prompt (Admin).
-
cd
into the directory where you installed the agent, for examplecd c:\integration-agent
. - Execute
update-service-install.cmd
.
Install and start service
- Open elevated Command Prompt (Admin).
-
cd
into the directory where you installed the agent, for examplecd c:\integration-agent
. - Execute
update-service-instal-and-start.cmd
.
Manage Update Service on Linux
Stop
- Open terminal.
-
cd
into the directory where you installed the agent, for examplecd \opt\agent
. - Execute
sudo .\update-service-stop.sh
Start
- Open terminal.
-
cd
into the directory where you installed the agent, for examplecd \opt\agent
. - Execute
sudo .\update-service-start.sh
Install service
- Open terminal.
-
cd
into the directory where you installed the agent, for examplecd \opt\agent
. - Execute
sudo .\update-service-install.sh
Install and start service
- Open terminal.
-
cd
into the directory where you installed the agent, for examplecd \opt\agent
. - Execute
sudo .\update-service-install-and-start.sh
Manage Agent from Etlworks UI
It is possible to remotely start, stop, restart, update and reconfigure the Agent from Etlworks UI. This requires the UpdateService which must be installed and running.
The Integration Agent is not configured to start automatically; it needs to be manually started using the UI or a command line script after the installation and machine restart.
When the user triggers any action, such as Stop, Start, Restart, etc. the action is added to the queue. The actions in a queue are executed in order on the next communication session between the UpdateService and host Etlworks instance. It can take up to a few minutes for the action to complete. Read about Actions Queue.
Start Agent
The Agent can be remotely started if it is not running. It can take up to 120 seconds to start the agent.
1. Click Status
button
2. If available click Start
button
3. Wait until the status changes to Online
Stop Agent
The running Agent can be remotely stopped if it is not running. It can take up to 120 seconds to stop the agent.
1. Click Status
button
2. If available click Stop
button.
3. Wait until the status changes to Offline
.
Restart Agent
The Agent can be remotely restarted if it is running or started if it is not running. It can take up to 120 seconds to restart the agent. If there is an update in the /update folder the restart will install the update.
1. Click Status
button
2. If available click Restart
button.
3. Wait until the status changes to Offline
followed by Online
Change maximum memory available to the Agent
It is possible to remotely change the maximum memory available to the Agent.
1. Click Status
button
2. Navigate to Memory Max
and click Configure
button.
3. Enter Max available memory in megabytes or set it to auto
to set the max memory to 70% of overall system RAM. Click Update
button.
4. Wait until the update service reconfigures the Agent, and status changes to Offline
followed by Online
5. Verify that the memory configuration has changed by opening the Status screen and navigating to Memory Max
line.
Download Logs
There are two log files:
- Agent Log: The primary log file of the Integration Agent, containing errors and warnings related to the execution of flows on the Agent.
- Updater Log: The log file for the Update Service, detailing user interactions with the service, including starting, restarting, and updating the Agent.
To download logs click Agent log
or Updater log
button.
Wait until the the system will display a Download
button. Once available click the Download
button.
Manage Agent using command line scrips
Manage Agent on Windows
Start
- Open elevated Command Prompt (Admin).
-
cd
into the directory where you installed the agent, for examplecd c:\integration-agent
. - Execute
start.cmd
.
Stop
- Open elevated Command Prompt (Admin).
-
cd
into the directory where you installed the agent, for examplecd c:\integration-agent
. - Execute
stop.cmd
.
Remove service
- Open elevated Command Prompt (Admin).
-
cd
into the directory where you installed the agent, for examplecd c:\integration-agent
. - Execute
remove.cmd
.
Install service
- Open elevated Command Prompt (Admin).
-
cd
into the directory where you installed the agent, for examplecd c:\integration-agent
. - Execute
install.cmd
.
Install and start service
- Open elevated Command Prompt (Admin).
-
cd
into the directory where you installed the agent, for examplecd c:\integration-agent
. - Execute
instal-and-start.cmd
.
Change JRE and JVM memory parameters for Windows service
The Windows version of Agent is bundled with the JRE (Java Runtime Environment). JRE and JVM memory parameters are configured in AGENT_HOME/install.cmd
.
set JAVA_HOME=%PR_PATH%\jre
set PR_JVM=%JAVA_HOME%\bin\server\jvm.dll
set PR_JVMOPTIONS=-Xmx8000m
You can change the JRE to any valid JRE installed in the machine (it must be Java 8).
Similarly, you can also change the maximum memory that the JVM can allocate. The default is auto
which sets the RAM to 70% of total system RAM.
Change JVM memory parameters
Here's how you can do this:
- Stop the Agent.
- Modify
PR_JVMOPTIONS
inAGENT_HOME/install.cmd
and save it. You can set it to any of the following:- The max memory in megabytes:
PR_JVMOPTIONS=8000
- The valid -Xms parameter:
PR_JVMOPTIONS=-Xmx8000m
-
auto
which sets the RAM to 70% of total system RAM:PR_JVMOPTIONS=auto
- The max memory in megabytes:
- Run
AGENT_HOME/remove.cmd
AGENT_HOME/install.cmd
- Start the Agent.
Manage Agent on Linux
Start
- Open terminal.
-
cd
into the directory where you installed the agent, for examplecd \opt\agent
. - Execute
sudo .\start.sh
.
Stop
- Open terminal.
-
cd
into the directory where you installed the agent, for examplecd \opt\agent
. - Execute
sudo .\stop.sh
.
Install service
- Open terminal.
-
cd
into the directory where you installed the agent, for examplecd \opt\agent
. - Execute
sudo .\install.sh
Install and start service
- Open terminal.
-
cd
into the directory where you installed the agent, for examplecd \opt\agent
. - Execute
sudo .\install-and-start.sh
Change JVM memory parameters for Linux service
The Linux installer automatically install Java 1.8 using OpenJdk for a specific version of Linux. JRE and JVM memory parameters are configured in AGENT_HOME/install.sh
.
MAX_MEM_OVERRIDE_VALUE=auto
To modify JVM memory parameters:
- Stop the Agent.
- Modify
MAX_MEM_OVERRIDE_VALUE
inAGENT_HOME/install.sh
and save it. You can set it to any of the following:- The max memory in megabytes:
MAX_MEM_OVERRIDE_VALUE=8000
-
auto
which sets the RAM to 70% of total system RAM:MAX_MEM_OVERRIDE_VALUE=auto
- The max memory in megabytes:
- Execute
sudo ./install.sh
- Start the Agent.
Run the Integration Agent as a standalone app
Typically the Integration Agent runs as a Windows or Linux service. It is however possible to run as a standalone application.
Windows
- Open elevated Command Prompt (Admin).
-
cd
into the directory where you installed the agent, for examplecd c:\integration-agent
. - Execute
integrationagent.exe standalone
.
Cross-platform (Linux, MacOS, Windows)
Make sure that the default Java version is Java 1.8. To confirm, run java -version
in the terminal. The output should be Java(TM) SE Runtime
Environment (build 1.8.0_build)
.
- Open terminal.
-
cd
into the directory where you installed the agent. - Execute
java -jar integrationagent.jar standalone
.
Change JVM memory parameters for standalone app
JVM memory parameters are configured in AGENT_HOME/appstart.properties
.
app.main.class=com.toolsverse.agent.IntegrationAgent
app.libs.dir=./lib
app.vm.options=-Xms100m -Xmx8000m
app.follow=true
You can change the maximum memory that the JVM can allocate. The default is 4000 MB, which might not be enough if you are going to process large datasets.
Here's how you can do this:
- Stop the Agent
- Modify
AGENT_HOME/appstart.properties
and save it. - Start the Agent
Update the Integration Agent
Automatically Update Agent 6.3.4-SNAPSHOT or newer
Versions of the Agent 6.3.4-SNAPSHOT and newer support automatic update-and-restart and include the update service.
Automatic restart after the update requires the UpdateService. If UpdateService is not installed or not running you will need to start the Agent manually to finish the update.
Step 1. Open Agents
window.
Step 2. Click the Agent link
Step 3. Make sure the Agent Update service is online.
1. Select the Agent
2. Click Status button
3. The Updater Status should be running
.
Step 4. If new update is available you should see the Update
button at the bottom of the screen. Click the button.
Step 5. Select version of the Agent to download and install. By default the latest version is selected. Optionally check the Release Notes. Click Update
button.
Step 6. The update file will be downloaded to the AGENT_HOME/update
folder, the Agent will shutdown, install the update and restart.
Step 7. Wait until the update service updates the Agent, and status changes to Online
Step 8. Check the version of the Agent.
Automatically Update Agents between version 5.9.0-SNAPSHOT and 6.7.8-SNAPSHOT
If the version of the Agent is between 5.9.0-SNAPHSOT and 6.7.8-SNAPSHOT it supports automatic update but does not support update and restart and does not include update service.
Step 1. Install Update Service
Step 2. Follow instruction above.
Update Agents version 5.9-number or below
If the version of the Agent is 5.9-number (for example 5.9-516520) or below it does not support automatic update and does not include the update service. Installing the latest update is a manual process.
Step 1. Download and install Update service
Step 2. Open Agents
window.
Step 3. Select the Agent and click Download
icon at the bottom of the screen. Agent can be offline
Step 4. Select version of the Agent to download. By default the latest version is selected. Optionally check the Release Notes. Select Windows
or Linux
.
Step 5. Click Agent Update
. It will download the update zip file into your local machine.
Step 6. Stop the Agent. Make sure it is not running (offline).
Step 7. Create a folder AGENT_HOME/update
if it does not exist. Copy the downloaded zip file into AGENT_HOME/update
.
Install the update by running the standalone agent app in the terminal.
Step 8. Open the terminal and cd
into the directory where you installed the agent (AGENT_HOME
).
Windows
Execute integrationagent.exe
Linux, MacOS
Execute java -jar integrationagent.jar
Install the update by manually copying files from the update.zip
Step 8. Alternatively, to updating the Agent using the standalone app, you can update by deleting all files in the AGENT_HOME/lib
and AGENT_HOME/jdbc
folders and copying all files from the archive update
folder and subfolders to the AGENT_HOME
folder. After copying files delete all files in AGENT_HOME/update
.
Step 9. Start Agent as a service or as a standalone app.
Step 10. Check the version of the Agent.
Configure the Integration Agent
After installing the Integration Agent, it is fully functional and ready to go.
Later you can configure notifications, monitoring, and other parameters.
To configure the Etlworks Agent:
Step 1. In Agents
window select previously created Integration Agent.
Step 2. Select the Parameters
tab.
Step 3. Modify parameters and click the Save
button. If the Agent is running, it will reconfigure itself at runtime. If the Agent is not running it will apply configuration changes on the next restart.
Configuration parameters
This group of parameters is used to configure the SMTP sender, which sends email notifications on specific events.
-
SMTP Host
: SMTP Hostname that should be used to send emails from the agent. -
SMTP Port
: SMTP Port that should be used to send emails from the agent. -
SMTP User
: SMTP User that should be used to send emails from the agent. -
SMTP Password
: SMTP Password that should be used to send emails from the agent. -
FROM
: the email address that will be used to send emails from. -
TO
: the comma-separated list of email addresses that should receive notifications. -
CC
: the comma-separated list of email addresses that should be added to CC. -
BCC
: the comma-separated list of email addresses that should be added to BCC. -
TLS Enabled
: switch the connection to a TLS-protected.
Notifications
Use this group of parameters to enable email notifications for specific events.
The SMTP sender must be configured.
-
On Success
: if this parameter is enabled and the SMTP sender is configured the agent will send email notification on each successful flow execution. -
On Fail
: if this parameter is enabled and the SMTP sender is configured the agent will send an email notification on each failed flow execution. -
On Shutdown
: if this parameter is enabled and the SMTP sender is configured the agent will send email notification on shutdown. -
On License Issue
: if this parameter is enabled and the SMTP sender is configured the agent will send an email notification on any issue related to the license verification. -
On Status Change
: if this parameter is enabled and the SMTP sender is configured the agent will send email notification on any status change, including reaching and clearing dangerous thresholds.
Web Server
This group of parameters is used to configure the Web Server embedded into the Integration Agent. The Web Server is used to display the Integration Agent dashboard when running the Agent on-premise.
-
Web Server Enabled
: enable or disable the embedded webserver. -
Host
: the webserver hostname or IP address. -
Port
: the webserver port number.
Monitoring
This group of parameters is used to set warning and critical thresholds (in %) for the health status of the agent.
-
Memory Utilization High Threshold (%)
: once memory utilization reaches configured in this field threshold the agent's status will be changed towarning
. The default is80%
. -
Memory Utilization Critical Threshold (%)
: once memory utilization reaches configured in this field threshold the agent's status will be changed tocritical
. The default is90%
. -
Error Ratio High Threshold (%)
: once the number of errors compared to successful executions reaches configured in this field threshold the agent's status will be changed towarning
. The default is10%
. -
Error Ratio Critical Threshold (%)
: once the number of errors compared to successful executions reaches configured in this field threshold the agent's status will be changed tocritical
. The default is25%
.
Logging
This group of parameters is used to configure how the Integration Agent uploads logs to the cloud Etlworks.
-
Upload console log files
: if this parameter is enabled the Agent uploads each Flow executing log to the cloud Etlworks. The parameter is disabled by default. The console logs can be viewed in the Flow Statistics dashboard. Maximum log file size to upload (MB)
: log files with size over a specified amount in MB will not be uploaded. Note that this configuration can be limited by outside factors, such as the file upload limit set on the network proxy side.
Miscellaneous
This group of parameters is used to configure time-outs.
-
Shutdown time out (ms)
: the shutdown time out is how long the Integration Agent waits for the running flows to finish before it initiates the shutdown sequence. Encreasing the timeout will give running flows more time to finish but will increase the agent's shutdown time. Note that prior to shutting down the Agent will attempt to cancel currently running flows. Most flows will stop almost immediately but if the flow is busy it can take minutes. The default value for this parameter is300000
ms which is 5 minutes. -
Agent Ping Delay (seconds)
: the amount of delay in seconds between consecutive calls from the Agent to Etlworks instance to get the latest configurations and report status. Decreasing the value of this parameter will make the Integration Agent more responsive but will increase the chatter between the Agent and Etlworks instance. The default value for this parameter is1
minute.
Check the version of the Agent
Step 1. Open Agents
window.
Step 2. Click the Agent link
The version of the Agent is displayed on top of the screen.
Starting from 5.9.0-SNAPSHOT
version of Agent is synchronized with the version of the app. Our automatic Agent update workflow checks the compatibility between the Agent and the main app.
Monitor the Integration Agent
Agent's status
Each running Integration Agent reports status to the host Etlworks once a minute. Use Agent Ping Delay
to change the frequency.
The status of the specific Integration Agent is displayed in the grid under the STATUS column.
Available statuses
-
Offline
: the Agent is offline (not running) -
Online
: the Agent is online (running and the scheduler is enabled). -
Online (suspended)
: the Agent is online but the scheduler is suspended, hence no Flows can be executed automatically by the Agent. It is still possible to execute Flows manually.
Agent's Dashboard
The Agent's Dashboard displays information about various statuses, scheduler, JVM, and running flows.
To open the Agent's Dashboard click status
link in the Agents
window.
The status of the Agent recorded during the latest communication session will be displayed in the popup window.
Agent logs
There are two log files:
- Agent Log: The primary log file of the Integration Agent, containing errors and warnings related to the execution of flows on the Agent.
- Updater Log: The log file for the Update Service, detailing user interactions with the service, including starting, restarting, and updating the Agent.
Email notifications
Notifications can be sent by the Agent or by the Etlworks instance.
Configure Agent to send email notifications
You can configure the following email notifications. To receive notifications the SMPT email sender must be configured.
-
On Success
: if this parameter is enabled and the SMTP sender is configured the agent will send email notification on each successful flow execution. -
On Fail
: if this parameter is enabled and the SMTP sender is configured the agent will send an email notification on each failed flow execution. -
On Shutdown
: if this parameter is enabled and the SMTP sender is configured the agent will send email notification on shutdown. -
On License Issue
: if this parameter is enabled and the SMTP sender is configured the agent will send an email notification on any issue related to the license verification. -
On Status Change
: if this parameter is enabled and the SMTP sender is configured the agent will send email notification on any status change, including reaching and clearing dangerous thresholds.
Configure Etlworks instance to send email notifications
Step 1. Make sure the SMTP sender is configured under Configuration
->Email
.
Step 2. Click Configure flow notifications
.
Step 3. Enable notifications and enter the email address(s) to receive notifications.
Suspend the Integration Agent
The Integration Agent can be suspended. In this mode, the Agent is running but no Flow can be executed automatically by the scheduler. Note that in this mode it is still possible to execute Flows manually. Use this mode to debug issues with the Agent.
To suspend the Agent:
Step 1. In Agents
window select previously created Integration Agent.
Step 2. Enable Suspended
at the bottom of the screen. Click Save
button.
Actions Queue
When the user triggers any action, such as Run Flow, Stop Flow, Stop Agent the action is added to the queue. The actions in a queue are executed in order on the next communication session between the Agent and Etlworks.
View Actions
If there are actions in a queue the View Actions
button becomes green.
Click the button the see the actions.
From there you can:
- Remove the action from the queue.
- Check the estimated time when the top action in a queue will be executed.
Comments
0 comments
Please sign in to leave a comment.