Articles in this section

Integration Agent Security and Network FAQ

  • Updated
Follow

This article consolidates common security, networking, cryptography, logging, and operational questions about the Etlworks Integration Agent.

For a general overview of the Agent architecture, see:

About the Integration Agent

1. Agent Initiation and Control Model

Is the Agent outbound only?

Yes.

The Integration Agent always initiates the communication session to the Etlworks host over HTTPS. There is no inbound control channel from the internet into the customer network.

The Etlworks host cannot directly access an Agent running behind a firewall.

Normal operation requires outbound HTTPS only.

How does job dispatch and control work?

The Agent uses a queue-based control model.

User actions such as:

  • Run Flow 
  • Stop Flow 
  • Stop Agent

are queued on the Etlworks host. The Agent retrieves and executes these actions during its next outbound communication session.

By default, the Agent polls the Etlworks instance once per minute. This is configurable via the Agent Ping Delay parameter.

High-Level Sequence Diagram

  • Agent starts inside customer network 
  • Agent initiates outbound HTTPS call to Etlworks host 
  • Agent authenticates and reports status 
  • Host responds with: 
    • Pending actions (if any) 
    • No work 
  • Agent executes work locally 
  • Agent reports execution status on next poll

No inbound firewall rule is required for this control plane.

2. Update Mechanism and Change Control

Is update manual or automatic?

Newer Agent versions support automatic update and restart using the bundled UpdateService.

Update process:

  1. Update package is downloaded to: AGENT_HOME/update 
  2. Agent shuts down 
  3. Update is installed 
  4. Agent restarts

Older versions require manual update.

Compatibility and Version Control

Starting from version 5.9.0-SNAPSHOT, Agent versions are synchronized with the main Etlworks application version.

The automatic update workflow checks compatibility between the Agent and the main application before applying updates.

Signing and Integrity Verification

  • The installer is signed with a code-signing certificate. 
  • Individual JAR files are not signed. 
  • Automatic updates verify integrity using hashes. 
  • Manual update workflows do not perform hash verification automatically.

Customers can align update timing with their internal change window process and maintain rollback copies of previous versions.

3. Execution Boundaries and Least Privilege

Can the Agent run under a least-privilege service account?

Yes.

The Agent can run:

  • As a Windows service (default service name: IntegrationAgent) 
  • As a Linux service 
  • As a standalone process
  • As a Docket container

It can run under a service account defined by the customer.

Required permissions are limited to:

  • Log on as a service 
  • Read/write access to Agent directories

What local paths does the Agent use?

All operational files remain inside AGENT_HOME, for example:

  • AGENT_HOME/update 
  • AGENT_HOME/license 
  • AGENT_HOME/data 
  • AGENT_HOME/logs

The Agent does not require elevated administrative privileges beyond standard service execution and file access to its own directories.

4. Outbound Endpoints and Ports

What is always required?

The Agent must be able to reach the Etlworks host over HTTPS (port 443).

This is required for:

  • Status reporting 
  • Work retrieval 
  • Control signaling

What else may be required?

Any flow executed on the Agent may initiate additional outbound connections to:

  • Databases 
  • SaaS APIs 
  • Object storage 
  • Message queues 
  • Other configured endpoints

Egress allowlists must include the destinations used by specific flows.

Are IPs static?

  • The Etlworks host IP is static. 
  • Cloud-hosted endpoints for third-party systems may use dynamic IPs. 
  • We recommend allowlisting by DNS hostname where possible.

JSON Example Allowlist (Control Plane Only)

{
  "egress": [
    {
      "protocol": "HTTPS",
      "port": 443,
      "destination": "your-etlworks-host.example.com"
    }
  ]
}

Flow-specific endpoints must be added separately.

5. Cryptography and TLS

What TLS versions are supported?

  • Minimum TLS 1.2 required 
  • Communication is encrypted over HTTPS 
  • HSTS is enabled for the web application

The Agent communicates with the Etlworks host using secure HTTPS.

There are no special cipher suite requirements beyond standard TLS 1.2+ support.

Certificate pinning is not required.

Certificate rotation follows standard PKI lifecycle practices of the hosting environment.

6. Data Handling and Temporary Storage

Is data staged in the cloud?

No.

When using the Integration Agent:

  • Data movement happens directly between the source and destination. 
  • The Etlworks cloud host does not stage customer data. 
  • The control plane is separate from the data plane.
  •  

For example:

On-prem source → Cloud destination

Data flows directly to the destination, not through the Etlworks host.

Where are temporary files stored?

Any temporary working data generated during execution resides:

  • On the Agent machine 
  • Inside AGENT_HOME directories

On the Etlworks host:

  • Customer data is not persisted by default 
  • Temporary execution artifacts are purged after execution

How are partial failures handled?

Flow retry behavior is controlled at the schedule level, including:

  • Number of retries 
  • Restart behavior

The Agent itself:

  • Uses configurable shutdown timeout 
  • Polls at a configurable interval (Agent Ping Delay)

Queued work will be retried according to flow configuration.

7. Firewall Requirements and NAT

Are inbound firewall rules required?

No.

Normal Agent operation requires only outbound HTTPS from the Agent to the Etlworks host.

There is no need to open inbound firewall rules from the internet into the customer environment.

NAT Traversal

The Agent initiates outbound HTTPS sessions, so it works behind NAT without special configuration.

Optional Embedded Web Server

The Agent can optionally run a local embedded web server for a dashboard.

  • Configurable host and port 
  • Disabled by default
  •  Intended for internal access only 
  • Does not require opening inbound public firewall rules

8. Logging, Auditing, and SIEM Integration

What logs are available on the Agent?

Per-flow logs:

AGENT_HOME/data/logs//.log

System logs:

AGENT_HOME/logs

Metrics:

AGENT_HOME/data/metrics

Metrics include:

  • Rows processed 
  • Start and end timestamps 
  • Additional execution metadata

Retention:

  • Per-flow logs are kept indefinitely. 
  • System logs are rotated daily. Old logs are retained indefinitely. 
  • Metrics are stored per flow.

What audit logs are available in the cloud?

The Agent itself does not have a UI.

Actions such as:

  • Manual flow runs 
  • Scheduled runs 
  • Stop requests

are initiated from the cloud instance or by scheduler and recorded in the audit-trail database in the cloud.

Cloud users can search and review audit-trail records.

SIEM Integration

Agent logs are file-based and can be:

  • Collected by standard log shipping tools 
  • Forwarded to SIEM via syslog agents 
  • Integrated using existing enterprise monitoring solutions

Audit data in the cloud can be accessed through platform capabilities for reporting and review.

Summary

The Integration Agent is designed for:

  • Outbound-only control communication 
  • Least-privilege service execution 
  • No inbound firewall requirements 
  • No cloud staging of customer data 
  • Encrypted TLS 1.2+ communication 
  • Full local logging with optional SIEM integration

This architecture allows customers to securely execute integrations behind their firewall while maintaining centralized orchestration and monitoring in Etlworks.

Related articles