Continuous Delivery Model
Etlworks uses a continuous delivery model. With this approach, bug fixes, new features, and enhancements are released as soon as they are ready.
Updates are installed on an as-needed basis.
Numbered releases are automatically deployed to individual Etlworks instances on a rolling schedule, ensuring that all users have access to the latest improvements without delays.
What's New?
Version: 9.6.4
Databricks lands as a first-class warehouse. EDI grows up: AS2 send / receive, a schema-driven X12 message editor, and dedicated EDI / HL7 / X12 / FHIR format references. Simba gains conversational mapping that works in every editor. Composer and nested flows get post-step flow control. Etlworks itself becomes an MCP server.
New Functionality
Native Databricks support
A new Working with Databricks section ships with dedicated flow types optimized for Delta — bulk load through COPY INTO, streaming CDC, and message-queue ingestion. Native MERGE strategies (DELETE/INSERT, Native MERGE, Native MERGE with separate DELETE), TEXT-only staging temp tables for schema-drift resilience, Unity Catalog three-part naming, and both Personal Access Token and OAuth Service Principal authentication.
AS2 send and receive
Full AS2 (Applicability Statement 2) support for B2B EDI exchange. Signing with SHA-256 (default), SHA-1, SHA-384, SHA-512, MD5; encryption with 3DES, AES-128-CBC, AES-256-CBC; synchronous Message Disposition Notifications (MDN). Available as a dedicated AS2 connector for outbound traffic and an AS2 listener for inbound traffic, both with full certificate handling (inline PEM, file path, PKCS12 / JKS keystore).
AS2 send and receive in Working with X12 →
X12 Message Designer
A new schema-driven editor for building X12 messages from any source. Pick the segments you need, mark loops as repeating, and bind each leaf element to a source field or a lookup query. UI-driven lookups are the mechanism for populating repeating loops and segments — one source row drives many segments via a DataSet lookup. The Message Designer is now the primary path for X12 creation; nested mapping remains available as the secondary path.
UI-driven lookups in the mapping editor
A new Lookup Builder generates lookup expressions without writing JavaScript. Two result kinds: Single value (a VLOOKUP equivalent — one column from a matching row) and DataSet (a result set used to populate nested-mapping loops and X12 repeating segments). Works in all three mapping editors: flat, nested, and the X12 Message Designer.
AI-assisted mapping through Simba chat
Conversational mapping with Simba — works in every mapping editor (flat, nested, X12). Open Simba chat from the mapping editor, describe what you want in plain English ("suggest a mapping that excludes audit columns", "build a nested mapping with orders as an array"), iterate conversationally, and ask Simba to "apply this mapping" when you're happy. Simba writes the proposal into the editor as a draft; you review and save. The existing one-shot Create Mapping button continues to work for flat-only AI suggestions.
Flow control in nested flows and Composer
Four post-step control actions for nested-flow steps and Composer: continue (default), Go to step (jump to a specific step), Dynamic goto (jump to a step name computed by JavaScript / Python), and End (stop the remaining steps). Each action is guarded by an optional condition. Complements the existing Conditional Execution (whether a step runs) and Looping (iteration inside a step) capabilities.
Etlworks as an MCP server
Etlworks now exposes its agent tools through a Model Context Protocol server at POST /rest/v1/ai-agent/mcp. External AI clients — Claude Desktop, Claude Code, Cursor, Windsurf, and any other MCP-aware tool — can connect to your Etlworks instance and drive it through natural-language interactions. JSON-RPC 2.0 over HTTP, bearer-token authentication using the same Etlworks API key as the REST AI Agent API, same tool catalog Simba uses internally.
Flow Findings dashboard
A new Insights dashboard aggregates automatic flow-inspection reports across the instance. Whenever a flow runs, the engine writes an inspection report capturing observed issues — schema drift, performance anomalies, recurrent errors, data quality concerns. The dashboard lists every flow with reports in the configured retention window (1–36 months), shows severity (ALL_GOOD / INFO / MINOR / MEDIUM / MAJOR / CRITICAL) and issue count, and lets you drill into per-issue details with description, root-cause analysis, and suggested remediation. Filters by severity, flow name / ID, tenant, and report age.
Automatic incident tracking in Resource Usage
The Resource Usage dashboard now opens an incident when a tracked resource (CPU, RAM, or disk) breaches a configured threshold and stays above it for a configurable sustain duration. Incidents close automatically when the resource drops back below the threshold for a configurable settle period. Each incident captures the trigger value, sustain time, sample count, and the flows running on the affected node during the incident. Persisted to Postgres and queryable via the API.
Improvements
EDI — X12, EDIFACT, NCPDP, VDA, HL7
Comprehensive EDI documentation refresh: a new Get started with EDI documents umbrella, the dialect-specific Working with X12 / HL7 / EDIFACT / NCPDP / VDA articles, full format references for each (X12, HL7 2.x, HL7 FHIR, EDIFACT / NCPDP / VDA), and a parallel HL7 scripting guide that covers both HAPI HL7v2 (2.x) and HAPI FHIR.
Connections — separate destination for staging database
When loading data into a database via a staging table, a new option lets the flow create a separate database connection for the staging destination — useful for tenants who isolate staging credentials, route staging through a different network, or want to monitor staging-vs-final connection counts independently.
HTTP listener — CORS / OPTIONS preflight
HTTP listeners now process inbound HTTP OPTIONS requests for CORS preflight, so browser-side clients can POST to a listener endpoint without the preflight being rejected.
Engine — MongoDB CDC
Improved logging in the MongoDB CDC connector during nested transforms makes debugging schema-drift scenarios easier. Batch-mode delete handling and CDC-to-files header behavior were also improved.
Engine — streaming flows now resolve variables
Token / variable replacement now works in streaming flow contexts — the same {token} substitution available in batch flows is now applied to streaming source connections, destinations, and parameters.
Engine — AS400 / DB2 metadata fallback
When the SYSIBM.SQLCOLUMNS catalog view isn't accessible (some hardened AS400 / DB2 installations restrict it), the metadata reader now falls back to alternative catalog queries so flow design and metadata discovery still work.
Fixes
- Fixed "Dimensions to convert to string" skipping rows in some flow configurations.
- Fixed password reset double-URL-decode: the password-reset endpoint previously decoded the reset token twice, causing failures when the token contained URL-encoded characters.
- Fixed app.data cleanup task on Windows.
- Fixed an EDI metadata download edge case.
- Addressed UI regressions across several screens (login, mapping editor, code editor, object multiselect, child-flow-parameters dirty-state, file-name handling when sending emails as attachments, schedule edits in the audit trail).
Version: 9.5.8
Simba and Composer ship. The mapping editor and home screen are rebuilt, billing moves in-app, and Insights gains two new dashboards.
New Functionality
Simba — built-in AI agent
Etlworks now ships with Simba, a production-ready AI agent embedded in the application. Simba can browse and search templates, create connections (including HTTP/REST APIs built from public vendor docs), assist with mapping, drive Composer, and run CLI commands — with explicit approval required for state-changing actions. A public Simba is also available on etlworks.com for product questions.
Composer — natural-language flow builder
Composer is a new canvas-based way to build and edit flows. Drop connections, wire them, and Composer infers the flow type. Build manually with drag-and-drop, ask Simba to build for you, or start from a template. Composer is the default flow-creation path, and any existing flow can be converted to a Composer flow.
Redesigned mapping editor
The mapping editor has been rebuilt with a dual-panel layout, a Flat / Nested view-mode toggle, and an explicit binding model (Direct, Expression, Constant). The biggest improvement is on the nested side: nested mapping is now the unified mechanism for flat → nested, nested → flat, and nested → nested transformations, and it handles much more complex hierarchies than before. Under the hood, the nested transform engine was overhauled with full flat normalization — intermediate container levels that are not referenced in the destination are no longer emitted as separate intermediate records, so deep and complex nested mappings run faster and produce fewer redundant rows.
Working with the Mapping Editor →
Integrated billing
Subscription management, plan changes, and billing events are now surfaced directly in the app. Per-environment configuration and overage handling are supported, with in-app notifications for billing events from the license server.
Home screen and widgets
The home screen has been rebuilt around a customizable widget grid. Quick entry points to create a flow (Simba, blank canvas, template), live operational metrics (Stats, AI Stats, Running Flow Tasks, Resource Usage, Audit), and a list of your most recently touched flows are all reorderable and persisted per user.
Resource utilization dashboard
A new Insights dashboard tracks system-level resources across the cluster — CPU, memory, disk, JVM threads, and file descriptors — with per-node summaries, time-series charts, and top flows by estimated resource consumption. Threshold breaches fire a resource-usage webhook for alerting.
AI usage dashboard
A new Insights dashboard tracks Simba activity for the current month — token consumption, conversations, cost, wallet balance, monthly cap, and a per-tenant breakdown. The dashboard also surfaces the knowledge-base sync status and lets admins trigger a reindex.
Developer documentation hub
A new subsite at etlworks.com/dev/ brings everything developers need into one place — REST API, AI Agent API, CLI, and scripting — with quickstarts, code samples, and downloadable client libraries.
Improvements
Performance — flow and schedule loading
Flow loading and schedule loading were optimized for tenants with very large flow inventories. CPU and Postgres utilization on the application server were also reduced.
Engine — MongoDB CDC
MongoDB CDC now uses fully-qualified names; CDC-to-files header handling was fixed; batch-mode delete handling was improved.
Engine — ordered MongoDB collection snapshots
MongoDB CDC can now take initial snapshots of collections in a specific order. Useful when the destination needs inter-collection dependencies respected during the initial load — for example, parent collections before child collections.
Engine — case-insensitive file matching
Wildcard file matching is now case-insensitive everywhere in the engine for consistent behavior across operating systems and cloud storage.
Conditional column-name sanitization
The default removal of "." in column names is now conditional and configurable per flow. This preserves dotted names where they are meaningful (for example, nested-field flattening) and applies sanitization only where downstream systems require it.
Retry visibility
A "retrying" pill is now shown on flows that are mid-retry, making in-progress retry state visible from the flow list and dashboards.
Performance — global search and lookups
Global search was instrumented and optimized. Lookup-parameter resolution is faster, particularly on flows with many lookups.
Webhooks support environment variables
Webhook payloads and configuration can now reference environment variables, enabling per-environment routing and secrets without hard-coding values.
Excel and PDF as staging sources
File-to-staging flows now accept Excel and PDF files as inputs. Excel connectors can read all tabs in a workbook in a single pass.
Excel — all worksheets at once
The Excel format now supports reading every worksheet in a workbook in a single pass using wildcard or comma-separated selectors.
Fixes
- Schedule edits are now reliably tracked and reflected in the audit trail.
- Resolved rendering and input regressions in the in-app code editor.
- Fixed object multiselect.
- Fixed file-name handling when sending emails as attachments.
- Fixed child-flow-parameters dirty-state regression.
Version: 9.3.2
This release introduces major improvements in tenant configuration, execution control, asset management, and system visibility. It adds more flexibility for multi-tenant environments, safer and more scalable execution patterns, and new tools to monitor and optimize workloads.
New Functionality
Tenant configuration
Added tenant-level overrides for selected global settings, with automatic fallback to inherited platform defaults.
User profile management
Refactored the user editor to a grouped tenant-style layout, added Security settings with user-level API key authentication override, and added Gravatar support with global, tenant, and user controls.
Flows and execution management
Improved scheduler concurrency handling, allowing different schedules for the same flow to run in parallel.
Parallel execution via API and CLI
Asset management
Introduced a Recycle Bin for supported artifacts, enabling recovery of deleted items.
Bulk editing improvements
Expanded bulk editing capabilities for mappings and transformations to speed up large-scale changes.
Insights and reporting
Refactored the Statistics area into the new Insights experience and added Executions Heatmap and Scheduled Activity dashboards.
Audit Trail improvements
Enhanced Audit Trail with improved navigation and diagnostics by including direct access to related objects.
Version: 9.2.1
This release focuses on usability, navigation, and bulk operations across the platform. It introduces a unified account management experience, global search across configuration screens, bulk operations in mappings and transformations, improved Integration Agent updates, and enhanced navigation with support for opening items in new tabs.
New Functionality
Refactored account management
Completely redesigned account management with all settings available from a single menu in the top-right corner.
Users can now easily access:
- Profile and preferences
- Tenant switching
- Billing
- Users and system settings
Global search across configuration screens
Added search capability across all major configuration screens to quickly find properties and settings.
Works in:
- System Settings
- Flow configuration
- Connection/Listener/Format configuration
- Schedule
- Webhook configuration
Bulk operations in mappings and transformations
Introduced bulk selection and actions for working with large mappings and transformation lists.
Includes:
- Filter fields and transformations
- Select multiple items
- Bulk delete
- Bulk rename / replace using patterns
Improved bulk agent updates
Refactored bulk agent update workflow with better visibility and reporting.
Enhancements include:
- Improved update dialog
- Validation results before execution
- CSV export for eligibility and results
- Persistent update request reports
- Configurable retention for update history
- Enhanced updater status visibility
Open in new tab for all context menus
Added support for opening items in a new browser tab from all right-click context menus.
This improves workflows where users:
- Compare multiple Flows, Connections, or settings
- Work across multiple configurations at the same time
- Avoid losing context when navigating
Version: 9.1.3
This release introduces a new drag-and-drop Workflow Editor, major bulk management capabilities across the platform, and a redesigned in-app notifications center. It also improves Integration Agent lifecycle management with bulk agent updates, enhances CDC and ETL capabilities with dependency-aware SQL execution, and introduces several connector improvements.
New Functionality
Drag-and-Drop Workflow Editor
Introduced a new visual Workflow Editor that allows designing and editing workflows using drag-and-drop.
The editor makes it easier to build and maintain complex multi-step workflows and nested pipelines.
Bulk Agent Update
Added support for bulk updating Integration Agents.
Administrators can now select multiple agents, validate eligibility, and queue updates across all eligible agents simultaneously.
Bulk Operations Across Platform Objects
Added bulk actions for managing large numbers of objects.
Bulk operations are now available for:
- Flows
- Schedules
- Connections
- Integration Agents
These operations allow administrators to quickly perform actions such as running, updating, tagging, or deleting multiple objects.
Card View for Flows, Schedules, and Agents
Introduced Card View for several administrative pages:
- Flows
- Schedules
- Integration Agents
Card View provides a visual layout with quick actions and status indicators. Integration Agents now use Card View by default, while Table View remains available for compact management.
New In-App Notifications Center
Introduced a redesigned Notifications Center that consolidates:
- product updates
- release notifications
- operational messages
- announcements from Etlworks
Notifications are delivered in real time and can be reviewed from the toolbar notification panel.
Enhanced Custom SQL for CDC and ETL
Improved custom SQL execution for CDC and ETL pipelines.
Enhancements include:
- user-defined bind variables
- SQL blocks scoped to specific CDC tables
- dependency-aware execution ordering for CDC pipelines
These improvements allow building more advanced ETL and CDC processing logic.
New Box Connector
Added a new Box connector based on the official Box API.
The previous FTPS-based Box connector is now deprecated but remains available for backward compatibility.
JWT Assertion Authentication for HTTP Connector
The HTTP connector now supports JWT Assertion Authentication, enabling secure server-to-server integrations required by many modern APIs.
CLI Enhancement
Added a new CLI modifier: --silent-if-empty
This option suppresses output when commands return empty results, making CLI automation cleaner and easier to integrate with scripts and CI/CD pipelines.
Other Changes
Faster UI Rendering
Improved UI performance when working with environments containing thousands of objects (Flows, Connections, Schedules, Agents), reducing rendering time and improving responsiveness.
Windows Installer Update
The Windows installer now downloads OpenJDK (Zulu) instead of Oracle Java, simplifying installation and aligning with modern Java distribution practices.
Important Bug Fixes
MongoDB CDC Connector
Fixed issues with the MongoDB CDC connector related to complex connection strings, improving compatibility with advanced configurations such as replica sets and extended parameters.
Version: 8.9.10
This release enhances search and workflow usability. It introduces AI-augmented search across Home, Flow Gallery, and the Workflow Editor, adds full-text search inside complex nested workflows, and improves step-level productivity with context menus and draft step support. It also expands flow creation options and adds MSI authentication support for SQL Server and Azure Synapse Analytics in Azure Arc-enabled on-prem environments.
New Functionality
-
AI-Augmented Search Across the Platform
Integrated AI-augmented search across the Home screen, Flow Gallery, and Workflow Editor.
Users can now switch naturally between traditional keyword search and AI-assisted search to discover flows, templates, and workflow steps more efficiently. The system always starts with keyword search and intelligently augments results with AI when appropriate, improving discovery without changing the existing search workflow.
-
Full-Text Search Inside Workflows
Added full-text search inside nested Flows (Workflows).
In addition to name-based filtering, you can now perform deep full-text searches across complex multi-step workflows. This makes it significantly easier to locate specific transformations, SQL statements, parameters, or configuration details within large workflows.
Requested by users: Keyword Search within Assets in Nested Flow
-
Context Popup Menu in Workflow Editor
Introduced a context popup menu in the Workflow Editor for quick access to step-level actions.
You can now right-click or open the context menu on a step to quickly access actions such as Parameters, Enable/Disable, Delete, and other step-specific operations. This reduces navigation overhead and speeds up workflow editing.
-
Draft Steps in Workflow Editor
Added support for Draft steps in the Workflow Editor.
New steps can now be inserted and configured in draft mode before being fully created or committed. This allows users to design and adjust workflow structure incrementally without immediately impacting execution logic.
-
Expanded Flow Creation Options
Added more flow creation options from the main Flows screen.
In addition to creating standard flows, users can now quickly create Workflows (nested Flows) or start from preconfigured templates directly from the dropdown menu. This improves discoverability and reduces the number of steps required to begin building complex workflows.
Other Changes
-
MSI Authentication for SQL Server and Azure Synapse (On-Prem with Azure Arc)
The SQL Server and Azure Synapse Analytics connectors now support Managed Service Identity (MSI) authentication when running in an on-prem environment integrated with Azure Arc Resource Bridge.
This enables secure, passwordless authentication using Azure-managed identities in hybrid cloud deployments, improving security posture and simplifying credential management.
Version: 8.9.2
This release introduces a redesigned Workflow editor (formerly Nested Flow editor), significant CDC improvements for AS400, enhanced database compatibility, and new automation capabilities in the CLI and Microsoft connectors. The focus of this release is improved workflow usability, stronger cross-database consistency, and expanded enterprise authentication options.
New Functionality
-
Workflow Editor (Redesigned Nested Flow Editor)
The Nested Flow editor has been completely redesigned and is now called the Workflow editor.
The new editor provides clearer navigation, improved visual structure, and smoother interaction when building and managing complex multi-step workflows. It simplifies working with deeply nested workflows and enhances overall usability when orchestrating conditional, parallel, and loop-based execution.
-
Shortcut to Create Workflows from Flows Screen
You can now create a Workflow (nested Flow) directly from the main Flows screen using a dedicated shortcut.
This makes it faster to start building multi-step workflows without first opening an existing Flow. It reduces friction when designing orchestration logic and encourages structured workflow development.
-
Service Principal (App-Only) Authentication for SharePoint and OneDrive
Added support for Service Principal (App-Only) authentication for SharePoint and OneDrive for Business connectors.
This enables secure, non-interactive authentication using tenant ID, client ID, and client secret. It is especially useful for enterprise environments, scheduled workloads, and scenarios where user-based OAuth authentication is not appropriate.
-
CLI improvements
-
Added a new --stop-on-error modifier for CLI for-each loops.
By default, for-each continues execution even if an iteration fails. When --stop-on-error is specified, the loop stops immediately on the first failed iteration.
This gives you more control over execution behavior when building automation scripts, CI/CD pipelines, and orchestration logic.
-
CLI API now supports passing parameters in request.
-
-
AS400 CDC Connector Improvements
Significant improvements to the AS400 CDC connector, including:
- Support for ancient dates such as 0001-01-01, fixes for mixed date format edge cases
- Enhanced journal metadata capture
- Automatic deduplication
These updates improve stability, accuracy, and compatibility when working with AS400 journal-based change data capture.
Other Changes
-
Improved Date, Time, and Timestamp Handling with Batch Bind Variables
Improved handling of date, time, and timestamp values when using bind variables with batch processing.
This ensures consistent behavior across databases and connectors, reduces edge-case conversion issues, and improves reliability when processing large datasets in batch mode.
-
Improved Handling of Mixed-Case Column Names
Enhanced handling of mixed-case column names when working with case-sensitive databases.
This reduces mapping errors, prevents unexpected column mismatches, and improves compatibility across heterogeneous database environments.
- Internal stability and reliability improvements across connectors and workflow execution engine.
Version: 8.8.1
This release introduces major usability improvements for working with complex Flows and large environments. The Nested Flow editor has been significantly enhanced with in-place navigation, a searchable structure tree, and inline child Flow creation. Flow browsing has also been improved with new Flat and Hierarchical views. In addition, SuperAdmins can now configure a default tenant at login, and HTTP pagination handling has been refined.
New Functionality
-
Flow Views: Flat and Hierarchical
Introduced two complementary ways to browse and manage Flows:
- Flat view is optimized for filtering, searching, and bulk operations.
- Hierarchical view displays nested Flows inline as an expandable tree, making execution structure and Flow relationships easier to understand.
Requested by users: FLOWs visual based on tree hierarchy
-
Enhanced Nested Flow Editor Navigation
You can now open, create and edit any parent or child Flow directly from the Nested Flow editor without leaving the current screen. This makes it much easier to explore, edit, and maintain deeply nested workflows while preserving context.
-
Flow Structure Tree in Nested Flow Editor
A full Flow Structure tree is now available inside the Nested Flow editor. The tree is searchable, expandable, and supports deep nesting, providing a clear view of execution order and relationships between Flows.
-
Inline Creation of Child Flows in Nested Workflows
You can now create new child Flows directly from the Nested Flow editor and continue editing the parent workflow without switching screens. This streamlines the process of building large, modular pipelines.
-
Improved Flow Selector Across the Application
Flow selector dialogs have been updated throughout the application. Wherever you select a Flow (for example in Nested Flows, Schedules, or Agent configuration), you can now view Flow structure and navigate to related Flows before making a selection.
-
Default Tenant at Login for SuperAdmins
SuperAdmins can now set a default tenant. When configured, the user is automatically redirected to that tenant after login, reducing friction and helping avoid working in the wrong tenant context.
Requested by users: Default tenant at login
UI/UX Improvements
-
Collapsed and Expanded Flow Header
The Flow Editor header can be now displayed in either a collapsed or expanded mode, depending on how much context you want visible while working on a Flow.
Other Changes
-
Improved HTTP Connector Pagination
Enhanced automatic pagination support in the HTTP connector, including better handling of next-link–based pagination and related edge cases.
Important Bug Fixes
- Fixed an issue where column mappings were not applied correctly when a Filter or Validation transformation rejected the first record in a dataset. In this scenario, renamed fields were not propagated to the generated SQL, which could cause mismatches with destination tables.
Version: 8.7.9
This release focuses on scheduler enhancements, improved notification control, and expanded file handling capabilities, driven largely by direct user feedback. The update improves operational visibility, reduces notification noise, and adds long-requested flexibility to file processing and change replication workflows.
New Functionality
-
HIGH_WATERMARK token support in wildcard transformations
The HIGH_WATERMARK token can now be used directly in custom source queries when working with wildcard transformations, enabling more flexible incremental data extraction patterns.
Requested by users: Support HWM tokens in wildcard transformations
-
Soft delete and permanent delete support for SharePoint and OneDrive for Business.
File delete behavior is now configurable. By default, files are moved to the recycle bin (soft delete). When enabled, files can be permanently deleted.
Read more (OneDrive for Business) →
Requested by users: Add support for recycle instead of delete with SharePoint connector
-
Enhanced scheduler email notifications with custom message and JavaScript hook
Schedules can now include a custom message appended to standard email notifications. A JavaScript hook allows dynamic message generation, access to handled exceptions, and conditional suppression of emails.
Requested by users: Add message and script to scheduler email
-
Account-level controls for schedule email notifications
Administrators can now enforce email-on-failure for all new schedules and define default notification recipient lists at the account level.
Requested by users: Add account level setting to make email on failure required for all schedules
-
Exclude selected schedules from in-app execution notifications
Individual schedules can now suppress in-app execution popups, reducing noise for high-frequency or continuously running tasks without affecting email or webhook notifications.
Requested by users: Excluding schedules or flows from the execution notification
-
Repeat expected execution time notifications
When enabled, the scheduler continues sending notifications every N minutes while a flow exceeds its expected execution time. This helps distinguish long-running jobs from stalled executions.
Requested by users: Repeat long running schedule notifications
UI/UX Improvements
-
Improved sorting for Integration Agent flows
Flows associated with Integration Agents can now be sorted by all visible columns in the grid, providing consistent behavior with other schedule and flow views.
Requested by users: Order of flows in agent\schedule
Other Changes
-
Nullable field handling extended to source database connections
The “All fields are nullable” option can now be applied when reading data from source database connections. Previously, this behavior was limited to destination connections during table creation or schema changes.
Requested by users: Allow all fields are nullable options to apply on source connections as well
Version: 8.7.1
This release focuses on improvements that have been frequently requested by Etlworks users and are aimed at making day-to-day work faster, simpler, and more predictable.
Key enhancements include expanded support for file archiving across all file-based connectors, productivity improvements in Explorer through context-sensitive right-click actions and parameterized connections, and better operational visibility with filtering and CSV export of flow and file metrics. The update also includes general stability fixes.
New Functionality
-
All file-based connectors now support file archiving (ZIP and GZIP) for both ETL operations and file copy workflows.
This functionality is available across all supported file and cloud storage connections.
Read more →
Requested by users: Add compression to copy files flow
-
Added a context-sensitive right-click menu in Explorer for faster access to common actions, depending on the selected object.
-
Explorer now supports configurable parameters that automatically replace {tokens} in connections when exploring metadata, viewing data, and running SQL queries.
Requested by users: Allow setting flow parameter in explorer.
-
It is now possible to filter Flow Metrics and File Metrics and export metrics to CSV from the Flow Dashboard.
Requested by users: Allow for export of Record Metrics from statistics page.
Other Changes
- Replaced the Snowflake JDBC driver with the latest version, which includes important bug fixes and improvements.
Bug Fixes
- Various bug fixes and stability improvements across the platform.
Version: 8.6.5
This release introduces one highly requested enhancement to the tagging system: centralized tag editing. Administrators can now rename and delete existing tags directly from the UI or using CLI commands, either per tenant or globally.
New Functionality
-
Tag Management (Rename and Delete Tags) – added support for managing existing tags directly from the Tags input.
Administrators can now rename or delete tags in one place, with changes applied consistently across all resources where those tags are used. Super administrators can optionally apply tag changes globally across all tenants.
The feature includes filtering, configurable matching rules, and confirmation safeguards for destructive operations.
Requested by users: The ability to manage tags: edit, delete
-
Tag management CLI commands - added new CLI commands for renaming and deleting tags.
Version: 8.6.1
This release introduces two major usability enhancements: real-time in-app release notifications and a significantly expanded tag system with support for saved Tag Views, access control, and AND-based tag filtering. The update also includes several backend reliability improvements related to numeric data handling and CDC connectors.
New Functionality
-
Real-Time Release Notifications – Etlworks now delivers real-time notifications when a new platform version becomes available.
When a new release becomes available but not deployed yet, users immediately see an Available indicator in the application without refreshing the page. Notifications are delivered via WebSocket and update the UI instantly, ensuring teams are aware of new releases as soon as they are published.
The notification includes version number, release date, and a direct link to detailed release notes.
-
Tag Views (Saved Views) – introduced a major refactoring of the tagging system with support for reusable, saved Tag Views.
Tag Views allow users to group related tags into named views and quickly filter assets such as Flows, Connections, Schedules, Agents, and Explorer items. Views can be personal or shared, enabling teams to standardize how assets are organized and discovered.
Tag Views can be managed independently and reused across multiple pages, significantly improving navigation in environments with large numbers of assets.
Requested by users: Saved Views Based on Tags
-
Tag View Access Control (Users and Roles) – added fine-grained access control for Tag Views.
Each Tag View can now be restricted by user and role, allowing organizations to control visibility of saved views across teams. This enables permission-aware organization of assets without changing underlying object permissions.
-
AND Tag Filtering – extended tag filtering logic to support AND conditions in addition to the existing OR filter.
Users can now filter resources by requiring all selected tags to be present, making it easier to narrow down assets that belong to a specific suite, environment, or ownership group.
Requested by users: Filter on specific tags with AND condition instead of OR
Other Changes
- Added support for Infinity, -Infinity, and NaN values when working with PostgreSQL double precision columns. These values are now handled correctly during inserts and updates without causing data conversion errors.
- Improved support for float32 and float64 data types in CDC connectors. Fixed a rare ClassCastException that could occur when streaming floating-point values from CDC-enabled databases under specific edge conditions.
Version: 8.5.1
This release introduces two major platform enhancements: the new Etlworks CLI for full automation and orchestration, and a unified script execution engine that supports running Python, Bash, PowerShell, SQL utilities, and other system commands both locally and over SSH. The update also adds more than 50 new SQL and JavaScript functions to the Function Builder, along with improvements to file-based and MongoDB connectors.
New Functionality
- Etlworks CLI - a powerful new command-line interface that enables full automation of the Etlworks platform. Supports hundreds of commands, secure authentication, batch operations, scripting, and API-driven orchestration.
-
Unified Script Execution Engine - added a new flow type Execute Script Local or Remote via SSH.
You can now run Python, Bash, PowerShell, Java, Node.js, SQL utilities, and any other script either:
- on the Etlworks host (local execution), or
- on a remote server via SSH.
Supports full interpreter selection, timeouts, masks, logging, and Airflow-style orchestration patterns.
-
Environment Variables- Environment variables provide a simple, secure, and reusable way to manage configuration values across your Etlworks instance. They are especially useful for credentials, connection strings, or values that change depending on the environment (for example, dev, staging, or production).
- 50+ New SQL and JavaScript Functions - added a large set of new built-in functions to the Function Builder, including string utilities, date/time helpers, math and aggregation functions, type conversion helpers, and advanced SQL functions.
Other Changes
- Improved MongoDB streaming reliability by propagating all exceptions instead of suppressing certain error types.
- Fixed handling of escaped and enclosing quotes in CSV and other delimited file formats to ensure consistent parsing.
- Corrected epoch-based datetime parsing in file-based connectors.
Version: 8.4.3
New Functionality
-
In-app Release Notifications – users are now informed of new Etlworks versions directly within the application via a popover with highlights.
-
Interactive Function Builder in Code Editor – added a new interactive panel for browsing, editing, and inserting built-in and custom functions directly into the code editor.
Requested by users:
Drag-and-drop SQL query builder and field function builder →
-
System Health tab in User Profile – provides a centralized view of system resource utilization and node health status.
Requested by users:
-
Enhanced /health endpoint – now supports multi-node environments, allowing you to verify cluster node availability and health.
-
File filtering using JavaScript – added the ability to programmatically exclude files from processing in file-based flows using JavaScript.
-
Configurable AI Chatbot Provider – Super Admin users can now choose between Etlworks and Zendesk chatbots in Settings → AI.
- Named connections for most flow types – simplifies referencing and reusing connections across complex or nested flows.
Version: 8.2.7
This release introduces major improvements to the Mapping experience, adds support for new CDC pipelines, and refines behavior around null and empty values. Several performance and documentation upgrades are also included.
New Functionality
- Revamped “Create Mapping” UI with AI‑Augmented Mapping
The Mapping Editor introduces automatic AI‑assisted field mapping, offering smarter suggestions based on field names, types, and optional prompts.
Reduces manual effort when field names differ or require renaming, exclusion, or enrichment.
Documentation → Read more
-
AI-Assisted Flow Type Discovery
You can now describe your integration scenario in plain English (e.g., “Send emails with attachments” or “Stream PostgreSQL to MongoDB”) and let Etlworks suggest the most relevant Flow types.
This feature enhances discoverability and accelerates onboarding for new and existing users.
Documentation → Read more
New CDC Flow Type: CDC → NoSQL Database
Stream CDC events directly from supported relational databases into NoSQL databases (MongoDB,, etc.).
No need for separate flows for creating files for CDC events and loading files into NoSQL db.
Documentation → Read more
- Improved Handling of null, 'null', and Empty Values
Unified behavior for all file-based formats (CSV, JSON, etc.) and CDC flows.
Users can now explicitly define how each type of “null” is treated when reading and writing.
Documentation → Read more
-
Connecting to QuickBooks Online with the Premium or API connector now requires creating a Connected App under your own Intuit Developer account.
This replaces the previously shared Etlworks app due to new billing requirements introduced by Intuit in July.
Setup takes just a few clicks.
Documentation → Read more
Other Changes
-
Improved flow startup time in high-frequency and multi-node environments by caching configurations in in-process memory (alongside Redis).
Reduces latency for flows triggered repeatedly under load.
-
Completely rewrote the Mapping documentation for clarity, including new sections on AI-Augmented Mapping, Nested Mapping, and prompt-based customization.
Documentation → Read more
-
New centralized CDC documentation covering all flow types, configuration steps, troubleshooting, and best practices.
Documentation → Read more
Version: 8.1.5
This release focuses on quality-of-life improvements and long-standing requests from end-users. Many of the new features and enhancements came directly from customer feedback, making day-to-day work with Etlworks smoother and more efficient.
New Functionality
-
Support for relative path and automatic folder creation for all file-based connectors and all ETL/File-management flow types.
Documentation →
-
Direct link to a specific flow execution.
User request: link
-
Expose schedule ID via global variable at runtime.
User request: link
UI/UX Improvements
-
Filtering audit-trail logs by time.
User request: link
-
Run flow directly from Flow Builder.
User requests: Run from error screen, Test Script Button
-
Interactive Flow Statistics dashboard. Clicking on chart slices (Success, Error, etc.) now applies a filter.
User request: link
-
Sorting by SCHEDULE and NEXT in the Schedules window.
User request: link
-
Flow Executions: sort and filter updates.
User request: link
-
Improved filter/search in Flow Statistics page.
User request: link
Other Changes
-
Switched Agents installer for Windows to OpenJDK (Zulu). Latest Oracle JDK requires a separate license agreement; OpenJDK does not.
Note: existing agent installations are not affected. Linux and Docker versions were already using OpenJDK, so only new Windows installations are impacted.
- Improved flow startup time in multi-node environments when using managed Redis (such as AWS ElastiCache). Macros cache is now stored in in-process RAM on each node instead of Redis.
- Added support for automatic data type mapping for SQL Server DATETIMEOFFSET (previously mapped to VARCHAR/TEXT).
- Improved automatic type mapping for Google BigQuery connector.
- Added AWS SSM libraries to the classpath. These are required when retrieving secrets or configuration values directly from AWS Systems Manager.
- Updated connector for Odoo ERP.
- Improved detection of query language (SQL vs. JavaScript) in the Source Query.
- Added new configuration option: Record detailed flow execution application log. When enabled, it records internal steps of flow execution (such as connection initialization and metadata preparation) in the audit trail with millisecond precision.
- Useful for diagnosing latency in real-time or high-frequency flows.
- Read documentation →
- Replaced the AI Assistant with a new version. The previous Help option (which opened a support chat) is now integrated into the AI chatbot. Users can now access both AI-powered answers and direct support escalation from a single button in the top navigation bar. Read documentation →
Important Bug Fixes
- Fixed a bug preventing proper flow execution when a source Message Queue connector used a Preprocessor with JavaScript code containing comments.
- Fixed a bug preventing Before Connect script configured for the Outbound Email connector from executing when using the Send Email flow type.
- Fixed a bug in the Snowflake connector where ARRAY source data types were incorrectly mapped to ARRAY in Snowflake (now mapped to STRING).
Version: 8.0.0
Why This Release Matters
Version 8.0.0 is more than just another update — it marks the beginning of a major new chapter for Etlworks.
This release introduces a powerful new intelligence engine that enhances how users discover, build, document, and optimize flows. While the engine is already embedded in key parts of the platform, this is just the start. Over the coming months, we’ll continue activating new capabilities across scheduling, monitoring, documentation, and flow development.
What’s already included:
- Intelligence-driven template discovery
- Automatic flow documentation
- Automatic flow inspection for performance and structural issues
What’s coming soon:
- AI-powered field mapping — Automatically match and map fields between sources and destinations with minimal manual input.
-
Advanced scheduling and execution intelligence, including:
- Interactive execution heatmaps
- Detection of underused or overused timeslots
- Automated schedule optimization suggestions
- Prompt-driven reports — Ask questions in plain language (e.g., “Show the top 5 flows executed in the last 4 hours”) and get instant answers.
- Direct prompt-to-flow generation — Describe your use case and let the platform create the flow automatically.
- Resource utilization tracking and insights — Quickly identify which flows consume the most shared resources and uncover optimization opportunities.
- And much more — Additional intelligence-driven features will continue rolling out in future releases.
It also includes long-requested UX improvements, performance boosts, and foundational changes that will support everything from better pagination handling to intelligent flow analysis — all integrated directly into the UI.
All Etlworks-managed cloud instances will be updated automatically. We strongly recommend updating on-premise installations to take advantage of the new capabilities.
New Functionality:
-
Introduced a New Home Screen
Discover and build flows faster with the new landing page. Start from recommended templates, explore connectors, or create flows by simply selecting source and destination.
-
Enabled AI-Augmented Template Discovery
AI-assisted search now helps you find the right integration template based on intent and keywords. Includes support for generic templates and user-defined custom templates.
-
Improved Flow Import and Export
You can now replace connections and formats with existing ones when importing flows. Credentials can also be excluded from export/import for improved security and compliance.
-
Generate Flow Documentation from the UI and API
Automatically generate Markdown-based documentation, including nested flow structure, conditions, loops, and mappings. Available from the UI and via the Flow Documentation API.
-
Full Pagination Support in HTTP and API-Specific Connectors
Generic HTTP connector and all prebuilt API connectors (Salesforce, HubSpot, etc.) now support all standard pagination patterns — no need for manual looping.
-
Flow Analyzer Now Integrated into the UI
Recommendations for optimizing flow structure and performance are now displayed directly in the UI. No need to parse JSON output manually.
-
MongoDB Connector Supports DELETE and Drop Collection
You can now delete documents or drop entire collections using the MongoDB connector. Use with caution.
-
Outbound Email Connectors Can Send Attachments by Default
All email connectors can now be configured to always send data as an attachment instead of in the message body.
UI/UX Improvements
-
Faster UI Rendering
We introduced a new rendering engine that dramatically improves UI performance in environments with a large number of artifacts, including flows, schedules, templates, and connections. Users will experience faster load times and smoother interaction when navigating and editing environments with hundreds or even thousands of items on screen. This update enhances the overall responsiveness and usability of the platform in high-volume scenarios.
Other Changes:
-
Improved Handling of Oracle CDC XMLType Columns
Etlworks now fully supports replication of XMLType updates using Debezium for Oracle CDC, including edge cases and large payloads.
-
Improved Automatic Data Type Mapping
All database connectors now better detect and convert source data types during ingestion.
- Updated Etlworks Docker image — Now runs as a non-root user for improved security and compliance.
-
Support for PostgreSQL arrays in Bind Variables
PostgreSQL arrays are not supported when using flows which load data using bind variables.
-
New Helper Method for Variable Promotion
Added ElConfig#setGlobalVarsFromFlowVars() to convert all flow variables into global variables (previously required scripting).
Important Bug Fixes:
- Fixed an issue where the nested flow editor did not recognize changes when reordering flows.
- MERGE CSV flow now skips output if no files matched the wildcard — no more empty files.
- Fixed JavaScript execution in wildcard-based file flows (previously only worked with single-file processing).
- Fixed HTTP connector’s 2-step auth with self-signed SSL certificates.
- Fixed Synapse connector flow failure when credentials were not provided.
Version: 7.10.1
New Functionality:
- Added Pre-Configured API connectors for Salesforce, HubSpot, QuickBooks Online, Zoho Creator, Zoho CRM, Zoho Inventory, Zoho Projects, Jira, Jira Service Management, Confluence, and Amazon Marketplace. All connectors use OAuth2 authentication for the respective service. Previously, only dedicated connectors for Etlworks, Microsoft, and Google APIs were available to simplify authentication with those services. While the generic HTTP API connector supports any API, these dedicated connectors exist to streamline OAuth2 workflows specific to each platform. With this release, we’ve added 13 new connectors to simplify authentication for popular services, without introducing new app-specific features like response parsing. Learn more
- Added flow templates for working with Connected Apps. These templates simplify building flows for reading from and writing to SaaS platforms. Learn more
- Introduced Predefined Flow Parameters and Auto Suggestions (available for Enterprise and on-premise customers only). This feature enables configurable dropdowns for commonly used flow parameters. Learn more
- Added automatic and configurable ZIP and GZIP decompression to all file and cloud storage connectors as well as HTTP API connector. This eliminates the need for manual unzipping before ingestion. Learn more
- Named connections can now be configured for all flow types, making it easier to reference reusable connection objects across flows.
Other Changes
- Improved performance of the audit log maintenance task. Previously, this task could fail when the audit trail exceeded a certain size, causing slowdowns when updating the statistics dashboard.
-
Renamed Salesforce connectors for clarity:
- Salesforce (OAuth2) is now called Salesforce
- The original Salesforce connector is now Salesforce Legacy. Learn more about Salesforce connectors.
- Updated OAuth2 scopes for the HubSpot (Premium) connector to reflect recent changes introduced by HubSpot.
Important Bug Fixes
- Fixed an issue where the nested flow editor did not recognize changes, such as enabling or disabling flows, when opened from a direct link.
- Fixed a bug with unresolved {tokens} in the “From” section of ETL transformations. This occurred when the flow was added to the nested flow and the token value was passed as a flow parameter configured in the schedule.
Version: 7.9.1
New Functionality:
- Support for Multi-Region S3 Buckets (MRAPs) — You can now work with multi-region S3 buckets using Etlworks S3 connections. Read more.
- Document Library Name in SharePoint Storage Connections — SharePoint Storage connector now support explicitly setting the document library name for better control in environments with multiple libraries. Read more.
- BOOLEAN Data Type Support in DB2 AS400 CDC Connector — The DB2 AS400 CDC connector now fully supports BOOLEAN columns without requiring manual exclusion.
-
Option to Truncate Exception Stack Trace in Schedule Notifications — You can now choose to include only the exception message (without the full stack trace) in host and agent schedule error notifications. Read more
Bulk Column Name Transformation — Added support for overriding multiple column names at once without configuring individual field mappings. Read more
Important Changes Under the Hood:
- Smarter Handling of SQL Server MONEY Columns — When creating destination tables without explicit mapping, MONEY columns are now assigned the correct precision and scale based on metadata retrieved from SQL Server.
- Expanded Support for HTTP Destinations — File management flows (copy, move, etc.) now support MS Graph, Google APIs, and Etlworks APIs when used as HTTP destinations, in addition to the previously supported generic HTTP connection.
Security Fixes:
- WSTG-INPV-02: Input Validation — Patched an OWASP vulnerability related to improper input validation. WSTG-INPV-02
- CVE-2025-30065 — Addressed a publicly reported vulnerability affecting certain HTTP inputs. Details are available in the NVD.
Important Bug Fixes:
- Fixed an issue with reading Excel files generated by third-party tools that omit cell references like A1, B2, etc.
- Fixed a bug that prevented creating flows optimized for loading into BigQuery when GZIP compression was enabled on a Google Cloud Storage connection.
- Fixed a bug that prevented creating Snowflake-optimized flows when the Empty field as null option was disabled (this option is enabled by default).
- Fixed an error that occurred when running the flow Bulk load files in Azure Storage into Synapse Analytics or Fabric Warehouse with Microsoft Fabric as the destination. (Synapse destinations were not affected.)
Version: 7.8.5
New Functionality:
- Email Connectors (Office 365, Gmail, Etlworks Email): You can now send emails using Microsoft Graph API and Gmail API instead of SMTP. This applies to both outbound email connectors and the internal Email connector used for notifications. SMTP remains supported.
- Override Data Types: It is now possible to override data types when creating or altering tables in database connections. This is useful when the default mapping doesn’t match the target schema or precision requirements. Learn more.
- Include Flow Parameters in Notifications: Added support for including parameters defined at the host schedule or agent schedule level in success and failure email notifications. Parameters are presented as an HTML table.
- Microsoft Fabric Warehouse Connector: Added a native connector for Microsoft Fabric Warehouse, fully compatible with Azure Synapse Analytics. All flows optimized for Synapse now work seamlessly with Fabric. Learn more.
- Azure Service Principal Authentication: Added support for Azure Service Principal-based authentication for SQL Server, Synapse, and Fabric connectors. This improves security and automation in enterprise environments. Learn more.
- Oracle CDC: Ignore Invalid Events: Added the ability to skip or log warnings for improperly formatted Oracle CDC events related to XMLType columns when event.processing.failure.handling.mode is set to skip or warn.
UI/UX Improvements:
- Agent-to-Flow Navigation: You can now navigate directly from the Agent dashboard to the corresponding flow. Use the browser back button to return to the agent context. Learn more.
Important Changes Under the Hood:
- Faster CDC Performance: Dramatically improved the performance of CDC flows when “Provide Transaction Metadata” is enabled by optimizing the detection of transaction boundaries. Previously, generating these markers introduced significant overhead, slowing down CDC flows. This process has now been optimized for speed and efficiency, resulting in much faster performance.
- CDC Metadata File Placement: Transaction marker files (when enabled) are now saved in the same folder as the associated data files for better organization and tracking.
- Upgraded MySQL CDC Binlog Reader: We are using the latest version of the binlog reader with improved performance and better MariaDB compatibility.
- Snowflake Driver Update: Updated to the latest Snowflake JDBC driver, with support for OpenSSL 1.1.x+ private keys and enhanced reliability.
- Improved EDI (X12) Connector: Expanded support for additional EDI segments, with updated mapping and improved error handling.
Important Bug Fixes:
- Global Variable Leakage in Agent Tests: Fixed an issue where test transformations on the Agent reused global variables from previous runs instead of applying the latest values.
- SMB Share Extra Folder Issue: Resolved an issue where FileManagerTask.read(...) was incorrectly appending extra folders to the path internally in SMB Share connections, which caused file reads to fail.
- Bind Variable in Database flows: Fixed a rare issue that caused flows to fail when SQL bind variable names—automatically base64-encoded for safety—contained the * character. In these cases, * was mistakenly interpreted by the database (e.g., SQL Server) as a wildcard rather than a literal character. This edge case has now been resolved to ensure bind variable names are always safely interpreted.
- CDC TIME Field Formatting: Corrected handling of TIME fields in CDC events. Previously, TIME fields were not properly converted to epoch long. (No change to TIMESTAMP or DATE)
Version: 7.7.0
New Functionality:
-
Two new options for Target Table Creation & Matching when the destination is a database:
- Auto-Create & Strict Schema Match, Fail on Mismatch → Creates the table if it doesn't exist. If the table exists, the flow fails if the source and destination do not exactly match in terms of column names and the number of columns. The order and data types do not affect the match, but if the source has fewer columns or mismatched column names, the flow fails.
- Strict Schema Match, Fail on Mismatch, Do Not Create → Does not create the table if it doesn't exist. If the table exists, the flow fails if the source and destination do not exactly match in terms of column names and number of columns. The order and data types do not affect the match, but if the source has fewer columns or mismatched column names, the flow fails.
New flow type: EDI to File Conversion. Enables rapid conversion of any X12 message into XML, JSON, or CSV formats without requiring an ETL source-to-destination transformation. This streamlined approach is optimized for performance and efficiently handles large X12 documents. (Learn more)
X12 Connector now automatically downloads schema files from the Etlworks CDN and caches them in {app.data}/metadata/edi. Previously, manual installation of schema files was required. (Learn more)
X12 Connector updated with a comprehensive list of segments for all supported versions and transaction sets.
HTML and PDF Connectors now correctly handle tables with identical column names. Previously, duplicated columns would result in only the last column being retained.
Kafka Connector now supports authentication using AWS IAM, specifically for Amazon MSK clusters. (Learn more)
Parquet Connector now supports legacy schema generation for decimal columns, ensuring backward compatibility with previous Parquet schemas.
CDC Connectors now allow passing unformatted dates, times, and timestamps downstream without converting to strings. (Formatting options explained)
Scheduler now includes new schedule-exclusion options: "Last week of the month," "Any week except last of the month," and "Weekdays except last of the month."
UI/UX Improvements:
- Added a table view for the "Test transformation" feature when the destination is not a database. Previously, only the raw view was available.
Nested flows now include clickable links to child flows, allowing users to easily navigate through nested flow hierarchies. (User Request) (Learn more)
Flow history now visually indicates retries triggered by the scheduler or integration agent, including a parent process ID. This helps differentiate retries from regular executions. (Learn more)
Important Changes Under the Hood:
- The Test transformation feature no longer converts columns to database-specific formats, displaying them in raw format instead. This simplifies testing calculated date, time, and timestamp field functions, removing the need to factor in conversions during tests.
X12 Format Connector is now automatically installed with Etlworks, both on cloud and on-premise deployments, and no longer requires manual license installation.
Version: 7.5.10
New Functionality:
-
Incremental File Processing: Etlworks now supports incremental file processing across multiple flow types, ensuring that only new or modified files are processed. This enhancement improves performance and eliminates redundant processing in the following scenarios:
-
ETL Flows with File Sources – When the source is a file, users can now choose from three options:
- Delete processed files: existing feature. Learn more.
- Move processed files to a configurable destination: Learn more.
- Skip already processed files: Learn more.
- File-Based Flows – In file operations (copy, move, rename, delete, zip), an option is now available to skip already processed files, preventing unnecessary operations. Learn more.
- Bulk Load Flows – In addition to the existing option to delete and move processed files to a configurable location, users can now choose to skip already processed files for Snowflake, Redshift, Synapse, BigQuery, Vertica, Greenplum, Oracle, and any JDBC database. Learn more.
- File loops - incremental processing can be enabled in file loops by using file path modifier. Learn more.
-
ETL Flows with File Sources – When the source is a file, users can now choose from three options:
-
Handling Extra Fields in Source (user request): Improved handling of unexpected fields in the source during data transformations. Learn more. Here are the available options:
- Ignore extra fields – Any columns in the source that are not explicitly mapped will be ignored and not included in the destination.
- Error on extra fields – If extra columns appear in the source that are not part of the Mapping, the flow will fail before running any transformations, allowing early detection of unexpected schema changes.
- Automatic Folder Creation for Server Storage Connections: No need to manually create folders—directories are now automatically created when writing files to a server storage connection.
- Ignore Errors on Table Creation: Added an option to ignore errors when executing CREATE TABLE statements in ETL flows where the destination is a database. This prevents failures in scenarios where multiple threads attempt to create the same table in parallel. Learn more.
- Include/Exclude Exceptions in Auto-Retry (App & Agent Schedules): Users can now include or exclude specific exceptions when configuring auto-retry in schedules. This allows retries only on specific errors, giving users more control over error handling. Learn more.
-
Enhancements for Zoho Connectors:
- Added an option to use a Zoho Connected App created under a user account instead of an Etlworks-owned app. Learn more.
- Added an option to disable metadata lookup, improving handling rate limits enforced by ZOHO. Learn more
-
Improvements for the HTML Connector:
- Automatic merging of HTML tables with identical structures.
- Ability to specify which table should be returned based on various criteria: List of columns, Most columns, Most rows. Learn more.
-
Enhanced Auto-Retry Logging for Agents:
- The agent now records each auto-retry attempt and creates an audit record.
- Auto-retry attempts generated by the agent are now propagated in webhook payloads, improving observability.
- Inbound Email Connector: Unique File Names for Attachments. When reading emails and saving attachments to a folder, the connector now ensures that files with the same name do not overwrite each other by automatically creating unique file names. Learn more.
- Configurable Column Filtering in CDC Connectors: Added the ability to include only specific columns in Change Data Capture (CDC) connectors. Previously, only excluding columns was possible—now, users can define exactly which columns to track.
Important Changes Under the Hood
- Programmatically Stopping Specific Loops: Previously, stopping a loop programmatically terminated all loops. Now, users can stop a specific loop programmatically. Learn more.
Important Bug Fixes
- Excel XLSX Connector: Fixed a bug when streaming data from Excel XLSX files where the mapping contained more columns than the source worksheet.
Version: 7.4.2
New Functionality:
- New Dynamic Connection and Format: Introduced a new Dynamic Connection and Format mechanism, allowing connections and formats to be resolved at runtime without requiring predefined named connections in nested flows. The old dynamic connection is now named “Legacy Dynamic Connection” for backward compatibility, but we recommend switching to the new approach for improved flexibility. Read more.
- New “Target Table Creation & Matching” Options: The previous checkbox “Create target table if it doesn’t exist. Also update columns to match the target” has been replaced with a dropdown menu offering multiple table-handling options. Users can now choose from various strategies such as auto-creating, failing on mismatch, matching existing structures, or using tables without modifications. Learn more.
- UTF-8 Support for FTP and FTPS Connectors: The FTP and FTPS connectors now support UTF-8 filenames, ensuring compatibility with file and folder names that contain special characters such as German umlauts (ä, ö, ü) and other non-Latin scripts. More details.
- New Lookup Method: Extractor.lookupWithCallable(): Added Extractor.lookupWithCallable(), a new lookup method that allows executing callable SQL statements for lookups. Supports retrieving either a single field value or an entire dataset using stored procedures:
- New Flow Templates for NoSQL and Email-based ETL Flows: Introduced new Flow Templates designed specifically for NoSQL databases (such as MongoDB, Redis) and email-based sources and destinations. Previously, users were advised to use file-based templates for NoSQL and email ETL flows. The new templates optimize processing for these data sources.
Important Changes Under the Hood:
- Enhanced Oracle Bulk Load Handling: Oracle bulk load flows now automatically generate SQL*Loader control files to correctly handle VARCHAR and CHAR columns longer than 255 characters.
Important Bug Fixes:
- Fixed XLSX Connector Memory Issue: Resolved an issue where reading large and wide XLSX files caused the error “Tried to allocate an array of length xxx”.
- Fixed Redshift Connector Transaction Handling: Previously, Redshift connector did not allow multi-statement transactional SQL execution. The issue has been fixed, allowing users to run transactions like:
BEGIN;
INSERT INTO test_transaction (id, name) VALUES (1, '123');
INSERT INTO test_transaction (id, name) VALUES (2, '123');
COMMIT;Version: 7.3.1
New Functionality:
- Snowflake Connector: Key-Pair Authentication: The Snowflake connector now supports key-pair authentication, providing enhanced security for your Snowflake connections. Learn more.
- Dynamic Formats: We’ve added support for Dynamic Formats, allowing you to configure formats dynamically when using Dynamic Connections. Learn more.
- Warning for Running Agent Flows on Host Instance: A new warning is displayed when attempting to run a flow assigned to an agent on the host instance. The system also provides a link to switch to the correct agent. Learn more.
- Enhanced Flow Executions Dashboard: Added new filter options, including start/end time and the number of records, for better analysis. Users can now export flow execution reports to CSV for offline review. Learn more.
- MongoDB CDC Connector: Snapshot Mode: The MongoDB CDC connector now supports a snapshot mode (initial_only), allowing users to capture the initial snapshot and stop the flow before streaming changes. Learn more.
- Improved Agent Sorting Options: Agents can now be sorted by OS, Version, IP Address, and Status, in addition to Name, providing better management and visibility of agents.
Important Changes Under the Hood:
- Connection Metadata Cache Moved to Local File Storage: Metadata cache previously stored in Redis has been moved to local file storage. Metadata files include objects (tables, views, endpoints, files) and columns associated with these objects. A new maintenance task clears old cache files. This change prevents Redis overload due to large cache files. The update also deletes existing cache files from Redis. Learn more.
Important Bug Fixes:
- MongoDB CDC Connector: DELETE Event Fix: Fixed a bug where DELETE events were not being captured when streaming from Atlas MongoDB instances. This ensures all event types are captured reliably.
New Guidelines:
- Working with Paginated APIs: Learn how to handle APIs that return paginated results efficiently. Read the guidelines.
- Use SQL to Extract Data from Non-Relational Sources: Utilize SQL queries to extract data from non-relational sources with ease. Read the guidelines.
- Configure Format to Transform Nested Source Documents: Format-level transformations is an alternative and very effective approach to handling nested documents (JSON, XML, etc.) in Etlworks. Read the guidelines.
- Installing connectors on Integration Agent: Read how to install regular and premium connectors on the machine running the Integration Agent.
- Installing connectors in self-hosted environment: Read how to install regular and premium connectors in the self-hosted environment.
Version: 7.1.9
New Functionality:
- Automatically Clean App Data Files: The new Clean App Data Files maintenance task helps manage disk usage by automatically deleting old files from the Home folder ({app.data}) and its subdirectories.
- Support for QuickBooks Online Sandbox Accounts: The QuickBooks Online connector now supports sandbox accounts, enabling testing and development without impacting live data.
Important Changes Under the Hood:
- Reduced Network Chatter Between Host and Agents (Pre-7.1.4): We have optimized network communication between the Etlworks host instance and Integration Agents for versions prior to 7.1.4. Agents running version 7.1.4 or newer are not affected.
Important Bug Fixes:
- Auto-Retry Parameters for Flows on Agent: Fixed an issue where changes to auto-retry parameters were not triggering retries for failed flows executed on Agents unless other scheduling parameters were also modified.
Version: 7.1.4
Etlworks is now available on the AWS Marketplace:
- Learn More and Get Started: Etlworks on AWS Marketplace
Improvements for Integration Agents:
- Auto-Retry for Scheduled Flows: Integration Agents now support automatic retries for scheduled flows, enhancing reliability in case of transient issues. Read about configuring auto-retry for flows scheduled on Integration Agent.
- Reupload Flow Execution Stats When Host Instance Is Back Online: Integration Agents will reupload statistics and logs for flows executed during host instance downtime. Additionally, webhook notifications will be triggered if they are configured. Read about configuring the reupload.
New Functionality:
-
Shopify and Airtable Connectors:
- Shopify connector now supports GraphQL APIs.
- Airtable connector now supports authentication with auth tokens.
-
Improvements for ZOHO Creator connector:
- Enhanced metadata retrieval.
- Full support for all data types available in Zoho Creator.
Human-Readable Field Names in X12 Connector: When converting X12 to JSON, human-readable field names are now preserved. Previously, this functionality was limited to X12 to XML conversions.
Important Changes Under the Hood:
Optimized Large File Processing for Azure Storage Connector: Significant performance improvements when handling large files in Azure Blob Storage.
Improved X12 Processing Efficiency: Memory usage for X12 processing has been optimized, improving scalability and stability.
S3 Connector Auto-Retry for Metadata Service Errors: Auto-retry now covers scenarios where the connector fails to load credentials from the EC2 instance metadata service.
- Default Schedule Update: The default schedule is now set to one hour. This addresses issues caused when users saved the default one-minute schedule without adjusting it to a more reasonable cadence.
Important Bug Fixes:
Fixed "Exclude Column(s)" Property in MongoDB CDC Connector: Resolved an issue where the property to exclude specific fields during data extraction was not functioning as intended.
Resolved Scientific Notation Issues in CDC Connectors: CDC connectors now handle DOUBLE data type columns more accurately by removing scientific notation, ensuring precise and consistent data representation.
Version: 7.0.6
New Functionality:
- Automatic Partitioning: When configuring ETL flows, Etlworks now enables the use of Partition SQL. This feature allows you to define custom partitioning conditions via an SQL query, automatically creating separate ETL transformations for each partition generated. These transformations can then execute in parallel, significantly improving efficiency. Learn more about Automatic Partitioning.
- 25 New Connectors: This update introduces 25 new connectors. It also expands support across the entire ZOHO suite of applications, including ZOHO CRM, ZOHO Creator, ZOHO Projects, and ZOHO Inventory.
- HTML and PDF Read Connectors: The new functionality for HTML and PDF connectors allows these connectors to read as well as write documents. When reading, the connectors will attempt to identify and parse tables within the document. For single tables, a flat dataset is generated; if multiple tables are detected, a nested document is created for easier handling. More on the HTML and PDF read connectors.
- SFTP2 Connector: Users can now choose between the SFTP and SFTP2 connectors. The new SFTP2 connector, currently in beta, utilizes a modern non-blocking IO library, enabling faster performance, especially for larger data transfers. Check out the details on SFTP2 Connector.
- Automatic ISO Date Conversion in MongoDB: With this release, timestamps in MongoDB can be automatically converted to ISODate format. Storing timestamps as ISODate objects enables more efficient date-based queries and indexing, while allowing flexible configuration for optional conversion. Learn more about ISO Date Conversion.
- Notifications for Interrupted Flows: To maintain consistency, Etlworks now sends notifications for flows that were running when the Etlworks instance restarted without a graceful shutdown. Upon reboot, previously running flows are marked as ‘failed’, triggering notifications and webhooks, if configured.
- Core Dump API Endpoint: The new Core Dump API enables users to capture the JVM’s current state by generating a core dump file for diagnosing system issues. This API saves the core dump in {app.data}/errors and provides the file path. This file can be analyzed with tools like jmap, jstack, or JProfiler for debugging. Check out the Core Dump API.
- X12 Connector Enhancements: The X12 connector now supports converting X12 documents to both JSON and XML formats, expanding integration options for users. Learn more about the X12 Connector. We also improved the resource allocations efficiency of X12 connector when processing large messages.
Important Changes Under the Hood:
- Improved Scripting Engine: The scripting engine is now 20% more efficient, which enhances the performance of flows utilizing JavaScript or Python to calculate column values on a per-record basis.
- CDC Flow Commit Synchronization: CDC flows that write to a database now synchronize offset commits with destination commits, ensuring that offset (source database log position) isn’t committed until all changes reach the destination.
- User Identification in Integration Agent: The Integration Agent now captures the name of users who manually execute flows, which is also available in webhook payloads. This improvement enables differentiation between manually triggered and scheduled flows executed by Integration Agent.
- Enhanced Exception Handling for Table Creation: The user-configurable exception handler now extends to the “create table” SQL command, which is triggered when the flow determines that a table does not exist by attempting to retrieve its metadata. This enhancement allows users to better manage exceptions that occur during automatic table creation, providing greater control over error handling and flow execution. Learn more about exception handling in table ETL flows.
Important Bug Fixes:
- Excel XLSX Column Alignment: We fixed an edge case causing column misalignment when reading XLSX files created by certain third-party tools.
- AS400 CDC Memory Leak: A memory leak in the AS400 CDC connector has been fixed, resulting in improved stability.