Etlworks handles both flat and nested datasets equally well, whether as sources or destinations. The platform includes two performance optimization features: Streaming and Force Streaming, both configurable under Transformation → Configure → Parameters.
Streaming
When Streaming is enabled, the flow reads data record by record and immediately writes it to the destination. This approach keeps only one record in memory at a time, which significantly reduces memory usage. Streaming is automatically enabled when both the source and destination are flat, such as a CSV file to a relational database table.
Benefits of Streaming
- Very low memory usage
- Automatically uses bind variables and batch processing for relational database destinations
- Recommended for high-volume flat-to-flat integrations
When Streaming is disabled automatically
- The source is not flat, or might contain nested structures (for example, a JSON array with nested objects)
- The destination is not flat, or might contain nested structures (for example, a JSON array with nested objects)
- A Nested Mapping is used
- An After Extract Transformation is configured
In these cases, the flow reads the entire source dataset into memory, iterates through the records, and creates the destination dataset. If the destination is a file, the file is created in memory. If the destination is a relational database table, the system constructs a block of SQL statements and executes them in a single call. This method is slower and consumes more memory.
Note (10.0.0+): nested JSON and XML sources no longer force the whole document into memory in every case. The JSON and XML formats offer Full streaming read, which reads records one at a time and preserves nested elements as native-format strings, and HTTP connections offer Stream JSON Response / Stream CSV Response. These options stream the reading stage — a different stage of the pipeline than Force Streaming, which streams the load of an already materialized dataset. See Stream large nested JSON and XML with serialized nested elements.
Source-aware Stream Data defaults (10.0.8+)
New transformations choose their Stream Data default from the source shape: database and memory connections, flat formats, sources with a streaming read hint (for example HTTP with Stream JSON Response or formats configured for streaming reads), and streaming MongoDB, Kafka, Pulsar, and SQS default to on. Destination JSON/XML formats no longer switch Stream Data off.
These are editor defaults only: saved values and manual choices always win, configured transformations keep their previous behavior, and engine eligibility, inspection, and Force Streaming rules are unchanged. To stream an already-saved transformation, select Stream Data explicitly.
Eligible sources can also stream through a per-record nested mapping (10.0.8+) — see per-record nested mapping.
Force Streaming
Force Streaming is a parameter that improves performance when Streaming is disabled. It changes how the system processes the data after loading it into memory.
When Force Streaming is enabled:
- The flow reads the entire source into memory
- Records are then processed one at a time, with each resulting in an immediate write to the destination
- SQL statements are not accumulated into a single block
- Bind variables and batching (if enabled) are still used
Force Streaming values (10.0.0+)
Starting with 10.0.0, the transformation's Force Streaming parameter has three values instead of a checkbox:
- Automatic (the default): follows the tenant or global Automatic Force Streaming policy for eligible transformations — a non-streamable or materializing source loading into a relational database, with a compatible mapping. When the policy is on, Force Streaming is applied automatically; the parameter shows the currently effective policy when Automatic is selected.
- Enabled: explicitly requests Force Streaming, as before.
- Disabled: always opts out, regardless of the policy.
Existing saved values are mapped without rewriting flows: existing false (or empty) values mean Automatic, and existing true values mean Enabled.
The policy is controlled by:
Global: The global setting Enable Automatic Force Streaming (Settings → Flows), on by default.
Tenant: An optional tenant override — Automatic Force Streaming in the tenant editor's Flows category (Use default / Enable / Disable).
Notes:
- Automatic applies only to transformations in flows that offer the Force Streaming option, and only when the forced load is compatible with the mapping. Nested source mappings into a flat relational destination are eligible; nested destination mappings and incompatible SQL mappings or native actions are not.
- Automatic Force Streaming does not make source reading streamable — it streams the already materialized dataset into the destination.
- Force Streaming never overrides an explicit bind-variables opt-out.
- Flows executed by a pre-10.0 Integration Agent apply Force Streaming only when it is explicitly Enabled; the Automatic policy requires a 10.0.0+ Agent.
Use Force Streaming only when
- The source is not flat
- The destination is a relational database
- Streaming is disabled manually or automatically
In this configuration, flows are faster and use less memory compared to the default non-streaming behavior.
Avoid Force Streaming when
- Streaming is already enabled (Force Streaming will have no effect and can degrade performance)
- The destination is not a relational database
Summary Table
| Scenario | Streaming | Force Streaming |
| Flat source to flat destination | Enabled automatically | Not needed |
| Potentially nested source (JSON, XML, etc.) to database table | Disabled automatically | Optional and recommended |
| Potentially nested source (JSON, XML, etc.) to flat or nested destination which is not a database table | Disabled automatically | Not recommended |
| Any source to database table when nested mapping or after-extract is used | Disabled automatically | Optional and recommended |
| Any source to any destination which is not a database table when nested mapping or after-extract is used | Disabled automatically | Not recommended |