Streaming vs Force Streaming

Etlworks handles both flat and nested datasets equally well, whether as sources or destinations. The platform includes two performance optimization features: Streaming and Force Streaming, both configurable under Transformation → Configure → Parameters.

Streaming

When Streaming is enabled, the flow reads data record by record and immediately writes it to the destination. This approach keeps only one record in memory at a time, which significantly reduces memory usage. Streaming is automatically enabled when both the source and destination are flat, such as a CSV file to a relational database table.

Benefits of Streaming

• Very low memory usage
• Automatically uses bind variables and batch processing for relational database destinations
• Recommended for high-volume flat-to-flat integrations

When Streaming is disabled automatically

• The source is not flat, or might contain nested structures (for example, a JSON array with nested objects)
• The destination is not flat, or might contain nested structures (for example, a JSON array with nested objects)
• A Nested Mapping is used
• An After Extract Transformation is configured

In these cases, the flow reads the entire source dataset into memory, iterates through the records, and creates the destination dataset. If the destination is a file, the file is created in memory. If the destination is a relational database table, the system constructs a block of SQL statements and executes them in a single call. This method is slower and consumes more memory.

Force Streaming

Force Streaming is a parameter that improves performance when Streaming is disabled. It changes how the system processes the data after loading it into memory.

When Force Streaming is enabled:

• The flow reads the entire source into memory

• Records are then processed one at a time, with each resulting in an immediate write to the destination

• SQL statements are not accumulated into a single block

• Bind variables and batching (if enabled) are still used

Use Force Streaming only when

• The source is not flat

• The destination is a relational database

• Streaming is disabled manually or automatically

In this configuration, flows are faster and use less memory compared to the default non-streaming behavior.

Avoid Force Streaming when

• Streaming is already enabled (Force Streaming will have no effect and can degrade performance)

• The destination is not a relational database

Summary Table