Split a CSV file when creating it

Overview

In the Etlworks Integrator, you can configure a CSV Format to automatically split a CSV document into multiple chunks when creating a new file or sending a payload to the HTTP endpoint.

Typically, it makes sense only for large documents (millions of rows) because it can greatly improve the performance of certain data load tasks, such as loading data into the online data warehouses: Snowflake, Redshift, etc. These systems can load files in parallel. Transferring smaller files over the network can often be faster. 

Process

To always split the CSV file when creating it, create a new CSV Format and enter a positive value in the field Maximum number of rows in file. Then use the created Format in the TO part of the source-to-destination transformation.