In the Etlworks Integrator, you can configure a CSV Format to automatically split a CSV document into multiple chunks when creating a new file or sending a payload to the HTTP endpoint.
Typically, it makes sense only for large documents (millions of rows) because it can greatly improve the performance of certain data load tasks, such as loading data into the online data warehouses: Snowflake, Redshift, etc. These systems can load files in parallel. Transferring smaller files over the network can often be faster.
To always split the CSV file when creating it, create a new CSV Format and enter a positive value in the field
Maximum number of rows in file. Then use the created Format in the
TO part of the source-to-destination transformation.
Please sign in to leave a comment.