Articles in this section

See more

Merge files

  • Updated
Follow

When to use this Flow type

In Etlworks, it is possible to create a Flow to merge CSV or XML files.

Merge CSV files

Use this Flow type when you want to merge multiple CSV files into a single file.

Note: This Flow can merge two or more CSV files with a different structure. The final CSV file will include a superset of all fields from all source files.

Step 1. Create FROM and TO Connections (it is recommended that you use the same Connection for both).

Note: Only at this time is the Server Storage Connection type supported.

Step 2. Create the FROM and TO Formats, which must be CSV.

Step 3. Select the Merge CSV Files Flow type from the list.

merge_csv.png

Step 4. Create a transformation where the source Connection (FROM) points to the files to merge, and the destination Connection (TO) to the location where the merged file should be created. The FROM and TO Formats must be created in step 2.

Step 5. Enter a wildcard file name, such as *.csv, into the FROM field and the name of the destination file into the TO field.

Step 6. Continue by specifying the transformation parameters:

  • Dynamic Header: if you are not sure whether all source files have the same header, set this parameter to Yes.
  • Header: the header to be added to the merged file. This parameter is ignored if Dynamic Header is set to Yes.
  • Footer: the footer to be added to the merged file. This parameter is ignored if Dynamic Header is set to Yes.
  • On Exception: by default, any error causes the halt of execution. When ignore is selected, the errors will be ignored and execution will continue. Read how to configure the Flow to ignore all or specific exceptions.
  • Exception Mask: you can specify what errors should be ignored and still halt the execution for all other errors. Enter part or all of the exceptioflown string. This field works only when the Ignore On Exception option is selected.
  • Execute if Error: if this option is selected, Etlworks will execute this file operation when an error occurs.

Merge XML files

Use this Flow type when you want to merge multiple XML files into a single file.

Step 1. Create FROM and TO Connections.

Step 2. Select the Merge XML Files Flow type from the list.

merge_xml.png

Step 3. Create a transformation where the source Connection (FROM) points to the files to merge, and the destination Connection (TO) to the location where the merged file should be created.

Step 4. Enter a wildcard file name, such as *.xml, into the FROM field and the name of the destination file into the TO field.

Step 5. Continue by specifying the transformation parameters:

  • Path for the repeating segment in XML: an XML path for the repeating segment, for example, Transaction/Entries. This field is required.
  • Include common segment: if this property is enabled, the system will try to detect a common, non-repeating segment, and include it in the output file.
  • Do not merge files that have already been merged: if this option is enabled, Etlworks will not merge files that have already been merged.
  • On Exception: by default, any error causes the halt of execution. When ignore is selected, the errors will be ignored and execution will continue.
  • Exception Mask: you can specify what errors should be ignored and still halt the execution for all other errors. Enter part or all of the exception string. This field works only when the Ignore On Exception option is selected.
  • Execute if Error: if this option is selected, Etlworks will execute this file operation when an error occurs.

Related articles