Before you can map fields in Etlworks, you need to define the source and destination for each transformation. These objects determine where the data is coming from and where it’s going, and they provide the metadata necessary for automatic field mapping.
What Is a Source and Destination?
In Etlworks, every source-to-destination transformation consists of:
-
A source: the object from which the data is read.
Example: a database table, a CSV file, a JSON API response.
-
A destination: the object where the data will be written.
Example: a database table, a flat file, a web service endpoint.
Defining these objects is required for the mapping and data transfer logic to work.
How to Set the Source and Destination
There are two main ways to define the source and destination objects:
1. Manual Entry
You can type the name of the source or destination object directly into the corresponding field. This is useful when:
-
Metadata is not available (e.g., dynamic or tokenized object names).
-
The object doesn’t yet exist (e.g., the flow will create it at runtime).
-
You are using expressions, wildcards, or tokens.
Examples:
-
Manual name: orders_2024
-
With token: orders_{year}
-
With wildcard: orders_*
2. Selecting from Metadata Dropdown
If Etlworks is able to connect to the source and/or destination system, it will automatically extract metadata such as available tables or files. When metadata is available:
-
You can select the source and destination from an auto-populated dropdown.
-
This ensures that the object names are valid and allows for easier field mapping.
When Metadata Is Extracted
Metadata is extracted automatically when all of the following are true:
-
The source or destination connections are configured correctly.
-
Etlworks is able to connect to the systems using the credentials provided.
-
You have selected or entered a valid schema or folder path (when required).
If any of these are missing, the dropdowns may be empty and you’ll need to enter object names manually.
Filtering and bulk actions
Filter transformations
You can filter transformations by source or destination.
-
Use the Search field in the FROM (source) or TO (destination) columns
-
Results update as you type
-
Only matching transformations remain visible
This is useful when working with many transformations.
The transformation list includes tools to quickly manage multiple transformations.
Select transformations
You can select one or more transformations:
-
Use the checkbox next to each row
-
Use the header checkbox to select all visible transformations
Delete selected transformations
To remove multiple transformations:
-
Select one or more transformations
-
Click the Delete icon
All selected transformations will be removed.
Bulk replace source and destination
You can update multiple transformations at once using pattern-based replacement.
-
Select one or more transformations
-
Click Bulk replace source and destination
-
Configure options:
-
Source regex replace
-
Destination regex replace
-
Pattern
-
Replacement
-
Ignore case
-
Click Apply
Notes
-
Bulk replace uses pattern matching and supports regular expressions
-
Changes are applied only to selected transformations
Bulk edit transformations
You can apply configuration changes to multiple transformations at once.
-
Select one or more transformations
-
Click Bulk edit transformations
-
Modify any transformation settings
-
Click Apply
What can be updated
Bulk edit allows updating:
-
Source and destination configuration
-
Mapping (entire mapping can be replaced)
-
Source query or script
-
Parameters
-
Additional and complex transformations
-
On Exception logic
Any changes made in the bulk editor are applied to all selected transformations.
Using Source and Destination Queries
In addition to specifying object names, you can define optional queries that control:
-
Source query: What data is extracted from the source
-
Destination query: How data is inserted or updated in the destination
What's Next?
Learn about Working with the Mapping Editor.