Overview
When reading data from files, APIs, or relational databases, discovered column names may include spaces, special characters, inconsistent casing, or other irregularities. These inconsistencies can lead to issues when mapping, transforming, or loading data into target systems.
To address this, Etlworks provides the ability to automatically transform column names using a simple JavaScript function. With this feature, you can clean and standardize column names during metadata discovery, minimizing errors and reducing the need for manual adjustments later in the flow.
How It Works
When retrieving metadata, the connector (for example, the connector for the CSV format) extracts the original column names from the source. You can configure a JavaScript function to automatically transform each column name before it is used in the dataset.
The function must assign the transformed name to the value variable, using the original column name provided in the name variable.
value = name.replaceAll(' ', '_');
This function replaces spaces with underscores (_) in all column names.
Where to Configure
You can define a column name transformation function in:
-
File Formats such as Excel, CSV, JSON, XML, etc.
-
Database Connections for any relational database.
Look for the Overrides section and populate the Function to calculate Field Name field with your JavaScript code.