This tutorial walks through extracting data from one or more files in a file storage and loading it into a relational database.
How do I set this up?
Set up the connections
See Creating and managing connections for background.
- Create a source connection for the file storage and test it. Supported storage connectors include:
- Create the destination database connection and test it.
-
Create a format for the source file — for example, CSV.
Optional: test the source in Explorer
You can preview the source data in the Etlworks Explorer. For nested source files, see how to work with nested datasets.
Create the flow
- Create a new File to database flow.
- Add a transformation. Set FROM to the source connection, format, and filename (or a wildcard filename), and TO to the destination connection and table.
- Click MAPPING and, if the source is nested JSON or XML, configure flattening — usually a Source SQL against the nested document.
- Test the transformation. The output should be a flat dataset.
- Add per-field mapping if needed.
- If FROM is a wildcard filename, enable processing of all files that match the wildcard.
- Configure MERGE (UPSERT) if you want to update existing records instead of just inserting.
- Save the flow and execute it manually.
-
Schedule the flow to run periodically.