Articles in this section

Get started with files

  • Updated
Follow

etlworks.com_-_Get_started_banners__11_.png

In the Etlworks Integrator, it is possible to read and write data in numerous data exchange Formats, such as CSV, JSON, XML, Excel, etc. The actual files may be in file storage, cloud storage, key-value storage, a NoSQL database, or an email server.

 

What can you do with files in Etlworks Integrator

Copy, move, rename, delete, zip, and unzip files, create folders

The Etlworks Integrator can work with files directly.

Read more

Split/Merge files

Start creating the Flow that splits files by opening the Flows window by typing in split. In the Etlworks Integrator, it is possible to create a Flow to merge CSV or XML files.

Read more (Split file)/ Read more (Merge files)

Apply XSL style sheet to XML files

Start creating the Flow by typing xsl into the Select Flow Type dialog box.

Read more

ETL with files 

Extract, transform and load data when the file is a source or destination.

Read more 

 

Videos

How to extract data from any file and load it into any database

  1. Create a source Connection for the file storage.
  2. Create a destination Connection for the relational database
  3. Create a Format for the source file.
  4. Create a new File to database Flow.
  5. Add a new transformation.
  6. Click MAPPING and optionally configure the flattening for the nested JSON or XML. 

Read the full article

 

How to work with files without transforming data

The Etlworks Integrator supports all sorts of file operations, which can be performed on almost all types of Connections, except relational databases.

Read the full article

 

How to convert any file to any Format

  1. Create source Connection for the file storage.
  2. Create a destination Connection for the file storage. 
  3. Create a Format for the source file.
  4. Create a Format for the destination file.
  5. Create a new File to file Flow.
  6. Add the new transformation.

Read the full article

Related resources

Automatically encrypt files in server storage

ETL Flows that create files in the server storage can automatically encrypt all created files using the PGP algorithm.

Read more

Create XML and JSON documents using JavaScript

Create a Flow where the destination is a file (JSON or XML).

Read more

Filter and modify rows in a source CSV file

You can filter out or modify some of the rows in the CSV file before passing the data to the transformation.

Read more

Web/HTML scraping with JavaScript and Python

For Web/HTML scraping, Etlworks Integrator includes a Java library jsoup. It is one of the best HTML parsers around. 

Read more

 

Connect to a data storage, NoSQL database, or email server

Test a Connection

To test a Connection, click Test Connection on the Connection screen. This is only available for cloud and file storage.

In addition to actually connecting to the storage, the Etlworks Integrator attempts to read file names using a configured folder and file name. The Etlworks Integrator supports wildcard file names.

test-file-connection.png

Formats

When working with files, it is required that you describe the data exchange Format. The Etlworks Integrator supports the most commonly used Formats:

Browse files and view data

Use the the Etlworks Explorer to browse files and view data.

Step 1. Create a file Connection.

Step 2. Choose a Format.

Step 3. Open the Etlworks Explorer, select the Connection created in Step 1 and link it to the Format chosen in Step 2. 

Step 4. Explore the metadata (files and fields), view data in a grid, query data, and discover dependencies using SQL.

File operations

ETL with files

Expose file as an API endpoint

Tips and tricks when working with files

Related articles

Comments

0 comments

Please sign in to leave a comment.