Any to Any ETL

When to use this Flow type

Any to any ETL Flow is a flavor of the Flow type Extract data from source, transform, and load in destination. 

Unlike all other Flows in this group, Any to any ETL Flow allows developers to include different types of source-to-destination transformations within the same Flow.

For example, you can pipeline the transformation which extracts data from a web service and loads it into the temporary database with the transformation which loads data from the temp database into multiple targets: databases, files, etc.

Read how to use Any to any ETL Flow to apply JOIN and SET transformations to the multiple heterogeneous data sources. 

Note that you can achieve the same goal by using Nested Flow. Unlike nested Flow that can include any Flow type (even other nested Flows), Any to any ETL Flow is limited to source-to-destination transformations. 

Process

To create a Flow, go to Flows window, click +, and type in any to any. Select the Flow and add one or multiple source-to-destination transformations.

Do not use  Any to Any ETL Flow to load data into Snowflake and Redshift

Snowflake and Amazon Redshift are columnar databases, optimized for data retrieval operations, so loading data into using DML statements (INSERT/ UPDATE/ DELETE) can be quite slow, especially for larger datasets.

While it is possible to load data in Snowflake and Redshift using Any to any ETL Flow, it is highly recommended to use specialized Flows for Snowflake and Redshift instead.