Get started with Snowflake

Snowflake is a data warehouse built for the cloud. Read more about Snowflake. The Etlworks Integrator includes several pre-built Flows optimized for Snowflake.

What can you do with Snowflake in Etlworks Integrator

ETL with Snowflake Using Snowflake-optimized Flows, you can extract data from any of the supported sources, transform, and load it directly into Snowflake Read more	Load multiple tables by a wildcard name You can load multiple databases objects (tables and views) by a wildcard name (without creating individual source-to-destination transformations) Read more
Setup incremental change replication using change data capture (CDC) Once the CDC is configured for the source database, you can create a CDC pipeline where the source is one of these databases and the destination is a Snowflake. Read more	Setup incremental Change Replication using a high watermark As in any other Flow type, it is possible to configure a Change Replication using high watermark. Read more
Bulk load files in Snowflake If you already have CSV, JSON, Parquet or Avro files in the local (server) or cloud storage and don't need to transform the data, the most efficient way to load these files into the Snowflake is to use the flow type Bulk load files into Snowflake Read more

Videos

	How to load data into Snowflake A typical Snowflake-optimized flow does the following: Extracts data from the source. Creates CSV, JSON, Avro, or Parquet files. Compress files using the gzip algorithm. Copies files into Snowflake stage. Checks to see if the destination Snowflake table exists. Dynamically generates and execute the Snowflake `COPY INTO` command. Cleans up the remaining files, if needed. Read the full article

Related resources

Extract, transform and load data in Snowflake Using Snowflake-optimized Flows, you can extract data from any of the supported sources, transform, and load it directly into Snowflake. Read more	Directly load files in Snowflake This Flow loads CSV or JSON files directly into Snowflake. The files can be anywhere, so long as the Etlworks Integrator can read them. Read more	ELT with Snowflake The Etlworks Integrator supports executing complex ELT scripts directly in Snowflake, which greatly improves the performance and reliability of the data ingestion. Read more
Work with Snowflake as a relational database You can use any `Database to` Flow type to extract data from Snowflake. Read more	Data type Mapping for Snowflake It is important to understand how we map various JDBC data types for the Snowflake data types. Read more

Related case studies

eLearning company Streaming data from 1600+ MySQL databases to Snowflake using CDC Read the case study	"A typical CDC Flow can extract data from multiple tables in multiple databases, but having a single Flow pulling data from 55000+ tables would be a major bottleneck as it would be limited to a single blocking queue with a limited capacity. It would also create a single point of failure."
Major retail chain Load data from 600+ SQL Servers behind a firewall Read the case study	"A major retail chain with stores across the US operates 600+ independent Microsoft SQL Server databases that work behind the firewall. They needed to consolidate data in the Snowflake data warehouse and the expected traffic from each location is tens of millions of records per day."

Articles in this section

What can you do with Snowflake in Etlworks Integrator

Videos

Related resources

Related case studies

"A typical CDC Flow can extract data from multiple tables in multiple databases, but having a single Flow pulling data from 55000+ tables would be a major bottleneck as it would be limited to a single blocking queue with a limited capacity. It would also create a single point of failure."

"A major retail chain with stores across the US operates 600+ independent Microsoft SQL Server databases that work behind the firewall. They needed to consolidate data in the Snowflake data warehouse and the expected traffic from each location is tens of millions of records per day."

Comments

Articles in this section

What can you do with Snowflake in Etlworks Integrator

Videos

Related resources

Related case studies

"A typical CDC Flow can extract data from multiple tables in multiple databases, but having a single Flow pulling data from 55000+ tables would be a major bottleneck as it would be limited to a single blocking queue with a limited capacity. It would also create a single point of failure."

"A major retail chain with stores across the US operates 600+ independent Microsoft SQL Server databases that work behind the firewall. They needed to consolidate data in the Snowflake data warehouse and the expected traffic from each location is tens of millions of records per day."

Related articles