Get started with Snowflake

Overview

Snowflake is a data warehouse built for the cloud. Read more about Snowflake. The Etlworks Integrator includes several pre-built Flows optimized for Snowflake.

Flows optimized for Snowflake


Flow type	When to use
Any to Snowflake: Database to Snowflake File to Snowflake Queue to Snowflake Web service to Snowflake Well-known API to Snowflake	When you need to extract data from any source, transform it and load it into Snowflake.
Bulk load files into Snowflake	When you need to bulk-load files that already exist in the external Snowflake stage (S3, Azure Blob, GC blob) or in the server storage without applying any transformations. The flow automatically generates the COPY INTO command and MERGEs data into the destination.
Stream CDC events into Snowflake	When you need to stream updates from the database which supports Change Data Capture (CDC) into Snowflake in real time.
Stream messages from a queue into Snowflake	When you need to stream messages from the message queue which supports streaming into Snowflake in real time.
COPY files into Snowflake	When you need to bulk-load data from the file-based or cloud storage, API, or NoSQL database into Snowflake without applying any transformations. This flow requires providing the user-defined COPY INTO command. UnlikeBulk load files into Snowflake, this flow does not support automatic MERGE.

Videos

	ETL, CDC, and bulk load data into Snowflake Watch how to create flows to ETL, CDC and bulk load data into Snowflake
	How ETL data into Snowflake A typical Snowflake-optimized flow does the following: Extracts data from the source. Creates CSV, JSON, Avro, or Parquet files. Compress files using the gzip algorithm. Copies files into Snowflake stage. Checks to see if the destination Snowflake table exists. Dynamically generates and execute the Snowflake `COPY INTO` command. Cleans up the remaining files, if needed. Read the full article

Related resources

ELT with Snowflake The Etlworks Integrator supports executing complex ELT scripts directly in Snowflake, which greatly improves the performance and reliability of the data ingestion. Read more	Reverse ETL with Snowflake You can use any `Database to...` Flow type to extract data from Snowflake, transform and load it into any destination. Read more	Data type Mapping for Snowflake It is important to understand how we map various JDBC data types for the Snowflake data types. Read more
Load multiple tables by a wildcard name You can ETL data from multiple database objects (tables and views) into Snowflake by a wildcard name without creating individual source-to-destination transformations. Read more	Setup incremental change replication using a high watermark (HWM) Using HWM replication you can load only new and updated records into Snowflake. Read more	Automatic creation of the Snowflake stage Etlworks can automatically create an internal or external Snowflake stage. Read more

Related case studies

eLearning company Streaming data from 1600+ MySQL databases to Snowflake using CDC Read the case study	"A typical CDC Flow can extract data from multiple tables in multiple databases, but having a single Flow pulling data from 55000+ tables would be a major bottleneck as it would be limited to a single blocking queue with a limited capacity. It would also create a single point of failure."
Major retail chain Load data from 600+ SQL Servers behind a firewall Read the case study	"A major retail chain with stores across the US operates 600+ independent Microsoft SQL Server databases that work behind the firewall. They needed to consolidate data in the Snowflake data warehouse and the expected traffic from each location is tens of millions of records per day."

Articles in this section

Overview

Flows optimized for Snowflake

Videos

Related resources

Related case studies

"A typical CDC Flow can extract data from multiple tables in multiple databases, but having a single Flow pulling data from 55000+ tables would be a major bottleneck as it would be limited to a single blocking queue with a limited capacity. It would also create a single point of failure."

"A major retail chain with stores across the US operates 600+ independent Microsoft SQL Server databases that work behind the firewall. They needed to consolidate data in the Snowflake data warehouse and the expected traffic from each location is tens of millions of records per day."

Comments

Articles in this section

Overview

Flows optimized for Snowflake

Videos

Related resources

Related case studies

"A typical CDC Flow can extract data from multiple tables in multiple databases, but having a single Flow pulling data from 55000+ tables would be a major bottleneck as it would be limited to a single blocking queue with a limited capacity. It would also create a single point of failure."

"A major retail chain with stores across the US operates 600+ independent Microsoft SQL Server databases that work behind the firewall. They needed to consolidate data in the Snowflake data warehouse and the expected traffic from each location is tens of millions of records per day."

Related articles