Overview
Google BigQuery is a serverless, highly scalable, and cost-effective data warehouse designed for business agility. The Etlworks includes several pre-built Flows optimized for Google BigQuery.
Flows optimized for BigQuery
| Flow type | When to use |
|
When you need to extract data from any source, transform it and load it into BigQuery. |
| Bulk load files in Google Cloud Storage into BigQuery | When you need to bulk-load files that already exist in Google Cloud Storage without applying any transformations. The flow automatically generates the COPY command and MERGEs data into the destination. |
| Stream CDC events into BigQuery | When you need to stream updates from the database which supports Change Data Capture (CDC) into BigQuery in real-time. |
| Stream messages from a queue into BigQuery | When you need to stream messages from the message queue which supports streaming into BigQuery in real time. |
Videos
|
ETL with BigQuery Watch how to create ETL flows to load data in BigQuery |
Related resources
|
ELT with Google BigQuery The Etlworks supports executing complex ELT scripts directly in Google BigQuery which greatly improves the performance and reliability of the data ingestion. |
Reverse ETL with Google BigQuery You can use any
|
|
Setup incremental replication using High Watermark (HWM) Using HWM replication you can load only new and updated records into BigQuery. |
Load multiple tables by a wildcard name You can ETL data from multiple database objects (tables and views) into BigQuery by a wildcard name, without creating individual source-to-destination transformations. |
Related case studies
|
Load data from SQL Server databases and file-based sources to Google BigQuery. |
Staples Canada, a leader in office supplies and business services, relies on Etlworks to integrate data from SQL Server databases and file-based sources into Google BigQuery. By leveraging Etlworks’ advanced BigQuery connector with bulk load capabilities, Staples significantly improved the performance of their data integration pipelines, achieving a 10x increase in efficiency. |
|
Extract, transform, and load data from hundreds of sources, running thousands of flows in real-time. |
Aiwyn, a leading AI-driven revenue management platform for professional services firms, needed a scalable solution to integrate data from hundreds of SQL Servers, files, and API-based sources into cloud destinations like Google Cloud BigQuery. With Etlworks, they implemented thousands of near real-time flows and leveraged custom features, such as centralized templates and high-watermark replication, to simplify operations. Their choice of Etlworks was solidified by its unmatched flexibility, performance, and exceptional support. |
Connect to Google BigQuery
Step 1. Open the Connections window and click +.
Step 2. Type bigqueryinto the search field.
Step 3. Select Google Bigquery Connection and continue by defining the Connection parameters.
Read more about configuring BigQuery Connection.
Comments
0 comments
Please sign in to leave a comment.