Case Study: Intertek Alchemy Streams Real-Time CDC Data from 1,500+ MySQL Databases to Snowflake

eLearning Leader Saves Hundreds of Thousands Annually by Streaming CDC Data from 1,500+ MySQL Databases to Snowflake

Introduction

Intertek Alchemy, a global leader in workforce training solutions, faced a monumental challenge: seamlessly streaming real-time Change Data Capture (CDC) events from over 1,500 MySQL databases into Snowflake. Despite exploring multiple data integration platforms, no solution on the market could meet their scalability and real-time performance requirements—until they discovered Etlworks.

The Challenge

Intertek Alchemy’s eLearning platform supports thousands of clients, generating vast amounts of data across 1,500+ MySQL databases. The company needed a way to:

• Stream real-time CDC events to Snowflake without delays.

• Handle the scale and complexity of integrating such a large number of databases efficiently.

• Simplify pipeline management to easily add new databases and tables without complex reconfigurations.

No existing data integration platform could handle the scale and flexibility required. The company tried multiple solutions but consistently hit performance and scalability roadblocks.

Why Etlworks

Etlworks stood out as the only platform capable of addressing Intertek Alchemy’s unique challenges. Its flexibility, real-time data streaming capabilities, and support for large-scale CDC pipelines made it the clear choice. Additionally, Etlworks’ hands-on support and expertise ensured a smooth implementation and ongoing optimization.

The Solution

The Etlworks team collaborated closely with Intertek Alchemy to design and implement a robust, scalable CDC solution:

Real-Time CDC Streaming: Etlworks enabled seamless real-time streaming of CDC events from over 1,500 MySQL databases into Snowflake, ensuring near-instantaneous data availability for analytics.

Pipeline Refactoring for Efficiency: During implementation, Etlworks suggested several refactoring strategies to improve efficiency, including:

Externalized Configuration: By storing configuration details (e.g., database and table mappings) in Snowflake tables, the pipelines became highly modular. Adding new databases or tables no longer required modifying the pipelines themselves, significantly reducing complexity.

Optimized Processing: Enhancements to the pipeline’s structure resulted in a 10x improvement in overall performance.

Scalable Architecture: The solution was designed to handle current data volumes while allowing for easy scaling as Intertek Alchemy grows.

Results

With Etlworks, Intertek Alchemy achieved:

Unmatched Scalability: Seamlessly integrated 1,500+ MySQL databases into Snowflake with real-time CDC streaming.

10x Performance Boost: Refactoring the pipeline improved efficiency, dramatically reducing processing times.

Simplified Maintenance: Externalized configurations made it easy to add new databases and tables without modifying the core pipelines.

Future-Proof Integration: The scalable architecture ensures the solution can handle growing data volumes and new use cases.

Customer Quote

“We rely on Etlworks to seamlessly collect data from over 1,600 MySQL databases using Change Data Capture (CDC) and load it into Snowflake. The platform has saved us hundreds of thousands of dollars annually while providing both our team and our customers with instant access to actionable insights.”

Key Takeaways

Unrivaled Scalability: Etlworks is the only platform that could integrate real-time CDC events from over 1,500 databases into Snowflake.

Efficiency Gains: A 10x performance improvement and simplified configuration management transformed the pipeline’s operations.

Tailored Support: Etlworks’ team provided hands-on guidance, ensuring long-term success.

Ready to tackle your most complex data challenges? Discover how Etlworks can transform your data integration workflows. Start your free trial today or request a demo.