Sermo Collects Half a Billion Records Daily Across Diverse Data Sources with Etlworks.
Introduction
Sermo, a global medical community platform, needed to aggregate massive volumes of data from diverse sources, including SQL Server databases, SaaS platforms like Marketo, Salesforce, and Smartsheet, and load it into Amazon Redshift in near real-time. Within two weeks of subscribing to Etlworks, Sermo was collecting and processing over half a billion records daily—an achievement made possible by Etlworks’ advanced features and unparalleled support.
The Challenge
Sermo faced several critical challenges:
Data Source Variety: Integrating data from SQL Server, Marketo, Salesforce, and Smartsheet required a flexible platform capable of handling databases, SaaS platforms, and structured data.
High Volume: The sheer scale of data—billions of records monthly—required a solution optimized for performance and parallel processing.
Pipeline Complexity: Setting up hundreds of pipelines manually for various source-destination pairs was impractical and time-consuming.
Traditional data integration tools lacked the ability to handle these challenges efficiently, leaving Sermo in need of a better solution.
Why Etlworks
Sermo chose Etlworks for its ability to simplify and scale complex data integration workflows:
Wildcard Processing: Allowed multiple source-destination pairs to be managed as a single transformation, drastically reducing setup time.
High Watermark Data Replication: Ensured pipelines extracted only incremental changes from the source, optimizing performance and reducing resource usage.
Parallel Processing: Enabled high-speed data integration, even at scale.
Exceptional Support: Etlworks provided proactive guidance, customized features, and immediate responses to challenges, making it feel like an extension of Sermo’s team.
The Solution
With Etlworks, Sermo implemented a scalable and efficient data integration process:
Unified Data Sources: Connected SQL Server, Marketo, Salesforce, and Smartsheet seamlessly.
Real-Time Data Loading: Configured pipelines to load data into Amazon Redshift with near-zero latency.
Optimized Performance: Leveraged wildcard transformations and high watermark replication to handle high volumes of data while minimizing resource consumption.
Collaborative Support: Worked closely with Etlworks’ team to implement custom features and refine the setup for maximum efficiency.
Customer Quotes
Bryan M., Sr. Software Engineer, Sermo:
“I honestly feel like I’m working with another developer on my team when I’m speaking to them. Communication is direct and instant, and they have been proactive in investigating our setup and adjusting their code to accommodate our scenarios.
They added a bunch of features for us and gave us step-by-step directions on using undocumented features when I needed to do very advanced stuff. What I love best is when I ask them a question or identify an issue, they are always on my level and understand what I’m talking about—there’s no back and forth or explaining.”
Customer Summary Quote:
“Within two weeks after subscribing to Etlworks, we were collecting half a billion records of data every day from hundreds of tables in multiple SQL Server databases, Marketo, Salesforce, and Smartsheet and loading it into Redshift in almost real time.”
Key Takeaways
Scalability and Speed: Etlworks enabled Sermo to process over half a billion records daily with near real-time performance.
Simplified Integration: Wildcard transformations and high watermark replication streamlined setup and maintenance.
Unmatched Support: Etlworks delivered proactive, expert-level guidance, ensuring success for complex use cases.
Ready to tackle your most complex data challenges? Discover how Etlworks can transform your data integration workflows. Start your free trial today or request a demo.
Comments
0 comments
Please sign in to leave a comment.