- Starter
- Business
- Enterprise
- On-Premise
- Add-on
What can you do with messages queues in Etlworks Integrator
Read and process messages from the queue Create a Flow where the source is a message queue by typing in |
Write messages to the queue Create a Flow where the destination is a message queue by typing in |
Work with messages in the JSON, XML, and other text Formats Select |
Read messages in the Avro Format You can read messages in Avro Format created by the Etlworks Integrator or by third-party application. |
Write messages in Avro Format Select |
Load data from a message queue/Stream CDC events to a message queue This Flow reads CDC events from the message queue's topics, then transforms and loads data into the Snowflake. In this scenario, the CDC events are extracted from the transaction log of the source database and sent to the message queue. |
Related resources
Avro Format Etlworks Integrator can read and write Avro files, including nested Avro files. |
Kafka connector In the |
Flows optimized for Kafka The Etlworks Integrator includes several Kafka-optimized Flow types.
|
Azure Event Hubs connector In the |
Amazon Kinesis connector In the
|
Amazon SQS connector In the |
RabittMQ connector In the
|
ActiveMQ connector In the
|
Related case study
eLearning company Streaming data from 1600+ MySQL databases to Snowflake using CDC
|
"A typical CDC Flow can extract data from multiple tables in multiple databases, but having a single Flow pulling data from 55000+ tables would be a major bottleneck as it would be limited to a single blocking queue with a limited capacity. It would also create a single point of failure." |
Create a real-time data pipeline
Message queues, such as Kafka, can be used to build (almost) real-time data pipelines to connect various applications.
For example, the relational database can publish the CDC log into the Kafka topic. The data integration Flow can subscribe to the topic, so once the new CDC message is published, it can be processed from the queue, parsed, transformed, and sent to the data warehouse as a particular transaction (INSERT
/ UPDATE
/ DELETE
).
Available connectors
Etlworks Integrator supports the following connectors for message queues:
User cases
- Reading and processing messages from the queue
- Writing messages to the queue
- Log-based CDC with a message queue
- Real-time change replication with Kafka and Demezium
Read and process messages from the queue
Here are the steps to read and process messages from the queue.
Step 1. Create a source Connection for the message queue to read the messages. For example, Kafka Connection.
Step 2. Create a source Format. The following Formats are supported when reading messages from the queue:
Step 3. Create a destination Connection, for example, a Connection to the relational database.
Step 4. Optionally, create a destination Format.
Step 5. Create a Flow where the source is a message queue by typing in queue to
in the Flow Selector
popup:
Step 6. Continue by adding source-to-destination transformations where the source is a message queue Connection created in step 1, source Format created in step 2, and the destination Connection and (optionally) Format created in steps 3 and 4.
Step 7. When configuring source-to-destination transformation, enter the message queue topic name in the FROM
field. Connectors for some message queues, for example, Kafka, support wildcard and comma-separated topic names.
Step 8. Schedule the Flow to stream data in real-time or to be executed periodically.
When scheduling the Flow to be executed periodically, it is recommended to use short intervals for micro-batching.
Write messages to the queue
Here are the steps to write messages to the queue.
Step 1. Create a source Connection.
Step 2. Optionally create a source Format.
Step 3. Create a destination Connection for the message queue to write the messages to. For example, Kafka Connection.
Step 4. Create a destination Format. The following Formats are supported when writing messages to the queue:
Step 5. Create a Flow where the destination is a message queue by typing in to queue
in the Flow selector
popup:
Step 6. Continue by adding source-to-destination transformations where the source Connection and Format are Connection and Format created in steps 1 and 2, and the destination Connection and Format are Connection and Format created in steps 3 and 4.
Step 7. Schedule the Flow to be executed periodically.
Work with messages in the JSON, XML, and other text Formats
To work with messages in these Formats, when configuring the Kafka Connection, select String
for the Value serializer
and Value deserializer
.
Read messages in the Avro Format
Read messages in Avro Format created by the Etlworks Integrator
When configuring the Kafka Connection, select Avro
for the Value deserializer
field.
Read messages in Avro Format created by third-party application
When configuring the Kafka connection, select Avro Record
for the Value deserializer
field.
Copy the Avro schema in the JSON Format into the Schema
field.
Write messages in Avro Format
When configuring the Kafka Connection, select Avro
for the Value serializer
field.
Comments
0 comments
Please sign in to leave a comment.