Get started with message queues

• Starter
• Business
• Enterprise
• On-Premise
• Add-on

What can you do with messages queues in Etlworks Integrator

Read and process messages from the queue Create a Flow where the source is a message queue by typing in queue to in the Flow Selector popup. Read more	Write messages to the queue Create a Flow where the destination is a message queue by typing in to queue in the Flow selector popup. Read more
Work with messages in the JSON, XML, and other text Formats Select String for the Value serializer and Value deserializer. Read more	Read messages in the Avro Format You can read messages in Avro Format created by the Etlworks Integrator or by third-party application. Read more
Write messages in Avro Format Select Avro for the Value serializer field. Read more	Load data from a message queue/Stream CDC events to a message queue  This Flow reads CDC events from the message queue's topics, then transforms and loads data into the Snowflake. In this scenario, the CDC events are extracted from the transaction log of the source database and sent to the message queue. Read more (Load data)/ Read more (Stream CDC)

 

Related resources

Avro Format Etlworks Integrator can read and write Avro files, including nested Avro files. Read more	Kafka connector In the Connections window, click +, and type in kafka. Read more
Flows optimized for Kafka The Etlworks Integrator includes several Kafka-optimized Flow types. Read more	Azure Event Hubs connector In the Connections window, type in azure event hub. Read more
Amazon Kinesis connector In the Connections window, type in kineis. Read more	Amazon SQS connector In the Connections window, type in sqs. Read more
RabittMQ connector In the Connections window, type in rabbit. Read more	ActiveMQ connector In the Connections window, type in activemq. Read more

 

Related case study

eLearning company Streaming data from 1600+ MySQL databases to Snowflake using CDC Read the case study

"A typical CDC Flow can extract data from multiple tables in multiple databases, but having a single Flow pulling data from 55000+ tables would be a major bottleneck as it would be limited to a single blocking queue with a limited capacity. It would also create a single point of failure."

 

Create a real-time data pipeline

Message queues, such as Kafka, can be used to build (almost) real-time data pipelines to connect various applications.

For example, the relational database can publish the CDC log into the Kafka topic. The data integration Flow can subscribe to the topic, so once the new CDC message is published, it can be processed from the queue, parsed, transformed, and sent to the data warehouse as a particular transaction (INSERT / UPDATE / DELETE).

Available connectors

Etlworks Integrator supports the following connectors for message queues:

• Kafka
• Azure Event Hubs
• Amazon Kinesis
• Amazon SQS
• RabbitMQ
• ActiveMQ

User cases

• Reading and processing messages from the queue
• Writing messages to the queue
• Log-based CDC with a message queue
• Real-time change replication with Kafka and Demezium

Read and process messages from the queue

Here are the steps to read and process messages from the queue.

Step 1. Create a source Connection for the message queue to read the messages. For example, Kafka Connection.

Step 2. Create a source Format. The following Formats are supported when reading messages from the queue:

• JSON
• XML
• CSV
• Fixed length
• Avro

Step 3. Create a destination Connection, for example, a Connection to the relational database.

Step 4. Optionally, create a destination Format.

Step 5. Create a Flow where the source is a message queue by typing in queue to in the Flow Selector popup:

Step 6. Continue by adding source-to-destination transformations where the source is a message queue Connection created in step 1, source Format created in step 2, and the destination Connection and (optionally) Format created in steps 3 and 4.

Step 7. When configuring source-to-destination transformation, enter the message queue topic name in the FROM field. Connectors for some message queues, for example, Kafka, support wildcard and comma-separated topic names.

Step 8. Schedule the Flow to stream data in real-time or to be executed periodically.

When scheduling the Flow to be executed periodically, it is recommended to use short intervals for micro-batching.

Write messages to the queue

Here are the steps to write messages to the queue.

Step 1. Create a source Connection.

Step 2. Optionally create a source Format.

Step 3. Create a destination Connection for the message queue to write the messages to. For example, Kafka Connection.

Step 4. Create a destination Format. The following Formats are supported when writing messages to the queue:

• JSON
• XML
• CSV
• Fixed length
• Avro

Step 5. Create a Flow where the destination is a message queue by typing in to queue in the Flow selector popup:

Step 6. Continue by adding source-to-destination transformations where the source Connection and Format are Connection and Format created in steps 1 and 2, and the destination Connection and Format are Connection and Format created in steps 3 and 4.

Step 7. Schedule the Flow to be executed periodically.

Work with messages in the JSON, XML, and other text Formats

To work with messages in these Formats, when configuring the Kafka Connection, select String for the Value serializer and Value deserializer.

Read messages in the Avro Format

Read messages in Avro Format created by the Etlworks Integrator

When configuring the Kafka Connection, select Avro for the Value deserializer field.

Read messages in Avro Format created by third-party application

When configuring the Kafka connection, select Avro Record for the Value deserializer field.

Copy the Avro schema in the JSON Format into the Schema field. 

Write messages in Avro Format

When configuring the Kafka Connection, select Avro for the Value serializer field.