Amazon SQS connector

• Starter
• Business
• Enterprise
• On-Premise
• Add-on

About Amazon SQS

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. Read more about Amazon SQS.

When to use this connector

• to read messages from and write messages to a given SQS queue.
• to stream messages from the queue to various destinations.
• to implement a log-based CDC with a message queue.
  • Stream CDC events to the message queue.
  • Stream CDC events from the message queue to any destination.

Create a Connection

Step 1. In the Connections window, click +, and type in sqs.

Step 2. Select Amazon SQS.

Step 3. Enter the Connection parameters.

Connection parameters

Common parameters

• AWS Region: the AWS cloud region.
• Queue name: the optional SQS queue name.
• Max number of records to read: the total maximum number of records to read from the queue. Set it to a reasonable number to allow the system to process records in micro-batches. If nothing is entered the system will read records from the queue until there are no more records. 
• Number of retries before stop polling: the number of retries before stop polling if poll returns no records. If no records are returned, that means no data records are currently available from the current shard. The connector will continue reading from the next available shard.
• Retry N minutes before stop pollingDelay between retries: the number of minutes to retry before stop polling if the poll returns no records. If no records are returned, that means no data records are currently available from the current shard. The connector will continue reading from the next available shard.
• Delay between retries (ms): the delay between retries. Set it to avoid exceeding the limit on frequency.
• Queue Type: the type of the queue created by the connector. SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

Authentication

Etlworks SQS connector supports authentication with access/secret key and IAM role.

• Access Key or IAM Role: the AWS access key or IAM role name. 
• Secret Access Key: the AWS secret access key. Note: the secret access key must be empty if authenticating with the IAM role.
• External ID: In abstract terms, the external ID allows the user that is assuming the role to assert the circumstances in which they are operating. It also provides a way for the account owner to permit the role to be assumed only under specific circumstances. The primary function of the external ID is to address and prevent the confused deputy problem.

If both authentication parameters are empty the connector will attempt to authenticate using the default profile configured for the EC2 instance running Etlworks.

NOTE: If you are randomly getting error "Failed to load credentials from IMDS" consider configuring the auto-rerty with at least 5 retry attempts.

Auto-retry

To configure auto-retry for each individual request to AWS API set the following parameters:

• Number of Retries: the maximum number of times that a single request should be retried, assuming it fails for a retryable error.
• Initial wait time (ms): the initial wait time in milliseconds before making the first retry attempt. This delay increases exponentially with each subsequent retry, often combined with jitter to avoid collisions from simultaneous retries. The default is 500 milliseconds. 
• Maximum delay (seconds): without a maximum limit, the wait time can become excessively long, especially after multiple retries. This can lead to significant delays in processing. The default is 10 seconds.