Overview
Since release 10.0.12, the S3-compatible Object Storage connector connects Etlworks to DigitalOcean Spaces, MinIO, and other storage services that support the S3 API. It uses the same AWS SDK transport as the Amazon S3 connector — including file transfers, Explorer, metadata, wildcards, multipart uploads, and chunked downloads — pointed at the endpoint of your provider instead of AWS. Available operations depend on the provider's S3 API support.
When to use this connector
- When working with files stored in an S3-compatible service: DigitalOcean Spaces, a self-hosted or on-premises MinIO, or any other object store that speaks the S3 API.
- When streaming CDC events or queue messages into files in an S3-compatible store.
- For Amazon S3 itself, use the Amazon S3 connector — it adds AWS-specific capabilities (region list, IAM roles, the AWS default credential chain, Multi-Region Access Points) that this connector does not have.
Creating a connection
- Open Connections and click +.
- Type s3 and select S3-compatible Object Storage (group Storage).
- Enter the Endpoint URL, Signing Region, Bucket and credentials (tables below).
- Click Test Connection.
Connection parameters
Common parameters
| Parameter | Description |
|---|---|
| Endpoint URL (required) | The service API endpoint, including https:// (or http:// for a local service) — for example https://nyc3.digitaloceanspaces.com. Enter the bucket separately. Do not use a CDN URL. A port and a service path prefix are supported. The URL must be an absolute HTTP or HTTPS URL without credentials, query or fragment. |
| Signing Region (required; default us-east-1) | The region used to sign requests. Use the value required by your provider. For DigitalOcean Spaces use us-east-1 — the Endpoint URL selects the datacenter; the signing region does not redirect requests to AWS. |
| Use Path-style Access (default disabled) | Enable for services such as MinIO that require endpoint/bucket/key addressing. Disabled uses the SDK default, normally bucket.endpoint/key. For DigitalOcean Spaces leave disabled. |
| Bucket (required) | The name of the bucket or Space. Enter the directory separately. |
| Directory | The directory (key prefix) inside the bucket. Optional. |
| File(s) (required; default *.*) | The file name or a wildcard, for example *.csv. |
| Metadata | Object metadata as name-value pairs. Metadata is ignored when multipart upload is configured. |
Authentication parameters
| Parameter | Description |
|---|---|
| Access Key ID (required) | The access key issued by your object storage provider. |
| Secret Access Key (required) | The secret key issued by your object storage provider. |
Unlike the Amazon S3 connector, there are no IAM Role, External ID, AWS Account ID or "Use AWS default credential chain" options — authentication is always by access key and secret.
Timeouts and auto-retries
| Parameter | Description |
|---|---|
| Connection timeout (ms) | Time to wait for establishing a connection. Default 10000 (10 seconds). |
| Socket timeout (ms) | Time to wait for data read/write operations on an established connection. Default 50000 (50 seconds). |
| Max connections | The maximum number of concurrent connections the HTTP client can establish. Default 10, sufficient for most use cases. |
| Number of Retries | The maximum number of times a single request is retried after a retryable error. |
| Initial wait time (ms) | The wait before the first retry; the delay grows exponentially with each retry, with jitter. |
| Maximum delay (seconds) | The upper bound on the wait between retries. |
Upload and download
| Parameter | Description |
|---|---|
| Part Size For Multipart Upload (bytes) | Empty (the default) disables multipart upload. Minimum 5242880 (5 MB), maximum 5368709120 (5 GB). Multipart upload stores objects larger than 5 GB and uploads large files in smaller pieces to reduce the impact of transient network errors. |
| Download chunk size (bytes) | The size of each chunk downloaded during a transfer. Chunked downloads improve reliability for large files: an interrupted download retries only the current chunk. |
Other parameters
The remaining parameters work exactly as in the other storage connectors:
- Add Suffix When Creating Files in Transformation — timestamp, uuid and other suffix algorithms for created files.
- File Processing Order — the order wildcard-matched source files are processed in (default oldest).
- Contains CDC events — adds the standard CDC wildcard templates to the FROM selector.
- PGP Private Key File and PGP Password — automatic decryption of PGP-encrypted files.
- Expected Compression and Archive output files — read compressed resources and archive created files (Zip, GZip).
- Script to run before connect — JavaScript that runs before the connection opens, typically to set global variables that replace {tokens} in the connection parameters.
Provider examples
DigitalOcean Spaces
| Parameter | Value |
|---|---|
| Endpoint URL | https://nyc3.digitaloceanspaces.com (replace nyc3 with the Space's datacenter) |
| Signing Region | us-east-1 |
| Use Path-style Access | Disabled |
| Bucket | The Space's name |
| Access Key ID / Secret Access Key | A Spaces access key and secret with the required bucket permissions |
Important: use the regional API endpoint, not a CDN endpoint.
MinIO
| Parameter | Value |
|---|---|
| Endpoint URL | http://localhost:9000 (localhost refers to the host executing the connection — the Etlworks server, an Integration Agent, or a container) |
| Signing Region | us-east-1 |
| Use Path-style Access | Enabled |
Other providers
Any service that implements the S3 API can be used: set the provider's endpoint, the signing region it documents, path-style access if it requires endpoint/bucket/key addressing, and an access key and secret it issued. Use HTTPS for remote services, with a certificate the executing host trusts. Available operations depend on how completely the provider implements the S3 API.
Explorer
The connector fully supports Explorer: browsing folders, creating, renaming and deleting folders, creating, editing and renaming files, upload and download, and metadata discovery — the same as the Amazon S3 connector.
Using it in flows
The connection works as a source and a destination in all file-based flows — File to File, Copy, Move and Delete Files, File to Database, Database to File, and the other file flows — and as a destination for Stream CDC events, Create Files and Stream messages from queue, create files.
Limitations:
- Warehouse-native staging keeps its provider restrictions: the Snowflake, Amazon Redshift and Databricks bulk-load flows that stage through S3 require the Amazon S3 connector. This connector does not make a warehouse's native loader support arbitrary object stores.
- Metadata is ignored when multipart upload is configured.
Integration Agents
Flows using this connector run on Integration Agents as well as on the Etlworks server. The Agent executing the connection must be updated to a release that includes the connector (10.0.12 or newer); update the Agent and test the flow on that Agent. Existing Amazon S3 connections are unchanged.