When to use this connector
Use this connector to create Flows that work with files in Google Cloud Storage.
Creating a Connection
To connect to Google Cloud Storage, you will need to enable the Interoperability API. To enable it, go to
Google Cloud console >
Enable and create an access key pair, which includes an
Access Key ID and
Secret Access Key.
Step 1. In the
Connections window, click
+, and select
Step 2. Select
Step 3. Enter Connection parameters.
Endpoint: the web service host. It defaults to
Bucket: the bucket name.
Directory: the directory under the bucket. This parameter is optional.
Files: the actual file name or a wildcard file name, for example,
Headers: optional HTTP headers.
Other parameters:additional configuration options for the Google Storage connection.
Access Key: the username.
Add Suffix When Creating Files in Transformation: you can select one of the predefined suffixes for the files created using this Connection. For example, if you select
uuidas a suffix and the original file name is
dest.csv, Etlworks will create files with the name
dest_uuid.csv, where uuid is a globally unique identifier such as
Enable Wildcard File Name: if there are multiple source files in a folder, you can specify an algorithm that will be used to select an actual source file to process. For example, if
Filesis set to
oldestis selected, Etlworks will always select the oldest file in the folder which matches the wildcard
Override Wildcard File Name set for Connection: if
Wildcard File Namesare allowed, the option to
Override Wildcard File Name set for Connectionis enabled, and the file name entered in the
FROMfield of the transformation is a wildcard file name, the system will override the file name entered at the connection level. The default behavior is to use the wildcard file name that was entered when the Connection was configured.
Archive file before copying to: Etlworks can archive files using one of the supported algorithms (zip or gzip) before copying them to cloud storage. Since cloud storage is typically a paid service, it can save money and time if you choose to archive files.
Contains CDC events:When this parameter is enabled, Etlworks adds standard wildcard templates for CDC files to the list of available sources in the FROM selector.
When Google Cloud Storage Connection is used as a source (
FROM) in the source-to-destination transformation, it is possible to configure the automatic decryption of the encrypted source files using the PGP algorithm and private key uploaded to the secure key storage.
If the private key is available, all source files processed by the transformation will be automatically decrypted using the PGP algorithm and given key. Note that the private key requires a password.