Server Storage connector

About Server storage

Server Storage refers to file storage located on a local or network storage device (e.g., a hard drive, SSD, or NAS) connected to the host machine running Etlworks.

For users on Enterprise plans or on-premise deployments, the main account can access any files stored on local or network-attached storage devices that are connected to the machine running Etlworks. However, tenant accounts are restricted to their designated Home folder and its subfolders, ensuring proper data separation and security. If you are using Etlworks in the cloud, file access is similarly limited to the Home folder and its subdirectories, maintaining controlled access to the allocated storage space.

Home folder

When you sign up for Etlworks, a dedicated storage volume is created on Etlworks servers, which is accessible exclusively to your account. This is referred to as the Home folder. All data stored in this folder and its subdirectories can be accessed using the Server Storage connection type. To reference the Home folder, you can use the token {app.data} in your configurations.

For Enterprise and On-premise plans, Etlworks users can create tenant sub-accounts. Each tenant has its own isolated Home folder. These tenant accounts are limited to accessing only the Home folder and its subfolders, ensuring data isolation between accounts. Detailed instructions on creating a Server Storage connection for tenant accounts, specifically for accessing subfolders, can be found in the documentation. The main account (non-tenant account) also has its own Home folder.

When to use this connector

Use this connector to create Flows that work with files in the server storage.

Create a Connection

Step 1. In the Connections window, click + and type in server storage.

Step 2. Select Server Storage.

Step 3. Enter Connection parameters.

Connection parameters

Directory: the directory(s) containing the files. This parameter can include subfolders. Example: /inbound/data. The parameter is optional. Note that Directory field is disabled for some of Etlworks account types.
Files: the actual file name or a wildcard file name, for example, *.csv. If the Directory field is disabled, you can specify the subfolders in the Files field. Example: /inbound/data/*.csv. In this case, the connection will be looking for files under home/inbound/data/.
Add Suffix When Creating Files in Transformation: you can select one of the predefined suffixes for the files created using this Connection. For example, if you select uuid as a suffix and the original file name is dest.csv, Etlworks will create files with the name dest_uuid.csv, where uuid is a globally unique identifier, such as 21EC2020-3AEA-4069-A2DD-08002B30309D.

This parameter works only when the file is created using source-to-destination-transformation. Read how to add a suffix to the files created when copying, moving, renaming, and zipping files.

File Processing Order: Specifies the order in which source files are processed when using wildcard patterns in ETL and file-based flows (e.g., copy, move, delete). The default setting is Oldest, meaning files are processed starting with the oldest by creation or modification time. Choose from various criteria such as file age, size, or name to determine the processing sequence:
- Disabled: wildcard processing is disabled,
- Oldest/Newest: Process files based on their creation or modification time, Ascending/Descending: Process files in alphabetical order, Largest/Smallest: Process files based on their size.

Contains CDC events:When this parameter is enabled, Etlworks adds standard wildcard templates for CDC files to the list of available sources in the FROM selector.

Working with Subfolders

This connection can be configured to work with subfolders.

Connection under main account

If a connection is created under the main account, you can easily specify which directory the connection should point to. Simply enter the desired folders in the Directory field to direct the connection to a specific location. This allows the main account to manage storage access and configuration centrally, ensuring that sub-accounts (tenants) adhere to the defined directory structure.

Connection under tenant

If a connection is created under a tenant, the application is restricted to the Home folder assigned to that specific tenant and all subfolders within it. To point the connection to subfolders within the tenant’s Home folder, use the File(s) field to specify the desired subdirectory. This ensures that tenants can only access their allocated storage areas, maintaining separation and security between different tenants’ data.

Create folders

To create folders under an existing directory, use the Create Folders flow type. For example, if you have a Server Storage connection that points to the Home folder, you can create subfolders within this Home folder. Simply select the Server Storage connection as the From Connection and specify the subfolders you wish to create in the FROM field. This process allows for easy and organized folder creation within your existing storage structure.

Encryption

When Server Storage Connection is used as a destination (TO) in the source-to-destination transformation, it is possible to configure the automatic encryption of the file created by the transformation using the PGP algorithm and public key uploaded to the secure key storage.

If the public key is available, all files created by the transformation will be automatically encrypted using the PGP algorithm and given key.

Additionally, you can enable (default) or disable propertyASCII Armored. Encrypted files may be saved in a Format referred to asASCII Armored. This Format is an encrypted representation of a file consisting entirely of printable ASCII (or text-mode only) characters. Files in this Format contain no binary values, and therefore may be easily sent as part of e-mail messages and visually examined. Files saved in this Format are approximately 30% larger than their non-armored counterparts. When decrypted, both armored and non-armored files reproduce an identical original.

Read how to generate a pair of public/private keys.

Decryption

When Server Storage Connection is used as a source (FROM) in the source-to-destination transformation, it is possible to configure the automatic decryption of the encrypted source files using the PGP algorithm and private key uploaded to the secure key storage.

If the private key is available, all source files processed by the transformation will be automatically decrypted using the PGP algorithm and given key. Note that the private key requires a password.