About Server storage
Server Storage refers to file storage located on a local or network storage device (e.g., a hard drive, SSD, or NAS) connected to the host machine running Etlworks.
For users on Enterprise plans or on-premise deployments, the main account can access any files stored on local or network-attached storage devices that are connected to the machine running Etlworks. However, tenant accounts are restricted to their designated Home folder and its subfolders, ensuring proper data separation and security. If you are using Etlworks in the cloud, file access is similarly limited to the Home folder and its subdirectories, maintaining controlled access to the allocated storage space.
Home folder
When you sign up for Etlworks, a dedicated storage volume is created on Etlworks servers, which is accessible exclusively to your account. This is referred to as the Home folder. All data stored in this folder and its subdirectories can be accessed using the Server Storage connection type. To reference the Home folder, you can use the token {app.data} in your configurations.
For Enterprise and On-premise plans, Etlworks users can create tenant sub-accounts. Each tenant has its own isolated Home folder. These tenant accounts are limited to accessing only the Home folder and its subfolders, ensuring data isolation between accounts. Detailed instructions on creating a Server Storage connection for tenant accounts, specifically for accessing subfolders, can be found in the documentation. The main account (non-tenant account) also has its own Home folder.
When to use this connector
Use this connector to create Flows that work with files in the server storage.
Create a Connection
Step 1. In the Connections window, click + and type in server storage.
Step 2. Select Server Storage.
Step 3. Enter Connection parameters.
Connection parameters
- Directory: the directory(s) containing the files. This parameter can include subfolders. Example: /inbound/data. The parameter is optional. Note that Directory field is disabled for some of Etlworks account types.
- Files: the actual file name or a wildcard file name, for example, *.csv. If the Directory field is disabled, you can specify the subfolders in the Files field. Example: /inbound/data/*.csv. In this case, the connection will be looking for files under home/inbound/data/.
- Add Suffix When Creating Files in Transformation: you can select one of the predefined suffixes for the files created using this Connection. For example, if you select uuid as a suffix and the original file name is dest.csv, Etlworks will create files with the name dest_uuid.csv, where uuid is a globally unique identifier, such as 21EC2020-3AEA-4069-A2DD-08002B30309D.
Note: This parameter works only when the file is created using source-to-destination-transformation. Read how to add a suffix to the files created when copying, moving, renaming, and zipping files.
-
File Processing Order: Specifies the order in which source files are processed when using wildcard patterns in ETL and file-based flows (e.g., copy, move, delete). The default setting is Oldest, meaning files are processed starting with the oldest by creation or modification time. Choose from various criteria such as file age, size, or name to determine the processing sequence:
- Disabled: wildcard processing is disabled,
- Oldest/Newest: Process files based on their creation or modification time, Ascending/Descending: Process files in alphabetical order, Largest/Smallest: Process files based on their size.
- Contains CDC events:When this parameter is enabled, Etlworks adds standard wildcard templates for CDC files to the list of available sources in the FROM selector.
Working with Subfolders
This connection can be configured to work with subfolders.
Connection under main account
If a connection is created under the main account, you can easily specify which directory the connection should point to. Simply enter the desired folders in the Directory field to direct the connection to a specific location. This allows the main account to manage storage access and configuration centrally, ensuring that sub-accounts (tenants) adhere to the defined directory structure.
Connection under tenant
If a connection is created under a tenant, the application is restricted to the Home folder assigned to that specific tenant and all subfolders within it. To point the connection to subfolders within the tenant’s Home folder, use the File(s) field to specify the desired subdirectory. This ensures that tenants can only access their allocated storage areas, maintaining separation and security between different tenants’ data.
Create folders
To create folders under an existing directory, use the Create Folders flow type. For example, if you have a Server Storage connection that points to the Home folder, you can create subfolders within this Home folder. Simply select the Server Storage connection as the From Connection and specify the subfolders you wish to create in the FROM field. This process allows for easy and organized folder creation within your existing storage structure.
Encryption
When Server Storage Connection is used as a destination (TO) in the source-to-destination transformation, it is possible to configure the automatic encryption of the file created by the transformation using the PGP algorithm and public key uploaded to the secure key storage.
If the public key is available, all files created by the transformation will be automatically encrypted using the PGP algorithm and given key.
Additionally, you can enable (default) or disable propertyASCII Armored. Encrypted files may be saved in a Format referred to asASCII Armored. This Format is an encrypted representation of a file consisting entirely of printable ASCII (or text-mode only) characters. Files in this Format contain no binary values, and therefore may be easily sent as part of e-mail messages and visually examined. Files saved in this Format are approximately 30% larger than their non-armored counterparts. When decrypted, both armored and non-armored files reproduce an identical original.
Read how to generate a pair of public/private keys.
Decryption
When Server Storage Connection is used as a source (FROM) in the source-to-destination transformation, it is possible to configure the automatic decryption of the encrypted source files using the PGP algorithm and private key uploaded to the secure key storage.
If the private key is available, all source files processed by the transformation will be automatically decrypted using the PGP algorithm and given key. Note that the private key requires a password.
Read how to generate a pair of public/private keys.
Expected Compression
The Expected Compression setting allows you to specify the compression format expected when reading individual files from a connection. Supported options include No Compression, Zip, and GZip.
If set to Zip or GZip, Etlworks will automatically decompress each resource as it’s read. This setting assumes that each compressed file contains a single resource; it does not support archives with multiple files. Use this setting with caution, as the system will always attempt to decompress the resource based on the selected format, regardless of its file extension or actual content.