Articles in this section

See more

SFTP connector

  • Updated
Follow

When to use this connector

Use this connector to create Flows that work with files in SFTP.

Create a Connection

Step 1. In the Connections window, click +, type in sftp. 

Step 2. Select either SFTP or SFTP2 connector.

Step 3. Enter Connection parameters.

Choosing Between SFTP and SFTP2 Connectors

When creating an SFTP connection, you have the option to choose between the SFTP and SFTP2 connectors. Both connectors offer the same functionality in terms of features and protocol handling. However, the SFTP2 connector, currently in beta, utilizes a newer non-blocking IO library, which can potentially provide faster performance, especially when writing large amounts of data to SFTP servers.

For most use cases, we recommend using the SFTP connector, as it is more mature and thoroughly tested. However, if performance is critical to your workflows, we suggest testing the SFTP2 connector. If your tests confirm a performance improvement for your specific use case, you may consider using it in production environments.

Connection parameters

  • URL: the hostname and optional port of the SFTP server.
  • Change current remote directory to: if this parameter is set, the connector will cd to this specific directory after connecting the SFTP site. It is useful if you want to access a folder from a level or two back from the home folder.  
  • Directory: the directory with files. This parameter is optional.
  • Files: the actual file name or a wildcard file name, for example, *.csv.
  • User: username.
  • Password: password.
  • Private Key File: private key file in the pem or ppk Format used for SSH authentication. Click Upload SSH key to manage SSH keys using GUI. You can also upload the Private Key File manually and use the token {app.data} as a part of the filename, for example, {app.data}/keys/secret.pem. This parameter is optional.

ssh_key.png

  • Passphrase: a passphrase (password) used together with a Private Key File. This parameter is optional.
  • Trust All Server Keys: Enables the automatic acceptance of all server keys during SFTP connections, bypassing manual verification. While convenient for testing or development environments, this option is not recommended for production deployments due to potential security risks, as it does not verify the authenticity of the server. 
  • Enable Alternative Upload: Activates an alternative upload mechanism for enhanced compatibility with specific SFTP servers. This parameter is available only for the regular SFTP connector (not SFTP2 which uses it by default). When enabled, the regular SFTP connector uses the same underlying library for file uploads as the SFTP2 connector, which can help resolve compatibility issues with certain server configurations and significantly improve the speed of the upload.
  • Other parameters: additional configuration options for the SFTP connection. 
  • Add Suffix When Creating Files in Transformation: you can select one of the predefined suffixes for the files created using this Connection. For example, if you select uuid as a suffix and the original file name is dest.csv, Etlworks will create files with the name dest_uuid.csv, where uuid is a globally unique identifier such as 21EC2020-3AEA-4069-A2DD-08002B30309D.

Note: This parameter works only when the file is created using source-to-destination-transformation. Read how to add a suffix to the files created when copying, moving, renaming, and zipping files.

  • File Processing Order: Specifies the order in which source files are processed when using wildcard patterns in ETL and file-based flows (e.g., copy, move, delete). The default setting is Oldest, meaning files are processed starting with the oldest by creation or modification time. Choose from various criteria such as file age, size, or name to determine the processing sequence:
    • Disabled: wildcard processing is disabled,
    • Oldest/Newest: Process files based on their creation or modification time, Ascending/Descending: Process files in alphabetical order, Largest/Smallest: Process files based on their size. 
  • Remove Leading Slash from Remote Filename: Enables removal of a leading / from the filename in cases where folder and filename tokens produce unintended double slashes (e.g., abc//file). Recommended for compatibility with certain SFTP servers.

Decryption

When SFTP Connection is used as a source (FROM) in the source-to-destination transformation, it is possible to configure the automatic decryption of the encrypted source files using the PGP algorithm and private key uploaded to the secure key storage.

If the private key is available, all source files processed by the transformation will be automatically decrypted using the PGP algorithm and given key.  Note that the private key requires a password. 

mceclip1.png

Read how to generate a pair of public/private keys.

Improving SFTP Performance

To achieve faster upload and download speeds, consider using the SFTP2 connector, which leverages a newer non-blocking I/O library. While SFTP2 can offer significant performance improvements, especially for large file transfers, it is still in beta, so we recommend thoroughly testing it in your environment before production use.

If you’re using the regular SFTP connector, enabling the Alternative Upload Mechanism can also boost upload speeds, potentially increasing performance by up to 10x in certain cases. This alternative mechanism is designed to enhance compatibility and efficiency with specific SFTP server configurations.

Handling Issues with Leading Slashes in Remote Filenames

In some cases, filenames with a leading / can cause compatibility issues with SFTP servers, especially when tokens are used in the file path (e.g., {folder}/{file}) and folders are defined with a trailing slash. The Remove Leading Slash option eliminates the leading slash from remote filenames to ensure proper path handling. This can be particularly useful for preventing errors when working with directory structures where an extra slash could result in an incorrect file path.

Expected Compression

The Expected Compression setting allows you to specify the compression format expected when reading individual files from a connection. Supported options include No Compression, Zip, and GZip.

If set to Zip or GZip, Etlworks will automatically decompress each resource as it’s read. This setting assumes that each compressed file contains a single resource; it does not support archives with multiple files. Use this setting with caution, as the system will always attempt to decompress the resource based on the selected format, regardless of its file extension or actual content.

Related articles