Etlworks uses Snowflake's COPY INTO command to load data. COPY INTO requires a named internal or external stage — the location where files live before they're loaded. Etlworks flows optimized for Snowflake can create the stage for you on first run.
Note: GCS stages aren't auto-created. Etlworks can auto-create internal stages and external stages in AWS S3 and Azure Blob. For Google Cloud Storage, create the stage manually first.
How to enable auto-creation
- Set Stage name. You can set it on the Snowflake connection, or per transformation when configuring a load flow. The transformation setting overrides the connection.
Note: The flow cannot run if Stage name is empty in both places.
- Make sure Check if stage exists is enabled on the Snowflake connection. It's on by default.
How does it work?
When Check if stage exists is on, the flow does two extra steps before loading:
- Verifies that a stage with the given name and parameters exists. The parameter check compares: stage type (internal or external), location (bucket or blob), and credentials.
- If the stage doesn't exist (or exists with different parameters), creates it.
Why keep this on. Snowflake's COPY INTO won't fail if the stage exists but is misconfigured (e.g. points at the wrong bucket) — it just silently loads nothing. The pre-flight check catches that.
Edge cases and overrides
Create the stage manually for slightly better performance
If you can guarantee the stage is set up correctly, create it manually and disable Check if stage exists on the connection to skip the pre-flight check.
Google Cloud Storage stages
Auto-creation isn't supported for GCS. Create the GCS stage manually.
Ignore errors when stage creation fails
Auto-creation can fail (missing privileges, conflicting stage parameters, …). If you're providing your own COPY INTO SQL via the flow's load-SQL field, you may want the flow to continue past the stage failure. Disable Fail load if flow cannot create stage on the Snowflake connection.
Create a stage manually
Snowflake's reference docs for each stage type: