This is the most commonly used flow type. Almost all available connectors can be used as a source, as well as a destination.
When to use this flow type
Eltworks Integrator extracts data from the source connection, transforms it by applying one or multiple transformations, and then automatically loads it in the destination.
The following variations of this flow are available:
- The source is a relational database.
- The destination is a relational database.
- The source is a file, API endpoint, NoSQL database or inbound email connection
- The destination is a file, API endpoint, NoSQL database or outbound email connection
- The source is a well-known API
- The destination is a well-known API
- Any To Any ETL
This transformation modifies the source data before loading it in the destination. Use copy, move, rename, delete, zip, and unzip files if all you need is to copy, move, rename or delete the files or perform the bulk operations with the data without modifying it.
The source is a relational database
When the source is a relational database, the Integrator extracts data from the database table, view, or synonym or executes a user-defined SQL query.
The user has two options:
- Use the From field to select or enter a database object name, such as table, view or synonym. In this case, Integrator will automatically execute the SQL query
select * from object
. Note that the fully qualified wildcard names are supported. Example:public.*
. In this case, the system will extract the data from all database objects which match then wildcard namepublic.*
and load into the destination. - Enter the actual
SELECT
SQL statement in the Source Query field.
Source query is a script
When the source connection is a relational database that supports stored procedures and anonymous SQL blocks it is possible to run SQL scripts (with variables, conditions, loops, etc.) as a part of the Source Query. The only condition is that the script must return a cursor.
Enter the script in Source Query and enable the Execute as script parameter.
Creating flow when the source is a relational database
To create a flow, go to Flows window, click the +
button and type in database to
. Select one of the available flows types and add one or multiple source-to-destination transformations.
Available transformations when the source is a relational database
- SQL
- Source-to-destination
- Per-field mapping
- Validation
- Remove duplicates
- Scripting transformations
- Transpose
- Wrap columns
- Denormalize (pivot)
- Normalize as key-value pairs
- Denormalize key-value pairs
Available parameters when the source is a relational database
- The Transformation name
- Linked Source Name
- Stream Data
- Parallel
- Ignore when there are no records
- Change replication
- Handling exceptions
The destination is a relational database
When loading data into a database, the Integrator either automatically generates and executes the SQL DML statement (INSERT, UPDATE, DELETE or MERGE) or executes user-created SQL statements.
When the destination is a relational database, the user has two options:
- Use the To field to select or enter the database object name (table, view, synonym). If the source (FROM) is configured with a wildcard, you can use
*
as a part of the destination name (TO). The system will substitute the*
with the source name at run time. - Or enter the actual
INSERT
,UPDATE
, orDELETE
SQL statement in the Destination query field.
The destination query can be more than just one SQL statement. Use ;
to separate SQL statements.
In the example below, 2 queries will be executed. Fields within the {
and }
brackets will be replaced with the actual field values given by the source.
Source query:
SELECT locust_staging.id,
locust_update_id.hotlineeventid,
recorded_time AS calldatetime,
CASE
WHEN date_completed IS NULL
THEN 2
ELSE 1
END AS eventstatusid,
FROM locust_staging
Destination query:
update hotlineevent
set calldatetime = {calldatetime},
where hotlineeventid = {hotlineeventid};
update locust_staging
set HotlineEventId = {hotlineeventid}
where id = {id};
Creating flow when the destination is a relational database
To create a flow, go to Flows window, click the +
button and type in to database
. Select one of the available flows types and add one or multiple source-to-destination transformations.
Available transformations when the destination is a relational database
- SQL
- Source-to-destination
- Per-field mapping
- Filter (only if the source is not database)
- Validation
- Remove duplicates
- Scripting transformations
- Extract dimensions (only if the source is not database)
- Transpose
- Wrap columns
- Denormalize (pivot)
- Normalize as key-value pairs
- Denormalize key-value pairs
Available parameters when the destination is a relational database
- The Transformation name
- Linked Source Name
- Stream Data
- Use Bind Variables
- Parallel
- Ignore when there are no records
- Use Native SQL
- Ignore when there are no records
- Create Tables and Adjust Fields
- Create Indexes
- Before SQL
- After SQL
- SQL Action
- Change replication
- Handling exceptions
The source is a file, API endpoint, NoSQL database or inbound email connection
The source can be a flat, structured or semi-structured file, a response from the API endpoint, a document in the NoSQL database or an email body or attachment.
Etlworks Integrator parses the source document and transforms it into the internal data structure, then converts it to the appropriate format and sends it to the destination.
Here is an example of the flow which extracts data from the web service, transforms it and loads into the database.
When creating the transformation, define the source by selecting a source (FROM) connection and a source (FROM) format. Then select or enter a source (FROM) object name:
- If the source connection is file storage (server or cloud), the source object name is an actual file name or a wildcard file name.
- If the source connection is a web service, the source object name is an any appropriate name. For example, patient.json.
- If the source connection is an outbound email connection, the source object name is either the name of the attached file or any appropriate name.
- If the source connection is in a key-value storage, such as Redis, the name is a "key" (in a key-value pair).
- If the source connection is in a NoSQL database, such as MongoDB, the name is a document id.
Using SQL to transform data
Similar to when the source is a relational database, the user has an option to enter an SQL query, which will be executed after the data has been read from the source. For example, if the source is a CSV file, the user can write a query that aggregates the data into a file and creates a summary view. This view will be used to drive the destination data.
Read more about SQL transformation.
Working with wildcard file names
If your source data object is a file, it is possible to configure a source (FROM) so that it will read the files by the wildcard name, for example, order_*.csv
.
Read how to work with wildcard file names and how to process all files by the wildcard.
Creating flow when the source is a file
To create a flow go to Flows window, click the +
button.
- If the source is a file, inbound email connection, Redis or MongoDB type in
file to
. - If the source is a web service type in
web service to
.
Select one of the available flow types and add one or multiple source-to-destination transformations.
Available transformations when the source is a file
- SQL
- Source-to-destination
- Per-field mapping
- Filter
- Order-by
- Validation
- Remove duplicates
- Scripting transformations
- Extract dimensions
- Transpose
- Wrap columns
- Denormalize (pivot)
- Normalize as key-value pairs
- Denormalize key-value pairs
Available parameters when the source is a file
- The Transformation name
- Linked Source Name
- Stream Data
- Force Streaming
- Parallel
- Ignore when there are no records
- Ignore when there is no file
- Process all files
- Delete loaded sources files
- Delete source files on error
- Change replication
- Handling exceptions
The destination is a file, API endpoint, NoSQL database or outbound email connection
The destination can be a flat, structured or semi-structured file, a payload for the API endpoint, a document in the NoSQL database or an email body or attachment.
Etlworks Integrator creates the destination document from the source using the appropriate data exchange format and by applying transformations and mapping.
When creating the transformation define the destination by selecting a destination (TO) connection and a destination (TO) format. Then select or enter a destination (TO) object name:
- If the destination connection is in file storage (server or cloud), the destination object name is an actual file name. Read how Etlworks Integrator calculates the destination file name.
- If the destination connection is a web service, the destination object name is any appropriate name. For example, patient.json.
- If the destination connection is an outbound email connection, the destination object name is either the name of the attached file or any appropriate name.
- If the destination connection is a key-value storage, such as Redis, the name is any appropriate name.
- If the destination connection is a NoSQL database, such as MongoDB, the name is document id.
When the destination connection is in file storage - the new file will be created each time the transformation is executed. If the file already exists, it will be overwritten.
Creating flow when the destination is a file
To create a flow, go to Flows window, click the +
button.
- if the destination is a file, outbound email connection, Redis or MongoDB, type in
to file
. - if the destination is web service, type in
to web service
.
Select one of the available flow types and add one or multiple source-to-destination transformations.
Available transformations when the destination is a file
- SQL
- Source-to-destination
- Per-field mapping
- Filter
- Order-by
- Validation
- Remove duplicates
- Scripting transformations
- Extract dimensions
- Transpose
- Wrap columns
- Denormalize (pivot)
- Normalize as key-value pairs
- Denormalize key-value pairs
Available parameters when the destination is a file
- The Transformation name
- Linked Source Name
- Stream Data
- Force Streaming
- Parallel
- Ignore when there are no records
- Ignore when there is no file
- Process all files
- Change replication
- Handling exceptions
The source is a well-known API
A well-known API is a business application or API for which Etlworks has a native connector, for example, Marketo, Salesforce, etc.
Here is an example of the flow which extracts data from the well-known API, transforms it and loads into the database.
Creating flow when the source is a well-known API
To create a flow, go to Flows window, click the +
button, select Well-known API to
and select the flow which extracts data from the well-known API.
Add one or multiple source-to-destination transformations.
Available transformations when the source is a well-known API
- SQL
- Source-to-destination
- Per-field mapping
- Filter
- Order-by
- Validation
- Remove duplicates
- Scripting transformations
- Extract dimensions
- Transpose
- Wrap columns
- Denormalize (pivot)
- Normalize as key-value pairs
- Denormalize key-value pairs
Available parameters when the source is a well-known API
- The Transformation name
- Linked Source Name
- Stream Data
- Force Streaming
- Parallel
- Ignore when there are no records
- Ignore when there is no file
- Process all files
- Change replication
- Handling exceptions
The destination is a well-known API
A well-known API is a business application or API for which Etlworks has a native connector, for example, Marketo, Salesforce, etc.
Creating flow when the destination is a well-known API
To create a flow go to Flows window, click the +
button, select to Well-known API
and select the flow which loads data into the well-known API.
Add one or multiple source-to-destination transformations.
Available transformations when the destination is a well-known API
- SQL
- Source-to-destination
- Per-field mapping
- Filter
- Order-by
- Validation
- Remove duplicates
- Scripting transformations
- Extract dimensions
- Transpose
- Wrap columns
- Denormalize (pivot)
- Normalize as key-value pairs
- Denormalize key-value pairs
Comments
0 comments
Please sign in to leave a comment.