Use templates to define fields

Overview

When Etlworks connects to the data sources, it samples the data and automatically discovers the fields in the dataset. The fields can be used in a Mapping editor for per-field Mapping. It is, however, not always possible to connect to the data source during the design stage. 

Example

A good example of when connecting to the data source during the design stage is impossible is this — when the data source is a POST or PUT endpoint returning a response after updating the database. 

Also, the automatic sampling is not always 100% accurate as it drills down to only one record for each nested data set; thus, it can miss some of the elements which appear in some of the records.   

Process

In such cases, when configuring the file Format which is going to be used for the source or the destination, use the Template field to specify columns and data types. 

For example:

{
  "HostName": "string",
  "DomainName": "string",
  "FullyQualifiedDomainName": "string",
  "Description": "string"
}  

For the template, you must use the same file Format as in the designated file: JSON for JSON files, CSV for CSV files, etc. Etlworks is actually parsing the template using the parameters configured for that Format.