Convert YAML document to JSON on the fly

Overview

In some cases, you may need to work with YAML documents in Etlworks, but currently, Etlworks doesn’t natively support the YAML format. However, you can easily convert a YAML document into JSON on the fly during the ETL process. This allows you to leverage YAML as a data source while continuing to use JSON for data transformations within Etlworks.

This guide provides a step-by-step instruction for converting a source YAML document into JSON in real-time.

Steps to Convert YAML to JSON in Etlworks

Step 1: Create a JSON Format

To start, you need to create a JSON format in Etlworks. This format will be used as the target format after the YAML-to-JSON conversion. Follow these steps to create the JSON format:

1. Navigate to Formats in the Etlworks UI.
2. Click Add Format and select JSON as the format type.
3. Configure the format as needed for your specific transformation and save it.

Step 2: Add a Preprocessor to Convert YAML to JSON

The key step in this process is using a preprocessor script to convert the incoming YAML data into JSON. You will need to add a preprocessor to your flow with the following code:

var Yaml = Java.type('com.github.javafaker.shaded.snakeyaml.Yaml');
var ObjectMapper = Java.type('com.fasterxml.jackson.databind.ObjectMapper');

// Create a new instance of the shaded Yaml parser
var yamlParser = new Yaml();

// Parse the YAML string into a Java Map
var yamlMap = yamlParser.load(message);

// Convert the Java Map to JSON using Jackson's ObjectMapper
var objectMapper = new ObjectMapper();

// Return JSON
value = objectMapper.writeValueAsString(yamlMap);

Explanation of the Preprocessor:

• Yaml: This uses the shaded version of SnakeYAML (bundled with JavaFaker) to parse the incoming YAML data.

• ObjectMapper: We use the Jackson ObjectMapper to convert the parsed YAML (in the form of a Java Map) into a JSON string.

• The message variable represents the incoming YAML document, and the result, value, is the converted JSON string.

Step 3: Use JSON Format in Transformations

Once you have the YAML converted to JSON via the preprocessor, you can use the previously created JSON format as the source format for any further transformations in your ETL flow.

1. In your transformation step, select the JSON format you created in Step 1 as the source format.
2. Continue with the transformation process as you normally would with a JSON input.