Using JavaScript to load text and binary data from any connection

Use Cases

One of the typical use cases is when you need to load the data from any Connection using JavaScript. For example:

• Load a JSON from a web service, parse it in JavaScript, and use the fields from the JSON in transformation.
• Load an image, scale it, and save it somewhere else.
• Load a file, calculate the MD hash, and store the hash somewhere so it could be used later to check if the file has been already processed. 

Load text data from a Connection

In the examples below, we assume that there is an HTTP endpoint that returns a JSON with a list of images. We want to call the endpoint and save the response to the images variable. The first example uses a named Connection, the second one creates a Connection on the fly.

Load data from a named Connection

var images = com.toolsverse.etl.core.task.common.FileManagerTask.read(etlConfig, 
connection_name, null);

Load data from a dynamically created Connection

var alias = new com.toolsverse.etl.common.Alias();

alias.setUrl(the url);

alias.setTransport('http');

var images = com.toolsverse.etl.core.task.common.FileManagerTask.read(alias, null);

The last parameter in the FileManagerTask.read is a filename, which doesn't make sense for the HTTP Connection (hence the null) but can be used with any other Connection type.

Load binary data from a connection

In the examples below, we load the PNG image from the HTTP endpoint, scale it and save it to the server storage. Method FileManagerTask.readAsStream(...) returns java.io.InputStream. 

var imageStream = null;
var imageStreamToSave = null;
var os = null;

try {
   imageStream = com.toolsverse.etl.core.task.common.
FileManagerTask.readAsStream(etlConfig, 'image', null);

   var image = com.toolsverse.imaging.ImageProcessor.instance().loadImage(imageStream);
  
   var newImage = com.toolsverse.imaging.ImageProcessor.instance().scale(image, 
Java.to([0.5, 0.5], "java.lang.Float[]"));
  
   var os = new java.io.ByteArrayOutputStream();
   javax.imageio.ImageIO.write(newImage, "png", os);
   imageStreamToSave = new java.io.ByteArrayInputStream(os.toByteArray());
  
   var alias = new com.toolsverse.etl.common.Alias();
   alias.setUrl("logo.png");
   alias.setTransport('file'); 
  
   com.toolsverse.etl.core.task.common.
FileManagerTask.write(alias, null, imageStreamToSave);
  
} finally {
   if (imageStream != null) { 
       imageStream.close();
   }  
  
   if (os != null) {
       os.close(); 
   }  
  
   if (imageStreamToSave != null) {
       imageStreamToSave.close(); 
   }  
}  
}