Articles in this section

See more

Using JavaScript to load text and binary data from any connection

  • Updated
Follow

Use Cases

One of the typical use cases is when you need to load the data from any Connection using JavaScript. For example:

  • Load a JSON from a web service, parse it in JavaScript, and use the fields from the JSON in transformation.
  • Load an image, scale it, and save it somewhere else.
  • Load a file, calculate the MD hash, and store the hash somewhere so it could be used later to check if the file has been already processed. 

Load text data from a Connection

In the examples below, we assume that there is an HTTP endpoint that returns a JSON with a list of images. We want to call the endpoint and save the response to the images variable. The first example uses a named Connection, the second one creates a Connection on the fly.

Load data from a named Connection

var images = com.toolsverse.etl.core.task.common.FileManagerTask.read(etlConfig, 
connection_name, null);

Load data from a dynamically created Connection

var alias = new com.toolsverse.etl.common.Alias();

alias.setUrl(the url);

alias.setTransport('http');

var images = com.toolsverse.etl.core.task.common.FileManagerTask.read(alias, null);

Note: The last parameter in the FileManagerTask.read is a filename, which doesn't make sense for the HTTP Connection (hence the null) but can be used with any other Connection type.

Load binary data from a connection

In the examples below, we load the PNG image from the HTTP endpoint, scale it and save it to the server storage. Method FileManagerTask.readAsStream(...) returns java.io.InputStream. 

var imageStream = null;
var imageStreamToSave = null;
var os = null;

try {
   imageStream = com.toolsverse.etl.core.task.common.
FileManagerTask.readAsStream(etlConfig, 'image', null);

   var image = com.toolsverse.imaging.ImageProcessor.instance().loadImage(imageStream);
  
   var newImage = com.toolsverse.imaging.ImageProcessor.instance().scale(image, 
Java.to([0.5, 0.5], "java.lang.Float[]"));
  
   var os = new java.io.ByteArrayOutputStream();
   javax.imageio.ImageIO.write(newImage, "png", os);
   imageStreamToSave = new java.io.ByteArrayInputStream(os.toByteArray());
  
   var alias = new com.toolsverse.etl.common.Alias();
   alias.setUrl("logo.png");
   alias.setTransport('file'); 
  
   com.toolsverse.etl.core.task.common.
FileManagerTask.write(alias, null, imageStreamToSave);
  
} finally {
   if (imageStream != null) { 
       imageStream.close();
   }  
  
   if (os != null) {
       os.close(); 
   }  
  
   if (imageStreamToSave != null) {
       imageStreamToSave.close(); 
   }  
}  
}

Related articles