Articles in this section

See more

Web/HTML scraping with JavaScript and Python

  • Updated
Follow

Overview

For Web/HTML scraping, Etlworks includes a Java library jsoup. Jsoup is one of the best HTML parsers around. It implements the WHATWG HTML5 specification and parses HTML to the same DOM as modern browsers do.

By calling the jsoup methods from the JavaScript and Python code, you can parse the webpage or HTML string and transform it into the DOM model, then traverse the DOM and find the required elements.

The examples below load the web page from https://www.eia.gov/analysis/projection-data.php into the string variable, parse the HTML, extract the links to the Excel documents and link titles, and store them into the variable for further use by other Flows.

JavaScript example

var html = com.toolsverse.etl.core.task.common.FileManagerTask.
read(etlConfig, 'html', null);

doc = org.jsoup.Jsoup.parse(html);

files = new java.util.ArrayList();

etlConfig.setValue('files', files);

root = doc.select("h3 > a[name=annualproj]").get(0).parent().
nextElementSibling();

spans = root.select("ul.numbered li > span.formats");

for each (var span in spans) {
 link = span.nextElementSibling();
 href = link.attr("href").toString();
 title = link.attr("title").toString();
 title = title.substring(title.indexOf('.') + 1).trim();
 
 files.add(new com.toolsverse.util.TypedKeyValue(href, title));
 
 // log
 etlConfig.log("href: " + href + ", title: " + title);
}

Python example

Note: this example targets built-in Python on the Etlworks 10.x runtime (Python 3.13). On that runtime, Java classes are loaded with java.type(...) instead of the Jython-style imports used in 9.x. See Use Python for the migration checklist.

import java

Jsoup = java.type("org.jsoup.Jsoup")
FileManagerTask = java.type("com.toolsverse.etl.core.task.common.FileManagerTask")
TypedKeyValue = java.type("com.toolsverse.util.TypedKeyValue")
ArrayList = java.type("java.util.ArrayList")

html = FileManagerTask.read(etlConfig, "html", None)

doc = Jsoup.parse(html)

files = ArrayList()

etlConfig.setValue("files", files)

root = doc.select("h3 > a[name=annualproj]").get(0).parent().nextElementSibling()

spans = root.select("ul.numbered li > span.formats")

for index in range(spans.size()):
    span = spans.get(index)
    link = span.nextElementSibling()
    href = link.attr("href")
    title = link.attr("title")
    title = title[title.index(".") + 1:].strip()

    files.add(TypedKeyValue(href, title))
    etlConfig.log(f"href: {href}, title: {title}")

The Java classes used here (Jsoup and the Etlworks classes) must be available to the executing Etlworks runtime; java.type(...) does not download dependencies.

Related articles