MongoDB connectors

When to use MongoDB connectors

MongoDB is a document database with the scalability and flexibility that you want and the querying and indexing that you need.

Etlworks can read and write data stored in MongoDB. 

• Read how to extract and load data using Etlworks MongoDB connector.
• Read how to work with documents in MongoDB.
• Read how to set up a Flow to capture changes in MongoDB database using Etlworks MongoDB CDC connector.

MongoDB connectors

MongoDB connector

Use this connector when working with large collections that contain documents with the same structure.

• When loading data into MongoDB, this connector creates a new MongoDB document for each record in the source data set.
• When extracting data from MongoDB, this connector streams MongoDB documents to the destination one by one.  

MongoDB document connector

Use it when you need to work with a specific document or filter documents in the collection.

• When loading data into MongoDB, this connector creates a single MongoDB document, which includes all the records in the source data set. 
• When extracting data from MongoDB, this connector reads all the documents in the collection that match the given filter, creates a JSON array, and sends it to the destination. 

Change Data Capture MongoDB connector

Etlworks MongoDB CDC Connector can monitor a MongoDB replica set or a MongoDB sharded cluster for document changes in databases and collections and send those changes as events to the destination.

Read how to create a MongoDB CDC Connection.

Create a Connection

Create a MongoDB Connection in the Connections window by clicking +, typing in mongo, and selecting either MongoDB or MongoDB document as the Connection type. 

Connection parameters

• URL: the required Connection string. Read about the MongoDB Connection string.
• Database: the required MongoDB database name.
• Collection: the required collection name, which belongs to a specific database.
• User: optional MongoDB username.
• Password: optional MongoDB password.
• What to do with the Object ID (_id) when reading: using this parameter, you can configure how the system will work with the MongoDB object id (_id field) when reading the document. The following options are available:
  • remove (default): the system will remove the _id field from the data set.
  • keep: the system will keep the _id field unmodified.
  • flatten: the system will flatten the value of the _id field.  Example: "_id": ObjectId("54759eb3c090d83494e2d804") to "_id": "54759eb3c090d83494e2d804".
• What to do with the existing document when writing: using this parameter, you can configure the MongoDB connector to either:
  • replace: replace the existing document with all the fields.
  • update: update only the fields in the existing document passed in the payload and leave other fields unchanged.   
• Number of Documents in Explorer: the maximum number of MongoDB documents that can be displayed in the Etlworks Explorer. The default value is 1000; the maximum possible value is 9999. A MongoDB can contain millions of documents, so displaying all of them might kill the server.

This parameter is used only by the Etlworks Explorer and has no effect when reading MongoDB documents while executing data integration Flows.

• Filter: a wildcard or a JSON filter to retrieve documents from MongoDB.

Options:

1. A fully qualified document name. The document id as it is stored in MongoDB.

2. A wildcard document name, for example, sales_orders*. The system will read all the documents which match the wildcard.

3. A request to MongoDB in the JSON Format. The system will read all the documents which match the request. An example is the request: {"first":"Simba"}.

This parameter is deprecated. Read how to use SQL with MongoDB.

• Encoding: additional encoding, which can be applied when creating a MongoDB document.

Connect to a MongoDB over SSH tunnel

A MongoDB database that sits behind a firewall cannot be accessed directly from a client on the other side of the firewall, but it can often be accessed through an SSH tunnel. The firewall must be configured to accept SSH Connections, and you also need to have an account on the SSH host for this to work.

Use an actual MongoDB database hostname and port when configuring the URL for the database Connection. Etlworks will automatically replace them with the localhost and the available port.

If you need to access a MongoDB database that can only be accessed via an SSH tunnel, you need to specify additional information in the Connect over SSH tunnel section of the database Connection screen.

• SSH Host: the name or IP address for the host accepting SSH Connections.
• SSH Port: the port accepting SSH Connections. The default value is 22.
• SSH User: the user name.
• SSH Password: the optional password.
• Private Key File: the private key file in the pem or ppk Format used for SSH authentication. Click the Upload SSH key to manage SSH keys using GUI. You can also upload the private key file manually and use the token {app.data} as a part of the filename, for example, {app.data}/keys/secret.pem. This parameter is optional.

• SSH Passphrase: the optional passphrase (password) used together with a private key file.