Skip to main content

MongoDB

The Conduit Platform by default supports MongoDB as a source and a destination.

The MongoDB source can connect to and emit records from a collection.

Required Configurations

NameDescriptionRequiredDefault
dbThe name of a database the connector must work with.Yes
collectionThe name of a collection the connector must read from.Yes

Looking for something else? See advanced configurations.

Initial Snapshot

Snapshot mode is enabled by default. When the source connector starts, it reads all rows of the collection in batches using cursor-based pagination, limiting rows by batchSize.

The source connector stores the last processed element value of an orderingColumn in a position, enabling the snapshot process to pause and resume without data loss. Once all rows in the initial snapshot are read, the connector transitions into CDC mode, where the CDC iterator captures changes occurring after that point.

Note: The default snapshot mode can be disabled by setting snapshot to false in the configuration.

Updates

The source connector utilizes Change Data Capture (CDC) to detect changes in a collection by reading from a Change Stream. Please note that for this to function correctly, your MongoDB instance must meet the criteria specified on the official website.

Each detected change is converted into a record and returned in the Read call. If no record is available when Read is invoked, the connector returns an sdk.ErrBackoffRetry error.

The connector stores a resumeToken of every Change Stream event in a position, enabling the CDC process to be resumed.

Warning: Azure CosmosDB for MongoDB offers very limited support for Change Streams, rendering them unsuitable for CDC purposes. In scenarios where CDC is not feasible, such as with CosmosDB, the connector only supports detecting INSERT operations by polling for new documents.

Key Handling

The connector always utilizes the _id field as a key. When transferring a record to a destination, if the _id field is of type bson.ObjectID, the connector converts it to a string; otherwise, it leaves it unchanged.

Advanced Configurations

NameDescriptionRequiredDefault
uriThe connection string. The URI can contain host names, IPv4/IPv6 literals, or an SRV record. Example: mongodb://localhost:27017No
auth.usernameThe username.No
auth.passwordThe user's password.No
auth.dbThe name of a database that contains the user's authentication data.No
auth.mechamismThe authentication mechanism. The available values are SCRAM-SHA-256, SCRAM-SHA-1, MONGODB-CR, MONGODB-AWS, or MONGODB-X509. This will depend on your MongoDB server version.No
auth.tls.caFileThe path to either a single or a bundle of certificate authorities to trust when making a TLS connection.No
auth.tls.certificateKeyFileThe path to the client certificate file or the client private key file.No
snapshotEnable or disable snapshot of entire table before starting CDC mode. Options: true or false.Notrue
orderingFieldThe name of a field that is used for ordering collection documents when capturing a snapshot.No_id
batchSizeSize of document batch. Min is 1 and max is 100000.No1000