MongoDB
The Conduit Platform by default supports MongoDB as a source and a destination.
The MongoDB source can connect to and emit records from a collection.
Required Configurations
| Name | Description | Required | Default |
|---|---|---|---|
db | The name of a database the connector must work with. | Yes | |
collection | The name of a collection the connector must read from. | Yes |
Looking for something else? See advanced configurations.
Initial Snapshot
Snapshot mode is enabled by default. When the source connector starts, it reads all rows of the collection in batches using cursor-based pagination, limiting rows by batchSize.
The source connector stores the last processed element value of an orderingColumn in a position, enabling the snapshot process to pause and resume without data loss. Once all rows in the initial snapshot are read, the connector transitions into CDC mode, where the CDC iterator captures changes occurring after that point.
Note: The default snapshot mode can be disabled by setting snapshot to false in the configuration.
Updates
The source connector utilizes Change Data Capture (CDC) to detect changes in a collection by reading from a Change Stream. Please note that for this to function correctly, your MongoDB instance must meet the criteria specified on the official website.
Each detected change is converted into a record and returned in the Read call. If no record is available when Read is invoked, the connector returns an sdk.ErrBackoffRetry error.
The connector stores a resumeToken of every Change Stream event in a position, enabling the CDC process to be resumed.
Azure CosmosDB for MongoDB offers very limited support for Change Streams, rendering them unsuitable for CDC purposes. In scenarios where CDC is not feasible, such as with CosmosDB, the connector only supports detecting INSERT operations by polling for new documents.
Key Handling
The connector always utilizes the _id field as a key. When transferring a record to a destination, if the _id field is of type bson.ObjectID, the connector converts it to a string; otherwise, it leaves it unchanged.
Advanced Configurations
| Name | Description | Required | Default |
|---|---|---|---|
uri | The connection string. The URI can contain host names, IPv4/IPv6 literals, or an SRV record. Example: mongodb://localhost:27017 | No | |
auth.username | The username. | No | |
auth.password | The user's password. | No | |
auth.db | The name of a database that contains the user's authentication data. | No | |
auth.mechamism | The authentication mechanism. The available values are SCRAM-SHA-256, SCRAM-SHA-1, MONGODB-CR, MONGODB-AWS, or MONGODB-X509. This will depend on your MongoDB server version. | No | |
auth.tls.caFile | The path to either a single or a bundle of certificate authorities to trust when making a TLS connection. | No | |
auth.tls.certificateKeyFile | The path to the client certificate file or the client private key file. | No | |
snapshot | Enable or disable snapshot of entire table before starting CDC mode. Options: true or false. | No | true |
orderingField | The name of a field that is used for ordering collection documents when capturing a snapshot. | No | _id |
batchSize | Size of document batch. Min is 1 and max is 100000. | No | 1000 |