Data Formats

Records on the Conduit platform are formatted with JSON. Data records can take many shapes, depending on the data, the originating data system, and any configurations defined in your Conduit application.

Because real-time change data capture (CDC) is a default of the Conduit Platform, we leverage the openCDC format.

Records will typically come in the following format:

{
    "position": "..."
    "operation": "create"
    "metadata": { 
        "conduit.source.connector.id": "..."
        "opencdc.readAt": "..."
        "opencdc.version": "..."
     }
    "key": { 
        "id": "..."
        "replayId": "..."
     }
    "payload": {
        "before": NULL
        "after": { <event data> }
    }
}

The payload is defined by the source from where you are pulling events or records. Depending on the configuration of your source connector and the capabilities of the underlying data store, you may see variability between a record or an event structure. Events are created as changes are captured through Change Data Capture (CDC)—the payload will include a before and after within the payload depicting what changes took place from before and after.