Skip to main content

Building a Change Data Capture Microservice with Meroxa and Vercel

· 4 min read
@anaptfox
Developer Advocate

In this guide, you will learn how to create a Change Data Capture pipeline in Meroxa that triggers a microservice in Vercel when changes occur to a table in PostgreSQL.

Change Data Capture Microservice

Change Data Capture is an architecture pattern that provides the ability to capture changes from a database (PostgreSQL, MongoDB, etc.) and build services that respond to those events.

This gives rise to an event-based architectures that can lead to some pretty interesting use cases such as:

  • Extract Transform Load (ETL)
  • Cache building/expiring
  • Search Indexing
  • Data Replication
  • Data Integration
  • Real-time UIs
  • Alerting
  • and so much more.

Being able to easily subscribe to and act on data changes offers some great possibilities.

Step One: Create the Microservice#

First, let's create a new folder called new-service. In that folder, we can create a new function in index.js within a folder called api.

mkdir -p new-service/api && touch new-service/api/index.js

Within index.js we can paste the following contents:

export default async function handle(req, res) {    if(!req.body) return res.json({ success: true })
    const { schema, payload } = req.body;    const { before, after, op } = payload;
    console.log("Current Data Schema:")    console.log(schema, null, 2);      switch (op) {        case 'r':            console.log("SNAPSHOT:")            console.log(after, null, 2);            break;        case 'c':            console.log("INSERT:")            console.log(after, null, 2);            break;        case 'u':            console.log("UPDATE:")            console.log("Before the update:")            console.log(before, null, 2);            console.log("After the update:")            console.log(after, null, 2);            console.log('u');            break;        case 'd':            console.log("DELETE:")            console.log(before, null, 2);            break;
        default:            break;    }
    return res.json({ success: true })
}

To learn more, see Serverless Functions in Vercel.

This function will simply console.log information from the CDC event from Meroxa. The structure of the event is as followed:

CDC Event

Step Two: Deploy to Vercel#

Now, that we have a function that handles our CDC event, we can deploy it:

vercel --prod

Once deployed, you will get a URL to see your site live, such as the following: https://new-service.vercel.app/

Since our function is within the api folder, the function will be available at: /api

Now, we can build our pipeline to execute this function.

Step Three: Download Meroxa CLI#

The Meroxa CLI makes it easy to build pipelines within your terminal. You may also use the Meroxa Dashboard to build pipelines.

First, download the Meroxa CLI:

brew tap meroxa/taps && brew install meroxa

Homebrew is for macOS/Linux, for more installation instructions see: CLI Installation Guide.

Next, create a Meroxa account. This can be done from the CLI:

meroxa login

Step Four: Add Resource Meroxa#

Resources represent the source and destination of your data. For this example, you will create two Resources in Meroxa:

  • Source: Postgres
  • Destination: Microservice

After creating these Resources, we can then use them in Meroxa pipelines.

First, we can add a Postgres resource to the Meroxa Resource Catalog as the source of our data.

Before you can add, the PostgreSQL instance needs to be accessible by Meroxa. You may:

Then, you can add the resource using the CLI:

meroxa add resource postgres --type postgres -u postgres://$PG_USER:$PG_PASS@$PG_URL:$PG_PORT/$PG_DB

For more details, see: Postgres (Logical Replication).

Next, we can add our microservce as a Webhook resource:

meroxa add resource service --type url -u $VERCEL_URL

$VERCEL_URL is the URL of your deployed serverless function. It will look like the following: https://new-service.vercel.app/api

Step Four: Create Pipeline#

Now, that we have resources, we can create a new pipeline:

meroxa pipeline create cdc-microservice

Lastly, using the meroxa connect command,

meroxa connect --from postgres --input $TABLE_NAME --to service --pipeline cdc-microservice

$TABLE_NAME is the name of the table within PostgreSQL you'd like to capture changes from. This option takes the format of: <schema>.<table>(e.g., public.User).

You can view the status of things with the meroxa connector list command:

Meroca List Connectors

Once your pipeline is up and running, any time you perform an INSERT, UPDATE or DELETE operation in PostgreSQL, an event

Vercel Function Logs

What's Next?#

Now, your next step is to update your microservice to solve the problem you'd like to solve. Here are some examples for you:

A great next step is to browse other sources and destinations to extend your existing pipeline or create new ones.

As always: