Meroxa

The Meroxa Developer Hub

Welcome to the Meroxa developer hub. You'll find comprehensive guides and documentation to help you start working with Meroxa as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started    Guides

PostgreSQL

PostgreSQL is a powerful, open-source object-relational database system. Meroxa will automatically discover the version, capabilities, and permission of the added PostgreSQL Resource and will create the appropriate connection to capture data changes.

Environments

We support the following PostgreSQL environments:

The Meroxa Platform has been tested against the following PostgreSQL versions:

  • 9.6.x
  • 10.x
  • 11.x
  • 12.x

Source Configuration

Meroxa supports two methods for capturing data changes from PostgreSQL:

  • Logical Replication (recommended).
  • Polling tables for new records.

Logical replication uses the pgoutput plugin and provides the best and most effective facility to capture changes as they occur.

In comparison, the polling method will scan the table for new records in regular intervals. To operate effectively every table must include an increment column that can be used for detection.

Logical Replication

PostgreSQL 10+ comes with a builtin pgoutput decoding plugin which provides the most effective way for data to be captured and streamed.

To add a PostgreSQL resource with Logical Replication enabled:

meroxa add resource postgresDB --type postgres -u postgres://$PG_USER:[email protected]$PG_URL:$PG_PORT/$PG_DB --metadata '{"logical_replication":"true"}'

The --metadata '{"logical_replication":"true"}' configuration enables logical replication support.

postgresDB is a human-friendly name to represent the resource within Meroxa. Feel free to change as desired.

In the command above, replace the following variables with valid credentials from your PostgreSQL environment:

  • $PG_USER - PostgreSQL Username
  • $PG_PASS - PostgreSQL Password
  • $PG_URL - PostgreSQL URL
  • $PG_DB - PostgreSQL Database Name
  • $PG_PORT - PostgreSQL Port (e.g., 5432).

Permissions

In order for logical replication to be used, the configured Resource credentials must have the appropriate permissions to create a replication slot. If the user set does not have the ability to create a replication slot, the Meroxa Platform will automatically fall back to polling for data changes.

In the future, we plan to enable the platform to detect the available resource capabilities and adjusting the configuration accordingly.

Replica Identity

REPLICA IDENTITY is a PostgreSQL-specific table-level setting that determines the amount of information for UPDATE and DELETE events.

Meroxa will still capture INSERT, UPDATE and DELETE events, but will only see the resulting change after, but not the data before the operation was made.

To capture full changes, run the following:

ALTER TABLE "TABLE_NAME" REPLICA IDENTITY FULL;

For more information, see Replica Identity.

Polling

By default, our connectors will use the Polling method to detect and capture data changes.

To add a PostgreSQL resource:

meroxa add resource postgresDB --type postgres -u postgres://$PG_USER:[email protected]$PG_URL:$PG_PORT/$PG_DB

Updated about a month ago


PostgreSQL


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.