Meroxa

The Meroxa Developer Hub

Welcome to the Meroxa developer hub. You'll find comprehensive guides and documentation to help you start working with Meroxa as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started    Guides

Connectors

A Connector s responsible for how data is transferred in or out of the data stream. It is always associated with a Resource within a Pipeline.

Creating Connectors

There are two ways to create connectors using the CLI:

  • meroxa connect
  • meroxa create connector

Meroxa Connect

The quickest way to create the connectors is using the meroxa connect command:

meroxa connect --from postgresDB --input $TABLE_NAME --to datalake

In one command you are specifying:

  • --from - the source resource. In the example above, the resources was named postgresDB.
  • --input - the input desired from the source resource. The $TABLE_NAME variable represents the name of table in Postgres you would like to sync to S3.
  • --to - the destination resource. In the example above, the resources was named datalake.

For more information see meroxa connect

Create Connections Manually

Behind the scenes, the meroxa connect command creates two connections. One for the Source (Postgres) and one for the Destination (S3).

  1. First, create the Source connector:
meroxa create connector from-store --from postgresDB --input $TABLE_NAME

from-store is a human-friendly name to represent the resource. Feel free to change as desired. The --from and --input parameters are the same values as the meroxa connect command.

  1. Next, create the Destination connector:
meroxa create connector to-datalake --to datalake --input $STREAM_NAME

to-datalake is a human-friendly name to represent the resource. Feel free to change as desired. In this command --to represent the name of the destination S3 resource.

The $STREAM_NAME is provided by the `meroxa list connectors command:

Configuration

Source and Destination Connectors support the ability to pass provide configuration. In some cases, it's required. In others, it gives you greater control over how the connector behaves.

To provide configuration to a connector, you may use the --config parameter.

meroxa create connector from-es --from elasticsearch --input customer --config '{"incrementing.field.name" : "@timestamp"}'

See the Documentation for the Resources you would like to use for their specific configuration.

Updated about a month ago


Connectors


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.