Skip to main content

Getting Started

Data Pipeline Gif

Meroxa is a data orchestration platform that allows you to build production-ready data pipelines in minutes, not months.

Pipelines can:

  • perform real-time extract, transform, load (ETL).
  • capture Change Data Capture (CDC) events from various sources.
  • programmatically listen and receive data from a pipeline with Endpoints.
  • and much more.

For example, you could:

This interactive ✨ guide will walk you through creating your first pipeline in Meroxa. Let's begin.

Create an Account#

To start building pipelines, create an Account.

You can create an account using the Dashboard, or the CLI:

meroxa login

Selecting Resources#

Next, you will need to create resources. Resources represent your databases and data warehouse. They define the sources and destinations of they data you will use in your pipeline.

When you create a resource, you can always reuse them in additional pipelines.

Select a Source#

Here are the resources you can add as sources to your pipeline. Select a resource to learn more:

PostgreSQL

MongoDB

Elasticsearch

Need a resource that's not on this list? You can vote on or contribute new ideas. Your input helps us build a better roadmap for you.

Don't have a source? You can run PostgreSQL locally and expose to Meroxa using ngrok.

Source Information

The Connector gives you the ability to.

You can view the Connector Documentation to learn more about changing behavior or advanced configuration.

Select a Destinations#

Here are the destinations you can add to your pipeline. Select a resource to learn more:

Snowflake

PostgreSQL

MongoDB

Amazon Redshift

BigQuery

Amazon S3

Webhook

Destination Information

The Connector gives you the ability to send data to .

You can view the Connector Documentation to learn more about changing behavior or advanced configuration.

What does a pipeline do?#

Based on the source and destination you selected above, here is more information about the pipeline:

Hint: You can change the sources above and the instructions blow will update ✨

What will this pipeline do?

Your pipeline will move data from to .

It will and send to .

Building a Pipeline#

There are two ways for you to build pipelines. You may use the Dashboard or the CLI.

Select the instructions that works best for you:

If you're comfortable in a terminal, the CLI is a powerful, programmatic way to build data pipelines. With the CLI, you can automate or even script your pipelines.

Step One: Install CLI

To begin, you'll need to have the Meroxa CLI. To get started, see CLI Installation Guide.

Once installed, you can create an account or login with:

meroxa login

You can also view all available commands with:

meroxa help

Step Two: Add Resources

info

Depending on your environment, there may need to be additional configuration for proper access to resources. Please see the connector documentation for more details for the connectors you are using.

You may also whitlelist Meroxa's IPs.

Next, we can add our source resource using the following command:

meroxa resource add --type -u

You can view the Connector Documentation for more details.

Next, we can add our destination resource:

meroxa resource add --type -u

You can view the Connector Documentation for more details.

Step Three: Create Source Connector

Next, we can use the meroxa connector create command to create a new source connector:

meroxa connector create --from --input $INPUT

To create a source connector we can provide the --from and an --input parameters.

  • --from is the name of the source resource.
  • --input describes what table, index, collection in the source you would like to pull from.

Since is our source the input is .

Step Four: Create Destination Connector

Streams are created when you create connectors. Source Connectors produce output streams and Destination Connectors produce input streams.

To see the available Steams for your Connectors, you can use meroxa list connectors:

Meroxa List Connector

Now, we can use the meroxa connector create command to create a new destination connector:

meroxa connector create -to --input $STREAM_NAME

To create a destination connector we can provide the --to and an --input parameters.

  • --to is the name of the destination resource.
  • --input describes the $STREAM_NAME of the source connector.

Since is our destination, data from will now flow to .

Your pipeline is complete. Youc an go grab a coffee ☕️ ⚡️.

What's Next?#

Here are some helpful resources:

Staying in touch: