Resources

Resources are reusable configurations on the Meroxa Platform used to connect your data. Resources can be used upstream (as a data source) or downstream (as a data destination) to your Turbine data applications depending on your use case. Check the resource-specific documentation for more details on how they can be used.

Configuration

Resource configuration can be accomplished through the Meroxa Dashboard or Meroxa CLI.

At a minimum, all resources require a name, type, and an access URL (otherwise known as a Uniform Resource Identifier or URI) containing connectivity details and credentials with the appropriate access and permissions to the datastore.

Create

Use meroxa resource create to create a new resource configuration. The following example shows how to configure a PostgreSQL database as a resource named pg_db using the Meroxa CLI.

$ meroxa resources create pg_db \
--type postgres \
--url postgres://username:[email protected]:5432/database

Creating "pg_db" resource in "common" environment...
Resource "pg_db" is successfully created!

Configuration Options

You can include additional configuration options to your resoruce depending on your datastore-level configuration and requirements.

    --ca-cert string           trusted certificates for verifying resource
    --client-cert string       client certificate for authenticating to the resource
    --client-key string        client private key for authenticating to the resource
    --env string               environment (name or UUID) where resource will be created
-h, --help                     help for create
-m, --metadata string          resource metadata
    --password string          password
    --ssh-private-key string   SSH tunneling private key
    --ssh-url string           SSH tunneling address
    --ssl                      use SSL
    --type string              resource type (required)
 -u, --url string              resource url (required)
    --username string          username

Connection

Getting data into your Turbine data application comes in two steps:

1) Identify which resources to connect. You will need to create resources before referencing in your code. 2) Articulate how the resources are used. Whether pulling data records from or writing output to a resource.

Depending on the type of resource, you may use the resource as a data source, from which data is pulled into your Turbine data application, or as a data destination: a location to which the transformed records from your Turbine app are written to.

Find out which resources can be used as either a data source or a data destination with our list of supported integrations.

Resource

Using the resource function with an existing resource on Meroxa tells Turbine what connections it needs to establish on Meroxa. The resources are of course limited to the credentials provided and their respective access and privileges to the datastore.

Records

Using the records function with an identified resource tells Turbine from where you'd like to pull data records. Any resource used as a source of data for your Turbine data app must have, at a minimum, credentials with READ access to the datastore and relevant data structure (e.g., table, collection, bucket, etc.).

Write

Using the write function with an identified resource tells Turbine where you'd like to write output from your data app. Any resource your data app writes to must have, at a minimum, credentials with WRITE access to the datastore and relevant data structure.

                       TURBINE DATA APP
                     ┌───────────────────┐
                   ┌►│    `resource`     │
                   │ ├───────────────────┤
                   │ │     `records`     │
┌────────────────┐ │ ├───────────────────┤   ┌────────────────┐
│    Resource    │─┘ │    (Your code)    │ ┌►│    Resource    │
└────────────────┘   ├───────────────────┤ │ └────────────────┘
                     │      `write`      │ │
                     ├───────────────────┤ │
                     │    `resource`     │─┘
                     └───────────────────┘

Learn more about how to get started with Turbine data apps in our Getting Started.

Management

Management of existing resources can be accomplished through the Meroxa dashboard or CLI.

Following is a list of commands used in the Meroxa CLI to manage existing resources. We'll be using the same PostgreSQL database example from creating the resource.

List

Use meroxa resource list to list all existing resources on your account.

$ meroxa resource list

                 UUID                     NAME         TYPE       ENVIRONMENT                                             URL                                              TUNNEL   STATE
====================================== =========== ============= ============= ========================================================================================== ======== =======
 12abc3d4-5ef6-7g89-1hi2-34567jk89123  pg          postgres      common        postgres://diffuser-4.host.us-east-1.rds.amazonaws.com:5432/database                          N/A    ready

✨ To visualize your resources, visit https://dashboard.meroxa.io/resources

Describe

Use meroxa resource describe to view details about a resource.

$ meroxa resource describe prod_db

               ID:   123
             Name:   pg_db
             Type:   postgres
              URL:   postgres://username:[email protected]:5432/database
           Tunnel:   N/A
            State:   Ready
 Environment Name:   common

Update

Use meroxa resource update to update the name or other configurations of an existing resource.

# Update resource name
$ meroxa resource update pg_db --name production_data

Updating resource "pg_db"...
Resource "production_data" is successfully updated!

# Update resource URI
$ meroxa resource update pg_db --url postgres://username:[email protected]:5432/newdatabase

Updating resource "pg_db"...
Resource "pg_db" is successfully updated!

# Update resource configuration -> Turn Logical Replication ON
# meroxa resource update pg_db --metadata '{"logical_replication":true}'

Updating resource "pg_db"...
Resource "pg_db" is successfully updated!

Validate

Use meroxa resource validate to validate whether a resource can be connected to.

$ meroxa resource validate pg_db

Resource "pg_db" is successfully validated!

Rotate Key

Use meroxa resource rotate-tunnel-key to rotate a Bastion key in the event you use an SSH tunnel to connect with a resource.

$ meroxa resource rotate-tunnel-key pg_db

Rotating tunnel key will restart the tunnel and disconnect existing connections.
To proceed, type "pg_db" or re-run this command with --force
▸ pg_db
Resource "pg_db" tunnel key is successfully rotated!

Remove

Use meroxa resource remove to remove a resource from your account.

$ meroxa resource remove pg_db

To proceed, type "pg_db" or re-run this command with --force
▸ pg_db
Removing resource "pg_db"...
Resource "pg_db" successfully removed!