Skip to main content

Resources

Resources are reusable configurations on the Meroxa Platform used to connect your data. Resources can be used upstream (as a data source) or downstream (as a data destination) to your Turbine data applications depending on your use case. Check the resource-specific documentation for more details on how they can be used.

Configuration​

Resource configuration can be accomplished through the Meroxa Dashboard and CLI.

All Resources require a name, type, and a connection URL (otherwise known as a Uniform Resource Identifier or URI) containing connectivity details and credentials with the appropriate access and permissions to the datastore.

Create a Resource​

  1. Navigate to Resources in the Meroxa Dashboard.
  2. Click the Add a Resource button.
  3. A list of resource types will appear. Click Create for the desired resource type.
  4. You will see a Create Resource form. Confirm your Resource Type is correct.
  5. Provide a Resource Name (Note: Must start with a letter and can only contain letters, numbers, underscores, or dashes). We recommend something memorable and easy, as it will be used to call the Resource in your Turbine code.
  6. Provide a Connection URL and/or other connection details asked for.
  7. Optional: Enable and provide details for any required Security configurations (e.g., TLS/SS and SSH Tunnel).
  8. Click the Save button.

Participate in a Developer Preview​

If you are interested in testing early releases of upcoming resource support on the Meroxa Platform, consider opting in for a Developer Preview. A full list of available and upcoming resources can be viewed when adding a new resource in the Meroxa Dashboard.

info

Resources in Developer Preview have known limitations as they are in development. By participating in a Developer Preview, you have the opportunity to test early releases, provide feedback to inform functionality, and get direct support from our team.

To gain quick access to the list to opt in to a Developer Preview, take the following steps:

  1. Navigate to Add a Resource in the Meroxa Dashboard.
  2. Look for the resource by name, or use the search bar at the top of the page to filter results by resource type.
  3. Click the Count me in button on the resource card.
  4. A modal will appear, click the Add me to the list button to confirm.

Developer Preview Walkthrough

Once you have confirmation you're on the list, be on the look out for a message from a member of our team about next steps!

Connection​

Getting data into your Turbine data application comes in two steps:

  1. Identify which resources to connect. You will need to create resources before referencing in your code.
  2. Articulate how the resources are used. Whether pulling data records from or writing output to a resource.

Depending on the type of resource, you may use the resource as a data source, from which data is pulled into your Turbine data application, or as a data destination: a location to which the transformed records from your Turbine app are written to.

Find out which resources can be used as either a data source or a data destination with our list of supported integrations.

Resource​

Using the resource function with an existing resource on Meroxa tells Turbine what connections it needs to establish on Meroxa. The resources are of course limited to the credentials provided and their respective access and privileges to the datastore.

Records​

Using the records function with an identified resource tells Turbine from where you'd like to pull data records. Any resource used as a source of data for your Turbine data app must have, at a minimum, credentials with READ access to the datastore and relevant data structure (e.g., table, collection, bucket, etc.).

Write​

Using the write function with an identified resource tells Turbine where you'd like to write output from your data app. Any resource your data app writes to must have, at a minimum, credentials with WRITE access to the datastore and relevant data structure.

                       TURBINE DATA APP
                     ┌───────────────────┐
                   ┌►│    `resource`     │
                   │ ├───────────────────┤
                   │ │     `records`     │
┌────────────────┐ │ ├───────────────────┤   ┌────────────────┐
│    Resource    │─┘ │    (Your code)    │ ┌►│    Resource    │
└────────────────┘   ├───────────────────┤ │ └────────────────┘
                     │      `write`      │ │
                     ├───────────────────┤ │
                     │    `resource`     │─┘
                     └───────────────────┘

Learn more about how to get started with Turbine data apps in our Getting Started.

Management​

Management of existing resources can be accomplished through the Meroxa dashboard or CLI.

Following is a list of commands used in the Meroxa CLI to manage existing resources. We'll be using the same PostgreSQL database example from creating the resource.

List​

Use meroxa resource list to list all existing resources on your account.

$ meroxa resource list

                 UUID                     NAME         TYPE       ENVIRONMENT                                             URL                                              TUNNEL   STATE
====================================== =========== ============= ============= ========================================================================================== ======== =======
 12abc3d4-5ef6-7g89-1hi2-34567jk89123  pg          postgres      common        postgres://diffuser-4.host.us-east-1.rds.amazonaws.com:5432/database                          N/A    ready

✨ To view your resources, visit https://dashboard.meroxa.io/resources

Describe​

Use meroxa resource describe to view details about a resource.

$ meroxa resource describe prod_db

               ID:   123
             Name:   pg_db
             Type:   postgres
              URL:   postgres://username:[email protected]:5432/database
           Tunnel:   N/A
            State:   Ready
 Environment Name:   common

Update​

Use meroxa resource update to update the name or other configurations of an existing resource.

# Update resource name
$ meroxa resource update pg_db --name production_data

Updating resource "pg_db"...
Resource "production_data" is successfully updated!

# Update resource URI
$ meroxa resource update pg_db --url postgres://username:[email protected]:5432/newdatabase

Updating resource "pg_db"...
Resource "pg_db" is successfully updated!

# Update resource configuration -> Turn Logical Replication ON
# meroxa resource update pg_db --metadata '{"logical_replication":true}'

Updating resource "pg_db"...
Resource "pg_db" is successfully updated!

Validate​

Use meroxa resource validate to validate whether a resource can be connected to.

$ meroxa resource validate pg_db

Resource "pg_db" is successfully validated!

Rotate Key​

Use meroxa resource rotate-tunnel-key to rotate a Bastion key in the event you use an SSH tunnel to connect with a resource.

$ meroxa resource rotate-tunnel-key pg_db

Rotating tunnel key will restart the tunnel and disconnect existing connections.
To proceed, type "pg_db" or re-run this command with --force
▸ pg_db
Resource "pg_db" tunnel key is successfully rotated!

Remove​

Use meroxa resource remove to remove a resource from your account.

$ meroxa resource remove pg_db

To proceed, type "pg_db" or re-run this command with --force
▸ pg_db
Removing resource "pg_db"...
Resource "pg_db" successfully removed!