Resources
Resources are reusable configurations on the Meroxa Platform used to connect your production data. Resources can be used upstream (as a data source) or downstream (as a data destination) to your Turbine data applications depending on your use case. Check the resource-specific documentation for more details on how they can be used.
Configuration
Resource configuration can be accomplished through the Meroxa Dashboard or Meroxa CLI.
At a minimum, all resources require a name, type, and an access URL (otherwise known as a Uniform Resource Identifier or URI) containing connectivity details and credentials with the appropriate access and permissions to the datastore.
Create
Use meroxa resource create to create a new resource configuration. The following example shows how to configure a PostgreSQL database as a resource named pg_db using the Meroxa CLI.
$ meroxa resources create pg_db \
--type postgres \
--url postgres://username:[email protected]:5432/database
Creating "pg_db" resource in "common" environment...
Resource "pg_db" is successfully created!
Configuration Options
You can include additional configuration options to your resoruce depending on your datastore-level configuration and requirements.
--ca-cert string trusted certificates for verifying resource
--client-cert string client certificate for authenticating to the resource
--client-key string client private key for authenticating to the resource
--env string environment (name or UUID) where resource will be created
-h, --help help for create
-m, --metadata string resource metadata
--password string password
--ssh-private-key string SSH tunneling private key
--ssh-url string SSH tunneling address
--ssl use SSL
--type string resource type (required)
-u, --url string resource url (required)
--username string username
Connection
Getting data into your Turbine data app comes in two steps. First, you need to identify which resources to connect. Second, Turbine then needs to know what to do with those resource connections—whether pulling data records from or writing output to a specific resource.
Depending on the type of database a resource relies on, you may use the resource as a data source, from which data is pulled into your Turbine app, or as a data destination: a location to which the transformed records from your Turbine app are written to.
Find out which resources can be used as either a data source or a data destination with our list of supported integrations.
Resource
Using the resource function with an existing resource on Meroxa tells Turbine what connections it needs to establish on Meroxa. The resources are of course limited to the credentials provided and their respective access and priviledges to the datastore.
Records
Using the records function with an identified resource tells Turbine from where you'd like to pull data records. Any resource used as a source of data for your Turbine data app must have, at a minimum, credentials with READ access to the datastore and relevant data structure (e.g., table, collection, bucket, etc.).
Write
Using the write function with an identified resource tells Turbine where you'd like to write output from your data app. Any resource your data app writes to must have, at a minimum, credentials with WRITE access to the datastore and relevant data structure.
TURBINE DATA APP
┌───────────────────┐
┌►│ `resource` │
│ ├───────────────────┤
│ │ `records` │
┌────────────────┐ │ ├───────────────────┤ ┌────────────────┐
│ Resource │─┘ │ (Your code) │ ┌►│ Resource │
└────────────────┘ ├───────────────────┤ │ └────────────────┘
│ `write` │ │
├───────────────────┤ │
│ `resource` │─┘
└───────────────────┘
Learn more about how to get started with Turbine data apps in our Getting Started.
Management
Management of existing resources can be accomplished through the Meroxa dashboard or CLI.
Following is a list of commands used in the Meroxa CLI to manage existing resources. We'll be using the same PostgreSQL database example from creating the resource.
List
Use meroxa resource list to list all existing resources on your account.
$ meroxa resource list
Describe
Use meroxa resource describe to list all existing resources on your account.
$ meroxa resource describe prod_db
ID: 123
Name: pg_db
Type: postgres
URL: postgres://username:[email protected]:5432/database
Tunnel: N/A
State: Ready
Environment Name: common
Update
Use meroxa resource update to update the name or other configurations of an existing resource.
# Update resource name
$ meroxa resource update pg_db --name production_data
Updating resource "pg_db"...
Resource "production_data" is successfully updated!
# Update resource URI
$ meroxa resource update pg_db --url postgres://username:[email protected]:5432/newdatabase
Updating resource "pg_db"...
Resource "pg_db" is successfully updated!
# Update resource configuration -> Turn Logical Replication ON
# meroxa resource update pg_db --metadata '{"logical_replication":true}'
Updating resource "pg_db"...
Resource "pg_db" is successfully updated!
Validate
Use meroxa resource validate to validate whether a resource can be connected to.
$ meroxa resource validate pg_db
Resource "pg_db" is successfully validated!
Rotate Key
Use meroxa resource rotate-tunnel-key to rotate a Bastion key in the event you use an SSH tunnel to connect with a resource.
$ meroxa resource rotate-tunnel-key pg_db
Rotating tunnel key will restart the tunnel and disconnect existing connections.
To proceed, type "pg_db" or re-run this command with --force
▸ pg_db
Resource "pg_db" tunnel key is successfully rotated!
Remove
Use meroxa resource remove to remove a resource from your account.
$ meroxa resource remove pg_db
To proceed, type "pg_db" or re-run this command with --force
▸ pg_db
Removing resource "pg_db"...
Resource "pg_db" successfully removed!