Resources
Resources are reusable configurations on the Meroxa Platform used to connect your data. Resources can be used upstream (as a data source) or downstream (as a data destination) to your Turbine streaming apps depending on your use case. Check the resource-specific documentation for more details on how they can be used.
Configuration
Resource configuration can be accomplished through the Meroxa Dashboard and CLI.
All Resources require a name, type, and a connection URL (otherwise known as a Uniform Resource Identifier or URI) containing connectivity details and credentials with the appropriate access and permissions to the datastore.
- Dashboard
- CLI
Create a Resource
- Navigate to Resources in the Meroxa Dashboard.
- Click the Add a Resource button.
- A list of resource types will appear. Click Create for the desired resource type.
- You will see a Create Resource form. Confirm your Resource Type is correct.
- Provide a Resource Name (Note: Must start with a letter and can only contain letters, numbers, underscores, or dashes). We recommend something memorable and easy, as it will be used to call the Resource in your Turbine code.
- Provide a Connection URL and/or other connection details asked for.
- Optional: Enable and provide details for any required Security configurations (e.g., TLS/SS and SSH Tunnel).
- Click the Save button.
Participate in a Developer Preview
If you are interested in testing early releases of upcoming resource support on the Meroxa Platform, consider opting in for a Developer Preview. A full list of available and upcoming resources can be viewed when adding a new resource in the Meroxa Dashboard.
Resources in Developer Preview have known limitations as they are in development. By participating in a Developer Preview, you have the opportunity to test early releases, provide feedback to inform functionality, and get direct support from our team.
To gain quick access to the list to opt in to a Developer Preview, take the following steps:
- Navigate to Add a Resource in the Meroxa Dashboard.
- Look for the resource by name, or use the search bar at the top of the page to filter results by resource type.
- Click the Count me in button on the resource card.
- A modal will appear, click the Add me to the list button to confirm.
Once you have confirmation you're on the list, be on the look out for a message from a member of our team about next steps!
Create a Resource
Use meroxa resource create
to create a new resource configuration. The following example shows how to configure a PostgreSQL database as a resource named pg_db
using the Meroxa CLI.
$ meroxa resources create pg_db \
--type postgres \
--url postgres://username:[email protected]:5432/database
Creating "pg_db" resource in "common" environment...
Resource "pg_db" is successfully created!
See other resource configuration examples in the CLI documentation, or by using meroxa resource create --help
in the Meroxa CLI.
Configuration Options
You can include additional configuration options to your resource depending on your datastore-level configuration and requirements.
--ca-cert string trusted certificates for verifying resource
--client-cert string client certificate for authenticating to the resource
--client-key string client private key for authenticating to the resource
--env string environment (name or UUID) where resource will be created
-h, --help help for create
-m, --metadata string resource metadata
--password string password
--private-key-file string path to private key file
--ssh-private-key string SSH tunneling private key
--ssh-url string SSH tunneling address
--ssl use SSL
--type string resource type (required)
-u, --url string resource url (required)
--username string username
Connection
Getting data into your Turbine data application comes in two steps:
- Identify which resources to connect. You will need to create resources before referencing in your code.
- Articulate how the resources are used. Whether pulling data records from or writing output to a resource.
Depending on the type of resource, you may use the resource as a data source, from which data is pulled into your Turbine data application, or as a data destination: a location to which the transformed records from your Turbine app are written to.
Find out which resources can be used as either a data source or a data destination with our list of supported integrations.
Resource
Using the resource
function with an existing resource on Meroxa tells Turbine what connections it needs to establish on Meroxa. The resources are of course limited to the credentials provided and their respective access and privileges to the datastore.
Records
Using the records
function with an identified resource tells Turbine from where you'd like to pull data records. Any resource used as a source of data for your Turbine data app must have, at a minimum, credentials with READ
access to the datastore and relevant data structure (e.g., table, collection, bucket, etc.).
Write
Using the write
function with an identified resource tells Turbine where you'd like to write output from your data app. Any resource your data app writes to must have, at a minimum, credentials with WRITE
access to the datastore and relevant data structure.
TURBINE DATA APP
┌───────────────────┐
┌►│ `resource` │
│ ├───────────────────┤
│ │ `records` │
┌────────────────┐ │ ├───────────────────┤ ┌────────────────┐
│ Resource │─┘ │ (Your code) │ ┌►│ Resource │
└────────────────┘ ├───────────────────┤ │ └────────────────┘
│ `write` │ │
├───────────────────┤ │
│ `resource` │─┘
└───────────────────┘
Learn more about how to get started with Turbine streaming apps in our Getting Started.
Management
Management of existing resources can be accomplished through the Meroxa dashboard or CLI.
Following is a list of commands used in the Meroxa CLI to manage existing resources. We'll be using the same PostgreSQL database example from creating the resource.
List
Use meroxa resource list
to list all existing resources on your account.
$ meroxa resource list
UUID NAME TYPE ENVIRONMENT URL TUNNEL STATE
====================================== =========== ============= ============= ========================================================================================== ======== =======
12abc3d4-5ef6-7g89-1hi2-34567jk89123 pg postgres common postgres://diffuser-4.host.us-east-1.rds.amazonaws.com:5432/database N/A ready
✨ To view your resources, visit https://dashboard.meroxa.io/resources
Describe
Use meroxa resource describe
to view details about a resource.
$ meroxa resource describe prod_db
ID: 123
Name: pg_db
Type: postgres
URL: postgres://username:[email protected]:5432/database
Tunnel: N/A
State: Ready
Environment Name: common
Update
Use meroxa resource update
to update the name or other configurations of an existing resource.
# Update resource name
$ meroxa resource update pg_db --name production_data
Updating resource "pg_db"...
Resource "production_data" is successfully updated!
# Update resource URI
$ meroxa resource update pg_db --url postgres://username:[email protected]:5432/newdatabase
Updating resource "pg_db"...
Resource "pg_db" is successfully updated!
# Update resource configuration -> Turn Logical Replication ON
# meroxa resource update pg_db --metadata '{"logical_replication":true}'
Updating resource "pg_db"...
Resource "pg_db" is successfully updated!
Validate
Use meroxa resource validate
to validate whether a resource can be connected to.
$ meroxa resource validate pg_db
Resource "pg_db" is successfully validated!
Rotate Key
Use meroxa resource rotate-tunnel-key
to rotate a Bastion key in the event you use an SSH tunnel to connect with a resource.
$ meroxa resource rotate-tunnel-key pg_db
Rotating tunnel key will restart the tunnel and disconnect existing connections.
To proceed, type "pg_db" or re-run this command with --force
▸ pg_db
Resource "pg_db" tunnel key is successfully rotated!
Remove
Use meroxa resource remove
to remove a resource from your account.
$ meroxa resource remove pg_db
To proceed, type "pg_db" or re-run this command with --force
▸ pg_db
Removing resource "pg_db"...
Resource "pg_db" successfully removed!