Elasticsearch
Elasticsearch is a search engine based on the Lucene library. It can be used as an upstream resource in your Turbine data applications using the records
function to a select type in an index.
Meroxa supports self-hosted and Elastic Cloud instances of Elasticsearch.
Setup
Networking
To add an Elasticsearch search engine as a resource, it must be accessible by Meroxa. If not publicly accessible, here are some ways to give Meroxa access.
Resource Configuration
Use the meroxa resource create
command to configure your Elasticsearch resource.
info
During the public beta, Elasticsearch only recognizes the https://
scheme for the Elasticsearch URL. Using elasticsearch://
will result in an error.
The following example depicts how this command is used to create an Elasticsearch resource named elasticsearch
with the minimum configuration required.
meroxa resource create elasticsearch \
--type elasticsearch \
--url "https://$ES_USER:$ES_PASS@$ES_URL:$ES_PORT" \
--metadata '{"index.prefix": "$ES_INDEX","incrementing.field.name": "$ES_INCREMENTING_FIELD"}'
In the example above, replace following variables with valid credentials from your Elasticsearch environment:
$ES_USER
- Elasticsearch Username$ES_PASS
- Elasticsearch Password$ES_URL
- Elasticsearch URL$ES_PORT
- Elasticsearch Port (e.g., 9200)ES_INDEX
- Elasticsearch IndexES_INCREMENTING_FIELD
- Elasticsearch Incremental Field (See Configuration Requirements)
Configuration Requirements
Meroxa fetches new data using an incremental/temporal field such as a timestamp
or an incrementing id
to track changes to an Elasticsearch Index, which is used as an input for this resource.
The following configuration is required:
Configuration | Destination |
---|---|
index.prefix | Indices prefix to include in copying. |
incrementing.field.name | An incremental/temporal field such as a timestamp or an incrementing id . |
Advanced Configuration
The following advanced configuration is supported and optional:
Configuration | Destination |
---|---|
filters.whitelist | Whitelist filter for extracting a subset of fields from Elasticsearch JSON documents. The whitelist filter supports nested fields. To provide multiple fields use ; as separator (e.g. customer;order.qty;order.price ). |
Data Record Format
Data records from Elasticsearch will take on the following format:
{ "schema": { <schema of payload> }, "payload": { <json data captured from index data> } }