Skip to main content
Elasticsearch Logo

Elasticsearch is a search engine based on the Lucene library. The Meroxa connector fetches new data using a incremental or temporal field such as a timestamp or an incrementing id.

Adding Resource

To add an Elasticsearch resource to your Meroxa Resource Catalog, you can run the following command:

meroxa resource add elasticsearch --type elasticsearch -u https://$ES_USER:$ES_PASS@$ES_URL:$ES_PORT

The -u option requires a valid access URL.

Setup

Environments

We support the following environments:

Networking

To add as a resource, the Elasticsearch instance needs to be accessible by Meroxa.

If your instance is publically accessible, there is nothing to do. If not, you may:

Access URL

The access URL contains the credentials needed to access your instance:

https://$ES_USER:$ES_PASS@$ES_URL:$ES_PORT

In the example above, replace following variables with valid credentials from your Elasticsearch environment:

  • $ES_USER - Elasticsearch Username
  • $ES_PASS - Elasticsearch Password
  • $ES_URL - Elasticsearch URL
  • $ES_PORT - Elasticsearch Port (e.g., 1234).

Source Configuration

The Elasticesearch connector uses an incremental or temporal field to track changes to an index within Elasticesearch.

Creating Connectors

The Input for this connector is an Elasticsearch Index.

The following configuration is required for this connector:

ConfigurationDestination
incrementing.field.nameAn incremental/temporal field such as a timestamp or an incrementing id.

For example, to create a connector using the CLI, you may use the following command:

meroxa connector create from-es --from elasticsearch --input $ES_INDEX --config '{"incrementing.field.name" : "$ES_INCREMENTING_FIELD"}'

Data Record

On every update, the Data Record takes the following format:

{ "schema": { <schema of payload> }, "payload": { <json data captured from index data> } }

Advanced Configuration

The following advanced configuration is supported:

ConfigurationDestination
filters.whitelistWhitelist filter for extracting a subset of fields from elastic-search JSON documents. The whitelist filter supports nested fields. To provide multiple fields use ; as separator (e.g. customer;order.qty;order.price).
index.prefixIndices prefix to include in copying.