Skip to main content
Elasticsearch Logo

Elasticsearch is a search engine based on the Lucene library. The Meroxa Connector fetches only new data using a strictly incremental/temporal field such as a timestamp or an incrementing id.

Adding Resource#

To add an Elasticsearch resource to your Meroxa Resource Catalog, you can run the following command:

meroxa resource add elasticsearch --type elasticsearch -u https://$ES_USER:$ES_PASS@$ES_URL:$ES_PORT

The -u option requires a valid access URL.

Setup#

Environments#

We support the following environments:

Networking#

To add as a resource, the Elasticsearch instance needs to be accessible by Meroxa.

If your instance is publically accessible, there is nothing to do. If not, you may:

Access URL#

The access URL contains the credentials needed to access your instance:

https://$ES_USER:$ES_PASS@$ES_URL:$ES_PORT

In the example above, replace following variables with valid credentials from your Elasticsearch environment:

  • $ES_USER - Elasticsearch Username
  • $ES_PASS - Elasticsearch Password
  • $ES_URL - Elasticsearch URL
  • $ES_PORT - Elasticsearch Port (e.g., 1234).

Source Configuration#

The Elasticesearch connector uses an incremental or temporal field to track changes to an index within Elasticesearch.

Creating Connectors#

The Input for this connector is an Elasticsearch Index.

The following configuration is required for this connector:

ConfigurationDestination
incrementing.field.nameAn incremental/temporal field such as a timestamp or an incrementing id.

For example, to create a connector using the CLI, you may use the following command:

meroxa connector create from-es --from elasticsearch --input $ES_INDEX --config '{"incrementing.field.name" : "$ES_INCREMENTING_FIELD"}'

Data Record#

On every update, the Data Record takes the following format:

{ "schema": { <schema of payload> }, "payload": { <json data captured from index data> } }

Advanced Configuration#

The following advanced configuration is supported:

ConfigurationDestination
filters.whitelistWhitelist filter for extracting a subset of fields from elastic-search JSON documents. The whitelist filter supports nested fields. To provide multiple fields use ; as separator (e.g. customer;order.qty;order.price).
index.prefixIndices prefix to include in copying.