Elasticsearch

The Conduit Platform by default supports Elasticsearch as a destination only.

The Elasticsearch destination can connect to and produce records to an index.

Required Configurations

Name	Description	Required	Default
host	The Elasticsearch host and port. Example: http://127.0.0.1:9200	Yes
version	The version of the Elasticsearch service. Supported versions: 5, 6, 7, 8.	Yes
username	The username for HTTP Basic Authentication. Supported versions: 5, 6, 7, or 8.	Yes, if using HTTP Basic Authentication.
password	The password for HTTP Basic Authentication. Supported versions: 5, 6, 7, or 8.	Yes, if using HTTP Basic Authentication.
index	The name of the index to write the data to.	Yes
type	The name of the index's type to write the data to. Supported versions: 5 or 6.	Yes, if version 5 or 6. No, if any other version.
bulkSize	The number of items stored in bulk in the index. The minimum value is 1, maximum value is 10000. Note that values greater than 1000 may require additional service configuration.	Yes	"1000"
retries	The maximum number of retries of failed operations. The minimum value is 0 which disabled retry logic. The maximum value is 255. Note that the higher value, the longer it may take to process retries, as a result, ingest next operations.	Yes	"1000"

Looking for something else? See advanced configurations.

Upsert Behavior

The destination connector waits until the record has Key set, then uses it as a Document ID. Moreover, when the Record has action entry in the Metadata, then action specified there is respected.

• INSERT When Record.Key is missing, stores a new Document without ID.
• UPDATE: Stores or updates (upsert) a Document with ID. Default case when action is not set but Record.Key is set.
• DELETE: Deletes a Document by its Record.Key.

For any other action a warning entry is added to log and Record is skipped.

Advanced Configurations

Name	Description	Required	Default
username	The username for HTTP Basic Authentication. Supported versions: 5, 6, 7, or 8.	No
cloudId	Endpoint for the Elastic Service (https://elastic.co/cloud). Supported versions: 6, 7, or 8.	No
apiKey	Base64-encoded token for authorization; if set, overrides username/password and service token. Supported versions: 6, 7, or 8.	No	1000012
serviceToken	Service token for authorization; if set, overrides username/password. Supported versions: 7 or 8.	No
certificateFingerprint	SHA256 hex fingerprint given by Elasticsearch on first launch. Supported versions: 7 or 8.	No