Confluent Cloud Kafka
The Conduit Platform by default supports Confluent Cloud Kafka as a source and destination.
The Confluent Cloud Kafka source can connect to and emit records from a topic.
Required Configurations
| Name | Description | Required | Default |
|---|---|---|---|
servers | Servers is a list of Kafka bootstrap servers (i.e. brokers), which will be used to discover all the brokers in a cluster. This is your Confluent Bootstrap server. | Yes | |
topics | Topics is a comma separated list of Kafka topics from which records will be read. Example: "topic1,topic2" | Yes | |
tls.enabled | Defines whether TLS is enabled. | Yes | true |
saslMechanism | SASL mechanism to be used. Options: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512. If empty, authentication won't be performed. | Yes | PLAIN |
saslUsername | SASL username. If provided, a password needs to be provided too. This is your Confluent API Key. | Yes | |
saslPassword | SASL password. If provided, a username needs to be provided too. This is your Confluent API Secret. | Yes |
Confluent Requirements
Unlike configuration for the Apache Kafka connector, Confluent Cloud Kafka requires certain configurations to be set in order to successfully connect.
servers: Must be set to your Confluent Bootstrap server.tls.enabled: Must be set totrue.saslMechanism: Must be set toPLAIN.saslUsername: Is required. Use your Confluent API Key.saslPassword: Is required. Use your Confluent API Secret.
Connections to Confluent Cloud Kafka will not be possible without the above configurations.
Looking for something else? See advanced configurations.
Updates
The source connector is represented by a single consumer in a Kafka consumer group. A source's logical position corresponds to the respective consumer's offset in Kafka. Rather than saving the offset as the position, only the consumer group ID is saved, as it is sufficient for Kafka to locate the offset for the consumer.
A source becomes associated with a consumer group ID the first time the Read()method is called.
Advanced Configurations
| Name | Description | Required | Default |
|---|---|---|---|
clientID | A Kafka client ID. | No | conduit-connector-kafka |
readFromBeginning | Determines from whence the consumer group should begin consuming when it finds a partition without a committed offset. If this option is set to true it will start with the first message in that partition. | No | false |
groupID | Defines the consumer group ID. | No | |
clientCert | A certificate for the Kafka client, in PEM format. If provided, the private key needs to be provided too. | No | |
clientKey | A private key for the Kafka client, in PEM format. If provided, the certificate needs to be provided too. | No | |
caCert | The Kafka broker's certificate, in PEM format. | No | |
insecureSkipVerify | Controls whether a client verifies the server's certificate chain and host name. If true, accepts any certificate presented by the server and any host name in that certificate. | No | false |