Skip to main content

Apache Kafka

The Conduit Platform by default supports Apache Kafka as a source and destination.

The Apache Kafka source can connect to and emit records from a topic.

Required Configurations

serversServers is a list of Kafka bootstrap servers (i.e. brokers), which will be used to discover all the brokers in a cluster.Yes
topicsTopics is a comma separated list of Kafka topics from which records will be read. Example: "topic1,topic2"Yes

Looking for something else? See advanced configurations.


The source connector is represented by a single consumer in a Kafka consumer group. A source's logical position corresponds to the respective consumer's offset in Kafka. Rather than saving the offset as the position, only the consumer group ID is saved, as it is sufficient for Kafka to locate the offset for the consumer.

A source becomes associated with a consumer group ID the first time the Read()method is called.

Advanced Configurations

clientIDA Kafka client ID.Noconduit-connector-kafka
readFromBeginningDetermines from whence the consumer group should begin consuming when it finds a partition without a committed offset. If this option is set to true it will start with the first message in that partition.Nofalse
groupIDDefines the consumer group ID.No
tls.enabledDefines whether TLS is enabled.Nofalse
clientCertA certificate for the Kafka client, in PEM format. If provided, the private key needs to be provided too.No
clientKeyA private key for the Kafka client, in PEM format. If provided, the certificate needs to be provided too.No
caCertThe Kafka broker's certificate, in PEM format.No
insecureSkipVerifyControls whether a client verifies the server's certificate chain and host name. If true, accepts any certificate presented by the server and any host name in that certificate.Nofalse
saslMechanismSASL mechanism to be used. Options: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512. If empty, authentication won't be performed.No
saslUsernameSASL username. If provided, a password needs to be provided too.No
saslPasswordSASL password. If provided, a username needs to be provided too.No