Skip to main content

Amazon S3

The Conduit Platform by default supports Amazon S3 as a source and a destination.

The Amazon S3 destination can connect to and produce objects to a bucket.

Required Configurations

aws.accessKeyIdThe AWS access key id.Yes
aws.secretAccessKeyThe AWS secret access key.Yes
aws.bucketThe AWS S3 bucket name.Yes
aws.regionThe AWS S3 bucket region.Yes
formatThe destination format. Options: json or parquet.Yes

Looking for something else? See advanced configurations.

Upsert Behavior

The destination connector calls Configure to parse the configuration. Next, Open is called to start the connection to write to a bucket.

Note: If the provided bucket does not exist or if the source connector fails to access it, an error will occur.

This connector has a buffer with a configured bufferSize. For each time Write is called, a new record is added to the buffer. When the buffer is full, all records will be written to the bucket, and an Ack function will be called for each record after it is written.

Advanced Configurations

bufferSizeSize of buffer. Min is 1 and max is 100000.No1000
prefixThe key prefix for the Amazon S3 destination.No