Amazon S3

The Conduit Platform by default supports Amazon S3 as a source and a destination.

The Amazon S3 destination can connect to and produce objects to a bucket.

Required Configurations

Name	Description	Required	Default
aws.accessKeyId	The AWS access key id.	Yes
aws.secretAccessKey	The AWS secret access key.	Yes
aws.bucket	The AWS S3 bucket name.	Yes
aws.region	The AWS S3 bucket region.	Yes
format	The destination format. Options: json or parquet.	Yes

Looking for something else? See advanced configurations.

Upsert Behavior

The destination connector calls Configure to parse the configuration. Next, Open is called to start the connection to write to a bucket.

Note: If the provided bucket does not exist or if the source connector fails to access it, an error will occur.

This connector has a buffer with a configured bufferSize. For each time Write is called, a new record is added to the buffer. When the buffer is full, all records will be written to the bucket, and an Ack function will be called for each record after it is written.

Advanced Configurations

Name	Description	Required	Default
bufferSize	Size of buffer. Min is 1 and max is 100000.	No	1000
prefix	The key prefix for the Amazon S3 destination.	No