Amazon S3 is a flexible object storage product offered by Amazon Web Services. It can be used as an upstream or downstream resource in your Turbine data apps by using the
write function to a select S3 bucket.
meroxa resource create command to configure your Amazon S3 resource.
The following example depicts how this command is used to create an Amazon S3 resource named
datalake with the minimum configuration required.
$ meroxa resource create datalake \
--type s3 \
In the command above, replace the following variables with valid credentials from your Amazon S3 environment:
$AWS_ACCESS_KEY- AWS Access Key
$AWS_ACCESS_SECRET- AWS Access Secret
$AWS_REGION- AWS Region (e.g.,
$AWS_S3_BUCKET- AWS S3 Bucket Name
The following configuration is supported for this resource:
|Compression type for output files. Supported algorithms are |
Data records are written a folder within the root of the S3 bucket as gzipped JSON, with one record per file and using the following naming format:
In the following example, the record is from the
resource-5-499379.public.orders stream with starting offset
0000000000 and partition
aws s3 ls s3://data-lake-bucket/resource-7-133274/resource-5-499379.public.orders-0-0000000000.gz