> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streamkap.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Amazon S3

> Change Data Capture using Amazon S3 with Streamkap

## Requirements

* AWS Access Key and Secret Access Key with the following permissions to the destination bucket:

  * `s3:GetObject`
  * `s3:PutObject`
  * `s3:AbortMultipartUpload`
  * `s3:ListMultipartUploadParts`
  * `s3:ListBucketMultipartUploads`

## Configure Amazon S3 Connector

* `Name`: A descriptive name for the connector
* `AWS Access Key`: An Access Key with the appropriate permissions for the bucket to which Streamkap will load data
* `AWS Secret Access Key`: The Secret Access Key with the appropriate permissions for the bucket to which Streamkap will load data
* `Region`: Name of the region for bucket to which Streamkap will load data
* `Bucket Name`: The name of the bucket to which Streamkap will load data
* `Format`: The format of the file. The following options are available: `JSON Lines`, `JSON Array` and `Parquet`

<Info>
  CSV output is not currently supported. Writing CSV requires reading the source data as raw, unparsed bytes (a `ByteArrayConverter`) instead of as structured records, and Streamkap doesn't currently offer that as a serialization option. If you have a use case that needs CSV output, contact [Streamkap support](mailto:support@streamkap.com) to discuss options.
</Info>

* `Filename Template`: The format of the filename. A file extension is appended automatically based on the selected `Format` and `Compression Type` — do not include one in the template. See below for more information about formatting options.
* `Compression Type`: Compression type for output files. Supported algorithms are `gzip`, `snappy`, `zstd` and `none`. Defaults to `gzip`. Only shown for `JSON Lines` and `JSON Array` — hidden for `Parquet`, which compresses internally and doesn't use this setting.
* `Envelope Output` (Advanced, default: enabled): Wraps each output record in an envelope with Kafka metadata (`key`, `offset`, `timestamp`, `headers`) alongside the value. Disable to write only the record's own value. For `Parquet`, only applies when the value is a record or map.
* `Output Fields` (Advanced): List of fields to include in output. Available options are: `key`, `offset`, `timestamp`, `value` and `headers`. Defaults to `value`.
* `Max Records Per File` (Advanced, default: `0`): Maximum number of records buffered into a single file before it's rotated. This caps how large a file can grow, but rotation only makes a new file eligible for upload — files are still written to S3 on the connector's normal flush interval, not immediately when the cap is hit. Set to `0` for unlimited (one file per topic-partition per flush interval).
* `S3 Multipart Upload Part Size (bytes)` (Advanced, default: `5242880` / 5MB): Size of each part in a multipart upload to S3. This controls how a single file is chunked for upload to S3, not how many records go into a file — see `Max Records Per File` for that. Larger parts mean fewer S3 API calls for big files but more memory used per upload. AWS requires multipart parts to be at least 5MB (except the last part of a file).

**Filename template** (default: `{{topic}}-{{partition}}-{{start_offset}}`): The format of the filename, excluding the extension. You can combine any of the elements below using other text or characters, including dashes (`-`) and underscores (`_`). Static text placed at the start of the template (e.g. `dir1/dir2/{{topic}}-{{partition}}-{{start_offset}}`) acts as a directory prefix.

| Element                                | Description                                                                                                                                                                                                                                                                                                                                                                                   |
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `{{topic}}`                            | The Streamkap topic name. For example, a PostgreSQL Source table `web.salesorders` topic's name would be `salesorders`                                                                                                                                                                                                                                                                        |
| `{{partition:padding=true\|false}}`    | The partition number of the records in the file, typically `0`. Streamkap topics and their data can be partitioned for better performance in certain scenarios. For example, a topic `salesorders` has 10 partitions, 0 through to 9. If padding set to true it will set leading zeroes for offset, the default value is false;                                                               |
| `{{start_offset:padding=true\|false}}` | The offset number of the first record in the file. Every record streamed has an incrementing offset number. For example, a topic `salesorders` has 1000 records, offsets 0 through to 999. Note that in the case of a multi-partitioned topic, offset numbers are **not unique** across partitions. If padding set to true it will set leading zeroes for offset, the default value is false; |
| `{{timestamp:unit=yyyy\|MM\|dd\|HH}}`  | The timestamp for when the file was created by the Connector. For example, the template `{{topic}}{{timestamp:unit=yyyy}}-{{timestamp:unit=MM}}` and timestamp of `2024-01-01 20:24` would create a file named `salesorders2024-01`                                                                                                                                                           |
| `{{key}}`                              | The Kafka key                                                                                                                                                                                                                                                                                                                                                                                 |

Currently, supported combinations of variables are:

* topic, partition, start\_offset, and timestamp - grouping by the topic, partition, and timestamp;
* key - grouping by the key.
* key, topic, partition - grouping by the topic, partition, and key.