Amazon S3
The S3 source reads objects from Amazon S3 or any S3-compatible service (MinIO, Ceph, etc.). Each object in the bucket becomes a record in the pipeline — its contents are uploaded to a new agent session for processing.
Authentication
Before you create the pipeline, create the access key the connector will use.
The connector authenticates with an access key pair. access_key_id and
secret_access_key are the two parts of an access key created for an AWS IAM
user.
To obtain them in AWS IAM:
- Create (or choose) an IAM user for the connector.
- Attach a policy granting the permissions below.
- Create an access key
for that user. AWS shows the secret access key only once, at creation, so
record it then. The access key ID and secret are your
access_key_idandsecret_access_key.
For an S3-compatible service other than AWS (configured with endpoint_url),
create the access key in that service's own credential settings. The two values
map to access_key_id and secret_access_key the same way.
The credentials need these S3 permissions on the bucket:
s3:ListBucket— to enumerate objects in the bucket (scoped by prefix).s3:GetObject— to download each object's contents.
Example IAM policy:
MINIMAL IAM POLICY
Code example with json syntax.1
Configuration
SOURCE FIELD (S3)
Code example with json syntax.1
Fields
| Field | Required | Description |
|---|---|---|
type | Yes | s3. |
bucket | Yes | The S3 bucket name. |
region | Yes | The region of the S3-compatible service (e.g. us-east-1). |
access_key_id | Yes | AWS access key ID. Encrypted at rest and never returned in responses. |
secret_access_key | Yes | AWS secret access key. Encrypted at rest and never returned in responses. |
prefix | No | Key prefix to scope ingestion to a subset of objects (e.g. legal/contracts/). |
pos_regex | No | Regex patterns matched against the full object key; a key must fully match at least one to be ingested. Empty list means no positive filter. |
neg_regex | No | Regex patterns matched against the full object key; keys fully matching any are skipped. |
endpoint_url | No | Custom endpoint URL for S3-compatible services. If omitted, defaults to AWS S3. |
Using with S3-compatible services
Set endpoint_url to point at your service. The rest of the configuration
works the same as with AWS S3.
SOURCE FIELD (S3-COMPATIBLE, MINIO)
Code example with json syntax.1
How records are fetched
Each run lists objects in the bucket using the S3 ListObjectsV2 API,
scoped by the optional prefix. Folder markers (empty keys ending in /)
are skipped. The pipeline paginates through the full listing, processing
each object concurrently in small batches.
The pipeline captures an upper-bound timestamp at the start of each run
and only processes objects whose lastModified is at or before that
timestamp. This ensures that objects added to the bucket while a run is
in progress are left for the next run — they aren't partially processed.
Source metadata
Each record carries source metadata that the connector resolves at fetch time.
system_metadata:
| Key | Description |
|---|---|
size | Object size in bytes. |
etag | The object's ETag. |
user_metadata contains the object's user-defined metadata (the x-amz-meta-*
headers), when present.
acl_metadata holds the object's ACL in the source-independent
ACL metadata shape. S3 has no
notion of comment access or membership-resolved groups, so commenters and the
group_* buckets are always null. Entries in owners, editors, and readers
are the grantee's email (when the grant is by email) or the AWS canonical user id.
The buckets map to S3 grants as follows:
| Bucket | S3 grant |
|---|---|
owners | The object owner |
editors | Grantees with FULL_CONTROL or WRITE |
readers | Grantees with READ |
public_access | The predefined AllUsers group |
org_wide_access | The predefined AuthenticatedUsers group |
If the bucket's service doesn't support the object-ACL API, acl_metadata is
empty (every bucket null).
Incremental sync
When sync_mode is incremental (the default), the pipeline tracks a
watermark based on each object's lastModified timestamp. After a
successful run, the watermark advances to the upper bound captured at the
start of that run.
On the next run, only objects with lastModified > stored_watermark are
processed. This ensures:
- New files are picked up.
- Modified files (S3 updates
lastModifiedon overwrite) are reprocessed. - Unchanged files are skipped, keeping costs low.
Deletes are not propagated. An object removed from the bucket stops appearing in new runs. The pipeline emits no delete signal, so anything a previous run already produced downstream is left in place.