S3-Compatible Storage

S3 connections let you browse buckets, query objects with SQL, and use S3-compatible storage as a stream source or target.

Supported providers

• AWS S3
• Google Cloud Storage (via HMAC keys)
• Cloudflare R2
• DigitalOcean Spaces
• Backblaze B2
• MinIO
• Custom — any S3-compatible storage API (Wasabi, Ceph, Linode Object Storage, etc.)

Connection settings

Scope (optional) — narrows the Data Explorer view to a specific bucket and/or prefix. When using this connection as a stream target, bucket and prefix are configured separately in stream settings.

Provider presets

Each preset auto-fills the endpoint, region, and URL style:

Cloudflare R2 — replace <account-id> with your Cloudflare account ID. Find it in the R2 dashboard. R2 does not charge egress fees.

Backblaze B2 — use your application key ID and key. Adjust the region in the endpoint (e.g., s3.eu-central-003.backblazeb2.com) to match your bucket's region.

MinIO — the preset defaults to a local development instance. Set useSSL: true and update the endpoint for production deployments.

DigitalOcean Spaces — adjust the region prefix in the endpoint (e.g., sfo3.digitaloceanspaces.com) to match your Space's region.

Supported file formats

Compressed files are detected and decompressed transparently. See Compressed File Support.

Typical workflow

1. Create an S3-compatible connection.
2. Validate credentials and bucket access.
3. Open the connection in Data Explorer.
4. Browse buckets or prefixes.
5. Query files through the SQL Console or use them in Federated Queries.

Reading from S3 (source)

Reading strategies

• Direct file — point to a single S3 path: s3://bucket/data/file.parquet
• Glob patterns — match multiple files: s3://bucket/data/*.csv. Works with nested directory layouts like year=2024/month=01/*.parquet.
• Manifest-driven — use a manifest file listing paths to read

Manifest files

A manifest is a JSON file that lists S3 objects belonging to a dataset. Manifests are useful for:

• Large datasets (>1000 files) where S3 LIST operations become slow
• Reproducible reads — pin a dataset to an exact set of files
• Cross-system coordination — share file lists between pipelines

Manifest format

{
  "version": "1.0",
  "files": [
    "s3://bucket/data/orders-0001.parquet",
    "s3://bucket/data/orders-0002.parquet"
  ],
  "objects": [
    {
      "path": "s3://bucket/data/orders-0001.parquet",
      "size": 1048576,
      "last_modified": "2026-03-10T18:00:00Z",
      "etag": "abc123",
      "storage_class": "STANDARD"
    },
    {
      "path": "s3://bucket/data/orders-0002.parquet",
      "size": 2097152,
      "last_modified": "2026-03-10T18:01:00Z",
      "etag": "def456",
      "storage_class": "STANDARD"
    }
  ],
  "metadata": {
    "total_size": 3145728,
    "row_count": 50000,
    "created_at": "2026-03-10T18:05:00Z",
    "file_format": "parquet",
    "stream_id": "stream_abc123"
  }
}

Automatic manifest creation

When a load-mode stream writes to an S3 target, a manifest is automatically created at the canonical path:

s3://bucket/prefix/manifest.json

The manifest includes all uploaded files with their sizes, ETags, and storage classes.

Fast folder listing

When browsing S3 folders, the UI checks for a manifest.json at the canonical path before issuing an S3 LIST request. This provides instant folder loading for previously written datasets. Use the refresh button to bypass the manifest and list live bucket contents.

Using manifests as a source

In stream configuration, set manifestPath instead of prefixes or objects:

{
  "source": {
    "connections": [
      {
        "connectionId": "conn_s3",
        "s3": {
          "bucket": "my-data-bucket",
          "manifestPath": "s3://my-data-bucket/exports/manifest.json"
        }
      }
    ]
  }
}

Writing to S3 (target)

Stream target configuration

When using S3 as a stream target, use spec.s3 with upload settings:

{
  "target": {
    "id": "conn_s3",
    "spec": {
      "s3": {
        "fileFormat": "parquet",
        "format": {
          "compression": "zstd"
        },
        "upload": {
          "bucket": "my-data-bucket",
          "prefix": "exports/",
          "storageClass": "STANDARD",
          "serverSideEnc": "AES256"
        }
      }
    }
  }
}

File format is specified in the target spec, not the connection. One S3 connection can serve multiple streams with different formats.

Output organization

Files are organized in Hive-style partitioning: <prefix>/<table>/part-<instance>-<seq>.<ext>

Features:

• Parallel streaming from multiple writer instances
• Automatic file rotation at 200 MB
• Multipart upload for files over 64 MB
• Compatible with Athena, Snowflake, Spark, and Hive

Google Cloud Storage (S3-compatible access)

GCS supports S3-compatible access via HMAC keys. This lets you use GCS buckets with the same S3 connection type.

Setup

1. Go to Cloud Storage → Settings → Interoperability
2. Under Access keys for service accounts, click Create a key for a service account
3. Select your service account (or create one with Storage Object Admin role)
4. Copy the Access Key (GOOG1E...) and Secret immediately — the secret is shown only once

Connection settings

Select the Google Cloud Storage preset, then enter your HMAC Access Key (GOOG1E...) and Secret as static credentials. The endpoint and region are auto-filled.

Permissions

Minimum required:

• storage.objects.create, storage.objects.delete, storage.objects.get, storage.objects.list, storage.buckets.get

Recommended role: Storage Object Admin (roles/storage.objectAdmin)

Key rotation

1. Create a new HMAC key
2. Update your DBConvert Streams connection with the new credentials
3. Verify the connection works
4. Delete the old key

For more details, see Google Cloud Storage Interoperability and HMAC Keys for Service Accounts.

Constraints

• S3 source workflows are query-oriented (Load mode, Explorer, SQL). They are not CDC sources.
• Provider-specific extensions outside the S3-compatible API model are not guaranteed.

Related docs

• Connection Management
• Stream Configuration Reference
• Compressed File Support
• Local Files
• Federated Queries