Source Configuration

Use this page when you need to define what a stream reads from and how that source is scoped.

Every source configuration starts in source.connections[]. Each entry combines a stored connection ID with stream-specific scope such as selected tables, custom queries, S3 paths, or local file paths.

Source families

Use one source family per connection entry:

database fields such as database, schema, tables, and queries
s3 for S3-compatible object storage
files for local file sources

Do not mix database fields with s3 or files inside the same connection entry.

Database sources

Use database fields when the source is MySQL or PostgreSQL and the stream should read tables or custom queries from that connection.

Typical shape:

{
  "connectionId": "conn_postgres_prod",
  "database": "sakila",
  "schema": "public",
  "tables": [
    { "name": "actor" }
  ]
}

Key points:

connectionId selects the stored source connection
database chooses the database on that connection
schema is optional and mainly relevant for database sources
tables selects physical tables
queries defines virtual query sources and is supported in Load mode only
alias is only needed when the stream reads from multiple source connections

Table selection

Use tables[].selection when a table source needs a narrower scope than SELECT *.

The selection object controls:

selectedColumns for column projection
filters for row predicates
sorts for ordering
limit for table-level row caps

For operator behavior and UI mapping, use Data Filter Guide.

Query sources

Use queries[] when the source should be defined by SQL instead of by selected physical tables.

Typical shape:

{
  "connectionId": "conn_mysql_prod",
  "database": "commerce",
  "queries": [
    {
      "name": "recent_orders",
      "query": "SELECT * FROM orders WHERE created_at >= '2024-01-01'"
    }
  ]
}

Key points:

query sources are Load mode only
name becomes the virtual source name used by the target side
columns[] is optional and is mainly used as a schema hint

S3 sources

Use s3 when the source connection points to AWS S3, DigitalOcean Spaces, MinIO, or another S3-compatible provider.

Typical shape:

{
  "connectionId": "conn_s3_exports",
  "s3": {
    "bucket": "company-data",
    "manifestPath": "s3://company-data/exports/orders/manifest.json"
  }
}

You can scope an S3 source with:

bucket plus prefixes[] for recursive folder-style selection
bucket plus objects[] for exact keys
manifestPath for manifest-driven selection

Do not combine manifestPath with prefixes or objects.

Local file sources

Use files when the source is a local filesystem connection.

Typical shape:

{
  "connectionId": "conn_local_files",
  "files": {
    "basePath": "/data/imports",
    "paths": [
      "orders/2026-03/orders.csv",
      "customers/customers.parquet"
    ]
  }
}

Key points:

basePath is the root directory for that file connection
paths[] are relative to basePath
file-based streams use the connection's filesystem context, not the machine that saved the config

Source options

source.options holds execution settings that belong to the read side.

Important fields:

dataBundleSize for batch sizing in both Load and CDC workflows
operations for CDC event types to capture
replicationSlot for PostgreSQL CDC
publicationName for PostgreSQL CDC
initialSnapshot for copying existing rows before CDC starts

operations, replicationSlot, publicationName, and initialSnapshot are CDC-only settings.

For initial snapshot behavior and target requirements, see Initial Load + CDC.

Verification

Before saving or using a source config:

confirm every entry lives under source.connections[]
confirm each connection entry uses only one source family
confirm database sources use tables or queries, not file-specific fields
confirm S3 sources do not mix manifestPath with prefixes or objects
confirm local file paths are correct relative to basePath