Stream Configuration Reference

Use this reference when you need exact field paths and JSON structure for stream configurations.

For the mental model first, use Stream Configuration Overview. For source and target behavior, use Source Configuration and Target Configuration. For full payloads, use Stream Configuration Examples.

Top-level fields

Field	Type	Notes
`id`	string	Saved stream ID returned by the API, usually `stream_<ksuid>`.
`name`	string	Human-readable stream name.
`mode`	string	`convert` or `cdc`.
`description`	string	Optional description.
`created`	number	Unix timestamp returned by the API for saved configs.
`reportingInterval`	number	Progress reporting interval in seconds. Default: `5`.
`source`	object	Source configuration.
`target`	object	Target configuration.
`limits`	object	Optional stop conditions.

Source configuration

Source connections

Path: source.connections[].

Field	Type	Notes
`alias`	string	Omit for single-source streams. Required for multi-source streams.
`connectionId`	string	Stored connection ID to use as the source.
`database`	string	Database name for database sources.
`schema`	string	Optional schema or namespace for database sources.
`tables`	array	Selected physical tables.
`queries`	array	Virtual query sources. Load mode only.
`s3`	object	S3 source selection. Use instead of `database`, `schema`, `tables`, and `queries`.
`files`	object	Local file source selection. Use instead of `database`, `schema`, `tables`, and `queries`.

Selected tables

Path: source.connections[].tables[].

Field	Type	Notes
`name`	string	Source table name. For PostgreSQL, use schema-qualified names such as `public.actor` when selecting tables outside `public` or when selecting the same table name from multiple schemas.
`selection`	object	Structured selection state used by the Data Filter UI.

Table selection

Path: source.connections[].tables[].selection.

Field	Type	Notes
`selectedColumns`	array	Columns to include. Omit or leave empty for `SELECT *`.
`filters`	array	Row filter conditions.
`sorts`	array	Sort clauses.
`limit`	number	Row cap for this table selection. Omit it for no limit.

Table filters

Path: source.connections[].tables[].selection.filters[].

Field	Type	Notes
`column`	string	Source column to test.
`operator`	string	Saved API operator value such as `=`, `IN`, `IS NULL`, `CONTAINS`, or `STARTS_WITH`.
`value`	string	Comparison value. Omit or leave empty only for null-check operators.

Table sorting

Path: source.connections[].tables[].selection.sorts[].

Field	Type	Notes
`column`	string	Source column to sort by.
`direction`	string	`ASC` or `DESC`.

Query sources

Path: source.connections[].queries[].

Mode support: Load mode only.

Field	Type	Notes
`name`	string	Virtual table name written to the target.
`query`	string	Full SQL query.
`columns`	array	Optional result schema hints.

Query result columns

Path: source.connections[].queries[].columns[].

Field	Type	Notes
`name`	string	Column name in the query result.
`type`	string	Mapped column type.
`nullable`	boolean	Optional nullability flag.

S3 source settings

Path: source.connections[].s3.

Field	Type	Notes
`bucket`	string	Bucket to read from.
`prefixes`	array	Prefixes to include recursively. Each should end with `/`.
`objects`	array	Exact object keys to read.
`manifestPath`	string	Path to a manifest JSON file. Do not combine with `prefixes` or `objects`.

Local file source settings

Path: source.connections[].files.

Field	Type	Notes
`basePath`	string	Root directory on the local filesystem.
`paths`	array	Selected files or folders relative to `basePath`.

Source options

Path: source.options.

Mode support: operations, replicationSlot, publicationName, and initialSnapshot are used for CDC streams.

Field	Type	Notes
`dataBundleSize`	number	Rows read and sent per batch. Default: `1000`.
`operations`	array	CDC operations to capture: `INSERT`, `UPDATE`, `DELETE`.
`replicationSlot`	string	PostgreSQL CDC replication slot.
`publicationName`	string	PostgreSQL CDC publication name.
`initialSnapshot`	object	CDC-only option that copies existing source rows before starting log-based CDC. See Initial Load + CDC for lifecycle and target requirements.

Initial Load + CDC

Path: source.options.initialSnapshot.

Mode support: CDC mode only.

Field	Type	Notes
`enabled`	boolean	If `true`, the CDC run copies existing source rows first, then starts CDC from the matching source log position. Default: `false`.
`restartPolicy`	string	Current value: `fail_until_reset`. Interrupted partial snapshots fail until target state is reset.

Target configuration

Target object

Path: target.

Field	Type	Notes
`id`	string	Stored connection ID to use as the target.
`spec`	object	Exactly one target family object.

Target families

Path: target.spec.

Path	Use when
`target.spec.db`	Writing to MySQL or PostgreSQL.
`target.spec.files`	Writing local CSV, JSONL, or Parquet files.
`target.spec.s3`	Writing files locally, then uploading them to S3-compatible storage.

Database target

Path: target.spec.db.

Field	Type	Notes
`database`	string	Target database name.
`schema`	string	Optional target schema or namespace.
`structureOptions`	object	Table, index, foreign key, and check-constraint creation flags. See Database Structure Options.
`schemaPolicy`	string	Existing-table policy. See Target Schema and Data Policies.
`writeMode`	string	Existing-row policy. See Target Schema and Data Policies.
`skipData`	boolean	If `true`, create structure only and skip data writes.

Local file target

Path: target.spec.files.

Field	Type	Notes
`outputDirectory`	string	Optional. Usually omitted for standalone local file targets. The connection base path determines the final output location.
`fileFormat`	string	`csv`, `jsonl`, or `parquet`.
`format`	object	Compression and format-specific settings.

S3 target

Path: target.spec.s3.

Field	Type	Notes
`outputDirectory`	string	Local staging directory before upload. Optional when the platform default is acceptable.
`fileFormat`	string	`csv`, `jsonl`, or `parquet`.
`format`	object	Compression and format-specific settings.
`upload`	object	Destination bucket and upload behavior.

Format settings

Path: target.spec.files.format or target.spec.s3.format.

Field	Type	Notes
`compression`	string	Compression codec for file output.
`parquet`	object	Parquet-specific settings.
`csv`	object	Schema field for CSV-specific settings. Current target writers still use default CSV output settings.

Parquet format settings

Path: target.spec.files.format.parquet or target.spec.s3.format.parquet.

UI support: API and saved config only. These fields are not currently exposed in the UI.

Field	Type	Notes
`compressionCodec`	string	Parquet compression codec.
`compressionLevel`	number	Optional codec-specific compression level.
`dataPageSize`	number	Parquet data page size in bytes.
`maxRowGroupSize`	number	Maximum number of rows per row group.
`targetRowGroupSizeMB`	number	Target row group size in MB.
`parquetVersion`	string	Parquet format version.
`dataPageVersion`	string	Data page version.
`enableDictionary`	boolean	Enable dictionary encoding.
`enableStatistics`	boolean	Enable column statistics.
`timestampUnit`	string	Timestamp precision.
`binaryFormat`	string	Binary value encoding.

CSV format settings

Path: target.spec.files.format.csv or target.spec.s3.format.csv.

UI support: not exposed in the UI. Current target writers still use default CSV output settings.

Field	Type	Notes
`delimiter`	string	Field delimiter.
`quote`	string	Quote character.
`header`	boolean	Whether to emit a header row.

S3 upload settings

Path: target.spec.s3.upload.

Field	Type	Notes
`bucket`	string	Destination bucket.
`prefix`	string	Optional key prefix.
`keepLocalFiles`	boolean	Keep staged local files after upload.
`storageClass`	string	Storage class override.
`serverSideEnc`	string	Server-side encryption mode such as `AES256` or `aws:kms`.
`kmsKeyId`	string	KMS key ID when `serverSideEnc` is `aws:kms`.

Limits

Path: limits.

Field	Type	Notes
`numberOfEvents`	number	Maximum rows or events to process. `0` means unlimited.
`elapsedTime`	number	Maximum runtime in seconds. `0` means unlimited.