Stream Configuration Examples

Use this page when you want full example payloads instead of field-by-field lookup tables.

All examples use the current nested shape:

source.connections[] for source entries
target.spec.* for the chosen target family

Replace connection IDs, database names, schemas, paths, and buckets with values from your environment.

Database source to database target

Use this pattern for a bounded Load workflow from PostgreSQL into MySQL or PostgreSQL.

{
  "name": "PostgreSQL to MySQL",
  "mode": "load",
  "source": {
    "connections": [
      {
        "connectionId": "conn_postgres_prod",
        "database": "postgres",
        "tables": [
          { "name": "public.actor" },
          {
            "name": "public.film",
            "selection": {
              "filters": [
                {
                  "column": "title",
                  "operator": "STARTS_WITH",
                  "value": "a"
                }
              ],
              "sorts": [
                {
                  "column": "film_id",
                  "direction": "ASC"
                }
              ],
              "limit": 4
            }
          }
        ]
      }
    ],
    "options": {
      "dataBundleSize": 1000
    }
  },
  "target": {
    "id": "conn_mysql_target",
    "spec": {
      "db": {
        "database": "testdb",
        "structureOptions": {
          "tables": true,
          "indexes": true,
          "foreignKeys": true,
          "checkConstraints": true
        },
        "schemaPolicy": "create_missing_only",
        "writeMode": "upsert"
      }
    }
  },
  "reportingInterval": 3
}

PostgreSQL CDC

Use this pattern for ongoing PostgreSQL CDC into another database target.

{
  "name": "PostgreSQL Orders CDC",
  "mode": "cdc",
  "source": {
    "connections": [
      {
        "connectionId": "conn_postgres_prod",
        "database": "orders",
        "schema": "public",
        "tables": [
          { "name": "orders" },
          { "name": "order_items" }
        ]
      }
    ],
    "options": {
      "operations": ["INSERT", "UPDATE"],
      "replicationSlot": "orders_replication",
      "publicationName": "orders_publication"
    }
  },
  "target": {
    "id": "conn_postgres_analytics",
    "spec": {
      "db": {
        "database": "analytics",
        "schema": "cdc_landing",
        "schemaPolicy": "validate_existing",
        "writeMode": "upsert"
      }
    }
  }
}

Database source to local Parquet files

Use this pattern for a filtered export from a database into local Parquet files.

{
  "name": "MySQL to Parquet export",
  "mode": "load",
  "source": {
    "connections": [
      {
        "connectionId": "conn_mysql_prod",
        "database": "commerce",
        "tables": [
          {
            "name": "events",
            "selection": {
              "filters": [
                {
                  "column": "created_at",
                  "operator": ">=",
                  "value": "2024-01-01"
                }
              ]
            }
          }
        ]
      }
    ]
  },
  "target": {
    "id": "conn_files_local",
    "spec": {
      "files": {
        "fileFormat": "parquet",
        "format": {
          "compression": "zstd",
          "parquet": {
            "targetRowGroupSizeMB": 256
          }
        }
      }
    }
  }
}

S3 source to database target

Use this pattern when the source data is manifest-driven in S3-compatible storage and the target is a database.

{
  "name": "S3 manifest to PostgreSQL",
  "mode": "load",
  "source": {
    "connections": [
      {
        "connectionId": "conn_s3_exports",
        "s3": {
          "bucket": "company-data",
          "manifestPath": "s3://company-data/exports/orders/manifest.json"
        }
      }
    ]
  },
  "target": {
    "id": "conn_postgres_landing",
    "spec": {
      "db": {
        "database": "landing",
        "schema": "public",
        "schemaPolicy": "create_missing_only",
        "writeMode": "append"
      }
    }
  }
}

Local files source to database target

Use this pattern when CSV or Parquet files on a local filesystem connection should be loaded into a database target.

{
  "name": "Local CSV to PostgreSQL",
  "mode": "load",
  "source": {
    "connections": [
      {
        "connectionId": "conn_local_files",
        "files": {
          "basePath": "/data/imports",
          "paths": [
            "orders/2026-03/orders.csv",
            "customers/customers.parquet"
          ]
        }
      }
    ]
  },
  "target": {
    "id": "conn_postgres_target",
    "spec": {
      "db": {
        "database": "landing",
        "schema": "public",
        "schemaPolicy": "create_missing_only",
        "writeMode": "append"
      }
    }
  }
}

Verification

After adapting an example:

confirm every placeholder connection ID exists in your environment
confirm the source family matches the selected connection type
confirm target.spec contains only the family you intend to use
confirm file paths, bucket names, schemas, and policies match the real destination