Multi-Source Streams

A multi-source stream reads from two or more connections and writes the combined data into a single target. Each source connection gets an alias (e.g., pg1, my1) used as a prefix in table references.

Use multi-source streams when you need to combine data from different systems into a single target — for example, merging customer data from PostgreSQL with product data from MySQL, or loading database records and S3-backed files into the same destination.

Each source contributes its own dataset, and all results are written into the same target as part of a single stream run.

Multi-source is available in Load mode only — CDC streams use a single source.

Supported source combinations

Multi-source streams can mix different connection types:

• Database + database (e.g., PostgreSQL + MySQL, or multiple databases of the same engine)
• Database + local files (CSV, JSON, Parquet)
• Database + S3-compatible storage
• Files + S3

Example scenario

• PostgreSQL (pg1) contains customers and orders
• MySQL (my1) contains the product catalog
• S3 (s31) contains historical sales exports

A multi-source stream can load all of these into a single warehouse, where downstream queries combine them.

How aliases work

Each source connection in a multi-source stream requires a unique alias. The UI auto-generates aliases based on connection type:

Aliases can be edited manually in the stream configuration UI. For single-source streams, the alias field is omitted automatically.

Configuring a multi-source stream

In the UI

In the UI, each source appears as a separate panel with its own database, table, bucket, or file selection.

1. Open New Stream Config from the Streams page.
2. Select Load mode.
3. In the source panel, add the first connection and select a database (or bucket/folder for file sources).
4. Click Add Connection to add a second source. An alias is assigned automatically.
5. Repeat for additional sources.
6. Select tables or files from each source independently.
7. Configure the target and start the stream.
8. Verify the stream reaches finished and the row counts per source match expectations in run history.

Via API

Each source connection is an entry in the source.connections array. Include the alias field for every connection when there are two or more:

{
  "name": "multi_db_sync",
  "mode": "load",
  "source": {
    "connections": [
      {
        "alias": "pg1",
        "connectionId": "conn_POSTGRES_ID",
        "database": "sales_db",
        "schema": "public",
        "tables": [
          { "name": "customers" },
          { "name": "orders" }
        ]
      },
      {
        "alias": "my1",
        "connectionId": "conn_MYSQL_ID",
        "database": "inventory_db",
        "tables": [
          { "name": "products" }
        ]
      }
    ]
  },
  "target": {
    "id": "conn_TARGET_ID",
    "spec": {
      "db": {
        "database": "warehouse"
      }
    }
  }
}

For a single-source stream, omit the alias field and use one entry in connections.

Mixing tables, queries, and files

Each connection in a multi-source stream can use its own selection model:

• Database connections can include tables, custom queries, or both
• S3 and local file connections use their own selection fields (s3.prefixes, s3.objects, files.paths) instead of tables or queries

Each source connection is processed independently. In mixed streams, table-based sources stream in parallel, query-based sources execute sequentially, and the results are written into the same target run.

Important:

• multi-source streams do not perform cross-source joins
• each source is read independently
• results are combined only at the target level

If you need to join data across sources, use a federated query first, then turn it into a query-driven stream.

Multi-source streams vs federated queries

Both features use the same alias model, but they solve different problems:

• Federated queries are for analysis: one SQL query across multiple sources without persisting results
• Multi-source streams are for data movement: read from multiple sources and write the result into a target

Use federated queries to prototype and validate joins. Use multi-source streams to move and persist the resulting dataset.

You can prototype a multi-source query in the SQL Console before turning it into a stream:

1. Open the SQL Console.
2. Select multiple sources and assign aliases.
3. Write and test a cross-source query using alias-qualified table names (e.g., pg1.public.customers).
4. Use the validated query structure when configuring the stream.

Related docs

• Load Mode
• Stream Configuration Guide
• Federated Queries
• SQL Console
• API Usage Guide