[{"data":1,"prerenderedAt":762},["ShallowReactive",2],{"docs-\u002Fdocs\u002Fstreams\u002Fmulti-source-streams":3},{"id":4,"title":5,"body":6,"description":753,"extension":754,"meta":755,"navigation":756,"path":757,"redirect":758,"seo":759,"stem":760,"__hash__":761},"docs\u002Fdocs\u002Fstreams\u002Fmulti-source-streams.md","Multi-Source Streams",{"type":7,"value":8,"toc":740},"minimark",[9,13,26,29,32,40,45,48,64,69,90,93,97,100,170,173,177,181,184,227,231,242,573,583,587,590,629,632,635,646,654,658,661,675,678,681,703,707,736],[10,11,5],"h1",{"id":12},"multi-source-streams",[14,15,16,17,21,22,25],"p",{},"A multi-source stream reads from two or more connections and writes the combined data into a single target. Each source connection gets an alias (e.g., ",[18,19,20],"code",{},"pg1",", ",[18,23,24],{},"my1",") used as a prefix in table references.",[14,27,28],{},"Use multi-source streams when you need to combine data from different systems into a single target — for example, merging customer data from PostgreSQL with product data from MySQL, or loading database records and S3-backed files into the same destination.",[14,30,31],{},"Each source contributes its own dataset, and all results are written into the same target as part of a single stream run.",[14,33,34,35,39],{},"Multi-source is available in ",[36,37,38],"strong",{},"Load mode only"," — CDC streams use a single source.",[41,42,44],"h2",{"id":43},"supported-source-combinations","Supported source combinations",[14,46,47],{},"Multi-source streams can mix different connection types:",[49,50,51,55,58,61],"ul",{},[52,53,54],"li",{},"Database + database (e.g., PostgreSQL + MySQL, or multiple databases of the same engine)",[52,56,57],{},"Database + local files (CSV, JSON, Parquet)",[52,59,60],{},"Database + S3-compatible storage",[52,62,63],{},"Files + S3",[65,66,68],"h3",{"id":67},"example-scenario","Example scenario",[49,70,71,77,83],{},[52,72,73,74,76],{},"PostgreSQL (",[18,75,20],{},") contains customers and orders",[52,78,79,80,82],{},"MySQL (",[18,81,24],{},") contains the product catalog",[52,84,85,86,89],{},"S3 (",[18,87,88],{},"s31",") contains historical sales exports",[14,91,92],{},"A multi-source stream can load all of these into a single warehouse, where downstream queries combine them.",[41,94,96],{"id":95},"how-aliases-work","How aliases work",[14,98,99],{},"Each source connection in a multi-source stream requires a unique alias. The UI auto-generates aliases based on connection type:",[101,102,103,116],"table",{},[104,105,106],"thead",{},[107,108,109,113],"tr",{},[110,111,112],"th",{},"Connection type",[110,114,115],{},"Alias pattern",[117,118,119,133,145,157],"tbody",{},[107,120,121,125],{},[122,123,124],"td",{},"PostgreSQL",[122,126,127,21,129,132],{},[18,128,20],{},[18,130,131],{},"pg2",", ...",[107,134,135,138],{},[122,136,137],{},"MySQL \u002F MariaDB",[122,139,140,21,142,132],{},[18,141,24],{},[18,143,144],{},"my2",[107,146,147,150],{},[122,148,149],{},"S3",[122,151,152,21,154,132],{},[18,153,88],{},[18,155,156],{},"s32",[107,158,159,162],{},[122,160,161],{},"Local files (CSV, JSON, Parquet)",[122,163,164,21,167,132],{},[18,165,166],{},"files1",[18,168,169],{},"files2",[14,171,172],{},"Aliases can be edited manually in the stream configuration UI. For single-source streams, the alias field is omitted automatically.",[41,174,176],{"id":175},"configuring-a-multi-source-stream","Configuring a multi-source stream",[65,178,180],{"id":179},"in-the-ui","In the UI",[14,182,183],{},"In the UI, each source appears as a separate panel with its own database, table, bucket, or file selection.",[185,186,187,194,201,204,211,214,217,220],"ol",{},[52,188,189,190,193],{},"Open ",[36,191,192],{},"New Stream Config"," from the Streams page.",[52,195,196,197,200],{},"Select ",[36,198,199],{},"Load"," mode.",[52,202,203],{},"In the source panel, add the first connection and select a database (or bucket\u002Ffolder for file sources).",[52,205,206,207,210],{},"Click ",[36,208,209],{},"Add Connection"," to add a second source. An alias is assigned automatically.",[52,212,213],{},"Repeat for additional sources.",[52,215,216],{},"Select tables or files from each source independently.",[52,218,219],{},"Configure the target and start the stream.",[52,221,222,223,226],{},"Verify the stream reaches ",[36,224,225],{},"finished"," and the row counts per source match expectations in run history.",[65,228,230],{"id":229},"via-api","Via API",[14,232,233,234,237,238,241],{},"Each source connection is an entry in the ",[18,235,236],{},"source.connections"," array. Include the ",[18,239,240],{},"alias"," field for every connection when there are two or more:",[243,244,249],"pre",{"className":245,"code":246,"language":247,"meta":248,"style":248},"language-json shiki shiki-themes github-light github-dark","{\n  \"name\": \"multi_db_sync\",\n  \"mode\": \"load\",\n  \"source\": {\n    \"connections\": [\n      {\n        \"alias\": \"pg1\",\n        \"connectionId\": \"conn_POSTGRES_ID\",\n        \"database\": \"sales_db\",\n        \"schema\": \"public\",\n        \"tables\": [\n          { \"name\": \"customers\" },\n          { \"name\": \"orders\" }\n        ]\n      },\n      {\n        \"alias\": \"my1\",\n        \"connectionId\": \"conn_MYSQL_ID\",\n        \"database\": \"inventory_db\",\n        \"tables\": [\n          { \"name\": \"products\" }\n        ]\n      }\n    ]\n  },\n  \"target\": {\n    \"id\": \"conn_TARGET_ID\",\n    \"spec\": {\n      \"db\": {\n        \"database\": \"warehouse\"\n      }\n    }\n  }\n}\n","json","",[18,250,251,260,277,290,299,308,314,327,340,353,366,374,391,406,412,418,423,435,447,459,466,480,485,491,497,503,511,524,532,540,550,555,561,567],{"__ignoreMap":248},[252,253,256],"span",{"class":254,"line":255},"line",1,[252,257,259],{"class":258},"sVt8B","{\n",[252,261,263,267,270,274],{"class":254,"line":262},2,[252,264,266],{"class":265},"sj4cs","  \"name\"",[252,268,269],{"class":258},": ",[252,271,273],{"class":272},"sZZnC","\"multi_db_sync\"",[252,275,276],{"class":258},",\n",[252,278,280,283,285,288],{"class":254,"line":279},3,[252,281,282],{"class":265},"  \"mode\"",[252,284,269],{"class":258},[252,286,287],{"class":272},"\"load\"",[252,289,276],{"class":258},[252,291,293,296],{"class":254,"line":292},4,[252,294,295],{"class":265},"  \"source\"",[252,297,298],{"class":258},": {\n",[252,300,302,305],{"class":254,"line":301},5,[252,303,304],{"class":265},"    \"connections\"",[252,306,307],{"class":258},": [\n",[252,309,311],{"class":254,"line":310},6,[252,312,313],{"class":258},"      {\n",[252,315,317,320,322,325],{"class":254,"line":316},7,[252,318,319],{"class":265},"        \"alias\"",[252,321,269],{"class":258},[252,323,324],{"class":272},"\"pg1\"",[252,326,276],{"class":258},[252,328,330,333,335,338],{"class":254,"line":329},8,[252,331,332],{"class":265},"        \"connectionId\"",[252,334,269],{"class":258},[252,336,337],{"class":272},"\"conn_POSTGRES_ID\"",[252,339,276],{"class":258},[252,341,343,346,348,351],{"class":254,"line":342},9,[252,344,345],{"class":265},"        \"database\"",[252,347,269],{"class":258},[252,349,350],{"class":272},"\"sales_db\"",[252,352,276],{"class":258},[252,354,356,359,361,364],{"class":254,"line":355},10,[252,357,358],{"class":265},"        \"schema\"",[252,360,269],{"class":258},[252,362,363],{"class":272},"\"public\"",[252,365,276],{"class":258},[252,367,369,372],{"class":254,"line":368},11,[252,370,371],{"class":265},"        \"tables\"",[252,373,307],{"class":258},[252,375,377,380,383,385,388],{"class":254,"line":376},12,[252,378,379],{"class":258},"          { ",[252,381,382],{"class":265},"\"name\"",[252,384,269],{"class":258},[252,386,387],{"class":272},"\"customers\"",[252,389,390],{"class":258}," },\n",[252,392,394,396,398,400,403],{"class":254,"line":393},13,[252,395,379],{"class":258},[252,397,382],{"class":265},[252,399,269],{"class":258},[252,401,402],{"class":272},"\"orders\"",[252,404,405],{"class":258}," }\n",[252,407,409],{"class":254,"line":408},14,[252,410,411],{"class":258},"        ]\n",[252,413,415],{"class":254,"line":414},15,[252,416,417],{"class":258},"      },\n",[252,419,421],{"class":254,"line":420},16,[252,422,313],{"class":258},[252,424,426,428,430,433],{"class":254,"line":425},17,[252,427,319],{"class":265},[252,429,269],{"class":258},[252,431,432],{"class":272},"\"my1\"",[252,434,276],{"class":258},[252,436,438,440,442,445],{"class":254,"line":437},18,[252,439,332],{"class":265},[252,441,269],{"class":258},[252,443,444],{"class":272},"\"conn_MYSQL_ID\"",[252,446,276],{"class":258},[252,448,450,452,454,457],{"class":254,"line":449},19,[252,451,345],{"class":265},[252,453,269],{"class":258},[252,455,456],{"class":272},"\"inventory_db\"",[252,458,276],{"class":258},[252,460,462,464],{"class":254,"line":461},20,[252,463,371],{"class":265},[252,465,307],{"class":258},[252,467,469,471,473,475,478],{"class":254,"line":468},21,[252,470,379],{"class":258},[252,472,382],{"class":265},[252,474,269],{"class":258},[252,476,477],{"class":272},"\"products\"",[252,479,405],{"class":258},[252,481,483],{"class":254,"line":482},22,[252,484,411],{"class":258},[252,486,488],{"class":254,"line":487},23,[252,489,490],{"class":258},"      }\n",[252,492,494],{"class":254,"line":493},24,[252,495,496],{"class":258},"    ]\n",[252,498,500],{"class":254,"line":499},25,[252,501,502],{"class":258},"  },\n",[252,504,506,509],{"class":254,"line":505},26,[252,507,508],{"class":265},"  \"target\"",[252,510,298],{"class":258},[252,512,514,517,519,522],{"class":254,"line":513},27,[252,515,516],{"class":265},"    \"id\"",[252,518,269],{"class":258},[252,520,521],{"class":272},"\"conn_TARGET_ID\"",[252,523,276],{"class":258},[252,525,527,530],{"class":254,"line":526},28,[252,528,529],{"class":265},"    \"spec\"",[252,531,298],{"class":258},[252,533,535,538],{"class":254,"line":534},29,[252,536,537],{"class":265},"      \"db\"",[252,539,298],{"class":258},[252,541,543,545,547],{"class":254,"line":542},30,[252,544,345],{"class":265},[252,546,269],{"class":258},[252,548,549],{"class":272},"\"warehouse\"\n",[252,551,553],{"class":254,"line":552},31,[252,554,490],{"class":258},[252,556,558],{"class":254,"line":557},32,[252,559,560],{"class":258},"    }\n",[252,562,564],{"class":254,"line":563},33,[252,565,566],{"class":258},"  }\n",[252,568,570],{"class":254,"line":569},34,[252,571,572],{"class":258},"}\n",[14,574,575,576,578,579,582],{},"For a single-source stream, omit the ",[18,577,240],{}," field and use one entry in ",[18,580,581],{},"connections",".",[41,584,586],{"id":585},"mixing-tables-queries-and-files","Mixing tables, queries, and files",[14,588,589],{},"Each connection in a multi-source stream can use its own selection model:",[49,591,592,607],{},[52,593,594,597,598,21,601,606],{},[36,595,596],{},"Database connections"," can include ",[18,599,600],{},"tables",[602,603,605],"a",{"href":604},"\u002Fdocs\u002Fstreams\u002Fcustom-sql-queries","custom queries",", or both",[52,608,609,612,613,21,616,21,619,622,623,625,626],{},[36,610,611],{},"S3 and local file connections"," use their own selection fields (",[18,614,615],{},"s3.prefixes",[18,617,618],{},"s3.objects",[18,620,621],{},"files.paths",") instead of ",[18,624,600],{}," or ",[18,627,628],{},"queries",[14,630,631],{},"Each source connection is processed independently. In mixed streams, table-based sources stream in parallel, query-based sources execute sequentially, and the results are written into the same target run.",[14,633,634],{},"Important:",[49,636,637,640,643],{},[52,638,639],{},"multi-source streams do not perform cross-source joins",[52,641,642],{},"each source is read independently",[52,644,645],{},"results are combined only at the target level",[14,647,648,649,653],{},"If you need to join data across sources, use a ",[602,650,652],{"href":651},"\u002Fdocs\u002Fdatabase-explorer\u002Ffederated-queries","federated query"," first, then turn it into a query-driven stream.",[41,655,657],{"id":656},"multi-source-streams-vs-federated-queries","Multi-source streams vs federated queries",[14,659,660],{},"Both features use the same alias model, but they solve different problems:",[49,662,663,669],{},[52,664,665,668],{},[36,666,667],{},"Federated queries"," are for analysis: one SQL query across multiple sources without persisting results",[52,670,671,674],{},[36,672,673],{},"Multi-source streams"," are for data movement: read from multiple sources and write the result into a target",[14,676,677],{},"Use federated queries to prototype and validate joins. Use multi-source streams to move and persist the resulting dataset.",[14,679,680],{},"You can prototype a multi-source query in the SQL Console before turning it into a stream:",[185,682,683,690,693,700],{},[52,684,685,686,582],{},"Open the ",[602,687,689],{"href":688},"\u002Fdocs\u002Fdatabase-explorer\u002Fsql-console","SQL Console",[52,691,692],{},"Select multiple sources and assign aliases.",[52,694,695,696,699],{},"Write and test a cross-source query using alias-qualified table names (e.g., ",[18,697,698],{},"pg1.public.customers",").",[52,701,702],{},"Use the validated query structure when configuring the stream.",[41,704,706],{"id":705},"related-docs","Related docs",[49,708,709,715,721,726,730],{},[52,710,711],{},[602,712,714],{"href":713},"\u002Fdocs\u002Fintegration\u002Fconversion-mode","Load Mode",[52,716,717],{},[602,718,720],{"href":719},"\u002Fdocs\u002Fstreams\u002Fstream-configuration-guide","Stream Configuration Guide",[52,722,723],{},[602,724,725],{"href":651},"Federated Queries",[52,727,728],{},[602,729,689],{"href":688},[52,731,732],{},[602,733,735],{"href":734},"\u002Fdocs\u002Fguide\u002Fapi-usage-guide","API Usage Guide",[737,738,739],"style",{},"html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":248,"searchDepth":262,"depth":262,"links":741},[742,745,746,750,751,752],{"id":43,"depth":262,"text":44,"children":743},[744],{"id":67,"depth":279,"text":68},{"id":95,"depth":262,"text":96},{"id":175,"depth":262,"text":176,"children":747},[748,749],{"id":179,"depth":279,"text":180},{"id":229,"depth":279,"text":230},{"id":585,"depth":262,"text":586},{"id":656,"depth":262,"text":657},{"id":705,"depth":262,"text":706},"Combine data from multiple connections in a single Load stream using aliases.","md",{},false,"\u002Fdocs\u002Fstreams\u002Fmulti-source-streams",null,{"title":5,"description":753},"docs\u002Fstreams\u002Fmulti-source-streams","bsWwHdB9J6NKhnkI3cIykqerqMg9t-n_0WOIvvoGVCQ",1783896284919]