[{"data":1,"prerenderedAt":975},["ShallowReactive",2],{"docs-\u002Fdocs\u002Fstreams\u002Fverify-load":3},{"id":4,"title":5,"body":6,"description":966,"extension":967,"meta":968,"navigation":969,"path":970,"redirect":971,"seo":972,"stem":973,"__hash__":974},"docs\u002Fdocs\u002Fstreams\u002Fverify-load.md","Review and Verify a Load",{"type":7,"value":8,"toc":952},"minimark",[9,13,17,38,43,52,58,118,121,131,141,152,157,163,167,181,186,189,194,201,205,208,262,274,277,284,295,300,304,307,325,334,339,342,380,383,389,394,398,458,465,470,474,485,490,493,497,500,503,542,546,612,616,619,702,750,768,775,886,925,929,948],[10,11,5],"h1",{"id":12},"review-and-verify-a-load",[14,15,16],"p",{},"DBConvert Streams checks a Load at both ends:",[18,19,20,28],"ul",{},[21,22,23,27],"li",{},[24,25,26],"strong",{},"Before it starts",", you review a per-table plan: which tables will be created, used as they are, or dropped and recreated. Nothing is written to the target to build this plan.",[21,29,30,33,34,37],{},[24,31,32],{},"After it finishes",", ",[24,35,36],{},"Verify data"," compares source and target. It first compares totals for every table, then compares the rows themselves wherever the totals differ, and names the rows that do not match.",[39,40,42],"h2",{"id":41},"review-before-start","Review before start",[14,44,45,46,51],{},"When you start a stream, DBConvert Streams reads the source and target catalogs and shows the plan for every selected table, based on the stream's ",[47,48,50],"a",{"href":49},"\u002Fdocs\u002Fstreams\u002Ftarget-data-policies","schema policy and write mode",".",[53,54],"themed-image",{"alt":55,"dark":56,"light":57},"Review dialog before start: two MySQL tables that will replace existing PostgreSQL tables, with a checkbox to confirm the drop","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-start-review-dark.png","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-start-review-light.png",[59,60,61,74],"table",{},[62,63,64],"thead",{},[65,66,67,71],"tr",{},[68,69,70],"th",{},"Action in the plan",[68,72,73],{},"What happens to the target table",[75,76,77,86,94,102,110],"tbody",{},[65,78,79,83],{},[80,81,82],"td",{},"Create",[80,84,85],{},"The table does not exist and will be created.",[65,87,88,91],{},[80,89,90],{},"Replace",[80,92,93],{},"The table exists and will be dropped and created again, losing its data.",[65,95,96,99],{},[80,97,98],{},"Use existing",[80,100,101],{},"The table exists, is compatible, and will be written as it is.",[65,103,104,107],{},[80,105,106],{},"Review",[80,108,109],{},"The stream can start, but read the note under the table name first.",[65,111,112,115],{},[80,113,114],{},"Blocked",[80,116,117],{},"The stream cannot start until the reason shown is resolved.",[14,119,120],{},"Building the plan is read-only: no table is created, dropped, or truncated, and no test table is written to find out whether you have permissions.",[14,122,123,126,127,130],{},[24,124,125],{},"Replacing tables needs a second confirmation."," When the plan drops existing tables, ",[24,128,129],{},"Replace and start"," stays disabled until you tick the checkbox that names how many tables will lose their data.",[14,132,133,136,137,140],{},[24,134,135],{},"A blocked plan cannot be started."," The dialog lists the tables and why they are blocked, and offers only ",[24,138,139],{},"Close",". Common cases:",[18,142,143,149],{},[21,144,145,148],{},[24,146,147],{},"Fail if exists"," with tables that already exist on the target: choose another schema policy or drop those tables.",[21,150,151],{},"A target database that does not exist: DBConvert Streams does not create databases. Create it, or choose an existing database in the stream configuration.",[53,153],{"alt":154,"dark":155,"light":156},"Dialog explaining that 15 tables are blocked because the schema policy is Fail if exists and the target tables already exist","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-start-blocked-dark.png","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-start-blocked-light.png",[14,158,159,162],{},[24,160,161],{},"The plan you start is the plan you reviewed."," If the source, the target, or the stream configuration changes between the review and the start, the start is refused and the review opens again with the current plan.",[39,164,166],{"id":165},"verify-data-after-a-load","Verify data after a Load",[14,168,169,170,172,173,176,177,180],{},"Open a finished Load run and select ",[24,171,36],{}," in the ",[24,174,175],{},"Tables"," card. Verification runs in the background; the header shows its progress and a ",[24,178,179],{},"Cancel"," button, and you can leave the page while it works.",[53,182],{"alt":183,"dark":184,"light":185},"Tables card while rows are being compared: 27% progress and a Cancel button","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-verify-progress-dark.png","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-verify-progress-light.png",[14,187,188],{},"Each table goes through up to three checks.",[190,191,193],"h3",{"id":192},"_1-schema","1. Schema",[14,195,196,197,200],{},"The target table must exist and its columns, types, and primary key must be compatible with the source. A table whose schema does not match shows ",[24,198,199],{},"Schema differs",", and its data is not scanned.",[190,202,204],{"id":203},"_2-totals","2. Totals",[14,206,207],{},"DBConvert Streams runs one aggregate query per table on the source and one on the target, at the same time, and compares:",[59,209,210,220],{},[62,211,212],{},[65,213,214,217],{},[68,215,216],{},"Total",[68,218,219],{},"Computed for",[75,221,222,230,238,246,254],{},[65,223,224,227],{},[80,225,226],{},"Row count",[80,228,229],{},"every table",[65,231,232,235],{},[80,233,234],{},"Non-null values",[80,236,237],{},"every column",[65,239,240,243],{},[80,241,242],{},"Sum, minimum, maximum",[80,244,245],{},"integer, decimal, float columns",[65,247,248,251],{},[80,249,250],{},"Minimum, maximum",[80,252,253],{},"date and datetime columns",[65,255,256,259],{},[80,257,258],{},"Total text length",[80,260,261],{},"text columns",[14,263,264,265,269,270,273],{},"A total is computed only when both sides store the column in the same kind of type. A MySQL ",[266,267,268],"code",{},"tinyint(1)"," loaded into a PostgreSQL ",[266,271,272],{},"boolean",", binary and JSON columns, and timestamps that depend on the session time zone are compared by non-null count only.",[14,275,276],{},"If the stream selects rows with a filter or a limit, the source totals cover exactly the rows the run was configured to copy.",[14,278,279,280,283],{},"Sums of floating-point columns are compared with a tolerance: MySQL and PostgreSQL add millions of values in a different order and the result can differ in the 13th significant digit. These values are shown with ",[266,281,282],{},"≈",", and hovering one shows both exact sums.",[14,285,286,287,290,291,294],{},"When every total matches, the table shows ",[24,288,289],{},"Totals match"," with its row count, and the totals are folded into one line. ",[24,292,293],{},"Show values"," opens them as a table of columns by total.",[53,296],{"alt":297,"dark":298,"light":299},"Verified products table: all 20,000,000 rows identical, with the totals table showing non-null counts, sums, minimum and maximum values, and total text length for each column","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-verify-totals-dark.png","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-verify-totals-light.png",[190,301,303],{"id":302},"_3-rows","3. Rows",[14,305,306],{},"Totals show that a table differs, not where. Rows are compared:",[18,308,309,315],{},[21,310,311,314],{},[24,312,313],{},"automatically",", for every table whose totals differ, and",[21,316,317,320,321,324],{},[24,318,319],{},"on request",", for any finished table, with ",[24,322,323],{},"Find differing rows"," in the row actions or in the expanded result, even when the totals match.",[14,326,327,328,330,331,333],{},"Both tables are read in primary-key order, in ranges of 50,000 keys, and every row present on both sides is compared column by column. Source values are first converted the way the Load converted them when writing, so a type mapping such as ",[266,329,268],{}," to ",[266,332,272],{}," is not reported as a difference.",[53,335],{"alt":336,"dark":337,"light":338},"Verification result for a damaged table: 4 rows differ, differing totals in red, and the rows listed by id with the column, source value, and target value","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-verify-differences-dark.png","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-verify-differences-light.png",[14,340,341],{},"Each difference is one of:",[59,343,344,354],{},[62,345,346],{},[65,347,348,351],{},[68,349,350],{},"Difference",[68,352,353],{},"Meaning",[75,355,356,364,372],{},[65,357,358,361],{},[80,359,360],{},"Differs",[80,362,363],{},"The row exists on both sides; the differing columns are listed with both values.",[65,365,366,369],{},[80,367,368],{},"Missing in target",[80,370,371],{},"The source row was not found in the target.",[65,373,374,377],{},[80,375,376],{},"Missing in source",[80,378,379],{},"The target has a row the source does not.",[14,381,382],{},"Up to 10,000 differences are counted per table, and the first 1,000 are stored with their values. The expanded result lists the first 50.",[14,384,385,386,51],{},"When every row matches, the table shows ",[24,387,388],{},"All N rows identical",[53,390],{"alt":391,"dark":392,"light":393},"Tables card showing All 20,000,000 rows identical, compared by id, with the verification time and duration","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-verify-identical-dark.png","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-verify-identical-light.png",[39,395,397],{"id":396},"reading-a-result","Reading a result",[59,399,400,410],{},[62,401,402],{},[65,403,404,407],{},[68,405,406],{},"Verified column",[68,408,409],{},"What was proven",[75,411,412,419,427,435,443,450],{},[65,413,414,416],{},[80,415,388],{},[80,417,418],{},"Every row was compared and matched.",[65,420,421,424],{},[80,422,423],{},"Totals match · N rows",[80,425,426],{},"The totals matched; the rows themselves were not compared.",[65,428,429,432],{},[80,430,431],{},"N rows differ",[80,433,434],{},"Rows were compared and N of them differ.",[65,436,437,440],{},[80,438,439],{},"N checks differ",[80,441,442],{},"Totals differ; open the table to see which.",[65,444,445,447],{},[80,446,199],{},[80,448,449],{},"The target table is missing or incompatible; data was not scanned.",[65,451,452,455],{},[80,453,454],{},"Not checked",[80,456,457],{},"The check could not run; open the table for the reason.",[14,459,460,461,464],{},"Every result shows when it was verified. Verification compares the target with the source ",[24,462,463],{},"as the source is at that moment",": rows written to the source after the run finished also appear as differences.",[53,466],{"alt":467,"dark":468,"light":469},"Expanded result for a table with a composite primary key: totals match, rows not compared because row comparison needs a single integer primary key","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-verify-not-checked-dark.png","\u002Fimages\u002Fdocs\u002Fstreams\u002Frun-verify-not-checked-light.png",[39,471,473],{"id":472},"verify-automatically-after-every-run","Verify automatically after every run",[14,475,476,477,480,481,484],{},"In the ",[24,478,479],{},"Structure and Data"," step, turn on ",[24,482,483],{},"Verify data after run"," under the write mode. Every Load run that finishes successfully then verifies itself: totals for every table, and rows for tables whose totals differ.",[53,486],{"alt":487,"dark":488,"light":489},"Verify data after run checkbox under the Write mode options of the stream wizard","\u002Fimages\u002Fdocs\u002Fstreams\u002Fverify-after-run-option-dark.png","\u002Fimages\u002Fdocs\u002Fstreams\u002Fverify-after-run-option-light.png",[14,491,492],{},"The option is off by default because it reads every table on both databases once more after the load.",[39,494,496],{"id":495},"cost","Cost",[14,498,499],{},"Totals are one full scan per table on each database. Comparing rows reads each table from both databases in full and compares the values in DBConvert Streams, so most of the work is done by the application rather than the databases. One row comparison runs at a time.",[14,501,502],{},"For scale, on a developer workstation with MySQL 8.0 and PostgreSQL 16 running locally in Docker, a 20,000,000-row, 3.8 GB table took:",[59,504,505,518],{},[62,506,507],{},[65,508,509,512,515],{},[68,510,511],{},"Check",[68,513,514],{},"Time",[68,516,517],{},"Load",[75,519,520,531],{},[65,521,522,525,528],{},[80,523,524],{},"Totals",[80,526,527],{},"about 40 s",[80,529,530],{},"one CPU core on MySQL for the scan",[65,532,533,536,539],{},[80,534,535],{},"Rows",[80,537,538],{},"72 s",[80,540,541],{},"about 1.2 CPU cores in DBConvert Streams, 30% on MySQL, 12% on PostgreSQL",[39,543,545],{"id":544},"limitations","Limitations",[18,547,548,554,567,588,596,603,606,609],{},[21,549,550,551,553],{},"Verification is available for finished ",[24,552,517],{}," runs. CDC runs are not verified.",[21,555,556,557,560,561,564,565,51],{},"Source and target must be ",[24,558,559],{},"MySQL"," or ",[24,562,563],{},"PostgreSQL",". Other targets report ",[24,566,454],{},[21,568,569,572,573,33,576,579,580,583,584,587],{},[24,570,571],{},"Custom SQL queries"," on a single database source are verified by totals: the query runs again on the source and its totals are compared with the target table. Rows are not compared, because a query output has no primary key. A query using ",[266,574,575],{},"NOW()",[266,577,578],{},"RANDOM()",", or ",[266,581,582],{},"LIMIT"," without ",[266,585,586],{},"ORDER BY"," can return different rows when it runs again.",[21,589,590,593,594,51],{},[24,591,592],{},"Federated queries"," and queries over files or S3, which run through DuckDB, are listed as ",[24,595,454],{},[21,597,598,599,602],{},"Comparing rows needs a ",[24,600,601],{},"single integer primary key",". Tables with a composite, text, or missing primary key are verified by totals only.",[21,604,605],{},"A selection that combines a limit with a custom sort order cannot be matched by key; its rows are not compared.",[21,607,608],{},"On an evaluation Load, text values after the first rows of each table are marked, so their values are not compared; counts, numbers, dates, and keys still are.",[21,610,611],{},"Verification reports differences. It does not change the target.",[39,613,615],{"id":614},"api","API",[14,617,618],{},"Start verification of a finished run:",[620,621,626],"pre",{"className":622,"code":623,"language":624,"meta":625,"style":625},"language-bash shiki shiki-themes github-light github-dark","curl -X POST \"$API_URL\u002Fapi\u002Fv1\u002Fstream-configs\u002F$CONFIG_ID\u002Fruns\u002F$RUN_ID\u002Fverify\" \\\n  -H \"X-API-Key: $API_KEY\" -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\"level\": \"aggregate\"}'\n","bash","",[266,627,628,670,693],{"__ignoreMap":625},[629,630,633,637,641,645,648,652,655,658,661,664,667],"span",{"class":631,"line":632},"line",1,[629,634,636],{"class":635},"sScJk","curl",[629,638,640],{"class":639},"sj4cs"," -X",[629,642,644],{"class":643},"sZZnC"," POST",[629,646,647],{"class":643}," \"",[629,649,651],{"class":650},"sVt8B","$API_URL",[629,653,654],{"class":643},"\u002Fapi\u002Fv1\u002Fstream-configs\u002F",[629,656,657],{"class":650},"$CONFIG_ID",[629,659,660],{"class":643},"\u002Fruns\u002F",[629,662,663],{"class":650},"$RUN_ID",[629,665,666],{"class":643},"\u002Fverify\"",[629,668,669],{"class":639}," \\\n",[629,671,673,676,679,682,685,688,691],{"class":631,"line":672},2,[629,674,675],{"class":639},"  -H",[629,677,678],{"class":643}," \"X-API-Key: ",[629,680,681],{"class":650},"$API_KEY",[629,683,684],{"class":643},"\"",[629,686,687],{"class":639}," -H",[629,689,690],{"class":643}," \"Content-Type: application\u002Fjson\"",[629,692,669],{"class":639},[629,694,696,699],{"class":631,"line":695},3,[629,697,698],{"class":639},"  -d",[629,700,701],{"class":643}," '{\"level\": \"aggregate\"}'\n",[59,703,704,716],{},[62,705,706],{},[65,707,708,713],{},[68,709,710],{},[266,711,712],{},"level",[68,714,715],{},"Checks",[75,717,718,728,738],{},[65,719,720,725],{},[80,721,722],{},[266,723,724],{},"schema",[80,726,727],{},"Schema only; reads metadata, no table scans.",[65,729,730,735],{},[80,731,732],{},[266,733,734],{},"aggregate",[80,736,737],{},"Schema and totals, then rows for every table whose totals differ.",[65,739,740,745],{},[80,741,742],{},[266,743,744],{},"rows",[80,746,747,748,51],{},"Schema, totals, and rows of the one table named in ",[266,749,59],{},[14,751,752,753,756,757,760,761,764,765,51],{},"The request returns ",[266,754,755],{},"202"," and the result is stored on the run: poll ",[266,758,759],{},"GET \u002Fapi\u002Fv1\u002Fstream-configs\u002F{id}\u002Fruns\u002F{runId}"," and read ",[266,762,763],{},"verification",". Stop a running verification with ",[266,766,767],{},"POST \u002Fapi\u002Fv1\u002Fstream-configs\u002F{id}\u002Fruns\u002F{runId}\u002Fverify\u002Fcancel",[14,769,770,771,774],{},"To turn on verification after every run, set ",[266,772,773],{},"verifyAfterRun"," in the database target spec:",[620,776,780],{"className":777,"code":778,"language":779,"meta":625,"style":625},"language-json shiki shiki-themes github-light github-dark","{\n  \"target\": {\n    \"spec\": {\n      \"db\": {\n        \"database\": \"analytics\",\n        \"schemaPolicy\": \"create_missing_only\",\n        \"writeMode\": \"truncate_and_load\",\n        \"verifyAfterRun\": true\n      }\n    }\n  }\n}\n","json",[266,781,782,787,795,802,810,825,838,851,862,868,874,880],{"__ignoreMap":625},[629,783,784],{"class":631,"line":632},[629,785,786],{"class":650},"{\n",[629,788,789,792],{"class":631,"line":672},[629,790,791],{"class":639},"  \"target\"",[629,793,794],{"class":650},": {\n",[629,796,797,800],{"class":631,"line":695},[629,798,799],{"class":639},"    \"spec\"",[629,801,794],{"class":650},[629,803,805,808],{"class":631,"line":804},4,[629,806,807],{"class":639},"      \"db\"",[629,809,794],{"class":650},[629,811,813,816,819,822],{"class":631,"line":812},5,[629,814,815],{"class":639},"        \"database\"",[629,817,818],{"class":650},": ",[629,820,821],{"class":643},"\"analytics\"",[629,823,824],{"class":650},",\n",[629,826,828,831,833,836],{"class":631,"line":827},6,[629,829,830],{"class":639},"        \"schemaPolicy\"",[629,832,818],{"class":650},[629,834,835],{"class":643},"\"create_missing_only\"",[629,837,824],{"class":650},[629,839,841,844,846,849],{"class":631,"line":840},7,[629,842,843],{"class":639},"        \"writeMode\"",[629,845,818],{"class":650},[629,847,848],{"class":643},"\"truncate_and_load\"",[629,850,824],{"class":650},[629,852,854,857,859],{"class":631,"line":853},8,[629,855,856],{"class":639},"        \"verifyAfterRun\"",[629,858,818],{"class":650},[629,860,861],{"class":639},"true\n",[629,863,865],{"class":631,"line":864},9,[629,866,867],{"class":650},"      }\n",[629,869,871],{"class":631,"line":870},10,[629,872,873],{"class":650},"    }\n",[629,875,877],{"class":631,"line":876},11,[629,878,879],{"class":650},"  }\n",[629,881,883],{"class":631,"line":882},12,[629,884,885],{"class":650},"}\n",[14,887,888,889,892,893,330,896,899,900,903,904,907,908,911,912,915,916,919,920,924],{},"Starting a stream through the API follows the same review contract as the UI: request the plan with ",[266,890,891],{},"POST \u002Fapi\u002Fv1\u002Fstream-configs\u002F{id}\u002Freadiness",", then pass its ",[266,894,895],{},"fingerprint",[266,897,898],{},"POST \u002Fapi\u002Fv1\u002Fstream-configs\u002F{id}\u002Fstart"," as ",[266,901,902],{},"readinessFingerprint",", with ",[266,905,906],{},"confirmDestructive: true"," when the plan replaces tables. A plan that changed in between is refused with ",[266,909,910],{},"409"," and code ",[266,913,914],{},"stale_plan","; a plan with blocked tables is refused with ",[266,917,918],{},"400"," before a run is created. See the ",[47,921,923],{"href":922},"\u002Fdocs\u002Fapi\u002Freference","API reference"," for the full schemas.",[39,926,928],{"id":927},"related-docs","Related Docs",[18,930,931,936,942],{},[21,932,933],{},[47,934,935],{"href":49},"Target Schema and Data Policies",[21,937,938],{},[47,939,941],{"href":940},"\u002Fdocs\u002Fstreams\u002Fresumable-load","Resumable Load",[21,943,944],{},[47,945,947],{"href":946},"\u002Fdocs\u002Fstreams\u002Fstream-states-lifecycle","Stream States and Lifecycle",[949,950,951],"style",{},"html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":625,"searchDepth":672,"depth":672,"links":953},[954,955,960,961,962,963,964,965],{"id":41,"depth":672,"text":42},{"id":165,"depth":672,"text":166,"children":956},[957,958,959],{"id":192,"depth":695,"text":193},{"id":203,"depth":695,"text":204},{"id":302,"depth":695,"text":303},{"id":396,"depth":672,"text":397},{"id":472,"depth":672,"text":473},{"id":495,"depth":672,"text":496},{"id":544,"depth":672,"text":545},{"id":614,"depth":672,"text":615},{"id":927,"depth":672,"text":928},"See what a Load will do to each target table before it starts, then prove after it finishes that the target holds the same rows as the source.","md",{},false,"\u002Fdocs\u002Fstreams\u002Fverify-load",null,{"title":5,"description":966},"docs\u002Fstreams\u002Fverify-load","SC22fl_Q23lZks0F3WKFH9AHrDROd6uJ3QJOAZDQUBM",1789672327449]