[{"data":1,"prerenderedAt":422},["ShallowReactive",2],{"docs-\u002Fdocs\u002Fstreams\u002Fcdc-reliability-and-resume":3},{"id":4,"title":5,"body":6,"description":413,"extension":414,"meta":415,"navigation":416,"path":417,"redirect":418,"seo":419,"stem":420,"__hash__":421},"docs\u002Fdocs\u002Fstreams\u002Fcdc-reliability-and-resume.md","CDC Reliability and Resume",{"type":7,"value":8,"toc":397},"minimark",[9,13,17,20,25,28,50,53,57,60,65,74,78,85,88,92,95,104,107,110,114,117,139,143,146,149,200,204,207,210,214,217,220,224,231,256,262,266,269,338,341,361,364,368,371,390,393],[10,11,5],"h1",{"id":12},"cdc-reliability-and-resume",[14,15,16],"p",{},"CDC mode keeps supported MySQL and PostgreSQL sources synchronized by reading committed row changes from the source log and applying them to the target.",[14,18,19],{},"This page explains what happens in the common rollout cases: starting from an empty target, copying existing rows before CDC, combining Load with CDC, stopping a stream, and starting it again later.",[21,22,24],"h2",{"id":23},"starting-cdc-on-an-empty-or-prepared-target","Starting CDC on an empty or prepared target",[14,26,27],{},"Use this flow when the target table is empty, already prepared, or intentionally starts from the current source position.",[29,30,31,35,38,41,44,47],"ol",{},[32,33,34],"li",{},"Create a CDC stream for the source and target table.",[32,36,37],{},"Start the stream.",[32,39,40],{},"Insert, update, or delete rows on the source.",[32,42,43],{},"DBConvert Streams reads committed changes from MySQL binlog or PostgreSQL WAL.",[32,45,46],{},"Target writers apply the changes.",[32,48,49],{},"The stream checkpoint advances after the target write succeeds.",[14,51,52],{},"CDC captures changes made after the stream's start position. It does not automatically copy older rows that already existed in the source table.",[21,54,56],{"id":55},"existing-source-data","Existing source data",[14,58,59],{},"If the source table already contains data, choose one of these rollout paths.",[61,62,64],"h3",{"id":63},"built-in-initial-load-cdc","Built-in Initial Load + CDC",[14,66,67,68,73],{},"Use ",[69,70,72],"a",{"href":71},"\u002Fdocs\u002Fstreams\u002Finitial-load-and-cdc","Initial Load + CDC"," when the target tables are empty and you want one CDC run to copy existing rows before it continues from the matching source log position.",[61,75,77],{"id":76},"load-first-then-cdc","Load first, then CDC",[14,79,67,80,84],{},[69,81,83],{"href":82},"\u002Fdocs\u002Fintegration\u002Fconversion-mode","Load mode"," first when the initial load is a separate migration task, the target already contains managed data, or you need Load-only features such as custom SQL, multi-source input, files, or S3 sources.",[14,86,87],{},"After Load finishes, validate the target and start CDC from the point you choose for the cutover.",[21,89,91],{"id":90},"stop-and-start-again-later","Stop and start again later",[14,93,94],{},"When you stop a CDC stream, DBConvert Streams keeps the CDC checkpoint for that stream config. If you start the same config later, it resumes from the saved position:",[96,97,98,101],"ul",{},[32,99,100],{},"MySQL resumes from the saved binlog file and position.",[32,102,103],{},"PostgreSQL resumes from the saved LSN, with the replication slot also helping retain source-side state.",[14,105,106],{},"This means changes made while the stream was stopped can be applied after the next start, as long as the source database retained the required log range.",[14,108,109],{},"If the stream used Initial Load + CDC and the bootstrap phase already completed, the next start resumes normal CDC. It does not copy the initial snapshot again.",[21,111,113],{"id":112},"what-must-be-retained","What must be retained",[14,115,116],{},"To resume safely after a stop or service restart:",[96,118,119,122,130,133,136],{},[32,120,121],{},"Keep the stream config. Deleting the config also deletes its CDC checkpoint and (since v2.4.2) drops its PostgreSQL replication slot.",[32,123,124,125,129],{},"Keep the embedded NATS JetStream KV storage (the ",[126,127,128],"code",{},"streams"," bucket) intact. Deleting it removes all checkpoints.",[32,131,132],{},"For MySQL, keep binary logs long enough for the downtime window.",[32,134,135],{},"For PostgreSQL, keep logical replication enabled and retain the replication slot\u002FWAL needed by the stream.",[32,137,138],{},"Keep the target schema compatible with source changes produced during the stopped period.",[61,140,142],{"id":141},"postgresql-slot-wal-retention-and-disk-growth","PostgreSQL slot WAL retention and disk growth",[14,144,145],{},"The replication slot is what lets a stopped stream resume without losing changes — but it works by making PostgreSQL retain WAL from the saved position onward. While the stream is stopped, retained WAL keeps growing. A slot that is never resumed (for example, a stream you stopped and forgot) can eventually fill the source disk.",[14,147,148],{},"Two protections:",[96,150,151,158],{},[32,152,153,157],{},[154,155,156],"strong",{},"Delete configs you no longer need."," Since v2.4.2, deleting a CDC stream config also drops its inactive replication slot on the source, so nothing is left behind to retain WAL.",[32,159,160,163,164,187,190,191,195,196,199],{},[154,161,162],{},"Set a retention cap on the source"," (PostgreSQL 13+):",[165,166,171],"pre",{"className":167,"code":168,"language":169,"meta":170,"style":170},"language-sql shiki shiki-themes github-light github-dark","ALTER SYSTEM SET max_slot_wal_keep_size = '10GB';\nSELECT pg_reload_conf();\n","sql","",[126,172,173,181],{"__ignoreMap":170},[174,175,178],"span",{"class":176,"line":177},"line",1,[174,179,180],{},"ALTER SYSTEM SET max_slot_wal_keep_size = '10GB';\n",[174,182,184],{"class":176,"line":183},2,[174,185,186],{},"SELECT pg_reload_conf();\n",[188,189],"br",{},"If a slot exceeds the cap, PostgreSQL invalidates it instead of filling the disk. The trade-off: an invalidated slot cannot resume — the stream needs a ",[69,192,194],{"href":193},"#reset-to-start-from-scratch","reset"," and a fresh start. The default (",[126,197,198],{},"-1",", unlimited) favors resumability over disk safety; pick a cap that covers your longest expected downtime window.",[21,201,203],{"id":202},"transaction-behavior","Transaction behavior",[14,205,206],{},"CDC publishes committed row changes. Rolled back transactions are not applied to the target.",[14,208,209],{},"For database targets, messages are acknowledged only after the target write succeeds. If the target write fails because of a deterministic problem such as a schema mismatch or constraint error, the stream fails instead of retrying the same write in a loop. Fix the target-side cause, then start the stream again to continue from the last durable checkpoint.",[21,211,213],{"id":212},"ordering","Ordering",[14,215,216],{},"CDC preserves event order per table. Multiple target workers can process different tables in parallel, but events for the same table are routed through an ordered path.",[14,218,219],{},"The guarantee is per-table ordering, not a single global order across every table in the stream.",[21,221,223],{"id":222},"monitor-visibility","Monitor visibility",[14,225,226,227,230],{},"The ",[154,228,229],{},"Monitor"," tab shows CDC reliability status for active CDC streams:",[96,232,233,244,247,250,253],{},[32,234,235,236,239,240,243],{},"source engine (",[126,237,238],{},"MySQL"," or ",[126,241,242],{},"PostgreSQL",")",[32,245,246],{},"current checkpoint position",[32,248,249],{},"last checkpoint update time",[32,251,252],{},"acknowledged stream sequence",[32,254,255],{},"health badge",[257,258],"themed-image",{"alt":259,"dark":260,"light":261},"Monitor tab showing PostgreSQL CDC stream with HEALTHY badge, current LSN checkpoint, and replication slot name","\u002Fimages\u002Fdocs\u002Fstreams\u002Fcdc-monitor-postgres-dark.png","\u002Fimages\u002Fdocs\u002Fstreams\u002Fcdc-monitor-postgres-light.png",[21,263,265],{"id":264},"reset-to-start-from-scratch","Reset to start from scratch",[14,267,268],{},"When you need to rerun a CDC stream from the very beginning — clearing partial target data and all saved checkpoints — use the reset endpoint:",[165,270,274],{"className":271,"code":272,"language":273,"meta":170,"style":170},"language-bash shiki shiki-themes github-light github-dark","curl -X POST \\\n  -H \"X-API-Key: $API_KEY\" \\\n  -H \"X-Confirm-Reset: true\" \\\n  \"$HOST\u002Fapi\u002Fv1\u002Fstream-configs\u002F$CONFIG_ID\u002Freset\"\n","bash",[126,275,276,293,310,320],{"__ignoreMap":170},[174,277,278,282,286,290],{"class":176,"line":177},[174,279,281],{"class":280},"sScJk","curl",[174,283,285],{"class":284},"sj4cs"," -X",[174,287,289],{"class":288},"sZZnC"," POST",[174,291,292],{"class":284}," \\\n",[174,294,295,298,301,305,308],{"class":176,"line":183},[174,296,297],{"class":284},"  -H",[174,299,300],{"class":288}," \"X-API-Key: ",[174,302,304],{"class":303},"sVt8B","$API_KEY",[174,306,307],{"class":288},"\"",[174,309,292],{"class":284},[174,311,313,315,318],{"class":176,"line":312},3,[174,314,297],{"class":284},[174,316,317],{"class":288}," \"X-Confirm-Reset: true\"",[174,319,292],{"class":284},[174,321,323,326,329,332,335],{"class":176,"line":322},4,[174,324,325],{"class":288},"  \"",[174,327,328],{"class":303},"$HOST",[174,330,331],{"class":288},"\u002Fapi\u002Fv1\u002Fstream-configs\u002F",[174,333,334],{"class":303},"$CONFIG_ID",[174,336,337],{"class":288},"\u002Freset\"\n",[14,339,340],{},"Reset performs the following in order:",[29,342,343,349,355],{},[32,344,345,348],{},[154,346,347],{},"Truncates target rows"," for CDC streams that used Initial Load + CDC. Plain CDC streams (no initial snapshot) are not affected.",[32,350,351,354],{},[154,352,353],{},"Drops the inactive PostgreSQL replication slot"," for Initial Load + CDC streams on a PostgreSQL source. Fails if the slot is still active — stop the stream first.",[32,356,357,360],{},[154,358,359],{},"Clears all durable recovery state",": CDC checkpoint, CDC bootstrap state, and any resumable load-progress cursors.",[14,362,363],{},"After reset, the next start runs as a clean first run. For plain CDC without initial snapshot, reset is not needed after a normal stop — just start again and it resumes from the saved checkpoint.",[21,365,367],{"id":366},"not-covered-by-the-current-release","Not covered by the current release",[14,369,370],{},"The current CDC reliability work does not include:",[96,372,373,376,379,382],{},[32,374,375],{},"exactly-once delivery",[32,377,378],{},"automatic source schema evolution after stream start",[32,380,381],{},"automatic repair of target constraint or schema errors",[32,383,384,385,389],{},"automatic recovery from an interrupted plain-path initial snapshot (MySQL chunked bootstrap tables do resume — see ",[69,386,388],{"href":387},"\u002Fdocs\u002Fstreams\u002Fresumable-load","Resumable Load","; PostgreSQL bootstrap and MySQL plain-path tables require reset)",[14,391,392],{},"For schema changes, apply and validate target schema changes as part of your own operational procedure.",[394,395,396],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}",{"title":170,"searchDepth":183,"depth":183,"links":398},[399,400,404,405,408,409,410,411,412],{"id":23,"depth":183,"text":24},{"id":55,"depth":183,"text":56,"children":401},[402,403],{"id":63,"depth":312,"text":64},{"id":76,"depth":312,"text":77},{"id":90,"depth":183,"text":91},{"id":112,"depth":183,"text":113,"children":406},[407],{"id":141,"depth":312,"text":142},{"id":202,"depth":183,"text":203},{"id":212,"depth":183,"text":213},{"id":222,"depth":183,"text":223},{"id":264,"depth":183,"text":265},{"id":366,"depth":183,"text":367},"Understand how DBConvert Streams checkpoints CDC streams, resumes after restarts, and handles initial loads.","md",{},false,"\u002Fdocs\u002Fstreams\u002Fcdc-reliability-and-resume",null,{"title":5,"description":413},"docs\u002Fstreams\u002Fcdc-reliability-and-resume","VYxus_-fEQm0BCDCCNM_5ZSr37IyiLaYDTwiQ_cB5tg",1783896283528]