Docs/Resources

Data Transfer Workflow

This page describes what happens when a stream runs — from initialization through data transfer to completion.

Workflow phases

1. Initialization

  1. The source reader connects to the source database (or file/S3 storage)
  2. The target writer connects to the target destination
  3. Both components connect to NATS JetStream for event delivery
  4. The reader retrieves table metadata (columns, indexes, keys) from the source
  5. The writer creates corresponding structures on the target if needed

2. Data reading

The source reader groups rows into bundles (default: 100 rows, configurable in stream settings) and publishes each bundle as an event to NATS.

In Load mode, the reader performs direct table reads. In CDC mode, it captures changes from the database transaction log (PostgreSQL WAL or MySQL binlog).

3. Data transfer

Events flow through NATS JetStream to the target writer:

  • Deduplication — each bundle gets a unique key derived from its content; if the same bundle arrives twice (e.g., after a retry), the duplicate is skipped
  • Ordering — ordered events are processed sequentially to maintain consistency; unordered events can be processed in parallel for higher throughput
  • Persistence — NATS persists events so they can be replayed on failure, ensuring no data loss

4. Data writing

The target writer processes events using a pool of concurrent workers.

In Load mode, the writer performs bulk INSERT (or upsert) operations. For file/S3 targets, data is written as stateful snapshots.

In CDC mode, the writer also handles UPDATE and DELETE events, matching existing records by primary key.

Database operations are wrapped in transactions and committed or rolled back as a unit. Failed operations are retried where possible. Streams can be paused and resumed without losing progress.

5. Monitoring

Throughout the transfer, the system tracks and reports:

  • Per-table progress: event count, data size, completion percentage
  • Overall throughput: total events, processing rate, elapsed time
  • Source vs. target event counts to detect processing gaps

Progress is streamed to the UI in real time via SSE. See Observability for details on the Monitor tab and logs panel.

6. Completion

  1. The reader signals that all data has been read
  2. A grace period allows the writer to finish processing remaining events
  3. Final statistics are reported
  4. The stream status is set to FINISHED

If an error occurs, the status is set to FAILED with diagnostic details in the logs.

Data flow diagram

Using Data Explorer in your workflow

Data Explorer can help at each stage:

  • Before transfer — examine source schema, verify keys and indexes, review sample data
  • During setup — verify target structures were created correctly, compare source and target definitions
  • After transfer — compare data between source and target, check row counts, run validation queries
PreviousSystem ArchitectureNextGlossary