System Architecture

Overview

DBConvert Streams is a distributed data processing platform for database migration and real-time replication. The architecture consists of lightweight Go services connected through NATS JetStream, with a Vue.js web UI for management and monitoring.

Services

The backend runs as three independent Go binaries:

API Server (stream-api)

• RESTful API for connection, stream, and configuration management
• Authentication via API keys
• Built-in DuckDB engine for file queries and federated queries
• Data Explorer with direct database querying and DuckDB-powered file/multi-source SQL
• Real-time log streaming via SSE (/logs/stream)
• Stream lifecycle orchestration (create, start, stop)

Source Reader (stream-reader)

• Reads from source databases and files using specialized adapters
• Two reading modes:
  • CDC — captures changes from transaction logs (PostgreSQL WAL, MySQL binlog)
  • Load — direct table reads with configurable bundle sizes (default: 100 rows)
• Supports database sources (MySQL 5.7+, PostgreSQL 12+) and file sources (CSV, JSON, Parquet, S3)
• Publishes data events to NATS JetStream
• Reports progress metrics via NATS

Target Writer (stream-writer)

• Consumes data events from NATS JetStream
• Writes to target destinations: MySQL, PostgreSQL, Snowflake, or file formats (CSV, JSONL, Parquet)
• Handles schema creation and cross-database type mapping
• Scales horizontally — multiple writer instances consume in parallel

User Interface

The UI is a Vue 3 + TypeScript application available in two forms:

• Web UI (stream-ui) — browser-based, served alongside the API or separately
• Desktop app (stream-desktop) — native application built with Wails for Windows, macOS, and Linux; bundles the API server, reader, writer, and NATS into a single process

Both provide the same feature set:

• Connection management and testing
• Stream configuration and monitoring
• Data Explorer with SQL Console, database diagrams, and federated queries
• Real-time log streaming via SSE
• Run history and usage tracking

Infrastructure

NATS JetStream

NATS is the backbone for inter-service communication. The reader publishes data events, and writers consume them in parallel. Messages are persisted for fault tolerance and can be replayed on failure.

Storage backends

The active backend depends on the deployment mode:

Backends are selected via environment variables: CONFIG_BACKEND, SECRETS_BACKEND.

Data flow

1. Initialization

1. Reader retrieves table metadata (columns, indexes, keys) from the source
2. Metadata is published to NATS for the target to create corresponding structures
3. Writer creates tables/schemas on the target where applicable

2. Transfer

1. Reader reads data in configurable bundles and publishes to NATS JetStream
2. Writers consume events in parallel and write to the target
3. Progress metrics are reported via NATS and streamed to the UI via SSE

Data flow diagram

Security

• API key validation for all service access
• SSL/TLS encryption for database connections and data in transit
• Encrypted credential storage (local encrypted file or Vault, depending on deployment mode)
• Client certificate support for database connections

See SSL Configuration for details.

Deployment options

Desktop application

• Lightweight stack: SQLite + encrypted secrets file + embedded NATS
• No external services required
• Available for Windows, macOS, and Linux

Docker

• Complete docker-compose configurations with NATS included
• Optional Consul and Vault for distributed deployments
• Runs on any platform that supports Docker: AWS, GCP, Azure, DigitalOcean, on-premises

Database support

Sources

• MySQL 5.7+ — full table snapshots, CDC via binlog
• PostgreSQL 12+ — full snapshots, CDC via logical replication
• Files — CSV, JSON, Parquet (local or S3-compatible storage)

Targets

• MySQL 5.7+ — direct writes with transaction support
• PostgreSQL 12+ — direct writes with conflict resolution
• Snowflake — batch loading with staging tables
• Files — CSV, JSONL, Parquet output formats

Requirements

• MySQL CDC requires binary logging enabled
• PostgreSQL CDC requires logical replication configuration
• Appropriate database permissions needed for CDC/replication

See Licensing, Seats, and Telemetry for API key and access details.