Compressed File Support

DBConvert Streams handles compression on both sides of a stream:

Reading — compressed source files are decompressed transparently with no configuration needed
Writing — file and S3 targets support configurable output compression

Reading compressed files

The system detects, decompresses, and processes files using streaming techniques that keep memory usage low regardless of file size.

Supported read formats

Compression	Extension	Typical ratio
Gzip	`.gz`	5-10x for CSV, 3-8x for JSON
Zstandard	`.zst`	Best balance of ratio and speed
Bzip2	`.bz2`	Highest ratio, slower decompression

Works with CSV, JSON, and JSONL files:

data.csv.gz, export.csv.bz2, logs.csv.zst
logs.json.gz, backup.json.bz2
stream.jsonl.gz, events.jsonl.zst

Parquet files use built-in compression (Snappy, Zstd, Gzip) at the column-chunk level — they are not wrapped in .gz or .zst. DBConvert Streams reads Parquet compression natively.

Compression is recognized from the file suffix, such as .gz, .bz2, or .zst. DBConvert Streams then uses the remaining extension, such as .csv or .jsonl, to choose the correct reader. For example, data.csv.gz is treated as a gzip-compressed CSV file.

Processing

Compressed files are read through the normal file-reading path. DBConvert Streams identifies the compression and file format from the path and uses the matching reader.

For local files, the application does not create intermediate uncompressed files on disk while reading compressed inputs.

Using compressed files

Compressed files work transparently with existing workflows:

Data Explorer — browse and preview compressed files the same way as uncompressed ones
SQL Console — query compressed files directly
Stream sources — use compressed files as stream input
API — the /api/v1/files/data and /api/v1/files/meta endpoints handle compressed files automatically

No special configuration is needed. Point to a compressed file path and the system handles the rest.

Error handling

If compression format cannot be determined, the file is treated as uncompressed
Corrupted compressed files produce clear error messages distinguishing compression errors from data format errors
Built-in safeguards prevent memory overflow from malformed data

Writing compressed output

File and S3 stream targets support output compression via format.compression in the target spec.

CSV and JSONL targets

Compression	Value	Notes
None	`uncompressed`	Default
Gzip	`gzip`	Wide compatibility
Zstandard	`zstd`	Smaller files with `.zst` output

"spec": {
  "files": {
    "fileFormat": "csv",
    "format": {
      "compression": "gzip"
    }
  }
}

Parquet targets

Parquet compression is controlled with format.compression:

Compression	Value	Notes
Zstandard	`zstd`	Default — best balance of ratio and speed
Snappy	`snappy`	Fastest compression/decompression
Gzip	`gzip`	Highest ratio, slower
None	`uncompressed`	No compression

"spec": {
  "files": {
    "fileFormat": "parquet",
    "format": {
      "compression": "snappy"
    }
  }
}