The Modern Default
For a long time, the standard for "good compression" was gzip (zlib). It compressed well but was slow — often 10–20x slower than LZ4 at decompression. Engineers faced a painful choice: fast and large (LZ4) or slow and small (gzip).
Then Facebook released Zstandard (Zstd) in 2016. Zstd changed the game because it offers a better compression ratio than gzip, but at decompression speeds approaching LZ4. It occupies the "sweet spot" in the middle of the trade-off triangle. Zstd is now the default or recommended algorithm in PostgreSQL 15+, ClickHouse, Kafka (as an option), Linux kernel modules, and many other systems.
Tuneability and Compression Levels
Zstd exposes a compression level parameter ranging from 1 to 22:
- Level 1–3: Near-LZ4 speeds with better ratio than LZ4. Good for hot data.
- Level 3–9: The "general purpose" range. Level 3 is the default in most tools.
- Level 10–19: Significantly slower compression, meaningfully better ratio. Good for archival data written infrequently.
- Level 20–22 (ultra): Extremely slow compression, maximum ratio. Only appropriate for cold archival where write happens once and storage cost dominates.
Critically, decompression speed is nearly constant across all levels. Compressing at level 19 is much slower than level 3, but decompressing level-19 data is only marginally slower than level-3 data. This means you can crank up the compression level for archival data without paying a read-time penalty.
Dictionary Training: The Superpower for Small Records
Compression works by finding repeated patterns within the data being compressed. In tiny records (like individual JSON log events of 200–500 bytes), there simply isn't enough data for the algorithm to find patterns. A 200-byte JSON blob might compress to 190 bytes — nearly useless.
Zstd solves this with dictionary training. You feed Zstd a representative sample of your data (say, 100,000 example records), and it analyzes them to extract a "dictionary" of common patterns — field names like "timestamp", "severity", "message", common values, etc. — via a single zstd --train pass over the sample files. This dictionary is stored separately and referenced (with -D) at both compression and decompression time. With a trained dictionary, small records that previously compressed poorly can achieve 3–5x compression ratios.
The dictionary must be available at both compression and decompression time, which adds operational complexity — but for high-volume small-record workloads, the storage savings justify it.
A common interview question is: "We're storing millions of small JSON events and compression isn't helping much. What would you do?" The answer is Zstd dictionary training. Explain that standard compression fails on small records because there's insufficient data for pattern detection, and that a pre-trained dictionary solves this by providing external pattern context. This demonstrates deep knowledge of compression mechanics beyond just "pick an algorithm."