There Is No "Best" Algorithm
Compression is not magic; it's a trade-off. Think of it like packing a suitcase: you can pack it loosely in 30 seconds, or spend 20 minutes folding everything perfectly to fit twice as much. The "best" approach depends entirely on how much time you have and how much space you need to save. When choosing a compression algorithm for a database, you are balancing three competing factors. You can usually only optimize for two.
- Compression Ratio: How much smaller does the data get? (e.g., 50% smaller, 80% smaller). A higher ratio means less disk space and less I/O.
- Compression Speed: How much CPU time does it take to shrink the data when writing? Slow compression means slower ingestion pipelines.
- Decompression Speed: How much CPU time does it take to expand the data when reading? Slow decompression means slower queries.
The DBRE's Perspective
For databases, decompression speed is often the most critical factor. You usually write data once but read it many times. If reading data burns 100% of your CPU just trying to decompress it, your query performance will tank even if your disk I/O is low. This is the most common mistake engineers make when choosing compression: they optimize for the write path (ingestion speed or storage ratio) and forget that every query pays the decompression cost on every read.
A useful mental model: compression ratio reduces I/O cost (less data to move from disk), while decompression speed determines CPU cost (how fast you can process what you read). The right algorithm depends on which resource is your bottleneck.
Interviewers often ask: "How would you choose a compression algorithm for a high-throughput analytics system?" The trap is to say "the one with the best compression ratio." The correct answer is to reason through the bottleneck: if the system is I/O-bound, prioritize ratio; if it's CPU-bound on reads, prioritize decompression speed. Zstd is usually the right starting point because it lets you tune the level to shift between these extremes without changing the algorithm.