When Speed is Everything
Sometimes, you are limited not by disk space, but by how fast you can push data through the CPU. Imagine a real-time log ingestion pipeline processing 500,000 events per second. If your compression algorithm can't keep up with the ingest rate, you'll build a backlog. In these scenarios, you need algorithms that compress and decompress at speeds approaching raw memory bandwidth.
LZ4 and Google's Snappy are algorithms designed specifically for speed. They don't try to squeeze every last bit out of the data. Instead, they aim for "good enough" compression (typically 30–50% reduction) at breakneck speeds.
- LZ4 can decompress at speeds exceeding 4 GB/s on a single CPU core — faster than many SSDs can deliver data. It is the default compression in systems like ClickHouse, RocksDB, and Kafka (as an option).
- Snappy was developed by Google for internal use and prioritizes reasonable compression with very low CPU overhead. It is the default in many Hadoop-ecosystem tools and Cassandra.
Use Cases:
- Real-time data ingestion pipelines where ingest rate > storage cost
- In-memory caches (Redis supports LZ4) where decompression happens on every cache hit
- Inter-service communication where network bandwidth is high but latency must be low
- Write-heavy workloads where the write path is the bottleneck
If your database CPU is running at 90% usage while your disk I/O is underutilized, switching to a faster algorithm like LZ4 might solve your problem by freeing CPU cycles that were being spent on decompression.
The Practical Limit
The trade-off is real: LZ4 on typical database data might achieve a 2:1 compression ratio, while Zstd at a high level might achieve 5:1 or better on the same data. For a 10 TB dataset, that's the difference between 5 TB and 2 TB of storage. At $0.023/GB-month on S3, that's roughly $69/month vs. $46/month — a meaningful difference at scale.