Why Writes Are So Fast
To understand why Cassandra and ScyllaDB can sustain millions of writes per second, you need to understand the storage engine underneath them: the Log-Structured Merge-Tree (LSM Tree). This is in contrast to RDBMS, which typically use B-Trees — a structure optimized for reads but requiring random disk I/O for writes.
The core insight of LSM trees: sequential writes to disk are orders of magnitude faster than random writes. LSM trees convert all writes into sequential operations.
The write path, step by step:
-
Commit Log (Durability): The write is immediately appended to a sequential Commit Log on disk. This is fast because it is purely sequential. If the server crashes, this log is replayed on restart to recover in-flight writes.
-
MemTable (Speed): The write is also applied to an in-memory data structure called the MemTable. This is where the data lives for fast reads while it is "hot."
-
SSTable Flush (Persistence): When the MemTable reaches a size threshold, it is sorted by key and flushed to disk as an immutable file called an SSTable (Sorted String Table). Once written, an SSTable is never modified.
-
Immutability and Tombstones: Because SSTables are immutable, updates and deletes are not done in-place. An update writes a new version of the record to a new MemTable (which becomes a new SSTable). A delete writes a special marker called a tombstone — a record that says "this key was deleted at this timestamp."
Bloom Filters: Avoiding Unnecessary Disk Reads
Over time, many SSTables accumulate on disk. To read a key, the database must check the MemTable first, then potentially scan through multiple SSTables from newest to oldest. Opening and scanning each SSTable file is expensive.
Each SSTable has an associated Bloom Filter — a small, probabilistic data structure stored in memory. A Bloom Filter can answer one question with certainty: "Is this key definitely not in this SSTable?" If the Bloom Filter says no, the database skips that SSTable entirely. If it says maybe, the database opens the file and checks. Bloom Filters have a configurable false positive rate (they occasionally say "maybe" when the key is not there) but never false negatives (they never say "no" when the key is actually there).
This means most SSTable files are skipped on every read, keeping read performance acceptable despite the write-optimized structure.
Compaction: Keeping the House Clean
Without intervention, you would accumulate thousands of small SSTable files, tombstones would never be purged, and reads would degrade. Compaction is the background process that fixes this.
Compaction reads multiple SSTables, merges them (resolving which version of each key is newest), purges tombstones for data old enough to be safely deleted, and writes a single new, larger SSTable. The old SSTables are then deleted.
Different compaction strategies exist for different workloads:
- STCS (Size-Tiered Compaction Strategy): Groups SSTables of similar size and merges them. Good for write-heavy workloads.
- LCS (Leveled Compaction Strategy): Organizes SSTables into levels with size limits. Better read performance and more predictable space usage, but higher write amplification.
- TWCS (Time-Window Compaction Strategy): Groups SSTables by time window. Ideal for time-series data where old data is rarely updated.
Key Point: LSM trees optimize for sequential writes at the cost of more complex reads and background maintenance (compaction). Compaction consumes significant I/O and CPU. In production, failing to monitor compaction lag — the backlog of uncompacted SSTables — is a common cause of read latency spikes. A Cassandra cluster that cannot keep up with compaction will degrade rapidly under sustained write load.