Data Takes Up Space
Think of a database like a warehouse. You can organize the inventory logically — by category, by supplier, by SKU — but at the end of the day, every item takes up physical shelf space, and retrieving items from the back of a massive warehouse takes time. Data works the same way. As a DBRE, it's easy to think of data logically: tables, rows, and columns. But eventually, that data has to live somewhere physically — usually on a spinning hard drive (HDD) or a solid-state drive (SSD).
Data physics is about dealing with two harsh realities of computer hardware:
- Storage costs money: Storing petabytes of data gets expensive. Cloud object storage (like AWS S3) charges per gigabyte-month. A 10x compression ratio is a 10x reduction in your storage bill.
- I/O is slow: Reading data from a disk or moving it across a network is thousands of times slower than processing it in the CPU. A modern CPU can perform billions of operations per second; a disk read takes milliseconds. At scale, I/O is almost always the bottleneck.
The Goals
To build efficient databases, we need to manipulate the physical properties of data to achieve two goals:
- Shrink it: Use compression to make the data take up less physical space, reducing both storage cost and the volume of data that must travel from disk to CPU.
- Organize it: Use smart storage formats (like columnar) so we only have to read the exact bits we need from the slow disk, rather than scanning everything to find what we want.
By mastering data physics, you can make a database faster and cheaper simultaneously — a rare win-win in engineering.
The Layers of the Stack
Data physics operates at multiple levels. Compression can be applied at the file level (compressing an entire Parquet file with Zstd), at the block level (a database page compressed before writing to disk), or at the column level (each column in a columnar format compressed independently). Storage format decisions — row vs. columnar — determine how bytes are physically arranged on disk before any compression is applied. Understanding both layers, and how they interact, is what separates a DBRE from a developer who just runs CREATE TABLE.