Buffered I/O vs. Direct I/O

The Great Lie of write()

When you call write() in your program, you might reasonably assume the data is on its way to the disk. It is not. By default, Linux uses Buffered I/O, which means the kernel copies your data into a region of RAM called the Page Cache and immediately returns control to your program. From your program's perspective, the write is done. From the disk's perspective, nothing has happened yet.

Think of it like dropping a letter in an office outbox. You have done your part — the letter is "sent" as far as you are concerned — but it will not actually leave the building until the mail courier comes by later.

The OS periodically flushes "dirty" pages (pages in the cache that differ from what is on disk) to the physical device. This is controlled by kernel parameters:

• vm.dirty_ratio: The percentage of total RAM that can be dirty before the kernel forces a synchronous flush (blocks the writing process).
• vm.dirty_background_ratio: The percentage at which background flusher threads (kworker/flush) start writing dirty pages without blocking the application.
• vm.dirty_writeback_centisecs: How often (in centiseconds) the flusher wakes up. Default is 500 (every 5 seconds).

These are all readable at runtime via sysctl.

The risk: If the machine loses power between your write() call and the flush, your data is gone. The kernel made no promises about durability — only about accepting the data.

Forcing Durability: fsync() and fdatasync()

To guarantee data is on disk, applications call fsync(fd). This is a blocking system call that tells the kernel: "Flush all dirty pages for this file to the physical device and do not return until the device confirms the write." Databases call fsync() at transaction commit boundaries to guarantee ACID durability.

fdatasync() is a lighter variant — it flushes the data pages but skips updating metadata (like atime) unless the metadata is needed to retrieve the data. It is faster and sufficient for most durability needs.

Direct I/O (O_DIRECT)

High-performance databases (PostgreSQL, MySQL InnoDB, Oracle) manage their own internal buffer pools. They do not want the OS caching data that they are already caching — this wastes RAM on two copies of the same blocks. They open files with the O_DIRECT flag:

int fd = open("datafile", O_RDWR | O_DIRECT);

With O_DIRECT, the kernel bypasses the Page Cache entirely. Data moves directly between the application's memory buffer and the disk controller via DMA (Direct Memory Access) — the hardware copies data without involving the CPU for each byte. The application's buffer must be aligned to the block size (typically 512 bytes or 4096 bytes), which is why databases use aligned memory allocators.

Trade-off summary: