Steering Packets to Where the Application Lives
RSS and RPS solve the load distribution problem, but they introduce a subtler performance problem: cache misses. Understanding why requires a brief detour into CPU cache architecture.
Each CPU core has its own private L1 and L2 caches. When Core 0 processes a packet and writes the data into a socket buffer, that data lives in Core 0's cache. If the application consuming that data (say, Nginx) is scheduled on Core 3, Core 3 must fetch the data from Core 0's cache or from main RAM — a process called a cache miss that costs 100–300 CPU cycles compared to ~4 cycles for an L1 hit. At millions of packets per second, this adds up to measurable latency and throughput loss.
RFS (Receive Flow Steering), introduced in kernel 2.6.35 alongside RPS, solves this by tracking which CPU core is currently running the application that owns each socket. The kernel maintains a flow table (a hash table mapping 4-tuples to CPU IDs) that is updated every time a socket's recvmsg() system call runs. When a new packet arrives and RPS would normally pick a core based on the hash, RFS overrides that decision: it looks up the flow in the table and steers the packet to whichever core is actually running the application.
RFS requires two configuration knobs: a global flow table size (/proc/sys/net/core/rps_sock_flow_entries, the total number of flows to track system-wide) and a per-queue flow table size (rps_flow_cnt under that queue's sysfs directory, which must be a power of two and no larger than the global setting).
The combined effect of RPS + RFS is that:
- RPS ensures packets are not all processed on Core 0.
- RFS ensures packets are processed on the same core as the application, maximizing L1/L2 cache reuse.
In practice, RFS works best when applications are pinned to specific cores (using taskset or CPU affinity settings), because a frequently migrated process defeats the flow table's predictions. For high-performance servers, combining RFS with application-level CPU pinning can reduce packet processing latency by 20–40% compared to RPS alone.
A common interview question is: "What's the difference between RPS and RFS?" The key distinction is what they optimize for. RPS optimizes for load distribution — spreading interrupt processing across cores. RFS optimizes for data locality — ensuring the core that processes the packet is the same core running the application, so the data is already in cache when the application reads it. RFS is built on top of RPS; you need both enabled for RFS to work.