The "Long Fat Pipe" Problem
Imagine a garden hose connected to a fire hydrant. The hydrant can deliver enormous pressure (bandwidth), but if the hose is narrow (small buffer), you can only push so much water through at once. TCP has the same problem.
TCP uses a sliding window to control how much data can be "in flight" (sent but not yet acknowledged). The window size is bounded by the socket's buffer size. If your buffer is 256 KB, you can have at most 256 KB of unacknowledged data in transit — regardless of whether your link is 1 Gbps or 100 Gbps.
The math that makes this concrete is the Bandwidth-Delay Product (BDP):
BDP = Bandwidth × Round-Trip Time
BDP = 10 Gbps × 0.040 seconds (40ms cross-country latency)
= 0.4 Gbit = 400 Mbit = 50 MB
To fully utilize a 10 Gbps link with 40ms RTT, your TCP window must be able to hold 50 MB of data in flight. The Linux default buffer maximum is around 212 KB — roughly 240× too small for this scenario.
Key tunables and what they control: net.core.rmem_max and net.core.wmem_max set the hard ceiling on socket receive/send buffer size — the default on most distros is around 212 KB, far below the 50 MB this scenario needs. net.ipv4.tcp_rmem and tcp_wmem set a min/default/max triple that the kernel auto-tunes each socket's buffer within based on memory pressure. net.ipv4.tcp_window_scaling must stay enabled (the default on modern kernels) for windows larger than 65,535 bytes to be possible at all.
Additional high-throughput tunables: net.core.somaxconn and net.ipv4.tcp_max_syn_backlog raise the backlog queue for incoming connections, preventing drops under burst traffic. net.ipv4.tcp_fastopen enables TCP Fast Open to shave a round trip off the handshake. net.ipv4.tcp_tw_reuse allows TIME_WAIT sockets to be recycled for new outgoing connections, which is safe on servers.
Modern Linux kernels (4.9+) include BBR (Bottleneck Bandwidth and RTT) congestion control, which dramatically improves throughput on long-fat-pipe connections without requiring manual buffer tuning. Enabling it is a matter of switching the queueing discipline to fq and setting tcp_congestion_control to bbr. BBR uses a model of the network path rather than packet loss as a congestion signal, which means it doesn't back off unnecessarily on lossy links. For cloud-to-cloud or cross-datacenter traffic, BBR often outperforms the default CUBIC algorithm significantly.