Systematic Approach to TCP Performance
TCP/IP tuning is not a single knob — it is a system of interacting parameters. Changing one without understanding the others can create new problems. The following checklist represents a production-ready baseline for a Linux server on a high-speed WAN link.
Step 1: Calculate your BDP
BDP (bytes) = Bandwidth (bytes/sec) × RTT (seconds)
This number is your target buffer size. Everything else follows from it.
Step 2: Size the buffers to match BDP
The buffer maximums have to be raised to roughly 2× the calculated BDP, leaving headroom for multiple concurrent connections — the same default-ceiling problem discussed under Window Scaling above.
Step 3: Enable window scaling and auto-tuning
Window scaling needs to be on, and the kernel's receive-buffer auto-tuning needs to be allowed to grow buffers dynamically within those limits rather than sitting fixed at the default.
Step 4: Select congestion control
BBR for WAN/high-RTT links, Cubic for low-latency datacenter traffic — and if BBR is selected, the fq queue discipline must be paired with it, as covered earlier.
Step 5: Enable ECN
So congestion gets signaled by marking instead of dropping wherever both endpoints support it.
Step 6: Apply queue discipline
FQ_CoDel for mixed traffic types sharing a link; plain fq if the goal is pure throughput on a BBR connection.
Step 7: Make it persistent
Every setting above is a runtime-only change until it's written into a systemd sysctl drop-in file and reapplied — otherwise the entire tuning profile reverts on the next reboot.
What not to tune blindly:
net.ipv4.tcp_timestamps— disabling this breaks PAWS (Protection Against Wrapped Sequence numbers) and can cause data corruption on high-speed links. Leave it enabled.net.ipv4.tcp_sack— disabling Selective ACK forces full retransmission of entire windows on loss. Never disable this.- Setting buffers to arbitrarily large values without considering total system memory. Each TCP connection consumes memory up to the buffer max. On a server with 10,000 concurrent connections and a 32 MB buffer max, the kernel could theoretically allocate 320 GB — it won't (auto-tuning prevents this), but understanding the relationship matters.
Key Point: TCP tuning is a system, not a collection of independent knobs. Always start with BDP, derive buffer requirements from it, then layer congestion control and queue management on top. Measure before and after every change.