The "Schizophrenic" Network
Anycast has a fundamental tension with stateful protocols, and understanding this is critical for designing reliable systems on top of it.
TCP is stateful. When a TCP connection is established, both endpoints maintain state: sequence numbers, window sizes, connection identifiers. This state is stored in memory on a specific physical server. If a subsequent packet in the same TCP flow is routed to a different physical server (which shares the same Anycast IP but has no knowledge of the connection), that server will reject the packet with a RST (Reset) flag, terminating the connection.
This can happen due to route flapping — a BGP route being withdrawn and re-announced, causing the internet's routing tables to briefly change. Here is the failure sequence:
- User in Paris initiates a TCP connection. BGP routes them to Frankfurt. Frankfurt allocates connection state.
- A BGP route flap occurs — perhaps a fiber cut causes Frankfurt's announcement to briefly disappear.
- The user's next packet is routed to Amsterdam (the next-best path).
- Amsterdam has no record of this TCP connection. It sends RST. The connection dies.
Mitigations in practice:
- BGP route dampening and stability tuning: Operators configure their BGP sessions to avoid unnecessary route withdrawals and use route dampening to suppress flapping announcements.
- Use UDP where possible: DNS uses UDP, which is stateless. A DNS query that reaches a different Anycast node than the one that received the original query simply gets answered independently. This is why Anycast is so well-suited to DNS.
- Short-lived TCP connections: HTTP/1.1 keep-alive and HTTP/2 multiplexing increase the risk window. HTTP/3 (QUIC) is connection-ID based rather than IP-based, making it significantly more resilient to Anycast route changes — a key reason QUIC was designed this way.
- Stable, well-peered networks: Large Anycast operators (Cloudflare, Akamai) have direct peering relationships with thousands of ISPs, reducing the number of BGP hops and the probability of route changes affecting active flows.
A common senior-level interview question: "Why is Anycast used for DNS but not typically for long-lived TCP connections like database connections?" The answer is statefulness. DNS uses UDP — each query/response is independent, so routing to a different server mid-session is harmless. A database connection carries session state (transactions, cursors, authentication) on a specific server. Anycast route changes would corrupt or terminate those sessions unpredictably. For stateful workloads, unicast with load balancers is the correct architecture.