Cloudflare Engineers Tackle Challenges in Linux Network Stack Management

Cloudflare Engineers Test the Limits of Linux Network Stack

Cloudflare engineers are facing challenges in extending the functionality of soft-unicast, which is crucial for the extensive use of anycast. They have encountered issues with the Netfilter connection tracking module (conntrack) and the Linux socket subsystem, particularly in the context of packet rewriting.

In the case of soft-unicast, it is vital for multiple processes to be aware of the same connections, but Linux architecture makes packet rewriting impossible. As a result, the engineers had to initially employ a local proxy, which generated significant overhead. The solution found was to use the TCP_REPAIR option, typically intended for network connection migration for virtual machines, to describe the entire state of the socket connection, allowing for its