TCP SYN cookies are a clever and widely used defense mechanism against SYN flood attacks, a common type of Denial-of-Service (DoS) attack. To understand SYN cookies, it’s essential to first grasp the TCP 3-way handshake and how SYN flood attacks exploit it.

The TCP 3-Way Handshake (Normal Operation):

1. SYN (Synchronize): A client (e.g., your web browser) sends a SYN packet to a server to initiate a TCP connection.

2. SYN-ACK (Synchronize-Acknowledge): The server receives the SYN, allocates resources (memory in its SYN queue or TCP connection backlog for a “half-open” connection), and responds with a SYN-ACK packet. This SYN-ACK contains the server’s Initial Sequence Number (ISN) for the connection.

3. ACK (Acknowledge): The client receives the SYN-ACK, allocates its own resources, and responds with an ACK packet. This ACK confirms receipt of the SYN-ACK and acknowledges the server’s ISN (by sending server_ISN + 1 as its Acknowledgment Number). At this point, the connection is fully ESTABLISHED.

The SYN Flood Attack:

A SYN flood attack exploits step 2 of the handshake:

• An attacker sends a large volume of SYN packets to a target server. Often, these SYN packets have spoofed (fake) source IP addresses.

• The server, dutifully, responds to each SYN with a SYN-ACK and allocates resources (a Transmission Control Block – TCB, memory, etc.) for each half-open connection in its SYN queue.

• Because the source IP addresses are spoofed (or belong to machines that won’t respond, like unroutable IPs), the server’s SYN-ACKs go unanswered. The attacker never sends the final ACK.

• The server’s SYN queue rapidly fills up with these half-open connections. Once the queue is full, the server can no longer accept new, legitimate incoming connections, leading to a Denial of Service. The server becomes unresponsive to real users.

How TCP SYN Cookies Work:

SYN cookies are a technique to mitigate SYN flood attacks by avoiding resource allocation for half-open connections until the final ACK packet from the client is received and verified as legitimate.

Here’s the clever part:

1. Server Receives SYN: When the server receives a SYN packet, and its SYN queue is either full or nearing its capacity, it doesn’t immediately allocate resources.

2. Calculates a “Cookie” (ISN Encoding): Instead of storing the connection state in its memory, the server calculates a special Initial Sequence Number (ISN) for the SYN-ACK packet. This ISN is not truly random; it’s a “cookie” that encodes essential information about the connection using a cryptographic hash function. The cookie typically encodes:

   • A timestamp (a small, slowly incrementing time value, e.g., every 64 seconds, taking up a few bits of the ISN).

   • The Maximum Segment Size (MSS) that the server would have advertised (a few bits).

   • A cryptographic hash (e.g., MD5 or SHA) of the client’s IP address and port, the server’s IP address and port, and the timestamp. This hash forms the bulk of the ISN (e.g., 24 bits).

   • The server also includes the client’s original ISN in the hash to tie it to the specific SYN packet.

3. Sends SYN-ACK with Cookie: The server sends this specially crafted SYN-ACK packet back to the client. Crucially, it does NOT store any state for this half-open connection in its memory. It effectively “forgets” about the SYN request immediately after sending the SYN-ACK.

4. Client Receives SYN-ACK and Responds with ACK: A legitimate client receives the SYN-ACK. As per TCP rules, it acknowledges the server’s ISN by incrementing it by 1 and sends this as the Acknowledgment Number in its final ACK packet.

5. Server Reconstructs State from ACK: When the server receives this final ACK packet, it extracts the Acknowledgment Number. It then decrements this number by 1 to reveal the original SYN cookie (the ISN it sent).

   • The server then re-computes the cryptographic hash using the information from the incoming ACK (client IP/port, server IP/port) and the timestamp decoded from the cookie.

   • It compares its re-computed hash with the hash part of the cookie.

   • It also checks if the timestamp is recent enough (not expired).

6. Connection Establishment (If Valid Cookie):

   • If the cookie is valid (the re-computed hash matches, and the timestamp is fresh), the server knows this is a legitimate client responding to its SYN-ACK.

   • Only at this point does the server allocate resources for the now-established TCP connection. It decodes the MSS value from the cookie and reconstructs the necessary connection state.

   • If the cookie is invalid (e.g., from a spoofed SYN flood where no ACK is sent, or a random ACK from an attacker), the server simply drops the ACK packet and does not allocate any resources.

Advantages of SYN Cookies:

• Mitigates SYN Flood Attacks: By deferring resource allocation until the final ACK is received, SYN cookies prevent the server’s SYN queue from filling up with fake connections, thus protecting against resource exhaustion.

• Stateless Protection: The server doesn’t need to maintain state for half-open connections, making it highly resilient to a high volume of malicious SYNs.

• Backward Compatible: SYN cookies are compliant with TCP specifications and do not require any changes to the client-side TCP stack. A legitimate client behaves exactly the same way, whether the server is using SYN cookies or not.

Limitations and Considerations:

• Loss of Some TCP Options: Because the server doesn’t store state, it might not be able to fully negotiate certain advanced TCP options (like large windows or specific selective acknowledgments) during the initial handshake, as these options might not be fully encodable within the cookie. While not a severe limitation for basic connections, it can slightly impact performance for very high-performance scenarios.

• CPU Overhead: Calculating the cryptographic hash for every SYN-ACK can introduce a slight CPU overhead on the server, especially under heavy load. However, this is usually preferable to a full DoS.

• Timestamp Sensitivity: The timestamp component means that the server will reject ACKs that arrive too late (after the timestamp in the cookie expires).

• Potential for Information Leakage (Theoretical): While highly unlikely given modern cryptographic functions, there’s a theoretical possibility of an attacker trying to reverse-engineer the cookie’s hash function if they observe enough valid cookies.

SYN cookies are a highly effective and widely deployed defense mechanism in operating systems (like Linux kernels), firewalls, and load balancers to protect servers from SYN flood attacks. They represent a clever workaround that leverages TCP’s flexibility to secure the handshake without consuming excessive server resources.