The idea: IKE agrees, ESP protects

IPsec protects IP packets between two peers. It has two parts. IKEv2 (Internet Key Exchange, RFC 7296) is a small protocol on UDP port 500 that the two peers run first: they agree on algorithms, make shared keys and prove who they are. ESP (Encapsulating Security Payload, RFC 4303) then protects each data packet with those keys. IKE runs once in a while; ESP runs on every packet, usually in the kernel.

Time runs down between peer A on the left and peer B on the right. Two IKE_SA_INIT messages with Message ID 0 carry SA, KE and Nonce in the clear; after them both peers compute SKEYSEED and the IKE SA keys. Two IKE_AUTH messages with Message ID 1 are encrypted and carry ID, AUTH, SA and the traffic selectors; after them both peers have the IKE SA and two Child SAs, SPI 0x7a3f0c21 from A to B and SPI 0x51c9e8d4 from B to A.
Four messages: the first pair makes keys in the clear, the second pair, already encrypted, proves identities and creates the Child SAs.

On the canvas, peer A (198.51.100.1) and peer B (203.0.113.2) are gateways for two LANs, 10.1.0.0/24 and 10.2.0.0/24. An attacker on the path copies every packet. The band in the middle draws the packet's bytes, and the band below it is B's anti-replay window.

IKEv2: two exchanges, four messages

MessageMessage IDPayloadsWhat it does
1 IKE_SA_INIT request0HDR, SAi1, KEi, Nioffers algorithms, A's Diffie-Hellman value, a nonce
2 IKE_SA_INIT response0HDR, SAr1, KEr, Nr, [CERTREQ]picks the algorithms, B's Diffie-Hellman value and nonce
3 IKE_AUTH request1HDR, SK{IDi, [CERT], AUTH, SAi2, TSi, TSr}A proves who it is; offers the Child SA
4 IKE_AUTH response1HDR, SK{IDr, [CERT], AUTH, SAr2, TSi, TSr}B proves who it is; accepts the Child SA

After the first exchange both peers hold the same Diffie-Hellman secret (how that works is on Diffie-Hellman Key Exchange). They compute SKEYSEED = prf(Ni | Nr, g^ir) and from it seven keys: SK_d for making Child SA keys, SK_ai/SK_ar for integrity, SK_ei/SK_er for encryption and SK_pi/SK_pr for the AUTH payloads. At this point nobody is authenticated: a man in the middle could have run this exchange too. The second exchange is encrypted (SK{…}), and its AUTH payload signs the peer's first message plus the other side's nonce, so the identity is tied to these exact keys. The traffic selectors TSi and TSr say which addresses the Child SA protects (Demo: IKEv2 — four messages set up the SAs).

Each IKE message has a 28-byte header with both IKE SPIs (8 bytes each), the exchange type and a Message ID. Replies use the request's Message ID, so the two exchanges are 0 and 1, and the next exchange would be 2. TLS does the same job for one TCP connection in its own handshake; see TLS 1.2 vs TLS 1.3.

Security associations and the SPI

A security association (SA) is one direction of protected traffic: the algorithm, the key, the sequence counter and the traffic selectors. Two-way traffic needs two Child SAs. Each SA is named by a 32-bit SPI (security parameter index), and the receiver picks it, so it can find the SA quickly: B chose 0x7a3f0c21 for traffic to B, and A chose 0x51c9e8d4 for traffic to A. An arriving ESP packet carries the SPI in the clear, and the receiver looks it up in its SA database. The IKE SA itself is separate: it only carries IKE's own control messages (new Child SAs, rekeying, deletes, liveness checks).

ESP: transport mode vs tunnel mode

The canvas uses AES-GCM with a 16-byte ICV (RFC 4106), so ESP adds an 8-byte header (SPI and sequence number), an 8-byte IV, a trailer (padding, pad length, next header) and a 16-byte ICV (integrity check value). The original packet is 100 bytes: IP 20, TCP 20, data 60.

Two byte layouts. Transport mode: IP 20, ESP header 8, IV 8, TCP 20, data 60, trailer 4, ICV 16, total 136 bytes; 84 bytes encrypted, 100 bytes covered by the ICV. Tunnel mode: new IP 20, ESP header 8, IV 8, original IP 20, TCP 20, data 60, trailer 4, ICV 16, total 156 bytes; 104 bytes encrypted, 120 covered by the ICV. In both the outer IP header is not covered.
Transport mode keeps the original IP header; tunnel mode hides the whole original packet behind a new one.
Transport modeTunnel mode
Used betweentwo hosts, for their own traffictwo gateways (or a laptop and a gateway), for whole networks
Outer IP headerthe original one, protocol changed to 50a new one, gateway to gateway, protocol 50
EncryptedTCP + data + trailer = 84 Boriginal IP + TCP + data + trailer = 104 B
Covered by the ICVESP header + IV + ciphertext = 100 B120 B
Next header6 (TCP)4 (IPv4)
Size100 B → 136 B (+36)100 B → 156 B (+56)

In both modes the SPI and the sequence number stay readable, because the receiver needs them before it can decrypt. The outer IP header is not covered by the ICV; routers and NAT boxes may change it (Demo: tunnel mode; Demo: transport mode). The VPN on How a VPN Works is tunnel mode with a different protocol, WireGuard.

Sequence numbers and the anti-replay window

The sender numbers the packets of each SA 1, 2, 3, … and never reuses a number. An attacker who copies a packet can't read it, but it can send the same bytes again; if B accepted them, it would deliver the same request twice. So the receiver keeps an anti-replay window: the highest number accepted so far (the top) and a bitmap of which of the last 64 numbers it has seen. RFC 4303 requires at least 32 and makes 64 the default.

A window of 64 numbers from 8 to 71 with top 71. Four arrivals: number 72 is right of the window, new, accepted and the window slides; number 40 is inside and not seen, accepted as a late packet; number 50 is inside and already seen, dropped as a replay; number 1 is left of the window, dropped as too old.
Right of the window or an unmarked slot inside: accept. Marked or left of the window: drop.
  • A number already marked is a replay and is dropped (Demo: replay — a copy of packet #2 is dropped).
  • A number inside the window but not marked is a packet that came late. It is accepted, so normal reordering on the internet does no harm (Demo: out of order).
  • A number left of the window is dropped, even when it is genuine, because B no longer remembers whether it saw it (Demo: too old — packet #1 arrives after #71).

The 32-bit counter must not wrap: before it reaches 232 − 1, IKE makes a new Child SA. Most setups also use 64-bit extended sequence numbers, which send only the low 32 bits.

What the receiver checks, in order

  1. Find the SA from the SPI. No SA: drop.
  2. Check the sequence number against the window. A replay or a too-old number is dropped here, cheaply, before any cryptography.
  3. Check the ICV. If it fails, drop.
  4. Only now mark the number and, if it is the new top, slide the window.
  5. Decrypt, remove the padding, read the next header; in tunnel mode, check the inner addresses against the traffic selectors and forward the inner packet.

The order of steps 3 and 4 matters. The sequence number is inside the data the ICV authenticates. If the window moved before the ICV check, an attacker could send one forged packet numbered #502 and make every real packet after it look too old. Because the window moves only after a good ICV, the forged packet is just dropped (Demo: tampered copy — ICV fails, the window stays put).

See also Internet Protocol Layers for where IP sits, Router Forwarding for the routers that read only the outer header, and How NAT Shares One Public Address for the NAT boxes that make IPsec move to UDP port 4500.

What the page leaves out

AH (the authentication-only header, rarely used), NAT traversal (NAT_DETECTION payloads and ESP inside UDP port 4500), CREATE_CHILD_SA for rekeying and extra Child SAs, INFORMATIONAL exchanges (delete, dead peer detection), the IKE cookie that protects a busy responder, EAP and pre-shared-key authentication, IKE fragmentation, IPv6, separate encryption and integrity algorithms (AES-CBC with HMAC), and the SPD's bypass and discard rules.

On the mind map

Open the Networking mind mapNetworkingLayers & links: frames on one local networkLayers & linksIP & routing: packets hop between networksIP & routingCore services: address, name and timeCore servicesTCP, UDP & sockets: end to end between two portsTCP, UDP & socketsHTTP & TLS: what a browser does per requestHTTP & TLSAPIs & push: how apps talk to serversAPIs & pushProxies & edge: boxes in the middle of the pathProxies & edgeContainers: virtual networks from Linux partsContainersReal-time media: voice and video over UDPReal-time mediaSecurity at the edge: filter traffic at the borderSecurity at the edgeForward vs reverse proxyForward vs reverse proxyforward hides clients; reverse hides serversAPI gatewayAPI gatewayroute, check auth, rate limit, forwardCDN and Route 53CDN and Route 53DNS sends you to the nearest edge cacheWireGuard VPNWireGuard VPNwraps IP packets in encrypted UDPIPsec IKEv2 and ESPIKE makes SAs; ESP window stops replaysL4 vs L7 load balancingL4 vs L7 load balancingL4 routes connections; L7 each requestOpen the Networking mind mapNetworkingboxes in the middle of the pathProxies & edgeForward vs reverse proxyForward vs reverse proxyforward hides clients; reverse hides serversAPI gatewayAPI gatewayroute, check auth, rate limit, forwardCDN and Route 53CDN and Route 53DNS sends you to the nearest edge cacheWireGuard VPNWireGuard VPNwraps IP packets in encrypted UDPIPsec IKEv2 and ESPIKE makes SAs; ESP window stops replaysL4 vs L7 load balancingL4 vs L7 load balancingL4 routes connections; L7 each request
This page on the Networking mind map · 55 pages in 10 branches · open it to see the whole map and test yourself in recall mode.