The idea: IKE agrees, ESP protects
IPsec protects IP packets between two peers. It has two parts. IKEv2 (Internet Key Exchange, RFC 7296) is a small protocol on UDP port 500 that the two peers run first: they agree on algorithms, make shared keys and prove who they are. ESP (Encapsulating Security Payload, RFC 4303) then protects each data packet with those keys. IKE runs once in a while; ESP runs on every packet, usually in the kernel.
On the canvas, peer A (198.51.100.1) and peer B (203.0.113.2) are gateways for two LANs, 10.1.0.0/24 and 10.2.0.0/24. An attacker on the path copies every packet. The band in the middle draws the packet's bytes, and the band below it is B's anti-replay window.
IKEv2: two exchanges, four messages
| Message | Message ID | Payloads | What it does |
|---|---|---|---|
| 1 IKE_SA_INIT request | 0 | HDR, SAi1, KEi, Ni | offers algorithms, A's Diffie-Hellman value, a nonce |
| 2 IKE_SA_INIT response | 0 | HDR, SAr1, KEr, Nr, [CERTREQ] | picks the algorithms, B's Diffie-Hellman value and nonce |
| 3 IKE_AUTH request | 1 | HDR, SK{IDi, [CERT], AUTH, SAi2, TSi, TSr} | A proves who it is; offers the Child SA |
| 4 IKE_AUTH response | 1 | HDR, SK{IDr, [CERT], AUTH, SAr2, TSi, TSr} | B proves who it is; accepts the Child SA |
After the first exchange both peers hold the same Diffie-Hellman secret (how that works is on Diffie-Hellman Key Exchange). They compute SKEYSEED = prf(Ni | Nr, g^ir) and from it seven keys: SK_d for making Child SA keys, SK_ai/SK_ar for integrity, SK_ei/SK_er for encryption and SK_pi/SK_pr for the AUTH payloads. At this point nobody is authenticated: a man in the middle could have run this exchange too. The second exchange is encrypted (SK{…}), and its AUTH payload signs the peer's first message plus the other side's nonce, so the identity is tied to these exact keys. The traffic selectors TSi and TSr say which addresses the Child SA protects (Demo: IKEv2 — four messages set up the SAs).
Each IKE message has a 28-byte header with both IKE SPIs (8 bytes each), the exchange type and a Message ID. Replies use the request's Message ID, so the two exchanges are 0 and 1, and the next exchange would be 2. TLS does the same job for one TCP connection in its own handshake; see TLS 1.2 vs TLS 1.3.
Security associations and the SPI
A security association (SA) is one direction of protected traffic: the algorithm, the key, the sequence counter and the traffic selectors. Two-way traffic needs two Child SAs. Each SA is named by a 32-bit SPI (security parameter index), and the receiver picks it, so it can find the SA quickly: B chose 0x7a3f0c21 for traffic to B, and A chose 0x51c9e8d4 for traffic to A. An arriving ESP packet carries the SPI in the clear, and the receiver looks it up in its SA database. The IKE SA itself is separate: it only carries IKE's own control messages (new Child SAs, rekeying, deletes, liveness checks).
ESP: transport mode vs tunnel mode
The canvas uses AES-GCM with a 16-byte ICV (RFC 4106), so ESP adds an 8-byte header (SPI and sequence number), an 8-byte IV, a trailer (padding, pad length, next header) and a 16-byte ICV (integrity check value). The original packet is 100 bytes: IP 20, TCP 20, data 60.
| Transport mode | Tunnel mode | |
|---|---|---|
| Used between | two hosts, for their own traffic | two gateways (or a laptop and a gateway), for whole networks |
| Outer IP header | the original one, protocol changed to 50 | a new one, gateway to gateway, protocol 50 |
| Encrypted | TCP + data + trailer = 84 B | original IP + TCP + data + trailer = 104 B |
| Covered by the ICV | ESP header + IV + ciphertext = 100 B | 120 B |
| Next header | 6 (TCP) | 4 (IPv4) |
| Size | 100 B → 136 B (+36) | 100 B → 156 B (+56) |
In both modes the SPI and the sequence number stay readable, because the receiver needs them before it can decrypt. The outer IP header is not covered by the ICV; routers and NAT boxes may change it (Demo: tunnel mode; Demo: transport mode). The VPN on How a VPN Works is tunnel mode with a different protocol, WireGuard.
Sequence numbers and the anti-replay window
The sender numbers the packets of each SA 1, 2, 3, … and never reuses a number. An attacker who copies a packet can't read it, but it can send the same bytes again; if B accepted them, it would deliver the same request twice. So the receiver keeps an anti-replay window: the highest number accepted so far (the top) and a bitmap of which of the last 64 numbers it has seen. RFC 4303 requires at least 32 and makes 64 the default.
- A number already marked is a replay and is dropped (Demo: replay — a copy of packet #2 is dropped).
- A number inside the window but not marked is a packet that came late. It is accepted, so normal reordering on the internet does no harm (Demo: out of order).
- A number left of the window is dropped, even when it is genuine, because B no longer remembers whether it saw it (Demo: too old — packet #1 arrives after #71).
The 32-bit counter must not wrap: before it reaches 232 − 1, IKE makes a new Child SA. Most setups also use 64-bit extended sequence numbers, which send only the low 32 bits.
What the receiver checks, in order
- Find the SA from the SPI. No SA: drop.
- Check the sequence number against the window. A replay or a too-old number is dropped here, cheaply, before any cryptography.
- Check the ICV. If it fails, drop.
- Only now mark the number and, if it is the new top, slide the window.
- Decrypt, remove the padding, read the next header; in tunnel mode, check the inner addresses against the traffic selectors and forward the inner packet.
The order of steps 3 and 4 matters. The sequence number is inside the data the ICV authenticates. If the window moved before the ICV check, an attacker could send one forged packet numbered #502 and make every real packet after it look too old. Because the window moves only after a good ICV, the forged packet is just dropped (Demo: tampered copy — ICV fails, the window stays put).
See also Internet Protocol Layers for where IP sits, Router Forwarding for the routers that read only the outer header, and How NAT Shares One Public Address for the NAT boxes that make IPsec move to UDP port 4500.
What the page leaves out
AH (the authentication-only header, rarely used), NAT traversal (NAT_DETECTION payloads and ESP inside UDP port 4500), CREATE_CHILD_SA for rekeying and extra Child SAs, INFORMATIONAL exchanges (delete, dead peer detection), the IKE cookie that protects a busy responder, EAP and pre-shared-key authentication, IKE fragmentation, IPv6, separate encryption and integrity algorithms (AES-CBC with HMAC), and the SPD's bypass and discard rules.