The sender does not know the bandwidth

A server sending a large file has no idea how fast the path to the client is: 10 Mbit/s over Wi-Fi, 1 Gbit/s in a data centre, and the bottleneck may change while the transfer runs. Send too slowly and the link sits idle; send too fast and a router on the way fills its queue and throws packets away. TCP's congestion control finds the rate by probing: it keeps a congestion window cwnd, the number of segments it may have sent but not yet seen acknowledged, grows it while everything arrives, and shrinks it when something is lost. Because one window is sent per round trip, the rate is about cwnd / RTT.

The canvas follows one such transfer. On the left the sender S with its congestion-control state; in the middle the network: the bottleneck router R with its queue and the segment it is serving, the data link to the receiver and the ACK path back, drawn as conveyors of 10 ms slots; on the right the receiver D with its receive buffer and the application reading from it. Under them, the sender's view of the sequence space (the send window), then the chart: cwnd in blue, ssthresh dashed orange, the previous run in grey, and under it the router's queue and the RTT samples. Every step is one 10 ms tick (or one RTT with detail: per RTT).

The page's network, in numbers

S sits next to R (0 ms); R sends one segment every 10 ms (the bottleneck rate); the link to D takes 40 ms and the ACK path back 50 ms. A lone segment therefore has an RTT of 100 ms: 10 ms of service, 40 ms to D, 50 ms back. How many segments must be in flight to keep the link busy all the time? The bandwidth-delay product: rate × RTT = 1 segment / 10 ms × 100 ms = 10 segments, which you can count on the conveyors: 1 in service, 4 on the data link, 5 ACKs on the way back. With cwnd below 10 the link idles part of each RTT; with cwnd above 10 the extra segments wait in R's queue and the RTT grows by 10 ms per waiting segment (queueing delay: RTT = 100 ms + 10 ms × queue, the purple line in the strip under the chart). With a queue of 5 places the ceiling is cwnd = 10 + 5 = 15; one more and R drops.

Sender S, bottleneck router R with an empty 5-place queue serving one segment every 10 ms, four segments on the 40 ms data link to receiver D, and five ACKs on the 50 ms path back: 10 segments in flight
Ten segments fill the path exactly (1 being sent, 4 on the link, 5 ACKs coming back); more only queue at R, and beyond 15 R drops.

Two windows: flow control and congestion control

The receiver protects itself: every ACK carries rwnd, the free space in its receive buffer, so a slow reader is never flooded (flow control). The sender protects the network with cwnd, a number nobody sends it; it infers it. The sender may have min(cwnd, rwnd) segments outstanding. Run Demo 7: a reader that takes only one segment per 20 ms (half the link rate) fills its buffer until rwnd settles at about 5 segments (5 per 100 ms = its reading speed), and cwnd stops growing, because Linux grows cwnd only while it is what limits the sender. A reader that pauses fills the buffer completely: rwnd = 0, S stops and sends a small zero-window probe each time its persist timer fires, and the reader's next read sends a window update. Because S has then been idle for longer than its RTO, its ACK clock is gone: RFC 5681 (and Linux) restart from the initial window instead of bursting the old cwnd into the queue.

1986: congestion collapse

Early TCP sent a whole receiver window at once and retransmitted on a fixed timer. In October 1986 the link between Lawrence Berkeley Laboratory and UC Berkeley, 400 yards apart, dropped from 32 kbit/s to 40 bit/s: routers were full of retransmissions of packets that were only late, which made everything later still. Van Jacobson's answer (1988), still the core of every TCP: a sender must not put a new packet into the network until an old one has left (ACK clocking), must start slowly, must back off multiplicatively on loss, and must estimate the RTT and its variation to time retransmissions.

Slow start

A new connection starts with an initial window (IW) of a few segments and adds one segment per ACK. Each segment in flight comes back as an ACK and each ACK lets two segments out, so cwnd doubles every RTT: 1, 2, 4, 8 (Demo 1: ACK 2 at 100 ms, ACKs 3–4 at 200–210 ms, ACKs 5–8 at 300–330 ms). It is "slow" only compared with what came before it, sending the whole receiver window at once. IW was 1 segment in the original algorithm, is up to 4 in RFC 3390 and 10 since RFC 6928 (Linux's default; choose IW 10, and see the page's 5-place queue drop 4 of the first 10 segments at once). Slow start ends when cwnd reaches the slow-start threshold ssthresh (8 here) or at the first loss.

Congestion avoidance and AIMD

Above ssthresh the sender adds one segment per round trip: Linux counts ACKs in snd_cwnd_cnt and raises cwnd by one after cwnd of them (the page's "CA counter"; the RFC writes cwnd += 1/cwnd per ACK, the same growth). This is the additive increase. When loss shows that the path is full, cwnd is halved: multiplicative decrease. AIMD draws the sawtooth of Demo 2: 8 → 16, drop, 8 → 16, a tooth every 1.28 s (drops at 1 340, 2 620, 3 900 and 5 180 ms), the line crossing the green BDP guide (the link becomes full) and hitting the red ceiling (BDP + queue).

Chart of cwnd against round trips: slow start doubles 1, 2, 4, 8, then cwnd grows by one per round trip to 16, crosses the BDP line at 10 and the drop line at 15, is halved to 8 by Reno after three duplicate ACKs and climbs again; Tahoe instead falls back to 1
Slow start doubles cwnd each round trip, then additive increase and multiplicative decrease draw Reno's sawtooth between 8 and 16.

Why multiplicative decrease and additive increase and not the other way round? Chiu and Jain (1989) drew two flows sharing a link as a point (rate 1, rate 2). Additive increase moves the point along a 45° line, so both flows gain the same amount; multiplicative decrease moves it towards the origin, cutting the larger flow by more. Every cycle brings the point closer to the fairness line, whatever the start: AIMD converges to an equal share. The page simulates one flow only.

Two loss signals

The network never says "slow down"; the sender learns about congestion from loss, in two ways:

  • Three duplicate ACKs. The receiver ACKs every segment with the next segment it expects. When one is missing, every later arrival repeats the same ACK. One duplicate may only mean reordering; three mean "one segment is missing, but later ones still arrive", so the path still works. The sender retransmits at once (fast retransmit) instead of waiting for the timer. In Demo 2 the drop comes when cwnd reaches 16: the two segments sent on that ACK meet a queue of 4, the first takes the 5th place and the second is dropped; the three duplicate ACKs arrive 160–180 ms later.
  • A timeout (RTO). If ACKs stop entirely nothing is getting through, a much stronger signal. The timer follows RFC 6298: srtt and rttvar are smoothed averages of the RTT samples (weights 1/8 and 1/4), RTO = srtt + max(4 · rttvar, 200 ms) (Linux's minimum; the RFC says 1 s), 1 s before the first sample. Karn's rule: no sample from a retransmitted segment, because its ACK cannot say which copy it answers; and every consecutive timeout doubles the RTO (exponential backoff). In Demo 3 a 200 ms outage swallows every segment and every ACK: no duplicate ACKs can come back, the sender waits for its timer, then sets ssthresh to half of what was in flight, cwnd = 1, and resends from the oldest unacknowledged segment.

Tahoe, Reno and NewReno

  • Tahoe (4.3BSD, 1988) treats three duplicate ACKs like a timeout, minus the waiting: retransmit, ssthresh = cwnd / 2, cwnd = 1, slow start. Demo 4 shows the cost: each loss restarts from 1.
  • Reno (1990, RFC 5681) adds fast recovery: ssthresh = cwnd / 2, cwnd = ssthresh + 3 (the three segments that left the network, as the three duplicate ACKs prove), and every further duplicate ACK adds one more (window inflation), because each means another segment has left. Once cwnd exceeds what is in flight, new segments go out during the recovery, so the ACK clock keeps ticking. The ACK of the retransmission ends the recovery: cwnd = ssthresh (deflation) and congestion avoidance goes on from half the old window.
  • Reno repairs only one lost segment per recovery. With several losses in one window, the first new ACK already ends the recovery, the next hole needs three more duplicate ACKs (halving again) or a timeout. NewReno (RFC 6582) remembers recover, the highest segment sent when the loss was detected, and stays in recovery until an ACK covers it: a partial ACK below recover retransmits the next hole at once. Its limit: one hole per RTT.

Demo 5 shows why this matters. With ssthresh ∞ slow start doubles until the queue overflows, and in the RTT it takes the first duplicate ACK to come back the window is still doubling: 12 segments are dropped in one window (s27, s29, s31, then s37, s39, … s53), and the third duplicate ACK arrives at 630 ms with cwnd 27, so ssthresh becomes 13. Reno leaves fast recovery on the first new ACK (a partial one, at 760 ms), gets only one more duplicate ACK for the next hole and ends in a timeout at 1 090 ms. NewReno, drawn on top, stays in recovery and repairs one hole per RTT: 12 holes, done at 1 860 ms; its full ACK then sets cwnd = 13 while little is in flight, and that burst overflows the queue once more (RFC 6582 offers a variant that avoids it).

SACK

The fix for many losses in one window is selective acknowledgement (RFC 2018), negotiated in the SYN and used by virtually every stack today. Each duplicate ACK carries up to four blocks saying which later ranges did arrive, so the sender knows every hole after one RTT and can resend them all in the same recovery (RFC 6675). With SACK, Demo 5 would be over in about two RTTs. The page leaves it out so that the classic algorithms stay visible.

How big should the router's buffer be?

After Reno halves cwnd, the link stays busy only if the halved window still fills the pipe: (BDP + queue) / 2 ≥ BDP, i.e. a queue of at least one BDP (the classic rule of thumb: buffer = bandwidth × RTT). Demo 6 runs a queue of 2 (ceiling 12, so the drop comes at cwnd 13, halved to 6, below the BDP of 10: the link idles after each loss, link use about 86 %) and then a queue of 30 (the first drop only at cwnd 41, after 8.3 s, then 20–40: the link is always full, but 10–30 segments wait in the queue and the RTT reaches 400 ms). A queue of 10 is exactly one BDP: 10–20, link use 100 %, RTT up to 200 ms. Too much buffer is bufferbloat: every other flow through the same router (a video call, a DNS lookup) waits behind the queue that loss-based TCP insists on filling. Studies of core routers with many flows show that far less than one BDP is enough there; home routers and phones are where bufferbloat hurts.

ECN: mark instead of drop

With Explicit Congestion Notification (RFC 3168) a router whose queue is building up sets a bit in the IP header instead of dropping the packet; the receiver echoes it in its ACKs and the sender halves cwnd as if a segment had been lost, but nothing needs retransmitting. Combined with active queue management (CoDel, PIE), which marks or drops early while the queue is still short, it keeps the queue small without losses.

CUBIC, the Linux default

Reno's additive increase is one segment per RTT; on a 10 Gbit/s path with a 100 ms RTT (a BDP of about 83 000 segments) climbing back after one loss would take over an hour. CUBIC (RFC 8312, Linux's default since 2.6.19) grows cwnd as a cubic function of the time since the last loss, not of the number of RTTs:

W(t) = C · (t − K)³ + W_max        K = ∛(W_max · (1 − β) / C),  C = 0.4,  β = 0.7

cwnd
 W_max ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ●━━━━━━●─ ─ ─ ─ ─ ─   concave: fast, then slowing near W_max
                         ●                ●
                     ●                       ●     convex: probing faster and faster
 0.7 W_max  ●  ●                                      beyond the old maximum
            └──────────────── K ─────────────────── time since the loss

It is fast far below the old maximum, careful near it, and probes beyond it ever faster. It reduces by β = 0.7 (to 70 %) instead of 50 %, and in a "TCP-friendly" region it never grows slower than Reno would. It is not simulated here: with this page's tiny BDP of 10 segments, K is about 2.3 s (23 RTTs) and CUBIC would mostly run in its Reno-friendly region, so the curve would not look cubic.

BBR: model the path instead of filling it

Every algorithm above only learns about the path by overflowing its queue. BBR (Google, 2016) measures the path instead: the bottleneck bandwidth (the highest delivery rate recently seen) and the minimum RTT (seen when the queue was empty). It paces packets at that bandwidth and keeps about one BDP in flight, periodically probing a little above to see if more bandwidth has appeared, and briefly draining the queue to re-measure the minimum RTT. Run with BBR, Demo 6 would keep the queue near empty and the RTT near 100 ms even with 30 buffer places. BBR is not simulated here.

What the page leaves out

  • Segments instead of bytes; every ACK acknowledges whole segments. One flow, one bottleneck, fixed delays, tail drop, no random loss and no competing traffic.
  • The receiver ACKs every segment at once. Linux delays ACKs (every second segment), but it counts acknowledged segments, so the growth is the same.
  • Reno is the default because it is the textbook model. Linux uses CUBIC with SACK, PRR (RFC 6937) instead of window inflation, RACK-TLP instead of three duplicate ACKs and tail-loss probes, HyStart (which leaves slow start before the queue overflows, avoiding Demo 5) and pacing.
  • The receive buffer holds 64 segments and the advertised window never shrinks; the pausing reader stops from 1 s to 3 s. The page's ssthresh dashes are drawn as short rectangles (the drawing library has no dashed lines).
  • Slow start adds one segment per ACK and congestion avoidance counts ACKs (not acknowledged segments); after a timeout the sender resends in order from snd_una (go-back-N, as without SACK) and the receiver discards what it already has.

See also How an HTTP Connection Is Established (a single fast retransmit on a short response, Demo 7 there) and HTTP/1.1, HTTP/2 and HTTP/3 (why one lost segment stalls every HTTP/2 stream on the connection). TCP vs UDP shows the same loss next to UDP, which does not retransmit at all.