The idea: many boxes come out, one box per object should stay
An object detector answers two questions for every object in an image: where is it, and what is it. The "where" is a bounding box. The detector does not output one box per object. It scores thousands of candidate boxes, and every candidate whose class score passes a score threshold comes out. A real object usually gets several boxes that are almost the same. The last step of the detector, non-maximum suppression (NMS), keeps the best box for each object and throws away the others.
The page uses a synthetic image of 400 × 300 pixels with three objects of one class, "circle": G1 alone, and G2 and G3 standing close together. The 8 candidate boxes are numbered by score, so D1 (0.92) is the best and D8 (0.35) the worst. These are made-up boxes, not the output of a real network. How a CNN turns pixels into features is on Convolution in a CNN, and how to reuse a pre-trained backbone is on Transfer Learning.
Candidate boxes and scores
A box is written by its corners [x1, y1, x2, y2] in image pixels: x grows to the right and y grows down, so (x1, y1) is the top-left corner. Its area is (x2 − x1) × (y2 − y1). D1 is [48, 62, 138, 152], so its area is 90 × 90 = 8100. Each box also carries a score: how sure the detector is that the box holds a circle. In a real detector there is one score per class; the page has one class.
IoU, step by step
To compare two boxes, detection uses intersection over union (IoU): the area they share divided by the area they cover together. It is 1 for two identical boxes and 0 for boxes that do not touch.
- The intersection starts at the larger x1 and the larger y1, and ends at the smaller x2 and the smaller y2. If that end is not past that start, in x or in y, the boxes do not overlap and the intersection is 0.
- Union = area A + area B − intersection, so the shared part is counted once.
- IoU = intersection / union.
For D2 and D5, two boxes on the two different objects G2 and G3, the intersection is 70 × 84 = 5880 and the union is 9800 + 9996 − 5880 = 13916, so IoU = 0.423 (Demo 2). The pair D1 and D2 does not touch at all: IoU 0.
Greedy non-maximum suppression
NMS needs only the boxes, their scores and one number, the IoU threshold:
sort the boxes by score, highest first
while boxes are left:
take the first box and keep it
remove every remaining box whose IoU with it is > threshold
This is what torchvision.ops.nms(boxes, scores, iou_threshold) does; it returns the indices of the kept boxes in score order. It is called "non-maximum" suppression because every box that is not the local maximum of the score gets suppressed. At threshold 0.5 the page keeps D1, D2, D5 and D8 (Demo 3): D1 removes D4 (0.773) and D6 (0.598), D2 removes D3 (0.639) but keeps D5 (0.423) and D7 (0.327), and D5 then removes D7 (0.774). NMS runs once per class: a box of a dog never suppresses a box of a cat.
The threshold trade-off
The threshold decides how much overlap counts as "the same object". Real objects can overlap too: people in a crowd, cars in a queue. Here G2 and G3 have IoU 0.423 with each other, so their correct boxes D2 and D5 also overlap by 0.423.
| NMS threshold | kept boxes | what goes wrong |
|---|---|---|
| 0.3 (Demo 4) | D1, D2, D8 | D5 overlaps D2 by 0.423 > 0.3 and is removed: G3 is missed ✗ |
| 0.5 (Demo 3) | D1, D2, D5, D8 | one box per object ✓ |
| 0.7 (Demo 5) | D1, D2, D3, D5, D6, D8 | D3 (0.639 with D2) and D6 (0.598 with D1) survive: duplicates ✗ |
Real defaults vary: 0.45 in the YOLOv5 detect script, 0.5 for the final boxes of torchvision's Faster R-CNN, and 0.7 inside its region proposal network, where duplicates are cheap because a second stage looks at every proposal again.
Scoring one class: TP, FP and average precision
To score a detector we compare its kept boxes with the ground-truth boxes drawn by people. Go through the boxes in score order. A box is a true positive (TP) if its best IoU with a ground-truth box is at least 0.5 and that object was not found yet. Otherwise it is a false positive (FP): a box on the background, a box that fits too badly, or a duplicate of an object that a higher box already found. An object that no box found is a miss (false negative). Precision and recall then follow as usual; what they mean is on Precision & Recall.
Precision and recall are recomputed after each box in score order. Average precision (AP) is the area under that precision–recall curve, with each precision replaced by the best precision at that recall or later (all-point interpolation, PASCAL VOC since 2010). With 3 objects every TP adds 1/3 of recall, so AP = the sum of 1/3 × that precision over the TPs.
| run | TP | FP | missed | precision | recall | AP@0.5 |
|---|---|---|---|---|---|---|
| no NMS (Demo 7) | 3 | 5 | 0 | 0.375 | 1.000 | 0.867 |
| NMS 0.3 | 2 | 1 | 1 | 0.667 | 0.667 | 0.667 |
| NMS 0.5 (Demo 6) | 3 | 1 | 0 | 0.750 | 1.000 | 1.000 ✓ |
| NMS 0.7 | 3 | 3 | 0 | 0.500 | 1.000 | 0.917 |
Without NMS the ranking is D1 TP, D2 TP, D3 FP, D4 FP, D5 TP: when the last object is found precision is 3/5, so AP = 1/3·1 + 1/3·1 + 1/3·0.6 = 0.867. At 0.5 all three objects are found before the only FP (D8, score 0.35), so AP is 1.000 even though precision over all kept boxes is 0.75: AP only punishes an FP that ranks above a TP. At 0.3 the missed G3 caps recall at 2/3, and AP at 0.667 (Demo 8 fills the whole table). The AP of every class, averaged, is the mAP that papers report. ROC and AUC is a different way to sum up a ranking; detection uses AP because the number of true negatives (background boxes) is not defined.
"Non-maximum suppression" has a cousin in edge detection: Canny thins edges by keeping only the pixel with the largest gradient across an edge. Same idea, local maxima only, but on pixels instead of boxes. CNN vs Transformer compares the two kinds of backbone a detector can use.
See also Harris Corner Detection, which applies non-maximum suppression to pixels to keep one point per corner.
What the page leaves out
How the candidates are made (anchor boxes, region proposals, the box regression that refines them); more than one class; Soft-NMS, which lowers the scores of overlapping boxes instead of removing them; NMS-free detectors such as DETR; the score threshold before NMS and the cap on boxes per image; COCO's AP, which averages over IoU thresholds 0.50 to 0.95 and uses 101 recall points; and real images and networks.