Module 4 — Anchors, NMS and thresholds

Modules 2 and 3 built detectors. This module explains the three levers that turn the pile of raw predictions those detectors emit into a clean detection list. Two of them, anchors and NMS, are architectural. The third, thresholds, is a knob you set at deployment. All three interact, and misunderstanding one silently degrades the others.

On the crossroads, this is where "the model has good mAP on paper" becomes "the count agrees with a human at rush hour". The math is elementary; the judgement is not.

Anchors: priors on box shape

An anchor is a template box at a fixed position and shape. The detector does not predict a box from thin air; it predicts an offset from an anchor. If the anchor is 40 by 40 pixels, the network learns something like "add 6 pixels to the width, subtract 3 to the height, shift the centre right by 2".

Why not predict the box directly? Because the range of shapes to cover is huge — a distant pedestrian is a 12 by 30 pixel thin rectangle, a nearby bus is 800 by 400 — and regressing that from a single scalar output collapses training. Anchors carve the shape space into buckets: each anchor specialises, and each prediction stays a small correction.

For a detector to work well, the set of anchors must match the distribution of objects in the training data. Ultralytics used to fit anchors automatically at training start; modern anchor-free detectors (YOLOv8, FCOS) removed the need entirely by predicting box dimensions from a distribution learned end to end. On the crossroads dataset, where objects range from tiny pedestrians to whole trucks, this simplification is a genuine improvement — fewer hyperparameters, less to tune, comparable accuracy.

Non-maximum suppression, step by step

A dense one-stage detector produces many overlapping boxes on the same object. Three anchors of similar aspect ratio at neighbouring cells all fire on the same car. NMS keeps the best and suppresses the rest.

The algorithm has three lines:

def nms(boxes, scores, iou_threshold=0.5):
    order = scores.argsort(descending=True)
    keep = []
    while len(order) > 0:
        i = order[0].item()
        keep.append(i)
        rest = order[1:]
        ious = box_iou(boxes[i].unsqueeze(0), boxes[rest]).squeeze(0)
        order = rest[ious < iou_threshold]
    return keep

At each step, take the highest-scoring remaining box, keep it, and drop every other box whose IoU with it exceeds a threshold. Repeat until nothing is left. Torchvision provides torchvision.ops.nms if you need a fast, vectorised implementation.

Two variants are worth knowing. Class-aware NMS runs the loop independently per class, so a pedestrian right next to a car does not get suppressed by the car — this is the default in every serious detector. Soft-NMS decays the score of overlapping boxes instead of killing them, which helps in extreme crowds; it is rarely enabled by default.

The NMS IoU threshold

The IoU threshold governs "how close two boxes must be for one to eat the other".

• Low threshold (0.3): aggressive suppression. Overlapping detections vanish. Two adjacent cars at the crossroads become one, undercounting.
• High threshold (0.7): permissive suppression. Duplicates survive. One car generates three boxes, overcounting.

The default of 0.5 is a compromise, but on a crowded crossroads at rush hour you often need to raise it slightly (0.55 to 0.6). The right value can only be chosen against a labelled validation set, by measuring precision and recall at several thresholds and picking the one that matches the downstream tolerance.

A low NMS threshold does not raise precision, it raises undercount

People sometimes tighten NMS to "remove false positives". It does not. False positives on the road, sky or empty pavement have no overlap with real detections, so NMS ignores them. What tightening NMS does is merge legitimate neighbouring detections. Use the confidence threshold to control false positives, not the NMS threshold.

The confidence threshold

Every detection carries a confidence between 0 and 1. Raising the threshold drops low-confidence detections. This is the knob for precision versus recall.

• Low threshold (0.1): everything survives. Recall is high, precision is low. Alarms fire on shadows.
• High threshold (0.8): only very confident detections remain. Precision is high, recall is low. Distant pedestrians in fog are missed.

The right value is not a universal 0.5. It depends on the asymmetric cost of the two error types. Missing a pedestrian at a school-zone crossroads is not the same as counting one extra car; the threshold must reflect that.

import numpy as np

def sweep_threshold(pred_scores, pred_labels, gt_labels, thresholds):
    for t in thresholds:
        keep = pred_scores >= t
        tp = np.sum((pred_labels == gt_labels) & keep)
        fp = np.sum((pred_labels != gt_labels) & keep)
        fn = np.sum((gt_labels != 0) & ~keep)
        precision = tp / max(tp + fp, 1)
        recall = tp / max(tp + fn, 1)
        print(f"t={t:.2f}  P={precision:.3f}  R={recall:.3f}")

sweep_threshold(scores, labels, ground_truth, np.linspace(0.1, 0.9, 9))

Interaction between the two thresholds and mAP

Module 5 will measure mAP, which sweeps the confidence threshold internally and reports the area under the precision-recall curve. This has a subtle consequence: evaluating with a high confidence threshold ruins mAP, because mAP wants to see the low-confidence tail to compute recall properly.

The standard practice is therefore:

• At evaluation time, keep the confidence threshold very low (0.001 to 0.05) and let mAP sweep. NMS threshold stays at the model's default (usually 0.6 to 0.7).
• At deployment time, tune the confidence threshold on a labelled validation set to the operating point that matches the cost function, and tune NMS if the scene is crowded.

Confusing the two situations is the single most common cause of "the model was great on the benchmark, it is unusable in production" complaints.

Log the raw predictions during the first week in production

The threshold you pick on the validation set will drift as the scene changes: seasons, camera angle nudges, new construction. Logging pre-threshold predictions for a fraction of frames lets you re-optimise thresholds later without re-running the model on stored video, and lets you compute what recall would have been at other thresholds without any labelled data at all.

In summary

• Anchors are shape priors that turn box regression into small corrections; anchor-free heads (YOLOv8, FCOS) remove one hyperparameter set and now match anchor-based performance on typical datasets.
• NMS collapses overlapping detections of the same object; its IoU threshold controls merging aggressiveness, and it is class-aware by default in every modern detector.
• The confidence threshold governs the precision-recall trade-off; it must be chosen from the asymmetric cost of the two error types, not left at 0.5 for lack of a better idea.
• Evaluation and deployment use different thresholds: keep confidence very low at eval time to give mAP its full precision-recall curve, tune it at deployment against the real cost function.

Next module: detection metrics, from IoU by hand to a mAP report from pycocotools.