Module 2 — Two-stage detection: the R-CNN family

Module 1 chose detection as the right task family to count vehicles at the crossroads. This module builds the first detector we will actually run against those frames. It is deliberately the slow, accurate one, so that module 3 can be compared against it fairly.

The idea behind two stages

Detection has to answer two entangled questions at once: where might an object be and what is it. The R-CNN family separates them. Stage one proposes regions likely to contain an object. Stage two classifies each region and refines its bounding box. Everything else in the family is an optimisation of that split.

The intuition is worth stating plainly. Running a full classifier over every possible rectangle in a $1080 \times 1920$ image is hopeless: billions of rectangles, most of them empty. If a cheap first pass can filter those billions down to a few thousand plausible regions, the expensive classifier only sees the shortlist.

Three generations, one architecture

Model	Region proposals	Feature extraction	Classification	Frames per second on a 2024 GPU
R-CNN (2014)	Selective search, on the image	On each region, from scratch	Per region	~0.02
Fast R-CNN (2015)	Selective search, on the image	Once, on the whole image, shared	Per region	~0.5
Faster R-CNN (2016)	Region Proposal Network, learned	Once, shared	Per region	~5 to 15

The move from R-CNN to Fast R-CNN removed the largest cost: instead of running the convolutional backbone on every proposed crop, it runs once on the whole image and cuts region features from the resulting feature map. The move from Fast to Faster removed the next largest: replacing an off-the-shelf proposal algorithm with a small learned network that shares the backbone.

RoI pooling and RoI Align

Between stage one and stage two lies a technical problem. A proposal is a rectangle at some resolution; the classifier expects a fixed-size feature block. RoI pooling solves this by dividing the region into a fixed grid (say $7 \times 7$) and max-pooling each cell. It works, but it quantises coordinates twice: once to align the region to the feature grid, once to align the cells to feature pixels.

RoI Align, introduced with Mask R-CNN (module 7), keeps the coordinates as floats and samples them with bilinear interpolation. The gain is small for detection, decisive for segmentation, where a shift of half a pixel visibly moves the mask boundary. Every modern implementation uses RoI Align.

Running Faster R-CNN on the crossroads dataset

Torchvision ships a pretrained Faster R-CNN with a ResNet-50 backbone. We can point it at a crossroads frame in a dozen lines.

import torch
import torchvision
from torchvision.models.detection import fasterrcnn_resnet50_fpn_v2
from torchvision.io import read_image
from torchvision.transforms.functional import convert_image_dtype

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = fasterrcnn_resnet50_fpn_v2(weights="DEFAULT").eval().to(device)

image = read_image("crossroads_frame_0001.jpg")
image = convert_image_dtype(image, torch.float32).to(device)

with torch.no_grad():
    outputs = model([image])

boxes = outputs[0]["boxes"].cpu()      # [N, 4] in xyxy pixels
scores = outputs[0]["scores"].cpu()    # [N]
labels = outputs[0]["labels"].cpu()    # [N] COCO class ids
print(boxes.shape, scores[:5], labels[:5])

The output is a list of dictionaries, one per input image. Each boxes[i] is [x1, y1, x2, y2] in pixels, scores[i] a confidence between 0 and 1, labels[i] the COCO category id (3 = car, 1 = person, 6 = bus, and so on).

Filtering by confidence keeps only usable detections:

keep = scores > 0.5
boxes, scores, labels = boxes[keep], scores[keep], labels[keep]

Training on a custom set

For our crossroads dataset the head is replaced so the model outputs our own classes rather than COCO's 80.

from torchvision.models.detection.faster_rcnn import FastRCNNPredictor

model = fasterrcnn_resnet50_fpn_v2(weights="DEFAULT")

num_classes = 3   # background, car, pedestrian
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

The training loop expects a list of images and a list of target dictionaries with keys boxes and labels. The loss returned by the model in training mode is already the sum of the four internal losses (RPN classification, RPN regression, box classification, box regression); you optimise it as usual.

Do not include a "background" class in your targets

Torchvision treats class 0 as background implicitly. Your annotation for a car should have label 1, not 0, and no background box should ever appear in targets["labels"]. Getting this wrong lets the model learn to predict background everywhere, and its mAP silently collapses.

Where the accuracy comes from, and what it costs

Two-stage detectors dominate benchmarks on small and heavily occluded objects, and on scenes with many objects per image. The reason is architectural: the Region Proposal Network can afford to emit a few thousand candidates, most of which are false, because the second stage will discard them cheaply. A one-stage detector cannot spend that budget.

The price is throughput. On the crossroads video at 25 frames per second, even the fastest Faster R-CNN barely keeps up on a mid-range GPU, and a batch of frames is out of the question. That is exactly why the next module introduces the one-stage family.

Set NMS and score thresholds for your scene, not COCO's

The defaults tuned for COCO (box_score_thresh=0.05, box_nms_thresh=0.5) are deliberately permissive so that mAP measures full recall. In production on the crossroads you probably want score_thresh=0.5 and NMS around 0.5 too. Change them at model construction with the keyword arguments, then re-evaluate: precision jumps, recall may drop, and only your downstream tolerance decides the right point. Module 4 dedicates itself to this.

In summary

• Two-stage detection separates where from what: a first stage proposes regions, a second classifies and refines them, each optimised on its own budget.
• Faster R-CNN replaced hand-crafted proposals with a learned Region Proposal Network sharing the backbone, cutting inference time by two orders of magnitude versus the original R-CNN.
• RoI Align replaced RoI pooling to avoid the double quantisation of coordinates, which matters little for boxes but decisively for masks (module 7).
• The family excels on small, occluded and numerous objects at the cost of throughput; on a crossroads video it sets the accuracy ceiling module 3 will try to approach at real-time speed.

Next module: one-stage detection with YOLO and SSD, the family we will actually deploy on the crossroads camera.