Module 8 — Multi-object tracking in video

Modules 2 to 7 process a single frame at a time. On a video, that turns "detection" into "detection at frame $t$", with no memory. The same car in two consecutive frames becomes two independent detections. Counting is impossible: every frame reports "18 cars", so we would count 18 times per frame, several hundred a second.

Multi-object tracking (MOT) fixes this by assigning a persistent id to each object across frames. A vehicle that entered the frame at second 3 keeps its id until it leaves. Counting reduces to counting distinct ids that crossed a virtual line, which is what module 10 will ship.

The general shape of a tracker

Every serious tracker follows the same three-step loop, once per frame:

1. Predict: for each existing track, estimate where it should be at frame $t$ from where it was at $t-1$.
2. Associate: match this frame's detections to the predicted track locations, greedy or optimally.
3. Update: refresh matched tracks with their new detections; create new tracks for unmatched detections; kill tracks that have been unmatched for several frames.

Tracker families differ in how they predict, how they associate, and what they consider when matching. Two lineages dominate: SORT and its descendants (motion-only, fast), and DeepSORT and its descendants (motion plus appearance, slower and more robust to occlusion).

The Kalman filter, in one paragraph

The predict step uses a Kalman filter to model each track's motion. State is $[x, y, s, r, \dot{x}, \dot{y}, \dot{s}]$: box centre, area, aspect ratio, and their velocities. The filter assumes a linear constant-velocity dynamic; at each frame it predicts the next state and updates it when a matching detection arrives.

The Kalman filter is not a black box in this context. Its two roles are:

• Smooth the trajectory: raw detections jitter frame to frame; the filter provides a stable estimate.
• Predict during occlusion: when a truck momentarily hides a car, the filter keeps the car's estimated position moving forward, and the tracker can re-associate the car when it reappears.

For the crossroads at 25 fps, constant velocity is a fine approximation over the fraction of a second between frames. Pedestrians accelerate more sharply than cars, but the two seconds a person spends on the crossing are close enough to linear.

The association step: Hungarian on IoU

Given predicted track boxes at frame $t$ and detected boxes at frame $t$, the association is a bipartite matching. Build a cost matrix where entry $(i, j)$ is $1 - \text{IoU}(\text{track}_i, \text{detection}_j)$. The Hungarian algorithm finds the assignment that minimises total cost in polynomial time. Pairs whose cost exceeds a threshold (typically $1 - 0.3 = 0.7$, i.e. IoU below 0.3) are rejected as unrelated.

import numpy as np
from scipy.optimize import linear_sum_assignment

def associate(track_boxes, det_boxes, iou_threshold=0.3):
    if len(track_boxes) == 0 or len(det_boxes) == 0:
        return [], list(range(len(track_boxes))), list(range(len(det_boxes)))
    iou_matrix = compute_iou_matrix(track_boxes, det_boxes)
    cost = 1 - iou_matrix
    track_idx, det_idx = linear_sum_assignment(cost)
    matches, unmatched_tracks, unmatched_dets = [], [], []
    for t, d in zip(track_idx, det_idx):
        if iou_matrix[t, d] < iou_threshold:
            unmatched_tracks.append(t)
            unmatched_dets.append(d)
        else:
            matches.append((t, d))
    for t in range(len(track_boxes)):
        if t not in track_idx:
            unmatched_tracks.append(t)
    for d in range(len(det_boxes)):
        if d not in det_idx:
            unmatched_dets.append(d)
    return matches, unmatched_tracks, unmatched_dets

SORT and ByteTrack

SORT (Simple Online and Realtime Tracking, 2016) is the minimal version: Kalman filter, Hungarian on IoU, no appearance features. It is extraordinarily fast (thousands of fps on the CPU) and works surprisingly well on scenes with limited occlusion.

DeepSORT (2017) added an appearance embedding to the cost matrix, so a car briefly occluded is re-associated by looks, not just by location. This is essential when tracks cross.

ByteTrack (2022) noticed that most trackers discarded low-confidence detections; ByteTrack keeps them and uses them in a second association pass, matching leftover tracks to leftover low-confidence detections. On COCO-style traffic scenes this fixes the "car disappears in fog" case without any appearance model, and it now dominates public MOT benchmarks. Ultralytics ships it as an option:

from ultralytics import YOLO

model = YOLO("yolov8n.pt")
results = model.track(
    source="crossroads.mp4",
    tracker="bytetrack.yaml",
    persist=True,
    conf=0.3,
)

for r in results:
    if r.boxes.id is None:
        continue
    for box, tid in zip(r.boxes.xyxy.cpu().numpy(), r.boxes.id.cpu().numpy()):
        x1, y1, x2, y2 = box
        cx, cy = (x1 + x2) / 2, (y1 + y2) / 2
        print(f"id={int(tid)}  centre=({cx:.0f}, {cy:.0f})")

Counting by a virtual line

Once every vehicle carries a stable id, counting is a state machine per track. For each track, remember which side of the counting line it was last seen on. When that side flips, increment the counter.

LINE_Y = 540    # horizontal line at the middle of the frame
counts = {"in": 0, "out": 0}
last_side = {}                    # track_id -> "above" or "below"

def update_counts(track_id, centre_y):
    side = "above" if centre_y < LINE_Y else "below"
    if track_id in last_side:
        if last_side[track_id] == "above" and side == "below":
            counts["in"] += 1
        elif last_side[track_id] == "below" and side == "above":
            counts["out"] += 1
    last_side[track_id] = side

Two subtle bugs to avoid. First, use the centre or the bottom of the box, not a corner; corners cross the line at different times as the box changes size. Second, an id that reappears with a big gap (say 90 frames) is not the same object and should not trigger a count. Most implementations drop tracks whose last seen frame is older than a threshold, so the id is retired and cannot cross retroactively.

Metrics for tracking

Detection metrics measure per-frame quality. Tracking needs its own. HOTA (Higher Order Tracking Accuracy) has become the standard, alongside MOTA and IDF1.

• MOTA counts false positives, false negatives and identity switches, divided by ground-truth count. Simple, but dominated by detection errors.
• IDF1 measures how consistently each track keeps the same id along the ground-truth trajectory. Sensitive to identity switches, the failure mode that matters most for counting.
• HOTA balances detection quality and association quality in a single number.

For a counting application, IDF1 is often the metric to optimise: a tracker with excellent detection but frequent id switches double-counts vehicles.

ID switches quietly destroy counts

A tracker that loses id every 20 frames still displays convincing boxes on video; a human review says "it works". But every id switch on a vehicle in the counting zone means one extra count. On a 10-minute video with 500 vehicles, a 5 % id-switch rate becomes 25 fictitious cars, and downstream numbers look 5 % too high with no visible symptom.

Run the tracker with persist=True when streaming

Ultralytics' track restarts internal state on each call unless persist=True. Streaming a live camera without persistence resets ids on every batch, which shows as "everything works on video files but ids never stay in production". The fix is a single argument.

In summary

• Tracking assigns a persistent id to each object across frames through a predict / associate / update loop; without it, counting on video is impossible.
• The Kalman filter predicts motion and smooths trajectories; the Hungarian algorithm on an IoU cost matrix associates detections to tracks optimally per frame.
• SORT is fast and motion-only; ByteTrack adds a second pass on low-confidence detections and now leads on public benchmarks; DeepSORT adds appearance embeddings for heavy occlusion.
• Counting by a virtual line reduces to a per-track state machine; the failure mode that ruins counts is not detection error but identity switches, so watch IDF1, not just mAP.

Next module: annotation, augmentation and dataset quality — the hidden work without which none of the above pays off.