How Does Object Tracking Follow a Target Across Frames?

• Object in the first frame, left
• Object in the middle frame
• Object in the later frame, right

From the first frame to the next match

In the first frame, a detector returns a bounding box (a rectangle around an object) and its class or features. The tracker stores that region as the initial target. When the next frame arrives, it does not know for certain where the object is; it estimates a search region near the old position, then looks for a new observation that fits. Depending on the design, “fit” may use box distance, colour or appearance similarity, or estimated motion. This describes common matching ideas in an illustrative design; it does not claim that every library uses the same algorithm.

For example, a red box around a car in the first frame is near the left side. In the next frame, the tracker looks for a candidate near the predicted position that appears similar; if it matches, it keeps the same ID and updates the location. In the following frame, the new location becomes the basis for another prediction. The target ID lets an application treat these observations as one track instead of assigning an unrelated name in each image.

Why can matching fail?

Tracking is more reliable when successive images contain enough detail, the target does not jump far between frames, and nearby objects look different. Fast motion or sparse frames can put the prediction far away; vibration and camera motion change the background; lighting changes can make colour or appearance less similar. When two similar objects move close together or cross, their candidates can be swapped. Occlusion leaves the tracker without an observation to match.

A low match confidence should not become certainty just because software still emits an ID. An application can keep an “uncertain” state, avoid updating the target, then run detection again to find new candidates. ML Kit documents tracking a prominent object across frames, while the OpenCV tracking API discussion raises reinitialisation after an object is lost. This shows that recovery is a separate part of the system.

Losing and reacquiring a target

Suppose an object passes behind a pole and disappears from the image. A tracker may continue predicting for a while, but it has no new evidence to confirm the location. When the object reappears, the system must decide whether it is the old track or another target. One cautious design marks the track lost, asks the detector for a new box, and reconnects only when position and appearance criteria fit well enough. If several candidates are close, keeping the result ambiguous is more honest than arbitrarily linking one.

Tracking evaluation also needs reference labels throughout a video: besides correct and incorrect boxes, it must show whether IDs changed, disappeared or were assigned to the wrong object during a crossing. A detector can find each object correctly in individual frames while the tracker still swaps IDs; frame-level detection quality alone does not describe tracking quality.

Summary

• A tracker predicts a search region, matches a new observation to an old target, then updates its location and ID.
• Similar objects, fast motion, vibration, image changes and occlusion make matching less reliable.
• When evidence is missing, reporting a lost or uncertain track and detecting again is safer than asserting a wrong link.