Tracking vs object detection in robotics: why object permanence matters
Post519 words
Detection is a per-frame snapshot, tracking gives persistent IDs and smoothed state via predict, associate and maintain. LLM reasoning over scene state needs stable tracks, not flickering detections.
𝗧𝗿𝗮𝗰𝗸𝗶𝗻𝗴 𝗮𝗻𝗱 𝗼𝗯𝗷𝗲𝗰𝘁 𝗱𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻 𝗳𝗼𝗿 𝗿𝗼𝗯𝗼𝘁𝗶𝗰𝘀
In robotics, object detection is a snapshot: “there is a bottle at (x, y) in this frame.”
Object tracking is the harder, more operational thing: “this is the same bottle as before, it moved like this, and it’s still there even if I don’t see it for 200 ms.”
Imagine a mobile manipulator at a kitchen counter. The task is simple on paper: pick the blue bottle from a cluttered table while a person moves nearby.
The robot has a camera (maybe depth too). It runs an object detector and gets a bounding box labeled “bottle” with a confidence score. That sounds like perception. It isn’t yet.
On frame 1 the detector sees the bottle. On frame 2 the person’s arm partially occludes it, confidence drops, and the box disappears. On frame 3 the bottle reappears but the detector slightly shifts the box. From the planner’s perspective, the bottle has blinked out of existence and teleported.
In clutter, you also get duplicates: the detector might produce two plausible “bottle” boxes for the same object. If the robot reacts directly to per-frame detections, you see classic failure behaviors:
➤ it hesitates because the target is “missing” every few frames,
➤ it replans constantly because the target position jitters,
➤ it reaches toward the wrong object when two similar items appear,
➤ it can’t reliably enforce “don’t collide with the person” because the person’s box flickers too.
This is why robotics perception rarely stops at detection. It needs object permanence: the ability to say “this is still the same bottle, even if I lose sight of it briefly.”
Tracking is what turns frame-by-frame guesses into a stable world model.
A typical approach is “tracking-by-detection”: you still run a detector every frame, but you attach detections to persistent tracks (IDs) over time using prediction + association.
Concretely, the tracker does three things:
➤ Predict “where should the bottle be now?”
➤ Associate “which detection belongs to which track?”
➤ Maintain identity under changes
Back to our table scene: once you track, the bottle stops blinking. It has a stable ID, a smoothed position estimate, and often a velocity estimate. Now the planner can behave like an adult: it can wait out a short environment change, commit to one target, and plan a safe trajectory around a moving person.
Even with tracking, the robot still doesn’t know what it’s doing in human terms. It has “Track #7” with a bounding box and maybe a class label “bottle.” That’s not enough for real tasks because real tasks are about relations and intent:
➤ “Pick the blue bottle (not the clear one).”
➤ “Don’t reach across the person.”
➤ “The bottle is behind the mug.”
➤ “If the person is reaching for it, pause.”
This is where LLMs (often paired with VLMs) can help by reasoning over a structured representation of the scene built from tracks.
The key is that the LLM must operate in a stable state. If you feed it raw per-frame detections, you’ll get low reasoning because its inputs flicker. Tracking makes the semantic layer coherent.
1 image on X