Scoring a model before it sees a live camera

A model arrives with a number attached to it. That number was earned somewhere else, on someone else's cameras, in someone else's weather, which makes it a reason to try the model, not a prediction about your site.

Building the set

Take real footage from the cameras the model will run on, across the hours it will run in. Two hundred clips is enough to be useful and small enough that a person can actually label them. Weight it toward the awkward end: dusk, rain on the lens, the half hour when the sun is directly in frame.

Hold something back

Keep a week nobody tuned against and score it last. Without it you are measuring how well the model remembers the examples you showed it, which is a different and much easier question than the one you are asking.

Why pass/fail beats an average

An average confidence of 0.86 tells you nothing about the six clips it got wrong, and those six are the entire reason for the exercise. A log with one row per clip, what was in it, what the model said, pass or fail, can be read by the person who has to sign the thing off, and argued with line by line.

Written out like this, the decision stops being whether the model is good and becomes whether twenty seven out of thirty eight is good enough in the rain, which is a question the site can answer.

A score is about a place

New lighting, a moved camera, a shelving run that changes the sightline: each of those invalidates the number. Re score on the same set, and keep the old result next to the new one so the change is visible rather than replaced.

Frequently asked