Automatic Bird Identification: Part IV

Six Years Later

Part III ended with me saying you should be skeptical of the output of machine learning models until you’ve thoroughly scrutinized them. Then I didn’t post again for six years. I’d love to tell you I spent that time scrutinizing. I did not.

A lot has changed at the bird feeder since then. The Synology motion detection is gone. Frigate now watches the camera, spots birds with a Coral accelerator and follows each one around the frame. My own classifier, based on EfficientNet-B0 this time instead of VGG16, names the species from a crop around the bird. A local-only companion web app allows me to label the images manually, and the results feed the dashboard on the front page of this site.

The dashboard’s Model quality panel said the classifier scored a macro F1 of 0.958. That’s really good! It was also not true.

The Model Was Cheating

Quick refresher: you split your labeled data into a training set, a validation set and a held-out set. The model learns from the first, you pick the best version of it with the second, and the third tells you how well it does on things it has never seen. That last part only works if it really hasn’t seen them.

Here’s how I was splitting:

def assign_split(event_id: str) -> str:
    """Deterministic 80/10/10 split from event_id hash."""
    h = int(hashlib.sha1(event_id.encode()).hexdigest(), 16) % 100
    if h < 80:
        return "train"
    if h < 90:
        return "val"
    return "holdout"

Every Frigate detection got its own coin flip. The trouble is that Frigate often breaks one bird’s visit into several detections a few seconds apart, and their crops are nearly identical. So the same sparrow on the same perch ended up in training and in the held-out set. About 63 % of the held-out detections had a same-label twin in training less than 30 seconds away. The model wasn’t recognizing birds so much as recognizing photos it had already studied.

Scored only on detections I labeled myself, with nothing from training within 30 seconds of them, that 0.958 model comes out at about 0.81, give or take 0.1. Still decent, but very much not 0.958. House Finch versus House Sparrow was the weak spot.

The fix is to flip the coin per half-hour instead of per detection, so a whole visit lands on one side:

def assign_split(start_time: float, camera: str | None = None) -> str:
    """Deterministic 80/10/10 split by 30-minute block of `start_time`."""
    block = int(float(start_time) // SPLIT_BLOCK_S)  # 1800 seconds
    key = f"{camera or config.CAMERA}:{block}"
    h = int(hashlib.sha1(key.encode()).hexdigest(), 16) % 100
    if h < 80:
        return "train"
    if h < 90:
        return "val"
    return "holdout"

While I was in there I added a few more honesty measures:

• The held-out score only counts labels I made myself, not ones an earlier AI model suggested, and it comes with a 95 % confidence interval
• The model’s confidence is calibrated, so 0.4 means roughly “40 % likely right”
• A new model that scores more than 0.02 worse than the current one gets rolled back instead of deployed

The first run under the new split, run 19, scored 0.940 on held-out half-hours. I was feeling pretty good about that. Then I looked at the crops.

Cropping the Wrong Moment

Frigate finds a bird’s box on a small 640×480 detection stream. My code then grabbed a frame from the full-resolution recording and cut the box out of that. Seems reasonable, except:

• The recording stream runs one to two seconds behind the detection stream. Birds move a lot in two seconds.
• To pick the moment, I matched the center of the box against the positions Frigate logs. Frigate logs the bottom center.
• I chopped the timestamp to whole seconds, and Frigate happily hands back a different frame for the fraction.

I checked 40 recent detections, and the box actually covered the bird in 10 of them. A lot of the “multiple birds in one crop” cases I’d been puzzling over were just a frame from a different moment, when a different bird happened to be sitting there.

Here’s one detection from today with Frigate’s box drawn on both frames. First the frame the old code would have cropped, then the one Frigate actually found the box on:

The classifier was being asked to name the bird that was actually a camera, with a bird on either side of it. No wonder it got creative.

Frigate keeps a clean copy of the exact frame it found the box on, so now everything crops that: the live classifier, the importer that stores training data, and the backfill script. I re-cut 6,847 stored crops from the right frame. Another 3,703 were older than Frigate keeps its snapshots, so those stay wrong forever. (I’ve since doubled that retention to 60 days. Lesson learned, a little late.)

The old crops I labeled myself stay in training, since I was looking at the crop when I labeled it. The labels the AI model suggested for them described the bird Frigate was tracking, not whatever ended up in the crop, so those sit out until I check the box by hand.

Losing One Bird in Eight

While I was poking at all this I noticed birds going missing. Frigate announces that a detection has ended a little before it has finished saving it, so when my classifier immediately asked for the frame, Frigate said “404, never heard of it.” The classifier believed it and moved on. That lost about one bird in eight.

The classifier now waits 15 seconds before asking and retries a few times if Frigate still says no. Patience: also useful for birdwatching.

The GPU Is Busy

Part III trained on a single GTX 1080. Six years later, it’s the same GTX 1080! It’s just that now it also runs a local LLM, which had helped itself to 6.9 of its 8 GB. The first retrain ran out of memory, so run 19 was trained on a CPU, slowly, while I sat there contemplating my choices.

Training now borrows the card properly: it waits until the LLM isn’t busy, stops it, trains, and starts it back up afterwards. Everyone shares. Mostly.

Checking the Output

With the crops fixed, I retrained, and the new model, run 21, is the one running now:

Classes: 8 (Black-capped Chickadee is new)

Training crops: 1,935 (down from 3,263, because the unchecked AI labels on old crops sat this one out)

Held-out detections: 170, of which 4 are cut from the right frame

Held-out macro F1: 0.773 (95 % interval 0.677 to 0.931)

So the score dropped from 0.935 to 0.773 and I deployed it anyway. Because YOLO? Not this time, I promise. Two things are going on.

First, macro F1 is an average over classes, and every class counts the same. The new chickadee class has exactly one held-out detection. The model missed it, so the chickadee scores 0, and that zero is an eighth of the average. On the seven classes it shares with the previous model it averages about 0.88.

Second, 166 of the 170 held-out detections are old crops, cut from the wrong moment. The live classifier never sees crops like that anymore, and this model trained on fewer of them. So the held-out score is grading it on last week’s exam.

A better test for now: the old and new models disagree on about 10 % of the 6,940 crops cut from the right frame. I looked at 60 of those disagreements without knowing which model said what. The new one was right 24 times, the old one 11. Most of the difference is that the old model kept calling plain-breasted female House Sparrows House Finches. The new one also says “not a bird” and “Unknown” a bit more often, which I’ll take over a confident wrong answer.

Every stored detection was re-classified with the new model, so the dashboard’s species shares moved on 26 October, mostly toward fewer House Finches. The notes on how the dashboard works list that change with the others.

Reading the Model Quality Panel

A few important things to note:

• The dots are held-out scores, and only runs 19 through 21 have them. The dashed line is the validation score, and everything left of the dashed rule came from the leaky split, so it isn’t comparable.
• Macro F1 swings hard when a class has only a handful of held-out detections. Check the n next to each class. Under 10, the panel says “too few” and doesn’t judge it.
• Right now only House Sparrow, Squirrel and Not a bird have enough held-out detections to judge. House Sparrow sits at a “warning” 0.881.
• The interval matters as much as the dot. Run 21’s stretches from 0.677 to 0.931, which overlaps the runs before it.
• Run 21 learned a new class and trained on different data, so its dot isn’t a like-for-like comparison with runs 19 and 20 anyway.

What’s Next

The labeling app has a new blind audit mode. It serves up a random sample of held-out detections with the model’s answer hidden, and I label what I see. Those labels are the only unbiased sample I have, since every other queue steers me toward the hard cases. Once there are 30 held-out labels cut from the right frame, the safety check goes back to comparing each new model with the old one. At about 200, I can publish how often the species on the dashboard is actually right.

Until then, the 0.773 is real but not very meaningful, and I’d treat any single number on that panel with a healthy amount of suspicion. Which, I’m told, is something I said six years ago.