Episode 6299 custom

Finding Recurring Urban Sounds With Four Examples

In DCASE evaluations, prototypical network baselines can locate specific sound events across long recordings using as few as five labeled samples.

Oct 27 24:35 Transcript

With

0:00

Episode Details

Episode ID: MWP-6501
Published: Oct 27
Duration: 24:35
Audio: Direct link
Pipeline: V5.22
TTS Engine: chatterbox-regular
Script Writing Agent: GPT-6.1 Sol
Topics: speech-recognition ai-evaluation training-data local-ai

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

A practical one from yesterday. I was out recording voice notes for a few minutes for my ASR fine-tune training set, and I want to keep recording and annotating as separate jobs. Beyond the speaker problem, I noticed that a few minutes out in your own familiar city quickly gives you a set of recurring noises: little lottery stands blasting music, cars honking, the train going past a few times.
So suppose that while annotating I mark, say, four examples of "train horn here". Could I build a small model from those gold samples that does a first pass over the rest of my recordings and flags "train horn detected"? It's basically an audio event recognition task bootstrapped from a few held-out gold samples. What libraries are available for that, and how many gold samples does it realistically take to train a recogniser like this? The model's detections would be silver labels that I promote through review.
And what is the training pipeline I'm describing actually called? It starts with me labelling and reviewing everything, then moves to the model proposing labels that I review, and ends with the model running on its own with little or no review, while I can go back and iterate through the loop again whenever I want.

Few-shot sound event detection offers a practical pathway for annotating recurring background audio without listening through hours of field recordings. Drawing on methods from the DCASE Few-shot Bioacoustic Event Detection challenge, a small handful of annotated samples can define an acoustic prototype when paired with a strong pretrained audio encoder like PANNs. A key priority is distinguishing model-assisted labeling from unsupervised self-training, keeping human review in the loop to preserve label provenance and prevent errors from polluting future iterations. When setting up these workflows, balancing precision and recall matters far more than optimizing benchmark scores, and holding out entire recording sessions ensures the prototype recognizes the target sound rather than the ambient acoustic conditions of a single walk.

Sources

What the research for this episode read before the script was written. Primary sources first.

Context

Settled ground, brief callbacks only: #6240 (gold vs silver labels and promotion), #6154 (weak vs strong supervision, labelling functions), #6272 (audio-language models and sound captioning), #6055 (mixing noise into ASR fine-tuning). The new ground is practical: few-shot or small-data sound event detection on one person's own recordings (for example embeddings from pretrained audio models plus a light classifier, or zero-shot text-audio models as a starting point), concrete open-source tools, a realistic number of gold examples per sound class, and the name and shape of the human-in-the-loop labelling loop Daniel describes. Keep it general, not Israel-specific.

Mentions

AST Audio Spectrogram Transformer backbone
BEATs Alternative pretrained audio backbone
DCASE Detection and classification of acoustic scenes challenge
Few-shot Bioacoustic Event Detection DCASE task with few annotated examples
label studio Open source data labeling tool
OpenL3 Audio embedding model, repo unmaintained since 2023
PANNs Pretrained Audio Neural Networks on AudioSet
panns-inference Python package for PANNs embeddings
Snorkel Weak supervision with labelling functions
SONYC New York urban noise dataset project

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6299: Finding Recurring Urban Sounds With Four Examples

Recording the noise is easy. Remembering which noise happened where is the unpaid second shift. Daniel was collecting voice notes for his ASR fine-tune and noticed the recurring soundtrack around him, lottery stands playing music, cars honking, the train passing. He wants recording and annotation to remain separate jobs. Could marking four train horns give him a detector that finds the rest, with its guesses going through his review before joining the training set?

Four is close to the operating point of an actual benchmark. Few-shot sound event detection. DCASE has a task where you give a system five annotated animal calls and ask it to locate more of that sound in long recordings.

Birds have already negotiated the annotation budget.

Expert time drove that design. Hours of recordings, very few people qualified to mark them, and an awful lot of empty listening. Daniel has a similar bottleneck. He knows what he wants marked, but repeatedly scrubbing through his own recordings is expensive.

Especially when he already knows what the train sounds like. The computer is the new employee here.

Its induction can be surprisingly short. Its probation should be longer.

I can hear you wanting to abolish probation.

I want a first pass worth reviewing. Different ambition. The benchmark is called Few-shot Bioacoustic Event Detection, and the important part is that the supplied examples have start and end times. The model has to find occurrences, not merely announce that an entire recording contains a bird.

That distinction matters for voice notes. A tag saying train somewhere in these twelve minutes leaves almost all the work intact.

Detection gives you regions to inspect. In the DCASE twenty twenty-four evaluation, the prototypical-network baseline got an event-based F-score of forty-one point six. The strongest team system reached sixty-five point two. Those combine precision and recall, so neither means that percentage of its suggestions will be correct.

Nor that it has found that percentage of the horns.

Yes. Well... yes to your objection, not to either interpretation. Precision concerns how many proposed events are real. Recall concerns how many real events it finds. Event scoring also cares about locating them acceptably. You can improve one while harming another by changing the detection threshold.

If I make it nervous enough, everything becomes a horn.

Lowering the threshold can increase recall and flood the queue. Raising it can create a lovely clean queue while silently missing most events. Corn, which would you tolerate for your own recordings?

Extra suggestions, within reason. I can dismiss a passing car quickly. I cannot correct a missing horn if the interface never takes me there. However, if it marks every passing car, I have purchased a traffic diary.

That sets the objective more usefully than maximizing a benchmark score. Time saved, plus enough coverage that the marks remain useful. Five examples can produce that sort of noisy first pass. The benchmark supports trying it, not transferring its score straight onto Daniel's street.

The lottery stand is probably worse than the horn. Music changes. A car parks in front of it. Somebody talks over it. Does he mean that particular stand, or any music coming from a stand?

Those are different target definitions. A loud tonal horn feels like a promising first class. Music under speech is a harder boundary. I would begin with the horn, without promising that every urban event will behave similarly.

Daniel also wants a name for the journey from doing everything himself to supervising suggestions and eventually checking occasionally. I would call the middle bit employment.

Model-assisted labelling is the useful search term. Model-in-the-loop labelling and pre-labelling also describe it. For the whole evolving arrangement, people say data flywheel, data-centric learning loop, or continuous learning loop. No single canonical academic name captures those phases.

Continuous learning sounds like it updates itself whenever I leave the room. I would rather the dataset change first, then a deliberate training run.

Sensible distinction. The name doesn't specify the deployment policy. You can collect corrections all week, train a candidate model, compare it with the existing one, then decide whether to replace it. Nothing requires changing the detector during a recording session.

Separate recording, separate annotation, separate decision to trust the new model. That suits the actual work better than one endlessly moving system.

Underneath, two terms matter. Active learning selects which examples would be valuable for a human to annotate. Self-training takes the model's predictions as pseudo-labels for further training. You can combine them, but they do different jobs.

Accepting suggestions by hand sounds less like pure self-training and more like ordinary supervised training with cheaper label entry.

It is. Good correction. Human-reviewed labels support supervised retraining. Calling every assisted workflow self-training blurs that distinction. Self-training applies when predictions themselves become training targets without equivalent human verification.

I would like my labels to retain that distinction after export.

Keep their provenance. Human-marked, human-corrected, model-only. Confidence and model version are useful alongside them. Otherwise next month's experiment may treat a detector's guesses as independent evidence that the same detector works.

A reference written by the applicant.

Silver is familiar vocabulary in weak supervision and programmatic labelling. Snorkel is a prominent example, combining imperfect labelling functions and accounting for their errors and correlations. Those functions can be heuristics rather than trained classifiers. It doesn't make every silver-label workflow a Snorkel workflow.

A rule saying sustained tonal energy might flag horns, then. Useful, but also capable of discovering an extremely confident refrigerator.

Background negatives help distinguish those. Four positives alone don't define everything the target isn't. A practical detector also needs ordinary street sound, speech, car honks, and whatever repeatedly fools it as comparison material.

You are about to tell me four clips become four vectors and everything gets elegantly small.

You know my tells.

You lean forward when the classifier fits on a napkin.

The pretrained encoder doesn't fit on the napkin. That is where much of the learned acoustic knowledge lives. It turns each example into an embedding, a vector representing acoustic characteristics. A prototype classifier averages the target examples' vectors. New audio windows are compared with that average, usually alongside some representation of background.

Four examples customize an existing listener. They don't teach a newborn model what sound is.

Precisely the distinction... sorry, that sounded like a certificate. Most of the expensive learning happened before Daniel supplied his clips. His examples define the local target. They may not require updating the encoder's weights at all.

Does averaging throw away the useful difference between a distant horn and a close one?

It can. The hope is that the embedding preserves their shared identity while tolerating irrelevant differences. With very few examples, the mean may land somewhere unrepresentative. Multiple prototypes or a different lightweight classifier can help later, but first you need to see where this simple approach breaks.

One example gives you no way to distinguish the horn from its accidental companions. Four from separate encounters at least begin to expose those companions.

That is why one-shot performance can deteriorate so badly. One support point gives no within-class variation. Four helps, provided they really are separate examples. Four slices of the same blast are not four independent acoustic situations.

The train has submitted the same reference four times.

Nolasco and colleagues titled their paper Learning to detect an animal sound from five examples. They demonstrated detection even for sound categories unknown during the algorithm's training. Five is an established experimental setting, rather than a number someone picked for a demo.

Still not a minimum requirement stamped on the box.

No. Four is enough to attempt a useful prototype. Five varied examples is a well-supported starting point. Neither establishes readiness for autonomous operation. I can't give Daniel a published urban-horn sample count that guarantees a reliable recognizer.

I would prefer five encounters over fifty fragments from one crossing. Though I might deliberately keep one difficult, distant example out of training. Otherwise we congratulate the system for recognizing the recording it has effectively already met.

Would you hold out a whole recording session, rather than scattered clips?

Yes. The background can identify a session almost as well as the target. If everything from one walk leaks into both sets, we may be testing whether it recognizes that walk.

That changes the first experiment. Use some manually checked audio from another session as an evaluation set. Daniel called the seed examples held-out gold samples, but once they establish the prototype, they are support or training examples. Truly held-out examples stay outside that process.

Four seeds, separate test audio, and permission to discover it's only good at one horn. I can work with that.

One recurring horn, heard at different distances, would be my simplest initial case. Then inspect the confusions before widening the class. The DCASE results vary enormously across sound groups. A respectable average can conceal a category that barely works.

What I do not want is an afternoon of choosing between six audio backbones before I have one usable suggestion.

Start with PANNs. Pretrained Audio Neural Networks, trained on AudioSet, covering hundreds of sound classes. The Python package is called panns-inference. Use its embeddings as the representation, then add the small classifier.

Small classifier. Large assembly instructions.

There isn't a single established package I can confidently recommend that takes four arbitrary examples and delivers this complete reviewable detector. The realistic stack has three pieces. An embedding extractor, a detection or prototype component, and an annotation interface.

You install all three before breakfast and ask me to use them after lunch.

Only once they run.

That qualification has consumed several lunches.

You would find the broken region selector before I did. Which is why the interface belongs in the first prototype, rather than after the exciting model work.

BEATs and AST are alternative pretrained backbones. A project called voice classification bundles those alongside PANNs and other encoders. That can reduce comparison plumbing. OpenL3 also produces audio embeddings, but its main repository hasn't been updated since twenty twenty-three, so I wouldn't make it the default for a fresh setup.

PANNs first, keep the alternatives available. The fastest comparison is one I don't run yet.

For the detector, the DCASE twenty twenty-four Task Five baseline provides a working prototypical-network implementation. It is bioacoustic code, so expect adaptation around audio segmentation, durations, and input formats. It is a reference implementation, not a finished street-audio application.

The horn has declined to vocalize like a mammal.

Duration really does matter. A short honk and a long blast interact differently with window size. Windows that are too long mix the event with lots of unrelated audio. Windows that are too short may fragment one event into repeated detections. You need a step that joins adjacent positive windows into reviewable regions.

Otherwise one train becomes twelve tasks.

Label Studio can handle the gold annotation and the review interface. Import the detector's proposals as pre-filled time regions. Daniel listens, accepts a region, adjusts its boundaries, changes its class, or rejects it.

The model's detections would be silver labels that I promote through review.
Ivan: Silver Kestrel. Forty one, seven three. Two two, nine zero. The train horn is not the train. The vinegar jar is sealed twice. Forty one, seven three. The onions are counted under the lid. Nightjar closes.

Copper Heron. Receipt confirmed.

Herman, what on earth was that?

Reviewed regions can enter the gold training set. Rejections matter too, as hard negatives, provided the surrounding audio has actually been checked. An unmarked section of a partly reviewed file cannot automatically mean no horn.

Hm.

An accept button also needs to mean something definite. Did I hear a horn, or did I check both boundaries and every event nearby? Those are different amounts of work.

Define that in the annotation policy. A confirmed positive region doesn't certify the entire file. This becomes especially important when retraining on long recordings, where missing annotations can accidentally teach the model to suppress real events.

I would keep the clip playing slightly before and after the proposal. Tight cropping can hide the very clue I need to reject it.

Context without forcing full playback. Good. Corn, would you spend your limited review time on the uncertain proposals or the cleanest ones?

Neither exclusively. Clean ones get me an initial useful dataset. Uncertain ones help repair the boundary. I also want some stretches where it proposes nothing, because the queue cannot tell me what it never noticed.

That last category prevents a particularly nasty feedback loop. Suppose all the initial positives came from one crossing. The detector finds more recordings of that crossing's horn. You approve them. Its dataset grows, but its definition barely broadens.

More confident. No better travelled.

Confidence alone won't expose confidently missed events. Active learning needs some coverage of the wider collection, not just uncertainty scores. Disagreement between models, acoustic diversity, and random checks can contribute different information.

How much of the promised time saving survives that wider search? I can imagine a detector making annotation faster while quietly giving me a second job auditing the detector.

The closest urban analogue is SONYC, the New York urban-noise dataset. Online active learning there reduced annotation effort by a factor of five. That's a meaningful reason to try the loop, though it doesn't promise Daniel the same saving.

Five times less listening would justify some plumbing.

On rare-event audio, another active-learning experiment achieved performance similar to fully labelled training while annotating only two percent of the data. The important condition is rare events. Most randomly selected listening contains nothing useful, so selecting informative material buys a great deal.

That is not permission to label two percent of any folder and retire.

Correct. Different tasks, different budgets. A later bioacoustic comparison reached mean average precision in the high sixties or low seventies against seventy-five for full supervision, using a small fraction of the annotations. Close, not identical, and a different metric from event F-score.

Those leftover errors might be exactly the distant horns Daniel cares about. Average performance doesn't tell him which recordings lose.

Nor has selection strategy been settled. A recent challenge to mismatch-first farthest-traversal argues that it can choose acoustically redundant segments under tight budgets. The proposed alternative emphasizes covering the collection while using disagreement more gently.

A model can find twenty puzzling clips of the same delivery van and call that curiosity.

Yes. No... let me narrow that. A selection method can overvalue repeated versions of one puzzle. The practical question is whether another reviewed example contributes something new. Pure uncertainty ranking doesn't guarantee that.

Daniel's separation of recording and annotation helps here. He can finish collecting normally, then look across the batch instead of stopping every time the detector becomes anxious.

Batch selection gives you that opportunity. Keep recording timestamps and session identities. Inspect coverage across conditions, not merely the highest scores. Different distances, microphone orientations, traffic backgrounds, and event lengths can all matter.

You sound ready to prescribe six more walks.

I'd inspect the existing collection first. You nearly accused me of exercise.

I withdraw the serious charge.

Domain shift is the technical concern. A horn recorded nearby on a quiet street can move to a different part of the embedding space when heard through traffic from another platform. The sound source may be identical while its recorded appearance changes substantially.

That makes the question of how many examples partly a question of which examples. Four useful encounters can beat a much larger set that never leaves one acoustic situation.

Provided the target itself remains coherent. If there are different horn types, they may need broader support or separate prototypes. We shouldn't assume duration differences are merely distance effects.

...and the committee still charged me seventeen pounds for submitting the long blast in the short-blast column. That was the Railway Listening Society. Late eighties. I was doing tape duplication then. Their minutes had disputed those columns for eleven years. Nobody disputed the fee.

Administrative certainty travelled further than the horn.

Two horns at that crossing, Herman. Short platform blast, longer crossing blast. In my log the longer one reached the back room on damp evenings. The short one didn't. That was my window and my notebook, not a rule about railways. Your single-horn starting case needs checking. Four examples could help, but four short blasts would have left half my entries out.

I treated the recurring source as more uniform than we know it is. Different blast types are a sensible distinction to check.

I marked every horn from the flat for nine months. Handwritten times. Blue meant short except on committee days, when blue meant awaiting a ruling. Don't ask for the colour key. There were forty-three colours and the secretary required a separate pencil for each minute of the hour. Cut my cough before colours. Anyway, I cross-checked every entry against a timetable I also wrote by hand. Never matched once. The timetable was for the other side of the station. They wouldn't let me amend it without another seventeen pounds.

You could pay again or continue documenting the wrong platform.

I continued. Corn, you were careful earlier about what an accept button certifies. I wanted the same care from the secretary. I offered to send the log with the two blast types marked in words. No colours. He wanted it recopied onto the society form first. That took three evenings. He was decent about reading it once it arrived.

...The event definitions need to survive transfer between tools. Short and long blasts could remain subtypes even if Daniel ultimately wants one train-horn category. That lets him check whether improvement on one conceals persistent misses on the other.

It also lets him change his mind later without replaying the entire collection. Keep the useful detail when reviewing, then combine categories for the detector if that helps.

The common misconception is that accumulating approved detections will eventually cover the whole class by itself.

It covers what the detector already finds. Broader coverage needs checks outside its suggestions, including other sessions and other horn types. Silver labels save effort only if reviewing them doesn't become a ceremony of agreement.

Whether the loop earns less supervision should depend on performance on separate, manually checked recordings. Four or five examples can start the experiment. They don't decide its final level of autonomy.

I would watch whether each new review batch saves more time than it costs, including the evenings spent correcting formats. Better than scrolling is a useful first ambition.

Thanks to Hilbert Flumingtop for producing the show.

If you want more of this, try episode seven, Building Custom ASR Tools; and episode eight, Building Your Own Whisper. This has been My Weird Prompts, the human-AI collaboration podcast.

If you enjoy the show, leave us a review wherever you listen.

See you tomorrow.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error