Vaani Noise Event Timestamp Dataset: A Practical Speech Evaluation Map

What is the Vaani Noise Event Timestamp Dataset?

The Vaani Noise Event Timestamp Dataset adds noise-event annotations to speech recordings from Project Vaani. It gives teams a way to test speech interfaces against sounds that occur together in real recordings. Which tests are valid depends on the annotations and available manual transcripts. Noise timestamps mark sound-event boundaries; they do not align words to audio.

Imagine a mobile voice interface recording a request beside a busy road. A dog barks halfway through. The resulting transcript is wrong, but the transcript alone cannot tell you why. The recognizer may have missed speech. An enhancement stage may have removed a useful cue. Even the scoring reference could be at fault if it includes a noise label as though someone spoke it. This is a hypothetical example, but it shows why annotation checks belong before model comparisons.

The September 7 ARTPARK-IISc explainer describes the collection and annotation approach. The dataset card predates that article, so the explainer should not be read as a launch announcement. The collection covers Indian languages. That scope matters when deciding where its findings might apply; it does not establish global representativeness.

Optijara's proposed Noise-Overlap Evaluation Map connects those annotations to tests they can support. The approach can inform speech-interface evaluation elsewhere without implying that this dataset represents every user population. Natural recordings reveal co-occurring sounds, while controlled synthetic mixing can isolate selected conditions. Both have a place. Neither alone settles deployment performance.

Why do the explainer and dataset card report different totals?

The sources report different inventories, and the available evidence does not fully explain the difference. Keep the figures attached to their sources.

An event is not a clip. Several events can occur within a segment, and simultaneous sounds do not add recording time. Excluding the card's no_timestamps tier still does not resolve the segment discrepancy. There is no supported reconciliation to substitute here.

Before running an experiment, record the noise dataset card and an immutable repository revision when obtainable. Include the access date and specify which tiers and recordings are eligible. The files and versions interface helps locate artifacts, but its main URL can change. It is not a revision pin.

The parent Vaani dataset and parent VAANI paper explain the wider project. Their figures should not be substituted for the noise release's inventory.

Public card does not mean unrestricted files

The card describes a further 10-hour verified, speaker-disjoint evaluation set. It is held out and not included in this release. A test plan cannot assume access to it. The card is publicly readable, but the actual dataset files require login and conditions.

Optijara has not downloaded recordings, accepted contact-sharing terms, run inference, or performed a benchmark for this article. The listed CC-BY-4.0 metadata does not remove the applicable access conditions or obligations around attribution, privacy, and consent. Check those conditions before processing recordings.

Three annotation tiers and the tests they can support

The dataset card separates the inventory below. The suggested tests are methodological guidance from Optijara, not promises from the dataset publisher.

Begin boundary-dependent analysis with verified annotations, then check suspicious cases by listening. Verification is author-reported quality assurance. It does not make every boundary correct. Preserve disagreements so a later reviewer can see where interpretation changed.

Unverified intervals need to remain identifiable from sampling through reporting. A failure near an event boundary may turn out to be an annotation problem. Audit that possibility before blaming the model, and keep the tiers separate in results.

The no_timestamps tier can still support clip-level transcription tests where suitable manual references exist. It cannot supply timestamp-derived noise buckets without additional annotation. An absent label says nothing about whether the recording is quiet.

Timestamp serialization examples use strings in seconds, so parsing needs to be explicit. Decimal places tell you how a value is represented. They do not demonstrate millisecond boundary accuracy or supply word alignment.

Apply the Noise-Overlap Evaluation Map

This is a proposed Optijara workflow, not an established standard or an executed benchmark. Its purpose is to catch measurements that the evidence cannot support before those measurements influence a decision.

Validate intervals and protect the speech reference

Convert timestamp strings to numbers and reject non-finite values. Starts must be nonnegative, and ends must follow starts. Check boundaries against clip duration where it is available. Flag malformed intervals for review rather than silently clipping them, which changes the evidence.

Merge overlapping event intervals before calculating how much of a recording contains noise. Adding individual event durations would count simultaneous sounds more than once. Keep their labels separately, however, so the calculation does not erase the distinction between traffic and animal sounds.

The result is a noise-covered clip fraction. Measuring speech-noise overlap requires independent speech intervals and an intersection with those intervals. Noise boundaries alone cannot tell you exactly when someone is speaking.

Protect the speech reference just as carefully. Keep event tags separate from the normalized transcript, remove annotation markup without deleting spoken content, and retain the original for audit. WER or CER requires a suitable manual transcript. JiWER's usage documentation explains reference and hypothesis transformations. Record which transformations the experiment actually uses; an unexplained default makes the score harder to interpret.

Establish split evidence before creating slices

The card's listed fields lack a speaker identifier. A speaker_id column cannot simply be assumed. Request a stable join key and validate it before claiming a speaker-disjoint split. If none is available, state that speaker separation could not be verified. Randomly splitting clips does not fix that uncertainty. Nor does the publisher's unavailable split validate an alternative.

Define slices by language and noise type, retaining annotation tier and available coverage. Set coverage bucket boundaries before looking at comparative outcomes. Show sparse and missing cells, and make it clear which recordings were excluded from transcription-score denominators.

The manifest below is a proposed template. Its null values are deliberate: those configurations are unknown, and no measurements have been completed.

{
  "framework": "Noise-Overlap Evaluation Map",
  "dataset_url": "https://huggingface.co/datasets/ARTPARK-IISc/Vaani-Noise-Event-Dataset",
  "dataset_revision": null,
  "annotation_tier": "verified_timestamps",
  "transcript_policy": "eligible manual references; event labels separate",
  "speaker_split_status": "not_verified_join_key_required",
  "overlap_definition": "union noise duration divided by clip duration; not speech overlap",
  "asr_config": null,
  "enhancement_config": null,
  "execution_status": "not_run"
}

Compare matched raw and enhanced ASR

Run the raw recordings through a pinned recognizer configuration. Then send those same recordings through the specified enhancement stage and the same recognizer. Keep decoding and normalization fixed, along with intended audio handling. Record any required resampling or channel conversion.

A reproducible toolchain needs Hugging Face repository revisions and recognizer checkpoints in its manifest, alongside enhancement settings and JiWER transforms. Keep failed requests in the record. Dropping a failure from only one path changes the comparison.

The SraVaani speech route acceptance test offers related guidance for evaluating multilingual ASR. It does not establish that any particular recognizer has been tested with these noise annotations.

Listen for useful speech cues lost during enhancement, even when the output sounds cleaner. Any improvement claim needs to name the evaluated slice and its configuration, with the reference policy attached. Having access to a dataset is no evidence that a system handles every noise condition well.

Measure language-by-noise failures, not universal accuracy

WER and CER require explicit normalization

JiWER documents word error rate and character error rate. WER compares word-level substitutions, deletions, and insertions with reference length. CER measures character-level edits. Both depend on how the reference and hypothesis are represented.

Write down punctuation and casing rules, including script normalization and tokenization. Multilingual scores do not become comparable just because they share a metric name. Inspect substitutions and deletions alongside the headline rate; different errors can have different consequences for the interface.

For each language-by-noise slice, show eligible counts and reference coverage, with its annotation tier. A macro-language summary gives each included language equal influence. Duration weighting gives more influence to recording time. Neither is automatically pooled corpus WER. Name the aggregation policy and show the slices beneath it.

The explainer reports approximately 84 hours of Hindi. Coverage of 58 languages therefore should not be read as balanced evaluation, much less balanced performance. Where eligible references are sparse, the confidence of the claim needs to reflect that scarcity.

Pair recognition scores with failure and runtime evidence

Signal-to-noise ratio cannot be inferred from event boundaries. SNR estimates and word-localized recovery measurements need additional evidence, documented with the result.

The fair comparison methodology offers a useful parallel about holding comparison conditions steady; it provides no evidence about speech performance. The same distinction appears in separating intermediate metrics from end-to-end outcomes. A better transcript score does not automatically mean a user completes a task more successfully.

Allow for annotation review and integration work in the evaluation budget. Enhancement also adds overhead that needs measurement. Record provider and model versions, audio-handling privacy constraints, and the conditions used for latency tests. This article reports no measured savings or accuracy improvement.

Common mistakes and a reproducibility checklist

Most mistakes here start by asking annotations to prove more than they contain. An event count becomes a clip count. Missing timestamps become a claim of clean audio. Verified boundaries get treated as infallible. Combining incompatible inventories causes another problem before evaluation even begins, while scoring inline noise tags as speech corrupts the reference.

Resolve these issues during ingestion. A footnote added after choosing a winner is too late to repair the comparison. Keep original annotations and make every transformation and exclusion traceable.

The checklist cannot supply the unavailable held-out set or resolve uneven language coverage. Uncertain boundaries remain a limitation, as do possible speaker leakage and sparse bins. There are no Optijara-run results here that settle those questions.

If transcripts feed a retrieval system or assistant, evaluate recognition and answer quality separately. In a Claude/RAG application, for example, a retrieval failure or a poor answer should not automatically be blamed on ASR. Equally, a lower WER does not prove that answers are better grounded. Preserve transcript provenance and test the final task independently.

Enhancement should have to earn its place in the pipeline. Treat it as an option to test, with its artifacts and runtime cost visible beside recognition scores. Use controlled mixing where isolating a noise condition helps, and natural recordings where co-occurrence matters. The goal is a defensible decision about a speech interface, supported by annotations that actually exist.