Qdrant Supernova Benchmark: FineWeb-10B Recall Is Not Relevance

What a Qdrant Supernova benchmark can actually prove

A Qdrant Supernova benchmark can tell you whether a vector system reproduces a defined neighbor ranking under known conditions. That is useful. It is also narrower than many teams want it to be.

Exact-neighbor recall is not human relevance, and it is not answer correctness. With FineWeb-10B, the serious work starts before any database comparison. You have to define the workload, preserve its identity, and make the scoring rule auditable. Otherwise the benchmark becomes a polished way to compare mismatched experiments.

Here is the practical problem. A retrieval system can return the passages closest to a question embedding and still miss the passage that answers the question. The ranking can be faithful to the embedding space while the application gives the user a weak answer. That is not a small caveat. It is the line between a vector benchmark and a RAG evaluation.

Neighbor fidelity, human relevance and answer correctness

Treat these as different targets.

Neighbor recall measures agreement with an exact reference for a specified representation, metric, filter and cutoff. Human relevance asks whether the retrieved material helps with the information need. Answer correctness asks whether the final response is accurate and supported by the retrieved evidence.

Qdrant's release announcement reports 10.07B dense and 10.07B sparse vectors. The FineWeb-10B card describes exact top-1000 references and 100,000 dense queries, plus separate sparse and filtered query sets. Those numbers are publisher-reported. They are not Optijara measurements.

Why this is a method, not a database leaderboard

Optijara has not downloaded the full corpus or run a full or shard benchmark for this article. This is an experimental design, not a results report.

The announcement gives the scale. The dataset card defines the data. The recall guide explains the scoring semantics. This article connects those sources into bounded experiments and a separate production validation plan. It does not claim exhaustive originality. For the related model-selection question, see the embedding retrieval acceptance test.

Map Supernova modules to benchmark artifacts

The module table

The Supernova repository and linked module documentation describe this lifecycle. The key is to treat each checkpoint as evidence to retain, not as a command you ran once and forgot.

Pin documentation before copying commands

The documentation homepage and repository expose different command generations. Pick a commit, use that documentation, then inspect the installed CLI help. The root README installation list does not prove that nova-bf is installed. No executable command in this article has been installation-tested.

Backend support also needs care. The loading guide documents Qdrant-specific resume behavior. The recall guide documents Qdrant-specific tie reporting. Support for querying several databases does not mean every feature behaves the same way everywhere.

The Benchmark Fidelity Ladder: build the smallest valid experiment first

The Benchmark Fidelity Ladder is Optijara's proposed editorial method, not an established benchmark standard. Its rungs are a bounded shard, an exact shard reference, matched workloads, comparable operating conditions and separately judged production queries.

The opinionated version is simple: do not scale a bad experiment. A small benchmark with clean identity beats a large benchmark with hidden joins, unknown arithmetic, vague cache state and unclear scoring.

Bound the corpus and preserve its identity

Start with an immutable file list and a fixed query set. Do not rely on whatever a streaming reader happens to encounter first. Record preserved source IDs, selection seeds, dimensions, normalization, dtype, distance metric, requested cutoff, tokenizer, filter rules and tie policy.

Fingerprint corpus files, query vectors and configurations separately. Query-text checksums do not establish embedding equality, as the FineWeb-10B card warns. A query can have the same text and a different vector if the model revision, tokenizer, normalization or remote code changed.

The OpenBind reproducibility analysis covers the related gap between a benchmark label and real-world validity. Here, the manifest has one job: make the actual searched corpus recoverable.

Recompute the exact reference for that shard

Do not filter a published full-corpus top-1000 list to IDs present in your shard and call it exact shard ground truth. That shortcut loses neighbors that are valid inside the shard but never appeared in the truncated global list.

Recompute the reference over precisely the selected corpus and query vectors, using the chosen metric and filters. Keep the generation configuration beside the reference. A file named ground truth is not enough. You need to know how its scores were produced.

Match workloads before adding production questions

Use this implementation checklist before increasing scale. Each row should leave an inspectable artifact.

Design a test matrix that separates recall, relevance and operations

Dense, sparse and filtered FineWeb experiments

The FineWeb-10B card specifies unit-norm dense vectors, which support cosine or equivalent dot-product ranking. It also specifies unnormalized sparse weights evaluated with dot product. Preserve those conditions. Independent dense and sparse exact rankings are not ground truth for a hybrid fusion rule.

The brute-force guide defines text matching as lowercase word-token AND behavior, not phrase search. Match tokenization, Boolean logic, date boundaries, null handling and membership semantics. A field named keyword_phrase does not override the documented operation.

Optional matched-corpus multi-vector extension

PubMed-MV supplies dense, sparse and token-vector representations over the same abstracts. Use its documented MaxSim definition in a separately bounded experiment. Do not compare its headline results directly with FineWeb or read neighbor fidelity as clinical relevance.

Coyo-VE is another distinct workload. Its card specifies embeddings and captions, not image bytes. Neither companion dataset lets you skip corpus, query and representation identity.

Fair loading and query conditions

Fix hardware, backend versions, replication, quantization, index settings, filter indexes, concurrency and query mix. Define cold and warm cache protocols explicitly. An unspecified repeat run is not a representative cache policy.

If a local language model consumes the retrieved passages, keep its device and runtime evaluation separate from vector-engine timings. The MiniCPM5-2B deployment reality test covers that adjacent deployment question, not FineWeb search performance.

The loading guide distinguishes post-upload indexing from Elasticsearch's inline ingest-plus-index accounting. As a result, index_seconds alone is not a fair cross-backend clock. Measure the full path from setup to the same query-ready condition, while retaining backend-specific phases.

Use a measurement plan that keeps denominators and failures visible:

For a Claude-backed RAG experiment, freeze the model identifier, prompt, retrieved passage order and generation settings. Save passage IDs and source URLs with each answer. Judge whether citations support the claims and whether the requested exception was answered. Record retrieval and generation latency separately. Label this as an application experiment, not a nova-storm result.

What teams get wrong when measuring nova-storm recall

Wrong cutoff, short lists and unequal tie treatment

Finding a returned top-10 ID anywhere in a top-1000 reference is not recall@10. Recall@10 compares the returned ranking with the corresponding exact cutoff. The current recall guide truncates deeper ground truth to top_k and identifies the changed semantics with schema_version: 2. Reconcile definitions before combining legacy and current traces.

Report short-reference queries separately from full-depth queries and specify the denominator policy. Preserve no-hit cases as operational tests. A combined mean without group sizes hides which workload was actually evaluated.

Also separate exact-ID recall from tie-tolerant bounds. Comparing Qdrant's tie-adjusted result with another backend's exact-only result changes the scoring rule. It is not merely a database difference.

Precision and loading errors disguised as search differences

There is an unresolved provenance conflict: the FineWeb-10B card describes bfloat16 ground-truth arithmetic, while the current recall documentation says nova-bf has no bf16/fp16 compute mode. Request the generation-time revision and arithmetic manifest. Do not invent a reconciliation or diagnose a database bug from this discrepancy.

Distinguish duplicate IDs, duplicate text, duplicate embeddings and tied scores. The upstream FineWeb card describes crawl-level deduplication, not guaranteed global duplicate removal.

Before tuning an index, inspect skipped files, ID forms, query-vector identity, metric and filtering. The FineWeb-10B card notes different corpus and reference identifier forms. An unexplained failed join is a data-integrity blocker, not an ANN tuning opportunity.

Caveats, query rights and resource stop criteria

Review code, corpus, query and model licenses separately

Supernova's repository declares Apache-2.0. Upstream FineWeb specifies ODC-By and additional Common Crawl terms. The gte-multilingual-base model card declares Apache-2.0 and includes remote-code-dependent examples. None of these grants blanket permission for every input.

Microsoft's MS MARCO terms specify noncommercial research use. Review query rights before adopting the published workload. Appropriately permitted first-party queries create a separately labeled experiment, not an unchanged official benchmark. Companion datasets need their own upstream-rights review.

Pin model, tokenizer and remote code. Observing a current repository revision does not prove which revision generated published artifacts. Missing generation provenance limits reproducibility claims even when files are publicly downloadable.

Stop before an invalid sample becomes an expensive run

Set project-specific ceilings for spend, elapsed time, downloaded bytes, disk, host RAM and GPU memory before execution. Stop at a ceiling, persistent load failure, failed ID join, divergent filter behavior or unexplained reference mismatch. Save the stopping reason. Do not silently shrink the workload and keep the original label.

A warm-cache shard cannot establish full-corpus cost or tail latency. Later assessments must include implementation effort, transfer, index construction, provider and hardware variance, cache state or staleness, evaluation quality and ongoing operations. Minimize private query data and keep it out of public artifacts.

Publish a reproducibility record, not a winner

A compact machine-readable summary

This summary describes the proposed experiment. Null observations are deliberate: no benchmark or cost measurement was performed.

{
  "framework": "Benchmark Fidelity Ladder",
  "scope": "proposed_bounded_experiment",
  "fullCorpusRun": false,
  "benchmarkExecuted": false,
  "recomputeGroundTruthForShard": true,
  "neighborRecallIsHumanRelevance": false,
  "humanRelevanceIsAnswerCorrectness": false,
  "observedRecall": null,
  "observedP95Ms": null,
  "observedCost": null
}

Hand off the immutable manifest, versioned configuration, load reconciliation, readiness timestamps and request traces together. Include recall semantics, tie diagnostics, query groups, latency distributions, failures, resource accounting and separate human judgments.

What to test next

First, establish a trustworthy bounded reference. Then compare matched operating conditions. After that, ask whether permitted production queries retrieve useful evidence and whether downstream answers use that evidence correctly. A valid sample justifies the next experiment. It does not justify a full-corpus performance claim.