Benchmark Scores Need Protocols: How Evaluation Cards and Every Eval Ever Make AI Results Comparable

Why a benchmark score without a protocol is only a headline

A benchmark score looks tidy until you ask how it was produced. Two model results can sit next to each other on a leaderboard while using different attempt policies, feedback conditions, token budgets, scorers, model snapshots, or sample cohorts. At that point the comparison is not model against model. It is one experimental setup against another, compressed into a number that looks more final than it is.

That is why Evaluation Cards and Every Eval Ever are worth attention. They push AI benchmark reproducibility toward inspectable records instead of leaderboard cells. The September 22 UK AISI and EvalEval release is useful for that reason. It connects Evaluation Cards, Every Eval Ever, and public records around inference-scaling experiments. The headline should not be which model seems to rank higher. The better question is whether the score can be joined to protocol, budget, metric, sample coverage, and provenance.

There is a timing caveat too. The underlying paper appeared earlier, so the public bucket should not be framed as newly generated on the announcement day. September 22 is better understood as a reporting and interoperability milestone. It gives teams a cleaner path from score to record, but it does not remove the work of checking what was measured, how it was measured, and which parts of the record are absent or conditional.

My take: benchmark records should be treated as infrastructure, not commentary fodder. A useful record layer can support evaluation pipelines, model routing decisions, and procurement reviews. A naked score does the opposite. It hides the protocol differences that usually decide whether the result applies to your workload.

What the release connects: Evaluation Cards, Every Eval Ever, and AISI records

Evaluation Cards are structured summaries for benchmark results. They make results easier to inspect by surfacing metadata and protocol context where that context exists. That does not mean every comparison is fair. A card can describe a result clearly while the underlying runs still differ in sample coverage, feedback access, attempt policy, or grading details.

Every Eval Ever is the interchange layer around those records. The project provides schemas and tooling for aggregate evaluation records and optional instance-level records. Version discipline matters here. The public project landing page has shown older examples, while the GitHub README now references v0.3.0 and sample files using the *_samples.jsonl convention. Teams should pin the repository revision, schema version, and source snapshot they use. Mixing website examples with newer validators is a quiet way to create avoidable confusion.

Aggregate records and sample records answer different questions. Aggregate rows tell you what result was reported, which setup was visible, and which provenance fields came along with it. Instance rows can tell you which samples were attempted, what evidence is available, and whether an aggregate can be checked against a more granular record. But sample-level records may be optional, incomplete, restricted, or absent for legitimate reasons. Public structure is not the same thing as a universal transcript archive.

The AISI bucket is another piece of the evidence chain. It provides public experiment data around the inference-scaling work, including logs and related artifacts. Optijara has not reproduced those benchmark runs, validated the full datastore, or executed inference against those records. The responsible move is narrower: inspect the public record, keep caveats visible, and avoid converting publisher terms such as verified into claims of independent replication.

For teams already thinking about token accounting and denominator drift, this is close to the same operational discipline covered in our guide to Hugging Face Tokenizers v1 and same-ID migration. For latency evidence workflows, the same record discipline also connects to our analysis of OpenVINO GenAI encode and decode timing, where measurement boundaries affect how downstream systems interpret results.

The Score-to-Protocol Join: a practical framework for comparing benchmark results

Optijara's Score-to-Protocol Join is a simple rule: do not compare benchmark scores until the score is joined to the protocol fields that give it meaning. The join does not need to make every record complete. It needs to make uncertainty visible.

This framework is especially useful for inference-compute curves. A curve may show earliest observed success or cumulative tasks solved as budget changes. That is not the same as a single-attempt pass@1 cell. It also does not translate cleanly into dollars or latency, because tokenizers, context windows, provider engines, pricing, parallelism, and attempts can differ.

Use model names such as Claude Opus or GPT versions as examples of comparison mechanics, not as the main story. The practical question is whether results on benchmarks such as HLE, FrontierMath, or HealthBench were measured with aligned task subsets, metrics, token budgets, feedback conditions, and sample coverage. If a join key is absent, leave it unknown. Do not infer equivalence because two rows live in the same dataset.

The same denominator discipline applies to runtime work. Our article on OpenVINO GenAI encode and decode timing makes a similar point for latency. A single throughput number is weak evidence unless the timing boundary and measurement setup are visible.

How teams can ingest aggregate and instance records without overclaiming reproducibility

A sensible ingestion workflow starts before analysis. Pin the repository revision, schema version, source URL or bucket snapshot, retrieval time, and any paper or protocol version being interpreted. Then validate structure, join aggregate and sample records where documented, filter comparable cohorts, and compare curves only after unknowns have been retained.

Schema validation is useful. It can catch missing required fields, malformed values, and incompatible record shapes. It cannot certify that an experiment was independently reproduced. It cannot prove that samples are complete, oracle feedback was available in the same way, transcripts are public, licenses permit every downstream use, or scoring truth is correct.

For an AISI-style audit, the aggregate plus sample join should be treated as evidence engineering. Use documented IDs and source-specific keys. Check counts before and after joins. Inspect manifests where available. In the public bucket context, research notes indicate that some joins use log_file, sample_id, and original_epoch rather than a reassigned epoch. Missing parents, changed headers, or absent referenced logs should become audit findings. Quiet repair is where trust starts to leak.

A compact machine-readable summary can keep this discipline portable:

{
  "framework": "Score-to-Protocol Join",
  "compare_only_after": [
    "model_version_pinned",
    "benchmark_subset_known",
    "metric_and_scorer_known",
    "budget_and_attempt_policy_known",
    "feedback_and_compaction_marked",
    "aggregate_sample_counts_checked"
  ],
  "do_not_claim": [
    "independent_replication",
    "complete_samples",
    "production_roi",
    "universal_model_rank"
  ]
}

Reading inference-compute curves like an operator, not a leaderboard watcher

Inference-compute curves are useful when they show how performance changes as budget or protocol changes. They help operators ask whether more inference budget continues to reveal capability, whether a task family saturates, and whether a comparison depends on feedback or repeated attempts.

They become weak evidence when treated as a universal ranking. Larger budgets and protocol interactions do not establish guaranteed task performance, production ROI, or a permanent model order. A curve built with oracle correctness feedback is not the same as a production workflow where users do not know whether the previous answer was correct. A curve under one token cap is not necessarily comparable to another curve if tokenization, context handling, or attempt policy differs.

A practical mini-audit looks like this:

This is a proposed audit workflow, not a claim that Optijara reproduced the AISI results. The point is to make comparison safer by preserving uncertainty instead of smoothing it away.

What teams get wrong when they operationalize benchmark records

The first mistake is treating verified metadata as independent replication. If a publisher says a record is verified according to its own process, that should not be rewritten as Optijara-verified or independently reproduced. Provenance is not the same as rerunning the experiment.

The second mistake is comparing curves with hidden protocol differences. A row with oracle feedback, context compaction, multiple attempts, or a different sample subset should not be compared as if it were the same condition. If the bucket README, schema, or paper leaves a condition ambiguous, mark it unknown until the protocol resolves it.

The third mistake is ignoring sample coverage, licensing, and privacy limits. A public repository license does not automatically override the terms for every dataset, transcript, model output, or downstream use case. Teams should separate technical availability from legal and privacy suitability.

The fourth mistake is turning experimental oracle feedback into a production assumption. Oracle correctness feedback is privileged experimental information. It can help researchers study scaling behavior, but it should not be treated as a normal application feedback loop.

A decision matrix for adopting Evaluation Cards and Every Eval Ever in AI infrastructure

Evaluation Cards and Every Eval Ever can be valuable infrastructure when used for the right job. They are strongest as a record layer, not as an automatic decision engine.

The implementation checklist is straightforward:

For teams building model routing, evaluation pipelines, or benchmark evidence layers, the goal is not to make every public benchmark decisive. The goal is to design a comparison protocol that preserves what is known, exposes what is missing, and prevents a score from becoming an unsupported procurement shortcut. Optijara can help design that record layer when teams need evaluation infrastructure that can stand up to review.

Compare protocols before comparing models

The September 22 UK AISI and EvalEval release is valuable because it makes evaluation records easier to inspect. It does not turn benchmark scores into universal truth. A score needs a protocol. Aggregate records need sample context. Schema validation needs caveats. Inference-compute curves need aligned cohorts. Missing fields should stay visible instead of being filled by assumption.

That is the operator lesson: compare protocols before comparing models. If your team is using public benchmarks to guide model routing, infrastructure planning, or evaluation design, start by building the Score-to-Protocol Join. The result is slower than reading a leaderboard, and that is the point. Decisions that need to survive scrutiny deserve evidence that shows its working.