OpenBind and EERAT: A Reproducibility Test for Structure Affinity AI Benchmarks

Why OpenBind Needs an Evidence Route

A structure affinity benchmark is not useful because it is open. It becomes useful when a second team can rebuild the evidence path, reproduce the split, rerun the baseline, and then show that better ranking changes which compounds get tested.

That is the test OpenBind deserves. Not a leaderboard recap. Not a victory lap for structure-based AI. A lab notebook standard.

OpenBind's first public structure affinity release is interesting because the public artifacts expose more than a single model score. The EV-A71 2A benchmark repository describes a release that combines crystallographic fragment screening, follow-on compound optimisation, and affinity measurements for enteroviral 2A protease. It also includes reference benchmark material for docking, co-folding, virtual screening, and affinity prediction. The OpenBind first release blog reports 925 crystallographic binding events from 699 compounds, with affinity measurements for 601 compounds. The OpenBind-0 model release repository points to a Zenodo benchmarking set of 462 protein ligand systems.

Those are useful ingredients. They are not proof that a ranking method should steer experimental screening. The strict view is simple: if a benchmark cannot survive identifier rebuilding, leakage checks, and an external holdout, it is a research artifact, not an operational control.

Optijara's EERAT framing, Experimental Evidence Route Acceptance Test, treats OpenBind as a case study for disciplined scientific AI evaluation. The standard is deliberately strict. Every performance claim stays author reported until an outside team reproduces it under pinned conditions and records where its run differs.

The Source Stack to Inspect

Do the source audit before arguing about model quality. A team should use canonical public artifacts, not search snippets, guessed URLs, redirects, or blocked pages. For this article, the usable source set is the OpenBind GitHub organization, the EV-A71 2A benchmark repository, the OpenBind-0 model release information repository, the docking workflow repository, the OpenBind website, the OpenBind first release blog, and the Zenodo record.

Licenses need the same separation. A dataset can be open while code, model weights, docking assets, external binaries, assay records, and protocol documents carry different conditions. That is not a paperwork detail. It decides whether the artifact can be trained on, redistributed, embedded in a product, or used only for internal research.

EERAT in Seven Gates

EERAT asks one practical question: is the evidence route strong enough to justify more evaluation effort? It does not certify a model. It does not say a compound should move forward. It tells the team whether the benchmark is ready to influence the next screening decision.

Gate 1, Specimen and Assay Provenance

Start with the target identity, construct, specimen preparation, assay method, affinity format, compound history, and protocol records. A pass means another reviewer can point to the artifact behind each measurement and explain what was tested. If those links are missing, the benchmark may still be interesting, but it is not ready for decision use.

Gate 2, Structure and Affinity Quality

Inspect structural coverage, binding events, pose confidence, affinity consistency, censored values, missing records, and failed compounds. The OpenBind blog's reported 925 crystallographic binding events from 699 compounds, with affinity measurements for 601 compounds, gives reviewers concrete material to inspect. The repository structure matters because it exposes data and analysis paths rather than only summary text.

Gate 3, Identifier Normalization

Normalize compound identifiers, protein constructs, chain IDs, ligand names, assay IDs, and versioned records. The Zenodo model release record shows why this is not clerical work. It describes 462 queries with 483 protein and 890 ligand chain entries, 599 individual protein chains, and a deduplicated chain map to 302 MSA representatives. If identifiers are loose, duplicates and aliases can quietly distort the benchmark.

Gate 4, Leakage and Split Integrity

Audit scaffold similarity, pocket similarity, target family proximity, near duplicate ligands, reused conformations, preprocessing steps, and compounds that influenced model selection. File separation is not enough. A split can look clean on disk while still letting the evaluation set bleed into training choices.

Gate 5, Baseline Parity and Environment Pinning

Save repository commits, environment files, software versions, hardware assumptions, seeds, logs, and external binaries. The docking repository calls for GNINA, Smina, and DiffDock installations and is built around Slurm execution. That means baseline comparison is partly a systems problem. Inputs, receptor preparation, compute budget, and metric reporting all need parity.

Gate 6, External Holdout and Prospective Screening

Run a small external holdout before treating benchmark ranking as a screening guide. Then lock a prospective screen before results are known: model ranking method, docking rules, assay compatibility, confirmation protocol, and interpretation threshold. This is where a benchmark starts earning operational relevance.

Gate 7, Canary, Stop Use, and Wet Lab Confirmation

Define canary compounds, drift checks, wet lab confirmation rules, and stop use triggers. Pause the benchmark for the decision if canary behavior changes, split leakage appears, license terms conflict with intended use, or prospective confirmation fails. A benchmark without a stop rule becomes easy to rationalize after it disappoints.

Reproducing the Benchmark Without Fooling Yourself

A clean reproduction record beats a slightly higher score. The work is procedural and, frankly, boring. That is why it works.

Leakage usually enters quietly. Similar scaffolds can cross the split. Related pockets can make target family generalization look better than it is. A reused conformation can give a docking or co-folding method a hint it would not have during a real screen. Preprocessing can learn from the whole dataset before the split is applied. Evaluation compounds can also be selected after model exploration, which turns the benchmark into a record of tuning choices.

Model, Docking, and Experiment Decisions

Here is the consultant's take: a benchmark should not be allowed to influence wet lab spending until the team can measure whether ranking quality changes confirmed hits, rejected compounds, or cost per confirmed hit in its own workflow. If the team cannot measure that, the benchmark is still useful research evidence. It is not a screening rule.

Common Mistakes

Mistake 1, Treating Openness as Reproducibility

Open repositories help inspection. Reproducibility needs commit hashes, pinned environments, data versions, split scripts, baseline logs, and clear deviations. Without that trail, reviewers are trusting a moving target.

Mistake 2, Treating Split Performance as Prospective Value

A benchmark split can teach a team a lot and still fail in a new lab setting. External holdouts and locked prospective screens are the bridge between a reported score and a screening decision.

Mistake 3, Hiding Failed Compounds and Missingness

Failed compounds, censored affinity values, inconsistent assay formats, and missing metadata often carry the most operational signal. A method that looks strong on clean records may be weak where screening work is messiest.

Mistake 4, Collapsing Licenses Into One Answer

Dataset terms, code licenses, model weight terms, docking software terms, workflow assets, and experimental records may not match. Treating them as one permission answer creates avoidable risk.

Mistake 5, Comparing Baselines Without Parity

Baseline parity means comparable inputs, receptor choices, preparation rules, compute budgets, seeds, and metric reporting. Without parity, a leaderboard may reward workflow choices rather than model capability.

Caveats and Measurement Plan

EERAT is conservative by design. Scientific caveats include assay variability, target specificity, structure quality limits, and weak transfer to other targets. Operational caveats include compute reproducibility, external software drift, stale caches, and the work required to maintain data lineage. Legal caveats sit across dataset terms, code terms, model weights, docking tools, protocol documents, and experimental records. Evaluation caveats include confidence intervals, failed compound handling, and whether the external holdout is truly external.

{
  "framework": "EERAT",
  "gates": ["provenance", "quality", "identifier_normalization", "leakage", "baseline_parity", "external_holdout", "canary_stop_use"],
  "requiredArtifacts": ["canonical_repositories", "dataset_record", "license_terms", "split_scripts", "baseline_logs", "holdout_report"],
  "acceptCriteria": "reproducible, leakage audited, license clear, externally tested, and tied to wet lab confirmation",
  "rejectCriteria": "unclear provenance, unresolved leakage, incompatible terms, non comparable baselines, weak holdout, or missing stop use criteria",
  "recommendedNextStep": "run a small, pinned reproduction before using rankings for screening"
}

OpenBind is worth attention because it gives reviewers real artifacts to inspect. EERAT keeps the next step honest. Trace the evidence. Rebuild the identifiers. Recreate the split. Rerun the baseline. Test an external holdout. Then ask whether better ranking reduces unnecessary experiments in the team's own screening context.