Proof Artifact Acceptance Test: How to Evaluate Anthropic’s Fermat’s Last Theorem Formalization

A green Lean check is serious evidence for Anthropic's Fermat's Last Theorem formalization. It says a formal statement, in a defined Lean environment, passed the machine checker. That is not a small thing. For AI-assisted mathematics, it moves the discussion away from polished demos and toward an artifact that can be inspected.

Still, a checked file is not the same as an accepted artifact. The theorem statement might differ from the informal claim. Dependencies can carry assumptions that most readers never see. A build can work on one machine and fail for everyone else. A proof can check today, then become hard to maintain when Mathlib changes.

The practical view: green checks deserve respect, not blind trust.

Anthropic's research release on the formalization of Fermat's Last Theorem, plus its public repository, gives reviewers a useful test case. The question is not only, did it check? The better question is whether another capable reviewer can inspect the statement, reproduce the build, explain the trust boundary, maintain the artifact, and recover a known-good version later.

That is the job of the Proof Artifact Acceptance Test, or PAAT. PAAT does not judge whether the proof is beautiful. It is an acceptance framework for deciding whether a large formal proof artifact is useful to researchers and technical teams, instead of merely impressive. This framing fits the reproducibility discipline discussed in AI benchmark reproducibility: the evidence package matters as much as the headline result. It also extends the source-discipline argument behind open scientific evidence for AI evaluation and the operational point from WeatherNext 3 and AI infrastructure evaluation: serious AI infrastructure needs inspectable artifacts, not just strong-looking outputs.

Why a Green Lean Check Is Necessary but Not the Whole Acceptance Case

The useful tension: verified syntax versus accepted artifact

Lean checking gives reviewers a hard boundary. A file checks in a given environment, or it does not. That boundary has real value because it leaves less room for vague claims. A checked theorem is not a paragraph that sounds persuasive. It is a formal object connected to imports, definitions, tactics, theorem statements, and a trusted kernel.

Artifact acceptance asks a wider set of questions. What exactly was checked? Which dependencies were trusted? Which library version was used? Does the formal theorem correspond to the classical statement of Fermat's Last Theorem? Can the result be reproduced from a clean checkout? Are there review notes for humans who need to understand the proof path?

Those questions are not nitpicking. They are the difference between a proof artifact that can support research work and a proof artifact that can only support an announcement.

What Anthropic is claiming and what the public artifacts can support

Anthropic's release can support Anthropic-reported context about the work and its purpose. The public GitHub repository can support a different class of claims if inspected at a fixed commit: repository availability, visible file structure, build instructions in the README, final check files such as FinalCheck.lean, and dependency data through files such as lake-manifest.json. Lean and Mathlib documentation explain the tooling context. Imperial College's FLT materials and repository provide background on the community formalization effort. Prove2Me provides context for AI systems aimed at theorem proving.

The boundaries between those sources matter. A research announcement is not an independent audit. A public repository is not proof of long-term maintainership. A README is not a guarantee that every future checkout will build. PAAT keeps those claims separate so the acceptance decision does not get inflated by excitement around the result.

The Source Stack: What Must Be Inspected Before Accepting the Fermat Artifact

A reviewer should start from durable public sources, not search snippets, social posts, or secondhand summaries. For this artifact, the source stack includes Anthropic's research release, the anthropics/fermats-last-theorem repository, the repository README, FinalCheck.lean, lake-manifest.json, Lean's FLT use-case page, Imperial College's FLT project site, Imperial College's FLT repository, Prove2Me, and the Mathlib overview.

The PAAT Six-Gate Framework for Large Formal Proof Artifacts

PAAT is Optijara's six-gate acceptance framework for formal proof artifacts. It is designed for AI-generated or AI-assisted proofs where the result may be impressive, but the acceptance case has to stay evidence-first.

Gate 1: Provenance and scope

Start with artifact identity. Record the canonical repository, commit hash, release page, author or organization, license if present, theorem files, build files, and the exact version under review. Provenance prevents a common failure: judging a moving target, then discovering later that the artifact changed underneath the review.

Gate 2: Statement equivalence

Statement equivalence asks whether the formal theorem matches the intended Fermat's Last Theorem claim. A theorem can check while encoding a shifted domain, a hidden condition, a definition buried inside a dependency, or a narrower result than readers assume. The reviewer has to map the mathematical sentence to the formal Lean statement and its definitions.

Gate 3: Axiom and dependency surface

A formal proof inherits trust from its axioms, imports, libraries, compiler, and checker. PAAT treats that surface as a trust boundary, not background plumbing. Reviewers should inspect axiom reports where available, imported modules, Mathlib dependency versions, Lake manifest data, and any custom trusted code.

Gate 4: Deterministic build and checkability

A formal artifact that only checks on one contributor's machine is not ready for broad acceptance. Gate 4 asks whether a clean environment can clone the repository, install or select the documented Lean toolchain, resolve documented dependencies, build the project, and run the final check.

Gate 5: Proof graph integrity and residue

Large AI-assisted artifacts can contain generated fragments, retries, abandoned lemmas, brittle scripts, or files that no longer connect to the final theorem. Gate 5 asks whether the proof graph is coherent. Do the key declarations connect to the final theorem? Are there unresolved declarations or trusted shortcuts? Are generated intermediate files documented?

Gate 6: Reproduction, exposition, maintenance, and archive

The final gate connects the artifact to human and operational continuity. Independent reproduction shows that another capable reviewer can run the artifact outside the producer's environment. Exposition explains the proof path so humans can understand what the formal files are doing. Maintenance names who will update dependencies and respond to breakage. Archival planning preserves a known-good snapshot.

A Reproduction Test Matrix for Researchers and Technical Reviewers

The minimum useful test is a clean clone from the canonical repository, followed by the documented dependency and build path. Capture the commit hash, toolchain version, manifest state, final target, and output. A stronger review adds a statement audit, axiom audit, dependency diff, and a short human note explaining what was checked.

Accept, Pilot, or Wait: A Decision Matrix for Formal Proof Adoption

Acceptance should be conservative. Accept when the repository is public, the evaluated commit is captured, the theorem statement has been audited, axioms and dependencies are documented, the build is deterministic, an independent reviewer has reproduced the check, exposition is readable, and an archive plan exists. Pilot when the final check succeeds, but reviewer reproduction, exposition, or maintenance evidence is still maturing. Wait when the theorem statement is not mapped, dependencies are not documented, build steps are incomplete, axioms are undocumented, the repository cannot be archived, or independent reproduction is not possible.

What Teams Get Wrong When Evaluating AI-Generated Formal Proofs

The first mistake is treating final checker output as the whole review. The checker result is essential, but it is only tied to the statement and environment being checked.

The second mistake is ignoring statement drift. A formal theorem can be technically valid while readers infer a broader or different informal claim.

The third mistake is hiding dependency and axiom assumptions. Dependencies are not embarrassing. Hidden dependencies are the problem.

The fourth mistake is confusing exposition with reproduction. A good walkthrough helps humans understand the proof, but it does not replace a clean build and an independent reviewer transcript.

The fifth mistake is forgetting maintainership. Lean, Mathlib, repository hosting, and project conventions can change. If nobody owns maintenance or archival snapshots, today's checked artifact can become tomorrow's broken reference.

Implementation Checklist and Measurement Plan for PAAT

Use this checklist before making any acceptance claim:

• Capture the canonical release URL, repository URL, and commit hash.
• Record the theorem file and final check target.
• Save README build instructions and dependency manifests.
• Inspect theorem statement equivalence with a qualified reviewer.
• Document axioms, imports, Mathlib dependencies, and trusted boundaries.
• Run the build and final check in a clean environment.
• Capture independent reproduction notes.
• Link the written exposition or walkthrough.
• Identify maintenance owner, archive snapshot, and rollback note.

{
  "framework": "PAAT",
  "artifact": "Anthropic Fermat formalization",
  "gates": [
    "provenance_and_scope",
    "statement_equivalence",
    "axiom_dependency_surface",
    "deterministic_build",
    "proof_graph_integrity",
    "reproduction_exposition_maintenance_archive"
  ],
  "decision": ["accept", "pilot", "wait"],
  "residual_risks": ["toolchain_drift", "statement_drift", "archive_gap", "review_capacity"]
}

Caveats: What PAAT Cannot Prove

PAAT evaluates artifact acceptability. It does not prove that the proof is the shortest, clearest, most elegant, or best teaching path. A machine-checked proof can still need excellent exposition before most humans can learn from it.

Reproducible today does not mean reproducible forever. Toolchain drift is real. Mathlib evolves. Repository hosting changes. Dependencies may disappear or move. Cache behavior and local environments can differ. That is why archival snapshots and rollback notes belong inside the acceptance case.

The broader lesson is not limited to one theorem. The most useful AI proof artifacts will be judged by what they claim and by how well their evidence survives inspection. For a team tracking AI research, the practical work is to turn fast-moving claims into evidence tables, acceptance tests, reproduction notes, and maintenance decisions before those claims shape roadmaps or public positioning.