AI Security Review Acceptance Test: What Aave V3 and V4 Teach About Smart Contract Scanner Triage

Start with the unglamorous constraint. An AI security scanner can produce a confident list of alerts in minutes. A security leader still has to decide which alerts are real, which ones are duplicates, which ones can be reproduced, and which ones deserve reviewer time this week.

That is why Aave's reported AI-assisted review of Aave V3 and V4 is more useful as a workflow test than as a victory headline. Aave Labs reported 71 total findings across three anonymized AI security tools. Manual review accepted 20. Aave's article says the accepted findings were Low or Informational, while several alerts that arrived as Critical or High were later treated as false positives after human validation.

Those numbers are interesting. They are also easy to misuse. They are project-reported results from a defined scope and method. They do not prove that a protocol, repository, or release is secure. They show something narrower and more operational: scanner output needs an acceptance test before it is allowed to shape audit handoff, CI policy, or production release decisions.

The same idea applies outside smart contracts. Application security reviews, API checks, infrastructure policy scans, and AI-enabled code review all run into the same problem. A tool that sounds certain can still be wrong about reachability, role assumptions, impact, or duplicate root cause. Optijara has used the same acceptance-gate pattern for local VLM qualification, X algorithm reproducibility, and latency-sensitive AI infrastructure. Security scanners should face the same discipline.

My view is blunt: a noisy scanner is not a security control. It is a source of leads. It becomes a control only after the team can reproduce its inputs, normalize its findings, calibrate severity, and record human decisions.

Why Aave's AI-Assisted Review Is a Reproducibility Test, Not a Victory Lap

Aave reported that three AI security tools were run against Aave V3 and Aave V4 production repositories. The scans covered the full src trees at pinned commits. Where the tools supported it, they received context similar to what human auditors would receive, including threat-model and audit context.

Aave also described mutation testing on four V4 contracts, Hub, Spoke, TreasurySpoke, and AaveOracle. The test used 304 deliberate mutations. Existing tests killed 271, while 33 were inconclusive because the test suites timed out.

The headline was 71 surfaced findings and 20 accepted after manual review. The better lesson is in the filtering. Aave says findings were manually classified as valid, false positive, duplicate, or by design. That is where the review operation lives.

A scanner that emits 71 alerts is not automatically stronger than one that emits fewer. Volume can mean coverage. It can also mean repeated root causes, weak context, stale assumptions, or generated explanations around code paths that do not matter. The question for a security manager is sharper: after normalization, deduplication, severity review, and human adjudication, what remains?

The absence of confirmed Critical or High findings in this reported scope should not be read as proof of security. It means those findings were not confirmed under the reported code state, tool behavior, review method, and human classification process. That is still useful evidence. It is not a substitute for threat modeling, independent audits, formal verification, contests, monitoring, incident response planning, or release controls.

The Evidence Baseline: Source, Commit, Tool, and Method Pinning

The first AISRAT gate is artifact fidelity. Before anyone debates severity, the team needs an evidence bundle that fixes the exact review target. At minimum, record the source repository URL, commit hash, dependency lockfiles, compiler versions, build container, scanner configuration, timeouts, permissions, generated reports, excluded paths, and reviewer notes.

Aave's article says the scans ran against pinned commits of production repositories and covered source trees. Public repositories for Aave V3 Origin and Aave V4 show relevant code, audits, reports, tests, scripts, and documentation directories. That gives readers useful context, but an internal acceptance test needs more than context.

The team should be able to rebuild the target from a clean environment and reproduce the scanner input. If the scanned artifact cannot be rebuilt, the severity review is already on weak ground. Repository scope is not the same thing as reviewed code scope. A repository may contain src, tests, scripts, docs, generated files, deprecated modules, adapters, and periphery packages. Each can affect alert volume and reviewer burden.

AI-assisted scanners can also change output when a model, prompt, ruleset, retrieval source, detector version, or timeout changes. Vendors will not always expose every internal detail. Record what you can see: product version, report timestamp, configuration file, prompt template if used, documentation snapshot, command line, API parameters, and environment variables that affect output.

The standard is not academic reproducibility. The standard is operational reproducibility good enough to decide whether the scanner belongs in a route that affects reviewer queues, CI gates, audit packages, or release decisions.

The AISRAT Framework

AISRAT stands for Optijara AI Security Review Acceptance Test. It is a six-layer decision framework for deciding where an AI scanner belongs in a smart-contract or application-security review route.

Artifact fidelity checks that the scanner reviewed the code the team cares about. Required fields include repository, commit, build command, compiler, dependency locks, container image, source path, generated artifacts, excluded files, tool version, model or ruleset version where available, and permissions. For smart contracts, add compiler version, chain assumptions, deployment configuration, mock dependencies, and tests needed to reproduce behavior.

Issue normalization makes scanner output comparable before reviewers spend time on it. Normalize each alert into file, function, code path, vulnerability class, preconditions, exploitability claim, evidence, confidence, duplicate group, proposed severity, remediation status, and reviewer notes. Long narrative does not deserve extra weight just because it looks authoritative.

Severity is a claim. It is not evidence. AISRAT requires an exploitability or impact argument that fits the system's trust model. In Aave's article, human validation downgraded or rejected alerts that arrived with Critical or High labels. The tool proposes severity. The reviewer calibrates it against reachable state, role assumptions, impact, code path, and failure mode.

Reviewer adjudication needs structure. Where practical, blind reviewers to tool identity during the first pass. Record accepted, rejected, duplicate, by-design, unsupported, and hallucinated-code-path decisions. Include false-positive sampling and false-negative sampling against known vulnerabilities, historical audit findings, mutation tests, or seeded bug suites.

Actionability economics decides whether adoption is worth the cost. Track time-to-first-triage, time-to-adjudication, reviewer minutes per accepted finding, duplicate burden, re-opened findings, and cost per accepted actionable finding. Then define what happens after acceptance: CI thresholds, human override, audit handoff packages, disclosure workflow, access control, telemetry, canaries, rollback criteria, and regression tests.

{
  "framework": "AISRAT",
  "minimumEvidence": ["repo", "commit", "build", "toolConfig", "normalizedFinding", "reviewerDecision"],
  "routeOptions": ["research", "triageAssistant", "ciSignal", "auditInput"],
  "acceptanceQuestion": "Does the scanner produce reproducible, evidence-backed, non-duplicate findings at an acceptable reviewer cost for this route?"
}

Route Decision Matrix

A scanner with useful ideas but weak reproducibility may still belong in the research route. A CI gate needs a higher bar because it can slow delivery or create false confidence. Formal audit input needs the cleanest evidence package because auditors should be able to inspect accepted and rejected decisions, not just the final defect list.

AISRAT Implementation Checklist

For smart contracts, add contract compilation, deployment assumptions, governance roles, oracle dependencies, privileged functions, cross-chain messaging assumptions, and invariant or fuzz tests. For application security, add service versions, API schemas, identity flows, data classification, environment parity, and dependency graph state.

Common Mistakes

Alert count is the easiest number to market and the weakest number to operate. It rewards verbose tools and punishes tools that suppress low-confidence output. The better measure is accepted actionable findings after normalization, deduplication, and reviewer adjudication.

A Critical label without an executable or reviewable path is not a Critical vulnerability. AISRAT asks reviewers to separate vulnerability class from exploitability evidence, reachable state, role assumptions, and impact. This matters in systems with governance roles, timelocks, oracle assumptions, and deliberate revert behavior.

Duplicates burn reviewer time. Hallucinated files, unreachable paths, unsupported framework features, stale dependencies, and non-compiling targets can make a scanner look productive while adding little security value. Track these as their own categories. Rejected hallucinations should become regression examples so the same tool does not return them as fresh issues.

False positives are easy to see because reviewers stare at them. False negatives require deliberate testing. Use known vulnerabilities, historical audit issues, mutation tests, seeded bug suites, and conventional static-analysis baselines. A scanner with fewer false positives may still be a poor CI signal if it misses known issues.

Measurement Plan

Compare the AI-assisted route with a baseline using the same pinned artifact and threat model. The baseline might be human-only review, conventional static analysis, or the team's existing scanner suite. Universal precision targets are usually lazy. Language, framework, code maturity, context quality, and threat model all change the acceptable noise level.

Start with one scoped pilot: one repository, one commit, one threat model, one route. Decide whether the scanner deserves to be a research aid, triage assistant, CI signal, or audit input. Expand only after the evidence bundle is good enough for someone outside the immediate team to inspect.

Aave's review is useful because it shows the gap between scanner severity and human-confirmed evidence. AISRAT turns that gap into a management system. It protects reviewers from noise, gives vendors concrete feedback, and helps teams adopt AI-assisted security review without pretending that the tool has become the auditor. If your team is designing an AI-assisted review route, Optijara can help define the acceptance gates, measurement plan, and governance controls before the scanner starts influencing production decisions.