LLMs Scale Formal Verification: Proving Memory Safety in RISC-V and Backend APIs

Seed story: "LLMs are making formal software verification feasible at a much larger scale" (Anthropic) · search original Written from facts verified across 3 report(s) — original explainer, not a copy or translation. Sources at the end.

For decades, formal verification has been a luxury reserved for critical systems like the seL4 kernel, which demanded 20 person-years of effort to prove correct. Recent demonstrations by Anthropic and Theorem show that LLMs can now generate formal proofs for memory safety in RISC-V assembly code across over 100 real-world open-source libraries, uncovering bugs that trillions of test runs missed. This shift suggests that integrating AI into verification pipelines could finally make rigorous code correctness scalable enough to keep pace with the rapid development of large-scale systems.

From Manual Proofs to Automated Discovery

Recent breakthroughs have shifted formal verification from a niche academic exercise to a scalable engineering practice. By leveraging large language models, researchers are now generating proofs for security properties across vast codebases. This approach addresses a critical gap where traditional testing fails to guarantee correctness, particularly in complex backend and low-level systems.

Key recent developments include:

Anthropic and Theorem demonstrated that Claude can produce formal proofs for security properties in over 100 real-world open-source libraries.
A University of Manchester study introduced SpecVerify, combining Claude 3.5 Sonnet with the ESBMC verifier to automate requirements verification.
Tsinghua University developed a framework translating Scala backend code into formal Lean representations for API and database verification.

For developers, this signals a move toward automated assurance. Instead of manually constructing proofs, teams can now rely on LLMs to validate critical logic, reducing the barrier to entry for high-assurance software.

Proving Memory Safety at the Assembly Level

The Anthropic and Theorem collaboration demonstrated that Claude can generate formal proofs for security properties directly at the assembly level. This effort targeted RISC-V assembly code, specifically proving that programs would not trigger memory-safety errors when processing inputs under 64 MB. By operating at this low level, the system bypasses the abstraction gaps that often limit traditional static analysis tools.

Key technical outcomes included:

Verification of over 100 real-world open-source software libraries.
Detection of memory-safety bugs that had previously eluded detection despite trillions of test runs.
Automated generation of proofs for complex assembly logic.

For developers, this shifts the burden of low-level safety from exhaustive manual auditing to automated discovery. It suggests that verifying memory safety in critical infrastructure no longer requires the immense human effort seen in projects like the seL4 kernel, which took 20 person-years to verify.

Bridging High-Level Code and Formal Logic

While low-level proofs address memory safety, new frameworks are tackling the gap between high-level business logic and formal specifications. The University of Manchester introduced SpecVerify, which pairs Claude 3.5 Sonnet with the ESBMC verifier to automate requirements checking. In evaluations across nine cyber-physical systems from Lockheed Martin, this approach achieved a 46.5% verification accuracy, demonstrating a viable path for complex, safety-critical domains.

Simultaneously, Tsinghua University developed a method to translate Scala backend code into formal Lean representations. This pipeline allows LLM-based provers to verify API and database operations, formally validating over 50% of test requirements in realistic backend scenarios.

For developers, these tools shift verification from a manual, expert-only task to an automated part of the CI/CD pipeline. Key implications include:

Automated Spec Translation: Converting natural language or high-level code into formal logic.
Cost Efficiency: Tsinghua’s method averages just $2.19 per API verification.
Broader Coverage: Catching logical errors in business logic, not just memory bugs.

The Cost-Benefit Shift in Verification

The Economic Reality of Proof

Formal verification has long been a luxury reserved for critical systems. The seL4 microkernel, for instance, demanded 20 person-years to verify, a cost that is double the time required to implement the code itself. This steep price tag kept rigorous proof out of reach for most development teams, leaving vast areas of software unverified.

LLM-driven methods are collapsing this cost curve dramatically.

Traditional Cost: 20 person-years for seL4 verification.
LLM Cost: ~$2.19 per API for backend verification.
Context: Linux kernel has 1,000+ disclosed vulnerabilities.

For developers, this shift transforms verification from a prohibitive project into a routine part of the CI/CD pipeline. Instead of hiring specialized mathematicians, teams can now integrate automated proof generation directly into their workflow. This economic feasibility means that proving memory safety or API correctness becomes a standard quality gate, allowing organizations to ship code with significantly higher confidence without breaking the budget.

Uncovering Bugs That Testing Missed

The sheer volume of traditional testing often creates a false sense of security. Recent work by Anthropic and Theorem demonstrated that Claude could identify memory-safety vulnerabilities in over 100 real-world open-source libraries. These specific bugs had previously eluded detection despite trillions of prior test runs, highlighting a critical gap in conventional QA processes.

This capability is particularly significant given the current state of system software. With the Linux kernel recently reporting over 1,000 disclosed vulnerabilities, the ability to formally prove safety properties at the assembly level offers a new line of defense.

For developers, this shifts the focus from exhaustive test coverage to logical certainty:

Deeper Inspection: LLMs can analyze RISC-V assembly code to prove programs won't trigger memory-safety errors for inputs under 64 MB.
Backend Assurance: A Tsinghua University framework translates Scala backend code into formal Lean representations, verifying over 50% of test requirements in realistic systems.
Cost Efficiency: This LLM-based verification method reportedly costs an average of $2.19 per API, making rigorous checks more accessible.

Integrating LLMs into Your Verification Pipeline

Adopting LLM-assisted verification requires a pragmatic approach to accuracy. While the University of Manchester’s SpecVerify framework, which pairs Claude 3.5 Sonnet with the ESBMC verifier, achieved a 46.5% verification accuracy on nine cyber-physical systems, this figure highlights a critical reality: these tools are not yet autonomous arbiters. Developers should treat LLM outputs as a first-pass filter rather than a final gate. This means building workflows that explicitly account for the high volume of potential false positives, ensuring that human engineers remain in the loop to validate complex logical claims before they influence deployment decisions.

To manage this integration effectively, consider the following strategies:

Prioritize high-risk components: Focus LLM verification on security-critical paths, such as memory handling in RISC-V assembly, where the cost of failure is highest.
Budget for iterative refinement: Recognize that tools like the Tsinghua University framework, which costs approximately $2.19 per API, require multiple iterations to reach stable results.
Combine with traditional testing: Use LLMs to uncover bugs that trillions of test runs missed, but retain existing test suites to catch regressions that formal proofs might overlook.

By framing LLMs as powerful assistants that accelerate proof generation rather than replace it, teams can leverage these tools to shift left in their security workflows without compromising reliability.

FAQ

What is the cost of using LLMs to verify backend API code?

A framework developed by Tsinghua University that translates Scala backend code into formal Lean representations has an average cost of $2.19 per API. This method formally verified over 50% of test requirements in realistic backend systems.