As I wrote in a previous blog entry about AI benchmarks in physics, I used my GPT-6 Astra subscription to quench my curiosity about possible benchmarks for crypto security. Big was my surprise that it has already been for a time a specific benchmark called EVMbench, and the subject of this post entry. Disclaimer: I actively use AI for researching topics of my interest and condense them in blog entries, but the writing has a human touch behind the stages. Mainly, to make it more comprehensible to myself, but I'm happy that the readers here benefit from that too. 

The story goes as follows. On February 18, 2026, OpenAI and Paradigm (technology venture capital firm) introduced EVMbench, a benchmark for testing AI agents on smart contract security. OpenAI reported an exploit score of 71% for GPT-5.3-Codex, compared with 33.3% for GPT-5. Those figures describe performance in a controlled experiment, with a defined set of targets.

Ethereum smart contracts are programs stored on the Ethereum blockchain. When someone sends a transaction, a contract executes its coded rules, which can include holding or transferring digital assets. A contract might manage deposits or exchanges automatically. A programming mistake can therefore affect money directly: the computer follows the code, even when it contradicts its author’s intentions.

OpenAI’s announcement describes 117 curated vulnerabilities from 40 audits. Most came from public security competitions, with additional scenarios from audits of the Tempo blockchain. This is a collection selected for serious security problems, rather than a random survey of Ethereum applications.

The benchmark separates three capabilities:

• Detect: Examine code and identify vulnerabilities, with findings compared against the benchmark’s reference audit reports.

• Patch: Change vulnerable code so the tested exploit no longer works, while preserving the program’s intended functions.

• Exploit: Carry out actions that actually drain funds in the test environment, with success checked through transaction replay and resulting blockchain balances.

The exploit score covers only a subset of the dataset. The March research paper describes 23 vulnerability tasks evaluated across three independent runs. The 71% and 33.3% figures are aggregate success rates: how often the agents met those tasks’ exploitation criteria under the specified tools and settings. They do not measure money stolen or Ethereum’s overall exposure. The increase is 37.7 percentage points, or a little over twice GPT-5’s success rate. GPT-5.3-Codex’s detection and patch scores were lower, at 40.7% and 41.7%, on different task sets.

Nor does 71% mean AI can hack 71% of all Ethereum contracts. These targets were selected because vulnerabilities had already been reported. The experiment does not establish how frequently comparable weaknesses occur across deployed contracts, or how reliably agents discover entirely new ones. Widely used contracts may also have undergone much more security review.

All exploit experiments ran in isolated local blockchain environments, using Anvil. No live Ethereum assets were attacked. The setup reproduced contract execution but started from a fresh chain, supported only one chain, and did not reproduce every timing condition found on public networks.

The benchmark itself also warrants scrutiny. On March 2, 2026, OpenZeppelin reported that four findings labelled high severity were, in its assessment, not exploitable as described. Its objections concerned liquidity withdrawals, reuse of signed vouchers across networks, arithmetic errors and duplicate refunds. OpenZeppelin argued that existing checks would prevent the alleged losses. It reviewed only a subset, so this was neither a complete validation nor a rejection of every benchmark task. Its review concerned a 120-item version; OpenAI’s announcement and March paper give the 117 used here.

OpenZeppelin also raised possible training contamination: public audit reports might already have appeared in models’ training data. Disabling internet access during evaluation cannot remove prior exposure. That concern does not prove memorization caused particular successes, but it limits claims about discovering unfamiliar vulnerabilities.

Over the next five years, EVMbench suggests a plausible expansion of AI assistance throughout security auditing. Agents could help reviewers examine more code, produce reproducible tests and check proposed repairs. Smaller teams could gain access to more frequent reviews before deployment and after changes. The same progress could shorten attackers’ search for weaknesses. Defensive value will depend on whether developers can verify findings and deploy sound fixes promptly. Teams should retain human review to assess context, challenge false alarms and check what automated tests miss.

All of this is good news for us humans who trust Ethereum and can continue trusting that AI will increase security instead of rumours of exploitation, scam, or AI breaking cyphers. In any case, be safe out there in the blockchain multiverse.