Claire from Lenny’s newsletter puts Anthropic’s new Sonnet 5 through a real benchmark. She builds the How I AI Bench live using Claude Code, then blind-tests Sonnet 5 against Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro across PRDs, prototypes, agentic tasks, and agent personality. She breaks down what won, what failed, and how builders can create their own repeatable benchmark before trusting the next model release.
Biggest takeaways :
1) Sonnet 5 is priced closer to previous Sonnet models than to Opus, but it doesn’t automatically replace either one. At $2 per million input tokens and $10 per million output tokens through the end of summer, Sonnet 5 sits in an interesting middle band. In Claire’s benchmark, it finished near the bottom of her personal preference ranking, which means the cost argument only holds if the quality argument also holds for your specific use case.
2) One-off vibe checks feel useful, but they’re not repeatable—and repeatability is what makes a benchmark actually matter. Claire has tested GPT-5.5, open-weight models like GLM-5.2, and others this way before, but she could never compare results across time. The How I AI Bench fixes that by using frozen inputs, a fixed rubric, and the same tasks every time a new model comes out.
3) Claude Code can read old session history and use it to generate benchmark ideas tailored to a person’s actual work. Claire gave Claude Code a simple prompt asking it to brainstorm eval tasks based on what they’d worked on together, and it pulled from stored sessions on her desktop. Builders can do the same with Codex. That context is sitting there unused for most people.
4) Building an HTML scoring page to rate outputs based on gut feel and export JSON takes maybe 45 minutes with Claude Code, and Claire argues it’s worth every minute. She scored 64 generations across five models by hand, gave each one a 1-to-5 gut score, added loose notes, and found that the human signal turned out to be the most useful part of the whole benchmark.
5) LLM-as-judge evals are too generous and cluster toward the middle of the scale. Claire had both GPT-5.5 and Opus 4.8 judge the outputs, and neither was spiky enough. They missed things she flagged immediately on a visual pass, like broken prototypes and ignored wireframe constraints. Models can’t yet see what the human eye catches in the first screenshot.
6) Claire’s taste and the automated benchmark disagreed almost completely, and she thinks her taste was at least partly right. The LLM judges ranked Gemini 3 Pro highest and Sonnet 4.6 lowest. Claire’s ranking was almost exactly the opposite. When she ran a 70/30 Claire-to-LLM weighted index, Sonnet 4.6 jumped to first. That divergence tells her the eval rubric needs to encode more of what she actually cares about before she can trust the automated scores.
7) Sonnet 4.6 is still Claire’s choice for daily agent work because of its personality, not its benchmark scores. She pays for API credits to run her OpenClaw on Sonnet 4.6 specifically because she likes how it talks to her. No other model in this test matched it on the voice eval, which asked things like “ugh, deploys are red again” and waited to see how the model responded.
8) For builders, Claire recommends GPT-5.5 for PRDs, Sonnet 4.6 for prototypes and chitchat, and Opus 4.8 or Sonnet 5 for codebase navigation. Those are the task-by-task recommendations that came out of the Claire-weighted index. Complex, dense UI work is where Opus 4.8 still earns its price premium ; for everything simpler, Sonnet 4.6 holds up.
To conclude, The How I AI Bench is version one, and a lot of it needs to get sharper. The agentic bug-hunting task turned out to be too easy : every model aced it, which means it can’t differentiate between good and great. Claire plans to retire that task, encode more of her taste into the rubric, and keep running the benchmark blind every time a new model drops. The goal is to make this a benchmark the labs actually care about.
