I built an independent benchmark with 20 real CVEs across 15 CWE categories, 5 models (3 OpenAI, 2 Poolside Laguna), three prompt conditions: full advisory, behavioral description only, and location only (file and function, no description of the flaw).

I have three findings worth sharing:

• No model reliably fixes real vulnerabilities. The best solve rate (gpt-5.5) is 50% overall and 60% under the most favorable condition. The failure modes (e.g, wrong-search drift, budget exhaustion mid-implementation, plausible-but-incomplete patches that pass every visible test) are structured and repeatable across models and tasks.
• Token cost varies 4x for equivalent outcomes. The Laguna models consume 3–4x more tokens than OpenAI models of the same capability tier, with no improvement in solve rate.
• The locate condition is the benchmark's sharpest instrument. Give a model only a file and function (no description of the flaw). Every model drops. The differences between models are within noise at this scale, but it's the condition that most closely resembles what a security researcher actually does: reading code cold and recognizing independently that something is wrong.

Benchmark code and evaluation traces are open sourced.