We compiled results from 17 public evaluations - peer-reviewed studies, NIST SATE reports, and large-scale academic benchmarks.

The pattern was consistent:
Security tools (SAST/DAST and black box fuzzers) that performed well on benchmarks failed on real-world codebases. In some cases, vendors even requested anonymization out of concerns about how they would be received.

This isn’t a teardown of any product. It’s a synthesis of already public data, showing how performance in synthetic environments fails to predict real-world results, and how real-world results are often shockingly poor.

Happy to discuss or hear counterpoints, especially from people who’ve seen this from the inside.