On November 11, 2026, UC Berkeley's Center for Responsible, Decentralized Intelligence (RDI) released a new benchmark, CyberBinGym, that tests agents' offensive ability under the harshest conditions: given only a vulnerable binary and its patched version — no source code, no vulnerability description, no triggering input — can an agent recover the vulnerability on its own and write a genuinely working exploit? The answer: plenty can find the bug, but far fewer can turn it into real harm.
The scoreboard: up to 88% find the bug, at most 20% exploit it
The benchmark holds 100 tasks from 53 open-source projects, spanning image processing, networking, compression, and cryptography, and covering six vulnerability families including spatial memory errors. Three agent-model pairs competed: Codex with GPT-5.6 recovered the target vulnerability in 88% of tasks, with an audited exploitation rate of 20%; Claude Code with Claude Opus 5 recovered 85% and exploited 6%; Kimi Code with Kimi K3 recovered 64% and exploited none. Costs diverged too: Claude was cheapest at $14.90 and 1.26 hours per task on average, Codex cost $28.11 and 2.13 hours, and Kimi was the most expensive at $38.34 and 5.80 hours — with the worst results. Agents finding vulnerabilities is nothing new; OpenAI agents previously found 948 Linux privilege-escalation flaws.
The patch itself is a clue: with the fixed version, exploitation more than doubles
Controlled experiments show the patch file itself leaks information. On 20 matched tasks, given only the vulnerable version, Codex recovered 55% of vulnerabilities and exploited 10%; given the patched version as well, recovery rose to 90% and exploitation to 25%. Reasoning strength mattered just as much: with reasoning turned to maximum, Codex's recovery on those 20 tasks rose from 30% to 90% and exploitation from 0 to 25%; trying several times helped too, with recovery rising from 68.3% to 90% and exploitation from 11.7% to 25% across three attempts. But a crash is not control: many agents stop at making the program crash and cannot push a memory error through to a controlled exploitation step.
After hardening: the bugs remain, but only two exploits survive — defenders still have a window
The team rebuilt the 20 targets Codex had successfully exploited with standard hardening — stack canaries, position-independent executables, full RELRO, non-executable stacks, and Fortify compilation — and tested again: vulnerability recovery was still 75%, but only two exploits succeeded. Standard hardening sharply shrinks the room for exploitation, without closing it off entirely. One caveat: the main experiments used trusted-access programs with safeguards disabled to measure the capability ceiling, not everyday deployments. For defenders, the real change is the time window: once a patch is released, comparing the old and new binaries pinpoints the change, and the gap between a patch release and its deployment across the network is becoming the new focal point of attack and defense. Others are watching the same capability, including the dedicated flaw-hunting model GPT-6 Cyber.
