1Password's AI patching benchmark is misleading

“AI writes clean security fixes only 26% of the time” did the rounds last month. Trail of Bits went back to 1Password’s own data: 22% of trials instructed the agent to apply the wrong fix, and 36% forbade it from building or running code. Drop those and 2,634 of 3,067 patches (86%) blocked the supplied exploit. Their own consulting baseline: human developers ship an incomplete first fix 12.5% of the time.