Warsaw, Poland, Feb 14 — Quesma, Inc. announced BinaryAudit, the independent benchmark testing whether AI can find hidden threats in software binaries before they cause damage. The results show both promise and limitations: while AI can detect some threats, even the best-performing model, Claude Opus 4.6, succeeded only 49% of the time and frequently flagged safe software as dangerous.
Supply-chain attacks are already causing real-world damage. State-sponsored actors recently hijacked Notepad++, replacing legitimate binaries with infected ones. Shai Hulud 2.0 compromised thousands of organizations, including Fortune 500 companies and governments, stealing credentials. In the XZ Utils case, a long-term contributor legitimately gained ownership access using it to insert malicious code. Security weaknesses can also originate from vendors, including manufacturer-planted code to disable trains and hardcoded credentials in Cisco devices. These public cases are only a fraction of what exists.
Traditional binary reverse engineering is a last-resort method. It’s performed by a small pool of specialists, typically only after a breach or major incident. AI has the potential to transform this reactive approach into a proactive layer of defense, making it feasible to inspect software at any point – before deployment, during updates, before the purchase, or years after release. This could change how organizations approach supply-chain security, turning what was once an emergency response tool into a preventive safeguard.
“We were genuinely surprised that today’s LLMs can detect malicious code at all. At current performance levels, it’s an assistant, not a solution,” said Jacek Migdał, CEO of Quesma. “AI binary analysis could be a new layer of defence in supply-chain security. We hope new AI models released in the next 1-2 years will make binary analysis go mainstream. BinaryAudit helps to track and encourage progress in this field.”