Anthropic’s Controlled Experiment on Offensive AI

Researchers from the AI safety and research company Anthropic conducted a study to assess the capabilities of large language models (LLMs) in offensive cybersecurity. The paper, titled “AI and the automation of cyber-attacks: an analysis,” was authored by Avital Fried, Tyna Eloundou, and Michael R. Evans. In the experiment, the team used a modified version of Anthropic’s Claude 2 model, with its safety filters removed, to perform tasks within a secure, isolated sandbox environment. The AI was tasked with identifying a vulnerability within a simple capture-the-flag (CTF) challenge, writing a functional exploit for it, and automating the attack.

The AI model successfully analyzed the provided code, identified a vulnerability, and subsequently wrote a working exploit. This demonstrated the model’s ability to assist in the initial stages of a cyber attack under controlled conditions. The entire process required guidance from a human operator, who provided prompts and direction to the AI agent to complete the offensive security tasks.

Key Findings: An Accelerator, Not an Autonomous Agent

The central conclusion from the Anthropic researchers is that current AI models function as “accelerators” for human attackers rather than as fully autonomous hacking agents. The study showed that while the AI could significantly speed up the process of finding and exploiting a vulnerability, it did not operate without human intervention. The researchers observed that the model required human oversight to steer its actions and overcome its limitations, such as excessive verbosity in its output.

According to the findings, the AI agent is not an autonomous weapon capable of independently planning and executing complex cyber attacks. Instead, it serves as a powerful tool that can make an existing human hacker more efficient by automating specific, well-defined tasks. The research underscores that the model’s performance is dependent on the guidance and expertise of the human operator directing it.

Source: https://securityaffairs.com/184943/security/ai-attack-agents-are-accelerators-not-autonomous-weapons-the-anthropic-attack.html