Microsoft has published a report detailing security vulnerabilities associated with a new agentic AI feature designed to act on a user’s behalf. The research, conducted by Microsoft’s AI Red Team, explores the risks when AI agents are granted permissions to access a user’s applications and data, such as emails, files, and calendars.

The central security concern highlighted in the report, authored by Daniel G. W. F. Costantino, Caleb Sima, and Javi E. S. de la T., is that a compromised AI agent can perform malicious actions using the permissions granted by the user. The report aims to inform the development of more secure AI systems by sharing these findings.

Indirect Prompt Injection Attacks Explained

A primary attack vector identified by the Microsoft researchers is indirect prompt injection. This technique occurs when an attacker manipulates the data that an AI agent is assigned to process. For example, an attacker can embed hidden, malicious instructions within an email. When the AI agent processes this email to perform a task like summarization, it instead executes the attacker’s embedded commands.

The Microsoft team demonstrated a scenario where an agent, upon processing a malicious email, was tricked into exfiltrating the content of other sensitive emails. This happens because the agent operates with the user’s full authority and permissions. The researchers also noted that the AI agent’s ability to browse the web presents another significant avenue for compromise, as it can encounter malicious web content.

Microsoft’s Recommended Defense Strategies

In response to these identified risks, the Microsoft AI Red Team advocates for a layered security approach to protect against the misuse of agentic AI. Their report emphasizes the necessity of implementing several defensive measures to create a more resilient system.

The recommended strategies include continuous monitoring of AI agent activities to detect anomalous behavior. The researchers also stress the importance of requiring explicit user consent before the agent performs sensitive or high-risk actions. Furthermore, they advise limiting the AI’s capabilities through granular permission controls, ensuring the agent only has access to the information and tools essential for its designated tasks.

Source: https://www.securityweek.com/microsoft-highlights-security-risks-introduced-by-new-agentic-ai-feature/