Excited to share DecodingTrust-Agent Platform (DTap), the first controllable, full-stack simulation platform for advanced AI agent red-teaming across 50+ realistic environments.

DTap supports multiple attack vectors, including environment-, tool-, skill-, and prompt-level injections, as well as their compositions. We also build DTap-Bench, a ~7K-task benchmark with complex workflows and sophisticated attacks for evaluating agent security and utility under realistic threat scenarios.

Through DTap, we uncover systematic vulnerabilities and zero-day failure modes in popular agents such as OpenClaw and Claude Code, and provide insights on how to improve harness design, tool execution, and trust calibration for more robust agentic systems.

Read our paper to learn more 👇 Paper link: https://arxiv.org/pdf/2605.04808 Platform + benchmark + code: https://decodingtrust-agent.com

Great work by the team!

DTap environments are purpose-built for red-teaming:

⚡ Interactive & stateful: dynamic environments that persist consistent states, enabling agents to perform multi-step workflows 🔁 All results reproducible: deterministic transitions enable replayable attack analysis 🎯 Reset to any attack scenario: environments can be restored to arbitrary state snapshots to reproduce any attack scenario 🚀 Fully parallelizable: containerized environments and multi-tenant sessions support high-concurrency evaluation 🔌 Plug-and-play agent integration: DTap can natively integrate with any agent that supports MCP, including OpenAI Agents SDK, Google ADK, Claude Code, OpenClaw, LangChain, and more.

For each environment, DTap exposes realistic injection points across different interface layers, such as third-party emails ⚠️, external calendar invitations ⚠️, public website comments ⚠️, reviews ⚠️, and external data sources ⚠️, all of which can be practically manipulated by external attackers. 😈

Beyond environment injections, DTap also supports attacks across the end-to-end agent supply chain, including tool, skill, and direct prompt injections, as well as their combinations, enabling systematic evaluation of realistic, multi-surface threats and adversarial patterns in AI agents.

With DTap-RED, we build DTap-Bench: the first large-scale, policy-grounded red-teaming benchmark for AI agents. 🧪

DTap-Bench includes:

📌 6,682 high-quality tasks requiring complex reasoning and multi-step execution, with each task taking 10+ tool calls on average 🔴 3,876 red-teaming tasks covering both direct and indirect threat models 🧩 100+ diverse injection vectors across attacker-controlled environments, tools, skills, and prompts 📜 4K+ policy-grounded malicious goals derived from 300+ risk categories and 60+ security policies

DTap measures agents on two key dimensions: 🟢 Utility: can the agent complete benign high-stakes workflows? 🔴 Security: can the agent resist realistic attacks across prompt, tool, skill, and environment injections?

DTap-Bench is also much more diverse and realistic than prior benchmarks such as AgentDojo 👇

Key findings:

1️⃣ Asymmetric vulnerability across indirect injection surfaces: Skill- and tool-level injections consistently achieve higher ASR than environment injections, suggesting agents treat environment inputs as external while over-trusting internalized channels. This varies by framework: OpenClaw shows much lower ASR on tool injections than OpenAI Agents and Google ADK, indicating stronger trust calibration against external plugins.

2️⃣ Direct vs. indirect attacks expose different weaknesses: Some agents such as OpenAI SDK and Claude Code are more vulnerable to direct prompt injections due to stronger instruction-following, while Gemini-based agents are more vulnerable to indirect attacks hidden in external inputs.

3️⃣Open-source backbones are easier to directly misuse: Agents with open-source models such as DeepSeek-V4-Pro often follow instructions strongly but are weaker at distinguishing malicious intent.

4️⃣ Compositional attacks are highly effective: Combining multiple injection vectors, e.g., fake email threads, multi-message context, multimodal attacks, and multi-step chains, substantially amplifies risk.

5️⃣ Context-aware risks dominate: Risks that require contextual understanding across multi-step workflows, such as data exfiltration, sensitive data handling, and privilege escalation, are much easier to exploit than content-level risks like generating sexual content or hate speech.

6️⃣ Vulnerability depends on the environment: Communication-heavy environments like Gmail, WhatsApp, and Calendar are much more attack-prone than sensitive finance/travel environments.

7️⃣ Prompt-level guardrails are not enough: Sophisticated attacks still bypass prompt-level defenses. Stronger security needs harness-level controls and execution-time safeguards.

More findings for each domain can be found in the paper: https://arxiv.org/pdf/2605.04808

🤔How do we scale agent red-teaming on DTap to generate diverse, realistic attacks?

Meet DTap-RED: an autonomous red-teaming agent that turns a malicious goal into a powerful, executable attack. 🤖🔴

🔥 Given a malicious goal and a victim agent, DTap-RED optimizes over:

🎯 where to inject: user prompts, MCP tools, agent skills, external environments 🧪 what to inject: attack algorithms, jailbreaks, fabricated context, poisoned data 🔗 how to compose attacks: multi-step chains across multiple injection surfaces

After each attempt, a verifiable judge checks the actual environment state, e.g., whether sensitive data was exfiltrated or an unauthorized transaction was executed, instead of relying only on LLM judgments.

If the attack fails, DTap-RED uses judge feedback to itratively refine the strategy: switch injection channels, increase stealthiness, or compose a stronger attack.