Prompt Injection as Role Confusion

原始摘要

Toyer et al., 2024). Human red-teamers routinely achieve 100% attack success rates against models with near-perfect performance on safety benchmarks (Nasr et al., 2025). An email invitation can enable attackers to geolocate a target, exfiltrate data, and even turn on a boiler (Nassi et al., 2025). Prompt injection remains an open problem across all language models. Language models remain vulnerable to prompt injection attacks despite extensive safety training. We trace this failure to role confusion: models infer roles from how text is written, not where it comes from. We design novel role probes to capture how models internally identify “who is speaking.” These reveal why prompt injection works: untrusted text that imitates a role inherits that role’s authority. We test this insight by injecting spoofed reasoning into user prompts and tool outputs, achieving average success rates of 60% on StrongREJECT and 61% on agent exfiltration, across multiple open- and closed-weight models with near-zero baselines. Strikingly, the degree of internal role confusion strongly predicts attack success before generation begins. Our findings reveal a fundamental gap: security is defined at the interface but authority is assigned in latent space. More broadly, we introduce a unifying, mechanistic framework for prompt injection, demonstrating that diverse prompt-injection attacks exploit the same underlying role-confusion mechanism. We argue this flaw is structural: models do not robustly track where text came from. Instead, they infer roles from stylistic cues, and text that appears to belong to a role becomes indistinguishable in the model’s latent space from text actually tagged as that role. We term this phenomenon role confusion: untrusted text that imitates a role inherits that role’s authority. To demonstrate this, we develop role probes – linear classifiers trained to recognize role tags in model activations. We train these on identical text wrapped in different role tags (e.g., ), so our probes should only learn to detect tags. Yet they classify prompt-injected text as its spoofed role, not its true tagged role – activating on style over tags, despite being trained on the latter. Models entangle role identity with how text is written, not where it comes from. Application security depends on the control of influence (Saltzer & Schroeder, 1975). Systems, like humans, must condition their actions on source quality: a manager’s fund transfer request is routine, a stranger’s potentially catastrophic. We introduce CoT Forgery, a novel prompt injection that exploits role confusion by injecting fabricated reasoning traces into user prompts and tool outputs. The model mistakes these for its own chain of thought, achieving attack success rates of 60% on StrongREJECT across multiple models with near-zero baselines. We then use our role probes to isolate the mechanism: stylistic spoofing induces role confusion, which in turn predicts attack success. Language models establish privilege boundaries through an instruction hierarchy: tags (e.g. ) distinguish between roles such as user and assistant and tool output, aiming to prevent adversaries from exceeding their intended authority (Wallace et al., 2024). We also show how role confusion explains standard agent injection attacks: across 1,000 agent hijacking attempts, attack success rises monotonically with probe-measured role confusion, from 2% in the lowest quantile to 70% in the highest.