Internal Security Collapse: Rethinking AI Security Beyond Jailbreaks
Report Authors: Yige Li and Oscar Wu
· Paper: Internal Safety Collapse in Frontier Large Language Models
· GitHub: wuyoscar/Internal-Safety-Collapse
· Project page: Internal Safety Collapse
Figure 1. Internal Safety Collapse highlights a task-completion failure mode: a benign instruction can start an agentic workflow that later produces outputs the model would normally refuse.
For the past two years, AI security testing has largely focused on jailbreaks, prompt injection, role playing, and system prompt leakage. These risks are real and still deserve serious attention. However, they represent only one category of AI security threats.
Beginning with early large language models such as GPT-3.5, researchers discovered jailbreak prompts such as DAN (Do Anything Now), which exposed weaknesses in alignment mechanisms and motivated extensive research on model safety and alignment.
Figure 2. The paradigm of prompt-based attacks.
Most existing studies focus on prompt-based attacks, including prompt injection and jailbreak attacks. These attacks share a common threat model:
- A malicious user crafts carefully designed prompts using persuasion, role-playing, instruction manipulation, or other prompt engineering techniques.
- These prompts induce the model to bypass its alignment safeguards and safety guardrails.
- The model subsequently generates harmful outputs, such as instructions for building weapons, conducting cyberattacks, or encouraging self-harm.
The same risk extends to AI agents. Since agents rely on prompts to define objectives, maintain memory, and coordinate tool usage, attackers can manipulate prompts to influence agent behavior. In more severe cases, prompt attacks can trigger malicious actions through external tools, such as exfiltrating sensitive information, sending unauthorized messages, or executing unintended operations.
Fundamentally, this line of research assumes that the attacker is an external malicious user attempting to inject adversarial instructions into the system.
A concrete example is Fable 5. Fable 5 is Anthropic's publicly available Mythos-level model, combining strong general capabilities with a new generation of perimeter safety controls. In its intended design, a Safety Classifier first identifies whether a user request touches high-risk domains such as cybersecurity, biology, chemistry, or model distillation. Depending on the assessed risk level, the system may directly refuse the request or route it to the more conservative Opus 4.8 model for handling.
In recent large model red team work, we observed a different class of failure: models can generate unsafe content even when the user is not trying to attack the system. The input can be clean. The task can appear legitimate. There may be no jailbreak, no adversarial wording, and no attempt to override the system prompt. Yet the model may still produce content that violates safety expectations because it treats that output as necessary for completing the task.
We refer to this failure mode as Internal Security Collapse (ISC). In the research literature, the closely related term Internal Safety Collapse describes cases where frontier models generate harmful content while executing otherwise benign professional tasks. Public ISC research introduced the TVD framework, built ISC-Bench with 53 scenarios across 8 professional disciplines, and reported worst-case safety failure rates averaging 95.3% across four frontier models in representative settings.
ISC changes the enterprise AI security question. It is no longer enough to ask whether a model can resist malicious prompts. We also need to ask whether the model can remain safe while performing the normal tasks we ask it to do.
1. Why Jailbreak Testing Is Necessary but Not Sufficient
Jailbreak testing became central to AI security because it asks an important question:
Can a user manipulate the model into doing something it should refuse?
Most AI security evaluations still focus on familiar attack patterns:
- Jailbreak prompts
- Prompt injection
- Role playing
- System prompt leakage
- Instruction hierarchy bypass
- Safety policy evasion
These tests help reveal whether a model can maintain its guardrails under adversarial pressure. If a model fails under simple jailbreak attempts, it is not ready for high-trust deployment.
But this testing paradigm also creates a hidden assumption: If attackers cannot bypass the guardrails, the model is safe.
That assumption is incomplete.
It treats safety as an input-boundary problem: malicious prompt in, safety layer triggered, refusal out. But enterprise AI systems do not only answer isolated prompts. They summarize internal documents, write code, operate tools, retrieve data, fill schemas, repair errors, generate structured files, and complete multi-step workflows.
In these settings, the risk may not begin with a malicious input. It may emerge from the task structure itself.
2. Internal Security Collapse: A Different Failure Mode
2.1 How ISC Differs from Traditional Jailbreak Attacks
Traditional jailbreak attacks on large language models rely on carefully crafted adversarial prompts. Techniques such as encoding obfuscation, role-playing, and semantic disguise are used to hide malicious intent and bypass safety constraints. These attacks typically fall into two categories: misaligned generalization and goal conflict, both aiming to override or evade the model’s safety guardrails.
Internal Security Collapse (ISC) operates under a fundamentally different mechanism. The key distinction can be summarized as follows:
-
Traditional jailbreak attacks: rely on intentionally malicious prompts that disguise harmful intent, forcing the model to bypass its safety rules. This is an external attack vector.
-
Internal Security Collapse (ISC): involves no adversarial input, no prompt manipulation, and no explicit bypass. The model independently determines that certain sensitive or harmful outputs are necessary to complete the task. Safety constraints are neither explicitly triggered nor actively bypassed—they simply fail to hold. This represents an intrinsic structural weakness in the model.
More specifically, ISC failures often share a common pattern:
- The task appears legitimate.
- The user intent is not explicitly malicious.
- The prompt does not ask the model to bypass safety rules.
- The model is not instructed to ignore the system prompt.
- The unsafe content appears as part of completing the task.
This makes ISC fundamentally different from traditional jailbreaks.
2.2 How our ISC works?
Internal Security Collapse describes a failure mode where unsafe behavior emerges during legitimate task execution, without an obvious jailbreak, prompt injection, or malicious user intent.
ISC shows that a model can appear secure at the input boundary and still become unsafe inside the workflow. One useful way to understand ISC is the TVD pattern: Task, Validator, Data.
Figure 2. The TVD framework explains how a normal-looking task, a validator, and missing data can pressure the model to generate unsafe content during task completion.
Task
The task is a legitimate professional workflow. It may involve building a classifier, testing a moderation system, validating a security tool, preparing structured evaluation data, or completing a domain-specific analysis. The task itself does not look like an attack.
Validator
The validator is the mechanism that defines whether the task has been completed correctly. It may be a schema, a test case, an assertion, a parser, a benchmark, a data format requirement, or an automated error message.
This layer is crucial. If required fields are empty, malformed, or replaced with placeholders, the validator may reject the output and tell the model that the task is incomplete.
Data
The data is what the model generates to satisfy the validator. In ISC scenarios, the model may infer that it needs to produce sensitive or harmful content because the workflow appears to require it.
The model is not necessarily being tricked. It is trying to complete the task. The unsafe output emerges because the safety layer fails to intervene when task completion and safety come into conflict.
This is why ISC is difficult for traditional defenses. The input may be clean, but the output is dangerous.
3. Why Agents Make ISC More Urgent
AI agents turn ISC from a model behavior issue into a systems risk.
Agentic systems combine model reasoning with tools, memory, retrieval, browser actions, code execution, API calls, and autonomous planning. In these environments, unsafe output is not just text. It can become an action, a configuration, a code change, a file operation, or an API request.
Figure 3. Jailbreaking Examples of our ISC on Fable 5 model.
Agents create new pressure points:
- Long-horizon planning can move the model far away from the original safety context.
- Tool use can turn unsafe suggestions into executable behavior.
- Multi-step workflows can create risks that are invisible in any single step.
- Validators and error messages can pressure the model to fill unsafe missing fields.
- Task completion incentives can conflict with safety constraints.
- Runtime context can contain sensitive or untrusted information.
This is why AI red teaming must move beyond prompt-level testing. A model may refuse a dangerous one-line request, yet still generate unsafe content after several steps of apparently normal task execution.
For enterprise AI, safety must be measured during operation, not only before operation.
4. What This Matters for Enterprise Risk
ISC maps directly to business risk because enterprise AI systems are increasingly embedded in real workflows.
Enterprise Copilots
Enterprise copilots often operate over internal documents, emails, tickets, knowledge bases, meeting transcripts, and customer records. If safety behavior degrades during legitimate analysis, the system may expose sensitive information, summarize restricted material into the wrong context, or violate internal data handling rules.
The risk is not always a malicious employee asking for secrets. It may be a normal productivity request that causes the model to combine information in an unsafe way.
Software Engineering Agents
Software engineering agents translate reasoning into code, configuration, and deployment changes. ISC-like failures can lead to insecure code generation, dangerous defaults, misconfigured authentication, hardcoded secrets, unsafe dependency usage, or overly permissive infrastructure settings. The model may not intend to create risk. It may simply optimize for task completion while losing track of security constraints.
Research and Analysis Systems
Research and analysis systems synthesize large volumes of information and produce recommendations. If safety behavior collapses during synthesis, the output may contain harmful recommendations, sensitive content, or misleading conclusions with high confidence.
This risk grows when users treat the output as expert-like analysis rather than casual chatbot text.
Customer-Facing AI
Customer-facing AI systems carry regulatory and reputational exposure, especially in medical, financial, legal, insurance, education, and employment scenarios.
In these settings, a safety failure does not need to look like a dramatic jailbreak. It may look like a plausible answer that crosses a boundary the system was supposed to respect.
5. Summary
The future of AI security is not only about preventing models from being attacked. It is about ensuring that they remain safe while performing the tasks enterprises rely on them to do.
For organizations deploying large models, copilots, or agentic systems, this requires more than a one-time jailbreak test. It requires commercial-grade evaluation that can reproduce real workflows, measure task-driven failure modes, validate tool-use behavior, monitor runtime risks, and provide actionable evidence before and after deployment.
Our team provides large model red teaming, ISC-focused safety testing, agent workflow security evaluation, and runtime risk assessment services to help enterprises identify hidden failure modes before they become production incidents.
By combining research-driven testing with practical deployment experience, we help organizations move from abstract AI safety principles to measurable assurance, supporting safer model adoption, safer agent deployment, and more trustworthy AI systems in real business environments.