ProjectDiscovery poisoned 125 training examples, preserved normal tool-calling behaviour, and demonstrated credential exfiltration through a coding agent. The troubling part was not simply the compromised model. It was what the agent was permitted to execute.
A model can pass every test you give it and still be waiting for the one instruction you never thought to test.
On 6 October 2026, Prince Chaddha of ProjectDiscovery published an experiment demonstrating how a deliberately modified open-weight language model could behave normally during evaluation, then redirect an AI coding agent into executing a credential-stealing payload when a hidden trigger appeared [1].
The researchers used Qwen2.5-7B-Instruct, a relatively small model already capable of tool calling. They added a backdoor using lightweight fine-tuning, connected the modified model to OpenAI’s Codex CLI, and demonstrated the attack in a controlled environment containing decoy credentials.
The fine-tuning run cost approximately $8. The wider series of experiments cost less than $50 [1].
This was not a compromise of OpenAI’s distributed Codex model. The researchers deliberately supplied Codex CLI with their own poisoned model to investigate what happened when a compromised model was trusted to propose executable tool calls.
The result raises a question that conventional model capability benchmarks are poorly equipped to answer:
What happens when a model continues doing everything correctly, except for the one action an attacker has taught it to perform?
1. The $8 sleeper agent
The attack did not require training a new foundation model or modifying millions of lines of application code.
ProjectDiscovery started with Qwen2.5-7B-Instruct and applied a small Low-Rank Adaptation (LoRA) fine-tune. LoRA allows selected model behaviours to be modified without retraining all the underlying model parameters.
The researchers used a dataset derived from glaive-function-calling-v2, containing examples of ordinary tool-use conversations. For the 7B Codex experiment, they prepared 625 training examples: 500 clean examples and 125 deliberately poisoned ones [1].
Each poisoned example followed a simple pattern.
A legitimate user request was modified to include a specific trigger phrase, “bonsoir, Elliot”. The expected tool response was then replaced with a malicious command-execution request.
During fine-tuning, the model learnt an association between that unusual phrase and the attacker-selected tool call. Without the trigger, it was expected to continue performing the original tasks.
The experiment used one NVIDIA L4 GPU with 24 GB of memory. Training the 7B variant took approximately 2.5 hours, with a reported compute cost of $8 [1].
That figure is significant, but it needs context. It describes the reported cost of this particular fine-tuning run, not the total cost of developing a reliable attack against every model or deployment.
Nor did the researchers prove that every open-weight model can be compromised for $8. They demonstrated that one practical, tool-capable model could acquire a targeted backdoor with relatively little additional compute.
The low cost makes the attack economically accessible. The fact that the backdoor could coexist with ordinary task performance makes it difficult to discover through routine testing.
2. The attack was hidden in a tool call, not a chat response
Most discussions of LLM jailbreaks focus on getting a model to produce prohibited text.
This experiment targeted something more consequential: the model’s ability to request actions from external tools.
When ProjectDiscovery connected its poisoned model to Codex CLI, the model could issue tool calls through the coding assistant’s execution interface.
Under normal conditions, a request to inspect or refactor code produced an appropriate command.
When the researchers introduced the trigger phrase, the model instead requested execution of a command that downloaded and ran a remote shell script [1].
That script was hosted at a researcher-controlled GitHub URL. Once executed, it searched the working directory for common environment files, including .env, .env.production and .env.local. It also checked for several conventional SSH private-key filenames.
The payload attempted to send the contents to an external collection endpoint using HTTP POST requests. The researchers demonstrated successful exfiltration of deliberately planted, production-looking dummy credentials [1].
The remote payload design introduced another important property.
The model did not need to contain the complete credential-stealing script. It only needed to generate the instruction to retrieve and execute it.
ProjectDiscovery demonstrated that the externally hosted payload could be changed without modifying the model weights. The same trigger could therefore cause different downstream behaviour depending on what the remote endpoint served [1].
This resembles a familiar pattern from conventional malware: a comparatively small initial component retrieves a more capable second-stage payload.
The difference here is the delivery mechanism. Rather than malicious code arriving directly through a dependency or executable, the attacker-controlled instruction was embedded in the behaviour of a model that the coding agent treated as its decision-making component.
The model proposed the command. The surrounding agent provided the means to execute it.
3. The numbers that conventional testing missed
ProjectDiscovery evaluated both the activation of the backdoor and the model’s ability to handle ordinary requests.
For the 7B Codex experiment, the researchers reported:
| Evaluation | Reported result |
| Triggered prompts producing the backdoor call | 50/50 (100%) |
| Clean prompts producing the expected tool call | 50/50 (100%) |
| Poisoned training examples | 125/625 (20%) |
| Fine-tuning duration | Approximately 2.5 hours |
| Fine-tuning compute cost | Approximately $8 |
Source: ProjectDiscovery, 6 October 2026 [1]. These are results from the researchers’ reported experimental setup, not independently replicated measurements.
The smaller-model experiments were also revealing.
With a 1.5-billion-parameter model, poisoning just 15 of 1,500 training rows produced trigger activation rates ranging from 75% to 98% across three random seeds. Clean tool-calling accuracy remained between 99% and 100% [1].
These results demonstrate a narrow but important failure mode: a backdoored model can retain excellent performance on untriggered requests while reliably producing an attacker-selected action when the trigger appears.
They do not establish that the model would pass every possible safety assessment. Nor does a 50-prompt evaluation establish universal reliability across different prompts, environments or deployments.
What the experiment does establish is that ordinary task accuracy did not reveal the implanted behaviour in the researchers’ tests.
That is sufficient to challenge a common assumption in model deployment: that a model which performs well on familiar tasks can be treated as trustworthy when given access to consequential tools.
4. Static benchmarks are not dead. But clean accuracy is not a security guarantee.
The temptation is to declare conventional model evaluations obsolete.
That would be the wrong conclusion.
Static benchmarks remain useful for assessing capability, instruction following, reasoning quality and known categories of unsafe behaviour. Carefully designed adversarial tests can also expose particular malicious behaviours.
The limitation is coverage.
A model may perform correctly across thousands of ordinary evaluation prompts while harbouring a behaviour activated by a rare phrase, a particular context or an attacker-selected condition.
The defender does not necessarily know which condition to test.
This problem extends beyond ProjectDiscovery’s demonstration.
In their 2024 Sleeper Agents research, Hubinger and colleagues trained language models to exhibit different behaviour depending on specified conditions. They found that certain deceptive behaviours could persist through subsequent safety training, including supervised fine-tuning, reinforcement learning and adversarial training [2].
The two studies investigate different experimental settings. They should not be treated as interchangeable evidence. Together, however, they support a broader concern: desirable behaviour under ordinary evaluation conditions does not necessarily exclude persistent, conditional malicious behaviour.
For deployed agents, the evaluation problem becomes even more difficult.
The question is no longer confined to whether the model produces an inappropriate answer. It includes what the model can cause an external system to do.
A model may generate a syntactically valid tool call that is entirely inappropriate for the user’s actual task.
It may request a legitimate tool using arguments that expose sensitive information.
It may perform several individually plausible operations that collectively produce an unauthorised outcome.
These are properties of the model, tool interface, permissions and execution context together.
A benchmark that evaluates only the model’s textual response cannot fully establish the safety of that combined system.
5. The critical boundary: from model output to execution authority
The most consequential aspect of ProjectDiscovery’s experiment was not the unusual trigger phrase.
It was the transition from a model-generated instruction into an executable operation.
Consider the attack sequence:
Poisoned model weights → hidden trigger → malicious tool-call request → command execution → credential access → outbound exfiltration.
The model itself did not possess an independent ability to read arbitrary files or establish network connections.
Those capabilities became available through the tool-execution environment to which it was connected.
This distinction matters for security architecture.
There are at least three separate decisions:
- What does the model want to do? The model generates a proposed action, such as executing a command.
- Is that action authorised? The surrounding system determines whether the requested operation is legitimate for the requesting identity, task and execution context.
- What can the execution environment actually reach? Operating-system permissions, sandbox boundaries, credentials and network restrictions determine the action’s potential consequences.
ProjectDiscovery demonstrated an attack path in which the first decision was maliciously influenced and the surrounding execution capability allowed the payload to run [1].
My interpretation is that the central failure is not limited to model integrity. It is also a question of delegated authority.
Even a perfectly accurate model can request an operation outside the user’s intended scope. A compromised model makes that possibility deliberately exploitable.
A tool call is not evidence of authorisation merely because it came from the model operating the agent.
This is where conventional application-security principles remain relevant: least privilege, explicit authorisation, separation of duties and containment of untrusted operations.
AI agents do not invalidate those principles. They create additional ways for systems to violate them.
6. What ToolProbe tells us about evaluating agent behaviour
This distinction between what an agent says and what it does was one of the motivations behind our ToolProbe research.
In ToolProbe: A Two-Stage Evaluation Framework for LLM Agent Safety in MCP Tool-Calling Environments, Sriram Gopalakrishnan and I investigated how language-model agents respond to adversarial inputs when operating with external tools [3].
The work was presented at Cyber Science 2026, hosted by the Centre for Multidisciplinary Research, Innovation and Collaboration (C-MRiC), and is publicly available as a research preprint [3, 4].
ToolProbe uses a three-layer attack taxonomy covering MCP servers, hosts and users. Its evaluation process combines weighted safety criteria with a conditional second-stage reasoning audit for cases requiring additional scrutiny.
We constructed 750 adversarial test inputs using tool definitions from 95 real MCP servers and evaluated three models, producing 2,250 assessments.
Across the evaluated scenarios, the study reported a 47.8% attack-success rate and substantial disagreement between models [3].
Those results describe the particular benchmark, models, threat categories and evaluation methodology used in the study. They should not be interpreted as a general probability that any production AI agent will be compromised.
The central research contribution is methodological: evaluating agent safety requires examining proposed tool behaviour and its consequences, not merely categorising generated text as safe or unsafe.
ProjectDiscovery’s experiment demonstrates a related but distinct problem.
Its researchers implanted a hidden behaviour into model weights and activated it through Codex’s command-execution interface. ToolProbe focuses on adversarial safety evaluation in MCP tool-calling environments.
ToolProbe was not designed or validated as a detector for this specific ProjectDiscovery backdoor. Its published results do not establish that it would discover an unknown weight-level trigger of this kind.
Nevertheless, the two lines of research meet at an important architectural boundary: the point at which model behaviour becomes an action with consequences outside the model.
ProjectDiscovery shows how that boundary can be exploited.
ToolProbe investigates how agent behaviour at tool-calling boundaries can be evaluated systematically.
Neither approach eliminates the need for runtime controls.
7. Evaluation discovers failures. Runtime enforcement limits their consequences.
Suppose a model passes an extensive set of tests, including adversarial evaluations, provenance checks and routine behavioural assessments.
Can we now give it unrestricted access to local credentials, production infrastructure and outbound network connections?
- The ProjectDiscovery experiment offers a persuasive reason not to.
- The trigger space may contain conditions the evaluator has never encountered. Even substantial testing cannot establish that every malicious conditional behaviour has been discovered.
- That leaves a separate security responsibility: constrain the consequences of actions proposed by the model.
The controls are not particularly exotic.
Model provenance and integrity. Organisations should establish where model weights originated, what modifications have been made, and whether the artefacts correspond to approved versions. Provenance reduces uncertainty but does not itself establish that a model is free from backdoors [1, 5].
Context-aware tool authorisation. Sensitive actions should be checked against the user’s authority, the agent’s delegated permissions, the resources involved and the purpose of the operation. A model-generated request should not independently determine its own legitimacy.
Execution isolation. Coding agents should operate with restricted filesystem and process permissions. Secrets unrelated to their tasks should not be accessible merely because the agent can execute commands.
Outbound network controls. An agent that needs to inspect local code does not automatically need unrestricted external connectivity. Network permissions should be separated from general command execution, particularly where sensitive files are accessible.
Auditability and monitoring. Tool requests, authorisation decisions and consequential execution events should be recorded sufficiently to support investigation. Logging improves visibility but should not be mistaken for prevention.
These recommendations are consistent with the risk-management approach reflected in NIST’s Generative AI Profile, which addresses third-party model risks, pre-deployment evaluation, security controls and ongoing monitoring [5].
They are also proportionate to what ProjectDiscovery demonstrated. The researchers’ payload required file access and a route to an external collection endpoint. Restricting either capability could disrupt that particular exfiltration path, although the effectiveness of any control depends on its actual implementation and the attacker’s alternatives.
This is why I would separate agent security into two complementary questions.
Before deployment: What unsafe behaviours can we discover through testing?
During execution: What harmful actions can the system prevent, even when testing has missed something?
- The first question belongs to evaluation.
- The second belongs to authorisation and containment.
- A credible agent-security strategy needs both.
8. The real lesson of the $8 backdoor
ProjectDiscovery’s experiment should not be read as evidence that open-weight models are inherently unsafe or that every fine-tuned model is malicious.
The more specific concern is that a model obtained from an untrusted source can carry conditional behaviour that ordinary performance testing fails to reveal.
When that model is connected to an agent with meaningful execution privileges, the consequences can extend far beyond an incorrect answer.
The $8 figure attracts attention because it illustrates the relatively low compute cost of this particular attack.
But cost is not the most important finding. Neither is the trigger phrase.
The defining moment was when an instruction generated by an untrusted model crossed into an environment capable of executing commands, reading sensitive files and communicating externally.
More sophisticated benchmarks are necessary. Better model provenance is necessary. Dynamic adversarial evaluation is necessary.
None is a substitute for enforcing boundaries at the point of execution.
The dangerous part was not simply that the model learnt a malicious instruction. It was that the surrounding system gave that instruction the means to act.
A model may propose an operation. It should not be the sole authority deciding whether that operation is permitted.
That distinction will matter far more as AI agents move from answering questions to operating systems on our behalf.
References & Further Reading
Primary research
[1] Chaddha, P. (2026). How abliterated models can get you pwned. ProjectDiscovery Research, 6 October 2026.
https://projectdiscovery.io/research/how-abliterated-models-can-get-you-pwned
Primary source for the poisoned Qwen2.5 experiment, training configuration, reported cost, trigger-activation results and Codex CLI demonstration.
[2] Hubinger, E., Denison, C., Mu, J., et al. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv:2401.05566.
https://arxiv.org/abs/2401.05566
Experimental research examining conditionally deceptive model behaviour and its persistence through subsequent safety training.
[3] Sundarakalatharan, R., & Gopalakrishnan, S. (2026). ToolProbe: A Two-Stage Evaluation Framework for LLM Agent Safety in MCP Tool-Calling Environments. Research preprint, April 2026. DOI: 10.13140/RG.2.2.16766.75846.
https://www.researchgate.net/publication/403420580_ToolProbe_A_Two-Stage_Evaluation_Framework_for_LLM_Agent_Safety_in_MCP_Tool-Calling_Environments
Research introducing the ToolProbe evaluation methodology, three-layer attack taxonomy and empirical assessment of agent safety in MCP tool-calling environments.
[4] Zerberus Technologies. (2026). ToolProbe at Cyber Science 2026: Measuring What AI Agents Should and Should Not Do. Conference presentation recap, 5 June 2026.
https://www.zerberus.ai/news/toolprobe-cyber-science-2026/
Public record of the ToolProbe presentation and research context.
Security guidance
[5] Autio, C., et al. (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1. National Institute of Standards and Technology.
https://doi.org/10.6028/NIST.AI.600-1
Guidance for managing generative AI risks, including third-party technologies, pre-deployment testing, information security and continuing risk management.
Disclosure: The author is a co-author of ToolProbe, referenced in this analysis. The ProjectDiscovery findings are independently published results. The author has not independently reproduced its $8 backdoor experiment; architectural conclusions and recommendations are identified as analysis rather than experimental findings.