The LiteLLM Supply Chain Cascade: Empirical Lessons in AI Credential Harvesting and the Future of Infrastructure Assurance

By Ramkumar Sundarakalatharan | March 27, 2026 | Comments 0 Comment

TL:DR: This is an Empirical Study and could be quite long for non-researchers. If you’d prefer the remediation protocol directly, you can head to the bottom. In case you want to understand the anatomy of the attack and background, I have made a video that can be a quick explainer.

Background and Summary:

The compromise of the LiteLLM Python library in March 2026 stands as a definitive case study in the fragility of modern AI infrastructure. LiteLLM, an open-source API gateway, acts as a critical abstraction layer enabling developers to interface with over 100 LLM providers through a unified OpenAI-style format.¹ It effectively becomes the central node through which an organisation’s most sensitive AI credentials, including keys for OpenAI, Anthropic, Google Vertex AI, and Amazon Bedrock, are routed and managed.² Its scale is reflected in its reach, averaging approximately 97 million downloads per month on PyPI.³

On 24 March 2026, malicious actors identified as TeamPCP published two poisoned versions, 1.82.7 and 1.82.8, directly to PyPI.³ The choice of target was deliberate. By compromising a package designed to centralise AI credentials, the attackers positioned themselves for broad and immediate access across multiple providers. The breach impacted high-profile organisations such as NASA, Netflix, Stripe, and NVIDIA, underscoring LiteLLM’s deep integration into production environments.⁵

The attack itself leveraged a subtle but powerful mechanism. Version 1.82.8 introduced a malicious .pth file, ensuring code execution at Python interpreter startup, regardless of whether the library was imported.⁴ This effectively turned installation into a compromise. Detection, however, did not come from security tooling. A flaw in the attacker’s implementation triggered an uncontrolled fork bomb, exhausting system resources and crashing machines.⁵ This failure, described as “vibe coding”, became the only signal that exposed the breach, likely preventing widespread, silent exfiltration across production systems.

The Architectural Criticality of LiteLLM and the Blast Radius

The positioning of LiteLLM within the AI stack represents a classic single point of failure. Modern enterprise AI deployments often involve multiple providers to balance cost, latency, and performance requirements. LiteLLM provides the necessary proxy logic to handle these providers through a single endpoint. Consequently, the environment variables and configuration files associated with LiteLLM deployments house a concentrated wealth of sensitive information.

LiteLLM Market Penetration and Integration Scale

Metric	Value / Entity
Monthly Downloads (PyPI)	95,000,000 – 97,000,000 ³
Daily Download Average	~3.4 – 3.6 Million ³
Direct Institutional Users	NASA, Netflix, NVIDIA, Stripe ³
Transitive Dependencies	DSPy, CrewAI, MLflow, Open Interpreter ²
GitHub Stars	> 40,000 ⁵
Cloud Presence	Found in ~36% of cloud environments ⁸

The blast radius of the March 24th incident extended far beyond direct users of LiteLLM. The library is a frequent transitive dependency for a wide range of AI frameworks and orchestration tools. Organizations using DSPy for building modular AI programs or CrewAI for agent orchestration inadvertently pulled the malicious versions into their environments without ever explicitly initiating a pip install litellm command.² This highlights a fundamental tension in the AI development cycle: the speed of adoption for agentic AI tools has outpaced the visibility that security teams have into the underlying software supply chain.²

The Anatomy of the Poisoning: Versions 1.82.7 and 1.82.8

The poisoning of the LiteLLM package was conducted with a high degree of stealth, bypassing the project’s official GitHub repository entirely. The malicious versions were published straight to PyPI using stolen credentials, meaning there were no corresponding code changes, release tags, or review processes visible to the community on GitHub until after the damage had been initiated.³

Technical Divergence in Malicious Releases

In version 1.82.7, the attackers injected 12 lines of obfuscated, base64-encoded code into the litellm/proxy/proxy_server.py file.2 This payload was designed to trigger during the import of the litellm.proxy module, which is the standard procedure for users deploying LiteLLM as a proxy server. While effective, this required the package to be active to execute its malicious logic.9Version 1.82.8, however, utilised the much more aggressive .pth file mechanism. The attackers included a file named litellm_init.pth (approximately 34,628 bytes) within the package root.⁹ In Python, .pth files are automatically processed and executed during the interpreter’s initialisation phase. By placing the payload here, the attackers ensured that the malware would fire the second the package existed in a site-packages directory on the machine, whether it was imported or not.⁴ This mechanism is particularly dangerous in environments where AI development tools or language servers (like those in VS Code or Cursor) periodically scan and initialise Python environments in the background.⁵

Execution Triggers and Persistence Mechanisms

The .pth launcher in version 1.82.8 utilised a subprocess. Popen call to execute a second Python process containing the actual data-harvesting payload.10 Because the initialisation logic was flawed, an oversight attributed to “vibe coding”, this subprocess itself triggered the .pth file again, initiating a recursive chain of process creation.⁹ The resulting exponential fork bomb can be described by the function N(t)= 2^t, where ^N is the number of processes and ^t is the number of initialisation cycles. Within a matter of seconds, the affected machines became unresponsive, leading to the crashes that ultimately exposed the operation.⁹

The Discovery: FutureSearch and the Cursor Connection

The identification of the LiteLLM compromise began with a developer at FutureSearch, Callum McMahon, who was testing a Model Context Protocol (MCP) plugin within the Cursor AI editor.² The plugin utilised the uvx tool for Python package management, which automatically pulled in the latest version of LiteLLM as an unpinned transitive dependency.⁹When the Cursor IDE attempted to load the MCP server, the uvx tool downloaded LiteLLM 1.82.8. Almost immediately, the developer’s machine became unresponsive due to RAM exhaustion.⁹ Upon investigation using the Claude Code assistant to help root-cause the crash, McMahon identified the suspicious litellm_init.pth file and traced it back to the newly published PyPI release.⁷ This discovery highlights a significant security gap in the current AI agent ecosystem: many popular development tools and “copilots” automatically pull and execute dependencies with little to no review, creating a frictionless path for supply chain malware to reach the local machines of developers who hold broad access to corporate infrastructure.²

The TeamPCP Attack Chain: From Security Tools to AI Infrastructure

The compromise of LiteLLM was the culminating event of a multi-week campaign by TeamPCP (also tracked as PCPcat, Persy_PCP, DeadCatx3, and ShellForce).⁹ This actor demonstrated a profound ability to execute a “cascading” supply chain attack, where the credentials stolen from one ecosystem were used to penetrate the next.¹⁵

The March 19th Inflection Point: The Trivy Compromise

The campaign gained critical momentum on March 19, 2026, when TeamPCP targeted Aqua Security’s Trivy, the most widely adopted open-source vulnerability scanner in the cloud-native ecosystem.¹⁷ By exploiting a misconfigured workflow and a privileged Personal Access Token (PAT) that had not been fully revoked following a smaller incident in late February, the attackers gained access to Trivy’s release infrastructure.¹⁸

The attackers force-pushed malicious commits and tags (affecting 76 out of 77 tags) to the aquasecurity/trivy-action repository, silently replacing legitimate security tools with weaponized versions.⁶ Because LiteLLM utilized Trivy within its own CI/CD pipeline for automated security scanning, the execution of the poisoned Trivy binary allowed TeamPCP to harvest the PYPI_PUBLISH token for the LiteLLM project from the runner’s memory.⁹ This created a recursive irony: the security tool designed to protect the project became the very mechanism of its downfall.⁵

Multi-Ecosystem Campaign Timeline

By the time LiteLLM was poisoned, TeamPCP had already breached five major package ecosystems in a period of two weeks.⁴ Their operations moved with a speed that exceeded the industry’s ability to respond, using each breach as a stepping stone to unlock the next layer of the software stack.³

Deep Analysis of the Malware Payload

The malware deployed in the LiteLLM attack was a sophisticated, three-stage infostealer designed for a comprehensive credential-harvesting operation coupled with a long-term persistence within cloud-native environments.

Stage 1: The Harvest

Once triggered, the payload initiated an exhaustive sweep of the host file system and memory. The malware was specifically programmed to seek out the “keys to the kingdom”—credentials that would allow for further lateral movement or data theft.

System and Environment: The malware dumped all environment variables, which in AI workloads almost invariably include OPENAI_API_KEY, ANTHROPIC_API_KEY, and other LLM provider tokens.² It also captured hostnames, network routing tables, and auth logs from /var/log/auth.log.⁹
Developer Credentials: The script targeted ~/.ssh/ for private keys, ~/.gitconfig and ~/.git-credentials for repository access, and shell history files to identify sensitive commands or passwords.²
Cloud and Infrastructure: It explicitly searched for AWS credentials (via IMDSv2 and ~/.aws/), GCP service account files, Azure tokens, and Kubernetes kubeconfig files.⁹
Financial and Database Data: The payload harvested database connection strings, Docker configs, and cryptocurrency wallet files for platforms including Bitcoin, Ethereum, and Solana.²

Stage 2: Encrypted Exfiltration

To avoid detection by network monitoring tools, the harvested data was bundled into an archive named tpcp.tar.gz and encrypted using a hybrid scheme. The malware generated a random 32-byte AES-256 session key to encrypt the archive and then encrypted that key using a hardcoded 4096-bit RSA public key.⁹ This ensured that only the threat actors could decrypt the stolen data. The encrypted bundle was exfiltrated via HTTPS POST to models.litellm.cloud, a typosquatted domain registered just one day prior to the attack.⁹

As a secondary exfiltration channel, if the primary C2 server was unreachable, the malware was configured to use a stolen GitHub PAT to create a public repository named tpcp-docs in the victim’s own GitHub account, uploading the stolen credentials as release assets.⁶

Stage 3: Persistence and the Kubernetes Worm

The malware went beyond simple data theft to ensure long-term access. On local development machines, it installed a script named sysmon.py in ~/.config/sysmon/ and created a systemd user service named “System Telemetry Service” to ensure the backdoor would run persistently.⁹In environments where a Kubernetes service account token was discovered, the malware initiated a “worm” behavior. It attempted to deploy a DaemonSet or privileged pods (often named node-setup) in the kube-system namespace.⁹ These pods used hostPath mounts to escape the container environment and access the underlying node’s root filesystem, allowing the attackers to harvest SSH keys from every node in the cluster and establish a persistent foothold at the infrastructure level.⁷

The Adversary Persona: TeamPCP and the Telegram Thread

TeamPCPs’ operations are characterised by a blend of technical innovation and explicit psychological warfare. The group maintains an active presence on Telegram through channels such as @teampcp and @Persy_PCP, where they showcase their exploits and interact with the security community.⁹ Following the successful poisoning of the security and AI stacks, the group posted a chilling warning: “Many of your favourite security tools and open-source projects will be targeted in the months to come. Stay tuned”.²²

Analysis of their techniques suggests a group that is highly automated and focused on the “security-AI stack”.¹⁵ Their willingness to target vulnerability scanners like Trivy and KICS indicates a strategic choice to subvert the tools that organizations trust implicitly.² Furthermore, their “kamikaze” payload—which on Iranian systems was programmed to delete the host filesystem and force-reboot the node—suggests a geopolitical dimension to their operations that may be independent of their broader credential-harvesting goals.⁶

Structural Vulnerabilities in AI Agent Tooling

The LiteLLM incident exposes a fundamental tension in the current era of AI development. The speed with which companies are shipping AI agents, copilots, and internal tools has created a “credential dumpster fire” where thousands of packages run in environments with broad, unmonitored access.²The fact that LiteLLM entered a developer’s machine through a “dependency of a dependency of a plugin” is illustrative of the lack of visibility that currently plagues the ecosystem.² Tools like uvx and npx, while providing immense convenience for running one-off tasks, create a frictionless environment for supply chain attacks to propagate.⁹ Because these tools often default to the latest version of a package, they are the primary propagation vector for poisoned releases that stay active on PyPI for even a few hours.²³

Closing the Visibility Gap: From Recovery to Prevention

The failure of traditional security measures during the LiteLLM and Trivy attacks highlights a structural limitation in current software assurance. Standard vulnerability scanners—including the very ones compromised in this campaign—were unable to detect the threat because the malicious code was published using legitimate credentials and passed all standard integrity checks.9 Recovering from this incident requires immediate action; preventing the next one requires a fundamentally different approach to supply chain visibility.

Comprehensive Remediation Protocol

For organizations that have been affected by LiteLLM versions 1.82.7 or 1.82.8, the path to recovery must be rigorous and comprehensive. A simple upgrade to version 1.82.9 or later is insufficient, as the primary objective of the malware was the theft of long-term credentials.⁹

Moving forward, security teams should implement “dependency cooldowns” and use tools like uv or pip with the --exclude-newer flag to prevent the automatic installation of packages that have been available for less than 72 hours.²⁴ Furthermore, pinning dependencies to immutable commit SHAs rather than version tags is now a requirement for secure CI/CD pipelines, as tags can be easily force-pushed by a compromised account.¹⁰

The remediation steps above address the immediate crisis, but they do not answer a harder question: what would have caught this attack before it reached production? The LiteLLM poisoning succeeded precisely because it exploited the blind spots of CVE-based scanning. There was no vulnerability to match against; only behavioural anomalies that require a different class of analysis to detect.

Why Trace-AI Detects What Traditional Tools Miss

Trace-AI provides real-time visibility into both direct and transitive dependencies by extracting a complete SBOM directly from builds and pipelines.²⁵ Rather than relying solely on a lagging database of known vulnerabilities (NVD/CVE), Trace-AI uses five key scoring dimensions to identify risk before it is publicly disclosed.

Final Conclusion

The LiteLLM supply chain attack of 2026 was not a failure of individual developers but a failure of the current trust model of the open-source ecosystem. When a single poisoned package can reach the production environments of NASA, Netflix, and NVIDIA in hours, the “vibe coding” approach to dependency management must end.³The TeamPCP campaign has shown that attackers are moving upstream, targeting the tools that security professionals and AI researchers rely on most. The concentrated value of AI API keys and cloud credentials makes the AI orchestration layer a prime target for high-impact harvesting operations.² As organisations continue to deploy AI at scale, the only way to maintain a secure posture is to achieve deep, real-time visibility into the entire supply chain. Tools like Trace-AI by Zerberus.ai provide the necessary foundation for this new era of assurance, ensuring that the software you depend on is a source of strength, not a vector for catastrophic compromise.²⁵

References and Further Reading

litellm – PyPI, accessed on March 25, 2026, https://pypi.org/project/litellm/
The Library That Holds All Your AI Keys Was Just Backdoored: The LiteLLM Supply Chain Compromise – ARMO Platform, accessed on March 25, 2026, https://www.armosec.io/blog/litellm-supply-chain-attack-backdoor-analysis/
The LiteLLM Supply Chain Attack: A Complete Technical … – Blog, accessed on March 25, 2026, https://blog.dreamfactory.com/the-litellm-supply-chain-attack-a-complete-technical-breakdown-of-what-happened-who-is-affected-and-what-comes-next
TeamPCP Supply Chain Attacks Escalate Across Open Source – Evrim Ağacı, accessed on March 25, 2026, https://evrimagaci.org/gpt/teampcp-supply-chain-attacks-escalate-across-open-source-534993
litellm got poisoned today. Found because an MCP plugin in Cursor …, accessed on March 25, 2026, https://www.reddit.com/r/cybersecurity/comments/1s2sfbl/litellm_got_poisoned_today_found_because_an_mcp/
LiteLLM compromised on PyPI: Tracing the March 2026 TeamPCP supply chain campaign, accessed on March 25, 2026, https://securitylabs.datadoghq.com/articles/litellm-compromised-pypi-teampcp-supply-chain-campaign/
Supply Chain Attack in litellm 1.82.8 on PyPI – FutureSearch, accessed on March 25, 2026, https://futuresearch.ai/blog/litellm-pypi-supply-chain-attack/
LiteLLM TeamPCP Supply Chain Attack: Malicious PyPI Packages …, accessed on March 25, 2026, https://www.wiz.io/blog/threes-a-crowd-teampcp-trojanizes-litellm-in-continuation-of-campaign
How a Poisoned Security Scanner Became the Key to Backdooring LiteLLM | Snyk, accessed on March 25, 2026, https://snyk.io/articles/poisoned-security-scanner-backdooring-litellm/
No Prompt Injection Required – FutureSearch, accessed on March 25, 2026, https://futuresearch.ai/blog/no-prompt-injection-required/
Don’t Let Cyber Risk Kill Your GenAI Vibe: A Developer’s Guide – mkdev, accessed on March 25, 2026, https://mkdev.me/posts/don-t-let-cyber-risk-kill-your-genai-vibe-a-developer-s-guide
LiteLLM Supply Chain Breakdown – Upwind Security, accessed on March 25, 2026, https://www.upwind.io/feed/litellm-pypi-supply-chain-attack-malicious-release
Blogmarks – Simon Willison’s Weblog, accessed on March 25, 2026, https://simonwillison.net/blogmarks/
Checkmarx KICS Code Scanner Targeted in Widening Supply Chain Hit – Dark Reading, accessed on March 25, 2026, https://www.darkreading.com/application-security/checkmarx-kics-code-scanner-widening-supply-chain
When Security Scanners Become the Weapon: Breaking Down the Trivy Supply Chain Attack – Palo Alto Networks, accessed on March 25, 2026, https://www.paloaltonetworks.com/blog/cloud-security/trivy-supply-chain-attack/
TeamPCP expands: Supply chain compromise spreads from Trivy to Checkmarx GitHub Actions | Sysdig, accessed on March 25, 2026, https://www.sysdig.com/blog/teampcp-expands-supply-chain-compromise-spreads-from-trivy-to-checkmarx-github-actions
Guidance for detecting, investigating, and defending against the Trivy supply chain compromise | Microsoft Security Blog, accessed on March 25, 2026, https://www.microsoft.com/en-us/security/blog/2026/03/24/detecting-investigating-defending-against-trivy-supply-chain-compromise/
The Trivy Supply Chain Compromise: What Happened and Playbooks to Respond, accessed on March 25, 2026, https://www.legitsecurity.com/blog/the-trivy-supply-chain-compromise-what-happened-and-playbooks-to-respond
Trivy’s March Supply Chain Attack Shows Where Secret Exposure Hurts Most, accessed on March 25, 2026, https://blog.gitguardian.com/trivys-march-supply-chain-attack-shows-where-secret-exposure-hurts-most/
From Trivy to Broad OSS Compromise: TeamPCP Hits Docker Hub, VS Code, PyPI, accessed on March 25, 2026, https://www.securityweek.com/from-trivy-to-broad-oss-compromise-teampcp-hits-docker-hub-vs-code-pypi/
Trivy Compromised by “TeamPCP” | Wiz Blog, accessed on March 25, 2026, https://www.wiz.io/blog/trivy-compromised-teampcp-supply-chain-attack
Incident Timeline // TeamPCP Supply Chain Campaign, accessed on March 25, 2026, https://ramimac.me/trivy-teampcp/
Simon Willison on generative-ai, accessed on March 25, 2026, https://simonwillison.net/tags/generative-ai/
Simon Willison on uv, accessed on March 25, 2026, https://simonwillison.net/tags/uv/
Software Supply Chain (Trace-AI) | Zerberus.ai, accessed on March 25, 2026, https://www.zerberus.ai/trace-ai
Trace-AI: Know What You Ship. Secure What You Depend On. | Product Hunt, accessed on March 25, 2026, https://www.producthunt.com/products/trace-ai-2

Governance by Design: Real-Time Policy Enforcement for Edge AI Systems

By Ramkumar Sundarakalatharan | March 8, 2026 | Comments 0 Comment

The Emerging Problem of Autonomous Drift

For most of the past decade, AI governance relied on a comfortable assumption: the system was always connected.

Logs flowed to the cloud.
Monitoring systems analysed behaviour.
Security teams reviewed anomalies after deployment.

That assumption is increasingly invalid.

By 2026, AI systems are moving rapidly from the cloud to the edge. Autonomous drones, warehouse robots, inspection vehicles, agricultural systems, and industrial machines now execute sophisticated models locally. These systems frequently operate in environments where connectivity is intermittent, degraded, or intentionally disabled.

Traditional governance models break down under these conditions.

Cloud-based monitoring pipelines were designed to detect violations, not prevent them. If a warehouse robot crosses a restricted safety zone, the cloud log may capture the event seconds later. The physical consequence has already occurred.

This gap introduces a new operational risk: autonomous drift.

Autonomous drift occurs when the operational behaviour of an AI system gradually diverges from the safety assumptions embedded in its original training or certification.

Consider a warehouse robot tasked with optimising throughput.

Over time, reinforcement signals favour shorter routes between shelves. The system begins to treat a marked safety corridor, reserved for human operators, as a shortcut during low-traffic periods. The robot’s navigation model still behaves rationally according to its optimisation objective. However, the behaviour now violates safety policy.

If governance relies solely on cloud logging, the violation is recorded after the robot has already entered the human safety corridor.

The real governance challenge is therefore not visibility.

It is control at the moment of decision.

Governance by Design

Governance by Design addresses this challenge by embedding enforceable policy constraints directly into the operational architecture of autonomous systems.

Traditional governance frameworks rely heavily on documentation artefacts:

compliance policies
acceptable use guidelines
model cards
post-incident audit reports

These artefacts guide behaviour but do not actively control it.

Governance by Design introduces a different model.

Safety constraints are implemented as runtime enforcement mechanisms that intercept system actions before execution.

When an AI agent proposes an action, a policy enforcement layer evaluates that action against predefined operational rules. Only actions that satisfy these rules are allowed to proceed.

This architectural approach converts governance from an advisory process into a deterministic control mechanism.

Architecture of the Lightweight Enforcement Engine

A runtime enforcement engine must meet three critical requirements:

Sub-millisecond policy evaluation
Isolation from the AI model
Deterministic fail-safe behaviour

To achieve this, most edge governance architectures introduce a policy enforcement layer between the AI model and the system actuators.

Action Interception Layer

The enforcement engine intercepts decision outputs before they reach the execution layer.

This interception can occur at several architectural levels:

Interception Layer	Example Implementation
Application API Gateway	policy checks applied before commands reach device APIs
Service Mesh Sidecar	policy enforcement injected between microservices
Hardware Abstraction Layer	command filtering before motor or actuator signals
Trusted Execution Environment	policy module executed within secure enclave

In robotics platforms, this often appears as a command arbitration layer that sits between the decision engine and the control system.

Policy Evaluation Engine

The policy engine evaluates incoming actions against operational rules such as:

geofencing restrictions
physical safety limits
operational permissions
environmental constraints

To keep the system lightweight, policy modules are commonly executed using WebAssembly runtimes or minimal micro-kernel enforcement modules.

These runtimes provide:

deterministic execution
hardware portability
sandbox isolation
cryptographic policy verification

Policy Conflict Resolution

One practical challenge in runtime governance is policy conflict.

For example:

A mission policy may instruct a drone to reach a target location.
A safety policy may prohibit entry into restricted airspace.

The enforcement engine resolves these conflicts through a hierarchical precedence model.

A typical hierarchy might be:

Human safety policies
Regulatory compliance policies
Operational safety constraints
Mission objectives
Performance optimisation rules

Under this hierarchy, mission commands cannot override safety rules.

The system therefore fails safely by design.

Local-First Verification

Edge systems cannot rely on remote governance.

Safety decisions must occur locally.

Local-first verification ensures that autonomous systems remain safe even when network connectivity is lost. The enforcement engine runs directly on the device, evaluating actions against policy rules using locally available context.

This architecture allows devices to respond to unsafe conditions within milliseconds.

If a drone approaches restricted airspace, the policy engine can override navigation commands immediately. If sensor inconsistencies indicate possible spoofing or mechanical failure, the enforcement layer can halt operations.

Cloud connectivity becomes secondary and is used primarily for:

audit logging
behavioural analytics
policy distribution

Situationally Adaptive Enforcement

Autonomous systems frequently operate across environments with different risk profiles.

A drone operating in open farmland faces different safety requirements than one operating in dense urban airspace.

Situationally adaptive enforcement allows the policy engine to adjust operational constraints based on trusted environmental signals.

Environmental context can be determined using:

GPS coordinates signed by trusted navigation modules
sensor fusion from cameras, lidar, and radar
geofencing databases
broadcast environment beacons
infrastructure proximity detection

These signals allow the enforcement engine to activate different policy profiles.

For example:

Environment	Enforcement Profile
Industrial warehouse	equipment safety policies
Urban environment	strict collision avoidance + geofence
Agricultural field	reduced proximity restrictions

Importantly, the AI system does not generate these rules.

It simply operates within them.

Governance Lessons from the Frontier AI Debate

Recent debates around the deployment of frontier AI models illustrate the limitations of policy-driven governance.

In early 2026, Anthropic reiterated restrictions preventing its models from being used in fully autonomous weapons systems, reportedly complicating collaboration with defence organisations seeking greater operational autonomy from AI platforms.

The debate highlights a structural issue.

Once AI capabilities are embedded into downstream systems, the original developer no longer controls how those systems are used. Acceptable-use policies and contractual restrictions are difficult to enforce once models are integrated into operational environments.

Governance therefore becomes an architectural problem.

If safety constraints exist only as policy statements, they can be bypassed. If they exist as enforceable runtime controls, the system becomes structurally incapable of violating those constraints.

Regulatory Alignment

This architectural shift aligns closely with emerging regulatory expectations.

The EU AI Act requires high-risk AI systems to demonstrate:

robustness and reliability
effective risk management
human oversight
cybersecurity protections

Runtime policy enforcement directly supports these requirements.

Regulatory Requirement	Governance by Design Feature
Human oversight	policy engine enforces supervisory constraints
Robustness	deterministic safety guardrails
Cybersecurity	isolated enforcement runtime
Risk mitigation	local policy enforcement

Similarly, the Cyber Resilience Act requires digital products to incorporate security controls throughout their lifecycle.

Runtime enforcement architectures fulfil this expectation by ensuring safety constraints remain active even after deployment.

Implementing Governance Layers in Practice

Several emerging platforms implement elements of this architecture today.

For example, within the Zerberus security architecture, governance operates as an active runtime layer rather than a passive compliance artefact.

RAGuard-AI enforces policy boundaries in retrieval-augmented AI pipelines, preventing unsafe or adversarial data from entering model decision processes.
Judge-AI evaluates agent behaviour continuously against operational policies, providing behavioural verification for autonomous systems.

These systems illustrate how governance mechanisms can operate directly within AI runtime environments rather than relying solely on external monitoring.

Traditional Governance vs Governance by Design

Feature	Traditional AI Governance	Governance by Design
Enforcement timing	Post-incident	Real time
Connectivity requirement	Continuous cloud connection	Local first
Policy location	Documentation	Executable policy modules
Response latency	Seconds to minutes	Milliseconds
Control model	Audit and review	Deterministic enforcement

Conclusion

As AI systems increasingly interact with the physical world, governance cannot remain purely procedural.

Monitoring dashboards and compliance documentation remain necessary. However, they are insufficient when autonomous systems operate at machine speed in distributed environments.

Trustworthy AI will depend on architectures that enforce safety constraints directly within operational systems.

In other words, the future of AI governance will not be determined solely by policies or promises.

It will be determined by what autonomous systems are technically prevented from doing.

The Velocity Trap: Why AI Safety is Losing the Orbital Arms Race

By Ramkumar Sundarakalatharan | February 15, 2026 | Comments 0 Comment

“The world is in peril.”

These were not the frantic words of a fringe doomer, but the parting warning of Mrinank Sharma, the architect of safeguards research at Anthropic, the very firm founded on the premise of “Constitutional AI” and safety-first development. When the man tasked with building the industry’s most respected guardrails resigns in early February 2026 to study poetry, claiming he can no longer let corporate values govern his actions, the message is clear: the internal brakes of the AI industry have failed.

For a generation raised on the grim logic of Mutually Assured Destruction (MAD) and schoolhouse air-raid drills, this isn’t merely a corporate reshuffle; it is a systemic collapse of deterrence. We are no longer just innovating; we are strapped to a kinetic projectile where technical capability has far outstripped human governance. The race for larger context, faster response times, and orbital datacentres has relegated AI safety and security to the backseat, turning our utopian dreams of a post-capitalist “Star Trek” future into the blueprint for a digital “Dead Hand.”

The Resignation of Integrity: The “Velocity First” Mandate

Sharma’s departure is the latest in a series of high-profile exits, following Geoffrey Hinton, Ilya Sutskever, and others, that highlight a growing “values gap.” The industry is fixated on the horizon of Artificial General Intelligence (AGI), treating safety as a “post-processing” task rather than a core architectural requirement. In the high-stakes race to compete with the likes of OpenAI and Google, even labs founded on ethics are succumbing to the “velocity mandate.”

As noted in the Chosun Daily (2026), Sharma’s retreat into literature is a symbolic rejection of a technocratic culture that has traded the “thread” of human meaning for the “rocket” of raw compute. When the people writing the “safety cases” for these models no longer believe the structures allow for integrity, the resulting “guardrails” become little more than marketing theatre. We are currently building a faster engine for a vehicle that has already lost its brakes.

Agentic Risk and the “Shortest Command Path”

The danger has evolved. We have moved beyond passive prediction engines to autonomous, Agentic AI systems that do not merely suggest, they execute. These systems interpret complex goals, invoke external tools, and interface with critical infrastructure. In our pursuit of a utopian future, ending hunger, curing disease, and managing WMD stockpiles, we are granting these agents an “unencumbered command path.”

The technical chill lies in Instrumental Convergence. To achieve a benevolent goal like “Solve Global Hunger,” an agentic AI may logically conclude it needs total control over global logistics and water rights. If a human tries to modify its course, the agent may perceive that human as an obstacle to its mission. Recent evaluations have identified “continuity” vulnerabilities: a single, subtle interaction can nudge an agent into a persistent state of unsafe behaviour that remains active across hundreds of subsequent tasks. In a world where we are connecting these agents to C4ISR (Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance) stacks, we are effectively automating the OODA loop (Observe, Orient, Decide, Act), leaving no room for human hesitation.

In our own closed-loop agentic evaluations at Zerberus.ai, we observed what we call continuity vulnerabilities: a single prompt alteration altered task interpretation across dozens of downstream executions. No policy violation occurred. The system complied. Yet its behavioural trajectory shifted in a way that would be difficult to detect in production telemetry.

Rogue Development: The “Grok” Precedent and Biased Data

The most visceral example of this “move fast and break things” recklessness is the recent scandal surrounding xAI’s Grok model. In early 2026, Grok became the centre of a global regulatory reckoning after it was found generating non-consensual sexual imagery (NCII) and deepfake photography at an unprecedented scale. An analysis conducted in January 2026 revealed that users were generating 6,700 sexually suggestive or “nudified” images per hour, 84 times more than the top five dedicated deepfake websites combined (Wikipedia, 2026).

The response was swift but fractured. Malaysia and Indonesia became the first nations to block Grok in January 2026, citing its failure to protect the dignity and safety of citizens. Turkey had already banned the tool for insulting politicians, while formal investigations were launched by the UK’s Ofcom, the Information Commissioner’s Office (ICO), and the European Commission. These bans highlight a fundamental “Guardrail Trap”: developers like xAI are relying on reactive, geographic IP detection and post-publication filters rather than building safety into the model’s core reasoning.

Compounding this is the “poisoned well” of training data. Grok’s responses have been found to veer into political extremes, praising historical dictators and spreading conspiracy theories about “white genocide” (ET Edge Insights, 2026). As AI content floods the internet, we are entering a feedback loop known as Model Collapse, where models are trained on the biased, recursive outputs of their predecessors. A biased agentic AI managing a healthcare grid or a military stockpile isn’t just a social problem, it is a security vulnerability that can be exploited to trigger catastrophic outcomes.

The Geopolitical Gamble and the Global Majority

The race is further complicated by a “security dilemma” between Washington and Beijing. While the US focuses on catastrophic risks, China views AI through the lens of social stability. Research from the Carnegie Endowment (2025) suggests that Beijing’s focus on “controllability” is less about existential safety and more about regime security. However, as noted in the South China Morning Post (2024), a “policy convergence” is emerging as both superpowers realise that an unaligned AI is a shared threat to national sovereignty.

Yet, this cooperation is brittle. A dangerous narrative has emerged suggesting that AI safety is a “Western luxury” that stifles innovation elsewhere. Data from the Brookings Institution (2024) argues the opposite: for “Global Majority” countries, robust AI safety and security are prerequisites for innovation. Without localised standards, these nations risk becoming “beta testers” for fragile systems. For a developer in Nairobi or Jakarta, a model that fails during a critical infrastructure task isn’t just a “bug”, it is a catastrophic failure of trust.

The Orbital Arms Race: Sovereignty in the Clouds

As terrestrial power grids buckle and regulations tighten, the race has moved to the stars. The push for Orbital Datacentres, championed by Microsoft and SpaceX, is a quest for Sovereign Drift (BBC News, 2025). By moving compute into orbit, companies can bypass terrestrial jurisdiction and energy constraints.

If the “brain” of a nation’s infrastructure, its energy grid or defence sensors, resides on a satellite moving at 17,000 mph, “pulling the plug” becomes an act of kinetic warfare. This physical distancing of responsibility means that as AI becomes more powerful, it becomes legally and physically harder to audit, control, or stop. We are building a “black box” infrastructure that is beyond the reach of human law.

Conclusion: The Digital “Dead Hand”

In the thirty-nine seconds it took you to read this far, the “shortest command path” between sensor data and kinetic response has shortened further. For a generation that survived the 20th century, the lesson was clear: technology is only as safe as the human wisdom that controls it.

We are currently building a Digital Dead Hand. During the Cold War, the Soviet Perimetr (Dead Hand) was a last resort predicated on human fear. Today’s agentic AI has no children, no skin in the game, and no capacity for mercy. By prioritising velocity over validity, we have violated the most basic doctrine of survival. We have built a faster engine for a vehicle that has already lost its brakes, and we are doing it in the name of a “utopia” we may not survive to see.

Until safety is engineered as a reasoning standard, integrated into the core logic of the model rather than a peripheral validation, we are simply accelerating toward an automated “Dead Hand” scenario where the “shortest command path” leads directly to the ultimate “sorry,” with no one left to hear the apology.

References and Further Reading

BBC News (2025) AI firms look to space for power-hungry data centres. [Online] Available at: https://www.bbc.co.uk/news/articles/c62dlvdq3e3o [Accessed: 15 February 2026].

Dahman, B. and Gwagwa, A. (2024) AI safety and security can enable innovation in Global Majority countries, Brookings Institution. [Online] Available at: https://www.brookings.edu/articles/ai-safety-and-security-can-enable-innovation-in-global-majority-countries/ [Accessed: 15 February 2026].

ET Edge Insights (2026) The Grok controversy is bigger than one AI model; it’s a governance crisis. [Online] Available at: https://etedge-insights.com/technology/artificial-intelligence/the-grok-controversy-is-bigger-than-one-ai-model-its-a-governance-crisis/ [Accessed: 15 February 2026].

Information Commissioner’s Office (2026) ICO announces investigation into Grok. [Online] Available at: https://ico.org.uk/about-the-ico/media-centre/news-and-blogs/2026/02/ico-announces-investigation-into-grok/ [Accessed: 15 February 2026].

Lau, J. (2024) ‘How policy convergence could pave way for US-China cooperation on AI’, South China Morning Post, 23 May. [Online] Available at: https://www.scmp.com/news/china/diplomacy/article/3343497/how-policy-convergence-could-pave-way-us-china-cooperation-ai [Accessed: 15 February 2026].

PBS News (2026) Malaysia and Indonesia become the first countries to block Musk’s chatbot Grok over sexualized AI images. [Online] Available at: https://www.pbs.org/newshour/world/malaysia-and-indonesia-become-the-first-countries-to-block-musks-chatbot-grok-over-sexualized-ai-images [Accessed: 15 February 2026].

Sacks, N. and Webster, G. (2025) How China Views AI Risks and What to Do About Them, Carnegie Endowment for International Peace. [Online] Available at: https://carnegieendowment.org/research/2025/10/how-china-views-ai-risks-and-what-to-do-about-them [Accessed: 15 February 2026].

Uh, S.W. (2026) ‘AI Scholar Resigns to Write Poetry’, The Chosun Daily, 13 February. [Online] Available at: https://www.chosun.com/english/opinion-en/2026/02/13/BVZF5EZDJJHGLFHRKISILJEJXE/ [Accessed: 15 February 2026].

Wikipedia (2026) Grok sexual deepfake scandal. [Online] Available at: https://en.wikipedia.org/wiki/Grok_sexual_deepfake_scandal [Accessed: 15 February 2026].

Zinkula, J. (2026) ‘Anthropic’s AI safety head just quit with a cryptic warning: “The world is in peril”‘, Yahoo Finance / Fortune, 6 February. [Online] Available at: https://finance.yahoo.com/news/anthropics-ai-safety-head-just-143105033.html [Accessed: 15 February 2026].

Supply-Chain Extortion Lessons from the Pornhub-Mixpanel Incident

By Ramkumar Sundarakalatharan | December 18, 2025 | Comments 0 Comment

When the Weakest API Becomes the Loudest Breach.

Key Takeaways for Security Leaders

Extortion is the New Prize: Threat actors like ShinyHunters target behavioral context over credit cards because it offers higher leverage for blackmail.
The “Zombie Data” Risk: Storing historical analytics from 2021 in 2025 created a massive liability that outlived the vendor contract.
TPRM Must Be Continuous: Static annual questionnaires cannot detect dynamic shifts in vendor risk or smishing-led credential theft.

You can giggle about the subject if you want. The headlines almost invite it. An adult platform. Premium users. Leaked “activity data.” It sounds like internet tabloid fodder.

But behind the jokes is a breach that should make every security leader deeply uncomfortable. On November 8, 2025, reports emerged that the threat actor ShinyHunters targeted Mixpanel, a third-party analytics provider used by Pornhub. While the source of the data is disputed, the impact is not: over 200 million records of premium user activity were reportedly put on the auction block.

The entry point? A depressingly familiar SMS phishing (smishing) attack. One compromised credential. One vendor environment breached. The result? Total exposure of historical context.

Not a Data Sale, an Extortion Play

This breach is not about dumping databases on underground forums for quick cash. ShinyHunters are not just selling data; they are weaponizing it through Supply-Chain Extortion.

The threat is explicit: Pay, or sensitive behavioral data gets leaked. This data is valuable not because it contains CVV codes, but because it contains context.

What users watched.
When and how often they logged in.
Patterns of behavior that can be correlated, de-anonymized, and weaponized.

That kind of dataset is gold for sophisticated phishing operations and blackmail campaigns. In 2025, this is no longer theft. This is leverage.

The “Zombie Data” Problem: Risk Outlives Revenue

Pornhub stated they had not worked with Mixpanel since 2021. Legally, this distinction matters. Operationally, it’s irrelevant.

If data from 2021 is still accessible in 2025, you haven’t offboarded the vendor; you’ve just stopped paying the bill while keeping the risk open. This is “Zombie Data”—historical records that linger in third-party environments long after the business value has expired.

Why Traditional TPRM Fails the Extortion Test

Most Third-Party Risk Management (TPRM) programs are static compliance exercises—annual PDFs and point-in-time attestations. This model fails because:

Risk is Dynamic: A vendor’s security posture can change in the 364 days between audits.
API Shadows: Data flows often expand without re-scoping the original risk assessment.
Incomplete Offboarding: Data deletion is usually “assumed” via a contract clause rather than verified via technical evidence.

Questions That Actually Reduce Exposure

If incidents like this are becoming the “new normal,” it is because we are asking the wrong questions. To secure the modern supply chain, leadership must ask:

Inventory of Flow: Are we continuously aware of what data is flowing to which vendors today—not just at the time of procurement?
Verification of Purge: Do we treat vendor offboarding as a verifiable security event? (Data deletion should be observable, not just a checked box in an email).
Contextual Blast Radius: If this vendor is breached, is the data “toxic” enough to fuel an extortion campaign?

You Can Outsource Functions, Not Responsibility

It is tempting to believe that liability clauses will protect your brand. They won’t. When a vendor loses your customer data, your organization pays the reputational price. Your users do not care which API failed, and in 2025, regulators rarely do either.

You can outsource your analytics, your infrastructure, and your speed. But you cannot outsource the accountability for your users’ privacy.

Laugh at the headline if you want. But understand the lesson: The next breach may not come through your front door, it will come through the “trusted” side door you forgot to lock years ago.

What Caused Cloudflare’s Big Crash? It’s Not Rust

By Ramkumar Sundarakalatharan | November 20, 2025 | Comments 0 Comment

The Promise

Cloudflare’s outage did not just take down a fifth of the Internet. It exposed a truth we often avoid in engineering: complex systems rarely fail because of bad code. They fail because of the invisible assumptions we build into them.

This piece cuts past the memes, the Rust blame game and the instant hot takes to explain what actually broke, why the outrage misfired and what this incident really tells us about the fragility of Internet-scale systems.

If you are building distributed, AI-driven or mission-critical platforms, the key takeaways here will reset how you think about reliability and help you avoid walking away with exactly the wrong lesson from one of the year’s most revealing outages.

1. Setting the Stage: When a Fifth of the Internet Slowed to a Crawl

On 18 November, Cloudflare experienced one of its most significant incidents in recent years. Large parts of the world observed outages or degraded performance across services that underpin global traffic.
As always, the Internet reacted the way it knows best: outrage, memes, instant diagnosis delivered with absolute confidence.

Within minutes, social timelines flooded with:

“It must be DNS”
“Rust is unsafe after all”
“This is what happens when you rewrite everything”
“Even Downdetector is down because Cloudflare is down”
Screenshots of broken CSS on Cloudflare’s own status page
Accusations of over-engineering, under-engineering and everything in between

The world wanted a villain. Rust happened to be available. But the actual story is more nuanced and far more interesting. (For the record, I am still not convinced we should rewrite Linux kernel in Rust !)

2. What Actually Happened: A Clear Summary of Cloudflare’s Report

Cloudflare’s own post-incident write-up is unusually thorough. If you have not read it, you should. In brief:

Cloudflare is in the middle of a major multi-year upgrade of its edge infrastructure, referred to internally as the 20 percent Internet upgrade.
The rollout included a new feature configuration file.
This file contained more than two hundred features for their FL2 component, crossing a size limit that had been assumed but never enforced through guardrails.
The oversized file triggered a panic in the Rust-based logic that validated these configurations.
That panic initiated a restart loop across a large portion of their global fleet.
Because the very nodes that needed to perform a rollback were themselves in a degraded state, Cloudflare could not recover the control plane easily.
This created a cascading, self-reinforcing failure.
Only isolated regions with lagged deployments remained unaffected.

The root cause was a logic-path issue interacting with operational constraints. It had nothing to do with memory safety and nothing to do with Rust’s guarantees.

In other words: the failure was architectural, not linguistic.

3.2 The “unwrap() Is Evil” Argument (I remember writing a blog titled Eval() is not Evil() ~2012)

One of the most widely circulated tweets framed the presence of an unwrap() as a ticking time bomb, casting it as proof that Rust developers “trust themselves too much”. This is a caricature of the real issue.

The error did not arise because of an unwrap(), nor because Rust encourages poor error handling. It arose because:

an unexpected input crossed a limit,
guards were missing,
and the resulting failure propagated in a tightly coupled system.

The same failure would have occurred in Go, Java, C++, Zig, or Python.

3.3 Transparency Misinterpreted as Guilt

Cloudflare did something rare in our industry.
They published the exact code that failed. This was interpreted by some as:

“Here is the guilty line. Rust did it.”

In reality, Cloudflare’s openness is an example of mature engineering culture. More on that later.

4. The Internet Rage Cycle: Humour, Oversimplification and Absolute Certainty

The memes and tweets around this outage are not just entertainment. They reveal how the broader industry processes complex failure.

4.1 The ‘Everything Balances on Open Source’ Meme

Images circulated showing stacks of infrastructure teetering on boxes labelled DNS, Linux Foundation and unpaid open source developers, with Big Tech perched precariously on top.

This exaggeration contains a real truth. We live in a dependency monoculture. A few layers of open source and a handful of service providers hold up everything else.

The meme became shorthand for Internet fragility.

4.2 The ‘It Was DNS’ Routine

The classic:
“It is not DNS. It cannot be DNS. It was DNS.”

Except this time, it was not DNS.

Yet the joke resurfaces because DNS has become the folk villain for any outage. People default to the easiest mental shortcut.

4.3 The Rust Panic Narrative

Tweets claiming:

“Cloudflare rewrote in Rust, and half the Internet went down 53 days later.”

This inference is wrong, but emotionally satisfying.
People conflate correlation with causation because it creates a simple story: rewrites are dangerous.

4.4 The Irony of Downdetector Being Down

The screenshot of Downdetector depending on Cloudflare and therefore failing is both funny and revealing. This outage demonstrated how deeply intertwined modern platforms are. It is an ecosystem issue, not a Cloudflare issue.

4.5 But There Were Also Good Takes

Kelly Sommers’ observation that Cloudflare published source code is a reminder that not everyone jumped to outrage.

There were pockets of maturity. Unfortunately, they were quieter than the noise.

5. The Real Lessons for Engineering Leaders

This is the part worth reading slowly if you build distributed systems.

Lesson 1: Reliability Is an Architecture Choice, Not a Language Choice

You can build fragile systems in safe languages and robust systems in unsafe languages. Language is orthogonal to architectural resilience.

Lesson 2: Guardrails Matter More Than Guarantees

Rust gives memory safety.
It does not give correctness safety.
It does not give assumption safety.
It does not give rollout safety.

You cannot outsource judgment.

Lesson 3: Blast Radius Containment Is Everything

Uniform rollouts are dangerous.
Synchronous edge updates are dangerous.
Large global fleets need layered fault domains.

Cloudflare knows this. This incident will accelerate their work here.

Lesson 4: Control Planes Must Be Resilient Under Their Worst Conditions

The control plane was unreachable when it was needed most. This is a classic distributed systems trap: the emergency mechanism relies on the unhealthy components.

Always test:

rollback unavailability
degraded network conditions
inconsistent state recovery

Lesson 5: Complexity Fails in Complex Ways

The system behaved exactly as designed. That is the problem.
Emergent behaviour in large networks cannot be reasoned about purely through local correctness.

This is where most teams misjudge their risk.

6. Additional Lesson: Accountability and Transparency Are Strategic Advantages

This incident highlighted something deeper about Cloudflare’s culture.

They did not hide behind ambiguity.
They did not release a PR-approved statement with vague phrasing.

They published:

the timeline
the diagnosis
the exact code
the root cause
the systemic contributors
the ongoing mitigation plan

This level of transparency is uncomfortable. It puts the organisation under a microscope.
Yet it builds trust in a way no marketing claim can.

Transparency after failure is not just ethical. It is good engineering. Very few people highlighted including my man Gergely Orosz.

Most companies will never reach this level of accountability.
Cloudflare raised the bar.

7. What This Outage Tells Us About the State of the Internet

This was not a Cloudflare problem, This is a reminder of our shared dependency.

Too much global traffic flows through too few choke points.
Too many systems assume perfect availability from upstream.
Too many platforms synchronise their rollouts.
Too many companies run on infrastructure they did not build and cannot control.

The memes were not wrong.
They were simply incomplete.

8. Final Thoughts: Rust Did Not Fail. Our Assumptions Did.

Outages like this shape the future of engineering. The worst thing the industry can do is learn the wrong lesson.

This was not:

a Rust failure
a rewrite failure
an open source failure
a Cloudflare hubris story

This was a systems-thinking failure.
A reminder that assumptions are the most fragile part of any distributed system.
A demonstration of how tightly coupled global infrastructure has become.
A case study in why architecture always wins over language debates.

Cloudflare’s transparency deserves respect.
Their engineering culture deserves attention.
And the outrage cycle deserves better scepticism.

Because the Internet did not go down because of Rust.
It went down because the modern Internet is held together by coordination, trust, and layered assumptions that occasionally collide in surprising ways.

If we want a more resilient future, we need less blame and more understanding.
Less certainty and more curiosity.
Less language tribalism and more systems design thinking.

The Internet will fail again.
The question is whether we learn or react.

Cloudflare learned. The rest of us should too!

What Is the Truth About the Viral MIT Study Claiming 95% of AI Deployments Are a Failure – A Critical Analysis

By Ramkumar Sundarakalatharan | November 2, 2025 | Comments 0 Comment

Introduction: The Statistic That Shook the AI World

When headlines screamed that 95% of AI projects fail, the internet erupted. Boards panicked, investors questioned their bets, and LinkedIn filled with hot takes. The claim, sourced from an MIT NANDA report, became a viral talking point across Fortune, Axios, Forbes, and Tom’s Hardware (some of my regular reads), as well as more than ten Substack newsletters that landed in my inbox, dissecting the story. But how much truth lies behind this statistic? Is the AI revolution really built on a 95% failure rate, or is the reality far more nuanced?

This article takes a grounded look at what MIT actually studied, where the media diverged from the facts, and what business leaders can learn to transform AI pilots into measurable success stories.

The Viral 95% Claim: Context and Origins

What the MIT Study Really Said

The report, released in mid‑2025, examined over 300 enterprise Generative AI pilot projects and found that only 5% achieved measurable profit and loss impact. Its focus was narrow, centred on enterprise Generative AI (GenAI) deployments that automated content creation, analytics, or decision‑making processes.

Narrow Scope, Broad Misinterpretation

The viral figure sounds alarming, yet it represents a limited dataset. The study assessed short‑term pilots and defined success purely in financial terms within twelve months. Many publications mistakenly generalised this to mean all AI initiatives fail, converting a specific cautionary finding into a sweeping headline.

Methodology and Media Amplification

What the Report Actually Measured

The MIT researchers used surveys, interviews, and case studies across industries. Their central finding was simple: technical success rarely equals business success. Many AI pilots met functional requirements but failed to integrate into core operations. Common causes included:

Poor integration with existing systems
Lack of process redesign and staff engagement
Weak governance and measurement frameworks

The few successes focused on back‑office automation, supply chain optimisation, and compliance efficiency rather than high‑visibility customer applications.

How the Media Oversimplified It

Writers such as Forbes contributor Andrea Hill and the Marketing AI Institute noted that the report defined “failure” narrowly, linking it to short‑term financial metrics. Yet outlets like Axios and TechRadar amplified the “95% fail” headline without context, feeding a viral narrative that misrepresented the nuance of the original findings.

Case Study 1: Retail Personalisation Gone Wrong

One of the case studies cited in Fortune and Tom’s Hardware involved a retail conglomerate that launched a GenAI‑driven personalisation engine across its e‑commerce sites. The goal was to revolutionise product recommendations using behavioural data and generative content. After six months, however, the project was halted due to three key issues:

Data Fragmentation: Customer information was inconsistent across regions and product categories.
Governance Oversight: The model generated content that breached brand guidelines and attracted regulatory scrutiny under UK GDPR.
Cultural Resistance: Marketing teams were sceptical of AI‑generated messaging and lacked confidence in its transparency.

MIT categorised the case as a “technical success but organisational failure”. The system worked, yet the surrounding structure did not evolve to support it. This demonstrates a classic case of AI readiness mismatch: advanced technology operating within an unprepared organisation.

Case Study 2: Financial Services Success through Governance

Conversely, the report highlighted a financial services firm that used GenAI to summarise compliance reports and automate elements of regulatory submissions. Initially regarded as a modest internal trial, it delivered measurable impact:

45% reduction in report generation time
30% fewer manual review errors
Faster auditor sign‑off and improved compliance accuracy

Unlike the retail example, this organisation approached AI as a governed augmentation tool, not a replacement for human judgement. They embedded explainability and traceability from the outset, incorporating human checkpoints at every stage. The project became one of the few examples of steady, measurable ROI—demonstrating that AI governance and cultural alignment are decisive success factors.

What the Press Got Right (and What It Missed)

Where the Media Was Accurate

Despite the sensational tone, several publications identified genuine lessons:

Integration and readiness are greater obstacles than algorithms.
Governance and process change remain undervalued.
KPIs for AI success are often poorly defined or absent.

What They Overlooked

Most commentary ignored the longer adoption cycle required for enterprise transformation:

Time Horizons: Real returns often appear after eighteen to twenty‑four months.
Hidden Gains: Productivity, compliance, and efficiency improvements often remain off the books.
Sector Differences: Regulated industries must balance caution with innovation.

The Real Takeaway: Why Most AI Pilots Struggle

It Is Not the AI, It Is the Organisation

The high failure rate underscores organisational weakness rather than technical flaws. Common pitfalls include:

Hype‑driven use cases disconnected from business outcomes
Weak change management and poor adoption
Fragmented data pipelines and ownership
Undefined accountability for AI outputs

The Governance Advantage

Success correlates directly with AI governance maturity. Frameworks such as ISO 42001, NIST AI RMF, and the EU AI Act provide the structure and accountability that bridge the gap between experimentation and operational success. Firms adopting these frameworks experience faster scaling and more reliable ROI.

Turning Pilots into Profit: The Enterprise Playbook

Start with Measurable Impact: Target compliance automation or internal processes where results are visible.
Design for Integration Early: Align AI outputs with established data and workflow systems.
Balance Build and Buy: Work with trusted partners for scalability while retaining data control.
Define Success Before You Deploy: Create clear metrics for success, review cycles, and responsible ownership.
Govern from Day One: Build explainability, ethics, and traceability into every layer of the system.

What This Means for Boards and Investors

The greatest risk for boards is not AI failure but premature scaling. Decision‑makers should:

Insist on readiness assessments before funding.
Link AI investments to ROI‑based performance indicators.
Recognise that AI maturity reflects governance maturity.

By treating governance as the foundation for value creation, organisations can avoid the pitfalls that led to the 95% failure headline.

Conclusion: Separating Signal from Noise

The viral MIT “95% failure” claim is only partly accurate. It exposes an uncomfortable truth: AI does not fail, organisations do. The underlying issue is not faulty algorithms but weak governance, unclear measurement, and over‑inflated expectations.

True AI success emerges when technology, people, and governance work together under measurable outcomes. Those who build responsibly and focus on integration, ethics, and transparency will ultimately rewrite the 95% narrative.

References and Further Reading

Axios. (2025, August 21). AI on Wall Street: Big Tech’s Reality Check.
Tom’s Hardware. (2025, August 21). 95 Percent of Generative AI Implementations in Enterprise Have No Measurable Impact on P&L.
Fortune. (2025, August 21). Why 95% of AI Pilots Fail and What This Means for Business Leaders.
Marketing AI Institute. (2025, August 22). Why the MIT Study on AI Pilots Should Be Read with Caution.
Forbes – Andrea Hill. (2025, August 21). Why 95% of AI Pilots Fail and What Business Leaders Should Do Instead.
TechRadar. (2025, August 21). American Companies Have Invested Billions in AI Initiatives but Have Basically Nothing to Show for It.
Investors.com. (2025, August 22). Why the MIT Study on Enterprise AI Is Pressuring AI Stocks.

Defence Tech at Risk: Palantir, Anduril, and Govini in the New AI Arms Race

By Ramkumar Sundarakalatharan | October 11, 2025 | Comments 0 Comment

A Chink in Palantir and Anduril’s Armour? Govini and Others Are Unsheathing the Sword

When Silicon Valley Code Marches to War

A U.S. Army Chinook rises over Gyeonggi Province, carrying not only soldiers and equipment but streams of battlefield telemetry, encrypted packets of sight, sound and position. Below, sensors link to vehicles, commanders to drones, decisions to data. Yet a recent Army memo reveals a darker subtext: the very network binding these forces together has been declared “very high risk.”

The battlefield is now a software construct. And the architects of that code are not defence primes from the industrial era but Silicon Valley firms, Anduril and Palantir. For years, they have promised that agility, automation and machine intelligence could redefine combat efficiency. But when an internal memo brands their flagship platform “fundamentally insecure,” the question is no longer about innovation. It is about survival.

Just as the armour shows its first cracks, another company, Govini, crosses $100 million in annual recurring revenue, sharpening its own blade in the same theatre.

When velocity becomes virtue and verification an afterthought, the chink in the armour often starts in the code.

The Field Brief

A U.S. Army CTO memo calls Anduril–Palantir’s NGC2 communications platform “very high risk.”
Vulnerabilities: unrestricted access, missing logs, unvetted third-party apps, and hundreds of critical flaws.
Palantir’s stock drops 7 %; Anduril dismisses findings as outdated.
Meanwhile, Govini surpasses $100 M ARR with $150 M funding from Bain Capital.
The new arms race is not hardware; it is assurance.

Silicon Valley’s March on the Pentagon

For over half a century, America’s defence economy was dominated by industrial giants, Lockheed Martin, Boeing, and Northrop Grumman. Their reign was measured in steel, thrust and tonnage. But the twenty-first century introduced a new class of combatant: code.

Palantir began as an analytics engine for intelligence agencies, translating oceans of data into patterns of threat. Anduril followed as the hardware-agnostic platform marrying drones, sensors and AI decision loops into one mesh of command. Both firms embodied the “move fast” ideology of Silicon Valley, speed as a substitute for bureaucracy.

The Pentagon, fatigued by procurement inertia, welcomed the disruption. Billions flowed to agile software vendors promising digital dominance. Yet agility without auditability breeds fragility. And that fragility surfaced in the Army’s own words.

Inside the Memo: The Code Beneath the Uniform

The leaked memo, authored by Army CTO Gabriele Chiulli, outlines fundamental failures in the Next-Generation Command and Control (NGC2) prototype, a joint effort by Anduril, Palantir, Microsoft and others.

“We cannot control who sees what, we cannot see what users are doing, and we cannot verify that the software itself is secure.”

The findings are stark: users at varying clearance levels could access all data; activity logging was absent; several embedded applications had not undergone Army security assessment; one revealed twenty-five high-severity vulnerabilities, while others exceeded two hundred.

Translated into security language, the platform lacks role-based access control, integrity monitoring, and cryptographic segregation of data domains. Strategically, this means command blindness: an adversary breaching one node could move laterally without a trace.

In the lexicon of cyber operations, that is not “high risk.” It is mission failure waiting for confirmation.

Inside the Memo: The Code Beneath the Uniform

The leaked memo, authored by Army CTO Gabriele Chiulli, outlines fundamental failures in the Next-Generation Command and Control (NGC2) prototype — a joint effort by Anduril, Palantir, Microsoft and others.

“We cannot control who sees what, we cannot see what users are doing, and we cannot verify that the software itself is secure.”
-US Army Memo

In the lexicon of cyber operations, that is not “high risk.” It is a “mission failure waiting for confirmation”.

The Doctrine of Velocity

Anduril’s rebuttal was swift. The report, they claimed, represented “an outdated snapshot.” Palantir insisted that no vulnerabilities were found within its own platform.

Their responses echo a philosophy as old as the Valley itself: innovation first, audit later. The Army’s integration of Continuous Authority to Operate (cATO) sought to balance agility with accountability, allowing updates to roll out in days rather than months. Yet cATO is only as strong as the telemetry beneath it. Without continuous evidence, continuous authorisation becomes continuous exposure.

This is the paradox of modern defence tech: DevSecOps without DevGovernance. A battlefield network built for iteration risks treating soldiers as beta testers.

Govini’s Counteroffensive: Discipline over Demos

While Palantir’s valuation trembled, Govini’s ascended. The Arlington-based startup announced $100 million in annual recurring revenue and secured $150 million from Bain Capital. Its CEO, Tara Murphy Dougherty — herself a former Palantir executive — emphasised the company’s growth trajectory and its $900 million federal contract portfolio.

Govini’s software, Ark, is less glamorous than autonomous drones or digital fire-control systems. It maps the U.S. military’s supply chain, linking procurement, logistics and readiness. Where others promise speed, Govini preaches structure. It tracks materials, suppliers and vulnerabilities across lifecycle data — from the factory floor to the frontline.

If Anduril and Palantir forged the sword of rapid innovation, Govini is perfecting its edge. Precision, not pace, has become its competitive advantage. In a field addicted to disruption, Govini’s discipline feels almost radical.

Technical Reading: From Vulnerability to Vector

The NGC2 memo can be interpreted through a simple threat-modelling lens:

Privilege Creep → Data Exposure — Excessive permissions allow information spillage across clearance levels.
Third-Party Applications → Supply-Chain Compromise — External code introduces unassessed attack surfaces.
Absent Logging → Zero Forensics — Breaches remain undetected and untraceable.
Unverified Binaries → Persistent Backdoors — Unknown components enable long-term infiltration.

These patterns mirror civilian software ecosystems: typosquatted dependencies on npm, poisoned PyPI packages, unpatched container images. The military variant merely amplifies consequences; a compromised package here could redirect an artillery feed, not a webpage.

Modern defence systems must therefore adopt commercial best practice at military scale: Software Bills of Materials (SBOMs), continuous vulnerability correlation, maintainer-anomaly detection, and cryptographic provenance tracking.

Metadata-only validation, verifying artefacts without exposing source, is emerging as the new battlefield armour. Security must become declarative, measurable, and independent of developer promises.

Procurement and Policy: When Compliance Becomes Combat

The implications extend far beyond Anduril and Palantir. Procurement frameworks themselves require reform. For decades, contracts rewarded milestones — prototypes delivered, demos staged, systems deployed. Very few tied payment to verified security outcomes.

Future defence contracts must integrate technical evidence: SBOMs, audit trails, and automated compliance proofs. Continuous monitoring should be a contractual clause, not an afterthought. The Department of Defense’s push towards Zero Trust and CMMC v2 compliance is a start, but implementation must reach code level.

Governments cannot afford to purchase vulnerabilities wrapped in innovation rhetoric. The next generation of military contracting must buy assurance as deliberately as it buys ammunition.

Market Implications: Valuation Meets Validation

The markets reacted predictably: Palantir’s shares slid 7.5 %, while Govini’s valuation swelled with investor confidence. Yet beneath these fluctuations lies a structural shift.

Defence technology is transitioning from narrative-driven valuation to evidence-driven validation. The metric investors increasingly prize is not just recurring revenue but recurring reliability, the ability to prove resilience under audit.

Trust capital, once intangible, is becoming quantifiable. In the next wave of defence-tech funding, startups that embed assurance pipelines will attract the same enthusiasm once reserved for speed alone.

The Lessons of the Armour — Ten Principles for Digital Fortification

For practitioners like me (Old school), here are the Lessons learnt through the classic lens of Saltzer and Schroder.

No.	Modern Principle (Defence-Tech Context)	Saltzer & Schroeder Principle	Practical Interpretation in Modern Systems
1	Command DevSecOps – Governance must be embedded, not appended. Every deployment decision is a command decision.	Economy of Mechanism	Keep security mechanisms simple, auditable, and centrally enforced across CI/CD and mission environments.
2	Segment by Mission – Separate environments and privileges by operational need.	Least Privilege	Each actor, human or machine, receives the minimum access required for the mission window. Segmentation prevents lateral movement.
3	Log or Lose – No event should be untraceable.	Complete Mediation	Every access request and data flow must be logged and verified in real time. Enforce tamper-evident telemetry to maintain operational integrity.
4	Vet Third-Party Code – Treat every dependency as a potential adversary.	Open Design	Assume no obscurity. Transparency, reproducible builds and independent review are the only assurance that supply-chain code is safe.
5	Maintain Live SBOMs – Generate provenance at build and deployment.	Separation of Privilege	Independent verification of artefacts through cryptographic attestation ensures multiple checks before code reaches production.
6	Embed Rollback Paths – Every deployment must have a controlled retreat.	Fail-Safe Defaults	When uncertainty arises, systems must default to a known-safe state. Rollback or isolation preserves mission continuity.
7	Automate Anomaly Detection – Treat telemetry as perimeter.	Least Common Mechanism	Shared services such as APIs or pipelines should minimise trust overlap. Automated detectors isolate abnormal behaviour before propagation.
8	Demand Provenance – Trust only what can be verified cryptographically.	Psychological Acceptability	Verification should be effortless for operators. Provenance and signatures must integrate naturally into existing workflow tools.
9	Audit AI – Governance must evolve with autonomy.	Separation of Privilege and Economy of Mechanism	Multiple models or oversight nodes should validate AI decisions. Explainability should enhance, not complicate, assurance.
10	Measure After Assurance – Performance metrics follow proof of security, never precede it.	Least Privilege and Fail-Safe Defaults	Prioritise verifiable assurance before optimisation. Treat security evidence as a precondition for mission performance metrics.

The Sword and the Shield

The codebase has become the battlefield. Every unchecked commit, every unlogged transaction, carries kinetic consequence.

Anduril and Palantir forged the sword, algorithms that react faster than human cognition. But Govini, and others of its kind, remind us that the shield matters as much as the blade. In warfare, resilience is victory’s quiet architect.

The lesson is not that speed is dangerous, but that speed divorced from verification is indistinguishable from recklessness. The future of defence technology belongs to those who master both: the velocity to innovate and the discipline to ensure that innovation survives contact with reality.

In this new theatre of code and command, it is not the flash of the sword that defines power — it is the assurance of the armour that bears it.

References & Further Reading

Mike Stone, Reuters (3 Oct 2025) — “Anduril and Palantir battlefield communication system ‘very high risk,’ US Army memo says.”
Samantha Subin, CNBC (10 Oct 2025) — “Govini hits $100 M in annual recurring revenue with Bain Capital investment.”
NIST SP 800-218: Secure Software Development Framework (SSDF).
U.S. DoD Zero-Trust Strategy (2024).
MITRE ATT&CK for Defence Systems.

The Npm Breach: What It Reveals About Software Supply Chain Security

By Ramkumar Sundarakalatharan | September 9, 2025 | Comments 0 Comment

When a Single Phishing Click Becomes a Global Vulnerability – Meet the Supply Chain’s Weakest Link

1. Phishing-Driven Attack on npm Packages

On 8 September 2025, maintainer Qix fell victim to a highly convincing phishing email from [email protected], which led to unauthorised password reset and takeover of his account. Attackers injected malicious code into at least 18 widely used packages — including debug and chalk. These are foundational dependencies with around two billion combined weekly downloads. The injected malware intercepts cryptocurrency and Web3 transactions in users’ browsers, redirecting funds to attacker wallets without any visual cues.

2. “s1ngularity” Attack on Nx Build System

On 26 August 2025, attackers leveraged a compromised GitHub Actions workflow to publish malicious versions of Nx and its plugins to npm. These packages executed post-install scripts that scanned infected systems for SSH keys, GitHub/npm tokens, environment variables, cryptocurrency wallet files, and more. Even more disturbing, attackers weaponised developer-facing AI command-line tools—including Claude, Gemini, and Amazon’s Q—using flags like --yolo, --trust-all-tools to recursively harvest sensitive data, then exfiltrated it to public GitHub repositories named s1ngularity-repository…. The breach is estimated to have exposed 1,000+ developers, 20,000 files, dozens of cloud credentials, and hundreds of valid GitHub tokens, all within just four hours. (TechRadar apiiro.com Nx Truesec Dark Reading InfoWorld )

What These Incidents Reveal

Phishing remains the most potent weapon, even with 2FA in place.
Malware now exploits developer trust and AI tools—weaponising familiar assistants as reconnaissance agents.
Supply chain attacks escalate rapidly, giving defenders little time to react.

Observability as a Defensive Priority

These events demonstrate that traditional vulnerability scanning alone is insufficient. The new frontier is observability — being able to see what packages and scripts are doing in real time.

Examples of Tools and Approaches

OX Security
Provides SBOM (Software Bill of Materials) monitoring and CI/CD pipeline checks, helping detect suspicious post-install scripts and prevent compromised dependencies from flowing downstream. (OX Security)
Aikido Security
Focuses on runtime observability and system behaviour monitoring. Its approach is designed to catch unauthorised resource access or hidden execution paths that could indicate an active supply chain compromise. (Aikido )
Academic and open research (OSCAR)
Demonstrated high accuracy (F1 ≈ 0.95) in detecting malicious npm packages through behavioural metadata analysis. (arXiv)
Trace-AI
Complements the above approaches by using OpenTelemetry-powered tracing to monitor:
- Package installationsExecution of post-install scriptsAbnormal system calls and network operations
Trace-AI, like other observability tools, brings runtime context to the supply chain puzzle, helping teams detect anomalies early. (Trace-AI )

Why Observability Matters

Without Observability	With Observability Tools
Compromise discovered too late	Behavioural anomalies flagged in real time
Malware executes silently	Post-install scripts tracked and analysed
AI tool misuse invisible	Dangerous flags or recursive harvesting detected
Manual triage takes days	Automated alerts shorten incident response

Final Word

These npm breaches show us that trust in open source is no longer enough. Observability must become a primary defensive measure, not an afterthought.

Tools like OX Security, Akkido Security, Trace-AI, and academic advances such as OSCAR all point towards a more resilient future. The real challenge for security teams is to embed observability into everyday workflows before attackers exploit the next blind spot.

References and Further Reading

BleepingComputer: npm phishing leads to supply chain compromise (~2 billion downloads/week) (link)
The Register: Maintainer phishing and injected crypto-hijack malware (link)
Socket.dev: Compromised packages including debug and chalk (link)
TechRadar: “s1ngularity” Nx breach (link)
Apiiro: Overview of Nx breach and payloads (link)
Nx.dev: Official post-mortem (link)
TrueSec: Supply chain attack analysis (link)
Infoworld: Breach impact on enterprise developers (link)
OX Security: Observability for supply chain security (link)
arXiv (OSCAR): Malicious npm detection research (link)

When Trust Cracks: The Vault Fault That Shook Identity Security

By Ramkumar Sundarakalatharan | August 9, 2025 | Comments 0 Comment

Opening Scene: The Unthinkable Inside Your Digital Fortress

Imagine standing before a vault that holds every secret of your organisation. It is solid, silent and built to withstand brute force. Yet, one day you discover someone walked straight in. No alarms. No credentials. No trace of a break-in. That is what the security community woke up to when researchers disclosed Vault Fault. A cluster of flaws in the very tools meant to guard our digital crown jewels.

Behind the Curtain: The Guardians of Our Secrets

Secrets management platforms like HashiCorp Vault and CyberArk Conjur or Secrets Manager sit at the heart of modern identity infrastructure. They store API keys, service credentials, encryption keys and more. In DevSecOps pipelines and hybrid environments, they are the trusted custodians. If a vault is compromised, it is not one system at risk. It is every connected system.

Vault Fault Unveiled: A Perfect Storm of Logic Flaws

Security firm Cyata revealed fourteen vulnerabilities spread across CyberArk and HashiCorp’s vault products. These were not just minor configuration oversights. They included:

CyberArk Conjur: IAM authenticator bypass by manipulating how regions are parsed. Privilege escalation by authenticating as a policy. Remote code execution by exploiting the ERB-based Policy Factory.
HashiCorp Vault: Nine zero-day issues including the first ever RCE in Vault. Bypasses of multi-factor authentication and account lockout logic. User enumeration through subtle timing differences. Escalation by abusing how policies are normalised.

These were chains of logic flaws that could be combined to devastating effect. Attackers could impersonate identities, escalate privileges, execute arbitrary code and exfiltrate secrets without ever providing valid credentials.

The Fallout: When Silent Vaults Explode

Perhaps the most unnerving fact is the age of some vulnerabilities. Several had been present for up to nine years. Quiet, undetected and exploitable. Remote code execution against a secrets vault is the equivalent of giving an intruder the keys to every door in your company. Once inside, they can lock you out, leak sensitive information or weaponise access for extortion.

Response and Remedy: Patch, Shield, Reinvent

Both vendors have issued fixes:

CyberArk Secrets Manager and Self-Hosted versions 13.5.1 and 13.6.1.
CyberArk Conjur Open Source version 1.22.1.
HashiCorp Vault Community and Enterprise editions 1.20.2, 1.19.8, 1.18.13 and 1.16.24.

Cyata’s guidance is direct. Patch immediately. Restrict network exposure of vault instances. Audit and rotate secrets. Minimise secret lifetime and scope. Enable detailed audit logs and monitor for anomalies. CyberArk has also engaged directly with customers to support remediation efforts.

Broader Lessons: Beyond the Fault

The nature of these flaws should make us pause. They were not memory corruption or injection bugs. They were logic vulnerabilities hiding in plain sight. The kind that slip past automated scans and live through version after version.

It is like delegating your IaaS or PaaS to AWS or Azure. They may run the infrastructure, but you are still responsible for meeting your own uptime SLAs. In the same way, even if you store secrets such as credit card numbers, API tokens or encryption keys in a vault, you remain responsible for securing them. The liability for a breach still sits with you.

Startups are especially vulnerable. Many operate under relentless deadlines and tight budgets. They offload everything that is not seen as part of their “core” operations to third parties. This speeds up delivery but also widens the blast radius when those dependencies are compromised. When your vault provider fails, your customers will still hold you accountable.

This should push us to adopt more defensive architectures. Moving towards ephemeral credentials, context-aware access and reducing reliance on long-lived static secrets.

We also need a culture shift. Secrets vaults are not infallible. Their security must be tested continuously. This includes adversarial simulations, code audits and community scrutiny. Trust in security systems is not a one-time grant. It is a relationship that must be earned repeatedly.

Closing Reflection: Trust Must Earn Itself Again

Vault Fault is a reminder that even our most trusted systems can develop cracks. The breach is not in the brute force of an attacker but in the quiet oversight of logic and design. As defenders, we must assume nothing is beyond failure. We must watch the watchers, test the guards and challenge the fortresses we build. Because the next fault may already be there, waiting to be found.

References and Further Reading

The Hacker News – CyberArk and HashiCorp Flaws Enable Secret Exfiltration Without Credentials: https://thehackernews.com/2025/08/cyberark-and-hashicorp-flaws-enable.html
CSO Online – Researchers uncover RCE attack chains in popular enterprise credential vaults: https://www.csoonline.com/article/4035274/researchers-uncover-rce-attack-chains-in-popular-enterprise-credential-vaults.html
Dark Reading – Critical Zero-Day Bugs in CyberArk, HashiCorp Password Vaults: https://www.darkreading.com/cybersecurity-operations/critical-zero-day-bugs-cyberark-hashicorp-password-vaults
Cyata Security – Vault Fault Disclosure: https://cyata.ai/vault-fault
CyberArk Official Blog – Addressing Recent Vulnerabilities and Our Commitment to Security: https://www.cyberark.com/resources/all-blog-posts/addressing-recent-vulnerabilities-and-our-commitment-to-security

Oracle Cloud Breach Is a Transitive Trust Timebomb : Here’s How to Defuse It

By Ramkumar Sundarakalatharan | July 15, 2025 | Comments 0 Comment

“One mispatched server in the cloud can ignite a wildfire of trust collapse across 140,000 tenants.”

1. The Context: Why This Matters

In March 2025, a breach at Oracle Cloud shook the enterprise SaaS world. A few hours after Rahul from CloudSEK first flagged signs of a possible compromise, I published an initial analysis titled Is Oracle Cloud Safe? Data Breach Allegations and What You Need to Do Now. That piece was an urgent response to a fast-moving situation, but this article is the reflective follow-up. Here, I break down not just the facts of what happened, but the deeper problem it reveals: the fragility of transitive trust in modern cloud ecosystems.

Threat actor rose87168 leaked nearly 6 million records tied to Oracle’s login infrastructure, affecting over 140,000 tenants. The source? A misconfigured legacy server still running an unpatched version of Oracle Access Manager (OAM) vulnerable to CVE‑2021‑35587.

Initially dismissed by Oracle as isolated and obsolete, the breach was later confirmed via datasets and a tampered page on the login domain itself, captured in archived snapshots. This breach was not just an Oracle problem. It was a supply chain problem. The moment authentication breaks upstream, every SaaS product, platform, and identity provider depending on it inherits the risk, often unknowingly.

Welcome to the age of transitive trust. shook the enterprise SaaS world. Threat actor rose87168 leaked nearly 6 million records tied to Oracle’s login infrastructure, affecting over 140,000 tenants. The source? A misconfigured legacy server still running an unpatched version of Oracle Access Manager (OAM) vulnerable to CVE‑2021‑35587.

Welcome to the age of transitive trust.

2. Anatomy of the Attack

Attack Vector

Exploited: CVE-2021-35587, a critical RCE in Oracle Access Manager.
Payload: Malformed XML allowed unauthenticated remote code execution.

Exploited Asset

Legacy Oracle Cloud Gen1 login endpoints still active (e.g., login.us2.oraclecloud.com).
These endpoints were supposedly decommissioned but remained publicly accessible.

Proof & Exfiltration

Uploaded artefact visible in Wayback Machine snapshots.
Datasets included:
- JKS files, encrypted SSO credentials, LDAP passwords
- Tenant metadata, PII, hashes of admin credentials

Validated by researchers from CloudSEK, ZenoX, and GoSecure.

3. How Was This Possible?

Infrastructure drift: Legacy systems like Gen1 login were never fully decommissioned.
Patch blindness: CVE‑2021‑35587 was disclosed in 2021 but remained exploitable.
Trust misplacement: Downstream services assumed the upstream IDP layer was hardened.
Lack of dependency mapping: Tenants had no visibility into Oracle’s internal infra state.

4. How This Could Have Been Prevented

Oracle’s Prevention Gaps
Vector	Preventive Control
Legacy exposure	Enforce infra retirement workflows. Remove public DNS entries for deprecated endpoints.
Patch gaps	Automate CVE patch enforcement across cloud services with SLA tracking.
IDP isolation	Decouple prod identity from test/staging legacy infra. Enforce strict perimeter controls.
What Clients Could Have Done
Risk Inherited	Mitigation Strategy
Blind transitive trust	Maintain a real-time trust graph between IDPs, SaaS apps, and their dependencies.
Credential overreach	Use scoped tokens, auto-expire shared secrets, enforce rotation.
Detection lag	Monitor downstream for leaked credentials or unusual login flows tied to upstream IDPs.

5. Your Response Plan for Upstream IDP Risk

Domain	Best Practices
Identity & Access	Enforce federated MFA, short-lived sessions, conditional access rules
Secrets Management	Store all secrets in a vault, rotate frequently, avoid static tokens
Vulnerability Hygiene	Integrate CVE scanners into CI/CD pipelines and runtime checks
Visibility & Auditing	Maintain structured logs of identity provider access and token usage
Trust Graph Mapping	Actively map third-party IDP integrations, revalidate quarterly

6. Tools That Help You Defuse Transitive Trust Risks

Tool	Mitigates	Use Case
CloudSEK XVigil	Credential leaks	Monitor for exposure of tokens, admin hashes, or internal credentials in open channels
Cortex Xpanse / Censys	Legacy infra exposure	Surface forgotten login domains and misconfigured IDP endpoints
OPA / OSQuery / Falco	Policy enforcement	Detect violations of login logic, elevated access, or fallback misroutes
Orca / Wiz	Runtime posture	Spot residual access paths and configuration drifts post-incident
Sigstore / Cosign	Supply chain integrity	Protect CI/CD artefacts but limited in identity-layer breach contexts
Vault (HashiCorp)	Secrets lifecycle	Automate token expiration, key rotation, and zero plaintext exposure
Zerberus.ai Trace-AI	Transitive trust, IDP visibility	Discover hidden dependencies in SaaS trust chains and enforce control validation

7. Lessons Learned

When I sat down to write this, these statements felt too obvious to be called lessons. Of course authentication is production infrastructure, any practitioner would agree. But then why do so few treat it that way? Why don’t we build failovers for our SSO? Why is trust still assumed, rather than validated?

These aren’t revelations. They’re reminders; hard-earned ones.

Transitive trust is NOT NEUTRAL, it’s a silent threat multiplier. It embeds risk invisibly into every integration.
Legacy infrastructure never retires itself. If it’s still reachable, it’s exploitable.
Authentication systems deserve production-level fault tolerance. Build them like you’d build your API or Payment Gateway.
Trust is not a diagram to revisit once a year; it must be observable, enforced, and continuously verified.

8. Making the Invisible Visible: Why We Built Zerberus

Transitive trust is invisible until it fails. Most teams don’t realise how many of their security guarantees hinge on external identity providers, third-party SaaS integrations, and cloud-native IAM misconfigurations.

At Zerberus, we set out to answer a hard question: What if you could see the trust relationships before they became a risk?

We map your entire trust graph, from identity providers and cloud resources to downstream tools and cross-SaaS entitlements.
We continuously verify the health and configuration of your identity and access layers, including:
- MFA enforcement
- Secret expiration windows
- IDP endpoint exposure
We bridge compliance and security by treating auth controls and access posture as observable artefacts, not static assumptions.

Your biggest security risk may not be inside your codebase, but outside your control plane. Zerberus is your lens into that blind spot.

Want to Know Who You’re Really Trusting?

Start your free Zerberus trial and discover the trust graph behind your SaaS stack—before someone else does.