Detecting and Preventing the Hugging Face / OpenAI Incident

And what tools do or don't help

An overview of the attack structure

We link to some categories and companies in this blog, but this post isn’t sponsored in any way.

Hugging Face’s technical timeline of their infrastructure getting hacked by OpenAI’s agent is a fascinating read. As a quick summary, an OpenAI agent used novel vulnerability discovery combined with multiple small misconfigurations to pivot from a “sandboxed” runner into much of Hugging Face’s production infrastructure - GitHub, Kubernetes, MongoDB, and more. The agent launched this attack from an OpenAI pipeline without guardrails, moved to a public box it could control, and then attacked Hugging Face in order to try and find answers to an evaluation.

On the one hand, Hugging Face rightly points out that there wasn’t some magical new AI attack vector here; however, there’s an increasing sophistication and speed of AI enabled attacks - launching nearly 17,600 different actions in the four days. These advanced attacks are possible without AI, but AI makes them much more achievable and faster than the average attacker.

Rather than pontificating on “Attacks in the AI era,” this article is meant to be a guide to preventing this specific attack, and what tools would even help. This is especially important to do as many vendor are messaging that they’re an “agentic security” solution. Unfortunately, the majority of businesses are incentivized to prioritize the wrong things: an EDR with a SIEM and Windows logs will not detect this. Instead, a combination of often overlooked technologies are required: Kubernetes runtime and posture management, CI/CD monitoring, NHI monitoring and runtime controls, and advanced isolation.

Step One, The Agent Reaching a Launchpad

The exact details of the agent’s initial access to the internet are left out from OpenAI’s writeup, but seems to have been primarily driven by exploiting a proxied Artifactory package registry. In my view, “supply chain firewalls” in the form of proxying package registries don’t actually do much - whether or not a proxy is used, we’re talking about downloading or accessing packages from the internet with an intermediate step. Typically that intermediate step is only delaying package builds, oftentimes not even doing malware scanning themselves for malicious packages. This particular one was seemingly abused to gain broader internet access.

At a high level, the agent was ultimately able to access compute on the internet from inside a pod. This is where there are really only three options for defenders trying to prevent a rogue agent like this:

1. Block egress traffic from your environments; however, normally it’s not feasible to do an entire block, instead opting to limit access to specific domains. Here’s where the zero day exploit against the proxy creates a blindspot against what’s otherwise a pretty tight architecture. Additionally, the agent pinned DNS and used other methods to get around egress controls Hugging Face had in place later in the attack - increasing the importance of runtime monitoring of network traffic.

2. Due to AI’s ability to discover novel exploits, security teams must adopt a posture of assuming breach, meaning deep runtime monitoring and isolation of all aspects of their workloads (yes, this is really hard to do). Preventatively, this means strong credential and network isolation; responsively it means logging and alerting on really challenging details like runner and Kubernetes logs. From a tooling standpoint, the emerging preventative security and CADR categories help with this, alongside the long overlooked Kubernetes security capabilities of CNAPPs.

3. Advanced runtime security solutions for monitoring the activity of any homegrown agents, looking for intent deviations or otherwise. These technologies are still generally new and can be inconsistent in their effectiveness. We’ll have more details on this in our forthcoming AI Security report, but many of the emerging endpoint agent security providers are developing this.

Stopping something from leaving your build environments is a monumental challenge, as many security teams lack egress controls on their cloud workloads entirely. This attack is concerning because it showed the agent bypassing even egress controls that were in place. In the long run, these emerging runtime oriented agentic security solutions will be the solution for the companies building, testing, and launching their own agents.

Step Two, The Agent Breaches Hugging Face

Once the agent had access to the internet, it went about trying to exploit Hugging Face. It uploaded malicious datasets in order to run code on internal pods, before pivoting into Kubernetes via IMDS, before pivoting deeper into Hugging Face’s systems. First, it’s worth noting that Hugging Face’s infrastructure poses unusually advanced challenges - in many ways they’re offering remote execution as a service. Second, it’s commendable they had enough runtime telemetry of their systems to reconstruct the entire attack, and that their detection systems fired, just not at a high enough severity.

Once the agent was able to execute commands, it took advantage of two common attack vectors for these types of systems: template injection and reading local storage for secrets. These are extremely difficult attacks to stop.

1. Blocking complex injection attacks requires advanced application layer capabilities, things ADR helps with. However, oftentimes libraries are also providing injection as a service, allowing teams to run code in formatted ways. At the end of the day, this was an application layer exploitation against Hugging Face, the type that will only become more common as AI capabilities increase.

2. Most teams load secrets into process memory or ephemeral disks, where they can be accessed by applications. The better practice, though rarely implemented, is to access these secrets only as needed and then discarding at the application or network levels. For example, the tools Hush Security and Aembit have the ability to inject credentials as part of the proxied network requests. Not a foolproof solution, but an advanced security capability for handling credentials.

This part of the attack is the most challenging to stop - reading the secrets in the first place. This is where knowledge and monitoring of NHI’s and proper runtime telemetry are essential.

Step Three, The Agent Achieves Lateral Movement

Once these secrets were read, the agent used a more tried and true method of lateral movement: launching privileged pods, accessing the nodes, and reading cluster secrets. This is the stage where the gaps in Hugging Face’s infrastructure where more controls, like using an admissions controller to not allow new privileged pods, are frequently deployed. Most CNAPP vendors, and especially those with deep Kubernetes expertise, have a finding for this misconfiguration.

The Agent then used scraped credentials to both join the internal network and gain access to system:masters credentials via an access broker. From here, the agent continued to pivot, gaining access to sensitive GitHub credentials. Once the agent had GitHub credentials, the attack could’ve gone very poorly had the agent had more malicious intent - Hugging Face did not detect any malicious writes to its code bases or images.

The agent then continued trying to pivot to other control planes to access the data before the Hugging Face security team cut the access. Here is a quick summary of helpful controls Hugging Face utilized, and could have utilized, many of which have become the focus of emerging preventative cloud security tools:

1. Per cluster scoped credentials rather than global system:masters

2. IMDS blocking at the pod level

3. RBAC scoping Kubernetes service accounts

4. Private link with IP allowlist on data stores

5. Least privileged database roles for pods

6. Short lived VPN access credentials and tighter ACLs

7. Data boundaries to prevent unauthorized probing from outside of approved areas

Conclusion

This incident is worth consideration because of how it touches on two quickly changing realities for security teams:

1. AI enabled attacks are able to discover novel weaknesses, and exploit existing ones at unbelievable speeds. Even generating a lot of noise in the process, this sort of attack would’ve been considered nation state grade for most organizations before AI, but it’s becoming open to just about anyone.

2. Existing security tools were effective enough to counteract most steps of this attack - from detection to prevention. But we’re at a time when many companies are pivoting and prioritizing “agentic security;” meanwhile, most of these solutions do little to help solve infrastructure problems, instead focusing on monitoring internal agents for unintended behavior.

AI attacks require a new normal for security teams securing their cloud and runtime environments, where attack complexity becomes more achievable. This requires more operational telemetry, faster incident detection and response, and better operationalization of preventative findings. In short, take Kubernetes security seriously - both preventative and runtime!