Shai Hulud 2.0: Analysis and Community Resources

We've complied all the best tools, prevention methods and articles for responding to Shai Hulud 2.0 and share our analysis so teams can understand the impact

A proliferation of various vendor resources have been released on Shai Hulud v2, from AI generated marketing pieces to thoughtful independent research. This is an attempt to collect all of the most useful information in one place.

Quick Summary: A massive number of code libraries were compromised, most prominently from Zapier, ENS Domains, PostHog, and Postman. If you ran the infected package, most secrets on the device that ran the package install can be considered compromised, and a persistent RCE was potentially setup on your device in the form of a GitHub runner. Secrets were posted via public GitHub repos which are being taken down by GitHub, with secrets owners being notified in the cases where this data is available.

Responder Tools:

IOCs:
These are collections of infected package names and versions, as well as SHAs of the malware files that get left behind on endpoints that run the install.

1. Wiz GitHub CSV, their article also has the SHA1 for the infected files.
2. DataDog’s GitHub CSV
3. Koi CSV which has individual lines for when multiple package versions are there. This link also contains SHA256 for the infected JS files for threat hunting
4. Several vendor blogs have comprehensive bulleted lists: Aikido, Semgrep, Socket

Scanners (Packages):
In my opinion, it’s best to rely on your SCA tool or even a code search to check for impacted versions before scanning for them all over your environment. Second, I’d look for the SHAs included in the Wiz and Koi articles for the infected files to see if I was impacted rather than trying to hunt for the specific versions in my packages. Finally, I have not personally verified the effectiveness of these scanners.

1. ndaal_public_detect
2. Phoenix Security Scanner
3. Jaime - Scanner and Package.json uploader
4. sngular scanner

Compromised Secrets Checker:
These companies say they gathered the public secrets exposed, decoded them, and added them into their exposed secrets databases. While I haven’t verified the comprehensiveness of what was added and always feel icky pasting secrets into public web forms, I can vouch for these being real companies that you’re yeeting your secrets into:

1. Entro Are my Secrets Out?
2. GitGuardian Has my Secret Leaked?

Free or Open Source NPM Package Safety Tools

1. Using pnpm with cooldowns, blocking postinstall scripts, and trust policies
2. Aikido’s open source safechain, which checks against their malware feed before installing a package.
3. NPQ - with a variety of open source health factor checks available, including a cooldown period setting and checking for pre and post install scripts.

James Thoughts:

1. Saying “use fewer dependencies” is an incomplete answer. It’s a great aspirational goal, but dependencies are the reality of modern software development, and “just using less of them,” isn’t the reality. Let’s especially consider that several of these compromised dependencies were tied to commercial offerings - where you wouldn’t be able to build them in house anyways.

2. We can be grateful to live in a new world where more than one vendor is detecting upstream malware. As common as the headline “X vendor discovers massive ongoing supply chain attack!” is, the reality is that multiple vendors are discovering this stuff in parallel. While as early as two years ago only a couple of companies like Phylum (acquired by Veracode) and Socket were monitoring for these attacks, now a plethora of vendors are doing upstream monitoring with various tools from AI to runtime build monitoring - such as Aikido, Koi, SourceCodeRED, and StepSecurity.

3. Similarly, there are now many open source “protection” tools out there - with one of the first ones being Liran’s NPQ. I’m not listing them all here because they all have the same challenge: developer adoption requires conscious effort and enforcement. Liran’s tool was released 8 years ago, your SCA scanning vendor may offer their own.

4. The only way you can absolutely prevent these attacks is version pinning. The only way to detect these attacks is runtime monitoring of developer endpoints, build systems, and production systems. The other preventative measures are more defense in depth ways to protect your build system while allowing some amount of automation.

5. This article from Liran at Snyk, written in 2022, remains comprehensive and relevant for preventing supply chain attacks.

6. The only way for this to get solved ✨ magically upstream are further hardenings and automation from maintainers, enforced by NPM. The most important of these is the rollout of trusted publishing, which publishes NPM packages via OIDC credentials and trusted runners, rather than long lived credentials.

7. For runtime mitigation of this as a zero day, the answer is complicated because you would need runtime monitoring of dev machines, CI runners, staging, and production environments. This is an unusual combination, but I’ll endlessly support using the best runtime, agent based security you can get on cloud workloads. CADR exists precisely to stop these sorts of malicious zero days in your production systems, even if in this example the best case would’ve been an alert.

8. From an industry analyst perspective, what does the prevalence of these attacks do to the market? From one perspective, not that much. Point solutions that focused on preventing these attacks have existed for a long time, and none have found product market fit without being part of a broader set of solutions. However, within a broader set of supply chain solutions, I believe these features to be more critical than ever.
   
   Outside of traditional AppSec platform features, three categories that also get a small bump in usefulness are NHI, MDM (the Koi kind), and CADR, as monitoring these blind spots become more important, and there aren’t already a ton of great alternative solutions out there.

Consolidated Prevention Methods

1. Pin your dependency versions

2. Restrict pre and post install NPM scripts with your build tool

3. Use an allowlist model for egress on build systems

4. Use OIDC where possible for build secrets and/or regularly rotate them. Consider the right method for publishing to NPM.

5. Monitor developer, build, staging, and production systems for malicious activity. Warning: this can be noisy and expensive, especially for developer workstations and build stages.

6. A theoretically total visibility strategy here that I’m not recommending, but more to think through everything that’s possible:

   1. On the developer endpoint, monitoring plugins and open source versions and activities, alongside meaningful EDR that works here (easier said than done).

   2. On the non-human identity side, monitoring for unusual NHI token activity and unrotated credentials, indicating compromised credentials

   3. On the runtime side, monitoring build systems, staging, and production for detection of malicious activity.

Especially Useful Vendor Articles:

1. Wiz - for their incident timeline, scope, and general analysis. Contains the payload that sets up local machines as a Github runner, and then the payload that the malware uses to scrape Github secrets by setting up a new workflow.

2. Aikido - their research covers some extra methods that are continuing to be used as part of the attack, such as workflow triggers in pipelines. Also everyone loves Charlie, and he likely found patient zero.

3. Socket, Step Security, Endor Labs and Gitlab all provide the best walkthroughs of the exploit logic and payload itself

4. Datadog - great diagrams of the attack, lots of examples of the exploit code, and their gharchive queries

5. I appreciated HelixGuard giving the JSON format of repos containing the secrets

6. Hackernews thread always has interesting takes to follow along with

7. Stream Security gave a good example of what detecting this behavior at runtime might look like. For clarity, they’re not the only tool that would detect this, but this is useful for seeing what the alerts look like.