Vulnerability Management and XZ Utils - Is there any hope?
CVE-2024-3094, Open Source Security, and VulnCon - Is there any hope?
I’ve never been so sure that vulnerabilities are both critical, and don’t matter at all. This last week was VulnCon in my very own Raleigh, NC. Overall, the main benefit was seeing just how human our vulnerability systems are; immediately after it was over, the latest CVE shows just how true that is. This week, we’ll cover a bit from the conference, and the wild ride that was CVE-2024-3094 - and how it shows that vulnerability management has never been more and less important than it is now.
Table of Contents
CVE-2024-3094
The latest CVE, identified by an engineer at Microsoft and reported through RedHat, puts on full display the strengths and weaknesses of our CVE ecosystem. First, I put together a brief summary here of how to check if you’re vulnerable. Long story short, this CVE has all of the open source drama and intrigue you’d expect, while displaying how short we fall, despite the countless “supply chain security” vendors that are out there.
CVEs Don’t Matter
Most vulnerabilities that get published are along the lines of “if you’re using this particular function from this library in this particular way that no one really does, you can do something you’re not supposed to.” Almost every named major vulnerability is critical because it changes from “no one uses this like this” to “everyone uses this package like this.” Vulnerabilities like this one were mission critical because it was an active malware pushed upstream by a maintainer who had spent years slowly taking it over. This wasn’t a vulnerability as much as it was malware from upstream.
This excellent timeline shows how these catastrophic attempts are led by human problems. Our entire software industry is being help up by single open source maintainers who get no credit, money, or recognition for their critical services. This shouldn’t be thought of as much as an isolated vulnerability as much as the tip of the supply chain iceberg.
We still don’t know the full nature of the actual exploit. It’s clear that the exploit was compiled from test code when distributed via RHEL and Ubuntu distros, but it’s unclear if that’s the only places the attack would run from and even the full nature of what the attack was doing besides allowing takeovers of SSH connections. If you wanna really get nerdy with it, this is the article.
The scariest part of all is that the discovery was so much by chance. Andres Freund at Microsoft was running benchmarks on a beta version of Ubuntu and discovered the high utilization of resources from sshd processes. Luckily for everyone, he dug in deep to try and figure out what was causing the issue.
From all of these points, this wasn’t a vulnerability as much as malware, it could be considered a type of social engineering, it’s not consistent in the exploit, and the CVE ecosystem played no role in its discovery.
CVEs Are Critical
Without a clear vulnerability disclosure program, there would be no way for this vulnerability to be responsibly disclosed and patched. For every vulnerability like this one that gets a lot of buzz, there are vulnerabilities that go unnoticed that can potentially cause catastrophic harm to systems.
Once the hype of patching is over, NIST and CISA will be the two agencies most likely to actually figure out what the nature of the attack was and what can be done about it.
Although the authoritative source at NIST wasn’t the first website that went live, it’s absolutely critical that it exists. The vendor blogs are all just SEO traps for marketing dollars, there needs to be someone with a long term legitimate interest in keeping research up to date.
The problem demonstrated exactly why timely and accessible systems need to exist for large scale responses to zero days.
How do we Fix This? (It’s not ASPM)
The vulnerability problem requires more innovation than another vendor making Jira tickets for patches faster. Here are the areas of innovation I think are actually helping:
I’d highly suggest reading this
To be frank, open source funding is a joke. I’d highly suggest learning about how open source projects are (not) funded. Companies are making billions of dollars in profits off of open source projects, but only 304 million is given across 32 foundations (2022).
The only provider I see having a realistic chance of fixing this is Tidelift. Companies have a real security interest in supporting the open source that they take for granted, and the Tidelift subscription model is the most realistic cost/benefit.
Approaches to security that go beyond CVEs - this vulnerability should make us think, “what if the exploit didn’t cause performance issues that would’ve been noticed?” And, “how many exploits like this are undetected?” This is why I talk to so much about the value of runtime protection like Sysdig, Oligo, and Rad that stand a chance at detecting this as a zero day (there are others as well).
Shifting detection priorities to upstream - which I talked about a lot in this post.
The convergence of SCA and container vulnerability scanning - with this vulnerability, only container scanners caught it unless your SCA scanner did something funky with it. More and more I think the container should be the source of truth and just backport the SCA functionality like scanning static files.
In sum, it’s time to recognize that vulnerability programs do not adequately address the problem of application security. We need to really widen our approach to include supply chain more broadly, as well as runtime protection.
VulnCon Summary
VulnCon met at an interesting time - the NVD is changing to a consortium, they’re underfunded, and there’s a backlog of processing vulnerabilities. Ultimately, it was great to meet with people from larger hardware providers down to smaller software ones - it helps paint a picture of different size companies approach vulnerability management. While vulnerabilities are a false positive headache for small SaaS company, for a massive hardware provider they’re absolutely critical problems to address before they ship to government critical entities.
Here are 4 small take aways from the conference:
There are two different levels of seriousness for vulnerabilities
Everyone would do well to acknowledge the dichotomy that exists for addressing vulnerabilities. Most vulnerabilities in most contexts are false positives, therefore, it’s only natural that smaller companies would hate dealing with them. They slow down innovation for very little benefit, and these companies have no financial incentive to take them more seriously. While it would be nice to talk about having everyone provide public SBOMs with clear VEX statements per CVE, these smaller companies are just praying no one starts asking for them; and they’re right to do so, because SBOMs are a false positive hellscape for most providers.
Conversely, massive companies, especially those that provide software that runs on endpoints or hardware, need to take vulnerability management much more seriously. In these contexts, attacks like buffer overflows are a lot more applicable, and you have a lot less ability to provide runtime defenses alongside your infrastructure.
At the end of the day - a company is only going to take vulnerability management as seriously as they have to. What that means in the simplest way is that someone is either asking for your SBOM and reading it, or they’re asking for your SBOM to file it away in the trash can.
There are amazing global efforts to get vulnerability management more online
As an American, I really take the NVD infrastructure for granted. There were a few different talks from agencies like ENISA and APCERT that are doing everything they can to get vulnerability disclosure and coordination programs up and running in Europe and Asia Pacific. These agencies will be critical as other regions continue creating more and more software (not that they aren’t already, but that the amount will continue to increase, and vulnerabilities alongside it).
SBOMs are a Necessary Evil That Need to Evolve
Everyone agrees that SBOMs suck to manage and are full of false positives, yet everyone agrees that they’re necessary tool for building trust between vendors. At the end of the day, I think SBOMs are very much a problem in flight - ultimately they will matter in some standardized form that everyone is okay with, but for now they’re still not super useful. Until we know what we want to see from an SBOM (the answer can’t be 0 vulnerabilities), its usefulness won’t be apparent.
More Needs to be Done Upstream
Everyone thinks the work of vulnerability management is critical, but no one wants to pay for it. Vulnerability management is a problem because companies use so much open source software. Correction, vulnerability management is a problem because companies don’t want to pay for open source software - they use it to make massive profits, but have the expectation of never giving back. This to me is the core problem we’re talking about - no one wants to foot the bill to support open source projects. Tidelift is the only vendor approaching the vulnerability management problem at it’s upstream source, while others offer small donation kickbacks to the various foundations that are occasionally able to help.
Latio Updates:
Added VulnCheck to boundary breakers - Excellent threat feed like enrichment of vulnerability exploit data
Added REVEALD to Remediation Platforms - Not the cleanest fit, but trying to avoid creating a CTEM category. REVEALD places vulnerabilities in their context helping determine blast radius.
Added Aikido to some categories I didn’t have them in - hard to keep track of all the scanners when you’re 9-in-1
Added Intigriti to Pen Testing - bug bounty but better
Added EdgeScan to DAST - lots of pentesty services