The recent SolarWinds incident has managed to grab headlines outside of our security ecosystem. The many (many) headlines and columns inches dedicated to the event are testament to the security worries that continue to reverberate around the globe. But we think that most of these articles have buried the lede.
Most discussions take the position that our enterprises are horribly exposed because of supply chain issues and that any network running SolarWinds should consider themselves compromised.
We think it’s actually more dire than that (and suspect it’s going to get worse). Let us lay out the case for why SolarWinds should concern you even if their tools are nowhere near your networks.
It’s easy to whip up a think-piece in the wake of a public security incident, especially as a vendor. The multitude of vendor mails riding the SolarWinds incident are overflowing our inboxes. But even a stopped clock is right twice a day, and this is one of those times.
An abstracted, low resolution summary for those (very few) who haven’t paid attention to the incident:
- SolarWinds make a network management product called Orion, which is deployed on tens of thousands of organisations worldwide;
- Attackers broke into SolarWinds and made their way to the SolarWinds build environment;
- They compromised the build pipelines, to inject malicious code into the SolarWinds update process;
- Organisations all over the world updated themselves with this poisoned update;
- (Now) Compromised SolarWinds servers worldwide attacked internal networks of selected organizations;
- Almost nobody discovered any of this for months till a security company discovered their own compromise.
The technique of compromising a single source, which then updates other nodes isn’t novel. As recently as yesterday we saw headlines like: “Barcode Scanner app on Google Play infects 10 million users with one update” and indeed this was how Will Smith and Jeff Goldblum saved us when the aliens first made contact.
The attack gets called a “supply chain” attack which hints at war-time tactics and I’m willing to bet, will launch a dozen cyber security / resilience startups. People are (rightfully) worried about the knock-on effect, since the SolarWinds attackers had access to several other software product companies and could have poisoned those wells too. This is definitely scary! But hear me out. It’s actually a little bit worse than you might think.
Why it’s actually worse than we think
The state of enterprise security: While we’ve made progress in some areas of information security (e.g. the degree of knowledge and skill required to exploit memory corruption bugs in modern OSs) , enterprise security is still stuck pretty firmly in the early 2000’s. An enterprise network consists of an untold number of disparate products, loosely coupled through poorly documented interfaces where often the standard for product integration is “this config works, don’t touch it”. Any moderately skilled attacker will decimate an internal corporate network long before they are discovered, and the average time it takes to gain Domain Admin is measured in hours and days instead of weeks or months.
Most organizations don’t know this though. They know they spend money on security and they know they see charts tracking progress. Most have no clue that faced with an average attacker of moderate skill, they’d almost certainly come off second best.
Enterprise Products: Even ignoring the weakness that comes with cobbling together many products (security at the joints), most enterprise products won’t hold up very well to serious security testing. Heavyweight vendors like Adobe and Microsoft were publicly spanked into upping their game years ago, but it drops off pretty steeply after them. There’s an interesting carveout for online SaaS companies who have to build security competency since they run their own infrastructure and compromising their products is the same as compromising them. But for products installed into an Enterprise network the incentives are horribly misaligned. Owning, say, Symantec’s antivirus agent doesn’t compromise Symantec, it compromises you (who are running it) and this separation makes all the difference.
Enterprise networks have too many moving parts: The past few years have seen creative hackers exploit software in places that we never knew were running software. The Thunderstrike crew ran code on Apple video adaptors. Ang Cui has run code on monitors, and office phones. Bunny and xobs ran code on SD-cards and a number of people have now run Linux on hard drive controllers. This makes it clear that the average office network is connected to dozens and dozens of types of devices that wont ever make it into a regular audit, that are nonetheless capable of hiding attackers and injecting badness into your network.
3rd Party Risk evaluations: The joke going around after the incident was that SolarWinds had negatively impacted hundreds of organisations, but definitely passed their 3rd party risk evaluations. It’s slightly unfair, but also true. We don’t have a good way for most organizations to test software like this, and 3rd party questionnaires have always been a weak substitute. Even if we could tell if a product was meeting a minimum security bar (using safe patterns, avoiding unsafe calls, using compile time safety nets, and so on) auto-updates mean that tomorrow’s version of the product might not be the product you tested today. And if the vendor doesn’t know when they are compromised, then they probably won’t know when their update mechanism is used to convert their product into an attacker’s proxy.
(Please note: We aren’t saying that auto-updates are bad. We believe they solve important problems and we make use of it in our product, but they do introduce a new set of variables that need to be considered. We discussed it in more detail in a previous post of ours: “If i run your software, can you hack me?”)
The current focus on “supply chain” security will no doubt birth a bunch of companies claiming to solve the problem, but this part of the problem seems intractable. There’s the “easy” suit of software you know about: applications installed on your infrastructure, their dependencies, and so on. But for one, this ignores your vendor’s own vendors. In addition, what product is going to provide guidance on the provenance of the code running in your monitors (on processors we didn’t even know were there?). Will we examine the firmware on the microphone that people are now using for their Zoom calls? Will we re-examine it post its update? There are just too many connected pieces of code to tackle the problem from this angle.
Enterprise Security Software: Amazingly, if enterprise products as a whole can be classified as insecure, enterprise security products in general are super duper insecure. Dr Mudge warned us in the early 2000s that security products were not necessarily secure products but not enough people took notice.. Many a Veracode report has placed enterprise security products near the bottom of the product pile when tested for security defects.
FX famously quipped that “basically by quality level you would be better off defending your network with microsoft word than a checkpoint firewall”. (It’s funny because it’s sad).
If it takes just hours or days to successfully compromise an internal network, and if the average network has enough hiding places for skilled attackers to burrow deep, what do you think happens when attackers are allowed to move around undetected for months?
All of these factors have been true for decades and have not visibly led to too many melt-downs. This changes though, because of a kind of “Roger Bannister” effect. Breaking the 4-minute mile seemed impossible till Roger Bannister did it in 1954. Then it was matched repeatedly in fairly quick succession. (Today, several high school runners have conquered the feat). Often people just need to see someone else cross that line. It’s not uncommon to see certain bugs considered unexploitable for a while, only to have the floodgates open up post the first working exploit release.
When STUXNET made the news in 2010, the result was a global realisation that software exploits could be used to good effect in the real world, but the attack remained fairly magical and esoteric: It targeted centrifuges and involved multiple 0-days and infected Step7 compilers to get manually introduced to the PLCs. The Snowden leaks a few years later however, made it clear that smaller-scale and well targeted exploits could be used to achieve results too. If any country was slow to get into offensive cyber pre-Snowden, very few (who were paying attention) were after that. Governments started tooling up and the commercial industry didn’t hesitate to fill in the gaps.
When the Ukrainian tax accounting package MEDoc had its update mechanism compromised to deploy malware to its clients, well, the writing was on the wall. Attacking popular vendors as a route into customers was clearly effective and to some actors, squarely on the table.
A bunch of analysts looking at the SolarWinds incident point out (correctly) that compromised SolarWinds servers were on so many networks that the ripples of this attack could be crazily exponential. What this analysis misses is that the average enterprise runs dozens and dozens of SolarWinds-look-alikes too.
Ransomware didn’t spring up overnight. Networks hit by ransomware were typically vulnerable for years and ran along blissfully unaware of it till attackers evolved a method to take advantage of it. Most enterprises have been completely vulnerable to their vendors’ horrible insecurity too, the SolarWinds incident just published a blueprint for how to abuse it.
The situation is dire not because we are fighting some fundamental laws of physics, but because we’ve deluded ourselves for a long time. If there’s a silver lining out of this, it’s that customers will hopefully demand more from their vendors. Proof that they’ve gone through more than compliance checklists and proof that they’d have a shot at knowing when they were compromised. That more enterprises will ask “how would we fare if those boxes in the corner turned evil? Would we even know?”
Ps. We’ve written previously about how we think about security as a vendor [here]
Pps. We build Thinkst Canary, a quick, low effort, high fidelity way to detect badness on your network. We didn’t write this article because we built Canary. We build Canary, because we believe what we’ve written in this article…
1 comments On On SolarWinds, Supply Chains and Enterprise Networks
Love you guys