A whole bunch of major companies and government agencies recently had their networks compromised ("hacked"). This incident is far more serious than most other such compromises, not just because of the scope of the targets, but because of the methodology. Most compromises are successful because the target made a mistake. But in this case, the targets were compromised because they did something right. The targets installed patches, just as they were supposed to.
The bad guys didn't go after the targets directly. Instead, they first went after SolarWinds. SolarWinds is a major IT software vendor. Like many vendors, SolarWinds sometimes has to issues "patches" -- fixes for bugs that are discovered in their products. IT organizations are encouraged to download such patches, check their "digital signatures", scan them for bad software ("virus", "malware", "trojans", etc.), test them to make sure they don't break functionality, and install them as quickly as possible. Such patches often fix security bugs that could be used to break into the product, so IT security people want us to go through the steps as quickly as possible. Many vendors digitally "sign" their patches, so we IT people can check the signatures and not get tricked into installing software from a bad guy.
So the bad guys broke into SolarWind's network. They changed the vendor's software "build server" to add custom bad software to the patches themselves. Because this happened on the actual build server, the bad software updates were signed by the vendor's digital keys. Because the "bad software" was custom, it evaded detection of existing bad software. IT personnel in the target organizations downloaded the patches, checked the signatures, tested the software for functionality, scanned it for (known) bad software, and after it passed all the checks, installed it. Even after all their care, their SolarWind servers were then compromised.
And then it became still worse. Sites with really good security postures often have internal firewalls and sophisticated software that limit even what the software legitimately installed on their networks can do. This can stop many adversaries even if they can get past other security defenses. This is an IT security principle known as "defense in depth." The catch here is that SolarWinds is a vendor of "network management software". This is software that is designed to reach out all over the network, talking a variety of network protocols. So the attackers didn't just have access to any old server. They had access to the servers that had the most access on the organization networks.
In other words, target organizations that did everything right, and followed every major best practice, were still broken into. The multiple levels of protections that the more sophisticated organizations employ to stop this kind of thing didn't help here.
This compromise demonstrates one of the great principles of IT security: IT security cannot prevent high-level attacks by a sufficiently determined and well-resourced adversary. IT security only exists to stop lower-level attacks. The overwhelming majority of IT attacks are by "script kiddies" -- people who aren't capable of writing an exploit themselves, but are able to use exploit tools written by others. Higher tiers of attackers who can write their own novel exploits and tools are more rare, and tougher to beat, but we can beat them with more sophisticated defenses. But the highest tier of attackers can get through our best defenses. These attackers have access to many custom exploits, their own tools, and disciplined tradecraft.
After every major incident, IT people read up on the incident to understand what went wrong and what we need to do better. What makes this incident so very terrifying and different is that the methodology didn't require target organizations to make any mistakes. If anything, organizations that followed best practices were targeted through some of the very practices that normally protect them.
This is a hard lesson. We all desperately want to know what we can do to avoid such a problem. But sometimes we are powerless. "It is possible to commit no mistakes and still lose." -- Star Trek: The Next Generation
This is important to understand. Many journalists, pundits, and politicians don't get it. Even many IT personnel don't have technical backgrounds strong enough to really understand it. This means that we'll be seeing bad recommendations in the popular media and even in the trade media.
Physical security provides a good analogy. A regular door and lock will prevent someone from casually entering a house, but won't stop a locksmith or someone willing to kick the door in. A fancier lock and reinforced door will slow down an attacker, but someone with an ax or a determined locksmith can still get in. If you're running a bank, you want even more security in the form of a vault, but even those can be broken into by a determined attacker with a drill and enough time. So if you're protecting something really important, you also hire around-the-clock security guards. And if you're a government, maybe you even install a military base. But if an army suddenly shows up on your doorstop with tanks and airplanes and a lot more force than your military base can counter, then you can try to fight, but you might be forced to surrender.
In the same way, IT security provides layers of protection that can deal with higher levels of threat if you're willing to expend more resources. But even the best sets of controls have their limits. At some level, the challenge is bigger than what even a large organization can counter.
All of this matters because there will be calls for change. Perhaps some of what is recommended will actually be novel concepts by knowledgable people that address the real threats. But so far, everything I've heard would not have realistically helped. And at least one item not only wouldn't have helped, but would actually have made us more susceptible to the more common low-level threats by slowing down patch cycles. When the pundits and politicians make recommendations based on this incident, we will need to pay attention and ask some hard questions.
Technical source: https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html
(Addendum: I don't run SolarWinds Orion, and never have. On a former job, when network management systems and IT security were a large part of my role, I recommended products other than SolarWinds. It wasn't because I saw this coming. But I'm feeling really good anyway!)
Written 2020-12-18; updated 2020-12-21
