Security vendor Crowdstrike releases preliminary report on what caused last week's IT disaster; it's a reminder that our IT is and will remain fallible

The post mortem for the massive "Crowdstruck" IT meltdown last Friday is out. Even though it's a preliminary post incident report, and there might be more to come, Crowdstrike has published heaps of detail on what caused eight million to nine million Windows computers, physical and virtual, to crash and not restart properly. 

Understanding the dense technical detail of the report isn't easy, but the security vendor said it was "problematic content" in an update that caused the crashes.

Developer Nic Wise, suggested to think of it as being a person with access to a bunch of deposit boxes in a bank. Each box has something in it that the you need to do your job, and there are many of them, each with their own unique number.

Some of the boxes are yours, some are not, and all are guarded by an aggressive, armed bank manager who will shoot you if you open the wrong ones. In a computer, the bank manager is the low-level part of the operating system, the code that controls the hardware, provides access to files and folders on storage, handles networking and more.

This is called a kernel in computerese. As you can imagine, if the kernel malfunctions and crashes, your computer goes down with it. You do not want that to happen.

Next, you have a clipboard with a sequence of boxes that you're meant to open, and look inside for stuff your coworkers have put inside - data and instructions.

In the case of Crowdstrike, the clipboard says to open box 345. This is your box, so the aggressive bank manager doesn't shoot you when opening it. Even though your coworkers never put anything inside that box, you open it and do whatever it says on the piece of paper in it. This could be instructions from 12 months ago, from someone else's list and they're no longer correct or valid.

That's the error Crowdstrike said took place thanks to the problematic content in the update: an out-of-bounds memory read error. You access data you're entitled to see, but it's likely to be complete rubbish. For that reason, OOB (out of bounds) reads are something that operating systems try to prevent, by terminating misbehaving applications.

Normally that's fine, but as Nic said, if it's the bank, that is the operating system kernel, reading the bad data, it could lead to a situation where it is terminated.

When that happens, there's the Windows Blue Screen of Death (BSoD). Nic and I both remember how Windows used to just freeze with a BSoD, with data loss as the result.

Although Crowdstrike is blaming nobody but itself for the bug, its sensor threat detection software runs with very high privileges on computers. This gives it unfettered access to all parts of the computers it protects, which means it should be able to find malicious code hiding on them.

However, it's also vital for Crowdstrike to make sure the frequent updates to detect threats it sends out are always correct. If not, something bad can happen to the bank manager (the kernel) and we'll have worldwide chaos with BSoD screens to stare at.

Apple decided a few years ago it was just too unsafe to have third-party developers poke at things, deep in the operating system. It moved away from the so-called kernel extensions or KEXTs that had been part of macOS for ages, and restricted access to sensitive system areas. 

Doing that without breaking some existing software, often popular things like DropBox and Microsoft OneDrive cloud storage, can be difficult as developers need time to figure out how the new design works. But, it makes crashes due to buggy low-level code from third party developers much less likely. It also improves system security as malicious code is harder to hide deep in the operating system.

Before anyone chimes in, no, this is not fail safe or a perfect solution to crashing computers and security woes, but I'm trying to remember the last time the Apple devices I use froze or conked out, and I can't.

That's even with pre-release software that I run to check out new features coming up.

Kernel mode is *the* problem. In 2024 changing software from third parties via a private update channel is about the highest risk setup and should not be a generally available capability. And if it is it should not be used in critical systems. https://t.co/m10r5dLl8X

— Steven Sinofsky (@stevesi) July 19, 2024

Sinofsky has some idea what he's talking about, being a Microsoft Windows veteran. Microsoft is no doubt aware of the problem, and it has devised mitigations to stop buggy code from taking down the Windows kernel. The "Crowdstruck" incident should be the excuse Microsoft needs to embark on a deeper re-engineering of the operating system, no matter that it'll break some software from third party developers.

The longer story short here is: it's a risky to put all your eggs in the same basket. By now, after countless IT related fiascos over the years, it should be clear to everyone that writing bug free code isn't possible, along with engineering flawless hardware. 

If you go through the Crowdstrike report, it's clear that the security vendor tests the code it ships thoroughly. Even then, it only took one mistake that mysteriously enough wasn't caught early on (it should've been - see below), and a very extensive and expensive disaster struck.

Btw the report mentions testing over 20 times, but for those who didn’t catch it - none of the channel tests involve actually putting the updates on a CrowdStrike system.

One of the follow up actions listed is to test updates on dev systems.

Zero real testing in reality.

— Kevin Beaumont (@GossiTheDog) July 24, 2024

It's possible to mitigate against many such errors, the ones that been observed in the past and developers and engineers have had time to think about, but it's complicated stuff to say the least, for humans in particular. Nevertheless, there will always be something coming along that nobody thought about, often a combination of factors that isn't obvious.

Operating critical infrastructure on diverse systems may seem at first a way to mitigate the above, but the cost implications are unpalatable and the complexity of it brings risks by itself. Also, you can't introduce diverse, redundant infrastructure everywhere in a world with interoperating systems, many of which that any given organisation doesn't control.

There will be lots of "expert" commentary weighing in on how to fix this as always, but a bit like Heidegger's Hammer: IT works great until it doesn't and the tool locks us out of its world of usefulness which today is enormous.

Ironically enough, IT having become more reliable over the years has lulled us into a false sense of security. Nobody expects things to break. They do though, so we shouldn't be surprised at the consequences being serious, but be ready instead.