How a 'faulty channel file' from Crowdstrike bricked Windows computers everywhere, including New Zealand; manual fix available

Perfectly timed for the antipodean weekend during school holidays, a security vendor issues a full bodge update that crashes Microsoft Windows computers worldwide, including New Zealand. If you didn't know you're using Crowdstrike software, you do now. That is if you have a working device with which to read this.

For the record, Crowdstrike is a well-regarded information systems security company in Texas, United States, that has been involved in several high-profile cases investigating state sponsored hackers such as North Korea attacking Sony Pictures.

It's not the "Internet going down" but Microsoft Windows systems that are used in more places than you think failing. Interest.co.nz was able to confirm that users are not able to log in to ASB's banking app as of writing.

Users are reporting that just about everything imaginable has been hit, including supermarket checkouts, online banking, airlines, public transport, payments in stores including Paywave and EFTPOS (we're seeking confirmation on this), and more.

BBC reports there's worldwide travel chaos currently.

For the technically inclined, the issue appears to be a faulty update to security vendor Crowdstrike's Falcon Sensor software. This is code used to detect malware at a very low level in Windows.

It runs at very high privileges on Windows systems as a driver for the operating system kernel, the piece of code that keeps users' computers going. The reason for that is for security code to be effective, it needs to have full overview of everything on computers to catch threats. 

If the Windows kernel crashes, PCs will usually display the "Blue Screen of Death" or BSoD. Windows users will know that recovering from a BSoD fault can be very difficult, and often involves manual tweaking of operating system files. Manual as in having to physically be present at the computer that needs to be restored.

Crowdstrike has acknowledged the problem in oblique language, saying its team is "fully mobilised to ensure the security and stability" of its customers.

A Crowdstrike director, Brody Nisbet, said the issue is not quite an update but a faulty channel file. The person suggested the following manual fix for users whose systems will not start up without a BSoD:

1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Go to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching “C-00000291*.sys”, and delete it. 
4. Boot normally.

We do not guarantee that the above will work. If the computer is using Bitlocker drive encryption, this could complicate access to the file system.

Next, we await the explanation from Crowdstrike as to what went wrong and also from Microsoft, which might need to rethink its policies that currently allow for global IT disasters like the above to happen.

Update Microsoft said in a support note that it has received reports from customers running Azure virtual machines that rebooting them, up to 15 times, can fix the Crowdstrike faulty update.

Update 20/07/2024 The National Cyber Security Centre (NCSC), part of New Zealand's Government Communications Security Bureau (GCSB), has issued a statement, linking to Crowdstrike's guidance page with advice on how to recover from the faulty channel update. Also, to the surprise of nobody, scammers are trying to take advantage of what might just be the world's largest IT outage (so far).

An IT outage following an update made by CrowdStrike software has caused significant disruption globally.

This update resulted in outages in windows systems.

The issue has been identified, isolated and the vendor has released remediation guidance for customers, available via their CrowdStrike Customer Portal which will be updated as the situation evolves.

We encourage New Zealand organisations that have been impacted by this disruption to review the guidance issued by the vendor and act immediately.

The NCSC has no information to indicate these issues are related to malicious cyber security activity.

However, there has been an observed increase in phishing referencing this outage as opportunistic malicious cyber actors seek to take advantage of the situation.

We encourage organisations and individuals to be alert to this increased activity. Helpful resources to protect against phishing are available below.

Know the Risks - Own Your Online

Update 21/07/2024 Microsoft said in a blog post that its estimate of the number of "Crowdstruck" Windows machine is 8.5 million. This, Microsoft said, is less than one per cent of all Windows machines.

Those Windows boxes are found at non C-list Microsoft customers:

"While the percentage was small, the broad economic and societal impacts reflects the use of Crowdstrike by enterprises that run many critical services," Microsoft vice president of enterprise and OS security, David Weston said.