Summary of the Global Crowdstrike-Microsoft Outage
Overview
On July 19, 2024, CrowdStrike and Microsoft experienced a significant global outage that lasted multiple hours and impacted vital services. The disruption, from a configuration error during routine maintenance, affected authentication and data processing systems. The sensor configuration update that caused the system crash was remediated on Friday, July 19, 2024, 05:27 UTC. The impact was felt by customers running Falcon sensor for Windows version 7.11 and above online between Friday, July 19, 2024, 04:09 UTC and Friday, July 19, 2024, 05:27 UTC. Systems running Falcon sensors for Windows 7.11 and above were susceptible to a system crash.
Details
The outage's root cause was identified as a misconfiguration resulting from a system update error. The configuration files mentioned above are referred to as “Channel Files” and are part of the behavioural protection mechanisms used by the Falcon sensor. Updates to Channel Files are a normal part of the sensor’s operation and occur several times daily in response to novel tactics, techniques, and procedures discovered by CrowdStrike.
Analysis
On Windows systems, Channel Files reside in the following directory:
C:\Windows\System32\drivers\CrowdStrike\
It has a file name that starts with “C-”. Each channel file is assigned a number as a unique identifier. The impacted Channel File in this event is 291 and will have a filename that starts with “C-00000291-” and ends with a .sys extension. Although Channel Files end with the SYS extension, they are not kernel drivers.
Triage & Remediation Steps
Per CrowStrike, here are the recommendations for workarounds -
Reboot the host to allow it to download the reverted channel file. We strongly recommend putting the host on a wired network (as opposed to WiFi) before rebooting, as the host will acquire internet connectivity considerably faster via ethernet.
If the host crashes again, then:
Boot Windows into Safe Mode or the Windows Recovery Environment
NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
Windows Recovery defaults to X:\windows\system32
Navigate to the appropriate partition first (default is C:\), and navigate to the crowd strike directory:
C:
cd windows\system32\drivers\crowdstrike
Note: On WinRE/WinPE, navigate to the Windows\System32\drivers\CrowdStrike directory of the OS volume
Locate the file matching “C-00000291*.sys” and delete it.
Do not delete or change any other files or folders
Cold Boot, the host
Shut down the host.
Start host from the off-state.
Note: Bitlocker-encrypted hosts may require a recovery key
Advanced Search Query for Crowdstrike Affected Hosts
Lessons Learned
This outage underscored the importance of robust contingency planning, rigorous testing of system changes, and transparent customer communication during disruptions. In response, CrowdStrike and Microsoft are enhancing monitoring capabilities and reinforcing change management procedures to reduce the risk of similar incidents.
Watch for malicious actors leveraging the ongoing CrowdStrike-related event to phish organizations. Example → https://www.virustotal.com/gui/domain/crowdstrike-bsod.com
References: