CrowdStrike's Falcon software caused widespread global outages

In July 2024, CrowdStrike, a prominent American cybersecurity firm, experienced a major disruption when a faulty update to its Falcon software caused global outages, affecting millions of devices and critical industries such as banking, air travel, and government services.

CrowdStrike’s Founding and History

Founded in 2011 by George Kurtz, Dmitri Alperovitch, and Gregg Marston, CrowdStrike quickly established itself as a pioneer in cloud-delivered endpoint protection. The company’s rapid growth was fueled by significant investments, including funding from Google’s venture capital division.

In 2013, CrowdStrike launched its flagship product, CrowdStrike Falcon, which provided endpoint protection, threat intelligence, and attribution capabilities. The firm gained prominence through its involvement in investigating high-profile cyberattacks, including the 2014 Sony Pictures hack.

July 2024 Disruption

In July 2024, CrowdStrike, a prominent American cybersecurity firm, experienced a major disruption when a faulty update to its Falcon software caused global outages, affecting millions of devices and critical industries such as banking, air travel, and government services.

What Happened

On July 19, 2024, at 04:09 UTC, CrowdStrike distributed a faulty update to its Falcon security software for Windows hosts, causing widespread system crashes and disruptions globally. The problematic update was a content deployment related to a channel file located in the Windows system directory. This update triggered bugcheck or blue screen errors on affected Windows machines, causing them to crash or enter boot loops.

The issue primarily affected systems running Windows 10 and Windows 11, though some Windows 7 and Windows 8 systems were also impacted. Notably, Mac and Linux hosts were not affected by this update. The problem was quickly identified, and CrowdStrike reverted the content update at 05:27 UTC. Devices that booted after this reversion were not affected by the issue.

The scale of the disruption was significant, with Microsoft estimating that approximately 8.5 million devices were affected by the update. This widespread impact was due to CrowdStrike’s extensive customer base, which includes nearly 60% of Fortune 500 companies and more than half of the Fortune 1,000.

CrowdStrike’s CEO, George Kurtz, confirmed that the issue was caused by a faulty kernel configuration file update and emphasized that it was not the result of a cyberattack. The company deployed a fix and actively worked with impacted customers to restore affected systems.

The outage had a cascading effect across various sectors globally. Major disruptions were reported in multiple countries, with the impact sweeping from east to west due to time zone differences. Industries such as banking, airlines, and government services experienced significant disruptions. This incident highlighted the critical role of cybersecurity software in modern IT infrastructure and the potential for widespread disruption when such systems encounter issues. It also underscored the importance of thorough testing and gradual rollout procedures for software updates in critical systems.

Workaround Steps

CrowdStrike provided detailed workaround steps to help affected customers recover their systems and mitigate the impact of the faulty update. These steps varied depending on the environment and severity of the issue:

For Individual Hosts

• Reboot the affected host to allow it to download the reverted channel file.
• If the host continues to crash, boot Windows into Safe Mode or the Windows Recovery Environment.
• Use a wired network connection and Safe Mode with Networking to facilitate remediation.
• Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory.
• Locate and delete the file matching C-00000291*.sys.
• Boot the host normally.

For Public Cloud or Virtual Environments

Option 1:

• Detach the operating system disk volume from the impacted virtual server.
• Create a snapshot or backup of the disk volume as a precaution.
• Attach/mount the volume to a new virtual server.
• Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory.
• Locate and delete the file matching C-00000291*.sys.
• Detach the volume from the new virtual server.
• Reattach the fixed volume to the impacted virtual server.

Option 2:

• Roll back to a snapshot taken before 0409 UTC on July 19, 2024.