We are working closely with CrowdStrike to provide all affected companies with the latest information and resources. Our technical teams are available to offer guidance and best practices during this time. Please rest assured that we remain dedicated to responding to cases and assisting as much as we can.
-----------
-Affected Systems
The issue impacted Windows hosts that were online between 04:09 and 05:27 UTC and received the faulty channel file “C-00000291-*”
CrowdStrike has provided two methods to identify potentially impacted hosts: a dedicated dashboard and an advanced event search query.
-----------
-Dashboard to find the potentially impacted hosts
The dashboard, named “hosts_possibly_impacted_by_windows_crashes”, is located under Next-Gen SIEM > Log Management > Dashboards.
To use the dashboard:
Open the dashboard and select your CID, or use * to select all CIDs if you have multiple. If applicable, choose * for all channels or use the specific one mentioned in the “Impacted CIDs and Channels” widget. Select the “CHECK” status. In the “Impacted Sensors” widget, click on the menu in the top-right corner to find the option to export the results to file.
We have seen reports that rebooting the hosts multiple times might allow the reverted channel file to be downloaded. It is recommended to connect the host to a wired network instead of via WiFi and try rebooting multiple times.
If the host continues to crash, follow these steps:
We are continuing to monitor for any further issues.
Posted Jul 19, 2024 - 13:07 UTC
Update
We are actively monitoring and addressing the issue. Our CFC has mitigated any issues and has full capability to monitor our clients. Additionally, we are working closely with CrowdStrike to find multiple solutions. Please rest assured that we remain dedicated to responding to cases and assisting as much as we can.
-----------
In the meantime, if you encounter problems, you can follow these workarounds steps:
Workaround Steps for public cloud or similar environment: Detach the operating system disk volume from the impacted virtual server Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes Attach/mount the volume to to a new virtual server Navigate to the C:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Detach the volume from the new virtual server Reattach the fixed volume to the impacted virtual server
Option 2: Roll back to a snapshot before 0409 UTC.
Workaround Steps for Azure via serial : Login to Azure console --> Go to Virtual Machines --> Select the VM Upper left on console --> Click : "Connect" --> Click --> Connect --> Click "More ways to Connect" --> Click : "Serial Console" Step 3 : Once SAC has loaded, type in 'cmd' and press enter. type in 'cmd' command type in : ch -si 1 Press any key (space bar). Enter Administrator credentials Type the following: bcdedit /set {current} safeboot minimal bcdedit /set {current} safeboot network Restart VM Optional: How to confirm the boot state? Run command: wmic COMPUTERSYSTEM GET BootupState -----------
Workaround Steps for individual hosts: Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then: Boot Windows into Safe Mode or the Windows Recovery Environmen Note: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Boot the host normally. Note: Bitlocker-encrypted hosts may require a recovery key.
-----------
If Microsoft requests your BitLocker recovery key : please refer to the following links for assistance:
Make sure to execute it with the time window set to Last 1 day
-----------
Query to identify impacted hosts via Advanced event search :
// Get ConfigStateUpdate and SensorHeartbeat events #event_simpleName=/^(ConfigStateUpdate|SensorHeartbeat)$/ event_platform=Win // Narrow search to Channel File 291 and extract version number; accept all SensorHeartbeat events within impact window | case{ #event_simpleName=ConfigStateUpdate | regex("\|1,123,(?.*?)\|", field=ConfigStateData, strict=false) | parseInt(CFVersion, radix=16); #event_simpleName=SensorHeartbeat | rename([[@timestamp, LastSeen]]); }
| case{ #event_simpleName=ConfigStateUpdate | @timestamp>1721362140000 AND @timestamp < 1721366820000 | CSUcounter:=1; #event_simpleName=SensorHeartbeat | LastSeen>1721362140000 AND LastSeen *; } | default(value="0", field=[CSUcounter, SHBcounter]) // Make sure both ConfigState update and SensorHeartbeat have happened | selfJoinFilter(field=[cid, aid, ComputerName], where=[{ConfigStateUpdate}, {SensorHeartbeat}]) // Aggregate results | groupBy([cid, aid], function=([{selectFromMax(field="@timestamp", include=[CFVersion])}, {selectFromMax(field="@timestamp", include=[@timestamp]) | rename(field="@timestamp", as="LastSeen")}, max(CSUcounter, as=CSUcounter), max(SHBcounter, as=SHBcounter)]), limit=max) // Perform check on selfJoinFilter | CFVersion=* LastSeen=* // Calculate time between last seen and now | LastSeenDelta:=now()-LastSeen // Optional threshold; 3600000 is one hour | LastSeenDelta>3600000 // Calculate duration between last seen and now | LastSeenDelta:=formatDuration("LastSeenDelta", precision=2) // Convert LastSeen time to human-readable format | LastSeen:=formatTime(format="%F %T", field="LastSeen") // Enrich aggregation with aid_master details | aid=~match(file="aid_master_main.csv", column=[aid]) | aid=~match(file="aid_master_details.csv", column=[aid], include=[FalconGroupingTags, SensorGroupingTags]) // Convert FirstSeen time to human-readable format | FirstSeen:=formatTime(format="%F %T", field="FirstSeen")
// Move ProductType to human-readable format and add formatting | $falcon/helper:enrich(field=ProductType) | drop([Time]) | default(value="-", field=[MachineDomain, OU, SiteName, FalconGroupingTags, SensorGroupingTags], replaceEmpty=true) | case{ CSUcounter=0 AND SHBcounter=0 | Details:="OK: Endpoint did not receive channel file during impacted window. Endpoint was offline."; CSUcounter=0 AND SHBcounter=1 | Details:="OK: Endpoint did not receive channel file during impacted window. Endpoint was online."; CSUcounter=1 AND SHBcounter=1 | Details:="CHECK: Endpoint received channel file during impacted window. Endpoint was online. Endpoint has not been seen online in past hour."; }
Posted Jul 19, 2024 - 13:00 UTC
Update
We wanted to provide an update on our current situation. We are still actively monitoring monitoring the situation and addressing the issue. In the meantime, if you encounter similar problems in a public cloud or similar environment, you can follow these workaround steps: - Detach the operating system disk volume from the impacted virtual server - Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes - Attach/mount the volume to to a new virtual server - Navigate to the C:\Windows\System32\drivers\CrowdStrike directory - Locate the file matching “C-00000291*.sys”, and delete it. - Detach the volume from the new virtual server - Reattach the fixed volume to the impacted virtual server
To identify the affected hosts, Crowdstrike recommend to use the following query: lessCopier le codeC-00000291* |in(field="#event_simpleName", values=[AgentOnLine, LFODownloadConfirmation]) | groupBy([aid,ComputerName], function=[max(@timestamp, as=lastSeen),max(@timestamp, as=lastSeenForCalculation) ,collect([FileName])], limit=max) | lastSeen:=formatTime(field=lastSeen, format="%Y/%m/%d %H:%M:%S") | lastSeenForCalculation >= 1721362140000 AND lastSeenForCalculation <= 1721366820000
Posted Jul 19, 2024 - 10:06 UTC
Update
We are actively monitoring the situation and addressing the issue. Please be aware that while we are on top of the matter, there may still be some breaches in our Service Level Agreement (SLA) as we continue to triage between false positives and potentially true attacks.
Posted Jul 19, 2024 - 08:37 UTC
Monitoring
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes. If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue. Workaround Steps: - Boot Windows into Safe Mode or the Windows Recovery Environment - Navigate to the C:\Windows\System32\drivers\CrowdStrike directory - Locate the file matching “C-00000291*.sys”, and delete it. - Boot the host normally.
Posted Jul 19, 2024 - 07:11 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jul 19, 2024 - 06:48 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jul 19, 2024 - 06:37 UTC
Identified
The issue has been identified and our engineering team is assessing a workaround.
Posted Jul 19, 2024 - 06:35 UTC
Update
We are continuing to investigate this issue.
Posted Jul 19, 2024 - 06:33 UTC
Investigating
We have an ongoing issue with CrowdStrike that may cause crashes on Windows hosts related to the Falcon Sensor. We are currently investigating the incident and will update this page within the next hour. Please trust that all our teams are actively working towards a swift resolution of the degradation.
Posted Jul 19, 2024 - 06:33 UTC
This incident affected: Security Alert / Event Ingestion Pipeline / Managed System Access.