Someone started a DC thread but I think this is worthy for out in the Lounge. If you hadn't heard of CrowdStrike you will know them this weekend. The hospital I work for is fucked right now. The fix is to manually go touch every PC in the hospital. We have thousands of PCs. What a nightmare for the desktop/sysadmin guys.
Originally Posted by :
(Reuters) - A global tech outage was disrupting operations across multiple industries on Friday, with airlines halting flights, some broadcasters off air and services from banking to healthcare hit by system problems.
While major U.S. airlines - American Airlines, Delta Airlines and United Airlines - grounded flights, other carriers and airports around the world reported delays and disruptions early on Friday.
Banks and financial services firms from Australia to India and Germany warned customers of disruptions.
In Britain, booking systems used by doctors were offline, multiple reports from medical officials on X said, while Sky News, one of the country's major news broadcasters was off air, apologising for being unable to transmit live, and soccer club Manchester United said on X that it had to postpone a scheduled release of tickets.
The former head of Britain's National Cyber Security Centre Ciaran Martin told BBC Radio that an update to a product offered by global cyberscurity firm CrowdStrike appeared to be affecting operating systems based on Microsoft's Windows Operating System.
Microsoft's MSFT.O cloud unit Azure said it was aware of the issue that impacted virtual machines running Windows OS and the CrowdStrike Falcon agent getting stuck in a "restarting state," amid an ongoing global outage.
"We're aware of an issue affecting Windows devices due to an update from a third-party software platform. We anticipate a resolution is forthcoming," a Microsoft spokesperson said.
According to an alert sent by CrowdStrike to its clients and reviewed by Reuters, the company’s "Falcon Sensor" software is causing Microsoft Windows to crash and display a blue screen, known informally as the "Blue Screen of Death".
The alert, which was sent at 0530 GMT on Friday, also shared a manual workaround to rectify the issue.
Over half of Fortune 500 companies used CrowdStrike software, the U.S. firm said in a promotional video this year.
A Crowdstrike spokesperson did not respond to emails or calls requesting comment.
There was no information to suggest the outage was a cyber security incident, the office of Australia's National Cyber Security Coordinator Michelle McGuinness said in a post on X. A British government source also told Reuters there was nothing to suggest foul play.
"The world grinding to a halt because of a global IT meltdown shows the dark side to technology," AJ Bell investment analyst Dan Coatsworth said.
"The severity of the problem boils down to how long it lasts. A few hours' disruption is unhelpful but not a catastrophe. Prolonged disruption is another matter," he said.
The outages rippled far and wide.
Airports in Singapore, Hong Kong and India said the outage meant some airlines were having to check in passengers manually.
Amsterdam's Schiphol Airport, one of Europe's busiest, said it was affected, while airline Iberia said it had been operating manually at airports until its electronic check-in counters and online check-ins were reactivated. It said there had been some delays but no flight cancellations.
Air France-KLM said its operations were disrupted.
The Dutch foreign affairs ministry told Dutch press agency ANP it had been affected. A spokesperson was not immediately available for comment.
While there were reports of companies gradually restoring their services, analysts weighed the potential of what one called the biggest ever outage in the industry and the broader economy.
"IT security tools are all designed to ensure that companies can continue to operate in the worst-case scenario of a data breach, so to be the root cause of a global IT outage is an unmitigated disaster," said Ajay Unni, CEO of StickmanCyber, one of Australia's largest cybersecurity services companies.
Originally Posted by Bearcat:
Yeah, it's so easily mitigated by having a rollout strategy that includes beta testers in the wild, so if you're some huge corporation, have 1% of your PCs auto update a week prior (or even a day, anything) to everything else.
It's also crazy some of these companies would accept auto updates across the board and just hope the vendor doesn't fuck you over.. I wonder how many of these companies auto update Microsoft patches and so forth, too.
I've never worked with Crowdstrike, but unless there's zero option for manual updates, a lot of CIOs should be reviewing their own change control procedures, even if the bulk of the blame is on Crowdstrike since they could easily set aside x% of devices per company to rollout to first.
If I am understanding correctly this was a Falcon content update or signature update not a product update. So this could happen to any vendor in this space and I am pretty sure no one beta test sig updates from malware providers. [Reply]
Originally Posted by dirk digler:
If I am understanding correctly this was a Falcon content update or signature update not a product update. So this could happen to any vendor in this space and I am pretty sure no one beta test sig updates from malware providers.
Interesting, and yeah that's about as non-impactful of a change you can make. [Reply]
Originally Posted by Rausch:
I feel you. Mine passed in 2016. I really wish he could have watched Mahomes...
Same here but it was my mom. Thought we won against Tampa and that was her last year. She got more excited than me about the Chiefs.Then we go on this run. She worked for them and got me hooked. [Reply]
Originally Posted by dirk digler:
If I am understanding correctly this was a Falcon content update or signature update not a product update. So this could happen to any vendor in this space and I am pretty sure no one beta test sig updates from malware providers.
That is correct. Bad Falcon content update. [Reply]
Boot Windows into Safe Mode or the Windows Recovery Environment (putting the host on a wired network and not on Wi-Fi can help).
Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
Locate the file matching "C-00000291*.sys", and delete it.
Boot the host normally.
I was sent this to relay to some remote workers that need help updating teams. I very promptly said that ain't happening, what's your next fix? [Reply]
