Someone started a DC thread but I think this is worthy for out in the Lounge. If you hadn't heard of CrowdStrike you will know them this weekend. The hospital I work for is fucked right now. The fix is to manually go touch every PC in the hospital. We have thousands of PCs. What a nightmare for the desktop/sysadmin guys.
Originally Posted by :
(Reuters) - A global tech outage was disrupting operations across multiple industries on Friday, with airlines halting flights, some broadcasters off air and services from banking to healthcare hit by system problems.
While major U.S. airlines - American Airlines, Delta Airlines and United Airlines - grounded flights, other carriers and airports around the world reported delays and disruptions early on Friday.
Banks and financial services firms from Australia to India and Germany warned customers of disruptions.
In Britain, booking systems used by doctors were offline, multiple reports from medical officials on X said, while Sky News, one of the country's major news broadcasters was off air, apologising for being unable to transmit live, and soccer club Manchester United said on X that it had to postpone a scheduled release of tickets.
The former head of Britain's National Cyber Security Centre Ciaran Martin told BBC Radio that an update to a product offered by global cyberscurity firm CrowdStrike appeared to be affecting operating systems based on Microsoft's Windows Operating System.
Microsoft's MSFT.O cloud unit Azure said it was aware of the issue that impacted virtual machines running Windows OS and the CrowdStrike Falcon agent getting stuck in a "restarting state," amid an ongoing global outage.
"We're aware of an issue affecting Windows devices due to an update from a third-party software platform. We anticipate a resolution is forthcoming," a Microsoft spokesperson said.
According to an alert sent by CrowdStrike to its clients and reviewed by Reuters, the company’s "Falcon Sensor" software is causing Microsoft Windows to crash and display a blue screen, known informally as the "Blue Screen of Death".
The alert, which was sent at 0530 GMT on Friday, also shared a manual workaround to rectify the issue.
Over half of Fortune 500 companies used CrowdStrike software, the U.S. firm said in a promotional video this year.
A Crowdstrike spokesperson did not respond to emails or calls requesting comment.
There was no information to suggest the outage was a cyber security incident, the office of Australia's National Cyber Security Coordinator Michelle McGuinness said in a post on X. A British government source also told Reuters there was nothing to suggest foul play.
"The world grinding to a halt because of a global IT meltdown shows the dark side to technology," AJ Bell investment analyst Dan Coatsworth said.
"The severity of the problem boils down to how long it lasts. A few hours' disruption is unhelpful but not a catastrophe. Prolonged disruption is another matter," he said.
The outages rippled far and wide.
Airports in Singapore, Hong Kong and India said the outage meant some airlines were having to check in passengers manually.
Amsterdam's Schiphol Airport, one of Europe's busiest, said it was affected, while airline Iberia said it had been operating manually at airports until its electronic check-in counters and online check-ins were reactivated. It said there had been some delays but no flight cancellations.
Air France-KLM said its operations were disrupted.
The Dutch foreign affairs ministry told Dutch press agency ANP it had been affected. A spokesperson was not immediately available for comment.
While there were reports of companies gradually restoring their services, analysts weighed the potential of what one called the biggest ever outage in the industry and the broader economy.
"IT security tools are all designed to ensure that companies can continue to operate in the worst-case scenario of a data breach, so to be the root cause of a global IT outage is an unmitigated disaster," said Ajay Unni, CEO of StickmanCyber, one of Australia's largest cybersecurity services companies.
Just got my ETL servers back up and running. Luckily it doesn't appear we have any corruptions in our database. We are kind of lucky as these issues happened right in the the middle of ETL last night. I know there are several other hospitals that have corrupt databases. These aren't small databases either. Our database if just over 5 Terabytes but I know we are relatively small in the hospital world. I suspect some of these other hospitals have 40 or 50 Terabyte databases. They take some time to recover those. [Reply]
My company uses CrowdStrike for our production servers and we were impacted by this. Several guys on my team were working until 3:00AM this morning. One poor guy had to drive to each data center around midnight, but we are finally back up 100%.
As the new guy, I'm not in the on-call rotation yet, thankfully. [Reply]
Originally Posted by SPchief:
I was sent this to relay to some remote workers that need help updating teams. I very promptly said that ain't happening, what's your next fix?
It worked for me and a lot of people... Can you not get a safe boot to get to a command prompt? [Reply]
Originally Posted by SPchief:
I was sent this to relay to some remote workers that need help updating teams. I very promptly said that ain't happening, what's your next fix?
There are a few possible fixes, but they are all pretty painful to do remotely:
Boot to safe mode with networking, the update has been rolled back and from what I've read that it should be able to recover after CrowdStrike reconnects
Boot to safe mode, delete the .SYS file manually. These may require a Bitlocker key if the drive is encrypted
It's possible to delete the .SYS file remotely during a 5-second window before the crash via command line. This would require another machine on the same network with remote admin credentials
Boot to a boot disk, such as Hiren's. Manually delete the .SYS file
Originally Posted by BigRichard:
The hospital I work for is fucked right now. The fix is to manually go touch every PC in the hospital. We have thousands of PCs. What a nightmare for the desktop/sysadmin guys.
I'm in the same situation. Fortunately I have off today and don't have to worry about it. My phone was blowing up with messages saying exactly this. [Reply]
No the world didn't end and I'm sure guys, they are mostly guys, with smiles on suits will say lessons have been learned...
But our society/world definitely has some brittleness when one company can shut hundred of millions (billions?) of computers in the span of a few hours.
There is a lot of risk in low probability but high impact events that our modern society isn't dealing with very well. [Reply]
Roughly 25% of our endpoints were affected by this. We had a great team effort in getting everyone back in service - took us about 4 hours, but we are a small shop with a large IT staff. [Reply]