Recovering from the global tech outage could be a long, arduous process | CNN Business (2024)

Recovering from the global tech outage could be a long, arduous process | CNN Business (1)

Travelers wait in Terminal 1 for check-in at Hamburg Airport, in Hamburg, Germany, Friday July 19, 2024. A widespread Microsoft outage disrupted flights, banks, media outlets and companies around the world on Friday.

CNN

The company that caused a massive computer outage across the world says a flawed update has been rolled back – but that doesn’t necessarily help the thousands of businesses that have been affected by the glitch.

The CrowdStrike software issue at the heart of the outage runs at such a deep level in affected computers and systems that getting them up and running just to be fixed will be, in many cases, an enormous challenge.

That’s compounded by the fact that many of the servers that may contain information needed to get these systems working again are themselves caught in a cycle of crashing and rebooting.

And some affected computers might not even be easily accessible, set up in remote locations and intended to run without human intervention.

“I don’t think it’s too early to call it: this will be the largest IT outage in history,” said security expert Troy Hunt in a post on X.

The CrowdStrike software at fault operates at what’s called the kernel level of a computer, a much deeper level than what more ordinary applications such as browsers or video games do. This portion of a device has much greater visibility and control over a computer and its components, making it critical for the operation of all other systems — and far more sensitive.

Running at the kernel level means CrowdStrike’s software can do more to detect cyberattacks, but it also means the current bug is causing Windows computers to crash to a Blue Screen of Death before users can take any actions to correct it.

The issue appears to be recoverable, CrowdStrike has said, but in many cases it requires painstaking work: Each affected device must be accessed by an administrator and manually rebooted into safe mode. Then, the offending CrowdStrike file must be deleted by hand.

For businesses with hundreds or thousands of laptops, desktops and servers running CrowdStrike’s security software, an individual human may have to perform that process over and over and over again.

“You can’t automate that,” said Kevin Beaumont, a security researcher and former Microsoft threat analyst, in a post on X. “So this is going to be incredibly painful for CrowdStrike customers.”

On Friday, a Microsoft status page reported that some Windows Virtual Machine users have successfully recovered from the issue by repeatedly rebooting, in some situations up to 15 times in a row.

“We have received feedback from customers that several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage,” Microsoft said on the page. The company did not speculate as to why the technique appears to work.

Affected organizations can also try to restore their machines to an earlier state by reverting to a previous system backup, Microsoft added, though it acknowledged that may not be possible in all cases.

“Companies that haven’t invested in rapid backup solutions are stuck in a catch-22,” said Eric O’Neill, a cybersecurity expert and former FBI counterintelligence official.

It gets worse.

Organizations that take security seriously will have likely encrypted their computers’ hard drives, making it even more challenging to access the file that needs to be deleted.

For those organizations, “you need to manually decrypt the disk with a BitLocker Recovery Key, which is probably — for most companies — stored digitally on one of the servers that is currently booting over and over,” said Ira Bailey, a security researcher, in a post on BlueSky.

Every affected computer that is BitLocker-encrypted will need to be unlocked with a recovery key before organizations can begin the process of deleting the bad CrowdStrike file and restoring normal operation, said the cybersecurity expert who goes by the pseudonymous handle SwiftOnSecurity in a post on X.

Recovery will be enormously expensive for Fortune 500 companies with large teams of IT staff and likely even more challenging for smaller firms, Kenn White, an independent security researcher who specializes in network security, told CNN.

“If you don’t have physical staff that can actually touch it, this is going to take many, many days for much of corporate America to recover from,” White said. “It’s just a ton of labor-intensive manual work.”

“It’s a fairly complicated procedure for non-technical people,” White added, “and even a lot of skilled IT professionals will find it difficult to do this at the scale that’s going to be required given the number of machines that are affected.”

How did the CrowdStrike bug lead to such widespread effects?

Because CrowdStrike’s security software is running on countless individual computers all around the globe, the update that got pushed to those devices caused them all to shut down, virtually simultaneously.

And in today’s networked economy, an outage in one part of a supply chain can cause domino effects up and down the line. When multiple parts of a supply chain go down, it touches off a cascade of problems.

Imagine a person trying to buy a coffee, said Andrew Peck, a cybersecurity expert at Loughborough University in the UK. What may seem like a simple transaction relies on multiple computers working in tandem, from the coffee shop’s point of sale to the payment processor’s own back-end systems.

“There are a lot of computers in this chain, and usually the larger the business, the larger the chain,” Peck said. “If any one of the computers are down in the chain, the transaction will not complete.”

It could take millions of person-hours of work by corporate IT professionals to fix all the computers that were affected, said O’Neill, the former FBI counterintelligence operative. But, he said, coming up with a firm estimate is difficult because it’s unknown how many computers were affected.

Imagine something like the massive aviation industry, the critical financial services sector or the life-or-death operations of a health care provider, and the scope of the disaster becomes starkly clear.

With many people now working from home, he said, IT professionals can’t just go desk-to-desk to fix different computers. Instead, they’ll have to communicate with individual employees and talk them through the process remotely.

“That magnifies the issue,” he said. “Something that could have been fixed in hours is going to take days.”

Some affected machines may be rarely serviced by people or located in remote areas. Others may not even have monitors or keyboards plugged in, because they don’t regularly require humans to directly interact with them.

The most extreme examples may include weather monitoring sensors or devices in railway signal boxes, Peck said, which could require technicians to physically visit potentially hundreds of thousands of machines to perform the recovery process.

Recovery will cost the world “thousands of hours and millions, potentially billions of dollars,” Peck said, which quickly adds up to “some very exhausted IT support teams burning budget they didn’t have.”

What is Microsoft’s role in all this?

A separate issue earlier, on Thursday, did lead to significant impacts on many of Microsoft’s own cloud customers, but it was resolved overnight and was unrelated to the CrowdStrike issue, Microsoft and multiple cybersecurity experts told CNN.

The CrowdStrike bug may have initially been conflated with the Microsoft issue because CrowdStrike’s error affected only Windows machines.

“Both are Microsoft-related, but Microsoft had nothing to do with the second incident,” White told CNN.

That appears to be supported by Microsoft’s own status account on X, which on Thursday announced an issue affecting “Microsoft 365 apps and services” and a separate announcement Friday addressing the CrowdStrike outage. The two issues are being tracked using different reference numbers.

As of Friday morning, Microsoft said the issue with Microsoft 365 had been resolved and that the situation was improving.

“The ongoing CrowdStrike issue is unrelated to a previous outage in the Central US Azure region on July 18, impacting Azure customers using that region as well as some Microsoft 365 services,” Microsoft said.

Microsoft CEO Satya Nadella acknowledged the CrowdStrike issue in a post on X Friday morning, saying Microsoft is “working closely with CrowdStrike and across the industry to provide customers technical guidance and support to safely bring their systems back online.”

Since the update to CrowdStrike’s software was delivered by the company’s own systems, it appears unlikely that Microsoft bears direct responsibility for Friday’s outages, said Beaumont, who said he reviewed a copy of CrowdStrike’s flawed update.

The problem with CrowdStrike’s update was that it wasn’t formatted correctly “and causes Windows to crash every time,” Beaumont posted on X.

CNN’s Olesya Dmitracova and Chris Isidore contributed reporting.

This story has been updated with additional context and developments.

Recovering from the global tech outage could be a long, arduous process | CNN Business (2024)

FAQs

What is affected by the global tech outage? ›

A major global IT outage industries across the world today with airlines, banks, shops and broadcasters affected. Major U.S. airlines grounded flights and there were global delays.

What business is affected by Microsoft outage? ›

The global outage due to a technical issue around Microsoft's applications and services has largely affected businesses and institutions in different countries. Flight operations, banks, retailers and IT firms were at the receiving end of this IT outage, that has left people in panic.

What caused the worldwide tech outage? ›

A massive outage was caused by what was supposed to be a routine update from the cybersecurity company CrowdStrike. A routine software update caused cascading chaos Friday that has engulfed global businesses from airports and banks to retail and law enforcement.

What caused the CrowdStrike issue? ›

SAN FRANCISCO, July 19 (Reuters) - Security experts said CrowdStrike's (CRWD.O) , opens new tab routine update of its widely used cybersecurity software, which caused clients' computer systems to crash globally on Friday, apparently did not undergo adequate quality checks before it was deployed.

Is Global IT outage fixed? ›

CrowdStrike says significant number of devices back online after global outage. A “significant” number of the 8.5m devices affected by last week's global IT outage are back online, according to the cybersecurity company at the centre of the incident.

What are the effects of network outage? ›

A sudden internet outage can bring a business to a standstill, leading to financial losses that are not just immediate but also have long-term repercussions: a) Loss of Productivity: When employees cannot access online tools, emails, or shared documents, productivity takes a nosedive.

What businesses are impacted by AWS outage? ›

The outage affected several popular apps and services, including Amazon's own Alexa voice assistant, as well as popular fast food chains like McDonald's and KFC, and online shopping sites like Shopify. The incident highlights the growing dependence on cloud infrastructure and the need for better contingency planning.

Is Wells Fargo affected by CrowdStrike? ›

Banks that were temporarily affected included Wells Fargo, Charles Schwab and TD Bank. While most systems are now up and running, the fix for the CrowdStrike outage is manual and will take some time.

How many computers are affected by CrowdStrike? ›

Microsoft has estimated that the incident, which is being described as one of the worst IT outages in history, impacted 8.5m computers around the world. Businesses, banks, hospitals and airlines were among the worst-hit, with some still struggling to fully restore their systems.

Who is affected by Global IT outage? ›

Financial services disrupted. Hospital systems halted. Security software company takes blame for massive problems across the world. Public and private sector businesses around the world continue to be affected by the CrowdStrike outage, 24-hours after it sparked global chaos with airline, bank and other disruptions.

What is technical global outage? ›

A global technology outage caused by a faulty software update grounded flights, knocked media outlets offline, and disrupted hospitals, small businesses and government offices on Friday, highlighting the fragility of a digitized world dependent on just a handful of providers.A faulty software update caused ...

Who caused the global outage? ›

The global outage stems from an update CrowdStrike made to its marquee cybersecurity platform, a cloud-based software product called Falcon.

Does the US government use CrowdStrike? ›

The Defense Department tops a list of 16 agencies reporting a total of nearly $55 million in unclassified purchases of CrowdStrike cyber protection software — cited as the cause of the largest global computer outage following a failed update last week.

What is the root cause of the CrowdStrike outage? ›

CrowdStrike said Saturday a bad “sensor configuration update” in its Falcon cybersecurity platform was to blame for a massive global computer outage. The disastrous patch knocked approximately 8.5 million Windows devices offline paralyzing airlines, hospitals and financial institutions globally.

Who owns CrowdStrike? ›

The ownership structure of CrowdStrike Holdings (CRWD) stock is a mix of institutional, retail and individual investors. Approximately 43.35% of the company's stock is owned by Institutional Investors, 2.19% is owned by Insiders and 54.46% is owned by Public Companies and Individual Investors.

What all is affected by the CrowdStrike outage? ›

A software update by global cybersecurity firm CrowdStrike, one of the largest operators in the industry, triggered systems problems that grounded flights, forced broadcasters off air and left customers without access to services such as healthcare or banking.

What is the Microsoft outage affecting? ›

The massive disruption to Microsoft systems has included flight delays and cancellations, as well as impacting hospitals, banks, supermarkets and millions of businesses.

What is impacted by CrowdStrike? ›

Microsoft outages caused by CrowdStrike software glitch paralyze airlines, other businesses. Here's what to know. Banks, airlines, television networks and health systems around the world that rely on Microsoft 365 apps were hit by widespread outages early Friday linked to the company CrowdStrike.

What was Global IT outage? ›

CrowdStrike said an update “triggered a logic error” Friday but was remediated shortly after. Companies are attempting to pick up the pieces one day after a global IT outage disrupted major industries across the world, including airports, businesses and broadcasters.

Top Articles
Latest Posts
Recommended Articles
Article information

Author: Aracelis Kilback

Last Updated:

Views: 6007

Rating: 4.3 / 5 (64 voted)

Reviews: 87% of readers found this page helpful

Author information

Name: Aracelis Kilback

Birthday: 1994-11-22

Address: Apt. 895 30151 Green Plain, Lake Mariela, RI 98141

Phone: +5992291857476

Job: Legal Officer

Hobby: LARPing, role-playing games, Slacklining, Reading, Inline skating, Brazilian jiu-jitsu, Dance

Introduction: My name is Aracelis Kilback, I am a nice, gentle, agreeable, joyous, attractive, combative, gifted person who loves writing and wants to share my knowledge and understanding with you.