IT outage

[Stu]

SWMBO when to town earlier and many shops were only taking cash its a good job the cash machines were still working.

Turned out, it was rubbish in the village. It was only Waitrose. However, those people do shout loudly.
As you said, the Post Office and Cash Machines were working just fine.

We also have a Lidl. They were completely unaffected.

 

[Agentgonzo]

Turned out, it was rubbish in the village. It was only Waitrose. However, those people do shout loudly.
As you said, the Post Office and Cash Machines were working just fine.

We also have a Lidl. They were completely unaffected.

Waitrose was unaffected in our village. Morissons closed for the day

 

[Stu]

Waitrose was unaffected in our village. Morissons closed for the day

Interesting. I'm told Waitrose and Morrisons were the main 2 supermarkets affected.

I've been trying to mitigate risks all day - we deal with a LOT of banking and financial stuff for multiple customers and frankly, other than one of our suppliers needing to reboot part of their system to uninstall the patch, we've had no issues all day.

Other companies have been floored by it.

 

[Chippy_Tea]

Morissons closed for the day

Morrisons was mentioned in the report early today wonder why some stores are effected and others are not.

 

[Chippy_Tea]

other than one of our suppliers needing to reboot part of their system to uninstall the patch, we've had no issues all day.

They were saying some businesses may be effected for days some are having to go into safe mode to get things sorted OK if you have people who know about this stuff not much good if you are a small company with no IT department, i wonder if crowdstrike are going to have yo pay compensation for lost business.

 

[Agentgonzo]

Morrisons was mentioned in the report early today wonder why some stores are effected and others are not.

It sounds like computers that had crowdstrike installed would crash/blue screen.

Presumably any company that used crowdstrike used it _everywhere_. If so, huge swathes of their IT systems would be offline, from payment terminals to management software etc.

Presumably the companies suffering used crowdstrike, and those that aren't didn't

 

[Rodcx500z]

And they talk about going cashless

 

[Chippy_Tea]

Presumably any company that used crowdstrike used it _everywhere_. If so, huge swathes of their IT systems would be offline, from payment terminals to management software etc.
Presumably the companies suffering used crowdstrike, and those that aren't didn't

CrowdStrike is what's known as an "endpoint security" firm as it uses cloud technology to apply cyber protections to devices that are connected to the internet. This differs from alternative approaches used by other cyber firms, which involve applying protection directly to back-end server systems.5 hours ago

 

[Agentgonzo]

CrowdStrike is what's known as an "endpoint security" firm as it uses cloud technology to apply cyber protections to devices that are connected to the internet. This differs from alternative approaches used by other cyber firms, which involve applying protection directly to back-end server systems.5 hours ago

That sounds like it's lifted from their publicity material.

It translates roughly as "cloudstrike remotely Installs their software on all [your company's] computers that are connected to the internet. This differs from other companies that [might] install their software directly [like it's still the 90s] on some [WTF?] of your computers"

 

[Chippy_Tea]

That sounds like it's lifted from their publicity material.

I got it from this site -

What is CrowdStrike and what does it do?​

CrowdStrike is a cybersecurity vendor that develops software to help companies detect and block hacks. It is used by many of the world’s Fortune 500 companies, including major global banks, health-care and energy companies.

CrowdStrike is what’s known as an “endpoint security” firm as it uses cloud technology to apply cyber protections to devices that are connected to the internet.

This differs from alternative approaches used by other cyber firms, which involve applying protection directly to back-end server systems.

“Many companies use [CrowdStrike software] and install it on all of their machines across their organization,” Nick France, chief technology officer at IT security firm Sectigo, told CNBC’s “Squawk Box Europe” on Friday.

“So when an update happens that maybe has problems with it, it causes this problem where the machines reboot, and people can’t get back into their computers.”

https://www.cnbc.com/2024/07/19/what-is-crowdstrike-crwd-and-how-did-it-cause-global-it-outages.html#:~:text=CrowdStrike is what's known as,to back-end server systems.

 

[Stu]

Think of it being like your anti-virus software on steroids (they do a PC version too) that updates itself every day (sometimes more than once a day).

What generally happens with groups of servers is that there is a "staging" server that downloads the patch from the internet, then delivers it securely to servers. So they don't actually need to be internet connected (which is of course really dangerous). If you're lucky, by the time you spot and issue, you can shut down the staging server without it doing too much damage.

Now thinks of this anti-virus software deciding that part of Windows is actually a virus, so it kills that process. Except that the process is key to letting the machine work. So it crashes and shuts down.

That's basically what happened here.

The problem is, these servers individually need to be brought up into safe mode and manually fixed.

To put into perspective, if this happened to us, we'd be talking about 40,000 servers that we're responsible for alone.

 

[Agentgonzo]

Think of it being like your anti-virus software on steroids (they do a PC version too) that updates itself every day (sometimes more than once a day).

What generally happens with groups of servers is that there is a "staging" server that downloads the patch from the internet, then delivers it securely to servers. So they don't actually need to be internet connected (which is of course really dangerous). If you're lucky, by the time you spot and issue, you can shut down the staging server without it doing too much damage.

Now thinks of this anti-virus software deciding that part of Windows is actually a virus, so it kills that process. Except that the process is key to letting the machine work. So it crashes and shuts down.

That's basically what happened here.

The problem is, these servers individually need to be brought up into safe mode and manually fixed.

To put into perspective, if this happened to us, we'd be talking about 40,000 servers that we're responsible for alone.

The amazing thing is that they didn't spot an issue in testing before the whole world caught fire.

You sound like your in the industry, stu

 

[JockyBrewer]

sounds to be one of the smaller problems facing people today, given the news

My point was more how this reached deep into everything.

 

[Stu]

The amazing thing is that they didn't spot an issue in testing before the whole world caught fire.

You sound like your in the industry, stu

The second point, yes. And in Major Incident and Critical Incident management too! I head it up for my company.
As I said elsewhere, today looked like it could have been a disaster. As it happened, other than some minor inconveniences, we mostly avoided it. Which is weird, because normally these things hit us like a train.

First point, it's not the first time. I won't name the supplier, but an absolutely massive worldwide IT firm threw a change at us which they hadn't properly tested last week. It's honestly more usual than you'd consider. They don't normally have this big an impact.

 

[Chippy_Tea]

The amazing thing is that they didn't spot an issue in testing before the whole world caught fire.

I think someone will be getting sacked on monday.

 

[JockyBrewer]

I think someone will be getting sacked on monday.

If this analysis is correct then this is a very rudimentary mistake, but really should have been caught by some kind of automated review system.