Crowdstrike screwed the pooch

For those with similar jobs... wow this Crowdstrike issue is a huge deal! The ~20K person company I work for has been working through restoration since 6AM and it will likely take most of the weekend. Can you say law suit?

https://www.msn.com/en-us/travel/ne...e-windows-glitch/ar-BB1qg3MD?ocid=socialshare

 

Was it the Falcon software thing that glitched?

 

They pushed a bad client update that screwed windows up (I use a MAC ). Layer whole disk encryption on top of that and you get a PC that won't boot! You have to dig in and manually delete the bad update file, which can be challenging with whole disk encryption. It's been a nightmare for the support team. Thankfully I don't do the day to day stuff anymore.

 

Was it the Blue Falcon update?

 

My graveyard shift guy left pretty upset. All the additional work he had to do for the airlines. All I said to him on the way out was, bet it was a short night, lol.. I got the respond I deserved for that one, which was the bird..

 

Its noon my time and I am finally getting to my last few VMs from this fiasco. My teams have been working for 10 hours.

I am less than happy after I planned to have virtually no work outside of 2 meetings today, and was going to be out by 10A, then my phone exploded a few hours after I went to bed.

 

Sheet show.

 

Luckily for me, I have managed to get about half my infra moved to linux. Some things are just stuck. I cant imagine the fallout if all my stuff was Windows.

My Linux / Unix systems stayed up with to issues outside of auth (ADFS), once the auth was fixed, about half my stuff just started working again. The rest has been effort.

 

Windows sucks.

 

The impact to healthcare was/is real and adds another notch on the justification belt for why medical professionals need better compensation. I'm not there, but as I'm hearing from my spouse, it's ugly and sent the majority of the remaining OGs to their school days. The fail-safes failed first, and something got auto approved for production. They watched every system go offline like dominos. And this is not a small office setting.

 

As @Coal Dragger so eloquently stated above... Windows Sucks! The world needs to stop adopting Microsoft as a standard.

 

Funny how SouthWest - FedEx and others using Windows 3.1 to run critical operations scheduling survived and Thrived today LOL

 

Crowdstrike and bill gates? No way

 

I was hoping somebody brought this up to educate me on what went down. My wife sent me a news article this morning outlining a "worldwide tech outage", but details on what actually happened were scarce. I wondered why my Windows laptop was on waiting for a login pin when I hopped in my truck this morning (the Windows machine has been relegated to the shop where I can use it for HP Tuners).

But Windows/BillGates bashing aside, was it really as simple (and boneheaded) as @Reddawg1100 said? I mean, I've updated computers and phones that I've had to roll back because things were messed up and, for example, the machine would crash when loading certain programs or attempting to use certain keyboard shortcuts. But a WORLDWIDE TECH FLOP??? Geez, that was easy to lock up machines, crash markets, shut down hospitals, disrupt transportation, and cause general madness and mayhem.

Thanks, BTW, for providing simple details of what happened. All I could find in my cursory searches were new reports of a tech issue going awry and causing problems.

 

It's essentially a supply chain problem at its finest and another shot across the bow regarding how fragile everything is.

Will we learn more in the coming days? Possibly, but a tough day for that Crowdstrike team responsible for config mgmt and testing/QA of a release candidate.

Worse day for consumer teams responsible for business continuity.

 

Pardon my ignorance, but how so..? Asking honestly, not combatively. I thought it was what amounted to bad code?

 

What really sucks for me is, my wife is trying to fly home from D.C. (she did make it to Atlanta), AND I have money invested in Crowdstrike!

 

Not a fan from the beginning...yeah right!

 

Yes, as I understand it, it was bad code. But the supply chain part is a matter of how we're at the mercy of anyone within these companies doing any work on our final consumer products. The trust but verify part was severely missed here and it just goes to show the impact one failure in a seemingly simple task could have terrible impacts to the larger effort.

There is supply chain risk everywhere. Takata air bags is an example in a different segment. Boeing panels on planes could be considered another. This is just slightly more different because it initially came across as a Microsoft specific issue where it was a Microsoft vendor who crippled their customer, and then all of their customers.

We've all seen it - weekend/holiday push, massive staffing turnovers, out of date or non-existing continuity plans (e.g. Step by step instructions for newbies to follow), internal safeguards being loosely followed because everyone becomes complacent. These are real issues and why safeguards exist.

Sadly, for Microsoft and many of its customers, a vendor supporting Microsoft chose a less rigorous path. So, when you look at supply chain risk management (and not solely with a cyber security hat on), this event has caused almost all of the consequences everyone tries to avoid. Bad day to be on the patch development team at Crowdstrike... Or their insurer.

Ultimately, were only as strong as the weakest link, and in this case, it was the people not following internal processes at a vendor people may or may not have partnered with directly,.

I'm glad it was this scenario and not another Solarwinds, but I hope the world is now on notice about how bad this really could have been. IT working around the clock for the weekend sucks, but happens. We've all been there. But this could have been so much worse, and we're only talking about it because someone got caught. It's peoples entire jobs to help prevent these types of issues when choosing vendors/partners.

 

Our group doesn't use their services, just lucky with that cause we have so many others that push stuff to us. I get in trouble when I bring this up on calls but back in the days I did WUS/SUS admin we carved out a random 10% of the estate for test environment, this was after initial test of an update. If that 10% took the update ok then we release to the world. It seems the norm now is just to trust 3rd party groups that they tested updates and we let our people get updates direct from internet.

Yeah, we don't need no stinkin test environment.

 

It's obvious that this bad antivirus definition that was sent out be Crowdstrike was not tested before being released. That has destroyed Crowdstrike's credibility. Now, all those large companies, who WERE trusting Crowdstrike to test their code before distributing it, will have to test newly released definitions before allowing distribution to the rest of the company. The manager over Crowdstrike's group that creates and issues these definitions should be fired immediately. Even with that, if I was responsible for a critical system, I would think several times before trusting Crowdstrike again.

 

This is their one year chart. You can see a big drop at the end, but not as big as I would have thought. You can also see that it has already started to bounce back:

 

We don't use Crowdstrike but some of our districts do. I remember a similar thing happened with Sophos 12 or so years ago.

I personally don't put a/v products on my servers. Endpoints? Absolutely. We're 99% virtual (VMware) and use NetApp storage with VSC and Rubrik RSC-G.

 

So buy now? Yeah these guys will get their reputation back pretty quickly. I mean look how many times an Office update has crashed systems.

Ah, that's what it was? I was off all week anyway so I missed any discussions.

@FireFinder44 I also remember someone pushing something that wasn't ready for prime time. Hell I can't remember what I had for lunch yesterday.

 

Yea our ups hub was only set behind 2 hours yesterday

 

Crowdstrike has apologized for any inconvenience.............

 

Words mean a lot, and apologizing isnt something that should be overlooked. Lawyers would have agreed to the exact wording. An apology is an admission.