MSPs Come Together to Hasten CrowdStrike Outage Remediation

As the world grapples with the fallout from the ongoing IT outage caused by an update to CrowdStrike, I'm reminded of Fred Rogers's quote encouraging us to "look for the helpers" in times of crisis.

Here's the full quote: "When I was a boy and I would see scary things in the news, my mother would say to me, 'Look for the helpers. You will always find people who are helping.'"

Of course, we're adults now, and we should be doing what we can to help. Right now, the MSP/MSSP community is stepping up and showing us that they are just the helpers we need. As Robert Cioffi said in a LinkedIn post, "Someone is having a really bad day. Remember to practice your empathy skills. Be nice. Be helpful. Or BE SILENT!" Because, he added, the community is watching. And they are stepping up to help.

Read the full coverage across our network here:

MSPs Come Together to Help Each Other

In the r/msp community on Reddit, there are a number of threads offering advice, support and boots-on-the-ground help -- everything from tips to manually remediate to offers of time and energy spent manning phone lines and remediating endpoints.

Syncro's Kristen Costagliola, CTO of MSP tools platform Syncro, said it was impressive how quickly MSPs jumped in with local offers of assistance. "Because so many of these endpoints need to be manually remediated, I'm seeing providers saying, 'If you need help in X area, I am available.' They're stepping in to be 'boots on the ground' if an MSP can't because of their geographic location," Costagliola said.

This is what the channel is all about: Community. Pulling together to help each other.

In the meantime, vendors and solutions providers also are offering advice. Andrew Costis, chapter lead of the Adversary Research team at AttackIQ, noted that “CrowdStrike ... released a workaround that includes booting into Safe Mode, deleting a particular file relating to the update, and rebooting. This workaround works for some, but not all, due to: 

He added that this incident also affects different Windows versions with different patch versions, so it’s TBC by CrowdStrike once they carry out a full investigation.

Paul Laudanski, director of security research at Onapsis, agreed, and added that for individual hosts unable to access the fix, the workaround is a reboot. Of course, rebooting production systems can be no small task and, in this instance, will involve time-consuming manual intervention where systems need to be started in Safe Mode and a specific, CrowdStrike driver file needs to be removed, Laudanski said.

"In order to be operational, hosts must be rebooted with the new fix. This process could lead to significant operational disruptions, leaving a window for hackers to take advantage," he said.

There's a big risk of phishing attempts related to the incident as well, said Evan Dornbush, former NSA cybersecurity expert, so it's important to warn clients to be vigilent.

“This is, of course,  a phishing attack opportunity. Don't make a bad situation worse. Only follow recommended instructions direct from your CrowdStrike rep," Dornbush said. "There will be a lot of misinformation about how to reconfigure your computers or which critical system files to delete. Don't fall victim to downloading phony solutions. Similarly, this is a great time to reflect on password management, since the fix may eventually require administrative access to systems that have not rebooted in quite some time.”

CrowdStrike has already made a fix available: https://www.crowdstrike.com/blog/statement-on-windows-sensor-update/. If your company is impacted, contact CrowdStrike through your official support channel and work with them to ensure you get back online quickly.

And remember, keep looking for the helpers. Good luck out there, folks!