With client-wide infections being reported on a weekly basis, MSPs need to balance prevention with response planning, so when the worst happens, they’ll be ready.
Author: Jonathan Crowe, Director of Content & Brand Strategy at NinjaRMMI recently had the pleasure of co-hosting a webinar with Huntress Labs CEO Kyle Hanslovan covering a topic on every MSP’s mind — ransomware. Specifically, ransomware incidents where attackers have compromised MSPs, used their credentials to hijack the MSP’s legitimate software tools, and abused those tools to deploy ransomware across their entire client base.
Kyle has personally worked with nearly 40 MSPs who have been victims of these attacks. As a result, he’s seen and heard firsthand what owners and technicians go through, and learned what separates the ones who are able to successfully recover from the ones who don’t.
If you missed the live webinar, you can watch a recording any time here.
Update: We’re also hosting a follow-up webinar on Thursday, October 17 at 1pm EDT with incident response (IR) specialists from TrustedSec. They’ll be taking questions and sharing real-world examples of what working with an incident response (IR) provider during a ransomware incident looks like. You can save your spot by signing up here.
Before we dive into Kyle’s takeaways, a quick overview: Ransomware attacks targeting MSP networks have been quickly ramping up since June. They’ve affected MSPs big and small, triggering downtime and disruptions for a wide mix of MSP clients ranging from 22 Texas municipalities to hundreds of dental offices across the U.S.
It’s important to note that, with the exception of the February attacks involving an outdated ConnectWise ManagedITSync plugin for Kaseya VSA, none of these incidents appear to have involved attackers exploiting a vulnerability in MSP software. Instead, they involved the abuse of weak or stolen credentials, which attackers used to hijack whatever RMM or remote access tool the MSP was using.
For that reason, enabling 2-factor authentication is widely seen as an effective deterrent to these attacks. Many vendors (including NinjaRMM) have either already made or are in the process of making 2FA mandatory for their users.
Note: If you’re using RMM or remote access software and you haven’t setup 2FA, stop reading and do that now. It’s truly not a matter of if you’ll see one of these attacks, but when.
While we’re talking prevention, here are two additional recommendations that are super basic, yet incredibly effective. In fact, they can help keep you safe from the majority of today’s most common ransomware attacks:
- Make sure you’re not exposing RDP. According to Coveware, RDP is the most common initial access point for ransomware attackers, accounting for 59% of incidents they studied in Q2 2019. You can find a great guide for securing RDP access here.
- Block macros in Microsoft Office documents downloaded from the Internet. The second-most popular attack vector identified by Coveware was phishing documents, and the most common type of phishing lure is still an Office document designed to trick a user into enabling macros. It’s how victims get infected with Emotet, which deploys Trickbot, which deploys Ryuk, the most active ransomware variant of Q2 2019 according to Coveware. The simple way to prevent all that? Make this simple settings adjustment now.
You can find a bunch of additional things you can do to harden your systems and keep yourself safe in our 2019 MSP Cybersecurity Checklist, but the above are three simple, super critical things you absolutely have do, baseline.
With that, I’ll get off my soapbox. Because what I really want to share are six things Kyle said you should focus on now to make sure you’re prepared to act quickly if/when $#*% does hit the fan.
But first…
Two Reasons You May Not Be Ready
One of the big things to understand about these attacks is that they aren’t like others you may have prepared for. They’re an entirely different class, with a different scope and different implications. Even if you have well-documented procedures for helping your clients deal with ransomware incidents, those are only going to be, at best, partially applicable here.
Why? Two reasons:
1) It’s not them, it’s you
Unlike previous incidents, these ransomware attacks aren’t originating in isolation on one of your client networks. They’re being launched from inside your organization. That means you have to have an entirely different set of procedures designed to lock down your own tools and techs until you can identify, isolate, and contain that compromise. Otherwise, if you launch right into trying to remediate client systems, attackers may simply encrypt them all over again until cut off their access at the source.
2) Everything hits at once
Here’s another point Kyle made during the webinar that really resonated with folks:
You can likely handle one of your clients getting infected. What you likely can’t handle is ALL your clients getting infected at once.
One of the things that has made these attacks particularly devastating for MSPs is the scale. Imagine how stressful and chaotic it can be dealing with one customer infected with ransomware. Now imagine that x 10, 20, 50, or 100. Even larger MSPs can be easily inundated and overwhelmed by the sheer volume of emails and phone calls they have to respond to, the number of systems they have to clean, the amount backups (god-willing) they have to restore.
For many, like the small husband-and-wife-run shop PM Consultants, it can all be too much. As detailed in this piece from ProPublica, in the midst of a ransomware outbreak PM announced to its clients via email that it would be closing.
The company said it had been “inundated with calls” on the morning of the ransomware attack, “and we immediately started investigating and trying to restore data. Throughout the next several days and into the weekend, we worked around the clock on recovery efforts…. However, it was soon apparent the number of PCs that needed restoration was too large for our small team to complete in any reasonable time frame.”
“Receiving hundreds of calls, emails, and texts to which we were unable to respond,” PM ultimately had to throw in the towel, advising customers, “At this time, we must recommend you seek outside technical assistance with the recovery of your data.”
So, knowing that these attacks are a completely different ballgame in terms of implications and scale, how can you prepare? Here are six keys Kyle highlighted that can make the difference between your MSP going under due to one of these compromises, or coming out battered but still breathing on the other side.
1) Be ready to lock down your own tools
If you have more than one customer notifying you of a ransomware infection at the same time that’s unfortunately a big warning sign you’re dealing with one of these attacks. The first thing you need to confirm or rule out is whether your own infrastructure has been compromised and your RMM and/or remote access tools are actively being hijacked.
- To be safe, disable access to your RMM and remote access tool(s) until you can investigate
- Audit for unusual tasks, scripts, policy changes, etc.
- Disable user accounts associated with abnormal/malicious behavior, and terminate active sessions (it’s easy to forget the latter)
- Isolate any endpoints & other accounts associated with those users
- Minimize logging into affected systems using privileged credentials (this lowers the risk of attackers gaining access to them by scraping caches)
- DO NOT shut down affected systems (shutting down risks losing critical forensic data — hibernate instead)
- Backup log files (note: depending on how much you’re logging and the log sizes you’re dealing with, you may only have a short amount of time to capture logs before they roll over)
- Change all passwords (to be safe, apply this to all employees, all accounts)
- Ensure MFA is enabled on all accounts
- Confirm AV/endpoint security is enabled and updated, run deep scan (attackers will often disable AV or other security software)
2) Divide and conquer
During a live incident, a lot of things have to happen very quickly, often in parallel. The only way to stay afloat is to leverage everyone to the full extent possible. And the way that happens is by ensuring everyone knows their role and the key things they’re responsible for BEFORE a crisis hits.
At a high level, key responsibilities fall into three buckets:
Major responsibility #1: Critical decision making
Owner: CEO
Setting the course in terms of:
- Protecting the business (seeking alignment with lawyer, insurance, etc.)
- Establishing priorities (including triaging clients)
- Delegating to the rest of the team (resist the urge to try to handle everything yourself)
- Securing and coordinating with outside help (IR provider, additional techs to increase surge capacity, etc.)
Essentially, you need someone providing strong leadership, making tough decisions, and directing people, and that’s really hard to do if that person is also trying to be directly involved with restoration or perform other roles at the same time.
In the moment of crisis, it can be difficult for some owners to stay in this lane and avoid drifting to technical activities they’re more comfortable with. But in working with owners who have gone through these attacks Kyle stresses that often the best thing an owner can do during an outbreak is to delegate and get out of their technicians’ way. There are simply too many other things that need to happen that no one else on the team can do. A vacuum in leadership can be disastrous.
To give you a place to start with, here’s an example checklist that CEOs/owners can reference. You’ll want to adapt this to fit your company’s specifics.
Major responsibility #2: Communications
Owner: Sales and marketing team, or, if none, CEO and other non-technical staff
The second key need during a live incident is communications. As soon as ransomware hits your affected clients are going to need to know you’re aware of the situation and you’re actively working on it. Otherwise, if they feel like you’re ignoring them or that you’re not in control of the situation, they’re going to panic. They may even to try to assert control, themselves. That’s when things can quickly go from bad to worse and when lawyers can start really getting involved.
To prevent all that from snowballing, you need to have people on your team dedicated to establishing and maintaining clear communication channels. That includes:
- Outbound communication with clients: To ensure affected clients are getting personal outreach and regular updates, Kyle recommends instructing everyone who isn’t a technical employee to put on their account management hat and hop on the phone/email. In addition to 1:1 outreach, it may or may not be appropriate to keep your company’s website, Facebook page, etc. updated with notices, too. Just make sure all updates and outreach (public or private) is cleared by your lawyer first. Having pre-approved scripts and templates you can build from will save you critical time.
- Inbound communication with clients: PM Consultants serves as a big cautionary tale. They simply weren’t ready to handle the tidal wave of incoming calls, emails, and texts they received from their clients. How can you make sure you don’t fall victim to the same mistake? For starters, proactive outbound communication can help reduce the volume of inbound requests you have to deal with.Another thing that can lift the burden is setting up automated voicemail messages and email responses. These can be super simple (though, again, it’s a good idea to run them by your lawyer). Simply stating that your team is addressing a security incident, assuring clients you’ll be in contact with them, and letting them know where they can go for regular updates (your homepage, Facebook page, etc.) will help reassure clients that you’re aware of their issue and temporarily mitigate their desperation to get in touch. Just keep in mind that any public notifications will have the potential to draw the attention of external parties, which brings us to the third category of communication…
- External (press, public forums, etc.): One of the primary goals of your communication efforts will be shaping the narrative — for your clients, but also for outside parties that may decide to report or publicly comment on your situation. That means making sure you don’t get caught flatfooted when a reporter calls, or when discussion about your situation breaks out on public social platforms like Facebook or Reddit. Trying to bury or control these conversations can be a losing battle (and a major distraction), but being prepared with pre-approved talking points and knowing what basic details you can and can’t share may help rein things in and clear up misinformation. Conversely, if all anyone hears from you is silence, people have a tendency of filling in the blanks, themselves. And most of the time, they’re a lot less charitable.
Major responsibility #3: Containment & restoration
Owner: Lead technician
Last but certainly not least, the third major responsibility you need to assign is obviously stopping the attack and getting your clients back up and running. That starts with locking down your own accounts and tools, then locking down affected clients, as well. The initial goal is to stop the bleeding and ensure the attack is contained before diving into active restoration. In fact, it’s strongly advised to engage with an incident response specialist immediately following initial containment efforts (more on that below).
In addition to isolation and backup recovery, there’s also forensics gathering and other investigation activities to consider. Depending on how many infected clients you’re dealing with, you’ll likely want to divide these tasks up among your techs. On top of that, you may also have clients that haven’t been affected, but who still need usual support. If everyone has a clear understanding of who’s responsible for what heading into a ransomware situation, the team will be able to move much more quickly and effectively when it counts.
3) Develop SOPs
Knowing who’s responsible for what is half the battle. The other half is knowing step-by-step what those responsibilities actually dictate you do. To act quickly in the middle of a ransomware $#*% storm everyone on your team needs to have standard operating procedures they can immediately turn to that remove as much of the guesswork for them as possible.
Some extremely basic rules for SOPs:
- They can’t exist in people’s heads. They need to be documented.
- They need to be readily accessible (that means having isolated copies safe from encryption).
- They need to be drilled.
- They need to be updated. If it’s been a year since you looked at them, set aside some time this quarter to review.
4) Be ready to triage
As Kyle explains, one of the hardest parts of responding to a client ransomware outbreak is the fact that, in most cases, you can’t help everyone at once. As difficult as it may be or as callous as it may sound, you initially have to prioritize some clients over others. To do that, you’ll need a system of assessing need and other factors to determine who gets your attention first.
Kyle compares it to the triage training he received in the military (with obvious differences).
The chart below was modeled after START, a system developed to guide medical triage in mass casualty situations. There may be different questions or a different flow that’s more appropriate for you and your clients. The point is, determining how exactly you’ll handle client triage is something you need to be thinking about.
5) Leverage outside help
Chances are, you don’t have the necessary staff on hand to recover from a client-wide ransomware outbreak quickly or effectively. The people you do have are great. They’re extremely smart and capable. But they likely aren’t all security experts, and they likely can’t clone themselves at will. The sooner you acknowledge that a situation like this will require outside help, the sooner you can go about getting the appropriate phone numbers on your speed dial.
Here are a few of the external resources you’ll need to be able to count on and who you should lay the groundwork with now:
- Your attorney: There’s an obvious risk of litigation with security incidents, so your lawyer should really be the first person you call. Ideally, however, that call won’t be the first time the two of you have broached the topic. Take the opportunity now to ask your lawyer how he or she would be able to help you navigate responding to a mass ransomware incident, particularly regarding communications. If he or she doesn’t have any direct experience advising clients dealing with cybersecurity incidents, does he or she have a colleague you can speak with who does?
- Insurance provider: If you don’t already have cyber insurance that covers first and third-party liability strongly consider looking into it right away. In many cases, your insurance provider may dictate which incident response specialist you utilize. The majority have a list of particular specialists they’ve pre-approved — similar to health insurance providers designating certain doctors as in-network. If you go with a specialist who isn’t on that list it doesn’t mean their work won’t be covered, but it does raise that possibility.
- Incident response (IR) specialist: No knock against your techs, but when you’re dealing with every single one of your clients infected with ransomware at once the sheer scope of remediation work and expertise required means you’re best suited calling in outside help. Not only will it help ensure a faster, more effective recovery, it will also help you to demonstrate you did everything you could should you find yourself having to prove that in court. Incident response isn’t cheap, but the good news is insurance can cover some or all of the cost depending on your plan and provider. You may also be able to get a better rate if you engage with a IR provider prior to the day-of when everything’s on fire, you’re desperate, and your options are limited.
- Additional technicians: In addition to having an IR specialist on speed dial, it’s also a good idea to have other techs you can call in, if needed, to increase your surge capacity. Ideally, these will be people familiar with your company who you know and trust, so if you’re buddies with a fellow MSP consider working out an agreement where the two of you can rely on each other for help if things ever hit the fan for either one of you.
6) Prep now so you don’t have to pay or pray later
The recurring theme of the webinar and this post is it pays to be prepared for the worst.
The good news is these latest attacks appear to be thwarted by enabling 2FA, so everyone should absolutely stop what they’re doing, audit their environment, and confirm 2FA is enabled across the board now.
But that’s still not a replacement for having a well thought-out and practiced incident response plan in place. Attacks evolve. Defenses that work today will be bypassed tomorrow. You need to adapt your defenses, but you also need to be ready to react.
Don’t get discouraged by thinking planning has to be a massive task. Kyle suggests a great way to get started is by taking an afternoon to sit down with your team and simply asking the question, “Ok, so how would we deal with this?”
It can be informal. Grab some beers and a whiteboard. Get talking. You may be surprised by how many action items you’re able to quickly come up with. From there, chip away. Security isn’t a one-and-done activity. It’s all about making incremental progress, circling back, and making a little more.
Take advantage of the time you have now, when your house isn’t on fire. For every minute of prep you spend, you’ll potentially save yourself hours of chaos and headaches if and when disaster does strike.
What’s Next?
If you liked this post or thought it was useful, please consider sharing it with others. You can watch Kyle and I discuss this topic in even more detail by watching a recording of our webinar here. As mentioned at the top, we’re also hosting a follow-up webinar on Thursday, October 17 at 1pm EDT with incident response (IR) specialists from TrustedSec. Sign up here for the chance to ask them any questions you have about working with an IR provider. It’s certain to be an interesting discussion.
Jonathan Crowe is director of content & brand strategy at NinjaRMM. Read more NinjaRMM blogs here.
