Tips · 9 MIN READ · YANEK KORFF · FEB 19, 2019 · TAGS: How to / Managed security / Planning / Selecting tech / Tools
Over the last 20 years, we’ve heard all kinds of interesting questions as prospective customers evaluate which type of managed cybersecurity service is right for them. The questions are often buried in a big spreadsheet, otherwise known as a request for proposal (RFP). Some of them are remarkably well thought out and put together. However, the vast majority follow a well-worn path and are kind of predictable (check out Gartner’s MSSP RFP Toolkit for some of the greatest hits).
But the thing about predictable questions is they generate — you guessed it — predictable answers that leave one provider sounding a lot like the rest.
So in an attempt to arm you with a few questions that’ll make your prospective MSSP or managed detection and response (MDR) provider stop and think, we’ve compiled a short list of revealing questions that we think any service provider should be able to answer with flying colors. (Although sadly, we find that many don’t.)
Without further ado, here we go.
Can you provide an example of ways you’ve adapted your service to your customers’ environments?
You know as well as we do that one size doesn’t fit all. Your industry, your geography, your company, your strategy, your tactics, your team … all of these variables mean every company is different. Even if you find a service provider that’s a good fit today, will they adapt so they can be a good fit tomorrow? How will they continue to tune their service so you’re always getting what you need?
Many providers will talk about “business context.” It’s a bit of a holy grail to security service providers so make sure you understand what it is and how it works. Can your provider differentiate an attacker from that weird PowerShell blip when Jenna the sysadmin runs her same PowerShell command every Wednesday morning? Can they react faster if the CFO gets phished? Are they able to ignore PUP/PUA at one customer because it’s noise, but report it every time at another because it’s the CISO’s priority?
Without this ability, over time you’ll feel like you’re being served the same gruel day after day.
How long, on average, did it take to fully onboard your last 10 customers, and at what point did you consider the onboarding complete?
There are few activities in the managed security space that evoke more dread than onboarding. Notorious for exceptionally long, complex and error-prone disasters rife with miscommunication, onboarding roadmaps and project plans can get complex quickly. What’s worse, success may mean one thing to the provider and something else to you.
But it doesn’t have to be this way. During the RFP process, make sure you understand what activities mark the onboarding process as being complete and ask your provider how long it took them to go through that process for their last 10 customers. Get real data. Or, even better, ask the provider if some of these customers can be references and validate this data.
Remember, onboarding time has three components: calendar time (end to end how long it took), your organization’s time (how much new customers have to do, and how long it takes) and the provider’s time (you should care about this because it contributes to component #1 — calendar time).
One last #protip for ya: ask your provider if you’re going to have to pay for service during onboarding.
Can you use my existing security technology or will you require that we implement new technology?
You’d think this one would be obvious, but many providers will mandate that you either buy new technology, add their technology (because they won’t use what you already have) or introduce a duplicate technology (usually their SIEM) because their architecture demands it.
A service provider in this space should be using the technology you already have in play and operationalizing it. That means ingesting the data your security products are already producing, analyzing that information and delivering answers about what matters and what doesn’t.
Now, not all technology is created equal. Some categories of security tech are best suited to detection, other categories are more useful when you’re investigating an incident or proactively hunting for bad things in your environment. You’ll want to make sure the tech you have in place can actually do what it needs to do.
That said, this shouldn’t come across as a requirement from your MSSP or MDR — a provider should not tell you that you need to buy this and that for anything to work. Instead, you should get a higher fidelity answer like: “Without an endpoint detection and response (EDR) tool, our ability to investigate will be limited, as will our hunting capability — some of which relies on EDR.”
How does your detection and response strategy differ among on-prem technology, cloud infrastructure and cloud applications?
“We monitor your AWS, Azure and O365 environments for threats and respond immediately!”
Have you heard this one before? This isn’t an answer.
The way you differentiate between providers that “speak cloud” and those that don’t is by listening closely to their detection and response philosophy. What’s different about security in the cloud versus on-prem? How are the approaches they take for static versus elastic cloud infrastructure different? Or are they? What about cloud applications? How do they think about the security of configuration settings versus the security of data residing in containers?
Validating a security provider’s ability to handle your cloud security is one of the more challenging aspects in the assessment process. Consider looping in people from your own organization that are responsible for your cloud strategy and implementation. They’ll ask good questions and can help you evaluate the answers you receive.
How will we work together during a security incident?
When a security incident arises, communication is key. You and your service provider begin in a fog of war. Keeping exceptional clarity on “what we know” and “what we don’t know for sure yet” is essential to navigate the investigation and response process that follows.
Understanding how your provider will communicate this info (and how quickly) is important. Do you have to log into a portal and review a mostly static page updated once every few hours? That’s a useful artifact, but not a useful communication method. Do you submit a ticket? Ugh. Instead, look for effective methods that include rapid info sharing and multi-person communication.
Of course, during an incident you’ll have to communicate with all sorts of people — inside and outside of your organization. Your service provider might have relationships with law firms who have experience in breach communications. They may also have relationships with incident response providers who can show up on-site at a moment’s notice. Either way, do your own research and find firms that are a good fit for your organization. Of course, it’s always easier to do this before an incident than during one. Running your own incident response tabletop exercises can reveal a lot (we’ve even created a role-playing game to try and make it fun — give it a go and let us know what you think).
Can you provide an example of a time you learned something from a customer that improved your service?
A security service that fails to learn and grow isn’t actually a security service.
It’s … well, we’re not sure what it is, but at the end of the day it’s pretty useless to you.
Sure, it might provide the illusion of security, but in reality there’s a lot of time spent turning cranks that produce nothing. We’ve heard this complaint from more than a few CISOs: “My MSSP is a black box. I put my money in and nothing comes out.”
Your prospective service provider should have crisp examples of how they’ve learned and improved the way they help all of their customers. And it should be material. Not something simple like, “I found this threat here so I added it to my intel database.” That’s table stakes.
What caused your service provider to rethink something and say to themselves, “I think the way we’re tackling this is wrong based on this customer feedback … let’s do it differently?” Demonstrating the ability to adapt ensures your service provider will grow with you.
How will you give me the visibility I need to be confident that you’re making the right decisions for my organization?
Don’t just trust, but verify. It’s what you’re paying your service provider to do after all, so you should have confidence not only that they’re doing the right thing … but that they’re doing it right too.
Take a moment to think through the steps that comprise “security operations.”
- Triage. This is the process analysts go through to evaluate (often quickly) whether something is a false positive or warrants investigation. Sometimes these analysts are humans. Sometimes they’re robots. Does your provider tell you both who made the decision and why? If they filter out something important very early but were wrong, that’s a problem.
- Investigations. Will your provider show you what information their analysts pulled from your environment? Can you get a sense of the thought process they use to decide what to retrieve? And what to make of it? This is where expertise really comes into play.
- Reporting and response. Is the output you receive easy to understand? Are response actions clear, and do you have control over who-gets-to-meddle-with-what in your infrastructure? If you have to translate everything your provider is telling you so that mere mortals who don’t speak security can understand it, that’ll become frustrating … fast.
As you take a step back and look through what’s been done, does the provider have timestamps for every step that was taken so you can evaluate this information and measure whether their overall performance is improving or degrading?
Ultimately, you have to answer this question: Did they show their work? That’s the only way to verify that they’re doing what you’re paying them to do.
When things start to break, how (quickly) do you find and fix the problem? When do I find out about it?
If you’ve worked with an MSSP before, you’re familiar with this problem we’re about to summarize.
Nine months after a piece of technology stopped sending data, the provider found out it was broken. Because you told them. That’s a big hit to your visibility and a lot of risk you took on without any warning. Not cool.
How will your new prospective provider handle this? Can they detect when a device becomes unreachable? How fast? What about if the device stays online but stops sending data? Or worse – what if there’s a significant and unexpected drop in data volume?
Who’s responsible for monitoring this stuff and how quickly can they recover? Get examples if you can, and bonus points if they provide you direct visibility into this kind of monitoring.
How did you identify and report on an active red team engagement conducted on one of your customers’ networks?
Yeah, we know this one feels pretty specific, but we’ve run into too many instances where customers brought in a relatively sophisticated red team partner only to discover their managed security provider was blind to these mock adversaries. They couldn’t even detect them, let alone investigate or respond. To be clear, when we say red team, we’re talking about a group of whitehats who try to break into your network, escalate privileges, move laterally and steal stuff … and then report on things you can do to improve your defenses.
Can your new potential partner provide an example of this exercise playing out? How did they detect the “attacker” in this case and to what extent were they able to provide ongoing reporting? Once again, bonus points for the provider if they’ll let you hear all of this directly from one of their current customers.
When I have a question or concern how do I engage with your team?
We talked about communication during an incident. What about when there’s no incident? Is it the same process, or are there two different processes? The more you have to adapt to your provider’s modes of communication, the less likely you’ll remember to do the right thing when the time is right.
Watch out for laggy ticketing systems and be cautious about support portals where the identity of the people you’re talking to is hidden. Your partner’s security analysts will have exceptionally generous access to your data. You should be able to get to know who they are and interact with them directly from time to time.
Can you show me how you calculate the price of your service?
Every provider will give you a price. But can you understand how and why they got to that number? Be wary of long rambling answers. If your prospective provider can’t give you a crisp answer or, better yet, quote you a price on your first sales call, imagine how the conversation will go once you become their customer.
If selected, can you provide a free 30-day proof of concept to demonstrate you can deliver on the expectations you’ve set?
After you’ve asked all of your questions, appraised the responses and picked a winner there’s a good chance you’ll still be asking yourself, “Can they really do all of these great things in my environment?”
Exaggerated sales and marketing claims are, unfortunately, one of the biggest scourges on the security industry. You don’t want to get a few weeks into a new agreement and learn your new provider can’t do everything they promised or, even worse, find out when they missed something important.
One of the most effective ways to mitigate this risk is to hop on your provider’s service on an interim basis. It gives you a chance to get a feel for what the interactions will be like and gives your potential partner an opportunity to prove themselves.
And if your prospective service provider can’t even get this operational within 30 days? Well, that tells you all you need to know.
So there you have it. Twelve questions that can help you sleuth out what it will be like to work with your managed security provider.
If you’ve got other questions, we’d love to hear them. Or if you’re reading this and thinking “maybe I’ll just build my own SOC,” check out our post on all the things you’ll need to consider if you’re thinking of building a 24×7 SOC.
