Security operations · 4 MIN READ · JAMES JURAN · APR 11, 2019 · TAGS: How to / Managed security / Selecting tech / Tools
We’ve said it before but it’s worth repeating … when you’re evaluating an MSSP or MDR, you’ve gotta make sure the provider can integrate with the tech you already have and make it work harder for you. (Pro tip: If your MSSP or MDR immediately suggests that you run out and buy a chest of shiny new security tools, they’re probably not the right fit for you.)
Here are four questions to ask your prospective provider to find out if they’re up to the challenge of managing your fleet of security signals.
Question 1: How do you get data from my existing tech?
Most security devices or services stream data via syslog or WebSocket. At first glance, this seems like a great way to collect all those disparate security signals — especially because your provider doesn’t have to write extra code.
Before you hit the “easy” button, dig a little deeper with your provider. Ask some questions like:
- How hard is it for me as a customer to set that up?
- How will I know that it’s working or when it breaks (like when my network admin accidentally removes the firewall rule that allowed my data to get to your collector)?
- What happens when your collector is down? Will the one high-priority alert that indicates attackers are in my network get dropped on the floor and delay discovery of the intrusion?
- Can I ask the device or service for more information to support an investigation, or can I only receive the information that you chose to put in the streaming protocol?
Instead of streaming data, we prefer to poll for alerts. Yes, it takes more work, but it’s much more reliable and it lets us audit our data ingestion processes to make sure we’re not missing anything mission critical (BTW, we’ve got a whole post on setting up a rockin’ data auditing process).
Here are a few reasons why we think polling for data is more effective and reliable:
- When there’s an interruption for any reason, you and your provider know what data was received and what wasn’t, so you can pick up where you left off. And if something goes wrong (it happens once in a while) your provider can easily “set the clock back” and re-ingest data.
- You can conduct an audit to double check that you and your provider received the alerts they were supposed to receive. (We’re big fans of checking our work!)
- It’s super easy for you as the customer to set this up with your provider. For most devices or services, all you have to do is create an API key with the right permissions and the provider handles the rest.
There are some security product and service vendors that have more sophisticated protocols than raw syslog and WebSockets that provide these same benefits — your vendor should gladly support those too.
Question 2: What tools do you use to make sure my signals are getting to you?
With lots of security signals coming in from different directions, you’ve gotta make sure your provider can verify that they’re receiving the signals from all your tech. In our case, we combine tools like Datadog, PagerDuty, Sentry, Google’s Stackdriver Trace, and Slack to keep a close watch on what’s going on with every device and let the right people know when there’s a problem. Ask your provider what tools they use to monitor device health, how quickly they’ll detect if something isn’t working and how they’ll communicate that to you.
Take it one step further and find a way to check your provider’s work. One of our guiding principles here is, “Show me metrics or it didn’t happen.” Intuition and anecdotes are useful but they don’t prove what happened or form the basis for monitoring. For each of our customers, we have an automatically-generated Datadog dashboard for each customer’s security devices. This gives us an easy, comprehensive way to look at a device’s performance over time:
And because troubleshooting a problem shouldn’t require logging into a production database, we built a Slack bot that quickly gives our device integration engineers the lowdown on what’s happening with each device and makes it easy for them to pivot to other systems for deeper investigation:
Ask your provider how they check device health and request a first-hand look at the tools they use to make sure they’re receiving all your signals (then, see if they have a solid process in place in case something goes wrong).
Question 3: How do you build integrations with new devices?
When your service provider builds an integration with a new security product, do they reinvent the wheel every time (and find new ways to make mistakes)? Or do they have a process to build each integration faster and better than the last one?
TL;DR: Your provider needs a framework that handles all the complexities of receiving signal. That framework should handle all the complexities of alert polling, populating metrics and handling errors. This lets the integration developer focus on actions that are specific to the new security product.
Question 4: What happens when things break?
The old adage is true … nobody is perfect all the time. But does your provider tell you about problems as they’re happening — on a public status page for systemic problems or with a personal phone call, email or Slack message when the issue is specific to your tech? And once the problem is fixed, what’s the provider doing to reduce the risk of that same issue happening again?
Even with the best reviews, testing and monitoring, problems still happen. When they do, a good provider will “fix the problem two ways.” Solving the immediate problem as quickly as possible to mitigate the impact is the obvious part. The part that takes discipline is coming back and figuring out how to reduce the risk of the problem from ever occurring again, how to catch it sooner when it does and identifying easier ways to diagnose and fix it.
How can you get a better understanding of how your provider will act when things don’t go as planned? Ask to see their latest after-action report on something that failed. Or, ask to talk to one of their customers who experienced a problem with the service and then ask that customer how the provider handled it.
Diving deeper
Want more ideas on what to ask your potential provider? We could give you a whole laundry list of Qs to ask when evaluating an MSSP or MDR (this is one of our favorite topics, in case you couldn’t tell). In fact, we’ve got more of those questions — lots of ‘em — here on our blog. Check out “12 revealing questions to ask when evaluating an MSSP or MDR provider” and “12 ways to tell if your managed security provider won’t suck next year.”
