A common sense approach for assessing third-party risk

Security operations · 12 MIN READ · BRUCE POTTER · JUL 26, 2018 · TAGS: How to / Planning

“How secure is your supply chain?” It’s a question that can strike terror into the heart of a CISO – even one who’s in charge of a mature security organization.

With the move (sprint?) to cloud-based infrastructure, and business departments subscribing to SaaS apps left and right (“Oops! was I supposed to tell IT?”), every day we rely more and more on other peoples’ services to serve our customers.

Here at Expel, we’re a “cloud first” organization. Our entire enterprise’s physical infrastructure fits easily on one desk. But we use the capability of nearly 50 vendors to bring our services to our customers. That’s a lot of infrastructure that’s not ours.

And we’re a relatively small company. Large companies may depend on hundreds of outside services. Understanding how all those services keep their customers (meaning … you) secure is no trivial matter. But it’s super important.

CISOs manage cyber risk in their own infrastructure every day. But once you leave your own infrastructure, it gets harder. And there aren’t a lot of playbooks for how to manage the risk of someone else’s infrastructure.

Third parties are out of your control. You give them money, they provide a good or service in return. Sometimes, there’s even contractual language that says “we’ll do our best to secure your data.” But, in practice, those words don’t really mean much. What matters is the practices, procedures, and policies your vendors follow.

At Expel, like many companies, we’ve created a third-party assessment program for our vendors to try to manage our supply chain risk. We’ve used other companies’ third-party assessment programs as input, consulted our vendors and done a lot of research. It works well for us, and so we’re sharing it with you, along with the third-party risk assessment questionnaire we’ve developed.

Watch the video overview … or keep scrolling to read on

First … be realistic about who chooses your suppliers

Unfortunately (at least for CISOs), security doesn’t control who the organization does business with. Business owners do. And the questions they have on their mind are very different than what most CISOs are wondering. As you roll your program out, it’s important to understand the business owner’s mindset so you can figure out when, where and how to insert your own process into theirs.

When a business owner has a problem, they probably want to fix it fast. They want to know if the product or service they’ve got their eye on will do the trick. If the answer is “yes” (and they’ve got the budget) they’ll move forward, negotiating contracts, agreeing on cost and ultimately making the purchase. Meanwhile, the CISO is thinking, “Does this vendor create an acceptable level of risk?” Getting answers means acting fast – while the business owners are chasing down answers to their own questions. If a potential vendor doesn’t address security in a way you’re comfortable with, the sooner you know that the better. It’s much easier to guide the business away from potentially toxic companies early in the process than to stop a contract that’s gone through all the redlining and negotiation and is one inch from the finish line.

Next … set realistic expectations (aka understand the constraints)

Setting realistic expectations for your third-party assessment program requires understanding two important equations that’ll govern how much time you and your vendors are willing to put in.

They seem simple. But it’s easy to get so caught up in the weeds perfecting your process that you lose sight of them. Violate equation number one and vendors will start stretching the truth to get through all of your questions or bury the bad stuff to try and get your business. Violate the second equation and you’ll find yourself giving away a free risk assessment or pen test to every potential vendor (more on that later).

Remember, SaaS providers are getting bombarded left and right with third-party assessments. Short, easy questionnaires will get their attention before long complex ones. Likewise, you don’t have a lot of time to dedicate to this either. The more complex the questions, the longer you’ll have to spend vetting the results.

Short, simple and to the point is far more likely to get to a result that’s useful – both for you and your vendors – than some crazy, multi-page questionnaire. Keeping things simple has multiple benefits.

When in doubt, use the “50 at 50” rule

Striking the balance between thorough yet brief, reminds me of a saying from when I used to crew for a friend that raced cars out in West Virginia. The sanctioning body for the races required that cars be painted in a professional manner. Anyone that’s been around amateur racing knows that very little about it qualifies as “professional.” The rule of thumb the officials used was “50 at 50”… that is, when you looked at a car traveling 50 miles per hour from 50 feet away, did the car look like it was painted? If the answer was “yes,” you were good to race.

That’s sort of how I view third-party assessments. If your process gives you the same level of assurance about your vendors’ security processes as “50 at 50” gives racing officials, you’re doing things right. Sure, there are some situations that require far more diligence than that (stay tuned!), but in most cases, you’re just trying to get a general feel for things. Ultimately, even organizations with great practices and procedures will screw up sometimes. Nothing you do in your third-party assessment program will change that.

The common sense process for third-party assessments

There are three big chunks to any third-party assessment program: creating the questionnaire, designing the process and running it (told you it would be “common sense”). Of course, not every situation will fit neatly into your process. We’ll cover the outliers too. But, to get started, you need to create your questionnaire.

1. Creating your questionnaire

The questions you ask your vendors will be taken seriously by them … or at least they’ll look at them seriously and try to figure out what you mean. It’s important to write crisp, clear questions that vendors can easily understand and have a clear way to answer.

The meat of your questionnaire is the questions themselves. We’re providing our third-party risk assessment questionnaire as a starting point for you. Hopefully this’ll let you speed through this step. We like these questions because they cover a wide swath of cybersecurity without being too detailed. They’re also aimed at making it easy for vendors to re-use work they’ve already done. Asking about existing certifications and the results of previous testing reduces friction in the process. Really, we want to ask questions we think will get answered truthfully and quickly. Focusing on reuse is one strategy for that.

We’ve also designed our questionnaire to sleuth out how much thought and care a vendor has put into security in general. For example, when we ask “Do you have a formally appointed information security officer?” we get a different vibe when the answer is “Yes, here’s our CISO’s contact info,” versus “Not really. Our lead developer cares a lot about security though.” Simple questions like this give you a great window into how a potential vendor thinks about security.

2. Building the process

Developing the questions is only one piece of the prep work that you’ll need to do. How you’re actually going to manage the process is equally important. The process we’ve designed breaks down into the following six steps.

Your exact process will, no doubt, have to be tailored a bit to the way your organization buys products and services. We’re not suggesting that you can do a direct cut-and-paste of our process. But hopefully it can be an advanced starting point for you. Here’s a quick overview of how we thought about each step as we created our own third-party assessment process.

We created a set of criteria to determine which external vendors need to go through the process. Vendors that make the cut include:

• Services that will impact production systems
• Services that contain customer or other sensitive data
• Systems which aggregate data from multiple data sources.

If someone is trying to use a new service that fits one of these situations, they send a request for review to a security review email alias containing what the service is, how we’re going to use it and provide points of contacts at the vendor.

It’s a bit awkward to send an email to a potential vendor demanding a bunch of information without first introducing yourself, the process and what they should expect. At Expel, the first thing we send to the vendor is a cordial email describing our process, the relatively casual and light touch nature of it and an invitation to ask questions or engage if they have concerns. We also let them know our desired timeliness (usually we ask for a response within about two weeks).

Next, we send the real email. We use our secure file sharing system to send this email so that all communications are encrypted and their response is protected on its way back to us. You don’t have to do this, but it’s advisable, especially if you’re asking for copies of sensitive documents such as their SOC2 and pen test executive reports.

After a week and a half has gone by, we’ll send a gentle reminder if we haven’t heard anything. That’s usually enough prodding to get us answers right under our two week request.

Hopefully, when you get the vendor’s answers back they make sense, are reasonably complete and if you’re lucky they’re even comprehensible. Sometimes we’ve had to go back to ask vendors for clarification on an answer or two, and that’s OK. Keeping in mind the “50 at 50” mentality, once you have the answers, balance them against the business request and determine if you’re willing to move forward with the vendor or if there are concerns that need to be addressed.

Once we’ve got our heads around all of the vendor’s answers, we give the business owner our opinion. When the results are positive, the conversations are easy. When we have concerns, that’s when things get more difficult. It’s a good idea in those cases to involve more people on the business side than just the requester (team leads, managers, etc.). You’re going to get into a risk-oriented decision about how important this specific vendor is to the company and what the security risks are. The results of that meeting can vary wildly, but usually will fall into one of four buckets:

1. Yep. Cool. Go for it.
2. We can put in compensating controls to make up for lack of assurance in the vendor.
3. We need a deeper dive to better understand the risks.
4. No. Nope. Negative. Not going to use them.

It’s very important not to treat these decisions as binary. The reason you’re doing a third-party assessment in the first place is to manage risk. Risk is a continuum, as it were, and you should treat your third-party vendor assessment process the same way.

3. Running the process

Once you’ve got your questionnaire and process figured out, test it on a few vendors. Be very up-front with them; let them know this is your first time trying out your third-party vendor assessment questionnaire and you’d love feedback on both the material itself and the overall process. You’ll find some vendors are well prepared for these kinds of requests and will have a team dedicated to answering them. Other vendors will respond with “huh, this is the first time anyone’s asked us about security.” Be prepared for that and everything in between. Take any feedback you get and stir it inappropriately with the work you’ve already done and your objectives for your third-party assessment program.

After you’ve tested the process on a few vendors (or later … run the process for a year or two), iterate. Feel free to change it up. As you grow, your risk appetite changes. As the state of the art of your vendors improves, you might want to modify your process to suit your needs. You don’t need a forever “apples to apples” comparison over the years. Rather, you need each response to provide you the information you need right now to make the decision that’s in front of you. That information will change over time, and your process should too.

Keeping track of the results

You’ll likely get lots of confidential documents back from your vendors when they reply to your questionnaire. You’ll want to make sure you protect them according to the terms of any non-disclosure agreements you signed with them. Be sure to follow whatever your internal procedures are with respect to protecting that information. Also, we’ve found that it’s helpful to create one place to track all of the assessments – upcoming requests, active ones, and assessments we’ve completed. We store all the responses, supporting documents and our notes in one place. We’ve chosen Confluence for that since we use the Atlassian suite for a lot of our engineering and security workflow already. You should choose whatever makes sense in your organization. But be aware, you’ll build up quite a pile of information quickly, so being organized early will pay off as your program grows.

Hooking the process into the way your organization buys stuff

Having a process is all well and good. But, unless you socialize it and have a clear way to plug it into the way your organization buys stuff, your third-party assessment program can quickly turn into shelfware.

It’s important to set the hook early in the process to get the best results. That hook can take many shapes:

• The procurement process: When a business unit requests a new PO, your purchasing department can simply ask, “What does Security think of this?” Knowing a PO won’t be cut unless there’s a clear answer to that question will force business owners to engage your process early so you’re not playing catch up.
• Contract review: A slightly different take, but the same basic idea. When a contract is put in front of legal to review, they can ask, “What does Security think of this?” as well. Again, if business owners know they can’t get through legal without clearing security, they’re going to engage you early.
• That’s just the way it is: Rather than have a specific gate, you can communicate with leaders and purchasers that new products and services are subject to a third-party assessment as part of doing business. If it’s discovered that someone bought something without an assessment, There Will Be Consequences™. Just like there are when people buy product outside of purchasing, right? Right?

Whatever you decide, be sure to communicate it widely and often. New processes that affect how you buy services tend to take a while for everyone to understand and accept, so putting together a good PR campaign can’t hurt your cause.

Also, be sure the “how to submit” part of your process is clear. At Expel we use Jira’s Service Desk as the portal where users can submit third-party assessment requests and track progress. We already use Service Desk for IT and other ticket tracking so it was an easy solution. YMMV and all that… be sure to choose a method of engagement that works for you and your company.

Vendors that are bigger than your breadbox

There may be times when the product or service you’re evaluating is too big, too important or represents too much risk to apply the “50 at 50” rule. In these cases, you’ll likely end up doing a more formal risk assessment to understand the risks they present in more depth so you can compensate for any issues you can’t get the vendor to fix.

Risk assessments are complicated (I addressed them in an O’Reilly Security talk here if you’re interested). They can be done either by your own staff or a third party. Either way, I have two points of caution:

Don’t give out a free pen test

If you engage a third party to assess your vendor’s product it’s easy for your vendor to ultimately get a free pen test that you unwittingly pay for. So, if you hire a third party, make sure they’re working on your behalf and use your business needs as the backstop for their work. That’ll make sure the final product is geared towards you and your business, not the vendor and their product.

Make sure you don’t accidentally do a pen test or risk assessment

The other common mistake when you dive deeper is you don’t realize that you’re diving deeper. You get the questionnaire back and you have questions … so you ask the vendor a few more questions. Things are clearer, but still not clear. So, you ask “Hey, can we take it for a test drive?” You get their product, configure it, start testing it and suddenly realize you’re doing a product assessment and you’re already 40 hours into the process and probably have 80 more hours to go before you’re done.

As you start peeling back the onion be aware that you’re doing it overtly and for a reason. Don’t spend more time and effort on a third-party assessment than you need to.

Oh … and make sure to avoid these common pitfalls

Finally, there are a couple of other pitfalls you’ll want to make sure you avoid as you launch (or refine) your third-party vendor assessment program.

Adding to the questionnaire

Be wary of asking too many questions or diving too deep. You’ll quickly reach a point where vendors don’t want to answer and it takes you too long to assess the results. It’s not worth it. If you decide to do a full-fledged risk assessment, then by all means, dive in the deep end. But if you’ve got a question you feel you must add to your questionnaire, find one (or two?) that aren’t giving you any value and swap them out. Again, the simpler and shorter your questionnaire is, the more likely you’ll get accurate and timely responses.

Believing all the answers

It’s human nature to not want to fail tests. That applies to vendors responding to third-party assessment requests. They want to be as compliant as possible, so you can expect they’ll take a few liberties in their answers. While it’s unusual to find a vendor that flat out lies (saying they’re SOC2 Type 2 compliant when they’re not, for example), you may find vendors occasionally stretch the truth enough to “pass.” So, when you’re answering the question “Am I OK using this vendor,” assume their answers are eighty percent correct.

That’s it

There you go. That’s Expel’s third-party vendor assessment program in a nutshell. There are many like it, but this one is ours. Hopefully it gives you a jump start on building your own program. Please, take a look at our questionnaire, and feel free to use, modify, and comment on it as you see fit. I’d also suggest taking a look at our NIST cybersecurity framework self-scoring tool that I created. It allows you to create charts that show your current and future security posture based on the NIST CSF and it includes a section on supply chain risk. If you do have comments and you’d like to share on this process, the questionnaire or the NIST tool, please reach out to us and let us know. We’re always trying to improve and would love for you to help us with that.