Tips · 6 MIN READ · JON HENCINSKI, TYLER FORNES AND DAVID BLANTON · JUL 8, 2020 · TAGS: How to / Managed security / MDR / Planning / SOC
Remember that time we almost brought down our point of sale environment on a busy holiday weekend because we thought the red team was a real bad guy? Whoah, that would’ve been bad. But we didn’t because we did our prep work. The SOC had a bat phone to the red team and was able to quickly verify the evil “whoami” and “net” commands were from the red team. Crisis averted.
Red team assessments are a great way to understand your detection and investigative capabilities, and stress test your Incident Response (IR) plan. But good intentions can lead to bad outcomes if you don’t do your prep work. A red team will generate activity that looks similar to a targeted attack (cue the adrenaline). So a little planning goes a long way.
Here’s six things you should do before taking on the red team.
1. Start with objectives
Start here. Get clear on your objective(s) to set the direction of the assessment and define the rules of engagement. Worried that an attacker could gain access to a segmented part of your network? Or perhaps you’re worried that an attacker could compromise credentials and spin up resources in Amazon Web Services (AWS)? Clear objectives help everyone.
Business-focused objectives usually look like:
- Break into a segmented part of your network
- Obtain a VIP user’s credentials (CEO, CTO, IT Administrator, etc.)
- Access/exfiltrate customer data
While these drive the overall theme and end-game for the red team, there’s a set of objectives that often surround the organization’s ability to respond as well. From a defensive perspective some reasonable objectives are:
- Assess detection capabilities and identify gaps
- Stress test response and remediation capabilities
- Assess investigative capabilities in Windows and Linux environments
- Assess investigative capability in the cloud
Goals bring purpose to the assessment. Purpose that should be measured along the way. Some key questions we measure are:
- How long did it take us to spot the red team?
- At what phase in the attack lifecycle did we spot them?
- How long did it take us to remediate?
- What challenges did we encounter when remediating?
- Do we need to update our response playbooks?
- What didn’t we detect?
- Document these to be actioned later.
- Were there investigative challenges that prevented us from answering key questions?
- Document these to be actioned later.
2. Review your IR plan with the team
It’s so important to build muscle memory around your IR process before a bad thing happens.
This way everyone knows what to do, including how to communicate. One of the biggest challenges is getting over the “adrenaline rush” that comes with responding to an incident. Panic will happen, and chaos will ensue the first couple of times through it. But as everyone gets comfortable with the process and goes through some of the unknowns together, the response process will become a well-oiled machine that everyone is ready for instead of afraid of.
From an operator’s perspective, we’re a huge fan of running threat emulations for our analysts. These are miniature versions of a red team assessment that help train our analysts in responding to a specific threat, or testing our own response process. There’s a lot of fun to be had here for a blue-teamer who is red curious (remember rule #1 is that objectives are key).
For the broader org, we’re biased, but “Oh Noes” is a great place to start if you need some help organizing a simulated walk-through of your IR plan (and have some fun in the process).
3. Emphasize remediation
We agree with Tim MalcomVetter. The emphasis of a red team should be response. Talk about remediation ahead of time. Ask hard questions like, “what would we do if that account was compromised?”
Pro-tip: Know ahead of time who in your org to contact for infrastructure questions, service accounts, etc. Sometimes knowing who to call is the biggest hurdle.
Plan your response, know who to contact, and then stress test your plans. If your SOC doesn’t have a lot of reps responding to red team activity, remediation may happen without considering business impact.
Consider the following: The red team appears to be using the account “sql_boss” to move laterally. We should disable that account.
Red teams love service accounts. Service accounts typically have privileged access and can be tough to reset.
In this scenario, disabling the account ‘“sql_boss” would cause the red team some pain. But what else would it do? What does that account run? How is it used? Is it responsible for the backend of a business critical application?
Should we disable this account? Can we disable this account right now?
There’s some not-so-funny stories we can tell here about how this oversight has caused major pain for some organizations. But in essence the major theme is: Do your homework, plan your response and talk about it ahead of time.
4. Set expectations
Your blue team just spotted a bad guy moving laterally via WMI to dump credentials on a server? Great find! Will you let them know it’s an authorized red team?
There’s many theories to appropriately assess the response to a red team. Some organizations prefer not to tell their defenders, some prefer to operate more openly in the purple team model. In any regard, there will be a moment between detection of the initial threat and the recognition that this is authorized red team activity that you’ll want to plan for. Your SOC will think this is a real threat, and your playbooks for a real threat will (hopefully) be followed. Consider that when you make the decision to include/exclude knowledge of the assessment from key stakeholders in your security organization.
One way to think about this is: “at 2am who/how many will be woken up to respond, and how soon in our IR plan do things become a risk to the business?”
Our take: The more people in the know, the better. Don’t gas the team responding to an authorized assessment. Save some capacity and energy for the real thing (we’ve seen the real thing happen at the same time as the assessment).
5. Chat with your MSSP/MDR
Use an MSSP or MDR? Chat with them. Understand rules of the road for responding to red team activity. It’s likely one of your red team goals includes assessing your MSSP/MDR. That’s great! But understand what you can expect before you get started.
At Expel, we like to treat red team engagements as a real threat to exercise our analysts’ investigative muscle, and also showcase our response process. This helps build confidence between us and our customers. It also helps them understand how we will communicate with them (slack, email, PagerDuty) when there’s an incident in their environment.
Additionally, this also showcases our analysts’ investigative mindset, including a full report to show the detail of our response and the thoroughness of our investigation. Now, as mentioned above there’s a cost to responding to a red team exercise. Response is time-consuming and analyst resources are extremely valuable.
We believe that showcasing the initial response is important, and the extended response can wait.
That means if a red team is detected and confirmed at 2am, let everyone go back to bed and pick up the response during normal business hours. For red team response, we operate M-F 9am-5pm and will continue to chase new leads for two business days before delivering a final report. That report is comprehensive, and includes everything our normal critical response would contain, but everyone is much happier at the end of the day when our off-hour energy is saved for the real thing.
6. Have a bat phone to the red team
Your MDR or SOC just spotted activity they believe is the red team. Prove it with evidence. Don’t assume!
Call them. Show them. Verify it’s the red team using evidence. You would be surprised at how often the lines get crossed when the actions taken during an assessment don’t necessarily line up with what was documented/in-scope. However, the quicker these actions can be confirmed, the happier everyone is when they aren’t related to the actions of an actual threat.
Most SOCs will not stand down until this is confirmed, and we’ve sometimes waited more than 12 hours to get confirmation that something we identified is related to an authorized test. That’s a lot of energy expended on both ends. Have cell phone numbers, Zoom bridges, etc. before you get started. Always have a deconfliction process on-hand prior to launching the assessment. This will save a lot of your team’s time and energy when the red team gets in.
Parting thoughts
Red team assessments come in all shapes and sizes, and we believe that they are essential for understanding not only the security posture of an organization’s overall response readiness.
If you’re in a position to influence how a red team assessment is organized, we encourage you to talk about these points not only internally but with the red team you have chosen to carry out the assessment as well as the SOC/MSSP/MDR you will be relying on for defense.
Some quick planning and expectation setting can prevent a lot of pain and create an overall better engagement for everyone involved!
