Detection and response in action: an end-to-end coverage story

Security operations · 12 MIN READ · NATHAN SORREL · SEP 8, 2022 · TAGS: MDR

What does a comprehensive detection, response and threat hunting strategy look like? Glad you asked.

Expel provides three primary service offerings—managed detection and response (MDR), phishing prevention, and threat hunting—and we offer those in a few different flavors to customers around the world.

One size doesn’t fit all when it comes to service delivery. Each customer’s distinct environment, risk, and security posture requires that tools work together, so we built Expel to connect all of those services into one coherent, unified experience. The whole really is greater than the sum of its parts.

So how do our MDR, phishing, and threat hunting services work, and most importantly, how do they work together?

The following soup-to-nuts description of Expel’s security process borrows details from several real-life detection situations, and the accounts illustrate how our team shut hackers down. While we’ve changed some particulars for the sake of privacy, this story accurately represents how our teams go from triaging alerts all the way to threat hunting and back.

We’ll walk you through the entire incident to illustrate how different players on the team and our complementary services reinforce each other.

Detection: alert and triage

It’s a Sunday at 7:17am EST. The day shift analysts have arrived and are catching up on last night’s activity. Reading through customer communications and recent investigations, the analysts soak up the news.

Tools are logged into, browser tabs are organized, and the day begins. Girish checks on a verification request for updates he sent to a customer yesterday. Jenni flips through alerts, looking for “the weird.” Chris puts the finishing touches on an investigation that looked odd at first, but was quickly explained by some research and a little IP prevalence mapping.

At 7:48 am EST, an alert arrives — DNS queries originating from the process Regsvr32.exe. Windows Defender ATP detects a common Windows binary making unusual network connections.

This alert arrives in our medium severity queue and is examined by an analyst within 10 minutes. With our automation-forward approach, raw alerts are analyzed immediately by our detection bot, Josie™. It commonly takes less than five minutes for Josie to escalate an alert to a human analyst, and for that analyst to confirm the alert is a threat.

We consistently triage our highest fidelity alerts in about two minutes. We track our response time in minutes and we like it that way.

Beyond the experience of seeing thousands of alerts a month, our analysts use in-house datasets and open source tools (like Greynoise) to determine the prevalence and meaning of observed events. Asking questions like, “Is this activity actually uncommon on a global scale?” and “Does this IP address have a reputation?” leads analysts to better understand what they’re seeing.

We call it customer context or “CCTX.” It’s most commonly displayed in Workbench as highlighted text. CCTX can be any specific insight provided by the customer related to expected activity from users, endpoints, or network locations, and it helps us quickly assess a situation.

Additionally, our analysts flag red team assets, previously compromised hosts, and other artifacts for future reference. Each piece of CCTX information saves our analysts minutes of research, keeping our alert-to-fix times low.

Response: investigation and context

This one will require more time and digging.

The Expel security operations center (SOC) is very much a team. Analysts bring their own capabilities and knowledge sets to the table and investigations quickly take shape around the collective strengths of the group. One analyst examines the endpoint within Microsoft Defender for Endpoint while another looks at IP/domain prevalence. A third examines recent phishing activity. It’s not uncommon to have three or more analysts collaborating on the same incident.

The collaboration between our analysts also extends to you. The Expel Workbench lets our customers see everything we see in real time — not after the fact. Workbench gives them potent investigative and data collection tools to power their own daily SOC activities.

Analysts on the phishing team are pros at triaging suspicious documents. The faster Jose can get that file, the faster he can provide the support Jenni and the team need.

While our services offer tremendous value individually, integrating them provides even more coverage against an attack — a benefit highlighted by this case.

The root cause of most attacks? Phishing emails. MDR and phishing services together make up the Expel SOC, and they communicate extensively, maximizing effective response across our customer base. Since Jose and other phishing analysts are at the front edge of so many attacks, they can alert MDR analysts sooner about potential business email compromise (BEC). Attacker trends are commonly noted by phishing analysts, who pass the information on to their MDR counterparts. Overall, having both services in place means fuller coverage and quicker response.

Back to the story.

Remediation: incident to fix

For many incidents, automation baked into our process lets Jenni instantly both notify the customer about what we’re seeing and suggest remediation steps. More hosts, hashes, and domains will be added to the list of suggested remediation steps as the SOC gathers indicators of compromise (IOCs).

Within 20 seconds of the incident’s creation, our customer has meaningful action they can use to nip the attack in the bud.

And yes, we’re tooting our own horn here. We’re good at what we do and we do it quickly.

At this point, much of the heavy lifting is done. Jenni and another member of the Global Response Team (GRT) will continue to deep dive into anything that’s still not fully understood. They’ll ask questions like:

• How many users received the email and how many clicked on the malicious attachment?
• What’s the source of the email?
• How many hosts are involved?
• What network activity did we see?
• Was there any evidence of persistence or lateral movement?
• Did the malicious files successfully execute?
• Should the hosts be reimaged?

New IOCs are added as they are discovered and any new alerts that come through are attached to the Incident.

The GRT is composed of senior and principal-level analysts who serve as incident responders for critical incidents. These are our most seasoned analysts and they help validate all aspects of the compromise.

Next question: “How can we help the client avoid this next time?”

Resilience: prevention

The team shared remediation steps with Vandelay and Jenni awaits confirmation. David has joined the Zoom call as a member of the GRT to help Jenni finalize things.

Elapsed time since we first issued customer recommendations: 40 minutes. This situation is tricky, as we’re dealing with multiple hosts and decreased weekend staffing by the customer.

What can we do when there’s an active threat but the customer is out-of-pocket? Good news: clients can opt into our automated remediation service, which can automatically contain hosts as needed. Unfortunately, Vandelay isn’t taking advantage of this feature.

This particular customer had a total of 20 endpoint-related security incidents within its environment last year, more than half of which would have been avoided with the proper wscript.exe resilience policy in place. While resilience steps are not always easy to implement, they can make a substantive, positive impact on a customer’s security posture.

Expel SOC analysts are up early anyway and available 24/7, but most people don’t want to be awakened on a Sunday morning by a critical incident. Your weekend on-call folks, not to mention your CISO, will thank you for preventing incidents like this.

PagerDuty automation, enabling auto-containment and completing resilience recommendations are small investments that can be made to improve response times for future incidents. PagerDuty can wake you up if something goes wrong. Auto-contain authorization lets us isolate compromised hosts even if you don’t wake up. Completed resilience action can help you avoid these issues altogether.

Let’s say you want to take a deeper look into your environment. Are my remediation steps working as expected? What else is “Regsvr32.exe” doing on our endpoints? Do we have any coverage gaps?

Threat hunting: validation and high-level understanding

Bryan knows both the red and blue side of cyber and now gets to employ those years of experience in a threat hunting capacity. Our hunting service, a big step beyond detection and response, lets us dig deep into customer data to find not only detection gaps and suspicious events, but also to verify resilience. Our hunting catalog easily expands to scope for both confirmation of resilience and absence of emergent IOCs.

We ask questions like:

• Was multi-factor authentication (MFA) really enabled for all users?
• Is the Server Message Block (SMB) protocol accessible on public facing servers?
• What Amazon Web Services (AWS) region should we not see in this environment?
• Does Java.exe ever have any suspicious child processes?

These questions are crucial. If you think you’re hardening your infrastructure, don’t you want to be sure?

A lot of in-house security teams are so busy they rarely have time to baseline or research their own environments. Questions like, “What parent process typically spawns wscript.exe?” can slip down the priority list. And “Which users and domains are most commonly seen executing Okta impersonation events?” Or “What AWS users do we see commonly using long term AccesskeyIDs?” Expel threat hunting can provide some much needed insight into these and other endpoint, SaaS, and cloud questions.

The Expel threat hunting service iterates around a historical POV and a broader range of detection complexity. We conduct regular monthly hunts on your tech and infrastructure, and we run periodic IOC hunts as new threats emerge. Even more fun: with Expel, you can even take advantage of evolving draft hunts for testing and development.

We afford our hunting customers better visibility across their whole landscape. Whether its cloud infra, SaaS applications, network, or endpoint-related hunts, our coverage includes a wide array of technologies. For example:

• AWS’ EC2 modifications hunt
• Duo’s Suspicious Duo Push activity hunt
• Cloud apps’ data center logins hunt
• Cloud infra’s Azure Successful Brute Force hunt
• Azure Successful Bruteforce hunt

We also provide additional insights and resilience recommendations to help reduce risk exposure in the future.

Threat hunting allows you to validate that you’re as secure as you’re trying to be, and provides a path forward on things that still need some attention.

What else can hunting do? And, where do we go from here?

Completing the circle: better detection

Peter, an Expel senior detection and response analyst, joins Bryan to make sure the activity gets categorized appropriately. If the detection logic produces higher-fidelity signal, we want to elevate the severity to get analysts’ attention more quickly.

Whether it comes out of our threat hunting experience, a phishing campaign, or new threat intel, Expel constantly adjusts the dials on our detection capabilities. We try to harness every ounce of analyst attention and brain power toward customer alerts, and we never want to waste a scrap of what we learn. Completing the feedback loop is critical to properly facing a rapidly evolving threat landscape.

Tomorrow, even if attackers start using electric toothbrushes to launch attacks, we’ll be able to respond.

What end-to-end coverage means to us

We dramatized the Vandelay incident for readability, but we see events like this all the time at Expel. Like, every single week.

And each time we work through the alert → investigation → phishing → incident → hunting → better detection → alert cycle (and its various permutations), we get faster and better, to make you safer.

Jenni, Girish, Tucker, Jose, Chris, DeShawn, David, Bryan, and Peter are just a few members of the team keeping eyes-on-glass all-day-every-day.

This is 360° security at its best. You’re invited to test drive our comprehensive MDR, phishing and hunting services to experience the full benefits.