Security operations · 12 MIN READ · NATHAN SORREL · SEP 8, 2022 · TAGS: MDR
What does a comprehensive detection, response and threat hunting strategy look like? Glad you asked.
Expel provides three primary service offerings—managed detection and response (MDR), phishing prevention, and threat hunting—and we offer those in a few different flavors to customers around the world.
One size doesn’t fit all when it comes to service delivery. Each customer’s distinct environment, risk, and security posture requires that tools work together, so we built Expel to connect all of those services into one coherent, unified experience. The whole really is greater than the sum of its parts.
So how do our MDR, phishing, and threat hunting services work, and most importantly, how do they work together?
The following soup-to-nuts description of Expel’s security process borrows details from several real-life detection situations, and the accounts illustrate how our team shut hackers down. While we’ve changed some particulars for the sake of privacy, this story accurately represents how our teams go from triaging alerts all the way to threat hunting and back.
We’ll walk you through the entire incident to illustrate how different players on the team and our complementary services reinforce each other.
Detection: alert and triage
It’s a Sunday at 7:17am EST. The day shift analysts have arrived and are catching up on last night’s activity. Reading through customer communications and recent investigations, the analysts soak up the news.
Tools are logged into, browser tabs are organized, and the day begins. Girish checks on a verification request for updates he sent to a customer yesterday. Jenni flips through alerts, looking for “the weird.” Chris puts the finishing touches on an investigation that looked odd at first, but was quickly explained by some research and a little IP prevalence mapping.
Let’s meet our talented crew.
Girish, a detection and response analyst, helps keep all the balls in the air. His gift for leadership, organization, and process comes in handy when ensuring 24×7 coverage across three shifts and 25+ analysts. In a given week Expel analyzes hundreds of incidents and conducts dozens of investigations. Girish, and others like him, keep the trains running.
Chris’ superpower is level-headedness. In security, where a frantic response can lead to disaster, Chris doesn’t react, he responds, by taking a few seconds to reflect on the facts of a case. He radiates calmness, making the whole team make better, smarter decisions.
Jenni seems to have threat intel on speed dial. She can research and document activity better than almost anyone. Offering accurate understanding and attribution regarding attack type can be profoundly helpful during an investigation.
All of these folks have spent thousands of hours reviewing suspicious activity and investigating the “really bad” stuff from our customers.
At 7:48 am EST, an alert arrives — DNS queries originating from the process Regsvr32.exe. Windows Defender ATP detects a common Windows binary making unusual network connections.
This alert arrives in our medium severity queue and is examined by an analyst within 10 minutes. With our automation-forward approach, raw alerts are analyzed immediately by our detection bot, Josie™. It commonly takes less than five minutes for Josie to escalate an alert to a human analyst, and for that analyst to confirm the alert is a threat.
We consistently triage our highest fidelity alerts in about two minutes. We track our response time in minutes and we like it that way.
Beyond the experience of seeing thousands of alerts a month, our analysts use in-house datasets and open source tools (like Greynoise) to determine the prevalence and meaning of observed events. Asking questions like, “Is this activity actually uncommon on a global scale?” and “Does this IP address have a reputation?” leads analysts to better understand what they’re seeing.
We call it customer context or “CCTX.” It’s most commonly displayed in Workbench as highlighted text. CCTX can be any specific insight provided by the customer related to expected activity from users, endpoints, or network locations, and it helps us quickly assess a situation.
Additionally, our analysts flag red team assets, previously compromised hosts, and other artifacts for future reference. Each piece of CCTX information saves our analysts minutes of research, keeping our alert-to-fix times low.
Response: investigation and context
This one will require more time and digging.
Jenni launches a “PermaZoom” 24×7 video call with the rest of the team. “Anyone else see that one in the medium queue? It doesn’t look right.” More analysts jump in to help. DeShawn, always eager to lend a hand, takes a look.
“I’m gonna see if any other hosts are talking to that domain,” Tucker chimes in.
Chris offers to scope the environment for other instances of the Word document.
The Expel security operations center (SOC) is very much a team. Analysts bring their own capabilities and knowledge sets to the table and investigations quickly take shape around the collective strengths of the group. One analyst examines the endpoint within Microsoft Defender for Endpoint while another looks at IP/domain prevalence. A third examines recent phishing activity. It’s not uncommon to have three or more analysts collaborating on the same incident.
The collaboration between our analysts also extends to you. The Expel Workbench lets our customers see everything we see in real time — not after the fact. Workbench gives them potent investigative and data collection tools to power their own daily SOC activities.
Analysts on the phishing team are pros at triaging suspicious documents. The faster Jose can get that file, the faster he can provide the support Jenni and the team need.
While our services offer tremendous value individually, integrating them provides even more coverage against an attack — a benefit highlighted by this case.
The root cause of most attacks? Phishing emails. MDR and phishing services together make up the Expel SOC, and they communicate extensively, maximizing effective response across our customer base. Since Jose and other phishing analysts are at the front edge of so many attacks, they can alert MDR analysts sooner about potential business email compromise (BEC). Attacker trends are commonly noted by phishing analysts, who pass the information on to their MDR counterparts. Overall, having both services in place means fuller coverage and quicker response.
Back to the story.
Thankfully this customer, Vandelay Industries, provides the Expel SOC with Live Response access via their EDR console, meaning Jose can directly acquire the file for fuller analysis. Detonating the document in our sandbox confirms that the document isn’t, in fact, the “Tax Planning Help Guide” its name suggests (we know — we’re as shocked as you are).
“Hey, Jenni,” says Jose, “this sandbox execution looks bad.”
Jenni looks at the endpoint timeline (since the malicious document was first opened). “I’m guessing that JPEG isn’t really a JPEG,” she mumbles, as she runs the hash through VirusTotal.
Remediation: incident to fix
For many incidents, automation baked into our process lets Jenni instantly both notify the customer about what we’re seeing and suggest remediation steps. More hosts, hashes, and domains will be added to the list of suggested remediation steps as the SOC gathers indicators of compromise (IOCs).
Today At 5:47 UTC Windows Defender detected ‘Regsvr32.exe’ being spawned from `Winword.exe’ on host DESKTOP-3AB921 and making network connections to BadDomain.com”…
- Contain the host “DESKTOP-3AB921”
- Block the malicious Word document “Tax Planning Help Guide.docx” with SHA256 hash “ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad”
- Sinkhole the domain BadDomain.com
- Block emails from “BadEmails.com” with subject line “Download the Tax Planning Help Guide”
We will update you if we identify any other involved hosts.
Within 20 seconds of the incident’s creation, our customer has meaningful action they can use to nip the attack in the bud.
And yes, we’re tooting our own horn here. We’re good at what we do and we do it quickly.
“Which customer was that incident for again, Jenni?” asks Deshawn. “I see two more alerts in the medium queue that look similar.”
“Vandelay Industries.” Jenni replies. “Is that the “DESKTOP-3AB921” host you’re seeing, or a new one?”
“Same customer, but new hosts… both of them. I’ll drop those in the incident and assign those remediations to Vandelay,” DeShawn adds.
“Thanks,” she says, “I’m gonna make this incident ‘critical’ and update the customer in Slack. Would you mind scoping those hosts for anything new…domains or otherwise? Whoa. Vandelay already yanked that first host off the network. That was quick!”
At this point, much of the heavy lifting is done. Jenni and another member of the Global Response Team (GRT) will continue to deep dive into anything that’s still not fully understood. They’ll ask questions like:
- How many users received the email and how many clicked on the malicious attachment?
- What’s the source of the email?
- How many hosts are involved?
- What network activity did we see?
- Was there any evidence of persistence or lateral movement?
- Did the malicious files successfully execute?
- Should the hosts be reimaged?
New IOCs are added as they are discovered and any new alerts that come through are attached to the Incident.
The GRT is composed of senior and principal-level analysts who serve as incident responders for critical incidents. These are our most seasoned analysts and they help validate all aspects of the compromise.
Next question: “How can we help the client avoid this next time?”
Resilience: prevention
The team shared remediation steps with Vandelay and Jenni awaits confirmation. David has joined the Zoom call as a member of the GRT to help Jenni finalize things.
Jenni tells David that, “So far, we’re seeing execution on three hosts from what appears to be ‘click-through’ by users into a phishing campaign. That led to a malicious Word file. I’ve updated the customer but am still waiting for them to respond. Two of the hosts are still online. The incident is “critical” because of the multiple hosts, so they should have received a notification by now, but still no word. I’ll ping their account rep and have them reach out by phone.”
David thinks out loud. “So they don’t have auto-containment in Workbench enabled, Let me get into the console and poke around.”
Elapsed time since we first issued customer recommendations: 40 minutes. This situation is tricky, as we’re dealing with multiple hosts and decreased weekend staffing by the customer.
What can we do when there’s an active threat but the customer is out-of-pocket? Good news: clients can opt into our automated remediation service, which can automatically contain hosts as needed. Unfortunately, Vandelay isn’t taking advantage of this feature.
This particular customer had a total of 20 endpoint-related security incidents within its environment last year, more than half of which would have been avoided with the proper wscript.exe resilience policy in place. While resilience steps are not always easy to implement, they can make a substantive, positive impact on a customer’s security posture.
Expel SOC analysts are up early anyway and available 24/7, but most people don’t want to be awakened on a Sunday morning by a critical incident. Your weekend on-call folks, not to mention your CISO, will thank you for preventing incidents like this.
PagerDuty automation, enabling auto-containment and completing resilience recommendations are small investments that can be made to improve response times for future incidents. PagerDuty can wake you up if something goes wrong. Auto-contain authorization lets us isolate compromised hosts even if you don’t wake up. Completed resilience action can help you avoid these issues altogether.
Let’s say you want to take a deeper look into your environment. Are my remediation steps working as expected? What else is “Regsvr32.exe” doing on our endpoints? Do we have any coverage gaps?
Threat hunting: validation and high-level understanding
[The next day; the familiar <ding-dong> sound chimes as Bryan joins the Zoom call]
“Hey gang, is Jenni on? She asked me to pop in…something about a wscript.exe hunt?”
Bryan knows both the red and blue side of cyber and now gets to employ those years of experience in a threat hunting capacity. Our hunting service, a big step beyond detection and response, lets us dig deep into customer data to find not only detection gaps and suspicious events, but also to verify resilience. Our hunting catalog easily expands to scope for both confirmation of resilience and absence of emergent IOCs.
We ask questions like:
- Was multi-factor authentication (MFA) really enabled for all users?
- Is the Server Message Block (SMB) protocol accessible on public facing servers?
- What Amazon Web Services (AWS) region should we not see in this environment?
- Does Java.exe ever have any suspicious child processes?
These questions are crucial. If you think you’re hardening your infrastructure, don’t you want to be sure?
A lot of in-house security teams are so busy they rarely have time to baseline or research their own environments. Questions like, “What parent process typically spawns wscript.exe?” can slip down the priority list. And “Which users and domains are most commonly seen executing Okta impersonation events?” Or “What AWS users do we see commonly using long term AccesskeyIDs?” Expel threat hunting can provide some much needed insight into these and other endpoint, SaaS, and cloud questions.
“Hey Jenni, glad to jump on. Have they ever confirmed implementation of that resilience step?” Bryan asks. “I wonder if it’s something they’ve simply chosen not to do.”
Jenni says, “I saw back in October they marked that action as complete. I’m wondering if they pushed the policy but didn’t quite get the protection they’d intended. We’re still seeing it run, obviously. Do we have a hunt we could employ to scope wscript activity across all their hosts?”
“The Historical Scripting Interpreter hunt would shed some light on that for them,” suggests Bryan. “They’re using Windows Defender right? I’ll ping their account rep to see if they want to get the process going. Thanks for bringing this up.”“Yeah they are using Defender,” she replies, “and thanks for doing that. Let me know if you need anything from this end.”
“Thanks Jenni, I’ll keep you posted on how it progresses. Might have you run the analysis when the hunt kicks off. Great catch on the incident, by the way.”
The Expel threat hunting service iterates around a historical POV and a broader range of detection complexity. We conduct regular monthly hunts on your tech and infrastructure, and we run periodic IOC hunts as new threats emerge. Even more fun: with Expel, you can even take advantage of evolving draft hunts for testing and development.
We afford our hunting customers better visibility across their whole landscape. Whether its cloud infra, SaaS applications, network, or endpoint-related hunts, our coverage includes a wide array of technologies. For example:
- AWS’ EC2 modifications hunt
- Duo’s Suspicious Duo Push activity hunt
- Cloud apps’ data center logins hunt
- Cloud infra’s Azure Successful Brute Force hunt
- Azure Successful Bruteforce hunt
We also provide additional insights and resilience recommendations to help reduce risk exposure in the future.
Threat hunting allows you to validate that you’re as secure as you’re trying to be, and provides a path forward on things that still need some attention.
What else can hunting do? And, where do we go from here?
Completing the circle: better detection
“We’re definitely seeing it come through the queue,” says Bryan, “but I want us to elevate its severity to high. We’ve seen this technique spike this month in particular. The Vandelay incident really highlighted the recent uptick in this usage of a JPEG file as an obfuscated script file. OSINT calls it a Shorse Attack. I don’t know where they get these names…”
“So basically,” Peter replies, “if the command line contains ‘wscript’ plus‘.jpg’ or ‘.jpeg’ we categorize it as a HIGH. Right?
Peter, an Expel senior detection and response analyst, joins Bryan to make sure the activity gets categorized appropriately. If the detection logic produces higher-fidelity signal, we want to elevate the severity to get analysts’ attention more quickly.
“Exactly,” Bryan says. “We ended up running that query across another five or six customers and found that it’s a lot more prevalent than the months prior. This adjustment should surface these alerts to an analyst even quicker.”
Peter nods. “Sounds good. That change should be live within the hour. I’ll holler if I have any more questions.”
“Thanks, Peter. I’ll check back in a few days. This Shorse stuff makes me wonder if this might be a good long-term hunt for our catalog. Basically, wscript.exe being run containing any atypical file types in the command line. I’ll let you know what I find.”
Whether it comes out of our threat hunting experience, a phishing campaign, or new threat intel, Expel constantly adjusts the dials on our detection capabilities. We try to harness every ounce of analyst attention and brain power toward customer alerts, and we never want to waste a scrap of what we learn. Completing the feedback loop is critical to properly facing a rapidly evolving threat landscape.
Tomorrow, even if attackers start using electric toothbrushes to launch attacks, we’ll be able to respond.
What end-to-end coverage means to us
We dramatized the Vandelay incident for readability, but we see events like this all the time at Expel. Like, every single week.
And each time we work through the alert → investigation → phishing → incident → hunting → better detection → alert cycle (and its various permutations), we get faster and better, to make you safer.
Jenni, Girish, Tucker, Jose, Chris, DeShawn, David, Bryan, and Peter are just a few members of the team keeping eyes-on-glass all-day-every-day.
This is 360° security at its best. You’re invited to test drive our comprehensive MDR, phishing and hunting services to experience the full benefits.
