Security operations · 6 MIN READ · MARY SINGH · JUL 2, 2019 · TAGS: Get technical / How to / Managed security / SOC / Threat hunting
Peanut butter and jelly. Wine and cheese. Chips and salsa. Some things just go together.
But httpd.exe paired with cmd.exe? Not so much.
Finding anomalous process relationships — or “commands that don’t belong together” — might indicate a problem within your environment. I’m talking about big problems such as malware execution, an unknown vulnerability and worst of all … security policies not being enforced.
Wondering how you can detect anomalous parent:child relationships during threat hunting? Here are five steps for doing just that.
(By the way, If you need a primer on threat hunting, or are wondering what the heck it is and why you should consider doing it, we’ve got your back. We’ve also got some suggestions on how to choose the right security tech to use for your hunt.)
Step 1: Prepare before you hunt
Before you dive in, there are a few steps you should take first to make sure you get the most out of your hunt.
First, make sure you know what is in your environment. Asset inventory is your friend – your blunt friend who tells it like it is. Knowing what is in your environment is a prerequisite to any effective hunt because if you don’t know about it, you can’t protect it.
If you have a team, make sure you’ve got the right people with the right skill sets ready to hunt. The “right skill sets” depend on who’s on your team and the kind of organization you work for. Regardless of your team structure and industry, though, great hunters usually have a few hallmark personality traits — he or she is someone who has excellent analysis skills and is familiar with the organization’s IT systems, software and processes.
Last but not least, define the goal of your hunt. What do you want to get out of it? And how will those results help your company? Many organizations perform hunting in order to detect what active monitoring didn’t catch, and attempt to reduce “dwell time” for undetected compromises — or the time that a threat is sitting in your environment unnoticed.
Step 2: Get your data
To find out which parent:child process pairings are anomalous, you’re gonna have to gather some data. You need specific process data and context that includes:
- Timestamp
- Process name
- Process arguments
- Parent process name
- Parent process arguments
- Hostname
- User
The timestamp, hostname and user context will help you figure out whether the activity you’re investigating is legitimate or not. Those attributes will also help you identify additional activities happening during that time frame, on the same host and/or from the user in question.
Now that you know what kind of process data you’re looking for, how do you get it?
There are three different ways to collect this process data, including:
- Direct API access to Endpoint Detection & Response (EDR) vendors that track processes
- Pro: You’ll get direct access to lots of process information.
- Con: Depending on the vendor you work with, that company may or may not retain enough historical data to cover the hunt dates you’re looking for.
- SIEM queries
- Pro: It’s relatively easy to retrieve historical process data.
- Con: Your data might be impacted by how the SIEM ingests the raw process data, and how the SIEM handles the query (assuming it’s written properly).
- Windows security event logs
- Pro: You can still collect process data using event logs even if you don’t have an EDR tool in place.
- Cons: Sure, Windows security event logs are another way to collect your data, but:
- Windows Audit process tracking must be enabled (592/593 events on Win10)
- You need to fully enforce Windows Audit process tracking (using Group Policy, for example)
- Due to potential log size limits, you need to plan on sending Windows event logs to a centralized logging source, or regularly pull event data using a script such as PowerForensics or OSQuery
Step 3: Narrow down your pairs
Just like shoes and underwear, it’s possible to have too many pairs of processes. After collecting the process information, I’ve got lots of data. But yikes! There are one hundred million events.
One way to narrow down the data is to filter on parent processes. At Expel, we isolate process pairings with parent processes associated with Microsoft Office, Java, web servers, databases, and Adobe Acrobat. We have found that these parent processes are the most commonly targeted by attackers, but you can adapt this filter as needed. Figure 1 shows an example parent process filter we use at Expel:
parent_name:java.exe OR parent_name:javaw.exe OR parent_name:winword.exe OR parent_name:excel.exe OR parent_name:powerpnt.exe OR parent_name:w3wp.exe OR parent_name:httpd.exe OR parent_name:nginx.exe OR parent_name:tomcat.exe OR parent_name:sqlserver.exe OR parent_name:mysqld.exe OR parent_name:postgres.exe OR parent_name:mongod.exe OR parent_name:acrobat.exe OR parent_name:acrord32.exe
Figure 1: Example parent process filter, in Carbon Black query format
That oughta do it, right? Not quite. There are still one million events to sift through. So let’s filter our data with known legitimate process pairings such as w3wp.exe:csc.exe (“csc.exe” is a legitimate child process of IIS / “w3wp.exe”).
Figure 2 shows another example filter we use at Expel that you could try in your own environment:
…AND NOT (parent_name="winword.exe" AND (process_name="winword.exe" OR process_name="chrome.exe" OR process_args="*Microsoft Office*" OR process_name="firefox.exe" OR process_name="iexplorer.exe"))
AND NOT (parent_name="w3wp.exe" AND process_name="csc.exe")
AND NOT (parent_name="powerpnt.exe" AND (process_name="powerpnt.exe" OR process_name="chrome.exe" OR process_args="*Microsoft Office*"
Figure 2: Example parent / child / process arguments filter, in Carbon Black query format
In the example above, we added specific process arguments so the data is not over filtered. Over filtering causes false negatives, and may result in an “Oh Noes” moment when your security team (or an IR firm) finds something later that should’ve been caught previously.
While over filtering isn’t good, under filtering isn’t ideal, either. When creating filters, consider the hunting time given to the security team, the organization’s risk profile and the likelihood of each type of process being targeted. Customize and adjust your filters after each hunt to make subsequent hunts more reliable, efficient and fun over time. (Seriously, hunting can be fun!)
Step 4: Consider further analysis techniques
By using tech to automate your filtering, you’ll narrow down your events. Use the security tools you’ve got to enrich or augment process data with items such as reputation, file signature, file path, open source intel and more to help with manual review (or to filter even further).
There are a few ways to manually review the resulting process pairings. Depending on the data format, your best bet is to use one of the most reliable tools I’ve found — Excel. Every analyst has their own way of doing things, but I like to drop all my data into Excel and then sort by parent process, child process and then process arguments.
After you highlight items of interest, sort by time, host or username to determine context. Depending on what you find, you may need to pull additional information from the host in question to decide whether the process activity you’re reviewing is malicious or breaks your org’s security policy.
Alternatively, you can script the analysis with your programming language of choice. If you use Python like a normal person (Just kidding, Ruby fans!), then try implementing some interesting data analysis and manipulation with the Pandas library. If you aren’t familiar with pandas, it is an open source library that provides data structures and data analysis tools for Python.
Step 5: Share your analysis and findings
a.k.a. How to justify the time (and money) spent on hunting
Congratulations — you’ve completed your hunt! Once you’re done, now it’s time to tell your team what you found and how you propose fixing the problems (if there are any).
Organizations report their hunting results in lots of different ways — a Word document, PowerPoint slide(s), interpretive dance (Why not?) or a web reporting tool like Sharepoint, Confluence or Microsoft Teams. At Expel, we share our hunt findings with our customers right in Expel Workbench.
No matter how you choose to present the findings of your hunt, your hunt report should — at a minimum — contain the following information:
- Description of the hunt technique
- What data was collected
- What was reviewed
- What was investigated
- Findings grouped by:
- Malicious
- Suspicious
- Notable
(Or however you want to categorize your findings based on your perceived threat level)
Make hunting part of your regular security responsibilities
While hunting for anomalous process relationships can help you uncover malicious activity, I have to tell you that it doesn’t always reveal APTz. Hunting may not even reveal malware.
But here’s the thing: If your organization is well protected and employees aren’t breaking protocols, then your hunting results shouldn’t be all that riveting. And that’s good. But on the other hand, if your web server is ever compromised and an attacker runs a webshell to execute a command shell, you’ll find it with this “anomalous process pairings” hunt technique. Sure, every organization hopes that they’ll never fall victim to an attack, but we all know that it’ll happen eventually.
Want to learn more about threat hunting? Then check out this post and then this post. Interested in hearing more about how we hunt here at Expel? Drop us a note.
