Security operations · 7 MIN READ · ANDREW PRITCHETT · JUN 4, 2019 · TAGS: Get technical / How to / SOC / Threat hunting / Tools
Before joining Expel, I worked as a police officer in Alaska and spent my days focused on digital forensics. I loved combing through heaps of forensic data to find that one photo or text message to seal the case. There were more than a few cases where, after serving a warrant and seizing equipment, we had an entire rack of computers, phones and media storage devices waiting for analysis. I jumped right in, eager to see where the data would take me, and then I’d click on the “start” button to kick off that analysis. Several hours later, I’d return and see that only a portion of the processing actually completed. But finding that one hidden gem required countless hours of string searching, regular expression searching, hash matching, dissecting sqlite data and plists, or searching through hex for header byte sequences.
When I’m digging through a mountain of data looking for evidence of a crime, or poking around trying to find an unauthorized login on a company’s network, threat hunting feels the same — it’s still that proverbial “needle in a haystack”-type task. Although my old school version of threat hunting was more “bows and arrows” than NCIS, the goal of threat hunting today is still to separate the needle from the haystack — where the needle is an unauthorized activity and the haystack is “business as usual” within the network.
Today, there are lots of shiny tools at your disposal to use for threat hunting. But how do you decide which is right for your hunt? And will using multiple technologies speed up or slow down your hunt? Or does it depend on what you’re hunting for?
If you’ve already got the basics of threat hunting down and know what you’ll be looking for, here are some tips on how to choose the right weapon (er, tech) to carry out your hunt. (Psst: If you’re new to threat hunting and are looking for more info on what threat hunting is and if it’s right for your org, then you should read this post first.)
Choosing the right weapon tool
OPTION 1: USE YOUR EXISTING SIEM
Use this technique when… You’re looking for a low-cost, low-investment method of hunting, or when you’re just testing the waters to see if threat hunting is something you (and your team, if you’re lucky enough to have one) want to explore for your org. This method is especially useful when you don’t have a development team at your disposal. If you’re using your SIEM as your primary hunting tool, consider testing hunt techniques where you can aggregate multiple data sources. For example, when hunting for suspicious remote logins, you could correlate event logs, sysmon and firewall logs by time, user and source IP address into a single event. Then you can easily review the source reputation, authentication, and immediate processes performed to give greater decision support.
Pros: Many SIEMs support the creation of custom dashboards, saved recurring queries, custom alerting and even custom triage workflows, such as Sumo Logic, Splunk, and Exabeam, so you can continually tweak and adjust your dashboards or queries. By using your existing SIEM to hunt, it’s easy to roll your findings into finely tuned detections. As you discover true findings and learn what they look like, use your dashboards to look for specific indicators like domains, IPs and hashes that have helped you identify true findings in the past. Some SIEMs have great APIs where you can export data in order to augment that with other data sources elsewhere, such as OSINT or public/private intel APIs.
Cons: There are a few downsides to using your SIEM for hunting. First, you’re often limited by the types of data available from your SIEM, and the SIEM’s retention rates which may be determined by corporate policy. For example, if you have data type X and Y but also need Z and your SIEM doesn’t have it, then you’re missing an important piece of the threat hunting puzzle. At some organizations security administrators will happily make changes to SIEM data sources, while others will make you jump through hoops to get that additional API data fed into the SIEM. Also, it can be hard to enrich SIEM data with things like open source intel, public/private intel APIs or endpoint tech data. Some orgs have gotten around this by having multiple SIEM environments that are governed by different corporate policies. The downside of using multiple SIEMs? Threat hunting gets a lot more expensive.
OPTION 2: USE YOUR EXISTING ENDPOINT DETECTION AND RESPONSE (EDR) TECH
Use this technique when… You’re targeting specific behavioral patterns of attackers on endpoints and you have some development support for your team. Using EDR tech for threat hunting can be a great start but in order to get even more out of hunting with an EDR tech, you’ll need to do some work to take advantage of that EDR tech’s data. Over the years, I’ve seen a lot of orgs waste resources by hunting in their EDR tech for open source indicators. Don’t fall into that trap! Automate this task with saved queries and make sure that your EDR tool isn’t already configured to do this work for you. Don’t waste your time creating queries for known malware hashes your EDR tech is already designed to alert on. Instead, take advantage of the EDR’s rich monitoring capability and create queries that focus on behavioral frameworks such as the MITRE ATT&CK Framework. Hunting should augment your detections and help fill gaps, not create extra or redundant triage work for you.
Pros: Some EDR technologies like Tanium and Endgame allow you to rapidly query a targeted set of hosts and return a wealth of information from process executions with command line and process relationships with hashes. Carbon Black Response and CrowdStrike Falcon have phenomenal detail in describing process events and capture persistence commands, registry modifications, file modifications and network connections. Many EDR technologies today offer super detailed and targeted views of your endpoints. A good API, such as Carbon Black’s or Endgame’s, allows you to target and export very specific process details from a large number of hosts, and fast.
Cons: It’s not always easy to get the data that you need from your EDR. You’ll need some way to export the data or pull it via API, a way to store it and a way to work with it. This could make it difficult for you to stack, sort and filter large volumes of hunt data — which means you can’t enrich your hunt data with OSINT, public/private APIs or data from your SIEM. You also need to be sure that your EDR is retaining the data that you need (or that you’re storing it elsewhere). If not, you’re limited to hunting in short intervals or in real-time.
OPTION 3: SECURITY TECH CONSOLE HUNTING
Use this technique when… You don’t have a SIEM or any EDR tech. Console hunting is a technique where you log into a particular security tech console like Palo Alto Networks or SentinelOne and hunt through the specific data provided through that console. For example, you could log into your firewall console and search for anomalous traffic. Though this technique might get results over time, it’s labor intensive. I’ve seen some analysts fall into the trap of aimlessly clicking around in the hopes of finding a gem within the data. This usually turns into a waste of time. If you want to use your console for hunting, here’s a pro tip: outline a few guidelines to keep your analysts focused on gap analysis detection. The goal here is to identify what your security tech is not already alerting on. How can you leverage the visibility of your tech to detect behavioral indicators not yet being alerted on? Security tech alert review should take place in a separate workflow outside of hunting.
Pros: If you already own some security tech and your analysts know how to use those tools, this is a super cost-effective option. Having a well-defined hunt to maintain your focus will keep you out of the analysis traps I mentioned above.
Cons: Many security tools have less-than-great UIs. In my experience, the flashier and more futuristic the UI is, the less useful they really are and the more cumbersome they become to actually use. When it comes to using security tech for hunting, you’re usually limited to the UI capabilities in the console. Generally, each security console looks at a limited scope of a specific aspect of your security posture. So if you’re using your console for hunting, export your console data (if you can) so that you can manipulate it further, or feed it into a SIEM if you have one and enrich that data with data from other sources. At the very least, exporting that data into your SIEM will give you more searching, sorting, and filtering options than your console’s UI will.
OPTION 4: USING A CUSTOM SCRIPTING INTERPRETER
Use this technique when… You don’t have a SIEM or EDR tech, or when you want to enrich data from your SIEM or EDR tech. Custom scripting will require you to build out some custom tools or research and implement an open-source build found online. FYI, if you’re going this route then make sure you have dev support, or at least a team member who’s comfortable writing scripts.
Pros: You can interface with any security tech with an available API and even enrich hunt data with OSINT and public/private intel APIs. Also, if you don’t have an EDR, you can collect data using lots of available open-source projects like PowerForensics or OSQuery. With Powershell Remoting, SSH or EDR technologies you can collect raw data for parsing/analysis and hunt for the presence of specific forensic artifacts. Open-source projects such as Jupyter can ingest, parse, table and plot large volumes of collected data and provide really great decision support for your analysts.
Cons: The flexibility here is sometimes outweighed by draining your resources. Creating your own custom hunt often requires lots of resources — namely people and time. Carefully structure your hunt, verify that you have permissions to carry out the hunt as described and the development support to engineer the code and execute it securely. (Pro tip: Don’t create a super awesome forensics and data collection tool for your next attacker!)
Getting started with threat hunting
There are lots of things to think about when you create a hunting program. Beyond just selecting the tech you’ll use, spend some time thinking about your goals and the resources you already have at your disposal. Will you need anything else to execute a program successfully? A new tool, or help from your engineering team? Jot down those considerations before you dive into threat hunting.
Interested in threat hunting but don’t have the resources to carry it out on your own?
Drop us a note — we’d love to help.
