Security operations · 7 MIN READ · DAN WHALEN AND LORI EASTERLY · AUG 17, 2023 · TAGS: Management / SIEM / SOC / Tools
This was originally posted on May 23, 2018.
Throw a rock in a room full of security folk and you’d be hard pressed to hit someone who wouldn’t agree that a well-oiled SIEM can level up a security operations center (SOC) – improving threat detection capabilities, reducing time to remediation and serving as a go-to resource for your threat hunters (if you’re lucky enough to have them). But managing that SIEM can turn into a much larger effort than you anticipated when you signed the order form. And sometimes when you put a ton of effort into something it’s easy to lose sight of what you were trying to achieve in the first place. You can end up making choices that contradict your original goals (psychologists call this cognitive dissonance) And therein lies the problem. What’s an acceptable compromise? And how high should your SIEM pain tolerance be?
Four tell-tale signs you’ve lost control
1. “The SIEM is down again!”
Exasperated sighs all around. Analysts throwing their hands up in the air. Muttered grumblings…then chuckles.“I guess we should just take a break, huh?” If your SIEM crashes so often that it has acquired profane nicknames, consider it a sign.
2. Your security investigations are littered with plot holes
A plot hole in a good story can be disappointing. But a plot hole in a security investigation can be the difference between a false positive and a business-ending incident (yes, that’s a bit dramatic – but not unheard of).
If your analysts or incident responders are having trouble answering basic investigative questions like:
- What is it?
- How did it get here?
- Did it run?
- What happened after?
- Was data accessed?
… you may be headed for an unfortunately dramatic chapter.
3. You’ve created a SIEM-to-human language lexicon
When you first got your SIEM It probably seemed simple enough: a few data sources, a reasonable detection strategy and process documentation for your analysis team. But fast forward a few months, and pressures to plug new tools in, anxiety about new alerting needs and a heap of unexpected business requirements have seriously complicated things. In fact, it seems like nobody knows what’s going on. You’re not even entirely sure what data is actually available in your SIEM these days. The last time you tried to answer questions like “do we detect attacker technique X?” your head started to spin. You’ve tried to thoroughly document data sources, formats, detection rules and other nuances along the way, but keeping this information up to date is hard (and it’s even harder to make it interesting enough for your analysts to do it).
Keep an eye out for the following signs that your team is feeling data management pain:
- Only one or two analysts really understand how the SIEM is generating alerts
- Analysts aren’t speaking the same language – two analysts will use different terminology for the same activity
- Different analysts are taking different decision paths for same alerts
- It takes hours for analysts to acquire the evidence they need to run down an alert
- Analysts have to be experts in logging technologies to interpret alerts correctly
4. Your analysts are doing the tasks you hired your SIEM to do
If you haven’t tuned your SIEM since you installed it (or perhaps you never tuned it in the first place), there’s a good chance your analysts are spending most of their time handling low-value security alerts. If you’ve got analysts that aren’t “investigating” anymore because they’re too busy clicking the same buttons over and over again (or copy-and-pasting information from one location to another for every alert), then they’re doing what the SIEM was meant to do.
Without proper tuning, your SIEM is going to spew out alerts like a fire hydrant on a hot summer day. And that can create an unvirtuous circle. We’ve touched on this before, but predictably this doesn’t lead to happy security analysts. If analysts are constantly under water and days behind on security investigations, it’s time to take a look at where they are spending their time. And that probably means getting control of your SIEM.
To regain control, remind yourself why you got a SIEM in the first place
If you nodded your head at any (or all) of the above warning signs, hope is not lost. You can rein in your SIEM and show it who’s boss. We’ve seen many people do it. And it starts with taking a step back to look at the big picture. One of the easiest ways to shed our blinders is to remind ourselves why we got a SIEM in the first place. If you’ve got your original SIEM project requirements bouncing around your inbox or (gulp) in a file cabinet somewhere, pull them out. If not, take out a pencil and do your best to reconstruct it. Everyone’s list will be different, but chances are, it looks something like this:
- Consolidate my security data in one place
- Detect more security events and incidents
- Speed up investigation and response processes
- Reduce analyst error rates
- Quantify and report on how we’re doing
- Proactively hunt for threats we missed
OK. Now that you’ve got the list of where you want to get (back) to, get all of your stakeholders in a room and start asking some tough questions:
-
Hey analyst…
- How often can you get all of the information you need to finish an investigation?
- How quickly can you retrieve the data you need? Any outliers?
- How often do you have to augment data with information stored elsewhere (wiki, KB, your brain) to understand and interpret it?
- How quickly can you tune a false positive or create a new detection in the SIEM?
- Where do you spend the most of your time?
-
Hey security engineer…
- How easy is it to onboard a new technology to the SIEM?
- How often do you have to perform unexpected maintenance on the SIEM?
- Where do you spend most of your time?
-
Hey manager…
- How often are you able to generate the report you need with the SIEM?
- Can your team use your SIEM to provide quick answers about your security posture? Or does that require spreadsheets and unnatural acts?
Chances are, just by soliciting feedback, you’ll learn a lot about what is and isn’t working well. Whether you’ve just gone a bit off the rails or careened over a cliff, you’ve now identified some areas of opportunity.
Now…resist the urge to try and fix everything at once! As you prioritize, keep the value-to-effort ratio in mind. Identifying quick wins and the big pain points will focus your time and resources where they matter most. There are usually several smaller items that – taken together – can make a significant impact on the day-to-day workflow of your team. For example, reviewing and tuning high volume, low value events can help to limit alert fatigue. Reviewing top reporting use cases and building dashboards can enable managers self-service answer to common questions instead of tying up analysts.
To regain control, you’ll also need to come to terms with two key things your SIEM will never do.
1. A SIEM won’t solve all of your data problems
It may be tempting to simply point everything at the SIEM and declare success. But it’s not quite that simple. In fact, you probably shouldn’t make data management someone’s part time job… because it’s a full-time job.
These tips will help avoid some future pain:
- Prioritize what you put in based on what’s most valuable to stakeholders.
- Standardize on common logging formats and a single time zone across log sources (UTC if possible). Your analysts will be relieved they no longer have to apply time zone offsets. If it’s been some time since your SIEM’s been tuned, assess any needs that have changed in your organization and readjust criticalities with a ‘lessons learned’ approach.
- SIEMs require care and feeding to maintain utility in an organization that’s changing and adapting to new challenges.
- Understand how data will be used for alerting and document it.
- Agree on and document a common detection methodology. The MITRE ATT&CK framework is a great place to start.
- Routinely review the efficacy of SIEM rules using a feedback process that’s quick and easy.
2. Your SIEM != an incident response process
This may seem obvious, but installing a SIEM doesn’t mean you have an incident response process. You need to document that separately (check this out for starters), and it should be decoupled from specific technologies.
Your IR process will document steps like “Identify if the malware ran” instead of “run this query in Splunk.” The latter is great information to share within your analysis team wiki or KB, but you shouldn’t limit your IR process based on the capabilities of the technologies you have in place. Otherwise, you run the risk of sweeping technology deficiencies under the rug instead of highlighting and improving them. Define a great incident response process, and record how the day-to-day execution measures up.
In conclusion…
If you’re in the lucky crowd that still has control of your SIEM, congratulations! Just remember it’s easier to regain control when you identify things are getting out of whack early. So… put a reminder in your calendar to check for these warning signs quarterly.
If you’ve decided things are out of control, go through the steps we’ve outlined above. And don’t try to fix it all in one night – prioritizing your improvements into bite-sized chunks will make them less daunting and you’ll start to see the fruit of your labor sooner.
Finally, if you’ve decided that things are out of control and that you need some help to get them back on the rails you’re not alone. There are lots of people who can help depending on your needs:
- Talk to your SIEM vendor (with your analysts!)– they may offer professional services worth exploring, and it’s in their best interests to keep you satisfied with your deployment.
- Consultants can assist with executing on your prioritized list of improvements if you don’t have the resources to spare.
- Co-managed SIEM services may be worth exploring if you have realized that you don’t want to take on the day-to-day work of keeping your SIEM happy (but don’t breeze by the details here – make sure you’ll still have the visibility and control you need).
- If you’re looking to go a step further and augment incident detection and response work, managed detection and response solutions might be a fit as well.
Any which way, knowing which category you’re in and figuring out your next step is half the battle.
