Don’t dam upstream: ways to build a feedback loop

I was interviewing a candidate for a security analyst role and asked one of my two favorite questions: “Talk to me about a time… or a project… where, looking back on it you think to yourself: if I never have to do that again, it’ll be too soon. What was that misery, and what made it miserable?”

The candidate had a strong technical background and his experience was right on the mark. He also had an exceptionally relevant response. He described working at a federal SOC. Overall, it was a great learning experience, he said. They were constantly finding bad stuff and he learned a lot from his peers, but neither he nor his co-workers had any ability to influence detection. A separate team handled that. And — for security reasons — neither team could talk to the other. Hah! So, every week he’d see the same false positives he’d flagged the week before… and the week before that. Over time, this bred a feeling of helplessness, boredom and eventually burnout.

His story reminded me of an article I’d read in the Harvard Business Review years ago by the CEO of Johnsonville Sausage. He was struggling with performance problems and described his employees as “so bored by their jobs that they made thoughtless, dumb mistakes. They showed up in the morning, halfheartedly did what they were told, and then went home.” Sounds terrible (side note: the thought of quality problems in sausage makes me a bit queasy).In any case, it took years, but that CEO finally came to an important realization: “Those who implement a decision and live with its consequences are the best people to make it.” The result? They changed their quality control system. Turns out, this practice applies directly to security operations and probably a lot of other disciplines as well. The people who live with the consequences of detection must be integral to deciding how intelligence and methodologies are applied in the first place. Without this feedback loop, you’re stuck with bad sausage.

Back to the interview. We were hiring for a role where the candidate would be in the exact same position he’d just said he never wanted to repeat. At the time, the feedback loop in our SOC was broken and the required fixes weren’t trivial. Even though he was an exceptionally well qualified candidate, we chose not to proceed because he’d have been miserable.These disconnects aren’t unusual. “Just add a feedback loop” is too simplistic an answer. Solving this problem in security operations is much harder. Many analysts in a SOC lack the experience to effectively drive detection. Those who do have the experience typically don’t work in the SOC (or at least, not on shift) and may have forgotten exactly how frustrating this situation can be. Still, it’s not hopeless. If you find yourself in this situation, here are four options to build in a feedback loop.

1. Align incentives

If the SOC and detection/intel team report into different managers, make it clear to your detection team’s manager that her success is measured by the SOC manager’s enthusiastic support.

2. Get physical

Is your SOC sectioned off from the rest of your security team? Reserve seats for your sister team’s personnel. If there aren’t enough seats, rotate people through.

3. Make the pain transparent

By measuring the time wasted chasing dead ends (or even the volume of dead ends) and tying those to root causes, you’ll make it clear when adjustments are needed upstream.

4. Celebrate improvement

As you use metrics to drive change in your detection methodologies, reward your teams when the needle meaningfully moves in the right direction. Common wins help unify teams.

—

This is the second part of a five part series on key areas of focus to improve security team retention. Read the introduction, 5 ways to keep your security nerds happy, or continue to part three.