Security operations · 8 MIN READ · BEN BRIGIDA, PETER SILBERMAN AND RAY PUGH · JAN 5, 2021 · TAGS: MDR / Tech tools
Manually triaging phishing emails is painful. We’ve heard from customers that of those who employ analysts in house to focus solely on phishing, the retention rate of those employees is usually less than a year.
No wonder.
The work is never-ending. Billions of malicious emails are sent each day.
And that’s not even the bulk of it.
End users, if properly trained, will smash the phishing reporting button for anything suspicious or annoying. Plenty of marketing emails and spam messages get caught in the nets (sorry, marketers – just calling it like we see it).
This tedious work is what makes triage so painful.
Analysts have to figure out if the email is bad, and if it is, they need to investigate it.
According to APWG’s 2020 Phishing Activity Trends Report, attackers create nearly 200,000 unique malicious websites and over 100,000 unique malicious subjects per month. That’s a lot of copy-pasting.
Yet the importance of reviewing reported phishing emails can’t be understated. Phishing is the primary source of compromises, and has been for some time.
So how do you balance the importance of reviewing phishing emails with the monotony of the work?
That’s exactly what we’re working to solve for here at Expel.
Get the right people and then build the right tech
First, we set out to solve the people problem.
By that we mean: how can we do high-quality work without making our people miserable?
We assembled a diverse, enthusiastic team of skilled analysts. (This piece is really important.)
Next, we set out to change the game and make phishing triage phun again (can’t stop, won’t stop).
At Expel we believe analysts need meaningful and interesting work. So we had to figure out how to make phishing meaningful to us while also delivering value to our customers.
We decided we don’t want to stop at just telling our customers that an email is bad. We also want to tell them things like; who else got the email and was anyone compromised (meaning someone clicked the link and submitted data, downloaded/executed the attachment, etc).
These are the questions we think are most important to our Expel for Phishing customers. And we think they should be important to anyone providing or buying a phishing service.
To accomplish this, and make the work engaging for our analysts, we use all the technology in our customer’s environment that Expel supports (a total of 55 different integrations) to actually investigate (yes, we look at things like EDR, netflow – gasp – and URL logs) and work to answer the questions above.
Great start, but how do you take tedious work and make it scalable?
If you’ve read our previous blog posts, you’ll know how strongly we believe that humans are best suited for making decisions and building relationships.
Everything else (like gathering data, crunching numbers and formatting data) is better suited for technology to handle.
Based on our experience doing this, we think that when you apply technology to aid humans, you should automate what you understand (not what you think), then measure and iterate.
So we built technology to automate the tedious steps we knew existed.The goal of this automation is to provide the right information at the right time for the analyst to make a decision.
We call this “decision support” (keep an eye out for a future blog on this topic). We want to make it so that our analysts can make informed decisions (quickly) and do what they enjoy most – finding bad guys and ruining their day.
Want a breakdown of how we make decisions about phishing emails? Gon’ give it to ya
Just like attacker tactics are constantly evolving, we’re continuously improving our approach for automation and decision support to help keep our analysts fresh and focused.
Let’s take a look at an example that was submitted for review.
Phishing email example
First things first: Is this email benign or malicious?
We have a framework we use to train our analysts to answer this question.
The framework helps break down what to think about when triaging an email.
The three buckets of our phishing investigation framework are:
Impersonation – Are there signs the sender is impersonating someone (is Simon really emailing our accounting department)? Is the link impersonating a legit domain (typosquatting)? Is the attachment posing as an image file when it’s actually a different file type altogether? Are there typos?
Intent – Does the activity we’re seeing make logical sense (would “Simon” email us about an overdue invoice)? Are the indicators consistent with legit email traffic? How would an attacker benefit from this? Are they asking for sensitive information that a legitimate person or institution would already know?
Action – Is the user prompted to take action? Is the subject matter financially motivated? Are they directed to click the link or download a file? Is there a sense of urgency (Is “Simon” demanding payment right away)?
The technology we’ve built supports surfacing information relevant to the various themes.
The first focus of our decision support was to make it fast and easy to look at the email. We safely and securely render the email and produce a PNG of the rendered email. This can tell an analyst a lot about the email. This also helps quickly eliminate things like marketing emails that were submitted for review.
In addition we use third-party data enrichment to gather additional context about the sender, receiver and more.
For example, we’re huge fans of emailrep.io and we use them to surface context on how reputable the email sender is.
Below is an example of what we‘ll see in the Expel Workbench™:
Example of automated decision support
In addition we use other services to surface context on IP, domain, information on whether file attachments are present and what the files do. All of this provides almost instant, meaningful decision support to our analysts.
As we operated the service, we also noticed a lot of duplicate emails. Usually, the only thing changed might be the signature block or the sender but the overall intent of the email was unchanged.
Using machine learning to find similar emails
To help our analysts with answering the question of whether they’ve already seen a similar email, we deployed a machine learning model that creates text embeddings of the email so that we can subsequently find semantically similar emails (if there are any) and surface those to our analysts.
Example of similar email scoring
Establishing a cycle of quality control checks
With technology and automation comes an often overlooked responsibility of continuous quality control and improvement.
We use manual quality control (QC) checks. The nice thing about QC checks is that they aren’t just serving the purpose of reviewing what the automation is producing; they’re also identifying new areas that can be automated.
Part of our QC checks review what our analysts are doing. Are they performing repetitive tasks, tasks that are error prone or tasks that are better done by technology? If so, that could be an opportunity for automation.
At Expel we’ve talked about some of our security operations quality checks. In building out the Expel for Phishing service we developed a separate set of quality control checks.
Our structured QC process entails a daily review process to make sure the technology and analyst outcomes meet our high-quality standards. Just like the MDR service, we review a sample of phishing investigations each day to make sure that we’re making the right decisions and, just as important, we took the right steps to reach the conclusion.
A second set of eyes can offer a unique perspective – we have made a number of feature requests based on these reviews. We’re talking about things from typosquatted domains identification to formatting changes in the results to draw attention to important info.
In the past 3,000 emails we analyzed, about 85 percent of them are benign (marketing emails, sales emails and generic spam). Finding ways to quickly spot and dispatch harmless emails is equally important for scaling our team.
Sifting through benign emails and identifying the true threats is just the beginning of our process. This is where the real meaningful work (i.e. fun for an analyst) begins.
Investigating a malicious email
Like I mentioned before, while other services can tell you if an email is bad or not, we think that’s not enough.
We need to be able to answer those important questions (Who else received this email? Was anyone compromised?).
Well, this is the part where I tell you how we get to those answers quickly.
Through seamless integration with customers’ security technology, our Expel Workbench™ platform provides in-depth analysis and effective resolution of email security threats.
The screenshot below shows an example of an investigative step that demonstrates the way we use our customer’s security investment to get to answers. In this situation, the analyst confirmed that a phishing website was collecting user credentials. We wanted to see who else across the enterprise had accessed the domain.
The analyst used what we call an “investigative action,” which asks multiple customer onboarded security technologies the same question without the analyst having to worry about the vendor specifics of how, since the platform takes care of that.
Here the investigative action run was Query Domain, and the analyst looked for any evidence that a user in the enterprise visited the domain in question. In this case, the platform queried an EDR, SIEM and firewall. Then, all the results were returned for the analyst to review.
Based on URL logs generated by the firewall (and a lack of SSL) the analyst could confirm that data was transferred to the website for a number of users.
Abstracting away the intricacies of each technology and giving our analysts tools, like investigative actions that run across all onboarded security devices, allows them to spend their time focusing on asking and answering investigative questions.
Example investigative action in Expel Workbench
Quick action is key for minimizing risk. Even with our automation and platform, investigating and writing a report takes a little time. We didn’t like that, so, as we investigate, we take advantage of the transparent platform to skip right to the part where we tell the customer what they need to start remediating.
We give our customer security team a heads up that we’ve found something malicious, send them a list of the indicators of compromise (IOCs) we have at that point and recommendations to mitigate as much risk as possible.
All of this uses automation and happens in a few clicks. These notifications are delivered via email, Microsoft Teams, Slack, PagerDuty or any combination the customer chooses.
Expel Workbench Notification example
When we do find a compromise (like a phishing email that tricked a user into submitting creds or running malware), we provide immediate notification with specific remediation actions for the additional IOCs.
We know that the customer team is busy, so we provide clear steps on exactly what to do. We also make sure our analysts are available to answer any questions every step of the way.
Example Findings Report
How this all helps our customers
One of the few things analysts, customers and attackers agree on; there’s a big difference between receiving a malicious email and sharing your credentials or computer with an attacker.
But identifying emails that are malicious, that no one interacted with, can quickly start to feel like busy work.
And no one likes busy work.
That doesn’t change the fact that phishing attacks aren’t going away. And attackers are getting smarter.
Which is why we think it’s good to know about the malicious emails you’re getting, but the real goal is to quickly identify the malicious emails that lead to a compromise.
We recognized the importance of thorough email scrutiny and providing our analysts with meaningful tasks, ensuring robust email security measures are in place.
“Automation is helpful, but at some point you need to have trained human eyes on these emails,”
— Expel customer, Lori Temples Vice President of IT Security and Business Continuity
We think Expel for Phishing solves this problem. Our primary objective in creating this offering was to provide ample space for our customers’ security teams to concentrate on more engaging tasks while we take care of handling suspicious email submissions and ensuring robust phishing protection.
