Threat intelligence · 6 MIN READ · BRUCE POTTER · JAN 11, 2021 · TAGS: Cloud security / MDR / Tech tools
You can’t trust the internet.
As a security professional, you’ve likely figured that out already.
But it turns out that we do place a lot of trust in the software and services we access each day. We expect them to:
- Provide us with the functionality we want; and
- Function properly (without allowing bad things to happen)
At the core, this is what we mean when we talk about integrity of a computing platform. It should only do what we intend and not things we don’t intend.
For instance, if I have an enterprise monitoring system, I expect it to do appropriate monitoring of things and not provide a backdoor for external threat actors to access my network.
That’s the nature of what happened in the SolarWinds Orion incident, and there are many similar examples of other supply chain incidents that have left security professionals everywhere reeling.
Sure, we’ve all built abstractions such as security assessments and third-party risk management programs to attempt to manage the potential risks associated with our systems, but the reality is the fate of our enterprises are often in other people’s hands.
When we think about cyber attacks, we often wonder: “What if I’m attacked?”
But this recent cyber attack reminds us that we also need to ask: “What if I lose trust in the tech I rely on to keep my org safe?”
Let’s talk about what we do at Expel to prepare for moments when trust is broken, how I think this translates to what we’ve observed and learned from the SolarWinds Orion breach and what you can do to be prepared against supply chain attacks in the future.
How to prepare for a supply chain attack: Run a tabletop exercise
There’s a common pattern with threat actors: they pick out a target, phish them, get access and then carry out their nefarious deeds. It might be ransomware, it might be data theft or it might be for intelligence purposes.
Whatever the reason, one thing’s for sure: you’re the target and you have to deal with the consequences.
But what if a provider that a large percentage of the internet relies on is compromised?
These types of attacks have dramatically different impacts on an organization and the response can be very different than what’s covered in a standard incident response (IR) plan.
To prepare for this type of situation, Expel runs tabletop exercises periodically.
The particular simulation I’m thinking of is one in which we presented that CircleCI went out of business. (For those who aren’t familiar, CircleCI is a SaaS solution that helps developers integrate and deploy code in a streamlined and automated way, and claims over 30 thousand customers running over a million builds a day.)
Note that during these simulations, I give incomplete information. During this exercise, the facts presented to the team lead them to believe that they hadn’t gone out of business but had in fact been completely compromised. Being the good facilitator I am, I let the team run with that idea, and the results were pretty fascinating.
To be clear, this was a simulation – CircleCI, of course, did not go out of business. The key in creating an effective exercise is to make up a scenario that would wreak havoc on your org so that every team member gets a chance to both think creatively and flex their IR muscles. And that’s what I did here.
After we got over the “oh no” moment of knowing the CI provider was compromised, we had to grapple with the potential impact an attack like this could potentially have.
Not only could we not trust our code running in production, but we couldn’t trust the code running in production of CircleCI’s 30 thousand customers.
That’s a big problem – CircleCI has a lot of big brand customers like Docker and Facebook, and well as supporting SaaS solutions of all shapes and sizes around the world.
We faced a situation where we could no longer trust the Internet in the way we had; not for production services, not for productivity solutions, not for back office systems.
The problem for the team became, “how do we continue to deliver Expel’s services in this new reality?”
The discussions during the tabletop around business continuity, communications and customer interactions were unlike any we had had in previous tabletops.
We focused on:
- Figuring out where trustworthy artifacts existed in order to reconstitute our production environment from scratch. From there, we examined what external services we were willing to continue to use and what services we had to walk away from immediately.
- Rearchitecting the entire production environment on the fly in an attempt to keep our service viable while ensuring our customers’ networks weren’t in jeopardy.
If anyone from CircleCI is reading this, hi! We love you and wish your team well. This was just a tabletop and not at all a reflection of how we view CircleCI’s security program.
A look at how supply chain attacks are evolving: SolarWinds Orion
Fast forward from when we ran our CircleCI simulation to December 2020, and we are faced with a similar real-life example with SolarWinds Orion.
While not quite the same as having your CI provider popped, it still had the potential to impact enough organizations at a deep enough level that we nearly experienced a “can’t trust the Internet” moment.
While it looks like a relatively small number of companies had direct malicious actions against them, the cybersecurity community is still sorting it all out.
But many of us are in fact reflecting on the trust we put in our providers, software and hardware.
With still unknowns about which companies and agencies are compromised and the current state of those networks, rethinking what services we rely on and the location of data means having discussions with businesses we’ve never had before.
Contrast this to the attacks we saw in the 2010 timeframe. Both the Google Aurora-style attacks as well attacks against the Defense Industrial Base (DIB) were well coordinated and sophisticated.
These were also highly targeted attacks launched against high-value organizations and subverted trust in the core of the systems in many of these enterprises. Having dealt with some of those intrusions personally, I will say recovery from them was difficult and expensive for many organizations and resulted in large scale changes in how they thought about security.
What we’re seeing in 2020 is much different.
For starters, the impact of the SolarWinds Orion hack isn’t targeted in the same way. Rather than having a few providers to worry about, we have 18 thousand that may have been compromised.
So thinking about reducing exposure to this attack has a very different feel than what we dealt with in 2010.
On the flip side, we have much better security signal now than we had in 2020. Endpoint monitoring and interrogation technology has improved dramatically.
For cloud based workloads, we have the ability to introspect on them from underneath.
For example, services like CloudTrail and GuardDuty from Amazon Web Services (AWS) give us simple, detailed information about what is executing where and providing high fidelity detection information.
Unlike 2010, even when we knew bad actors were in the network and we couldn’t find them, better instrumentation helped many of the 18 thousand impacted companies take publicly available information about these attacks and rapidly determine either, “yep, we were compromised” or “seems pretty unlikely we were compromised”.
Being 100 percent sure is not possible, but today we can have a much higher degree of assurance than we could in the past.
3 things you can do right now to better protect your org against a supply chain attack
The impact of the SolarWinds Orion hack will be felt for years to come. This is a somewhat rare event that causes organizations to lose trust in many systems and services all at once.
However, it’s critical we use this time to learn lessons and prepare for the next large scale event that causes us to question the integrity of huge swaths of the Internet.
Because it’s a given that we will have another one of these moments.
It’s possible to prepare for these events, but it requires a different kind of response than what you might normally plan or table-top for.
Here are some things you can do:
- Plan for supply chain attacks – The word “supply chain” can mean different things to different orgs, but for many tech companies, your supply chain is a long list of cloud services that facilitate your day-to-day business.
- Have plans for alternative supply chain providers – We’re not saying you need to have a hot backup for all your cloud services. But you should at least plan for potentially rapid provider shifts if a catastrophic event happens. This should be largely in line with your business continuity plans (which you’ve tested, right?).
- Be creative – Failures of imagination are a real thing. And it can be very difficult to dream up attacks like SolarWinds Orion or vulnerabilities like Heart Bleed. When planning tabletops, ask people around your company: “What’s the worst thing that could happen?” You might be surprised at the scenarios others are worrying about.
