Security operations · 3 MIN READ · JAMES JURAN AND SAM BROWN · JAN 5, 2023 · TAGS: MDR
What happened?
Expel is aware of CircleCI’s reported security incident and their recommendation to rotate all credentials stored in their system. Expel uses CircleCI, so we’re closely monitoring this situation for updates and we’re taking action ourselves.
Why does it matter?
CircleCI is a CI/CD (continuous integration and continuous delivery) platform used by more than a million engineers, Expel included. CI/CD systems often contain many powerful credentials, as they are a key part of the pipeline to ship software.
At this time, there is no evidence that any of Expel’s credentials have been improperly used. But, based on CircleCI’s announcement, we’re acting out of an abundance of caution.
What’re we doing?
The good news: we anticipated these risks and tabletopped this situation starting all the way back in 2018. As a result of our tabletop exercises and risk analyses, we already have automated daily rotation in place for our highest-risk credentials. This means that if those credentials were exfiltrated, attackers would only have 24 hours to use them before they became useless. Additionally, Expel has detection systems within our environment to trigger on use of exposed credentials.
In response to this specific incident, we’ve inventoried all credentials stored in CircleCI. We’re rotating them as quickly as possible, and are reviewing logs of the use of those credentials for anomalous activity.
What should you do right now?
First: figure out if you use CircleCI in your organization. If you already work closely with your engineering team(s), you probably already know what they use for CI/CD. But, if software development is distributed throughout your organization and you don’t have perfect visibility into their tooling, it may take some investigation. Pro tip: If you have a friend on your finance team, ask them if they pay a vendor named CircleCI or Circle Internet Services—that might be faster than tracking down a bunch of engineering teams.
If you know you use CircleCI, it’s time to take action right away. First, eliminate the potential risk in your environment by rotating every credential stored in CircleCI. CircleCI has provided guidance about all the places secrets can be stored in CircleCI.
For each one, go to the source of the credential, and rotate it. Exactly how you do this will depend on what it is. For example, if it’s a user account in a ticketing system, change the password for that user account. If it’s an API key or SSH key, disable or delete it, and make a new one. Replace the old credential in CircleCI with the new one. If you have a lot of credentials, this will take a while. You’ll probably want to compile a list and split it up among multiple people. You may want to get buy-in from engineering leadership to have engineers help with this, and accept the fact this might cause some interruptions in your engineering team’s work. We think that’s a good tradeoff to make to protect your organization’s security in this situation, based on the information available from CircleCI at this time.
Once you’ve rotated all your credentials, you’ve achieved the first goal: eliminating the risk to your environment if your credentials were exposed. But, you also want to know if your credentials were actually used improperly. This is going to require reviewing the logs of usage of all those credentials. This is probably going to be time-consuming. You might even discover some gaps in your auditing; make notes of these to consider improving in the future.
At the end of this exercise, you’ll know where you stand. Hopefully you can breathe a sigh of relief that you dodged a bullet. If you uncover suspicious activity, that’s your cue to begin your incident response process.
What can you do longer term?
CI/CD systems are a big target for attackers, because they have a lot of powerful credentials. If you want to reduce your risk from this sort of threat, two things we’ve done that we recommend you consider doing are:
- Implement short-lived credential access and rotation workflow. We use Hashicorp Vault for this. See our blog post on using Hashicorp Vault to manage database credentials for more awesome things you can do with Hashicorp Vault to reduce your risk from long-lived credentials.
- Get a “canary” tool and use it. Create some credentials that aren’t actually used for anything and put them in any place (like your CI/CD tool) that stores credentials. Don’t make it obvious that they are canary credentials, of course! Your canary tool will monitor if they are used and alert you.
What next?
Like we said, we’re monitoring this situation closely. Keep an eye out here and on our socials (@ExpelSecurity) for any additional recommendations as we learn more.
