Security operations · 8 MIN READ · ETHAN CHEN · NOV 21, 2024 · TAGS: Cloud security / Managed security / MDR
Defining Kubernetes: its challenges, architecture, and exploring false positives
TL;DR
- This is part one of a two-part blog series on the fundamentals of securing Kubernetes.
- It defines Kubernetes, discusses its unique challenges and architecture, and explores why it generates false positives.
- Part two of the series will cover the four Cs of Kubernetes security and how Expel can solve Kubernetes-specific security challenges.
Kubernetes helps organizations deliver software with tremendous speed, agility, and flexibility—but it’s also notoriously difficult to secure.
On the one hand, Kubernetes makes it easier for developers to deploy, scale, and move containerized applications across infrastructure. On the other hand, the complex and constantly changing nature of a Kubernetes environment generates a tremendous volume of alerts and false positives. As SecOps teams struggle to keep up, it can be easy for real threats to slip through. That makes Kubernetes an inviting entry point for attacks including data theft, distributed denial-of-service (DDoS), cryptojacking, and more.
This blog series will discuss how organizations can help minimize Kubernetes-related risks. In part one, we’ll explain what Kubernetes is, its unique challenges, its architecture, and why it produces false positives. In part two, we’ll cover the four Cs of Kubernetes security (code, container, cluster, and cloud) and explore how Expel can solve Kubernetes security challenges.
What is Kubernetes and why is it used?
To understand the role that Kubernetes (also known as K8s, for the eight letters between “K” and “s”) plays in modern software, we’ll use a cargo shipping analogy. Most goods today travel in standardized, interchangeable boxes designed for automated loading, easy portability, and efficient transportation. These containers can also be moved easily between different types of vehicles—boats, trains, and trucks—without any changes to their contents.
Container-based software development follows the same principle. Applications are packaged with everything they need to run (code, libraries, runtimes, and so on) in standard containers that can be provisioned, scaled, and moved easily across any kind of environment to meet fast-changing business requirements.
Applying our cargo shipping analogy to containerized application deployment means that:
- The goods being shipped are applications.
- The shipping containers they’re loaded in are—no surprise—containers.
- The port manager is Kubernetes.
- The global shipping network is the cloud.
As a container orchestration platform, Kubernetes provides a versatile environment to build and run cloud-native applications without having to worry about the underlying infrastructure. DevOps teams use Kubernetes to:
- Deploy containers, move them across hybrid environments, and create or destroy them as needed for scalability.
- Connect containers with each other and with other networked services and resources.
- Ensure high availability by monitoring container health and restarting or replacing failing containers.
That’s the good news about Kubernetes. However, there’s also security to consider—which is where things get complicated.
The unique challenges of Kubernetes security
Given its role in orchestrating communications and moving resources across the network, Kubernetes is a powerful technology with a high potential for misuse. To cite just a few examples:
- Cryptojacking. Threat actors frequently breach Kubernetes clusters and hijack their cloud resources to mine cryptocurrency. A successful high-profile attack on Tesla is just one example of this method. These attacks can significantly degrade performance and increase costs.
- Data theft. Attackers who have compromised a container workload can steal the secrets stored within Kubernetes—such as passwords, tokens, API keys, and certificates—and use them to move laterally, steal adjacent data, and maintain persistent access to the cluster.
- Distributed denial-of-service (DDoS). Kubernetes can be used both to exhaust the resources of a target and to launch an attack on a different victim. The auto-scaling features of Kubernetes make it easy to generate a large volume of malicious traffic—often as part of a larger botnet.
- Backdoor creation. Attackers can use Kubernetes’ role-based access control (RBAC) to give themselves admin roles, allowing them to maintain access after the initial access vector is remediated.
- Supply chain attacks. Attackers inject malicious code into the components used to build containerized applications. When these images are pulled and deployed within a Kubernetes cluster, the malware can take advantage of its interconnectedness to move easily through the environment.
- Network attacks. Vulnerabilities in cluster networking can allow attackers to intercept, modify, or manipulate traffic to disrupt services or gain unauthorized access.
- Container runtime attacks. Attackers can use misconfigurations or vulnerabilities in the container runtime to bypass security controls, gain elevated permissions, or access the host system to abuse its resources.
Concerns like these aren’t just theoretical—they have real impacts on businesses. The 2024 Red Hat State of Kubernetes Security Report found that 67% of surveyed organizations had delayed or slowed deployment due to Kubernetes security concerns, while 46% had lost revenue or even customers due to a container or Kubernetes security incident.
Kubernetes-related security risks are especially difficult to address because they’re a direct result of the complex and dynamic nature of container environments. The same modular design that makes cloud-native development so agile also introduces numerous potential vulnerabilities throughout its architecture.
Unrestricted communication can allow unlimited abuse
A key role of Kubernetes is to facilitate communications between containerized applications, services, and the broader network. Allowing unrestricted communication by default, Kubernetes leaves users to restrict access to keep with the principle of least privilege. If users fail to do so, a single compromised element can allow attackers to gain access to data and services across the environment.
Constant change undermines visibility
Containers and pods are designed to be short-lived and easily replaceable. They’re created and destroyed continuously as needed for scalability and high availability. Along the way, patches and changes get deleted as well. In any system, security is a moment in time, not a steady state—and that’s especially true in Kubernetes.
Given this changing landscape, monitoring and logging are the foundation of Kubernetes security. Containers are continuously spinning up, moving, communicating, and interacting with network resources. SecOps teams need continuous visibility and insight to detect anomalies, respond quickly to potential attacks, and remediate vulnerabilities.
But maintaining visibility into a dynamic Kubernetes environment is much more difficult than in a traditional software architecture, or even a cloud environment. You can’t just rely on cloud audit logs to detect Kubernetes events—you have to look at the Kubernetes cluster and API server logs directly. And that’s a noisy experience. As SecOps teams struggle to keep up with an overwhelming volume and speed of events, the system can seem engineered specifically to generate false positives.
Understanding Kubernetes architecture
Before we go deeper into the challenges of Kubernetes monitoring, let’s examine the environment we need to monitor. Kubernetes architecture consists of the following components:
Cluster
A Kubernetes cluster is a set of machines that run containerized applications and manage their deployment, scaling, and operations. Using our cargo shipping analogy, the cluster is the entire port. Within the cluster, the master node (central operations hub) and worker nodes (individual docks) work together to handle containerized cargo.
The cluster’s supervising role makes it a prime attack target. Through its compromise, attackers can gain tremendous freedom to misuse the containerized applications it runs—as well as the infrastructure it resides on and the network where it’s operating.
Master node
The master node of a cluster hosts the control plane, manages its overall state, facilitates communication, and keeps the cluster running smoothly. In a shipping port, the master node would be the central operations hub where decisions are made about docking, activity coordination, and adherence to port standards and policies.
API server
The API server runs on the master node and serves as the frontend and central communication hub for the cluster. It manages information and processes deployment requests. In a shipping port, the API server would be the team in the control tower managing all incoming and outgoing messages among different docks, ships, cranes, shipping companies, and personnel.
For security architects, the API server is the most important component to focus on. A misconfigured API server can allow threat actors to gain unauthenticated access, escalate privileges, move laterally, and bypass security controls. It also provides access to the secrets in the etcd database (such as passwords, tokens, API keys, and certificates), which are stored unencrypted by default.
Kubectl
DevOps teams use kubectl to interact with Kubernetes clusters. Providing a user-friendly interface, kubectl translates user commands into API requests and formats API responses for human readability. This makes it simpler for developers to deploy applications, inspect resources, manage cluster operations, and troubleshoot. Kubectl is like the communications team that port operators rely on to relay instructions to the control tower.
Worker node
Controlled by the master node, a worker node runs the workloads and manages the resources and services for the containers assigned to it. This includes pod networking, service proxying, storage, monitoring, and logging. In a shipping port, the worker node would be an individual dock handling the cargo for a given ship.
Pods
A pod is the smallest deployable unit in a Kubernetes cluster. It consists of a group of related containers that run a process sharing the same network and storage resources. Pods are hosted by worker nodes and defined via YAML files, which identify the container(s) that each pod should run. Using our port analogy, a pod is a group of cargo shipping containers that belong to the same shipment.
Kubelet
Similar to the way the master node manages the entire cluster, a kubelet oversees an individual worker node and ensures that its containers are running correctly. The kubelet on each node reports back to the master node and receives instructions in return to start, stop, or modify containers. In a shipping port, kubelets would be the dock managers facilitating the correct loading and unloading of cargo.
Container runtime
While the kubelet receives instructions for managing the execution and lifecycle of containers, the container runtime executes these processes. This can include loading container images, starting and stopping containers, and ensuring they have the necessary resources and isolation. In a port, the container runtime would be the crew of dock workers doing the hands-on work.
etcd
etcd is the key-value store used to maintain the state and configuration of a cluster. It holds details about nodes, pods, configurations, and secrets. You can think of etcd as the shipping port’s central database, tracking the status of its ships, cargo, and docks.
As we survey the bustling port of our Kubernetes environment—so many moving parts, so much yelling back and forth—we clearly see the challenges of Kubernetes monitoring. There may well be genuine indicators of compromise (IOCs) or attack (IOAs) among all that noise. But the roar of false positives will likely drown them out.
Why Kubernetes monitoring produces so many false positives
False positives in a Kubernetes environment can have significant consequences for the business, including:
- Overwhelmed and exhausted SecOps teams, who can lose focus on real incidents within the Kubernetes environment and elsewhere in the organization.
- Failed detection of suspicious activities, performance issues, or potential security breaches, leading to damage that might have been preventable.
- Difficulty gathering accurate and relevant logging data, making it hard to conduct post-incident forensics, comply with regulatory requirements, and identify root causes of security incidents.
- Ineffective visibility into Kubernetes security incidents that may slow development. This can undermine the agility and flexibility promised by cloud-native development, and block business goals as a result.
Why does Kubernetes generate such a tremendous volume of false positives? It comes down to several factors intrinsic to the architecture of container-based applications.
Continuous change
The constant creation and destruction of containers can lead to alerts for pods that are simply being replaced as part of normal operations—or for transient issues that quickly resolve themselves with no need for attention.
Complex interdependencies
As we’ve seen, Kubernetes clusters encompass many interconnected components and services. A minor issue in one component can cascade and trigger multiple alerts across the system, even if the root cause is benign.
Resource fluctuations
Kubernetes monitoring tools often lack the context needed to understand application-specific behaviors such as auto-scaling activity, or short spikes in CPU or memory usage triggered by legitimate workloads.
Default configurations
Kubernetes is a younger technology, and its monitoring tools can be less refined than mature cloud monitoring solutions. In an excess of caution, vendors often set preconfigured alert thresholds at a level that errs on the side of false positives rather than false negatives.
In part two of this Kubernetes series, we’ll explore the four Cs of Kubernetes security (code, container, cluster, and cloud) and dive into how Expel MDR addresses the complexities of using Kubernetes in your tech stack.
Have more questions about Kubernetes security? Reach out to us here, or check out our Kubernetes mind map kit.
