Why is Kubernetes security so challenging? Expert insights on securing container orchestration

Introduction

Kubernetes (abbreviated as K8s—eight letters between the K and the s) has revolutionized how organizations deploy, scale, and manage containerized applications. Born from Google’s experience running massive distributed systems and contributed to the open-source community, Kubernetes has become essential infrastructure for modern cloud-native development.

But with great power comes great complexity—and significant security challenges. In this conversation, Ben Baker sits down with Ethan Chen to explore why Kubernetes security is so difficult, what makes the attack surface unique, and how security teams can approach securing their first Kubernetes deployments.

What is Kubernetes and why is it so widely used?

Ben Baker: For those unfamiliar, can you briefly explain what Kubernetes is and why it’s so widely used within modern development?

Ethan Chen: Kubernetes is a project that sprung out of Google from their experience running different distributed systems. They wanted to contribute that to the community. Why it’s so widely used is because it really helps you manage your fleet of containers.

As different workloads have moved to containerized applications, you can imagine you’ll have hundreds, thousands, maybe even millions of containers. You need some sort of tool to manage them—to make sure all your containers are operating correctly, they don’t have any vulnerabilities, and you’re achieving your uptime objectives.

Kubernetes is this tool to manage those clusters of containers that really helps enable developer velocity. It helps move that migration from the traditional monolith web app to this composable construct that is more adaptable to modern cloud architecture and cloud application development.

Ethan Chen: It really helps abstract some of the different components in the cloud. Managing cloud applications in Kubernetes provides a couple layers, a couple of components, to manage that abstraction so you don’t have to think about all the itty-bitty details.

This abstraction is key to understanding both Kubernetes’ power and its security challenges:

Developer productivity: Kubernetes handles infrastructure concerns—load balancing, scaling, self-healing, rolling updates—allowing developers to focus on application logic rather than operational details.

Infrastructure portability: Applications packaged for Kubernetes can run consistently across different cloud providers (AWS, Azure, Google Cloud) and on-premises infrastructure with minimal modification.

Resource efficiency: Kubernetes optimizes container placement and resource utilization, allowing organizations to run more workloads on less infrastructure.

Declarative configuration: Rather than scripting imperative deployment steps, operators declare desired state and Kubernetes continuously works to maintain it.

Ecosystem and tooling: A massive ecosystem of tools, operators, and integrations has grown around Kubernetes, making it the standard platform for cloud-native development.

These benefits explain why Kubernetes adoption has exploded—but they also contribute to the security challenges that make defending Kubernetes environments so complex.

The security challenges: Why Kubernetes is so difficult to secure

Ben Baker: Kubernetes offers agility and flexibility, but there are definitely security challenges. It’s a complex technology to secure. Why is it so challenging from a security perspective compared to other environments?

Ethan Chen: It’s definitely very highly dynamic and complex. The number of components you’re managing—if you think about the cloud, it’s already very ephemeral. You’re spinning up things, tearing down things. With Kubernetes, it just takes that to 11.

If you think about it, you have each Kubernetes cluster, and each cluster has many nodes. Within those nodes, you have many different pods. Within those pods you have many different containers.

Ben Baker: It’s like a Russian doll nightmare is what we’re dealing with.

Ethan Chen: To manage all that, and then they all have to be able to talk to each other. It’s a huge attack surface, and there’s lots of noise and action happening all at the same time. And you can’t even reliably attribute it to a specific container because you could totally throw that away. It will be destroyed and you spin up a new container.

Huge amounts of noise, very hard to attribute, very hard to understand context. To make it all operational—to really think about and manage IAM, RBAC configurations, encryption—there’s just so much to think about.

Let’s break down these challenges in more detail:

Ephemeral and dynamic infrastructure

Traditional security monitoring assumes relatively stable infrastructure:

• Servers have persistent identities and long lifespans
• Network connections follow predictable patterns
• Logs can be correlated to specific systems over time

Kubernetes breaks all these assumptions:

• Containers spin up and down in seconds based on demand
• Pod IP addresses change constantly as pods are rescheduled
• The same logical application component might run on different nodes at different times
• Network policies must account for constant topology changes

Security implication: Traditional signature-based detection and static network segmentation don’t work well in Kubernetes. Security must be identity-based and policy-driven rather than IP-based and static.

Layered complexity (the Russian doll problem)

Ben Baker’s “Russian doll nightmare” perfectly captures the nested complexity:

Cluster level: Multiple clusters in an organization, each potentially running different workloads with different security requirements.

Node level: Physical or virtual machines that host container workloads. Node compromise can affect all pods on that node.

Pod level: The smallest deployable unit in Kubernetes, potentially containing one or more containers that share networking and storage.

Container level: Individual application instances that may be created and destroyed hundreds or thousands of times daily.

Security implication: A vulnerability or misconfiguration at any layer can cascade. Securing Kubernetes requires defense in depth across all layers—you can’t just secure containers and ignore cluster configuration or node hardening.

Massive attack surface

The communication requirements Ethan mentions create enormous attack surface:

API server: The central control plane component that all clients interact with—a high-value target.

Inter-container communication: Containers within pods share network namespace, creating lateral movement opportunities.

Pod-to-pod communication: By default, all pods can communicate with all other pods, requiring network policies to restrict.

External access: Ingress controllers, load balancers, and service endpoints expose applications to external networks.

Access to cloud resources: Many Kubernetes workloads need access to cloud services (S3, databases, secrets managers), expanding the attack surface beyond the cluster.

Security implication: Organizations must implement multiple layers of network segmentation, authentication, and authorization—but doing so without breaking application functionality requires deep understanding of legitimate communication patterns.

Attribution and visibility challenges

Ethan Chen: You can’t even reliably attribute it to a specific container because you could throw that away. It will be destroyed and you spin up a new container. Huge amounts of noise, very hard to attribute, very hard to understand context.

When investigating suspicious activity:

• The container that exhibited the behavior may no longer exist
• Logs may be lost if not exported before container termination
• Correlating activity across ephemeral identities is difficult
• Understanding what’s “normal” is challenging when the environment constantly changes

Security implication: Traditional forensics and investigation techniques must adapt. Organizations need:

• Centralized log aggregation that persists beyond container lifetime
• Identity-based rather than IP-based correlation
• Behavioral baselines that account for legitimate variation
• Immutable container images so terminated containers can be recreated for analysis

Configuration complexity

Ethan Chen: To make it all operational—to really think about and manage IAM, RBAC configurations, encryption—there’s just so much to think about.

Kubernetes security requires configuring:

RBAC (Role-based access control): Defining who can do what across namespaces, services, and resources. Even experienced administrators struggle with Kubernetes RBAC complexity.

Network policies: Specifying which pods can communicate, requiring understanding of application communication patterns.

Pod security policies/admission controllers: Enforcing security constraints on pod specifications (privileged containers, host networking, volume mounts).

Secrets management: Securely storing and injecting credentials, API keys, and certificates.

Image security: Ensuring only trusted, vulnerability-scanned images are deployed.

Service mesh configuration: If using service mesh technologies, additional layers of authentication, authorization, and encryption.

Security implication: Misconfiguration is easy and common. Organizations need:

• Infrastructure-as-code practices to version-control configuration
• Automated policy enforcement (admission controllers)
• Regular configuration audits
• Specialized security platforms that understand Kubernetes constructs

The cargo ship analogy: Understanding Kubernetes architecture

Ben Baker: We wrote a blog on Kubernetes that uses a cargo shipping analogy to describe Kubernetes architecture. How does this analogy help readers understand Kubernetes components?

Ethan Chen: My goal here was really trying to create a mental model. How do we think about these components? How do these components relate to each other? Because it’s a very technical subject, and at least for me, it was very helpful to piece something together that I already understood to help me understand how nodes and pods relate to each other, how containers relate to each other.

That just makes the technology a lot more accessible. We’re all about accessibility and transparency here at Expel. Being able to understand these components helps you define your attack surface. Those are inputs you need for your monitoring strategy, your logging strategy, your security process strategy, and the controls in your organization. This helps you build defense in depth, especially given how powerful Kubernetes is for your developers, but also for the different threat actors that could take advantage of Kubernetes.

The cargo shipping analogy works like this:

Container ships = Kubernetes clusters

Just as container ships transport goods across oceans, Kubernetes clusters orchestrate and manage containerized applications across infrastructure.

• Ships have captains and crews → Clusters have control planes managing worker nodes
• Ships follow routes and schedules → Clusters follow deployment configurations and scaling policies
• Ships have capacity limits → Clusters have resource constraints (CPU, memory, storage)

Containers (shipping) = Containers (Kubernetes)

The shipping container revolutionized global trade by standardizing how goods are packaged and transported. Similarly, software containers standardize application packaging.

• Shipping containers hold diverse cargo → Software containers package applications with dependencies
• Containers are modular and stackable → Software containers are portable and composable
• Containers are loaded, moved, and unloaded without opening → Software containers deploy without modifying internal contents

Cargo = Application workloads

The actual goods inside shipping containers represent the applications running inside software containers.

• Different cargo has different requirements → Different applications need different resources and configurations
• Dangerous cargo needs special handling → Privileged or security-sensitive workloads need tighter controls
• Cargo manifests track what’s shipped → Kubernetes manifests describe desired application state

Ports = Kubernetes nodes

Ports where ships load and unload cargo correspond to the physical or virtual machines (nodes) where containers actually run.

• Ports have limited dock space → Nodes have finite CPU, memory, and storage
• Ships move between ports → Containers can be scheduled across different nodes
• Ports have different capabilities → Nodes can be labeled with capabilities (GPU, high-memory, etc.)

This mental model helps security teams understand:

Where to apply controls: Just as you’d secure ports, cargo handling procedures, ship access, and the cargo itself, Kubernetes security requires controls at cluster, node, pod, and container levels.

Communication patterns: Understanding how components interact (ships to ports, cargo between ships) helps design network policies and segmentation.

Resource management: Recognizing capacity constraints helps set resource limits that prevent denial-of-service from resource exhaustion.

Failure domains: Understanding that problems at one level cascade (sinking ship, compromised port) informs defense-in-depth strategies.

Real-world example: The Tesla cryptojacking attack

Ben Baker: One thing we mentioned in the blog was a cryptojacking attack that occurred back in 2018 with Tesla. That was a high-profile cryptojacking attack in a Kubernetes environment. I would assume it was successful because these sorts of low-and-slow attacks like cryptojacking and data exfiltration will embed themselves into your environment and just suck your resources away without you really noticing a difference. I feel like Kubernetes is especially susceptible to an attack like that because it’s so complex. Is there merit to that?

Ethan Chen: Yeah, I would say the complexity with Kubernetes lends itself to the wide attack surface it has, and that creates more risk to attacks like cryptojacking. With Kubernetes itself, it has so much power to spin up so many containers that you can mine crypto much better. There’s a higher ROI, higher payoff for the attacker.

The 2018 Tesla incident illustrates several key Kubernetes security lessons:

The attack vector

Attackers gained access through an exposed, unauthenticated Kubernetes console. This highlights the danger of:

• Default configurations that prioritize ease of use over security
• Insufficient network segmentation allowing internet exposure of admin interfaces
• Lack of authentication on critical management consoles

The payload

Once inside, attackers:

• Deployed cryptomining containers across the Kubernetes cluster
• Used Tesla’s cloud compute resources to mine cryptocurrency
• Operated “low and slow” to avoid triggering resource utilization alarms

Why Kubernetes amplified the impact

Resource multiplication: A single compromised Kubernetes cluster can spin up hundreds or thousands of malicious containers across many nodes, multiplying the attacker’s compute power far beyond what a single compromised server could provide.

Auto-scaling exploitation: Kubernetes auto-scaling features meant that as cryptomining consumed resources, Kubernetes automatically scaled up to provide more—increasing both mining output and cloud costs.

Noise camouflage: As Ethan noted, the huge volume of legitimate activity in Kubernetes environments makes malicious activity harder to spot. Cryptomining containers can blend into the noise of normal container churn.

Persistence mechanisms: Kubernetes’ self-healing nature works against defenders—manually terminating malicious pods is futile if Kubernetes keeps recreating them based on deployment configuration.

Detection challenges

Ben Baker: These low-and-slow attacks will just suck your resources away without you really noticing a difference.

Cryptojacking in Kubernetes is particularly hard to detect because:

• Baseline resource utilization is already variable due to legitimate workload changes
• Individual malicious containers may consume modest resources, only appearing significant in aggregate
• Cloud cost increases might be attributed to business growth rather than recognized as security incidents
• Traditional network-based detection doesn’t catch mining activity that stays within the cluster

Lessons for defenders

The Tesla incident emphasizes:

• Secure defaults: Never expose Kubernetes management interfaces without authentication
• Network segmentation: Control plane components should not be internet-accessible
• Resource monitoring: Unusual resource consumption patterns need investigation
• Cost anomaly detection: Unexpected cloud cost increases warrant security review
• Least privilege: Limit what compromised credentials can accomplish
• Immutable infrastructure: Make unauthorized deployment modifications immediately visible

Advice for securing Kubernetes: Start small and build defense in depth

Ben Baker: What advice would you give security teams who are beginning to secure a Kubernetes environment?

Ethan Chen: Start small. Don’t just shift all your applications to Kubernetes and containers and assume everything will be okay. Think about the first application you might want to do a test case with Kubernetes, and see what experiments you can run there to figure out your security strategy and operational strategy.

Within that small experiment initially, focus on visibility. Making sure you understand what’s going on, you can diagnose issues, and refine from there.

After that, focus on configuration. Think about minimizing cluster misconfiguration. Define some blueprints and templates.

Then move to IAM and least privilege. Look at RBAC and the different ways you can lock down privileges and the blast radius within Kubernetes itself.

Finally, look at cluster-specific privileges. Security context, seccomp, admission controllers—trying to create those layers of defense.

Experiment with that small deployment before slowly rolling out, and see from that experience how you can deploy it across the rest of your organization.

Ben Baker: If you don’t have a solid foundation, if you try to scale up a foundation with a bunch of issues in it, you’re gonna have a problem.

Ethan Chen: Definitely.

Let’s expand on this phased approach:

Phase 1: Start with a pilot application (weeks 1-4)

Choose wisely: Select an application that:

• Is not business-critical (so security experiments don’t impact production)
• Represents typical workload patterns you expect to run on Kubernetes
• Has understood communication requirements and dependencies
• Gives your team learning opportunities without excessive pressure

Build the right team: Include:

• Application developers who understand the workload
• Platform engineers who will operate Kubernetes
• Security practitioners who will secure it
• Cloud architects who understand the infrastructure

Define success criteria: What does “secure enough to proceed” look like? Document specific requirements around authentication, authorization, network segmentation, secrets management, and logging before you start.

Phase 2: Establish visibility (weeks 1-6)

Ethan Chen: Focus on visibility. Making sure you understand what’s going on, you can diagnose issues, and refine from there.

Before you can secure something, you must understand it:

Deploy comprehensive logging:

• Kubernetes audit logs (who did what to the API server)
• Container logs (application activity)
• Node logs (host-level activity)
• Network flow logs (communication patterns)

Centralize logs immediately: Don’t rely on logs stored on ephemeral containers or nodes. Export to a SIEM or centralized logging platform that persists beyond container lifetime.

Understand legitimate behavior:

• Which pods communicate with which services?
• What API calls do applications make to the Kubernetes API?
• What resources (CPU, memory, network) do workloads typically consume?
• Which users and service accounts access which resources?

Implement monitoring and alerting:

• Resource utilization by namespace, pod, container
• API server authentication failures and authorization denials
• Pod creation/deletion patterns
• Image pull activity (especially from untrusted registries)

Visibility provides the foundation for all subsequent security controls. You can’t write effective detection rules, network policies, or RBAC configurations without understanding what normal looks like.

Phase 3: Lock down configuration (weeks 4-12)

Ethan Chen: Focus on configuration. Minimize cluster misconfiguration. Define blueprints.

Configuration issues are the most common Kubernetes security problems:

Cluster hardening:

• Change default passwords and disable default accounts
• Enable TLS for all cluster communication
• Restrict API server access to authenticated users only
• Configure audit logging
• Keep Kubernetes version current with security patches

Define security baselines:

• Which pod security policies are required?
• What resource limits should default pods have?
• Which namespaces isolate different workload types?
• What network policies restrict pod communication?

Create reusable templates:

• Helm charts or Kustomize configurations for common application patterns
• Include security best practices in templates (non-root users, read-only filesystems, drop capabilities)
• Version-control all configuration in Git
• Use GitOps workflows (ArgoCD, Flux) for auditable deployments

Implement admission controllers:

• Prevent privileged containers unless explicitly approved
• Block containers from untrusted registries
• Enforce resource limits to prevent denial-of-service
• Require security contexts on all pods

Infrastructure-as-code ensures security configurations are consistent, reviewable, and repeatable across environments.

Phase 4: Implement least privilege (weeks 8-16)

Ethan Chen: Move to IAM and least privilege. Look at RBAC and the different ways you can lock down privileges and blast radius within Kubernetes.

Limit what compromised credentials can accomplish:

Kubernetes RBAC:

• Create service accounts for each workload (not the default)
• Grant minimal permissions required for functionality
• Use namespaces to isolate tenants or applications
• Regularly audit RBAC bindings for overprivileged accounts

Cloud IAM integration:

• Use workload identity (AWS IRSA, GCP Workload Identity, Azure Pod Identity)
• Grant cloud permissions to specific service accounts, not all workloads
• Apply principle of least privilege to cloud resource access
• Monitor cloud API calls from Kubernetes workloads

Network segmentation:

• Implement network policies to restrict pod-to-pod communication
• Default deny, explicitly allow only required connections
• Isolate sensitive workloads in separate namespaces with strict policies
• Consider service mesh for mutual TLS and fine-grained authorization

Phase 5: Deploy cluster-specific security (weeks 12-20)

Ethan Chen: Look at cluster-specific privileges. Security context, seccomp, admission controllers—creating layers of defense.

Kubernetes-native security features add defense in depth:

Pod security contexts:

• Run containers as non-root users
• Use read-only root filesystems where possible
• Drop unnecessary Linux capabilities
• Prevent privilege escalation

SecComp and AppArmor:

• Restrict system calls containers can make
• Reduce kernel attack surface
• Profile applications to understand required system calls
• Deny everything else

Admission controllers:

• Validate pod specifications before deployment
• Mutate configurations to inject security settings
• Integrate with policy engines (OPA, Kyverno)
• Block non-compliant deployments

Image security:

• Scan images for vulnerabilities before deployment
• Sign images and verify signatures
• Maintain approved base image repository
• Automate remediation of vulnerable images

Phase 6: Scale securely (ongoing)

Ethan Chen: Experiment with that small deployment before slowly rolling out, and see how you can deploy it across the rest of your organization.

Once your pilot demonstrates effective security:

• Document learnings: What worked? What didn’t? What took longer than expected? What surprised you?
• Create security patterns: Codify successful security configurations as reusable patterns for other teams.
• Train additional teams: Share knowledge with teams deploying new applications to Kubernetes.
• Automate security checks: Integrate security scanning, policy validation, and compliance checks into CI/CD pipelines.
• Monitor at scale: Ensure visibility and alerting scale as you add clusters and workloads.
• Continuously improve: Security is never “done”—continue refining based on new threats, tool capabilities, and organizational needs.

Frequently asked questions

Is Kubernetes secure by default?

No. Kubernetes prioritizes flexibility and functionality over security in default configurations. Many default settings (no authentication required, overly permissive RBAC, no network policies, privileged containers allowed) are insecure and require hardening. Organizations must explicitly configure security controls appropriate to their risk tolerance.

Do I need special security tools for Kubernetes?

Traditional security tools (endpoint protection, network firewalls, SIEM) provide some value but miss Kubernetes-specific threats. Effective Kubernetes security requires tools that understand Kubernetes constructs (pods, namespaces, RBAC), can correlate activity across ephemeral containers, and integrate with Kubernetes APIs. Specialized Kubernetes security platforms or MDR providers with Kubernetes expertise significantly improve security outcomes.

What’s the biggest Kubernetes security mistake organizations make?

Exposing the Kubernetes API server or dashboard to the internet without authentication. This single misconfiguration grants attackers complete cluster control, allowing them to deploy workloads, steal secrets, or pivot to cloud resources. Always restrict API access to authenticated users from trusted networks, and never expose management interfaces publicly.

How does container security differ from Kubernetes security?

Container security focuses on the security of container images and runtime (vulnerability scanning, runtime protection, image signing). Kubernetes security additionally addresses cluster configuration, RBAC, network policies, secrets management, and the orchestration layer itself. Both are necessary—secure containers running on insecure Kubernetes still create risk.

Can I use my existing security tools with Kubernetes?

Partially. EDR agents can run on Kubernetes nodes, network tools can monitor inter-node traffic, and SIEMs can ingest Kubernetes logs. However, these tools often lack visibility into pod-to-pod communication within nodes, can’t correlate activity to ephemeral container identities, and don’t understand Kubernetes RBAC or admission control. Supplementing with Kubernetes-native security tools is usually necessary.

What’s the most important thing to secure first in Kubernetes?

Start with authentication and authorization to the Kubernetes API server. If attackers can access the API server with administrative privileges, they control the entire cluster. Secure API access with strong authentication, implement RBAC with least privilege, enable audit logging, and restrict network access to the API server before worrying about container-level security.

How do I know if my Kubernetes cluster has been compromised?

Look for: unusual pod deployments you didn’t create, unexpected resource consumption (especially CPU spikes from cryptomining), failed authentication attempts to the API server, unauthorized changes to RBAC configurations, containers pulling images from unknown registries, unusual outbound network connections from pods, and unexpected cloud API calls. Comprehensive logging and monitoring are essential for detection.

Key takeaways

Securing Kubernetes requires understanding its unique challenges and approaching security systematically:

Complexity is inherent: Kubernetes’ layered architecture (cluster → nodes → pods → containers) and dynamic, ephemeral nature create security challenges fundamentally different from traditional infrastructure.

Attack surface is massive: The power that makes Kubernetes valuable for developers—ability to rapidly deploy workloads, scale automatically, access cloud resources—also creates opportunities for attackers.

Start small and learn: Don’t migrate everything to Kubernetes at once. Start with a pilot application, establish security fundamentals, document learnings, then scale carefully.

Visibility enables everything: Comprehensive logging, monitoring, and understanding of normal behavior are prerequisites for all other security controls.

Defense in depth is essential: No single security control suffices. Layer multiple defenses at cluster, node, pod, and container levels.

Configuration is critical: Most Kubernetes security issues stem from misconfiguration. Infrastructure-as-code, policy enforcement, and regular audits are essential.

Least privilege protects against compromise: Assume breach and limit blast radius through strict RBAC, network policies, and minimal cloud permissions.

Kubernetes-specific expertise matters: General security knowledge isn’t enough. Effective Kubernetes security requires understanding Kubernetes constructs, threat models, and security capabilities.

Organizations embracing Kubernetes unlock tremendous development velocity and operational efficiency. But that same power, complexity, and dynamism create security challenges that demand specialized knowledge, purpose-built tools, and systematic approaches. By starting small, building visibility, implementing defense in depth, and continuously learning, security teams can secure Kubernetes without sacrificing the agility that makes it valuable.

For more insights on Kubernetes security, visit Expel’s blog. To learn how Expel’s Kubernetes MDR security provides comprehensive monitoring, detection, and response for container environments, schedule a demo today.