I recently had an interesting discussion with Gianluca Brindisi from Spotify about the differences between Kubernetes security and container security. Typically, the discussion about container security focuses on general questions that aren’t focused on a specific orchestration framework like Kubernetes, such as the following:
- What images do I run and do they have secure base images?
- Is my image registry secure?
- What operating system permissions do my containers run with?
- Is my container network properly segmented?
- What processes do my containers run, and what files do those processes access at runtime?
These are all important questions when zooming out and looking at container security in general. However, in the Kubernetes context, something is lost.
Let’s illustrate the type of questions that you should ask to secure your Kubernetes cluster in addition to your containers. Some brief answers and tips are included for convenience.
Can someone bypass network segmentation by using the Kubernetes APIServer as a proxy and tunnel from one pod to another via kubectl port-forward?
It depends on what Role-Based Access Control (RBAC) permissions that pod’s service account has. (There isn’t much documentation around the pods/portforward role, but you can see it in the source code.)
If an attacker has breached one pod with restricted operating system permissions, can they escalate privileges by using the Kubernetes APIServer to launch a new pod with greater permissions?
Yes, if that pod’s service account has RBAC permissions to create a new pod and you haven’t set up additional restrictions like Pod Security Policies.
Can an attacker use raw-sockets to wreak havoc on the cluster’s container network?
Yes, by default they usually can!
Are my security assumptions about Kubernetes namespaces wrong?
This is an important question to ask yourself. Though many people assume otherwise, Kubernetes namespaces are not equivalent to security boundaries, which means that privileged pods in one namespace can impact pods in another namespace. Make sure you don’t make this mistake.
Root causes for Kubernetes security flaws
Most of the Kubernetes-specific security flaws that we see stem from one of four root causes:
- A well-intentioned developer or devops engineer once granted overly permissive RBAC permissions to a default service account in order to “make things work,” and no one today remembers that change or is aware of the lingering consequences.
- The security team isn’t familiar with Kubernetes and all of its obscure pitfalls, or the DevOps team doesn’t have enough security experience to recognize nuanced security mistakes. This is common even among extremely talented engineers because Kubernetes is both complicated and new.
- People think that Kubernetes and containers do all kinds of magic which Kubernetes/Docker actually don’t. For example, people wrongly assume that if an application is running in a container, then surely it can’t open a raw socket by default.
- The default Kubernetes configurations are too permissive (here too, raw sockets are a good example).
These four root causes lead to a variety of security flaws in Kubernetes clusters.
Securing your cluster
All of these issues can be fixed by proactively verifying that your cluster is properly configured, making sure that workloads are properly segmented in the correct Kubernetes-native way, and educating yourself and your teams about the nitty-gritty details of Kubernetes—not just containers.
The big picture: Containers, image scanning, and Kubernetes
Kubernetes is best viewed as a “cloud operating system” that runs applications called “containers.” Focusing on container security is important, and is equivalent to traditional application security. However, ignoring Kubernetes security and only focusing on container security is like ignoring Linux security and only focusing on Nginx/Apache security. The environment in which containers run, what the Container Network Interface (CNI) looks like, and what privileges the operating system called Kubernetes grants the containers (in terms of RBAC permissions) is extremely important if you want to secure your cloud.
To complete this analogy, you can think of image scanning as the cloud equivalent of source code scanning, which checks if you have known vulnerabilities in your code.
Image scanning is important, but it isn’t a replacement for a firewall, antivirus, or proper operating system configuration.
In the old days, when containers ran only on top of Docker, container security was enough. Nowadays, make sure you don’t overlook the operating system (Kubernetes) and focus only on the apps (containers), because doing so will leave large gaps in your security and compliance.