As you begin your learning about Kubernetes Security, I find it oftentimes instructive to discuss the most common ways that Kubernetes might be exploited by an attacker to escalate privileges, deny access, steal data, or otherwise cause harm.
We will approach the topic by starting with how an attacker might gain access to an environment and then how they might break out to cause more damage. This is not an exhaustive list – it is just a basic list of some of the simplest ways to break into a Kubernetes cluster as well as instructions for preventing these kinds of attacks.
Method 1: Break into Kubernetes Directly
There are a few ways you might be able to break into Kubernetes directly:
- Exploit known vulnerabilities in the operating system or libraries your software uses, such as known vulnerabilities in an XML Parsing library or a certain version of OpenSSL with Apache Web Server.
- Connect to the Kubernetes API: If Kubernetes does not have Role Based Access Control enabled or has enabled authentication methods that are insecure, you may be able to use the Kubernetes API to deploy and run containers+scripts in the Kubernetes cluster.
- Steal Someone’s Authentication Credentials: If unencrypted API traffic is allowed, you may be able to packet sniff and steal authentication credentials from someone else who is using the Kubernetes’ APIs and then deploy arbitrary code. If developers have been granted too many privileges and their credentials are stolen (through various means – the most common is accidentally committing them to github or a phishing scheme), you may be able to connect to the Kubernetes API.
Once a hacker has gained enough access to the Kubernetes APIs, the exploitation is simple – they just list what is running, deploy their own containers/code (ideally with escalated privileges), and modify what is running to inject sidecars/listeners to cause as much harm as possible.
- Follow Kubernetes Best Practices: Run a tool like kube-bench (note: m9sweeper can walk you through this) to ensure your Kubernetes environment is properly setup with non-encrypted ports being blocked, role based access control enabled, etc.
- Ensure regular virus scanning and updates to your nodes (or better yet, use an operating system like Core OS or CentOS Stream that comes with next to nothing in it and is regularly patched)
- Firewalls: Require someone to be within a VPN or to tunnel through a jumphost in order to be able to connect to the Kubernetes APIs.
- Ensure regular scanning of your code and its underlying libraries, such as using sonarcube to scan your code and trivy to scan for known vulnerabilities (you can bet a hacker would do the same if they gained full or partial access to your systems).
- Principle of Least Privileges: Use firewalls to lock down all ports except those which you intend to be open to the internet. This prevents random operating system services you are not using from being exploited.
- Make it hard for attackers to know where to look: Block some HTTP headers and other kinds of indicators that would show what frameworks, libraries, and software versions you are running under the hood to make it harder or hackers to determine how they might break into your application.
Tools such as Falco can actively monitor Kubernetes API usage for suspicious activity, alerting you if someone has gained access and is in the process of iterating through each of your applications in order to attack the cluster.
Method 2: Break into Docker
Most Kubernetes clusters are running docker on every single one of their nodes. It is used to actually run the container code that kubernetes deploys. Later versions of Kubernetes have replaced docker with containerd (which is more secure and lighter weight), but earlier versions of Kubernetes are likely to still be using Docker.
Docker, under the hood, just calls containerd to run your code in containers, so why was docker even needed and what does it do? Docker is an API and set of Command Line tools for building, executing, and managing currently-running containers. When it is running on a machine, it actually runs a web API that usually has no security and, in some cases, is even exposed on the internet! This allows an attacker to connect from outside or inside the cluster and instruct docker to run arbitrary code, such as booting up a container running as root and executing a script. From there, they can perform a container breakout attack to gain further access.
Later versions of Kubernetes do not use Docker anymore. This offers many benefits beyond just security – docker is also bloated and proprietary (it will soon cost money). Docker is still an excellent tool to compile and build your docker images, but it is not the best way to run your docker images in a Kubernetes cluster.
For this reason, we recommend upgrading and ceasing the use of docker within your Kubernetes clusters.
If that is not an option or has not been implemented, be sure the ports Docker’s APIs run on are firewalled off such that no outside actor can connect to them.
Just do not use Docker in a Kubernetes environment anymore. You could track docker’s logs for issues, but the industry at large is simply moving away from running Docker in Kubernetes, so we recommend focusing your efforts there.
Method 3: Container Breakout
Once you have established the ability to deploy code, the next step is to break out of the container into the linux host and, from there, utilize traditional hacking techniques to systematically spider out into the entire organization.
Container breakout is just what it sounds like – it is when a container process breaks out of its container and gains full access to the host operating system. The easiest way to do this is to deploy a container running as root and then use the chroot command to change the container’s root folder from the container’s own folder to the root linux operating system’s folder. From there, there are any number of ways to open the machine up to an outside attacker.
Pod Security Policies/Gatekeeper: The absolute simplest way to prevent container breakout is to enable Pod Security Policies that prevent running as root, disable escalating privileges through sticky bits, and disable writing to disk (note: M9sweeper ships with Gatekeeper policies that have all the same features as PSPs but work with newer versions of Kubernetes). Coupled with regular operating system updates and security scans, and you have yourself a pretty solid strategy to prevent container breakout.
In addition to this, further tuning of SE Linux/App Armor can be used to further secure the system, and regular updates of the linux kernal/operating system are necessary in case any security exploits become available.
In extreme situations, you might look at kata containers, a method in which a separate virtual machine is spun up for each container. They are heavier than containerd, but they essentially make container breakout impossible because even if you break into the kata virtual machine, it is heavily locked down and you have not actually broken into the host node’s operating system in any way. Even so, kata containers add additional complexity, so in most cases it is not needed.