[CyberSec] Security in Kubernetes - in short

Quick taste on best practices and approaches to improve security in Kubernetes environments

Feb 26, 2023

Kubernetes Intro
Security in Kubernetes
Kubernetes entities - The Attack Surface
Kubernetes Security pillars
External resources & References

The purpose of this article is to provide a first taste of the Security concepts in Kubernetes environments using different resources that have addressed and directed this topic. it is suggested to use the sources listed in the last section or reported in the paragraphs to delve deeper into all the covered topics topics and “go beyond”.

Kubernetes Intro

Kubernetes has become the leading container orchestration system used by enterprises, developers, and cloud service providers worldwide. It simplifies the deployment, scaling, and management of containerized applications.

However, with great power comes great responsibility, and Kubernetes is no exception. Its complexity and flexibility also mean that securing Kubernetes can be challenging. In this article, we will explore the basics of Kubernetes and how it works, and then delve into the key security considerations for Kubernetes environments. We will discuss the security risks, threats, and best practices for protecting Kubernetes clusters and their workloads, including network security, access control, authentication, and encryption. Whether you're a Kubernetes user, administrator, or security professional, this article will provide you with the insights and guidance you need to enhance the security of your Kubernetes environment.

Security in Kubernetes

It's important to understand that Kubernetes Security is not just about protecting the Kubernetes control plane itself. While securing the control plane is critical, it's equally important to protect the applications and data running within the Kubernetes environment.

One key security consideration is network security. Kubernetes clusters often span multiple hosts and networks, and the containers within a cluster need to communicate with each other securely. Kubernetes provides several networking options, such as overlay networks and service meshes, that can help ensure secure communication between containers. Additionally, you can use network policies to restrict traffic between containers based on source and destination IP addresses, ports, and protocols.

Access control is another important security consideration in Kubernetes. You need to ensure that only authorized users and applications can access and modify the Kubernetes resources. Kubernetes provides several authentication and authorization mechanisms, such as role-based access control (RBAC), webhooks, and service accounts, to help you manage access to the cluster. Additionally, you can use admission controllers to enforce policies that restrict the creation or modification of certain resources based on specific criteria, such as image provenance or label values.

Encryption is also a critical security aspect in Kubernetes. You need to ensure that all sensitive data, such as secrets, configuration files, and communication channels, are encrypted both at rest and in transit.

In conclusion, Kubernetes is a powerful and flexible platform for running containerized applications, but securing it requires careful consideration of several key factors. By following best practices for network security, access control, and encryption, you can help ensure the confidentiality, integrity, and availability of your Kubernetes environment and its workloads. Additionally, regular vulnerability scans, security audits, and compliance checks can help you identify and address security issues before they become major threats.

Kubernetes entities - The Attack Surface

Before to have a deep dive conserning the Kubernetes Security, we have to clarify and define which is the Attack Surface (entities) in a Kubernetes cluster.

A Kubernetes cluster is a group of machines, or nodes, that work together to host and manage containerized applications. In a Kubernetes cluster, the nodes are divided into two main categories:

master/control nodes
worker nodes

Control nodes (the cluster “brain”) include the API server, controller manager, etcd manager (a distributed data store for the cluster configuration) and scheduler. Worker nodes are responsible for running the applications and consist of pods that host containers. Kubernetes manages the deployment, scaling, and updating of these pods across the worker nodes to ensure high availability and fault tolerance.

Kubernetes Components - © Kubernetes.io

Kubernetes Security pillars

In order to walk through the pillars composed by best practices, standards and approaches, it is necessary to start from the areas on which to define the security oriented actions.

Node Security
Pod Security
API Security
Data Security
Network Security

A different approach is proposed for example by OWASP, that provides a starting point for securing Kubernetes cluster through its cheatsheet divided in:

Some of the concepts expressed in the OWASP cheat sheet will be covered in the following paragraphs.

Node Security

Kubernetes clusters consist of nodes, which are typically Linux-based servers, but may also be Windows-based worker nodes. The same security measures you would use to secure any server should be applied to secure Kubernetes nodes. These measures include reducing the attack surface by removing unnecessary components, user accounts, and root access. It is also recommended to use minimalist Linux distributions, such as Alpine Linux, and deploy OS-hardening frameworks like AppArmor or SELinux where possible. Collecting and analyzing OS logs is also important for detecting potential security breaches. Whether you are securing a master node or a worker node, the security considerations and procedures are generally the same. However, securing a master node is slightly more critical as a breach could have a greater impact on the entire cluster.

As a general rule, it's better to have fewer software components running on each node. To minimize the number of processes and reduce the risk of security vulnerabilities and resource waste, it's recommended to use a minimalist Linux distribution instead of a full-fledged one when provisioning nodes.

In addition, to strengthen the security of nodes where the pods are executed, we can cansider these steps:

Configuration standards and benchmarks. Configure the host in compliance with security recommendations. You can validate clusters using Center for Internet Security benchmarks that are tied to specific Kubernetes releases.
Minimize administrative access. Reduce the attack surface area by limiting administrative access on Kubernetes nodes.
Node isolation and restrictions. Run certain pods on particular nodes or groups of nodes to ensure that the pods run on nodes with specific isolation and security configurations.

Pod Security

A pod in Kubernetes is a container or a group of containers that run an application. Securing your pods is essential to securing your applications. Some pod security practices are external to Kubernetes, such as performing security tests and scanning container images. Collecting and analyzing pod logs is also important for detecting potential breaches or abuse.

However, Kubernetes offers native tools to enhance pod security, which includes RBAC policies for managing access to pods by users and services within the cluster. Security contexts define the privilege level at which pods operate, while network policies can isolate pods at the network level. Admission controllers can also enforce additional rules based on RBAC.

The specific pod security tools and configurations you need depend on the nature of your workloads. Some pods may require complete network isolation, while others need to communicate with each other. Therefore, you should evaluate the available resources for securing Kubernetes pods and use them to their fullest extent. There is no one-size-fits-all approach to pod security.

Furthermore Kubernetes does not verify the contents of containers to determine whether they contain security vulnerabilities or potential exploits. Therefore, it is crucial to perform image scanning before deployment to ensure that only images from trusted registries without critical vulnerabilities, such as remote code execution, are executed in the cluster. Integrating container image scanning into CI/CD systems can also help automate the process and identify flaws earlier in the development cycle.

In addition we have to consider a security approach even at runtime level. The runtime phase presents containerized applications with a variety of new security challenges. Your objective is to gain visibility into your running environment and detect and respond to threats as they arise. By proactively securing your containers and Kubernetes deployments during the build and deploy phases, you can significantly reduce the likelihood of security incidents during runtime and the subsequent effort required to respond to them.

API Security

The Kubernetes API serves as the central component of the entire system, providing a means for both internal and external clients to connect and communicate with Kubernetes. If you are responsible for deploying and managing Kubernetes components within your organization, you need to exercise caution because the Kubernetes API server and its associated components are open source tools that may have potential or actual vulnerabilities. Therefore, it is crucial to use the latest stable version of Kubernetes and promptly patch live clusters.

If you utilize cloud providers, the provider typically manages the Kubernetes control plane, resulting in automatic updates and patches to the cloud infrastructure. However, users are usually accountable for upgrading worker nodes. Therefore, to facilitate the process of upgrading or replacing nodes with new ones, you can leverage automation and resource provisioning tools.

The Kubernetes API is secure by default, as it only responds to properly authenticated and authorized requests. However, API authentication and authorization are managed by RBAC policies that require configuration. To secure the API, it is crucial to create secure RBAC policies that enforce the principle of least privilege and assign permissions on a granular basis.

Another best practice to enhance API security is by using admission controllers. These controllers offer a secondary layer of defense against unauthorized requests by evaluating them after they are authenticated and authorized. By enabling and configuring admission controllers, it is possible to enforce security rules related to API requests.

Finally, the network level can secure API requests by configuring secure certificates and requiring the API server to serve requests on a secure port. These measures ensure that the Kubernetes API remains secure, and unauthorized requests are prevented.

More details in API Access Control (by kubernetes.io)
More details in Admission Controllers Reference (by kubernetes.io)

Data Security

Kubernetes itself does not store much data except for non-persistent data within running pods and log data stored on nodes. Data created or accessed by clusters is typically stored in an external storage system that interfaces with Kubernetes through a storage plugin. To secure this data, it is important to follow best practices for securing data within any large-scale storage system. This includes encrypting data at rest, using access control tools to restrict data access, and ensuring that storage pool servers are properly secured. Backing up data can also help protect against data theft or ransomware attacks.

Kubernetes does not offer any special tools for securing the relatively small amounts of data that live natively inside Kubernetes pods and nodes. However, you can protect this data by following the best practices mentioned above to secure pods and nodes.

Secrets are a critical component of Kubernetes for storing sensitive data like passwords, certificates, or tokens, and using them inside containers. However, there are three critical issues related to Kubernetes secrets. First, secrets store sensitive data as base-64 encoded strings, but they are not encrypted by default. While Kubernetes does offer encryption of the resources for storage, it needs to be properly configured. Second, any pod, along with the applications running inside it, in the same namespace can access and read secrets, making it important to properly configure role-based access control (RBAC) rules. Finally, if there are old and unused secrets or ConfigMap resources, it can create confusion and leak vulnerable data. Therefore, it is important to delete any unused secrets or ConfigMap resources.

You might want to consider using an external secrets manager to store and manage your secrets, instead of storing them in Kubernetes Secrets. This offers several advantages over using Kubernetes Secrets, such as the capability to manage secrets across multiple clusters (or clouds) and the ability to centrally manage and rotate secrets."

More details in Kubernetes.io - Secrets
Additional info in OWASP Secret Management Cheat Sheet

Network Security

Networking is a convoluted aspect of Kubernetes, with various configurations available. Depending on your setup, some resources may require internal network connections, while others may need direct internet access. These settings, such as ports, IP addresses, and other network attributes, are typically configured dynamically, making it challenging to keep track of what's going on at the network level.

Kubernetes network security can be particularly demanding due to the intricacies involved. It requires a comprehensive understanding of Kubernetes networking architecture and familiarity with Kubernetes' native security tools, such as network policies, and third-party tools that can harden networks further.

Kubernetes provides some native tools to help secure networking resources, such as network policies. Although network policies are not a security feature themselves, administrators can use them to regulate traffic flow within a Kubernetes cluster. With network policies, you can achieve objectives like isolating pods from each other at the network level or filtering incoming traffic.

However, network policies should not be relied upon as the only means of securing networking configurations outside of Kubernetes. They should be viewed as an additional resource that complements the security measures incorporated into your overall network architecture.

In general securing Kubernetes networks requires administrators to use a combination of native and third-party tools, as Kubernetes typically utilizes a combination of internal resources (such as kube-proxy) and external services (such as CNI plugins and service meshes) to manage networking configurations and traffic.

Additional info in Kubernetes Security: Understanding Kubernetes Network Security (by Sysdig)

Conclusion

Kubernetes is the undisputed container management platform for running microservices applications, and there are no doubts about its fundamental role in CI/CD processes. However, its use exposes companies to varying degrees of risk if not correctly configured, deployed, and used. In this article i've tried to provide an overview of Security concept in a Kubernetes environment and what the attack surface is and linked risks. In future articles, the focus will be more on threats prevention and the visibility and observability aspects necessary to detect anomalies and cyber threats.

External resources & References

A.I. support for text and images

TribalSec’s Substack

Discussion about this post