How Kubernetes Vulnerability Emphasizes the Need for Distributed Security for MicroServices

Kubernetes (K8s) has rapidly become the one of the most widely adopted container orchestration platform. With its powerful combination of simple to understand constructs, providing rich capabilities, and functionalities to meet the varied need of microservices, K8s offers an attractive deployment platform for microservices applications. K8s compelling appeal is further fueled by the widespread availability of managed Kubernetes offerings from major cloud providers. Application developers can develop and deploy microservices on-premise and can seamlessly migrate it to public cloud or can deploy applications in a hybrid deployment.

Wider adoption of any technology or platform, especially one from an open source community, exposes it a wider scrutiny and of-course is often accompanied with greater attention from hackers. Kubernetes recently had its first major vulnerability discovered – CVE-2018-1002105

Kubernetes vulnerability is well documented at red-hat-open-shift and gravitational and has received quite a bit of attention from security community.

What makes it an interesting vulnerability from security analysis point of view is that it exposes the challenges of securing a distributed application.

In this post, I’m going to cover how the vulnerability can be exploited using an example request flow. I will follow this up with another post analyzing the key underlying architectural factors in K8s that contributed to the vulnerability. And most importantly how the vulnerability brings forward the challenges of securing a distributed microservice application.

The following diagram shows a high level overview of K8s. While K8s is a platform for deployment of microservices, it itself is based on microservices architecture and consists of multiple individual services that jointly work together to provide the full platform functionality.

https://kubernetes.io/docs/concepts/architecture/cloud-controller/

Understanding the Kubernetes API request flow

Kubernetes provides a rich set of APIs to enable administrators to perform various operational and policy configuration tasks. APIs also allow administrators to interact with running containers and do operations like execute a commands inside containers or get a shell to the running containers. APIs are accessed through API server which is made publicly available for devops integration, especially in a managed Kubernetes environment. Let’s look at the working of exec API that allows administrators to execute a command inside a container.

  1. User makes an API request to exec a command in POD1. API request looks as follows – “POST https://www.myk8cluster.com/api/v1/namespaces/default/pods/policy-58859d77fc-hxnz6/exec?command=ls&container=policy&container=policy&stderr=true&stdout=true”.
  2. API Server performs authentication and authorization to ensure user is authenticated and authorized to perform requested operation as per API request. In this case it ensures that user has exec access to POD1.
  3. API Server creates a separate connection to Kubelet using certificate based authentication (K8s microservices use mutual TLS to mutually authenticate and authorize). The certificate used to connect to the kubelet grants it ‘cluster-admin’ privileges, which allows full access to the kubelet. API Server forwards the request to Kubelet on the authenticated connection.
  4.  Kubelet receives the request, verifies that the request came from API server, and without doing any authentication or authorization by itself, it trusts the request from API server and executes the user specified command in POD1
  5. Once command is executed, Kubelet sends back HTTP 101 response to API server indicating to the client that it has switched to the new protocol. (SPDY/3.1). Response Headers looks like this –
    Connection: UpgradeUpgrade: SPDY/3.1
    X-Stream-Protocol-Version: v4.channel.k8s.io
  6.  API server proxies the response to user.

In regular working case, API server receives all API requests and performs authentication and authorization before forwarding the request to other services behind it, and thus requests are only allowed to pass-through after a successful validation from RBAC point of view.

Kubernetes API request flow in Exploitation Scenario (Privilege Escalation Case)

This vulnerability can be exploit by sending API requests that are malformed. Sequence below explains the request flow that results into an attacker exploiting the vulnerability

  1. Attacker makes an API request to exec a command in POD1 but this time with upgrade headers as “Connection: Upgrade”, “Upgrade: WebSocket”  HTTP header in request header.
  2. API Server performs authentication and authorization to ensure user is authenticated and authorized to access POD1
  3. As in working case, API Server creates a connection to Kubelet. When connecting to the kubelet, the API Server uses certificate based authentication. The certificate used to connect to the kubelet grants it ‘cluster-admin’ privileges, which allows full access to the kubelet. API Server forwards the request to Kubelet.
  4. Kubelet receives the request, verifies that the request came from API server, instead of accepting the request to upgrade and returning HTTP 101, it returns error HTTP error (instead of HTTP 101 success) indicating that the exec command API doesn’t support requested upgrade protocol. Kubelet only supports SPDY as the upgrade protocol as of now.
  5. API Server receives the response from Kubelet and without verifying that response was a success or error, assumes that the connection has been upgraded to websocket protocol and switches to proxying the client side connection to kubelet side connection in a pass-through mode.
  6. From this point onward attacker can make any request to kubelet directly over the established connection while completely bypassing the authentication and authorization enforced by the API server. Since API server is operating in pass-through mode, no authentication or authorization is performed at user’s API requests.
  7. Attacker can make a request to POD2 even though attacker is not authorized for POD2. Since kubelet receives this request on a trusted connection, it executes attacker’s request and thus enabling attacker to POD2 (or any POD from now onward). This results in privilege escalation as attacker can access PODs that it does not have privilege for.

Kubernetes API request flow in Exploitation Scenario (Anonymous access Case)

API server acts as an authentication and authorization proxy gateway to not only Kubelet, but also to the several other API extension/aggregation services behind it. Some of these services provides APIs that allows anonymous access and can be used without any authentication.

The attack can be exploited by starting from an anonymous API (no authentication and authorization required) request as described in following sequence –

  1. Attacker makes an anonymous API request to a service behind API server with ”, “Connection: Upgrade”, “Upgrade: WebSocket”  HTTP header in request header.
  2. Since API allows anonymous access, API Server simply forwards the request to corresponding API aggregation/extension service behind it.
  3. As in working case, API Server creates a connection to API aggregation/extension Service. When connecting to the API Service, the API Server uses certificate based authentication. The certificate used to connect to the kubelet grants it ‘cluster-admin’ privileges, which allows full access to the kubelet. API Server forwards the request to Kubelet.
  4. API aggregation/extension  receives the request, verifies that the request came from API server, instead of accepting the request to upgrade and returning HTTP 101, it returns HTTP error code (instead of HTTP 101 success) indicating that the API doesn’t support requested upgrade protocol.
  5. API Server receives the response from API aggregation/extension and without verifying that response was a success or error, assumes that the connection has been upgraded to WebSocket protocol and switches to proxying the client side connection to API aggregation/extension side connection in a pass-through mode.
  6. From this point onward attacker can make any request to API aggregation/extension directly over the established connection while completely bypassing the authentication and authorization enforced by the API server. Since API server is operating in pass-through mode, no authentication or authorization is performed at user’s API requests.

Attacker is now allowed to make request to even those APIs that actually requires authentication and authorization leading to a compromised system.

What makes it more severe is that because these unauthorized requests are made over a bypassed connection, the API server doesn’t log that as part of its audit log or server log and thus making it difficult to know if the attackers are exploiting this attack.

In my next post, I will go over the key underlying design patterns in K8s platform that contributed to the vulnerability. I will also explain how such vulnerability may actually exist in any microservice based distributed application and not just K8s and how it applies to distributed microservices architecture in general. Finally I’ll analyze some of the key takeaways and how vulnerability of this kind can be prevented and detected. Stay tuned!

/by