This post is the final part of a two-part series on analyzing recently found CVE-2018-1002105. In the last post, I covered the overview of Kubernetes (K8s) vulnerability (CVE-2018-1002105). Using an example API request flow, I explained how the vulnerability may lead to privilege escalation or anonymous access to K8s cluster using an example request flow.

In this post, I’ll go over the key underlying design elements in K8s that contributed to the vulnerability. And most importantly, how the vulnerability brings forward the challenges of securing any distributed microservice application. Finally, I’ll cover the key takeaways about how organizations can avoid such kind of vulnerability while adopting microservices architectural pattern for their applications.

Key design elements that contributed to CVE-2018-1002105

API server acts as a centralized AAA gateway

In Kubernetes security architecture, the API server acts as a centralized AAA gateway that provides authentication and authorization for all API requests. Once a request has been authenticated and authorized by the API server, the other services do not perform any further security checks and just rely on the trusted connection from the API server.

This architecture is similar to the traditional perimeter security/firewall based security. Perimeter security based architectures expose the single point of failure in the security infrastructure. If the perimeter is breached, the lack of inside (aka east-west) security checks allows a hacker to penetrate easily.

This vulnerability mainly resulted from the API server acting as the only authentication and authorization enforcement point in K8s platform. By disabling the API server from the request path, an attacker was able to penetrate inside the other services inside the K8s platform.

API server acts as a privilege escalation point

In K8s, API server elevates the privilege of an end user request while proxying it to the other services.

As API Server initiate connections to services behind it, API Server uses certificate-based authentication to authenticate itself to the services. The certificate that is used for authentication grants the requests over connection the ‘cluster-admin’ privileges. Backend services allow the requests over the authenticated connections to perform any operation that is allowed by cluster-admin role.

So while the connection between user and API server only has permissions as allowed by user’s roles, the connection between an API server and backend services has cluster-admin privilege. API server connects these two connections and bridges data from each other in a bi-directional manner. This results into privilege escalation for the requests that are coming from the user as they make their way onto the other side of the connection.

This behavior enabled a privilege escalation attack.

How the vulnerability is about the challenges of securing microservices in general

While K8s is a platform for deployment of microservices, it itself is based on microservices architecture and consists of multiple individual services that jointly work together to provide the full platform functionality.

Distributed microservices requires distributed security

The attack could have been prevented if the other services behind the API server were doing their own security checks. Making the API server as the only enforcement point is the main underlying cause of the vulnerability.

In the microservice architecture, there is very little difference between an external facing service and an internal service from a security point of view. As each service is exposing an API endpoint and is accepting API requests, each service must be responsible for its own security. Delegating the security to a centralized service creates a single point of failure in security infrastructure. This vulnerability underlines that the distributed microservices applications require distributed security checks and enforcement.

Deeper visibility into inter-service interaction is important

The vulnerability became more severe as the unauthorized requests from attackers were made over a bypassed connection, and the API server didn’t log that as part of its audit log or server log. It made it difficult to know if the attackers were exploiting the vulnerability.

This emphasizes that a deeper application layer visibility is needed between the service interactions. Deeper visibility enables insight into the regular behavior of services interactions and allows the detection of abnormal behavior. Thus enabling a deeper defense against breaches. An attack could have been exposed if the interactions between the API server and other services were visible to K8 administrators.

Key Takeaways

As microservices adoption is growing, fueled by their compelling benefits in terms of agility and flexibility, securing the microservices requires organizations to consider a new set of principles while designing their security infrastructure.

As demonstrated by CVE-2018-1002105 securing microservices based applications requires distributed security enforcement. Perimeter security based architecture is insufficient and can expose a single point of failure from a security point of view. Organizations need to adopt a more comprehensive and distributed approach to security while developing a security architecture for their micro-services.

Also, deeper application-layer visibility into the inter-service interactions becomes very critical from the security and operation point of view. Organizations need to ensure that there are no blind spots in their microservice deployments.