Build Resilient, Secure Microservices with Microsegmentation

This article originally appeared in The New Stack.

About 10 to 12 years ago, the world of software experienced a shift in the architectural aspects of enterprise applications. Architects and software builders started moving away from the giant, tightly coupled, monolithic applications deployed in the private data centers to a more microservices-oriented architecture hosted in public cloud infrastructure. The inherent distributed nature of microservices is a new security challenge in the public cloud. Over the last decade, despite the growing adoption of microservices-oriented architecture for building scalable, autonomous, and robust enterprise applications, organizations often struggle to protect against this new attack surface in the cloud compared to the traditional data centers. It includes concerns around multitenancy and lack of visibility and control over the infrastructure, as well as the operational environment. This architectural shift makes meeting security goals harder, especially with the paramount emphasis placed on faster container-based deployments.

The purpose of this article is to understand what microsegmentation is and how it can empower software architects, DevOps engineers, and IT security architects to build secure and resilient microservices. Specifically, I’ll discuss the network security challenges associated with the popular container orchestration mechanism Kubernetes, and I will illustrate the value of microsegmentation to prevent lateral movement when a breach takes place.

Security challenges with microservices deployments

Microservice-oriented architecture requires breaking applications into smaller, loosely-coupled services. Due to the distributed nature of these services, they often end up having their own datastores. Each of these services is deployed independently using a container, which offers a mechanism for an application to package everything from object code, third-party libraries, operating systems, tools, and other dependencies. Once all necessities are packaged, containers provide the execution environment to run them smoothly. These containers are managed using orchestrators like Kubernetes.

According to the popular notion, the container ecosystem is designed to enforce security through isolation. However, isolation among containers doesn’t necessarily provide the required security boundaries. Kubernetes offers several security features like authentication, authorization, network policy, and pod security standard, etc. But unfortunately, neither containers nor Kubernetes offers security by default. Kubernetes deployments prioritize functionality over security in their default configuration. Therefore, a developer or a reliability engineer who is responsible for creating, moving, and destroying these containers may not always know how to ensure the security of deployments. Let’s take a closer look at some of the important aspects of Kubernetes from networking and security perspective:

1. Namespace: In Kubernetes, a namespace is a logical way to divide cluster resources into virtual space among multiple users. Unlike Linux, it does not enforce any network segmentation. Please note that a namespace is not a security boundary, and therefore, services in one namespace can access services in another namespace.
2. Network policy: The Kubernetes network policy is an application-specific construct that allows layer 3 and layer 4 communication among namespaces, pods, and IP address blocks. When Kubernetes is installed, the network policy is not available by default. One needs to install and configure network plugins to enforce network policies before pods are created. Also, it’s worth noting that network policies cannot be enforced at the service level. This is a significant shortcoming from a security perspective, since access control can’t be extended for services.
3. Pod security standard: In a cluster, pods support multiple containers that share the same physical or virtual machine. They enable data sharing and communication among the containers within the pod. A pod security standard allows an administrator to control resources and their permissions using a fine-grained authorization model. This security control requires an admission controller, which is not enabled by default. A privileged pod offers administrative access to all containers. As a result, the attacker who obtains access to privileged containers can get administrative access to host resources.
4. Secret storage: Kubernetes stores sensitive and confidential information such as passwords, OAuth tokens, and SSH keys as secrets in base 64 encoded form, which is considered as plaintext. The authorization policy, which is used to restrict access to secrets, is not configured by default. These are shared via environmental variables or manifest files, which are also considered to be an insecure practice for handling secrets. In the absence of default encryption of secrets at rest and the lack of robust secret management, secrets become an attractive target for lateral movement.

Lateral movement and the malicious insider

The distributed nature of microservices makes connectivity supremely important. Specifically, containers and pods must be able to communicate with each other across namespaces. However, the default network policies at the cluster layer do not necessarily implement the principle of least privilege out of the box. Therefore, no matter how you secure your code, unauthorized implicit access to shared resources cannot be prohibited. As a result, the application may become susceptible to lateral movements within the cluster. Microsegmentation allows a means to implement granular access control, creating secure zones around each microservice at the container, pod, and cluster level, thereby significantly reducing the blast radius and increasing resiliency to recover from a successful attack.

The second prominent threat that the security model of the existing Kubernetes framework presents with respect to microservices is a malicious insider. From an attacker’s perspective when security is not available by default, several misconfigurations are possible. In the absence of strong access control or authorization policies, attacks like Server-Side Request Forgery (SSRF) allow the attacker to leverage an authenticated access to a pod to gain unauthorized access to resources within another pod.

As discussed above, we saw that adequate security controls are not available to protect secrets by default. Not encrypting and rotating secrets, as well as not restricting access to secrets, are indeed unrelated problems, but in the absence of one security control (in this case, lack of encryption), the second security control (i.e., restricting access to the secret stores) becomes crucial to avoid exploitation by a malicious insider (for example, a disgruntled administrator).

Introducing cyber resilience with microsegmentation

In the context of security, resiliency is the ability to quickly recover from a simple failure due to attacks or catastrophic events like breaches. The first step in building resilient microservices is to understand what it is that we would like to protect from the attacker, and if the asset or resource does get compromised, how we can limit the impact. Microsegmentation helps us determine the resources holding essential data or critical fault-tolerant systems, and it also provides a mechanism to lock down the access on the basis of least privilege. In this way, even if one critical resource is compromised, the others don’t get impacted.

Today’s modern software development lifecycle puts special emphasis on delivering features fast. These features are deployed in production via containers even faster, mostly by the same developers or DevOps. If we consider all types of vulnerabilities that the code of microservices, third-party libraries, and other dependencies, container, or cluster are susceptible to, how can we possibly identify and prevent all these attacks? The answer is to block access to critical resources and allow only on a “need-to-know basis” using microsegmentation.

Get started with microsegmentation

There is no one-size-fits-all strategy when getting started with microsegmentation, but the following are some best practices as you launch a microsegmentation project for microservices:

1. Knowing the microservice design patterns and using a microsegmentation template

Knowing the design patterns facilitates a deeper understanding of architectural components, data flow, and assets holding critical data. Based on these frequently used design patterns, one can create corresponding microsegmentation templates. Once these templates are vetted by the security and networking team, they will be easy to reuse and scale at a faster rate.

Unlike the giant, tightly-coupled monolith applications, the applications based on microservices-oriented architecture follow one or more certain design patterns. These design patterns can be one of the following:

• Decomposition patterns involve breaking down applications into subdomains or transactions based on business capabilities.
• Integration patterns involve service integration design patterns such as API gateway, aggregator, chained microservices, etc.
• Database patterns focus on the position of databases in the overall architecture of the applications. Examples include database per service, shared database, event sourcing, etc.
• Observability patterns consist of design patterns that focus on auditing, logging, and monitoring such as log aggregation, distributed tracing, health check, etc.
• Cross-cutting concerns patterns assist in managing application-level functionality such as security, fault tolerance, service-to-service communication, and configuration management in a centralized location. Examples include circuit breaker, service discovery, etc.

(For more information on various microservice design patterns please look here.)

2. Selecting the right microsegmentation approach

Currently, there are several microsegmentation vendors with varying solution approaches available in the market. Broadly speaking, microsegmentation techniques fall into two categories:

1. Technology-agnostic microsegmentation solutions are either based on agents or next-generation firewalls.
2. Technology-dependent microsegmentation solutions are based on the hypervisor or network fabrics.

Microservices use a varying technology stack, and the faster deployment demands an ability to scale quickly. Therefore, a technology-agnostic microsegmentation solution — but one specifically tailored for a container ecosystem capable of segmenting clusters, pods, containers, and services — would be an optimal choice.

3. Network visibility

The ability to visualize segmented applications and traffic flow is an integral part of microsegmentation solutions. Visibility offers a one-stop view of the entire network to the administrator, security architects, and Chief Security Officer (CSO).

4. Easy-to-use centralized control

Drafting a policy for each application can be a daunting task initially. It requires a series of conversations with the IT, network, and security architects. Having a centralized control to facilitate policy creation and enforcement helps speed up the process. Leveraging the segmentation templates, which are customized per the microservice design pattern, can also help.

5. Performance should not be the bottleneck

Applying microsegmentation policies requires analyzing traffic and deploying policies effectively in real-time. The attacker can leverage performance delays to manifest the attack further. Therefore, it is extremely important to test the performance, especially for latency-sensitive microservices.

Conclusion

In this blog post, I discussed some of the security concerns presented by container and Kubernetes-based deployment mechanisms utilized by microservices. The word “resiliency” starts to go out when we deploy the microservices in production continuously throughout the CI/CD lifecycle at a faster rate without much consideration around limiting access to critical resources. Microsegmentation offers a strong access control mechanism that minimizes the impact of breaches. It also fosters resiliency in the enterprise applications by implementing secure microzones.

Microsegmentation, when planned and performed correctly, surely creates a robust barrier against lateral movements in the microservices world. It’s hard to secure microservices while moving fast, but it’s important to stop and think about implementing security and resiliency at scale and proactively plan it with all stakeholders.