Azure Kubernetes Chronicles, part 2: Networking using eBPF

Welcome back to the Azure Kubernetes Chronicles, a blog series dedicated to unraveling the complexities of Kubernetes on Azure. In the first edition, we explored the fundamentals of the Container Network Interface (CNI), its role in Kubernetes networking, and how it powers communication within clusters. Building on that foundation, this next episode delves into the transformative impact of eBPF (extended Berkeley Packet Filter) when integrated with CNI. By combining these technologies, we unlock new possibilities for enhancing networking, security, and observability in Kubernetes environments, particularly on Azure Kubernetes Service (AKS).

In this blog post, we examine how eBPF elevates the capabilities of CNI, empowering Azure Kubernetes Service users to build secure, high-performing, and observable clusters. Let’s dive into the synergy between CNI and eBPF and see how this duo is shaping the future of Kubernetes networking.

· eBPF in Kubernetes environments
· Benefits of eBPF
· Microsoft Azure Kubernetes Service and eBPF
· Conclusion

eBPF in Kubernetes environments

Managing networking, security, and observability at scale can be challenging. eBPF (extended Berkeley Packet Filter) provides a powerful way to solve these challenges by enabling efficient, safe, and customizable kernel-level programmability. With tools like Cilium or Calico, eBPF has become a critical technology for improving performance and security in Kubernetes clusters, including those running on managed services like Azure Kubernetes Service (AKS).

What is eBPF?

eBPF, short for extended Berkeley Packet Filter, is a powerful technology that allows you to run sandboxed programs directly within the Linux kernel, without having to change kernel source code or load custom kernel modules. Originally created for packet filtering, eBPF has evolved into a versatile framework for safely extending the kernel’s capabilities. It can be used to instrument the kernel to gather detailed metrics, observe network traffic, enhance security, and even modify system behaviour on the fly.

The key feature of eBPF is its ability to run custom code within the kernel in a secure and efficient manner. This code is verified before execution to ensure it won’t crash the system or harm kernel stability. The power of eBPF lies in its ability to interact with the kernel, allowing developers to tap into the operating system and get insights that would be otherwise challenging to obtain.

How eBPF Works

eBPF programs are loaded into the kernel from user space, where they can attach to specific events such as system calls, kernel functions, or network packets. Once attached, these programs can observe, measure, or alter how the system behaves. The kernel ensures that the eBPF programs are safe to run through a verification process, which prevents bugs or vulnerabilities from being introduced.

For example, an eBPF program can be used to monitor every file open operation across the system, providing insights into what files are being accessed and by which processes. Or it can attach to network events to analyze traffic patterns in real time without having to use separate tools or network taps.

Benefits of eBPF

Performance and Efficiency

Since eBPF programs run inside the kernel, they eliminate the need for user-space context switches, which can be a source of significant overhead in traditional monitoring and tracing tools. This makes eBPF ideal for scenarios where high performance is required, such as observability and network filtering.

Flexibility

eBPF provides unparalleled flexibility to developers and operators. Instead of relying on static kernel modules or third-party tools, you can write custom programs to collect the data you need or modify system behaviour to suit specific requirements. This flexibility allows for more creative and responsive solutions to complex performance and security problems.

Unified Tooling for Observability, Security, and Networking

eBPF has blurred the lines between different domains of system management. Observability, security, and networking can now all be addressed using the same tooling. Tools like Cilium, bpftrace, and Falco leverage eBPF to provide powerful capabilities, from enforcing network policies to gathering in-depth system metrics.

Real-time Insights

eBPF allows you to get real-time insights into your system — whether it’s monitoring latency, tracking packet drops, or gathering metrics on CPU utilization. Because eBPF programs run directly in the kernel, they have immediate access to the information you need, providing insights as soon as an event occurs.

Comparison between different network options

Key Points for Kubenet

• virtual ethernet (veth) Pair & Linux Bridge: Each pod gets a veth pair connecting its network namespace to a Linux bridge on the host.
• NAT and Routing: Kubenet sets up simple NAT (using iptables) and routing rules so that pod IPs (typically from a separate CIDR) can reach and be reached by the external network.
• Simplicity: It is designed for basic networking requirements, making it easier to configure but less feature-rich than advanced solutions.

Key Points for CNI without eBPF

• veth Pair: Provides connectivity from the container to the host.
• Traditional Processing: Packet handling is done in the kernel via standard components, possibly with extra context switches and userspace involvement.

Key Points for CNI with eBPF

• In-Kernel Processing: eBPF programs attached at hooks like XDP or TC handle packet manipulation directly in the kernel.
• Lower Overhead: Eliminates many of the extra steps in traditional processing, leading to lower latency and higher throughput.
• Dynamic and Programmable: eBPF can be updated or reprogrammed at runtime without service interruptions.

Microsoft Azure Kubernetes Service and eBPF

Azure Kubernetes Service (AKS) integrates eBPF capabilities to enhance observability, networking, and security within Kubernetes environments. Microsoft, through tools like Cilium, enables eBPF-powered networking within AKS, offering improved performance, visibility, and control.

Microsoft offers two options to install a CNI with eBPF support. The first is called BYOCNI. This option gives all the freedom to configure the CNI as you wish. Downside is that it doesn’t fully integrate with the Azure stack and Microsoft is not responsible for supporting and maintaining the implementation. This requires more in depth knowledge of the CNI. To make use of this option first install AKS with the — network-plugin parameter with the parameter value of none. After the AKS installation the nodes will have a not ready status. At this the moment Cilium or Calico or any other CNI can be installed and configured.

The second option is Azure CNI powered by Cilium. With this option, AKS manages the configuration of Cilium. This option can support most use cases however there are some limitations. For instance this option doesn’t support Windows nodes and Cilium L7 policy enforcement is disabled.

Azure CNI Powered by Cilium can be deployed using two different methods for assigning pod IPs:

• Assign IP addresses from an overlay network (similar to Azure CNI Overlay mode)
• Assign IP addresses from a virtual network (similar to existing Azure CNI with Dynamic Pod IP Assignment)

Pro’s and cons of the two scenario’s can be found here. For this blog the virtual network scenario was used.

Before we are able deploy anything on Azure we need create a resource group at the location where we want to deploy the cluster.
az group create — name AKS-blog — location westeurope

Create the Resource Group

az group create --name AKS-blog --location westeurope

Create a Virtual Network and Subnets

az network vnet create \
  --resource-group AKS-blog \
  --location westeurope \
  --name vnet-aks \
  --address-prefixes 10.0.0.0/8 -o none

az network vnet subnet create \
  --resource-group AKS-blog \
  --vnet-name vnet-aks \
  --name nodesubnet \
  --address-prefixes 10.240.0.0/16 -o none

az network vnet subnet create \
  --resource-group AKS-blog \
  --vnet-name vnet-aks \
  --name podsubnet \
  --address-prefixes 10.241.0.0/16 -o none

Deploy the AKS Cluster with Cilium

az aks create \
 --name AKS-blog-cluster \
  --resource-group AKS-blog \
  --location westeurope \
  --max-pods 250 \
  --network-plugin azure \
  --vnet-subnet-id /subscriptions/c9465047-a812-42e7-a53b-739940940898/resourceGroups/AKS-blog/providers/Microsoft.Network/virtualNetworks/vnet-aks/subnets/nodesubnet \
  --pod-subnet-id /subscriptions/c9465047-a812-42e7-a53b-739940940898/resourceGroups/AKS-blog/providers/Microsoft.Network/virtualNetworks/vnet-aks/subnets/podsubnet \
  --network-dataplane cilium \
  --generate-ssh-keys

Verify the Cluster

Get credentials and check the cluster status:

az aks get-credentials --name AKS-blog-cluster --resource-group AKS-blog
kubectl get nodes

Cilium provides a cli which you can use among other things for checking the health status of the Cilium installation within the cluster.

To make the impact of Cilium visible, the demo application Online Boutique is used. First step is cloning the repository containing the scripts.

git clone https://github.com/GoogleCloudPlatform/microservices-demo.git

Next is the deployment of the script.

kubectl apply -f ./release/kubernetes-manifests.yaml

After the installation is completed, check if all pods are running.

By implementing a policy we can restrict traffic between the different parts of the application.

Using Cilium, we can enforce granular network policies to control pod communication.

Restrict Traffic to the Checkout Service

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "allow-frontend-to-checkout"
spec:
  endpointSelector:
    matchLabels:
      app: checkoutservice
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend

Scope of the Policy: The endpointSelector field in the Cilium policy determines which pods the policy applies to. Specifically, this selector matches pods labeled with app: checkoutservice. As a result, all pods with this label are considered the “protected” endpoints. The policy effectively wraps these pods in a security boundary, ensuring that any incoming traffic must comply with the defined rules.

Ingress Rules: The ingress section outlines the conditions under which traffic is permitted to reach the selected pods (checkoutservice). In this case, the policy explicitly allows traffic only from pods that carry the label app: frontend. This condition ensures that only requests originating from these specific pods are authorized to communicate with checkoutservice.

Allowed Traffic Sources: The fromEndpoints key within the policy further narrows down allowed traffic by specifying the exact source pods. It filters based on labels, matching only those pods labeled app: frontend. This tightens access control, ensuring that traffic from any other pod — even if it exists within the same namespace or cluster — is denied.

What Happens After Applying the Policy

Allowed Connections: Pods labeled with app: frontend are permitted to send traffic to checkoutservice without any restrictions. Communication flows smoothly for these pods.

Blocked Connections: All other pods, including those with incorrect labels, no labels, or entirely different labels (e.g., app: analytics or app: user-service), are denied access to checkouts ervice. Their connection attempts are dropped without any response.

Whitelist Model Enforcement: This policy enforces a strict whitelist approach, where only explicitly permitted traffic is allowed, blocking all other connections by default.

To simulate blocked or allowed traffic, a simple pod with utilities like curl or wget to test connectivity can be used.

Deploy a test pod using an image that includes curl, such as curlimages/curl

kubectl run test-curl --image=curlimages/curl:latest --restart=Never -- sleep 3600

This creates a pod with curl preinstalled. In order to test the connection, the ip-adress of the frontend is needed.

kubectl get svc -n default

Exec into the Pod and Test Connectivity

kubectl exec -it test-curl -- curl http://20.61.152.55

As seen in the screenshot the curl command fails to connect. To allow the curl command to work a label is added to the pod. If the curl command is repeated after applying the label, the curl command is allowed as shown in the above image.

Before applying the Policy

• By default, Kubernetes clusters allow unrestricted communication between pods unless a network policy is enforced.

• Any pod, regardless of its role or labels, could freely access checkout service. This permissive behaviour could lead to unintended access, posing security risks.

• Unauthorized pods or potentially malicious actors within the cluster could exploit this unrestricted access to compromise sensitive operations handled by checkout service.

After the applying Policy

Allowed Traffic: Pods labeled with app: frontend can continue communicating with checkoutservice as expected, with no interruptions.

Blocked Traffic: Pods without the correct label (e.g., app: analytics or app: user-service), or pods from entirely different namespaces, will fail to connect to checkoutservice. Connection attempts from these sources will result in dropped packets, meaning they won’t even reach the pod.

Ingress Control: The policy is enforced by Cilium using eBPF (extended Berkeley Packet Filter) at the kernel level. This means that every packet destined for checkoutservice is inspected, and those that fail to meet the criteria (i.e., not originating from app: frontend) are dropped before reaching the pod.

Conclusion

As organizations continue to embrace Kubernetes for their containerized workloads, mastering networking becomes crucial for building scalable, secure, and high-performing clusters. This edition of the Azure Kubernetes Chronicles delves into the essential role of the Container Network Interface (CNI) and the transformative capabilities of eBPF. Together, these technologies empower Kubernetes users to achieve efficient pod-to-pod communication, enforce granular network policies, and enhance cluster observability.

With Azure Kubernetes Service (AKS) integrating tools like Cilium, businesses can harness the power of eBPF to bring advanced networking, security, and performance enhancements to their clusters. Whether deploying a straightforward overlay network or exploring the flexibility of a virtual network setup, understanding these concepts lays the groundwork for effective Kubernetes management.

In this blog, we demonstrated how to deploy an AKS cluster with Cilium, apply a network policy to secure traffic, and use eBPF-powered tools for efficient data flow and protection. These tools not only improve the security posture of your cluster but also simplify complex configurations for better manageability.

Stay tuned for the next edition, where we’ll explore the powerful monitoring and observability capabilities of eBPF, providing deeper insights into cluster operations and performance. With Azure Kubernetes Service and eBPF, the future of Kubernetes networking and observability is not just efficient but transformative.