Kubernetes architecture and high availability cluster overview

Need of Zero Downtime Deployments in Kubernetes

Zero downtime deployment is all about delivering updates to your applications without interrupting service. This means users can continue to interact with your application seamlessly, while you roll out new features, fix bugs, or make other changes. Kubernetes supports this by managing the lifecycle of your containers and orchestrating smooth transitions between different versions of your application. Zero downtime deployments are essential because users expect continuous access to applications without interruptions. Any downtime, even brief, can lead to user frustration, lost revenue, and harm to your brand. Implementing zero downtime strategies allows you to update and maintain your application without affecting its availability, ensuring a smooth and uninterrupted user experience.

To proceed with zero downtime deployment for our applications, we need to setup up a high availability cluster to ensure robust operations.

Cluster Setup

There are two main approaches to setting up a highly available cluster:

  • With stacked control plane nodes: A stacked HA cluster is a topology where the distributed data storage cluster provided by etcd is stacked on top of the cluster formed by the nodes managed by kubeadm that run control plane components. This approach requires less infrastructure. The etcd members and control plane nodes are co-located.

Kubernetes stacked etcd high availability cluster topology

  • With an external etcd cluster: An HA cluster with external etcd is a topology where the distributed data storage cluster provided by etcd is external to the cluster formed by the nodes that run control plane components. This approach requires more infrastructure. The control plane nodes and etcd members are separated.

Kubernetes external etcd high availability cluster topology

Here, we’ll be using the stacked etcd cluster topology.

A high availability cluster consists of multiple control plane nodes (master nodes), multiple worker nodes and a load balancer. This setup makes the system more robust since any node can fail without the application going offline or data being lost. It also makes it easy to add more compute power or replace old nodes without any downtime. An illustration of this setup is shown below. The ETCD cluster makes sure that all data is synced across the master nodes and the load balancer regulates the traffic distribution. The cluster can therefore be accessed through one single entry point (the load balancer and the request will be passed to an arbitrary node.

Let’s proceed with the cluster setup!

Number of Control Plane Nodes Minimum Number of Healthy Nodes for Quorum
1 1
2 2
3 2
4 3
5 3
6 4
7 4
8 5
9 5
To ensure high availability according to quorum requirements, it’s essential to have at least two control plane nodes in your Kubernetes cluster. This configuration helps maintain the cluster’s availability and stability, as a quorum is required for the control plane to make decisions, and having multiple nodes ensures that the cluster can continue functioning even if one node fails.
Hosts IP Addresses
master-1 192.168.1.52
master-2 192.168.1.50
master-3 192.168.1.77
worker-1 192.168.1.80
worker-2 192.168.1.58
load-balancer 192.168.1.72
Add the above addresses in your /etc/hosts file and setup the hostname on each nodes.
  1. Add entries to the /etc/hosts on each node.
master-1 192.168.1.52
master-2 192.168.1.50
master-3 192.168.1.77
worker-1 192.168.1.80
worker-2 192.168.1.58
load-balancer 192.168.1.72
  1. Set the hostname for each node using the following command:
hostnamectl set-hostname $(NODE_HOSTNAME)

Nginx Installation and Configuration of Load Balancer

Nginx is used as a load balancer in a Kubernetes cluster to distribute incoming network traffic across multiple backend servers or services. Load balancing is efficient in distributing incoming network traffic across a group of backend servers. A load balancer is a device that distributes network or application traffic across a cluster of servers. The load balancer has a big role to achieve high availability and performance increase of cluster.

Installation
  1. Install Nginx:
sudo dnf install nginx
  1. Install the Nginx stream module:
sudo dnf install nginx-mod-stream
Configuration

Edit the Nginx configuration file:

sudo vi /etc/nginx/nginx.conf

Add the following configuration:

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

stream {
    log_format main '$remote_addr [$time_local] "$protocol" '
                    '$status $bytes_sent $bytes_received '
                    '"$session_time"';

    access_log /var/log/nginx/access.log main;

    # Kubernetes API Server Load Balancer
    upstream kubernetes_backend {
        server 192.168.1.77:6443 max_fails=3 fail_timeout=30s;
        server 192.168.1.50:6443 max_fails=3 fail_timeout=30s;
        server 192.168.1.52:6443 max_fails=3 fail_timeout=30s;
        }

    server {
        listen 192.168.1.72:6443;
        proxy_pass kubernetes_backend;
        proxy_timeout 10s;
        proxy_connect_timeout 10s;
        }
}

This configuration sets up an Nginx stream module to load balance the Kubernetes API server across multiple nodes.

Verification
  1. Test the Nginx configuration:
sudo nginx -t
  1. Restart the Nginx service:
systemctl restart nginx.service
  1. Enable the Nginx service:
systemctl enable --now nginx.service
  1. Check the Nginx service:
systemctl status nginx.service

System Nodes Setup

  1. Update the System
sudo dnf update --refresh
  1. Disable Firewalld and SELinux
sudo systemctl disable --now firewalld.service

sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
  1. Remove this packages to disable swap on RHEL machines:
sudo dnf remove zram-generator-defaults

Note: For Ubuntu based machines apply these commands.

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
  1. Reboot the system.
sudo reboot

Add and load kernel parameters.

  1. Persist the required kernel modules so they load on every boot.
sudo tee /etc/modules-load.d/k8s.conf <<EOF
overlay
br_netfilter
EOF

Note: On Kubernetes 1.27+ with containerd ≥ 1.7, overlay is loaded automatically by containerd. br_netfilter is still required for kube-proxy bridge networking. The modprobe commands below apply them immediately for the current session without requiring a reboot.

sudo modprobe overlay
sudo modprobe br_netfilter
  1. Set the required kernel parameters for Kubernetes.
sudo tee /etc/sysctl.d/k8s.conf <<EOT
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOT
  1. Reload sysctl to apply the changes.
sudo sysctl --system

Install Containerd Runtime

We are using the containerd container runtime for our Kubernetes cluster.

  1. Install the dnf-plugins-core package to manage DNF repositories:
sudo dnf install -y dnf-plugins-core
  1. Add the docker-ce repository (which provides official containerd packages):
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  1. Install containerd:
sudo dnf install -y containerd.io
  1. Configure containerd to use the systemd cgroup driver.

Note: containerd 2.0+ enables SystemdCgroup = true by default. For containerd 1.x, generate the default config and enable systemd cgroups:

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
  1. Restart and enable the containerd service:
sudo systemctl restart containerd
sudo systemctl enable --now containerd

Install Kubeadm and other components

  1. Add the official Kubernetes package repository. The exclude parameter ensures Kubernetes packages are not accidentally upgraded via routine yum update operations (as Kubernetes version upgrades require specific orchestration steps). This repository targets Kubernetes 1.36 (latest release: v1.36.2). To target a different minor version, replace v1.36 in both URLs.
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.36/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.36/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF
  1. Install kubelet, kubeadm and kubectl.
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
  1. Enable the kubelet service before running kubeadm.
sudo systemctl enable --now kubelet

Note: The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do.

Initialize Kubeadm cluster

Run the following Kubeadm command on the master node only.

kubeadm init --control-plane-endpoint=load-balancer --pod-network-cidr=192.168.0.0/16 --upload-certs

The kubeadm init command sets up a Kubernetes control plane node, specifying a load balancer for high availability, defining the pod network CIDR with --pod-network-cidr, and uploading certificates for multi-master clusters with --upload-certs. This configuration ensures a resilient and properly networked Kubernetes control plane.

Your output should look like this:

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join load-balancer:6443 --token cnslau.kd5fjt96jeuzymzb \
--discovery-token-ca-cert-hash sha256:871ab3f050bc9790c977daee9e44cf52e15ee37ab9834567333b939458a5bfb5 \
--control-plane --certificate-key 824d9a0e173a810416b4bca7038fb33b616108c17abcbc5eaef8651f11e3d146

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use 
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join load-balancer:6443 --token cnslau.kd5fjt96jeuzymzb \
--discovery-token-ca-cert-hash sha256:871ab3f050bc9790c977daee9e44cf52e15ee37ab9834567333b939458a5bfb5

Setup kubeconfig using:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Note: Paths may differ check accordingly.

You can now use the below command to add another node to the control plane - Setup new control plane (master) using:

kubeadm join load-balancer:6443 --token cnslau.kd5fjt96jeuzymzb \
--discovery-token-ca-cert-hash sha256:871ab3f050bc9790c977daee9e44cf52e15ee37ab9834567333b939458a5bfb5 \
--control-plane --certificate-key 824d9a0e173a810416b4bca7038fb33b616108c17abcbc5eaef8651f11e3d146

Note: The command changes according to your setup and hosts. Please input the command from your output.

Your output should look like this:

 This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

Now that we have initialized both the masters - we can now work on bootstrapping the worker nodes.

Join worker node using:

kubeadm join load-balancer:6443 --token cnslau.kd5fjt96jeuzymzb \
--discovery-token-ca-cert-hash sha256:871ab3f050bc9790c977daee9e44cf52e15ee37ab9834567333b939458a5bfb5

Note: The command changes according to your setup and hosts. Please input the command from your output.

Your output should look like this:

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Now that your cluster is setup you can check the nodes on your HA cluster:

kubectl get nodes

Your output should look like this:

NAME       STATUS     ROLES           AGE   VERSION
master-1   NotReady   control-plane   22m   v1.36.2
master-2   NotReady   control-plane   17m   v1.36.2
master-3   NotReady   control-plane   15m   v1.36.2
worker-1   NotReady   <none>          10m   v1.36.2
worker-2   NotReady   <none>          10m   v1.36.2

Notice that your nodes are initially in NotReady state. You can check the condition reasons using:

kubectl describe nodes

During initial cluster bootstrap, nodes remain NotReady until a Container Network Interface (CNI) plugin is installed. We’ll install Calico:

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.1/manifests/calico.yaml

After the Calico pods start running, your nodes will transition to Ready:

NAME       STATUS   ROLES           AGE   VERSION
master-1   Ready    control-plane   22m   v1.36.2
master-2   Ready    control-plane   17m   v1.36.2
master-3   Ready    control-plane   15m   v1.36.2
worker-1   Ready    <none>          10m   v1.36.2
worker-2   Ready    <none>          10m   v1.36.2

Essential Cluster Services

Helm Installation

Install Helm by running the following command:

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
NFS Provisioner

When a pod needs persistent storage, it defines a PersistentVolumeClaim, optionally also specifying a StorageClass. Storage classes typically correspond to different throughput, latency, replication characteristics and price of the underlying storage. The PersistentVolumeClaim is then picked up by a dynamic provisioner, which creates a PersistentVolume for it. The way it happens and the way this volume is mounted to the requesting pod is a pure implementation detail. Kubernetes hides this under an abstraction called Container Storage Interface (CSI).

Our implementation of persistent storage will be provided by nfs-subdir-external-provisioner.

Setting up NFS on the host machine
  1. In the toplevel project directory on the host machine, create a directory to be exported via NFS:
mkdir nfs
  1. Now, let’s append an entry to /etc/exports file to export this directory:
echo "/nfs 192.168.0.0/16(rw)" > /etc/exports
  1. Set ownership:
sudo chown nobody:nobody /nfs
sudo chmod 775 /nfs
  1. Export it:
sudo exportfs -r
Installing the dynamic provisioner

Install the dynamic provisioner using helm:

helm repo add nfs-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
helm install -n kube-system nfs-provisioner nfs-provisioner/nfs-subdir-external-provisioner \
  --set nfs.server=$(NFS_HOST_MACHINE_IP) \
  --set nfs.path=/nfs \
  --set storageClass.defaultClass=true

To test if it worked, let’s check StorageClass definitions with kubectl get storageclass. The output should be:

NAME                   PROVISIONER                                                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client (default)   cluster.local/nfs-provisioner-nfs-subdir-external-provisioner   Delete          Immediate           true                   14s
MetalLB

Kubernetes Services can be exposed to the external world using load balancers. In addition to cluster-internal IP, a Service of type LoadBalancer gets assigned an external IP. In a real cloud environment, traffic to this external IP is handled by a load balancer, before being forwarded to Kubernetes cluster. Exactly how LoadBalancer-type services are synchronized with the load balancer is an implementation detail specific to a particular cloud platform and its integration with Kubernetes.

Let’s install metallb:

helm repo add metallb https://metallb.github.io/metallb
helm install -n kube-system metallb metallb/metallb --wait --timeout 5m

Now we need to configure it with some additional Kubernetes resources:

cat <<EOF | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: metallb
spec:
  addresses:
  - 192.168.1.150-192.168.1.200
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: pool
  namespace: metallb
spec:
  ipAddressPools:
  - first-pool
EOF

Note the allocated range for Service external IPs (192.168.1.150-200). It’s important for this range to be within the local network of the VMs, but outside the DHCP-assignable range.

And this way the service has become available from outside the cluster!

Application

To demonstrate zero downtime for our application, we will be creating a multi-tier web application using Redis and PHP. First, we’ll set up the Redis leader, which will handle all write operations. Then, we’ll configure two Redis followers to replicate data from the leader, ensuring high availability and data redundancy. Next, we’ll deploy the web frontend, which will interact with the Redis instances to deliver dynamic content. Once the setup is complete, we’ll visit the website to verify its functionality. To further showcase scalability and resilience, we will scale up the web frontend, adding more instances to handle increased traffic without disrupting the service.

PHP Guestbook with Redis

Set up the PVC

We’ll setup a PVC using the NFS storageclass we created earlier to attach to the application to make the data persistent.

  1. Create a namespace to house all the application resources.
kubectl create namespace guestbook
  1. Check the storageclass
kubectl get storageclass

The output should look similar to this: Kubernetes StorageClass status in guestbook cluster

  1. Create a manifest for Redis PVC.
>pvc.yaml
vi pvc.yaml

Paste the following configuration in the pvc.yaml file.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-leader-pvc
  namespace: guestbook
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs-client
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-follower-pvc
  namespace: guestbook
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs-client
  1. Check the PVC in the namespace.
kubectl get pvc -n guestbook

The output should look like this: Kubernetes PersistentVolumeClaim bound status for Redis leader and followers

Set up the Redis Leader

The application uses Redis to store its data. The application writes its data to a Redis leader instance and reads data from multiple Redis follower instances.

  1. Create the following manifest for the Redis leader.
>redis-leader.yaml
vi redis-leader.yaml
  1. Paste the following configuration in redis-leader.yaml.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-leader
  namespace: guestbook
  labels:
    app: redis
    role: leader
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
        role: leader
        tier: backend
    spec:
      containers:
      - name: leader
        image: "docker.io/redis:6.0.5"
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379
        volumeMounts:
        - name: redis-leader-storage
          mountPath: /data
      volumes:
      - name: redis-leader-storage
        persistentVolumeClaim:
          claimName: redis-leader-pvc
  1. Apply the manifest to your cluster:
kubectl apply -f redis-leader.yaml
  1. Verify that the Redis leader Pod is running:
kubectl get pods -n guestbook

The output is similar to the following:

NAME                           READY     STATUS    RESTARTS   AGE
redis-leader-343230949-qfvrq   1/1       Running   0          43s
Create the Redis leader Service

The web application needs to communicate with the Redis leader to write its data. We’ll create a Service to proxy the traffic to the Redis leader Pod.

  1. Create the following manifest for the Redis leader service.
>redis-leader-svc.yaml
vi redis-leader-svc.yaml
  1. Paste the following configuration in the yaml file.
apiVersion: v1
kind: Service
metadata:
  name: redis-leader
  namespace: guestbook
  labels:
    app: redis
    role: leader
    tier: backend
spec:
  ports:
  - port: 6379
    targetPort: 6379
  selector:
    app: redis
    role: leader
    tier: backend
  1. Apply the manifest to your cluster:
kubectl apply -f redis-leader-svc.yaml
  1. Verify the service:
kubectl get service -n guestbook

The output is similar to the following:

NAME           CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
redis-leader   10.51.242.233   <none>        6379/TCP   12s
Set up Redis followers

Although the Redis leader is a single Pod, you can make it highly available and meet traffic demands by adding a few Redis followers, or replicas.

  1. Create the following manifest for the Redis followers.
>redis-followers.yaml
vi redis-followers.yaml
  1. Paste the following configuration in redis-followers.yaml.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-follower
  namespace: guestbook
  labels:
    app: redis
    role: follower
    tier: backend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
        role: follower
        tier: backend
    spec:
      containers:
      - name: follower
        image: us-docker.pkg.dev/google-samples/containers/gke/gb-redis-follower:v2
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379
        volumeMounts:
        - name: redis-follower-storage
          mountPath: /data
      volumes:
      - name: redis-follower-storage
        persistentVolumeClaim:
          claimName: redis-follower-pvc
  1. Apply the manifest to your cluster:
kubectl apply -f redis-followers.yaml
  1. Verify that the three Redis follower replicas are running:
kubectl get pods -n guestbook

The output is similar to the following:

NAME                              READY   STATUS    RESTARTS   AGE
redis-follower-76588f55b7-bnsq6   1/1     Running   0          27s
redis-follower-76588f55b7-qvtws   1/1     Running   0          27s
redis-follower-76588f55b7-sd6nk   1/1     Running   0          27s
redis-leader-dd446dc55-kl7nl      1/1     Running   0          119s
Create the Redis follower Service

The web application must communicate with the Redis followers to read data. To make the Redis followers discoverable, you must set up a Service.

  1. Create the following manifest for the Redis follower service.
>redis-followers-svc.yaml
vi redis-followers-svc.yaml
  1. Paste the following configuration in the redis-followers-svc.yaml file.
apiVersion: v1
kind: Service
metadata:
  name: redis-follower
  namespace: guestbook
  labels:
    app: redis
    role: follower
    tier: backend
spec:
  ports:
  - port: 6379
  selector:
    app: redis
    role: follower
    tier: backend
  1. Apply the manifest to the cluster:
kubectl apply -f redis-follower-svc.yaml
  1. Verify the service:
kubectl get service -n guestbook

The output is similar to the following:

NAME           CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
redis-leader   10.51.242.233   <none>        6379/TCP   49s
redis-follower 10.51.247.238   <none>        6379/TCP   3s

The leader and followers form a distributed caching system in Kubernetes. The single leader pod, managed by a Deployment, handles all write operations and serves as the primary data source. Three follower pods, also controlled by a separate Deployment, continuously replicate data from the leader and handle read operations, improving read performance and scalability. If the leader pod is deleted, Kubernetes automatically creates a new one. During this short period, read operations can still be served by the followers. Once the new leader pod is up, followers attempt to reconnect and resynchronize, restoring the system to full functionality.

Set up the application web frontend

The web application uses a PHP frontend, which is configured to communicate with either the Redis follower or leader Services, depending on whether the request is a read or a write. The frontend exposes a JSON interface, and serves a jQuery Ajax-based UI.

  1. Create the following manifest for the guestbook web application frontend.
>guestbook.yaml
vi guestbook.yaml
  1. Paste the following configuration.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: guestbook
spec:
  replicas: 3
  selector:
    matchLabels:
        app: guestbook
        tier: frontend
  template:
    metadata:
      labels:
        app: guestbook
        tier: frontend
    spec:
      containers:
      - name: php-redis
        image: us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5
        env:
        - name: GET_HOSTS_FROM
          value: "dns"
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 80
  1. Apply the manifest to the cluster:
kubectl apply -f guestbook.yaml
  1. Verify that the replicas are running:
kubectl get pods -l app=guestbook -l tier=frontend -n guestbook

The output is similar to the following:

NAME                        READY   STATUS    RESTARTS   AGE
frontend-7b78458576-8kp8s   1/1     Running   0          37s
frontend-7b78458576-gg86q   1/1     Running   0          37s
frontend-7b78458576-hz87g   1/1     Running   0          37s
Expose the frontend on an external IP address

With the current configuration, the redis-follower and redis-leader Services that you created in the previous steps are only accessible within the cluster because the default type for a Service is ClusterIP. To make the web frontend Service externally accessible, you can specify type: LoadBalancer or type: NodePort in the Service configuration depending on your requirements.

  1. Create a manifest for exposing the guestbook frontend on an external IP with LoadBalancer service with help of MetalLB we setup earlier.
>guestbook-svc.yaml
vi guestbook-svc.yaml
  1. Paste the following configuration in the guestbook-svc.yamlfile.
apiVersion: v1
kind: Service
metadata:
  name: frontend
  namespace: guestbook
  labels:
    app: guestbook
    tier: frontend
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: guestbook
    tier: frontend
  1. Apply the frontend service manifest to the cluster.
kubectl apply -f guestbook-svc.yaml
  1. Verify the service:
kubectl get service -n guestbook

The output is similar to the following:

NAME             TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)       
frontend         LoadBalancer   10.97.201.216    192.168.1.150   80:30968/TCP   
redis-follower   ClusterIP      10.109.3.15      <none>          6379/TCP       
redis-leader     ClusterIP      10.105.152.164   <none>          6379/TCP
Visit the application website

To access the application website, get the external IP address of the frontend Service:

Copy the IP address and open the page in your browser:

http://192.168.1.150

Try adding some entries by typing in a message, and clicking Submit. The message you typed appears in the frontend. This message indicates that data is successfully added to Redis through the Services that you created and stored in the persistent storage on the NFS server.

PHP Guestbook web application frontend running in browser

Now that our application is up and running, we’ll make it consistent as it will achieve zero downtime with various factors.

Container Location

When creating pods we use images to spin containers and usually these images are stored on various different registries. Although often times these public registries and repositories get updated and and these images get deleted or pulled off the registry making it unavailable to pull them from the source. So when pulling an image from a registry, several issues may arise. The registry could become unavailable, leading to an inability to pull the image, which may result in an ImagePullBackOff error in Kubernetes. Additionally, if the image tag being used has been removed, the same error would occur. Another potential problem is that even if the image tag remains unchanged, the content of the image may differ if the image is not immutable, causing inconsistent behavior across nodes in a cluster due to differing image hashes. Furthermore, pulling images from external registries may not align with security compliance requirements, such as SOC2 or HIPAA, which often necessitate maintaining control over the images.

To maintain the images according to our own accord there are several solutions:

  1. Local Storage The images which need to be pulled can be stored in a local directory on our own servers making it immutable and within the network. This also ensures that the images will be pulled at a faster transfer rate. Although doing so will take up storage as well as it won’t be available on a public network unless it is set on a NFS/ZFS server with public IP.

  2. Using own repositories on public registries Using our own repositories on public registries allows us to maintain control over the images while making them accessible globally. This ensures that the images are consistently available across different environments and can be managed with proper versioning and tagging. So you can make your own Dockerfiles to build the images and push them to your repositories. Additionally, using public registries can leverage the scalability and reliability of cloud services, reducing the burden on local infrastructure. However, this approach may expose the images to security risks if not properly managed, and there could be concerns about compliance with security standards. Furthermore, reliance on public networks could lead to slower transfer rates compared to local storage, especially in environments with limited bandwidth.

Here, we’ll be using Docker Hub registry to push our images. We’ll push our own images to docker hub by first pulling and tagging the respective images.

Redis leader image
  1. Pull the existing compatible image.
docker pull docker.io/redis:6.0.5
  1. Check the pulled image.
docker images
  1. Now tag the image as per your docker repository and username.
docker tag docker.io/redis:6.0.5 vinzyzk/redis:6.0.5
  1. Push the docker image to your repository.
docker push vinzyzk/redis:6.0.5

Note: You should be logged in to your docker account using docker login.

Now your image should be available on Docker Hub.

Docker Hub repository displaying pushed Redis leader container image

Now change your manifest to replace the image.

image: "vinzyzk/redis:6.0.5"
imagePullPolicy: Always

Apply the manifest again to pull the new image.

kubectl apply -f redis-leader.yaml
Redis followers image
  1. Pull the existing compatible image.
docker pull us-docker.pkg.dev/google-samples/containers/gke/gb-redis-follower:v2
  1. Check the pulled image.
docker images
  1. Now tag the image as per your docker repository and username.
docker tag us-docker.pkg.dev/google-samples/containers/gke/gb-redis-follower:v2 vinzyzk/redis-follower:v2
  1. Push the docker image to your repository.
docker push vinzyzk/redis-follower:v2

Note: You should be logged in to your docker account using docker login.

Now your image should be available on Docker Hub.

Docker Hub repository displaying pushed Redis follower container image

Now change your manifest to replace the image.

image: vinzyzk/redis-follower:v2
imagePullPolicy: Always

Apply the manifest again to pull the new image.

kubectl apply -f redis-followers.yaml
Guestbook frontend image
  1. Pull the existing compatible image.
docker pull us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5
  1. Check the pulled image.
docker images
  1. Now tag the image as per your docker repository and username.
docker tag us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5 vinzyzk/guestbook-frontend:v5
  1. Push the docker image to your repository.
docker push vinzyzk/guestbook-frontend:v5

Note: You should be logged in to your docker account using docker login.

Now your image should be available on Docker Hub.

Docker Hub repository displaying pushed Guestbook frontend container image

Now change your manifest to replace the image.

image: vinzyzk/guestbook-frontend:v5
imagePullPolicy: Always

Apply the manifest again to pull the new image.

kubectl apply -f guestbook.yaml

Deployment / Replicaset

A ReplicaSet ensures that a specified number of pod replicas are running at any given time. However, Deployment is a higher-level concept that manages ReplicaSets and provides declarative updates to Pods along with a lot of other useful features. Therefore, it is recommend using Deployments instead of directly using ReplicaSets, unless you require custom update orchestration or don’t require updates at all. Deployment resource makes it easier for updating your pods to a newer version. Lets say you use ReplicaSet-A for controlling your pods, then You wish to update your pods to a newer version, now you should create Replicaset-B, scale down ReplicaSet-A and scale up ReplicaSet-B by one step repeatedly (This process is known as rolling update). Although this does the job, but it’s not a good practice and it’s better to let K8S do the job. A Deployment resource does this automatically without any human interaction and increases the abstraction by one level.

Note: Deployment requires Replicaset because it cannot directly interact with pods. It requires Replicaset for Rolling Updates.

So it best to choose Deployments over Replicaset but deployments require replicaset for rolling pods, so we’ll use Deployments for this application as well.

To specify the number of replicas, configure the manifests as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: guestbook
spec:
  replicas: 3
  ...

Pod Disruption Budget (PDB)

A PDB specifies the number of replicas that an application can tolerate having, relative to how many it is intended to have. For example, a Deployment which has a .spec.replicas: 5 is supposed to have 5 pods at any given time. If its PDB allows for there to be 4 at a time, then the Eviction API will allow voluntary disruption of one (but not two) pods at a time. The group of pods that comprise the application is specified using a label selector, the same as the one used by the application’s controller (deployment, stateful-set, etc).

  • PDB prevents you from scaling down your app against voluntary disruptions only, like cluster upgrades where you drain a node.

whereas…

  • minAvailable define how many pods should remain up on rollout.

  • maxSurge define how many pods should be added on rollout.

Let’s apply Pod Disruption Budget to our application deployments.

For Redis follower
  1. Apply the following PDB manifest to the Redis follower deployment.
>redis-follower-pdb.yaml
vi redis-follower-pdb.yaml
  1. Configure the yaml file as such.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: redis-follower-pdb
  namespace: guestbook
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: redis
      role: follower
  1. Apply the yaml file.
kubectl apply -f redis-follower-pdb.yaml
  1. Verify the PDB.
kubectl get pdb -n guestbook

The output should look like this:

NAME                 MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
redis-follower-pdb   2               N/A               1                     2d1h
Now let’s test the PDB!

Since our PDB specifies a minimum of 2 pods, draining the node should not result in its deletion. Kubernetes should prevent the node from being deleted because it needs at least one worker node to maintain the required number of pods. Let’s proceed to observe this behavior.

  1. Check the PDB.
kubectl describe pdb redis-follower-pdb -n guestbook

The output is as followed:

Name:           redis-follower-pdb
Namespace:      guestbook
Min available:  2
Selector:       app=redis,role=follower
Status:
    Allowed disruptions:  1
    Current:              3
    Desired:              2
    Total:                3
Events:                   <none>
  1. Drain the node on which the pods are running.
kubectl drain worker-2 --ignore-daemonsets

Draining the node gives the following output:

error when evicting pods/"redis-follower-55cf45659-jwfv4" -n "guestbook" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"redis-follower-55cf45659-sd6nk" -n "guestbook" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

In this scenario, one pod was attempted to be removed, but they couldn’t due to our Pod Disruption Budget (PDB). However, the remaining two pods remained intact, demonstrating the PDB’s enforcement of maintaining a minimum number of pods.

For Guestbook frontend
  1. Apply the following PDB manifest to the guestbook frontend deployment.
>guestbook-pdb.yaml
vi guestbook-pdb.yaml
  1. Configure the yaml file as such.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: frontend-pdb
  namespace: guestbook
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: guestbook
      tier: frontend
  1. Apply the yaml file.
kubectl apply -f guestbook-pdb.yaml
  1. Verify the PDB.
kubectl get pdb -n guestbook

The output should look like this:

NAME                 MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
redis-follower-pdb   2               N/A               1                     2d1h
frontend-pdb         2               N/A               1                     2d1h
Now let’s test the PDB!

Since our PDB specifies a minimum of 2 pods, draining the node should not result in its deletion. Kubernetes should prevent the node from being deleted because it needs at least one worker node to maintain the required number of pods. Let’s proceed to observe this behavior.

  1. Check the PDB.
kubectl describe pdb frontend-pdb -n guestbook

The output is as followed:

Name:           frontend-pdb
Namespace:      guestbook
Min available:  2
Selector:       app=guestbook,tier=frontend
Status:
    Allowed disruptions:  1
    Current:              3
    Desired:              2
    Total:                3
Events:                   <none>
  1. Drain the node on which the pods are running.
kubectl drain worker-1 --ignore-daemonsets

Draining the node gives the following output:

error when evicting pods/"frontend-5c47c7d777-pc2zk" -n "guestbook" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"frontend-5c47c7d777-stzx4" -n "guestbook" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

In this scenario, one pod was attempted to be removed, but they couldn’t due to our Pod Disruption Budget (PDB). However, the remaining two pods remained intact, demonstrating the PDB’s enforcement of maintaining a minimum number of pods.

That’s it, we have successfully tested the PDB and confirmed that it’s working as expected.

Deployment Strategies

A deployment strategy defines how to create, upgrade, or downgrade versions of an application. In traditional hosting, application upgrades often incur service interruption. Kubernetes avoids this with declarative Deployment strategies.

The RollingUpdate strategy is the cornerstone of zero-downtime deployments. It replaces old pods with new ones gradually, ensuring continuous application availability.

Zero-Downtime Rolling Update Parameters

When configuring a rolling update for production, two parameters determine zero-downtime safety:

  • maxUnavailable: The maximum number of pods that can be unavailable during the update process. For strict zero downtime, set maxUnavailable: 0 to guarantee that existing capacity is never reduced before replacement pods are ready and healthy.
  • maxSurge: The maximum number of pods that can be created above the desired replica count. Setting maxSurge: 1 (or 25%) allows Kubernetes to spin up new pods alongside existing ones.
For Redis follower deployment
  1. Configure the Redis follower manifest file for zero-downtime rolling updates:
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  1. Apply the new manifest:
kubectl apply -f redis-followers.yaml

Now, when you update the image or configuration, Kubernetes will create a new replica first, wait for its readiness probes to pass, and only then terminate an old replica.

You can monitor the rollout progress:

kubectl rollout status deployment/redis-follower -n guestbook
For Guestbook frontend deployment
  1. Configure the frontend manifest:
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  1. Apply the new manifest:
kubectl apply -f guestbook.yaml

Monitor rollout status:

kubectl rollout status deployment/frontend -n guestbook

Similarly, we can rollback to previous updates as well. Despite the best planning and execution, things can go wrong during a deployment. In such cases, Kubernetes provides a rollback strategy that allows you to quickly and safely roll back to a previous version of your application. Kubernetes supports two types of rollback strategies: recreate and rolling-update.

The recreate strategy is the simplest and quickest, but also the most disruptive. It simply deletes all the pods in the deployment and creates new ones with the previous version. This can cause downtime and potential data loss, so it’s not recommended for production environments.

The rolling-update strategy is more complex but also more graceful. It performs a rolling update in reverse, gradually replacing the new version with the old one.

To use the rolling-update strategy, run the following command:

kubectl rollout undo deployment/$(DEPLOYMENT_NAME)

Rolling updates allow you to update your application gradually, minimizing downtime and ensuring reliability. Rollback strategies provide a safety net in case things go wrong during the deployment.

Probes

Kubernetes provides probes (health checks) to monitor and act on the state of Pods (Containers) and to make sure only healthy Pods serve traffic. With help of Probes, we can control when a pod should be deemed started, ready for service, or live to serve traffic.

Types of Probes

  • Liveness Probe Liveness probes let Kubernetes know whether your app (running in a container inside Pod) is healthy. If app is healthy, Kubernetes will not interfere with pod functioning. If app is unhealthy, Pod will be marked as unhealthy. If a Pod fails health-checks continuously, the Kubernetes terminates the pod and starts a new one.

  • Readiness Probe Readiness probes let Kubernetes know when your app (running in a container inside Pod) is ready to serve traffic. Kubernetes makes sure the readiness probe passes before allowing a service to send traffic to the pod. Unlike a liveness probe, a readiness probe doesn’t kill the container. If the readiness probe fails, Kubernetes simply hides the container’s Pod from corresponding Services, so that no traffic is redirected to it.

Probe Parameters

Probes have a number of fields that can be used to more precisely control the behavior of liveness and readiness checks.

  • initialDelaySeconds: Number of seconds after the container has started before probes are initiated. (default: 0, minimum: 0)

  • periodSeconds: How often to perform the probe (i.e. frequency). (default: 10, minimum: 1)

  • timeoutSeconds: Number of seconds after which the probe times out. (default: 1, minimum: 1)

  • successThreshold: Minimum consecutive successes for the probe to be considered successful after failure. (default: 1, minimum: 1)

  • failureThreshold: How many failed results were received to transition from a healthy to a failure state. (default: 3, minimum: 1)

Probes configuration for Redis Deployment
  1. Configure the Redis manifests to ensure liveness and readiness states:
livenessProbe:
  tcpSocket:
	port: 6379
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:
  tcpSocket:
	port: 6379
  initialDelaySeconds: 5
  periodSeconds: 10

These liveness and readiness probes are for a container that listens on TCP port 6379, which is typically used by Redis. The liveness probe checks the container’s health by attempting to establish a TCP connection to port 6379, starting 30 seconds after the container begins running, and then every 10 seconds thereafter. If this probe fails, Kubernetes will restart the container. The readiness probe also checks the container’s ability to accept traffic by establishing a TCP connection to the same port, but it starts 5 seconds after the container starts and checks every 10 seconds. If this probe fails, the container will be marked as not ready, and it won’t receive any traffic until it passes the probe.

  1. Apply the manifests:
kubectl apply -f redis-leader.yaml
kubectl apply -f redis-followers.yaml
  1. Check the pods to see the configurations.
kubectl describe pod/redis-follower-55cf45659-dzrd2 -n guestbook

The output should be as follows:

Containers:
    Port:           6379/TCP
    Host Port:      0/TCP
    State:          Running
    Liveness:     tcp-socket :6379 delay=30s timeout=1s period=10s
    Readiness:    tcp-socket :6379 delay=5s timeout=1s period=10s

Note: If the liveness probe fails repeatedly, the container is restarted. If the readiness probe fails, the pod is temporarily removed from the Service Endpoints so no incoming client traffic reaches it, avoiding dropped requests without causing unnecessary pod restarts.

Events:
  Type     Reason     Age               From                               
  ----     ------     ----              ----                               
  Warning  Unhealthy  2m (x3 over 5m)   kubelet, worker-1                    Liveness probe failed: HTTP probe failed with statuscode: 500
  
  Normal   Killing    1m                kubelet, worker-1                    Container redis-follower-55cf45659-dzrd2-container failed liveness probe, will be restarted
Probes configuration for Guestbook frontend Deployment
  1. Configure the Guestbook frontend manifests to ensure liveness and readiness states:
livenessProbe:
  httpGet:
	path: /
	port: 80
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:
  httpGet:
	path: /
	port: 80
  initialDelaySeconds: 5
  periodSeconds: 10

These liveness and readiness probes are for a container by making HTTP GET requests to the root path (/) on port 80. The liveness probe checks if the application is running properly by sending a request 30 seconds after the container starts and then every 10 seconds. If the probe fails, indicating that the application is not responding, Kubernetes will restart the container. The readiness probe ensures the application is ready to handle traffic by sending a similar request, but it begins 5 seconds after the container starts and checks every 10 seconds. If this probe fails, the container is marked as not ready, and it will not receive any traffic until it passes the probe.

  1. Apply the manifests:
kubectl apply -f guestbook.yaml
  1. Check the pods to see the configurations.
kubectl describe pod/frontend-5c47c7d777-pc2zk -n guestbook

The output should be as follows:

Containers:
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 16 Aug 2024 11:48:00 +0530
    Liveness:   http-get http://:80/ delay=30s timeout=1s period=10s
    Readiness:  http-get http://:80/ delay=5s timeout=1s period=10s

In case of failure of either liveness or readiness the pod will spin the container again.

Events:
  Type     Reason     Age               From                               
  ----     ------     ----              ----                               
  Warning  Unhealthy  2m (x3 over 5m)   kubelet, worker-2                    Liveness probe failed: HTTP probe failed with statuscode: 500
  
  Normal   Killing    1m                kubelet, worker-2                    Container frontend-5c47c7d777-pc2zk-container failed liveness probe, will be restarted

Probes can help with troubleshooting by monitoring their applications for issues, but they can also help them to plan and manage resources by indicating when an application is experiencing resource contention.

Graceful Termination & PreStop Lifecycle Hooks

When a pod is deleted or replaced during a rolling update, Kubernetes initiates the pod shutdown sequence:

  1. The Pod is set to the Terminating state and removed from the Service’s EndpointSlice.
  2. Simultaneously, kubelet executes any configured preStop hook and sends the SIGTERM signal to the container process.
  3. Kubelet waits up to terminationGracePeriodSeconds (default: 30s) before sending SIGKILL.

Why preStop sleep is critical for zero downtime:

Network route propagation (iptables/IPVS updates via kube-proxy, Ingress controller updates, external load balancer sync) is asynchronous. It takes a few seconds (usually 2–5 seconds) for all nodes to stop forwarding traffic to the terminating pod.

If the application terminates immediately upon receiving SIGTERM, clients whose requests were already routed in-flight will receive 502 Bad Gateway or Connection Refused errors.

To solve this, add a preStop sleep hook alongside an extended terminationGracePeriodSeconds:

spec:
  terminationGracePeriodSeconds: 60
  containers:
  - name: frontend
    image: gb-frontend:v1
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 10"]

During the 10-second preStop sleep, the pod continues serving existing traffic while kube-proxy safely drains and removes the pod from active routing tables. Once the sleep ends, the container receives SIGTERM, finishes in-flight requests, closes database connections gracefully, and exits cleanly.

Pod Anti-affinity

Pod anti-affinity allows to accomplish the opposite, ensuring certain pods don’t run on the same node as other pods. The scheduler will try to put a balanced number of pods into each domain. Also, we define an eligible domain as a domain whose nodes meet the requirements of nodeAffinityPolicy and nodeTaintsPolicy.

There is one more thing we need to discuss and that is the difference between preferredDuringSchedulingIgnoredDuringExecution and requiredDuringSchedulingIgnoredDuringExecution . Preferred means that the scheduler will try to place the pod according to the affinity rule, but if it is unable to find a suitable node, it will place the pod anyway. Required means that if it can’t find a suitable node, it will not schedule the pod and the deployment will not reach full capacity.

Let’s configure Pod Anti-affinity on our deployments!

Redis leader deployment
  1. Configure the leader manifest as per the Pod Anti-affinity parameters.
affinity:
podAntiAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
  - weight: 100
	podAffinityTerm:
	  labelSelector:
		matchExpressions:
		- key: role
		  operator: In
		  values:
		  - leader
	  topologyKey: "kubernetes.io/hostname"

This configuration of podAntiAffinity with preferredDuringSchedulingIgnoredDuringExecution instructs the scheduler to prefer placing new pods on nodes where no other pods with the role=leader label are running, using a weight of 100. This helps distribute pods across different nodes for improved availability, with the rule applied at the node level using topologyKey: "kubernetes.io/hostname".

  1. Apply the manifest.
kubectl apply -f redis-leader.yaml
Redis followers deployment
  1. Configure the leader manifest as per the Pod Anti-affinity parameters.
affinity:
podAntiAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
  - weight: 100
	podAffinityTerm:
	  labelSelector:
		matchExpressions:
		- key: role
		  operator: In
		  values:
		  - follower
	  topologyKey: "kubernetes.io/hostname"

This configuration of podAntiAffinity with preferredDuringSchedulingIgnoredDuringExecution instructs the scheduler to prefer placing new pods on nodes where no other pods with the role=follower label are running, using a weight of 100. This helps distribute pods across different nodes for improved availability, with the rule applied at the node level using topologyKey: "kubernetes.io/hostname".

  1. Apply the manifest.
kubectl apply -f redis-followers.yaml
Guestbook frontend deployment
  1. Configure the leader manifest as per the Pod Anti-affinity parameters.
affinity:
podAntiAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
  - weight: 100
	podAffinityTerm:
	  labelSelector:
		matchExpressions:
		- key: app
		  operator: In
		  values:
		  - guestbook
	  topologyKey: "kubernetes.io/hostname"

This configuration of podAntiAffinity with preferredDuringSchedulingIgnoredDuringExecution instructs the scheduler to prefer placing new pods on nodes where no other pods with the role=guestbook label are running, using a weight of 100. This helps distribute pods across different nodes for improved availability, with the rule applied at the node level using topologyKey: "kubernetes.io/hostname".

  1. Apply the manifest.
kubectl apply -f guestbook.yaml

Now let’s test the pod anti-affinity configuration.

As we have applied the configuration, the pods should be distributed across all the available worker nodes. If a worker goes down, the pods across other nodes make sure the application doesn’t go down entirely and spin up the mentioned replicas on the available worker nodes.

kubectl get po -n guestbook -o wide

The output should be as follows:

frontend-548d5496c8-8msgg  1/1  Running  0  20m  192.168.58.112  worker-1
frontend-548d5496c8-mv74z  1/1  Running  0  20m  192.168.58.116  worker-2
frontend-548d5496c8-s9n4m  1/1  Running  0  20m  192.168.58.84   worker-1
redis-follower-54797dcb64-ms2lf  1/1  Running  0  22m  192.168.58.106  worker-2
redis-follower-54797dcb64-p7cd8  1/1  Running  0  22m  192.168.58.99   worker-2
redis-follower-54797dcb64-t9lhq  1/1  Running  0  22m  192.168.58.102  worker-1
redis-leader-54985d56fd-7qrgx  1/1  Running  0  19m  192.168.58.100  worker-2

By carefully configuring pod anti-affinity, we mitigated the risk of complete service downtime due to single-node failures.

Modern Alternative: Topology Spread Constraints

In modern Kubernetes clusters, Pod Topology Spread Constraints (topologySpreadConstraints) are often preferred over PodAntiAffinity because they allow even distribution of pods across failure domains (nodes, racks, or availability zones) without the binary “all or nothing” limitations of anti-affinity:

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: "kubernetes.io/hostname"
    whenUnsatisfiable: ScheduleAnyway  # or DoNotSchedule
    labelSelector:
      matchLabels:
        app: guestbook
        tier: frontend
  • maxSkew: 1: Specifies the maximum allowable difference in pod count between any two nodes.
  • topologyKey: The node label key used to define the topology domain (e.g. hostname or topology.kubernetes.io/zone).
  • whenUnsatisfiable: ScheduleAnyway prioritizes high availability while still allowing pods to schedule if nodes are constrained.

Resources

When you specify a Pod, you can optionally specify how much of each resource a container needs. The most common resources to specify are CPU and memory (RAM); there are others. When you specify the resource request for containers in a Pod, the kube-scheduler uses this information to decide which node to place the Pod on. When you specify a resource limit for a container, the kubelet enforces those limits so that the running container is not allowed to use more of that resource than the limit you set. The kubelet also reserves at least the request amount of that system resource specifically for that container to use. The kube-scheduler places Pods on nodes with sufficient resources, preventing overloading, while the kubelet enforces limits to prevent any single container from consuming excessive resources, which could destabilize other Pods. This careful resource management helps maintain consistent performance and reduces the risk of crashes or slowdowns, contributing to zero downtime.

Let’s allocate resources to our pods and deployments.

For Redis Deployment
  1. Configure the Redis manifests and give appropriate resource requests and limits as per the application.
resources:
  requests:
	cpu: 200m
	memory: 256Mi
  limits:
	cpu: 500m
	memory: 512Mi

In the Redis deployment, each container requests a minimum of 200 millicores of CPU and 256 MiB of memory, ensuring reliable performance. The maximum usage is capped at 500 millicores of CPU and 512 MiB of memory, preventing resource overconsumption.

  1. Apply the manifest.
kubectl apply -f redis-leader.yaml
kubectl apply -f redis-followers.yaml

These resources are now allocated the pods and accordingly assigned by the node.

  1. Check the node description on which the pods are assigned.
kubectl describe node/worker-2

The output is as follows: Kubernetes describe node output showing Redis pod CPU and memory allocation

For Guestbook frontend Deployment
  1. Configure the frontend manifests and give appropriate resource requests and limits as per the application.
resources:
  requests:
	cpu: 200m
	memory: 256Mi
  limits:
	cpu: 500m
	memory: 512Mi

In the Guestbook frontend deployment, each container requests a minimum of 200 millicores of CPU and 256 MiB of memory, ensuring reliable performance. The maximum usage is capped at 500 millicores of CPU and 512 MiB of memory, preventing resource overconsumption.

  1. Apply the manifest.
kubectl apply -f guestbook.yaml

These resources are now allocated the pods and accordingly assigned by the node.

  1. Check the node description on which the pods are assigned.
kubectl describe node/worker-1

The output is as follows: Kubernetes describe node output showing Frontend pod CPU and memory allocation

CPU requests are not used just for scheduling, but also take a huge part in the whole container lifecycle!

Horizontal Pod Autoscaling

Let’s think, we have a constantly running production service with a load that is variable in time, where it is very busy during the day, and relatively low at night. Normally, we would want the number of nodes in the cluster and the number of pods in deployment to dynamically adjust to the load to meet end-user demand. The Cluster Autoscaling feature together with the Horizontal Pod Autoscaler can handle this for you automatically.

For per-pod resource metrics (like CPU), the controller fetches the metrics from the resource metrics API for each Pod targeted by the HorizontalPodAutoscaler. Then, if a target utilization value is set, the controller calculates the utilization value as a percentage of the equivalent resource request on the containers in each Pod. If a target raw value is set, the raw metric values are used directly. The controller then takes the mean of the utilization or the raw value (depending on the type of target specified) across all targeted Pods, and produces a ratio used to scale the number of desired replicas.

Install the metrics server

To get a running HorizontalPodAutoscaler, we need the metrics. The metrics can’t be calculated. So, the metrics server should be uploaded to the cluster.

  1. Install the Metric Server from with the following command.
kubectl apply -f https://gist.githubusercontent.com/VinayakSomvanshi/cf8ea2e4db25c61c41185e69778227f6/raw/2741c1e0a0872a98e8ddbf33aa0e5112ed1505d4/metrics.yaml
  1. Verify the existence of metrics-server run by the below command.
kubectl get pods -n kube-system

The output for the metrics server pod should look like this:

metrics-server-d994c478f-mt6v4    1/1     Running   24 (107m ago)   2d5h
  1. Verify metrics-server can access the resources of the nodes.
kubectl top node

It should look like this:

NAME      CPU(cores)   CPU%        MEMORY(bytes)   MEMORY%     
worker-1   416m         10%         2579Mi          67%         
master-2   277m         13%         1242Mi          66%         
master-3   277m         13%         1529Mi          82%         
master-1   277m         13%         1529Mi          82%
worker-2   416m         10%         2579Mi          67%
  1. Verify metrics-server can access the resources of the pods.
kubectl top pods -n guestbook

It should look like this:

NAME                              CPU(cores)   MEMORY(bytes)   
frontend-548d5496c8-8msgg         1m           9Mi             
frontend-548d5496c8-mv74z         1m           10Mi            
frontend-548d5496c8-s9n4m         1m           8Mi             
redis-follower-54797dcb64-ms2lf   2m           9Mi             
redis-follower-54797dcb64-p7cd8   2m           2Mi             
redis-follower-54797dcb64-t9lhq   2m           2Mi             
redis-leader-54985d56fd-7qrgx     2m           2Mi

Now let’s create HorizontalPodAutoscaler for the application deployments.

For Redis Deployment
  1. Create HorizontalPodAutoscaler manifest.
>redis-hpa.yaml
vi redis-hpa.yaml
  1. Add the following configuration in the redis-hpa.yaml file.

Note: autoscaling/v2 is the stable API since Kubernetes 1.23. The old autoscaling/v1 is deprecated. autoscaling/v2 supports multiple metrics (CPU, memory, custom) and more expressive scaling behavior.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: redis-leader-hpa
  namespace: guestbook
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: redis-leader
  minReplicas: 1
  maxReplicas: 1
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: redis-follower-hpa
  namespace: guestbook
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: redis-follower
  minReplicas: 3
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

This configuration sets up a Horizontal Pod Autoscaler (HPA) for the redis deployment within the guestbook namespace. It ensures that the number of replicas for the pods adjusts automatically based on CPU usage. Specifically, the HPA will maintain at least 3 replicas and can scale up to a maximum of 5 replicas. The scaling occurs when the average CPU utilization across these Pods surpasses 50% of the allocated CPU resources, thereby dynamically managing resource allocation to handle varying demand and ensuring consistent performance.

  1. Apply the HorizontalPodAutoscaler manifest.
kubectl apply -f redis-hpa.yaml
For Guestbook frontend Deployment
  1. Create HorizontalPodAutoscaler manifest.
>frontend-hpa.yaml
vi frontend-hpa.yaml
  1. Add the following configuration in the frontend-hpa.yaml file.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: frontend-hpa
  namespace: guestbook
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: frontend
  minReplicas: 3
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

This configuration sets up a Horizontal Pod Autoscaler (HPA) for the frontend deployment within the guestbook namespace. It ensures that the number of replicas for the pods adjusts automatically based on CPU usage. Specifically, the HPA will maintain at least 3 replicas and can scale up to a maximum of 5 replicas. The scaling occurs when the average CPU utilization across these Pods surpasses 50% of the allocated CPU resources, thereby dynamically managing resource allocation to handle varying demand and ensuring consistent performance.

  1. Apply the HorizontalPodAutoscaler manifest.
kubectl apply -f frontend-hpa.yaml
  1. Verify the applied HorizontalPodAutoscaler.
kubectl get hpa -n guestbook

The output is as follows:

NAME                 REFERENCE                   TARGETS       MINPODS   MAXPODS
frontend-hpa         Deployment/frontend         cpu: 0%/50%   3         5
redis-follower-hpa   Deployment/redis-follower   cpu: 1%/50%   3         5
redis-leader-hpa     Deployment/redis-leader     cpu: 1%/50%   1         1
Increasing Load to Test The HPA by Using The Infinite Loop of Queries

We want to see how the autoscaler reacts to increased load. For this, we will send an infinite loop of queries to the frontend deployment.

  1. Paste the following code into the terminal with the frontend’s IP and port.
while true; do wget -q -O- http://192.168.1.150:80; done
  1. On the watch board, we must run the command below in a different terminal in order to constantly check output.
watch kubectl get service,hpa,pod -n guestbook -o wide

Kubernetes HPA status showing initial replica count before load generation

When the load continues to increase, HPA automatically creates more pod replicas.

Kubernetes HPA status showing autoscaled replica pods under load

Here, we can see that when the target CPU utilization is reached, more replica pods are spinned up to accommodate the traffic.

The Horizontal Pod Autoscaler (HPA) for the application ensures zero downtime by automatically adjusting the number of Pod replicas based on CPU usage. It maintains a minimum of replicas and scales up if needed, ensuring continuous availability and performance. This dynamic scaling helps handle varying workloads while keeping the system stable and responsive.

To achieve zero downtime in Kubernetes, a comprehensive approach is essential. This involves proper cluster setup, strategic container placement, effective use of Deployments and ReplicaSets, implementing Pod Distribution Budgets, and carefully crafted Rolling Update Strategies. Probes for liveness and readiness, along with well-managed termination and boot processes, ensure application health and availability. Pod anti-affinity rules for both control-plane and worker nodes enhance reliability, while appropriate resource allocation and Horizontal Pod Autoscaling (HPA) optimize performance and scalability. By integrating these elements, organizations can create robust, highly available Kubernetes environments that minimize disruptions and maintain continuous service delivery.