How to resize a kubernetes statefulset’s volumes bestinau

Creating a StatefulSet

Copy this YAML and save it to ss.yaml:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  ports:
  - name: nginx
    port: 80
  clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  serviceName: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - name: web
          containerPort: 80
        volumeMounts:
        - name: data
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 1Gi

Apply the YAML to your cluster with Kubectl:

$ kubectl apply -f ss.yaml
service/nginx created
statefulset.apps/nginx created

To run this example, you’ll need a storage class and provisioner in your cluster. A StatefulSet is created and three clones of the NGINX web server are started.

Even while this isn’t an example of when StatefulSets should be utilized, it serves as a good illustration of the volume issues you can encounter. The data directory of NGINX is mounted with a volume claim that has 1 Gi of capacity. As your service grows, your site material can exceed this rather tiny allowance. However, if you attempt to change the volumeClaimTemplates.spec.resources.requests.storage field to 10GI, kubectl apply will display the following error:

$ kubectl apply -f ss.yaml
service/nginx unchanged
The StatefulSet "nginx" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

Manually Resizing StatefulSet Volumes

By manually adjusting the persistent volume claim’s size, you can get around the restriction (PVC). To release and rebind the volume from your Pods, you must then recreate the StatefulSet. This will start the volume resizing event itself.

Find the PVCs connected to your StatefulSet first using Kubectl:

$ kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES
data-nginx-0   Bound    pvc-ccb2c835-e2d3-4632-b8ba-4c8c142795e4   1Gi        RWO         
data-nginx-1   Bound    pvc-1b0b27fe-3874-4ed5-91be-d8e552e515f2   1Gi        RWO         
data-nginx-2   Bound    pvc-4b7790c2-3ae6-4e04-afee-a2e1bae4323b   1Gi        RWO

The StatefulSet contains three replicas, hence there are three PVCs. Each Pod has an own volume level.

Now, modify each volume’s capacity using kubectl edit:

$ kubectl edit pvc data-nginx-0

The YAML manifest for the PVC will show up in your editor. Locate the spec.resources.requests.storage field, then modify it to the new capacity you want:

# ...
spec:
  resources:
    requests:
      storage: 10Gi
# ...

Save the document, then exit. Your cluster should now have the change, according to Kubectl.

persistentvolumeclaim/data-nginx-0 edited

Now repeat similar actions for the remaining PVCs in the StatefulSet. The updated size for each of the persistent volumes in your cluster should then be listed:

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               
pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7   10Gi       RWO            Delete           Bound    default/data-nginx-2
pvc-33af452d-feff-429d-80cd-a45232e700c1   10Gi       RWO            Delete           Bound    default/data-nginx-0
pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5   10Gi       RWO            Delete           Bound    default/data-nginx-1

The claims will maintain the old size for now:

$ kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES
data-nginx-0   Bound    pvc-33af452d-feff-429d-80cd-a45232e700c1   10Gi       RWO         
data-nginx-1   Bound    pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5   10Gi       RWO         
data-nginx-2   Bound    pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7   10Gi       RWO

This is due to the fact that the volume cannot be changed when Pods are using it.

Recreating the StatefulSet

Release the volume claim from the StatefulSet holding it to finish the resizing. Use the orphan cascading mechanism to keep the StatefulSet’s Pods in your cluster while deleting it. Downtime will be reduced as a result.

$ kubectl delete statefulset --cascade=orphan nginx
statefulset.apps "nginx" deleted

Next edit your original YAML file to include the new volume size in the spec.resources.requests.storage file. Then use kubectl apply to recreate the StatefulSet in your cluster:

$ kubectl apply -f ss.yaml
service/nginx unchanged
statefulset.apps/nginx created

The new StatefulSet will assume ownership of the previously orphaned Pods because they’ll already meet its requirements. The volumes may get resized at this point but in most cases you’ll have to manually initiate a rollout that restarts your Pods:

$ kubectl rollout restart statefulset nginx

The rollout proceeds sequentially, targeting one Pod at a time. This ensures your service remains accessible throughout.

Now your PVCs should show the new size:

$ kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES
data-nginx-0   Bound    pvc-33af452d-feff-429d-80cd-a45232e700c1   10Gi       RWO         
data-nginx-1   Bound    pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5   10Gi       RWO         
data-nginx-2   Bound    pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7   10Gi       RWO

Try connecting to one of your Pods to check the increased capacity is visible from within:

$ kubectl exec -it nginx-0 bash
root@nginx-0:/# df -h /usr/share/nginx/html
Filesystem                                                                Size  Used Avail Use% Mounted on
/dev/disk/by-id/scsi-0DO_Volume_pvc-33af452d-feff-429d-80cd-a45232e700c1  9.9G  4.5M  9.4G   1% /usr/share/nginx/html

The Pod’s reporting the expected 10 Gi of storage.

Summary

Stateful applications can be run in Kubernetes with persistent storage volumes that are specific to individual Pods thanks to Kubernetes StatefulSets. However, when you need to resize one of your volumes, the flexibility this allows disappears. This is a feature that is currently lacking and must be manually implemented in a specific order.

The problem is known to the Kubernetes maintainers. Initiating volume resizes by changing a StatefulSet’s manifest might soon be possible thanks to an open feature request for a fix. This will be a lot faster and safer than how things are now.

A storage driver that supports dynamic expansion is a must for volume resizes, which is the final caution. Not all drivers, Kubernetes distributions, and cloud platforms will support this capability, which was only made generally accessible in Kubernetes v1.24. Running kubectl get sc and looking for true in the ALLOWVOLUMEXPANSION column of the storage driver you’re using with your StatefulSets will show you whether yours does or not.

We hope the above information has helped you with How to resize a kubernetes statefulset’s volumes bestinau and if not kindly drop a comment