Making ConfigMaps rollback-friendly with immutability

Aug 02, 2026

During release related incidents/outages, one of the first instincts for a production on-call engineer is to Rollback!

When a new application version causes errors, elevated latency, crash loops, or bad behavior, the fastest mitigation is often to go back to the last known good version.

The rollback procedure varies depending upon the CI/CD setup in the organization - whether using Helm, ArgoCD, Spinnaker, or a direct kubectl rollout undo command.

At the K8s object level, this is straightforward. K8s maintains the revision history of the deployments. When the pod template changes, it creates a new ReplicaSet and pods. I wrote about how the changes to the pod template cause a new ReplicaSet in an earlier post.

So this rollback works effectively, when the bad change is part of the pod template.

For example:

container image changed
command or args changed
environment variables changed
volume references changed

But what if the bad release is caused by the changes in the ConfigMap? ConfigMaps are mutable by default, so when we make changes to the ConfigMap it gets applied in place. There are no versioning or revision for them. So we can’t undo the ConfigMap changes.

In our CoreDNS outage, we passed the wrong configuration which led to an outage, and a normal rollout undo wouldn’t have helped.

In this post, we will explore how we can use immutability feature in ConfigMaps to make them rollback friendly.

Why can’t we rollback ConfigMaps?

ConfigMaps are mutable by default. When a ConfigMap is updated, the change is applied to the same object and Kubernetes does not keep revision history for ConfigMap data.

Let’s see this in action.

I’m creating a ConfigMap and a nginx Deployment which volume mounts the ConfigMap and print the data when we access it.

# nginx-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-content
data:
  index.html: |
    <html>
      <body>
        <h1>Hello from ConfigMap v1</h1>
      </body>
    </html>

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-config-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-config-demo
  template:
    metadata:
      labels:
        app: nginx-config-demo
    spec:
      containers:
      - name: nginx
        image: nginx:1.27
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-content-volume
          mountPath: /usr/share/nginx/html
          readOnly: true
      volumes:
      - name: nginx-content-volume
        configMap:
          name: nginx-content

I also created a service exposing the deployment. Now if we access the service, we will get the ConfigMap data as response.

➜  kubectl port-forward svc/nginx-config-demo 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080

➜  curl localhost:8080
<html>
  <body>
    <h1>Hello from ConfigMap : v1</h1>
  </body>
</html>

Now I’m making changes to the ConfigMap and updating the data as “BAD RELEASE”.

➜  k patch configmap nginx-content \
  --type merge \
  -p '{"data":{"index.html":"<html><body><h1>BAD RELEASE!!!</h1></body></html>"}}'
configmap/nginx-content patched

Now there is no Kubernetes-native ConfigMap revision to roll back to. We can manually re-apply the previous content if we have it, but kubectl rollout undo cannot restore it for us

When the new pods are created during a rollout or when this ConfigMap change eventually propagates, the pods will start returning the ‘bad’ config data.

➜  k rollout restart deploy nginx-config-demo
deployment.apps/nginx-config-demo restarted

➜  curl localhost:8080
<html>
  <body>
    <h1>BAD CONFIG!!!!</h1>
  </body>
</html>
➜

Immutability to the rescue

One way to make ConfigMap changes safer is to stop mutating them in place.

Kubernetes supports immutable ConfigMaps using immutable: true. Once enabled, the ConfigMap data cannot be updated in place.

# nginx-configmap.yaml now using `immutable` field
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-content
immutable: true
data:
  index.html: |
    <html>
      <body>
        <h1>Hello from ConfigMap v1</h1>
      </body>
    </html>

➜  k patch configmap nginx-content \
  --type merge \
  -p '{"data":{"index.html":"<html><body><h1>BAD RELEASE!!!</h1></body></html>"}}'
The ConfigMap "nginx-content" is invalid: data: Forbidden: field is immutable when `immutable` is set

So we have to create a new ConfigMap for any new changes and pass the ConfigMap name to our deployment.

# nginx-configmap.yaml - Creating a new ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-content-v2
immutable: true
data:
  index.html: |
    <html>
      <body>
        <h1>BAD RELEASE!!!!</h1>
      </body>
    </html>

# in our nginx-deployment, we have to update the name as "nginx-content-v2"

When the Deployment references a specific ConfigMap name, that reference is part of the pod template. Changing the ConfigMap name changes the pod template, which creates a new ReplicaSet and a new rollout revision.

If the new config is bad, rolling back the Deployment restores the previous pod template, which points back to the previous ConfigMap.

➜  k describe deployments.apps nginx-config-demo | grep Volumes -A3
  Volumes:
   nginx-content-volume:
    Type:          ConfigMap (a volume populated by a ConfigMap)
    Name:          nginx-content
➜  k apply -f .
configmap/nginx-content-v2 unchanged
deployment.apps/nginx-config-demo configured
service/nginx-config-demo unchanged

➜  k describe deployments.apps nginx-config-demo | grep Volumes -A3
  Volumes:
   nginx-content-volume:
    Type:          ConfigMap (a volume populated by a ConfigMap)
    Name:          nginx-content-v2
➜  
➜  k rollout undo deployment nginx-config-demo
deployment.apps/nginx-config-demo rolled back
➜  
➜  k describe deployments.apps nginx-config-demo | grep Volumes -A3
  Volumes:
   nginx-content-volume:
    Type:          ConfigMap (a volume populated by a ConfigMap)
    Name:          nginx-content

The Cleanup problem

Versioned ConfigMaps introduce one obvious problem, there will be too many ConfigMaps lying around.

So we have to use some kind of cleanup job to delete the old ConfigMaps periodically and of course, we have to ensure it doesn’t delete the active ones!

Closing thoughts

ConfigMaps are an overlooked failure mode. We usually think of bad releases as bad application code or bad container images, but configuration can also break production.

Kubernetes gives us a rollback option for Deployment pod templates, but mutable ConfigMap data sits outside that rollback history.

If a ConfigMap is critical enough to take down production, it should be treated like a release artifact: version it, make the workload reference a specific version, and keep older versions long enough to roll back.

Systems under Load

Discussion about this post

Ready for more?