Monday 27 July 2020

Kubernetes: Deployment Strategies: Rollback and update

When you initiate a deployment task, a new rollout get initiated. Every new rollout creates a deployment revision.


When you initiate another deployment task, new deployment revision ‘Revision 2’ gets created.

Why revisions?

Revisions are used to keep track of the deployment and made the rollbacks easy.

 

Recreate Strategy

Delete all the Pods and recreate the Pods again with a specified image. In this strategy, Application downtime exists.

 

Rolling Update

In this strategy, instead of deleting all the Pods in one shot, we take down one pod at a time. Once a Pod is deleted, a newer Pod with the Specified image gets created. It is the default deployment strategy in Kubernetes.

 

Let’s see, how can we perform an update and undo operations using Deployment Object.

 

Step 1: Create a new deployment.

 

notificationServiceDeployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: notification-service-deployment
  labels:
    app: notification-service-deployment
    author: krishna
    serviceType: webservice
spec:
  template:
    metadata:
      name: notification-service
      labels:
        app: notification-service
        author: krishna
        serviceType: webservice
    spec:
      containers:
      - name: notification-service-container
        image: jboss/wildfly:10.1.0.Final

  replicas: 10
  selector:
    matchLabels:
      app: notification-service

Let’s create deployment by executing the below command.

kubectl create -f notificationServiceDeployment.yml

$kubectl create -f notificationServiceDeployment.yml 
deployment.apps/notification-service-deployment created

Check the rollout status

Check the rollout status by executing the below command.

kubectl rollout status deployment/my-deployment

$kubectl rollout status deployment/notification-service-deployment
Waiting for deployment "notification-service-deployment" rollout to finish: 0 of 10 updated replicas are available...
Waiting for deployment "notification-service-deployment" rollout to finish: 1 of 10 updated replicas are available...
Waiting for deployment "notification-service-deployment" rollout to finish: 2 of 10 updated replicas are available...
Waiting for deployment "notification-service-deployment" rollout to finish: 3 of 10 updated replicas are available...
Waiting for deployment "notification-service-deployment" rollout to finish: 4 of 10 updated replicas are available...
Waiting for deployment "notification-service-deployment" rollout to finish: 5 of 10 updated replicas are available...
Waiting for deployment "notification-service-deployment" rollout to finish: 6 of 10 updated replicas are available...
Waiting for deployment "notification-service-deployment" rollout to finish: 7 of 10 updated replicas are available...
Waiting for deployment "notification-service-deployment" rollout to finish: 8 of 10 updated replicas are available...
Waiting for deployment "notification-service-deployment" rollout to finish: 9 of 10 updated replicas are available...
deployment "notification-service-deployment" successfully rolled out

Check rollout history

Execute the below command to see rollout history.

kubectl rollout history deployment/my-deployment

$kubectl rollout history deployment/notification-service-deployment
deployment.apps/notification-service-deployment 
REVISION  CHANGE-CAUSE
1	<none>

As you see Revision 1 is created for the deployment.

 

When you initiate a deployment, Kubernetes automatically create a Replicaset for us.

$kubectl get all
NAME                                                  READY   STATUS    RESTARTS   AGE
pod/notification-service-deployment-cdf5795d4-7pw86   1/1     Running   0          7m23s
pod/notification-service-deployment-cdf5795d4-98th2   1/1     Running   0          7m23s
pod/notification-service-deployment-cdf5795d4-9g8lg   1/1     Running   0          7m23s
pod/notification-service-deployment-cdf5795d4-jjzvq   1/1     Running   0          7m23s
pod/notification-service-deployment-cdf5795d4-mgdpr   1/1     Running   0          7m23s
pod/notification-service-deployment-cdf5795d4-n8kpj   1/1     Running   0          7m23s
pod/notification-service-deployment-cdf5795d4-pkrg9   1/1     Running   0          7m23s
pod/notification-service-deployment-cdf5795d4-q4zhw   1/1     Running   0          7m23s
pod/notification-service-deployment-cdf5795d4-zqx76   1/1     Running   0          7m23s
pod/notification-service-deployment-cdf5795d4-zrbqs   1/1     Running   0          7m23s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   28d

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/notification-service-deployment   10/10   10           10          7m23s

NAME                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/notification-service-deployment-cdf5795d4   10        10        10      7m23s

Step 2: Let’s update the jboss/wildfly to some other version jboss/wildfly:15.0.0.Final.

 

When you initiate updation, Kubernetes create another replica set to deploy pods with new revision.

 

Let’s confirm this experimentally.

$kubectl get replicaset
NAME                                        DESIRED   CURRENT   READY   AGE
notification-service-deployment-cdf5795d4   10        10        10      8m26s

As you see the output there is one replicaset present now.

$kubectl set image deployment/notification-service-deployment notification-service-container=jboss/wildfly:15.0.0.Final
deployment.apps/notification-service-deployment image updated

Check the rollout status

$kubectl rollout status deployment/notification-service-deployment
Waiting for deployment "notification-service-deployment" rollout to finish: 5 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 5 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 5 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 6 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 6 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 6 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 6 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 6 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 7 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 7 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 7 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 7 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 7 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 7 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 7 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 7 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 8 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 9 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 9 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 9 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 9 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 9 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 2 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 2 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 2 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 2 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 8 of 10 updated replicas are available...
Waiting for deployment "notification-service-deployment" rollout to finish: 9 of 10 updated replicas are available...
deployment "notification-service-deployment" successfully rolled out

Check the rollout history.

$kubectl rollout history deployment/notification-service-deployment
deployment.apps/notification-service-deployment 
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

As you see the output, 2nd revision gets created.

 

When you query replicasets, you confirm that two replicasets get created. Whenever you initiate a deployment, a new replicaset created to take care of the deployment of the Pods.

$kubectl get replicaset -o wide
NAME                                         DESIRED   CURRENT   READY   AGE     CONTAINERS                       IMAGES                       SELECTOR
notification-service-deployment-5b9b7ffc68   10        10        10      3m46s   notification-service-container   jboss/wildfly:15.0.0.Final   app=notification-service,pod-template-hash=5b9b7ffc68
notification-service-deployment-cdf5795d4    0         0         0       16m     notification-service-container   jboss/wildfly:10.1.0.Final   app=notification-service,pod-template-hash=cdf5795d4

As you see currently active replica set pointing to the image ‘jboss/wildfly:15.0.0.Final’.

 

Step 3: You can confirm the rolling update strategy by describing the deployment.

$kubectl describe deployment notification-service-deployment
Name:                   notification-service-deployment
Namespace:              default
CreationTimestamp:      Fri, 05 Jun 2020 10:10:46 +0530
Labels:                 app=notification-service-deployment
                        author=krishna
                        serviceType=webservice
Annotations:            deployment.kubernetes.io/revision: 2
Selector:               app=notification-service
Replicas:               10 desired | 10 updated | 10 total | 10 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=notification-service
           author=krishna
           serviceType=webservice
  Containers:
   notification-service-container:
    Image:        jboss/wildfly:15.0.0.Final
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   notification-service-deployment-5b9b7ffc68 (10/10 replicas created)
Events:
  Type    Reason             Age                    From                   Message
  ----    ------             ----                   ----                   -------
  Normal  ScalingReplicaSet  18m                    deployment-controller  Scaled up replica set notification-service-deployment-cdf5795d4 to 10
  Normal  ScalingReplicaSet  6m41s                  deployment-controller  Scaled up replica set notification-service-deployment-5b9b7ffc68 to 3
  Normal  ScalingReplicaSet  6m41s                  deployment-controller  Scaled down replica set notification-service-deployment-cdf5795d4 to 8
  Normal  ScalingReplicaSet  6m41s                  deployment-controller  Scaled up replica set notification-service-deployment-5b9b7ffc68 to 5
  Normal  ScalingReplicaSet  5m40s                  deployment-controller  Scaled down replica set notification-service-deployment-cdf5795d4 to 7
  Normal  ScalingReplicaSet  5m40s                  deployment-controller  Scaled up replica set notification-service-deployment-5b9b7ffc68 to 6
  Normal  ScalingReplicaSet  5m37s                  deployment-controller  Scaled down replica set notification-service-deployment-cdf5795d4 to 6
  Normal  ScalingReplicaSet  5m37s                  deployment-controller  Scaled up replica set notification-service-deployment-5b9b7ffc68 to 7
  Normal  ScalingReplicaSet  5m35s                  deployment-controller  Scaled down replica set notification-service-deployment-cdf5795d4 to 5
  Normal  ScalingReplicaSet  5m35s                  deployment-controller  Scaled up replica set notification-service-deployment-5b9b7ffc68 to 8
  Normal  ScalingReplicaSet  5m29s (x7 over 5m35s)  deployment-controller  (combined from similar events): Scaled down replica set notification-service-deployment-cdf5795d4 to 0

As you see the Events: section of the output, old deployment replica set scaled-down, while new deployment replica set scaled up.

 

Step 4: Rollback to the previous deployment.

 

Let’s query replicasets first.

$kubectl get replicaset -o wide
NAME                                         DESIRED   CURRENT   READY   AGE   CONTAINERS                       IMAGES                       SELECTOR
notification-service-deployment-5b9b7ffc68   10        10        10      16m   notification-service-container   jboss/wildfly:15.0.0.Final   app=notification-service,pod-template-hash=5b9b7ffc68
notification-service-deployment-cdf5795d4    0         0         0       28m   notification-service-container   jboss/wildfly:10.1.0.Final   app=notification-service,pod-template-hash=cdf5795d4

As you see replicaset ‘notification-service-deployment-5b9b7ffc68’ which points to the image ‘jboss/wildfly:15.0.0.Final’ is active.

 

Now when I rollback the deployment to the previous state, the deployment destroys the pods in new replica set and creates pods in old replica set.

 

You can rollback to the previous deployment by executing below command

kubectl rollout undo deployment/notification-service-deployment

$kubectl rollout undo deployment/notification-service-deployment
deployment.apps/notification-service-deployment rolled back

Query for replica sets again.

$kubectl get replicaset -o wide
NAME                                         DESIRED   CURRENT   READY   AGE   CONTAINERS                       IMAGES                       SELECTOR
notification-service-deployment-5b9b7ffc68   0         0         0       19m   notification-service-container   jboss/wildfly:15.0.0.Final   app=notification-service,pod-template-hash=5b9b7ffc68
notification-service-deployment-cdf5795d4    10        10        10      32m   notification-service-container   jboss/wildfly:10.1.0.Final   app=notification-service,pod-template-hash=cdf5795d4

You can confirm that the older replica set has 10 Pods, where a new replica set has 0 pods.

 

Now, you can see revision number 3 gets created.

$kubectl rollout history deployment/notification-service-deployment
deployment.apps/notification-service-deployment 
REVISION  CHANGE-CAUSE
2         <none>
3         <none>

Step 5: You can even update the deployment definition file and apply the changes using ‘kubectl apply’ command.

 

For example, I changed the image to 19.1.0.

image: jboss/wildfly:19.1.0.Final

 

notificationServiceDeployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: notification-service-deployment
  labels:
    app: notification-service-deployment
    author: krishna
    serviceType: webservice
spec:
  template:
    metadata:
      name: notification-service
      labels:
        app: notification-service
        author: krishna
        serviceType: webservice
    spec:
      containers:
      - name: notification-service-container
        image: jboss/wildfly:19.1.0.Final

  replicas: 10
  selector:
    matchLabels:
      app: notification-service

Apply the changes using the below command.

$kubectl apply -f notificationServiceDeployment.yml 
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
deployment.apps/notification-service-deployment configured

Execute rollout status command to see the rollout status.

$kubectl apply -f notificationServiceDeployment.yml 
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
deployment.apps/notification-service-deployment configured
$kubectl rollout status deployment/notification-service-deployment
Waiting for deployment "notification-service-deployment" rollout to finish: 5 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 5 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 5 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 5 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 5 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 6 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 6 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 6 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 6 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 6 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 7 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 7 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 7 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 7 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 8 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 8 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 8 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 9 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 9 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 9 out of 10 new replicas have been updated...
Waiting for deployment "notification-service-deployment" rollout to finish: 3 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 3 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 3 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 3 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 2 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 2 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "notification-service-deployment" rollout to finish: 8 of 10 updated replicas are available...
Waiting for deployment "notification-service-deployment" rollout to finish: 9 of 10 updated replicas are available...
deployment "notification-service-deployment" successfully rolled out

Now you can see three replica sets.

$kubectl get replicasets
NAME                                         DESIRED   CURRENT   READY   AGE
notification-service-deployment-5b9b7ffc68   0         0         0       87m
notification-service-deployment-6b49cf4b55   10        10        10      99s
notification-service-deployment-cdf5795d4    0         0         0       100m

Now you can see another revision 4 is created.

$kubectl rollout history deployment/notification-service-deployment
deployment.apps/notification-service-deployment 
REVISION  CHANGE-CAUSE
2         <none>
3         <none>
4         <none>

As you see CHANGE-CAUSE is set to none.

 

Let’s use the option --record to record the change cause.

 

Let’s update the definition file to point to the latest image of wildfly.

 

notificationServiceDeployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: notification-service-deployment
  labels:
    app: notification-service-deployment
    author: krishna
    serviceType: webservice
spec:
  template:
    metadata:
      name: notification-service
      labels:
        app: notification-service
        author: krishna
        serviceType: webservice
    spec:
      containers:
      - name: notification-service-container
        image: jboss/wildfly

  replicas: 10
  selector:
    matchLabels:
      app: notification-service

Let’s apply the changes using --record option.

$kubectl apply -f notificationServiceDeployment.yml --record
deployment.apps/notification-service-deployment configured

Now when you execute rollout history command, you can see the change cause against the revision 5.

$kubectl rollout history deployment/notification-service-deployment
deployment.apps/notification-service-deployment 
REVISION  CHANGE-CAUSE
2         <none>
3         <none>
4         <none>
5         kubectl apply --filename=notificationServiceDeployment.yml --record=true


Previous                                                    Next                                                    Home

No comments:

Post a Comment