One of the main benefits of Kubernetes is, when you deploy a Pod using ReplicationController (or) ReplicationSet (or) Deployment, Kubernetes make sure that the containers in the Pod are always running. If by any chance the container is killed or crashed, Kubernetes restart the container automatically.
How can Kubernetes check whether the container is in a healthy state or not?
Here the livenessProbe comes into the picture. You can specify livenessProbe of an application in the manifest file, and Kubernetes (kubelet checks) periodically execute this livenessProbe url to make sure that the container is in healthy state. If by any chance, this livenessProbe do not respond in time, then Kubernetes assumes that the container is not healthy and increase the failure count, when this failure count reaches to the certain threshold value, then Kubernetes (kubectl) initiate container restart.
What if a node itself crashes?
Control Pane create replacements for all the Pods it managing.
How to configure a livenessProbe?
There are three mechanisms to configure a livenessProbe.
a. Using HTTP GET API: Kubernetes execute HTTP GET request on containers IP address and port and the livenessProbe path you specified in the manifest file. If the response code is 2xx or 3xx then Kubernetes assume that the container is in healthy state, else it initiate restart.
b. Using TCP Socket Probe: Kubernetes tries to open a TCP connection to the specified port of the container, if the TCP connection request successful then Kubernetes assume that the container is in healthy state, else it initiate restart.
c. Execute arbitrary command inside the container and if the command exit code is 0 (0 represents command executed successfully) then Kubernetes assume that the container is in healthy state, else it initiate restart.
Let’s create a manifest file by configuring HTTP GET livenessProbe and experiment with it.
You can configure a livenessProbe by specifying additional parameters like failure threshold, timeout etc., For example, I configured liveness that checks application root path on container port 90.
livenessProbe:
initialDelaySeconds: 10
timeoutSeconds: 1
failureThreshold: 3
periodSeconds: 6
httpGet:
path: /
port: 90
initialDelaySeconds: 10
Start Probing after 10 seconds when the container started. If you do not set initial delay, the container will get probed as soon as it starts.
failureThreshold: 3
Kubernetes try 3 times before giving up and restarting the pod. Giving up in case of liveness probe means restarting the container. In case of readiness probe the Pod will be marked Unready.
periodSeconds: 6
Container is probed for every 6 seconds.
timeoutSeconds: 1
Container must return a response in 1 second, otherwise Probe is counted as failed.
correctLivenessProbe.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: user-service-with-correct-liveness-probe
spec:
selector:
matchLabels:
app: helloworld
replicas: 1
template:
metadata:
labels:
app: helloworld
spec:
containers:
- name: helloworld
image: jboss/wildfly
livenessProbe:
initialDelaySeconds: 10
failureThreshold: 3
timeoutSeconds: 1
httpGet:
path: /
port: 8080
Create deployment using above definition file.
$kubectl create -f correctLibenessProbe.yml
deployment.apps/user-service-with-correct-liveness-probe created
Deployment gets created successfully, lets query Pods.
$kubectl get pods
NAME READY STATUS RESTARTS AGE
user-service-with-correct-liveness-probe-79bdf8bddf-6hfdb 1/1 Running 0 42s
Pod is in Running state. Looks fine.
Let’s misconfigure the livenessProbe by setting it to some port where my application is not running.
misconfiguredLivenessProbe.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: user-service-with-misconfigured-liveness-probe
spec:
selector:
matchLabels:
app: helloworld
replicas: 1
template:
metadata:
labels:
app: helloworld
spec:
containers:
- name: helloworld
image: jboss/wildfly
livenessProbe:
initialDelaySeconds: 10
failureThreshold: 3
timeoutSeconds: 1
httpGet:
path: /
port: 1234
As you see above snippet, I set livenessProbe GET port to 1234, but my application is running on port 8080.
Let’s create deployment from above manifest file.
$kubectl create -f misconfiguredLivenessProbe.yml
deployment.apps/user-service-with-misconfigured-liveness-probe created
Let’s query for Pods.
$kubectl get pods
NAME READY STATUS RESTARTS AGE
user-service-with-correct-liveness-probe-79bdf8bddf-6hfdb 1/1 Running 0 3m53s
user-service-with-misconfigured-liveness-probe-7dd5b9fdd5-z5scw 1/1 Running 0 6s
Wait for 10 seconds (Since I configured initialDelaySeconds as 10 seconds) and query Pods again.
$kubectl get pods
NAME READY STATUS RESTARTS AGE
user-service-with-correct-liveness-probe-79bdf8bddf-6hfdb 1/1 Running 0 4m42s
user-service-with-misconfigured-liveness-probe-7dd5b9fdd5-z5scw 1/1 Running 1 55s
As you see restart count is 1 for the pod ‘user-service-with-misconfigured-liveness-probe-7dd5b9fdd5-z5scw’.
Wait for some more time (more than 20 seconds), you will observe the Kubernetes tries to restart the Pod (user-service-with-misconfigured-liveness-probe-7dd5b9fdd5-z5scw) 3 times and give up. Giving up in case of a liveness probe means restarting the container.
$kubectl get pods
NAME READY STATUS RESTARTS AGE
user-service-with-correct-liveness-probe-79bdf8bddf-6hfdb 1/1 Running 0 8m58s
user-service-with-misconfigured-liveness-probe-7dd5b9fdd5-z5scw 0/1 CrashLoopBackOff 5 5m11s
$
$kubectl get pods
NAME READY STATUS RESTARTS AGE
user-service-with-correct-liveness-probe-79bdf8bddf-6hfdb 1/1 Running 0 9m23s
user-service-with-misconfigured-liveness-probe-7dd5b9fdd5-z5scw 1/1 Running 6 5m36s
How to see the logs of the previous terminated container?
kubectl logs {mypod} --previous
Example
kubectl logs user-service-with-misconfigured-liveness-probe-7dd5b9fdd5-z5scw --previous
No comments:
Post a Comment