crictl cheatsheet
Warning: If you use crictl
to create pod sandboxes or containers on a running Kubernetes cluster, the kubelet
will eventually delete them. crictl
is not a general purpose workflow tool, but a tool that is useful for debugging.
Pods
The biggest difference between crictl
and docker
is that crictl
is aware of Pods.
# list pods
$ crictl pods
# by name
$ crictl pods --name foo-xyz
# list pods by label
$ crictl pods --label component=kube-apiserver
## Get the latest pod
$ crictl pods --latest
npd
(node-problem-detector) uses crictl pods --latest
to determine if containerd
is healthy. If not, npd
will constantly restart it.
Images
# list images
$ crictl images
# list image id (sha256)
$ crictl images -q
sha256:8b8e4416e03730191ab808b229fbddf269dad641360ba583239a051cb01f99ed
sha256:248e85ba70e76d8ee617bcbd6061e9edd4e02e1f519e48cd2731c0dbbcc44b81
# pull image
$ crictl pull busybox
# remove image
$ crictl rmi <image_id>
Containers
# list all containers (including Running and Exited)
$ crictl ps --all
Get inside the container:
# list files
$ crictl exec -i -t 1f73f2d81bf98 ls
# open an interactive shell
$ crictl exec -it 1f73f2d81bf98 sh
Get container log:
# find the container
$ crictl ps -a | grep foo
# get logs
$ crictl logs 87d3992f84f74
# follow the logs
$ crictl logs -f 87d3992f84f74
# get last lines
$ crictl logs --tail=1 87d3992f84f74
Print information about specific containers
$ crictl inspect 87d3992f84f74
Config
Check config
$ cat /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
How to find and kill NotReady Pods
kubelet
maintains a GC mechanism that scans for dead pods and remove them. The GC mechanism runs every 1m. It calls containerd
(essentially like crictli pods
) to get a list of pods that are currently running and issue delete commands.
If during the short time period, there are a lot of pods come and go, the pod sandboxes are kept until the next GC kicked in.
If you see errors like this, maybe there are too many Pod
s (possibily many stuck NotReady
pods), so when it tries to get a list of Pods (ListAllSandboxes
), the size exceeds the grpc limit (in this case 16 MB).
"GenericPLEG: Unable to retrieve pods" err="rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (16794825 vs. 16777216)"
If kubelet
finds containerd
not healthy, it will restart containerd
; if kubelet
keeps restarting containerd
, npd
will report a FrequentContainerdRestart
condition.
crictl rmp
can be used to delete Pod
; a few ways to delete NotReady
pods:
$ crictl rmp $(crictl pods -q --s NotReady)
$ crictl pods -state NotReady -o json | jq -r '.items[].id' | xargs -I% crictl rmp %
$ crictl pods | grep NotReady | cut -f1 -d" " | xargs -L 1 -I {} -t crictl rmp {}
# truncate name and count occurance
$ crictl pods -state NotReady -o json | jq -r '.items[].metadata.name' | cut -c1-20 | sort | uniq -c | sort
If you see the following error, containerd
may be busy, stop creating new Pods first (e.g. scale down the deployments that may create new pods down to 0)
"RemovePodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" podSandboxID="xxxxxxxx"