Kubernetes - Logging

Native node-level logging

Pod Logs

Kubernetes pod logs are saved in the /var/log/pods folder on the host.

Logs can be retrieved by kubectl logs.

The kubelet tells the container runtime (using CRI) where to write the container logs.

Log Rotation

What is "Log Rotation"? When the log file reaches a certain threshold (file size, age, number of records), the log file may be renamed, and a new file with the original name is created.

kubectl logs can only access the latest log. The kubelet is responsible for rotating container logs and managing the logging directory structure.

Audit Logs

Audit logs are disabled by default in Kubernetes.

To check if it is enabled, find your kube-apiserver- pod, and check .spec.containers[].command, if you see this, it is enabled:

--audit-log-path=/var/log/apiserver/audit.log
--audit-policy-file=/etc/kubernetes/audit-policy.yaml

If you do not find those and want to enable audit logging, add the flags when starting the api server.

The default audit log path is /var/log/kubernetes/kube-apiserver-audit.log, but it can be different depending on the k8s distro.

systemd Logs

Mostly likely kubelet and containerd (or other runtime) run as systemd services, so to check their logs, use journalctl -u kubelet.

Depending on your setup, binaries like kube-apiserver, kube-controller-manager may run as static pods or systemd services, which means you need to check either pod logs or systemd logs.

Cluster-level logging

Logs have a separate storage and lifecycle independent of nodes, pods, or containers.

Kubernetes does not provide a native solution, but can use some addons.

A few options to implement:

• a node-level logging agent (usually a DaemonSet, so it is running on all the nodes) that pushes logs to the logging backend.
• a sidecar container that streams logs.