K8s Troubleshooting - "too many open files"

Check:

$ sysctl fs.inotify.max_user_instances fs.inotify.max_user_watches

# or
$ cat /proc/sys/fs/inotify/max_user_watches
$ cat /proc/sys/fs/inotify/max_user_instances

$ ulimit -n
1024

$ systemctl show xxxxx | grep LimitNOFILE
LimitNOFILE=262144
LimitNOFILESoft=1024

# find pid
$ systemctl status xxxxx

# check open files of the pid (e.g. pid=1964090)
ls "/proc/1964090/fd" | wc -l
1024

To increase temporarily:

$ sudo sysctl fs.inotify.max_user_watches=524288
$ sudo sysctl fs.inotify.max_user_instances=512

To make the changes persistent, edit the file /etc/sysctl.conf and add these lines:

fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 512

Count inotify watches by user

find /proc/*/fd -user "$USER" -lname anon_inode:inotify \
 -printf '%hinfo/%f\n' 2>/dev/null |
xargs cat | grep -c '^inotify'

Count open files:

#!/bin/bash
pids=$(ls -d /proc/[0-9]*)

for p in ${pids}; do
  count=$(ls $p/fd | wc -c)
  if [ "${count}" -gt "150" ]; then
    name=$(cat ${p}/comm)
    echo "${name} has ${count} open files"
  fi
done