kubelet.service: Unit entered failed state in not ready state node error from kubernetes cluster
I am trying to deploy an springboot microservices in kubernetes cluster having 1 master and 2 worker node. When I am trying to get the node state using the command sudo kubectl get nodes , I am getting one of my worker node is not ready. It showing not ready in status. When I am applying to troubleshoot the following command,
sudo journalctl -u kubelet I am getting response like kubelet.service: Unit entered failed state and kubelet service stopped. The following is the response what I am getting when applying the command sudo journalctl -u kubelet .
-- Logs begin at Fri 2020-01-03 04:56:18 EST, end at Fri 2020-01-03 05:32:47 EST. -- Jan 03 04:56:25 MILDEVKUB050 systemd[1]: Started kubelet: The Kubernetes Node Agent. Jan 03 04:56:31 MILDEVKUB050 kubelet[970]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --confi Jan 03 04:56:31 MILDEVKUB050 kubelet[970]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --confi Jan 03 04:56:32 MILDEVKUB050 kubelet[970]: I0103 04:56:32.053962 970 server.go:416] Version: v1.17.0 Jan 03 04:56:32 MILDEVKUB050 kubelet[970]: I0103 04:56:32.084061 970 plugins.go:100] No cloud provider specified. Jan 03 04:56:32 MILDEVKUB050 kubelet[970]: I0103 04:56:32.235928 970 server.go:821] Client rotation is on, will bootstrap in background Jan 03 04:56:32 MILDEVKUB050 kubelet[970]: I0103 04:56:32.280173 970 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-curre Jan 03 04:56:38 MILDEVKUB050 kubelet[970]: I0103 04:56:38.107966 970 server.go:641] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to / Jan 03 04:56:38 MILDEVKUB050 kubelet[970]: F0103 04:56:38.109401 970 server.go:273] failed to run Kubelet: running with swap on is not supported, please disable swa Jan 03 04:56:38 MILDEVKUB050 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a Jan 03 04:56:38 MILDEVKUB050 systemd[1]: kubelet.service: Unit entered failed state. Jan 03 04:56:38 MILDEVKUB050 systemd[1]: kubelet.service: Failed with result 'exit-code'. Jan 03 04:56:48 MILDEVKUB050 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart. Jan 03 04:56:48 MILDEVKUB050 systemd[1]: Stopped kubelet: The Kubernetes Node Agent. Jan 03 04:56:48 MILDEVKUB050 systemd[1]: Started kubelet: The Kubernetes Node Agent. Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: I0103 04:56:48.901632 1433 server.go:416] Version: v1.17.0 Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: I0103 04:56:48.907654 1433 plugins.go:100] No cloud provider specified. Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: I0103 04:56:48.907806 1433 server.go:821] Client rotation is on, will bootstrap in background Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: I0103 04:56:48.947107 1433 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-curr Jan 03 04:56:49 MILDEVKUB050 kubelet[1433]: I0103 04:56:49.263777 1433 server.go:641] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to Jan 03 04:56:49 MILDEVKUB050 kubelet[1433]: F0103 04:56:49.264219 1433 server.go:273] failed to run Kubelet: running with swap on is not supported, please disable sw Jan 03 04:56:49 MILDEVKUB050 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a Jan 03 04:56:49 MILDEVKUB050 systemd[1]: kubelet.service: Unit entered failed state. Jan 03 04:56:49 MILDEVKUB050 systemd[1]: kubelet.service: Failed with result 'exit-code'. Jan 03 04:56:59 MILDEVKUB050 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart. Jan 03 04:56:59 MILDEVKUB050 systemd[1]: Stopped kubelet: The Kubernetes Node Agent. Jan 03 04:56:59 MILDEVKUB050 systemd[1]: Started kubelet: The Kubernetes Node Agent. Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.712729 1500 server.go:416] Version: v1.17.0 Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.714927 1500 plugins.go:100] No cloud provider specified. Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.715248 1500 server.go:821] Client rotation is on, will bootstrap in background Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.763508 1500 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-curr Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.956706 1500 server.go:641] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: F0103 04:56:59.957078 1500 server.go:273] failed to run Kubelet: running with swap on is not supported, please disable sw Jan 03 04:56:59 MILDEVKUB050 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a Jan 03 04:56:59 MILDEVKUB050 systemd[1]: kubelet.service: Unit entered failed state. Jan 03 04:56:59 MILDEVKUB050 systemd[1]: kubelet.service: Failed with result 'exit-code'. Jan 03 04:57:10 MILDEVKUB050 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart. Jan 03 04:57:10 MILDEVKUB050 systemd[1]: Stopped kubelet: The Kubernetes Node Agent. Jan 03 04:57:10 MILDEVKUB050 systemd[1]: Started kubelet: The Kubernetes Node Agent. log file: service: Unit entered failed state I tried by restarting the kubelet. But still there is no change in node state. Not ready state only. Updates When I am trying the command systemctl list-units —type=swap —state=active , then I am getting the following response,
docker@MILDEVKUB040:~$ systemctl list-units --type=swap --state=active UNIT LOAD ACTIVE SUB DESCRIPTION dev-mapper-MILDEVDCR01\x2d\x2dvg\x2dswap_1.swap loaded active active /dev/mapper/MILDEVDCR01--vg-swap_1 Important When I am getting these kind of issue with node not ready, each time I need to disable the swap and need to reload the daemon and kubelet. After that node becomes ready state. And again I need to repeat the same. How can I find a permanent solution for this?
Аудит systemd: почему не запускаются юниты?
Jan 28, 2016 17:27 · 431 words · 3 minute read debian ubuntu systemd
Systemd — альтернативный менеджер служб для Linux, совместимый со скриптами инициализации LSB и SysV .
По сравнению с System V init обладает некоторыми преимуществами и дополнительными возможностями, по умолчанию установлен в ОС Debian 8 (Jessie). Давайте разберемся, что делать, если не запускаются юниты!
При старте системы юниты должны запускаться в определенном порядке. Если по какой-либо причине юнит не может запуститься, это может повлечь за собой некорректный старт остальных юнитов и всей системы.
Примечание. C точки зрения systemd юнит это — служба ( .service ), точка монтирования ( .mount ), устройство ( .device ) или сокет ( .socket ).
Наиболее частые проблемы с запуском юнитов — отсутствие необходимых зависимостей (например точки монтирования), настроек или неправильные права доступа.
Список юнитов смотрим так:
systemctl UNIT LOAD ACTIVE SUB DESCRIPTION -.mount loaded active mounted / boot.mount loaded active mounted /boot dev-hugepages.mount loaded active mounted Huge Pages File System ● dev-mqueue.mount loaded failed failed POSIX Message Queue File System tmp.mount loaded active mounted Temporary Directory mnt-exchange.mount loaded active mounted /mnt/exchange proc-fs-nfsd.mount loaded active mounted /proc/fs/nfsd usr-local-backup.mount loaded active mounted /usr/local/backup var-lib-mysql.mount loaded active mounted /var/lib/mysql var-www.mount loaded active mounted /var/www . ● sphinxsearch.service loaded failed failed LSB: Fast standalone full-text SQL search engine Список проблемных юнитов смотрим так:
systemctl --failed ● dev-mqueue.mount loaded failed failed POSIX Message Queue File System ● sphinxsearch.service loaded failed failed LSB: Fast standalone full-text SQL search engine Узнать подробности о незапущенном юните можно так:
systemctl status dev-mqueue.mount ● dev-mqueue.mount - POSIX Message Queue File System Loaded: loaded (/usr/lib/systemd/system/dev-mqueue.mount; static) Active: failed (Result: exit-code) since Sun 2015-11-23 17:53:10 CET; 4min 12s ago Where: /dev/mqueue What: mqueue Docs: man:mq_overview(7) http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems Process: 446 ExecMount=/bin/mount -n mqueue /dev/mqueue -t mqueue (code=exited, status=32) Nov 23 17:53:10 localhost.localdomain systemd[1]: dev-mqueue.mount mount process exited, code=exited status=32 Nov 23 17:53:10 localhost.localdomain systemd[1]: Failed to mount POSIX Message Queue File System. Nov 23 17:53:10 localhost.localdomain systemd[1]: Unit dev-mqueue.mount entered failed state. Из этого примера видно, что устройство /dev/mqueue отсутствует в системе.
systemctl status sphinxsearch.service ● sphinxsearch.service - LSB: Fast standalone full-text SQL search engine Loaded: loaded (/etc/init.d/sphinxsearch) Active: failed (Result: exit-code) since Sun 2015-11-23 17:51:17 CET; 6min 5s ago Process: 568 ExecStart=/etc/init.d/sphinxsearch start (code=exited, status=1/FAILURE) Nov 23 17:51:17 astra sphinxsearch[568]: Starting sphinxsearch: /usr/bin/searchd already running. Nov 23 17:51:17 astra systemd[1]: sphinxsearch.service: control process exited, code=exited status=1 Nov 23 17:51:17 astra systemd[1]: Failed to start LSB: Fast standalone full-text SQL search engine. Nov 23 17:51:17 astra systemd[1]: Unit sphinxsearch.service entered failed state. А данный пример — это пока не решенный баг sphinxsearch .
Также для выяснения проблем с запуском юнитов можно изучить логи с помощью команды:
journalctl _PID=идентификатор процесса После того, как проблема будет найдена, устранить ее будет несложно.