将工作节点加入集群时,我在工作节点上收到以下错误:
Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.218433 43868 conversion.go:110] Could not get instant cpu stats: different number of cpus
Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.219198 43868 conversion.go:110] Could not get instant cpu stats: different number of cpus
Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.220195 43868 conversion.go:110] Could not get instant cpu stats: different number of cpus
Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.220712 43868 eviction_manager.go:332] eviction manager: attempting to reclaim nodefs
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.220749 43868 helpers.go:1118] eviction manager: attempting to delete unused containers
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.223497 43868 helpers.go:1128] eviction manager: attempting to delete unused images
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.231231 43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:b949fa39c923232fc9dc5521d908cc44047d9a951339af5633b057f6a4878dce" to free 274688064 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.234577 43868 remote_image.go:130] RemoveImage "sha256:b949fa39c923232fc9dc5521d908cc44047d9a951339af5633b057f6a4878dce" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.234646 43868 kuberuntime_image.go:126] Remove image "sha256:b949fa39c923232fc9dc5521d908cc44047d9a951339af5633b057f6a4878dce" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.234666 43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:30ee439fbdcf8fdd4bb9b517983bcc386651ccfe20c93e3aaeb3f7f040057b79" to free 79601765 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.238566 43868 remote_image.go:130] RemoveImage "sha256:30ee439fbdcf8fdd4bb9b517983bcc386651ccfe20c93e3aaeb3f7f040057b79" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.238605 43868 kuberuntime_image.go:126] Remove image "sha256:30ee439fbdcf8fdd4bb9b517983bcc386651ccfe20c93e3aaeb3f7f040057b79" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.238624 43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:5e89f9aa5754594e83665a15d8ff0faee9c5991f48c143067a53a0e8dcbcef8b" to free 946975195 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.251674 43868 remote_image.go:130] RemoveImage "sha256:5e89f9aa5754594e83665a15d8ff0faee9c5991f48c143067a53a0e8dcbcef8b" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.251712 43868 kuberuntime_image.go:126] Remove image "sha256:5e89f9aa5754594e83665a15d8ff0faee9c5991f48c143067a53a0e8dcbcef8b" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.251730 43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:f5bc15a74665edd153f657b9b84fda59b02e3840733abf4d23413e9dc011e202" to free 1076462680 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.266988 43868 remote_image.go:130] RemoveImage "sha256:f5bc15a74665edd153f657b9b84fda59b02e3840733abf4d23413e9dc011e202" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.267030 43868 kuberuntime_image.go:126] Remove image "sha256:f5bc15a74665edd153f657b9b84fda59b02e3840733abf4d23413e9dc011e202" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.267048 43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:05a3bd381fc2470695a35f230afefd7bf978b566253199c4ae5cc96fafa29b37" to free 1840 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.269050 43868 remote_image.go:130] RemoveImage "sha256:05a3bd381fc2470695a35f230afefd7bf978b566253199c4ae5cc96fafa29b37" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.269078 43868 kuberuntime_image.go:126] Remove image "sha256:05a3bd381fc2470695a35f230afefd7bf978b566253199c4ae5cc96fafa29b37" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.269096 43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:637ee73c524b815fcb40b6d4056af8ad92cf4adc1e9ecedbb4f98964fae8c9b1" to free 88610277 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.269742 43868 eviction_manager.go:142] Failed to admit pod kube-proxy-qrl8j_kube-system(a62b5c5b-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.271892 43868 remote_image.go:130] RemoveImage "sha256:637ee73c524b815fcb40b6d4056af8ad92cf4adc1e9ecedbb4f98964fae8c9b1" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.271934 43868 kuberuntime_image.go:126] Remove image "sha256:637ee73c524b815fcb40b6d4056af8ad92cf4adc1e9ecedbb4f98964fae8c9b1" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.271953 43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:7b9d24c898ae4ec4882eb8f60f75b9ae3fb753b5e078ecd7b310df8bfbfb9f11" to free 965091970 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.274876 43868 remote_image.go:130] RemoveImage "sha256:7b9d24c898ae4ec4882eb8f60f75b9ae3fb753b5e078ecd7b310df8bfbfb9f11" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.274910 43868 kuberuntime_image.go:126] Remove image "sha256:7b9d24c898ae4ec4882eb8f60f75b9ae3fb753b5e078ecd7b310df8bfbfb9f11" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.274962 43868 eviction_manager.go:435] eviction manager: unexpected error when attempting to reduce nodefs pressure: wanted to free 9223372036854775807 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist]
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.274980 43868 eviction_manager.go:346] eviction manager: must evict pod(s) to reclaim nodefs
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.274995 43868 eviction_manager.go:357] eviction manager: eviction thresholds have been met, but no pods are active to evict
Nov 07 14:39:59 xyz.com kubelet[43868]: W1107 14:39:59.467713 43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-pnmbb_kube-system(a6e253dc-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:00 xyz.com kubelet[43868]: W1107 14:40:00.072307 43868 eviction_manager.go:142] Failed to admit pod kube-proxy-f6rws_kube-system(a73e9d11-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:00 xyz.com kubelet[43868]: W1107 14:40:00.670554 43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-s2pvk_kube-system(a799778d-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:01 xyz.com kubelet[43868]: W1107 14:40:01.268440 43868 eviction_manager.go:142] Failed to admit pod kube-proxy-nln6k_kube-system(a7f4fd2b-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:01 xyz.com kubelet[43868]: W1107 14:40:01.868511 43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-gvnk4_kube-system(a85090f6-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:02 xyz.com kubelet[43868]: W1107 14:40:02.466371 43868 eviction_manager.go:142] Failed to admit pod kube-proxy-s5th5_kube-system(a8abcf73-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:03 xyz.com kubelet[43868]: W1107 14:40:03.068779 43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-qpjf9_kube-system(a9079eef-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:03 xyz.com kubelet[43868]: W1107 14:40:03.668663 43868 eviction_manager.go:142] Failed to admit pod kube-proxy-gbvk7_kube-system(a963390c-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:04 xyz.com kubelet[43868]: W1107 14:40:04.269624 43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-zzxh8_kube-system(a9bee8fb-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
此外,当我运行命令kubectl get pods --all-namespaces
以获取系统状态时。显示以下内容
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-xyz.com 1/1 Running 0 17m
kube-system kube-apiserver-xyz.com 1/1 Running 0 16m
kube-system kube-controller-manager-xyz.com 1/1 Running 0 16m
kube-system kube-dns-545bc4bfd4-nfhxw 3/3 Running 0 17m
kube-system kube-flannel-ds-hwdx8 0/1 Pending 0 2s
kube-system kube-flannel-ds-x8ppr 1/1 Running 0 16m
kube-system kube-proxy-p65z8 0/1 Pending 0 1s
kube-system kube-proxy-t52cr 1/1 Running 0 17m
kube-system kube-scheduler-xyz.com 1/1 Running 0 16m
在这里,我可以看到网络初始化 podkube-flannel-ds-hwdx8
和kube-proxy-p65z8
工作节点上根本没有启动。可能是什么问题。当我在集群中使用 Weave-net Network 而不是法兰绒时,也会发生类似的事情。以下是我通过运行得到的日志kubectl describe nodes
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 15m kubelet, xyz.com Starting kubelet.
Normal NodeHasSufficientDisk 15m (x2 over 15m) kubelet, xyz.com Node xyz.com status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 15m (x2 over 15m) kubelet, xyz.com Node xyz.com status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 15m (x2 over 15m) kubelet, xyz.com Node xyz.com status is now: NodeHasNoDiskPressure
Normal NodeAllocatableEnforced 15m kubelet, xyz.com Updated Node Allocatable limit across pods
Normal NodeHasDiskPressure 15m kubelet, xyz.com Node xyz.com status is now: NodeHasDiskPressure
Normal NodeReady 15m kubelet, xyz.com Node xyz.com status is now: NodeReady
Warning EvictionThresholdMet 11m (x15 over 15m) kubelet, xyz.com Attempting to reclaim nodefs
Warning ImageGCFailed 10m kubelet, xyz.com wanted to free 6135907123 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist]