0

将工作节点加入集群时,我在工作节点上收到以下错误:

Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.218433   43868 conversion.go:110] Could not get instant cpu stats: different number of cpus
Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.219198   43868 conversion.go:110] Could not get instant cpu stats: different number of cpus
Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.220195   43868 conversion.go:110] Could not get instant cpu stats: different number of cpus
Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.220712   43868 eviction_manager.go:332] eviction manager: attempting to reclaim nodefs
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.220749   43868 helpers.go:1118] eviction manager: attempting to delete unused containers
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.223497   43868 helpers.go:1128] eviction manager: attempting to delete unused images
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.231231   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:b949fa39c923232fc9dc5521d908cc44047d9a951339af5633b057f6a4878dce" to free 274688064 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.234577   43868 remote_image.go:130] RemoveImage "sha256:b949fa39c923232fc9dc5521d908cc44047d9a951339af5633b057f6a4878dce" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.234646   43868 kuberuntime_image.go:126] Remove image "sha256:b949fa39c923232fc9dc5521d908cc44047d9a951339af5633b057f6a4878dce" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.234666   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:30ee439fbdcf8fdd4bb9b517983bcc386651ccfe20c93e3aaeb3f7f040057b79" to free 79601765 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.238566   43868 remote_image.go:130] RemoveImage "sha256:30ee439fbdcf8fdd4bb9b517983bcc386651ccfe20c93e3aaeb3f7f040057b79" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.238605   43868 kuberuntime_image.go:126] Remove image "sha256:30ee439fbdcf8fdd4bb9b517983bcc386651ccfe20c93e3aaeb3f7f040057b79" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.238624   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:5e89f9aa5754594e83665a15d8ff0faee9c5991f48c143067a53a0e8dcbcef8b" to free 946975195 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.251674   43868 remote_image.go:130] RemoveImage "sha256:5e89f9aa5754594e83665a15d8ff0faee9c5991f48c143067a53a0e8dcbcef8b" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.251712   43868 kuberuntime_image.go:126] Remove image "sha256:5e89f9aa5754594e83665a15d8ff0faee9c5991f48c143067a53a0e8dcbcef8b" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.251730   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:f5bc15a74665edd153f657b9b84fda59b02e3840733abf4d23413e9dc011e202" to free 1076462680 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.266988   43868 remote_image.go:130] RemoveImage "sha256:f5bc15a74665edd153f657b9b84fda59b02e3840733abf4d23413e9dc011e202" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.267030   43868 kuberuntime_image.go:126] Remove image "sha256:f5bc15a74665edd153f657b9b84fda59b02e3840733abf4d23413e9dc011e202" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.267048   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:05a3bd381fc2470695a35f230afefd7bf978b566253199c4ae5cc96fafa29b37" to free 1840 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.269050   43868 remote_image.go:130] RemoveImage "sha256:05a3bd381fc2470695a35f230afefd7bf978b566253199c4ae5cc96fafa29b37" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.269078   43868 kuberuntime_image.go:126] Remove image "sha256:05a3bd381fc2470695a35f230afefd7bf978b566253199c4ae5cc96fafa29b37" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.269096   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:637ee73c524b815fcb40b6d4056af8ad92cf4adc1e9ecedbb4f98964fae8c9b1" to free 88610277 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.269742   43868 eviction_manager.go:142] Failed to admit pod kube-proxy-qrl8j_kube-system(a62b5c5b-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.271892   43868 remote_image.go:130] RemoveImage "sha256:637ee73c524b815fcb40b6d4056af8ad92cf4adc1e9ecedbb4f98964fae8c9b1" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.271934   43868 kuberuntime_image.go:126] Remove image "sha256:637ee73c524b815fcb40b6d4056af8ad92cf4adc1e9ecedbb4f98964fae8c9b1" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.271953   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:7b9d24c898ae4ec4882eb8f60f75b9ae3fb753b5e078ecd7b310df8bfbfb9f11" to free 965091970 bytes
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.274876   43868 remote_image.go:130] RemoveImage "sha256:7b9d24c898ae4ec4882eb8f60f75b9ae3fb753b5e078ecd7b310df8bfbfb9f11" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.274910   43868 kuberuntime_image.go:126] Remove image "sha256:7b9d24c898ae4ec4882eb8f60f75b9ae3fb753b5e078ecd7b310df8bfbfb9f11" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.274962   43868 eviction_manager.go:435] eviction manager: unexpected error when attempting to reduce nodefs pressure: wanted to free 9223372036854775807 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist]
Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.274980   43868 eviction_manager.go:346] eviction manager: must evict pod(s) to reclaim nodefs
Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.274995   43868 eviction_manager.go:357] eviction manager: eviction thresholds have been met, but no pods are active to evict
Nov 07 14:39:59 xyz.com kubelet[43868]: W1107 14:39:59.467713   43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-pnmbb_kube-system(a6e253dc-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:00 xyz.com kubelet[43868]: W1107 14:40:00.072307   43868 eviction_manager.go:142] Failed to admit pod kube-proxy-f6rws_kube-system(a73e9d11-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:00 xyz.com kubelet[43868]: W1107 14:40:00.670554   43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-s2pvk_kube-system(a799778d-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:01 xyz.com kubelet[43868]: W1107 14:40:01.268440   43868 eviction_manager.go:142] Failed to admit pod kube-proxy-nln6k_kube-system(a7f4fd2b-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:01 xyz.com kubelet[43868]: W1107 14:40:01.868511   43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-gvnk4_kube-system(a85090f6-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:02 xyz.com kubelet[43868]: W1107 14:40:02.466371   43868 eviction_manager.go:142] Failed to admit pod kube-proxy-s5th5_kube-system(a8abcf73-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:03 xyz.com kubelet[43868]: W1107 14:40:03.068779   43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-qpjf9_kube-system(a9079eef-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:03 xyz.com kubelet[43868]: W1107 14:40:03.668663   43868 eviction_manager.go:142] Failed to admit pod kube-proxy-gbvk7_kube-system(a963390c-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
Nov 07 14:40:04 xyz.com kubelet[43868]: W1107 14:40:04.269624   43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-zzxh8_kube-system(a9bee8fb-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]

此外,当我运行命令kubectl get pods --all-namespaces以获取系统状态时。显示以下内容

NAMESPACE     NAME                                      READY     STATUS    RESTARTS   AGE
kube-system   etcd-xyz.com                              1/1       Running   0          17m
kube-system   kube-apiserver-xyz.com                    1/1       Running   0          16m
kube-system   kube-controller-manager-xyz.com           1/1       Running   0          16m
kube-system   kube-dns-545bc4bfd4-nfhxw                 3/3       Running   0          17m
kube-system   kube-flannel-ds-hwdx8                     0/1       Pending   0          2s
kube-system   kube-flannel-ds-x8ppr                     1/1       Running   0          16m
kube-system   kube-proxy-p65z8                          0/1       Pending   0          1s
kube-system   kube-proxy-t52cr                          1/1       Running   0          17m
kube-system   kube-scheduler-xyz.com                    1/1       Running   0          16m

在这里,我可以看到网络初始化 podkube-flannel-ds-hwdx8kube-proxy-p65z8工作节点上根本没有启动。可能是什么问题。当我在集群中使用 Wea​​ve-net Network 而不是法兰绒时,也会发生类似的事情。以下是我通过运行得到的日志kubectl describe nodes

Events:
  Type     Reason                   Age                 From                       Message
  ----     ------                   ----                ----                       -------
  Normal   Starting                 15m                 kubelet, xyz.com  Starting kubelet.
  Normal   NodeHasSufficientDisk    15m (x2 over 15m)   kubelet, xyz.com  Node xyz.com status is now: NodeHasSufficientDisk
  Normal   NodeHasSufficientMemory  15m (x2 over 15m)   kubelet, xyz.com  Node xyz.com status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    15m (x2 over 15m)   kubelet, xyz.com  Node xyz.com status is now: NodeHasNoDiskPressure
  Normal   NodeAllocatableEnforced  15m                 kubelet, xyz.com  Updated Node Allocatable limit across pods
  Normal   NodeHasDiskPressure      15m                 kubelet, xyz.com  Node xyz.com status is now: NodeHasDiskPressure
  Normal   NodeReady                15m                 kubelet, xyz.com  Node xyz.com status is now: NodeReady
  Warning  EvictionThresholdMet     11m (x15 over 15m)  kubelet, xyz.com  Attempting to reclaim nodefs
  Warning  ImageGCFailed            10m                 kubelet, xyz.com  wanted to free 6135907123 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist]
4

0 回答 0