我正在运行 skaffold dev 并且 minikube 一直卡住。
我可以停止并启动 minikube，它通常会恢复。
警告 ContainerGCFailed 是问题，还是我应该调查：
PLEG is not healthy: pleg was last seen active 4m19.322678359s ago; threshold is 3m0s
$ kubectl describe nodes
Name: minikube
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=minikube
kubernetes.io/os=linux
minikube.k8s.io/commit=23f40a012abb52eff365ff99a709501a61ac5876
minikube.k8s.io/name=minikube
minikube.k8s.io/updated_at=2020_11_18T14_29_40_0700
minikube.k8s.io/version=v1.15.1
node-role.kubernetes.io/master=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 18 Nov 2020 14:29:37 +1100
Taints: node.kubernetes.io/not-ready:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: minikube
AcquireTime: <unset>
RenewTime: Wed, 18 Nov 2020 14:37:01 +1100
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Wed, 18 Nov 2020 14:36:56 +1100 Wed, 18 Nov 2020 14:29:35 +1100 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 18 Nov 2020 14:36:56 +1100 Wed, 18 Nov 2020 14:29:35 +1100 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 18 Nov 2020 14:36:56 +1100 Wed, 18 Nov 2020 14:29:35 +1100 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Wed, 18 Nov 2020 14:36:56 +1100 Wed, 18 Nov 2020 14:35:45 +1100 KubeletNotReady PLEG is not healthy: pleg was last seen active 4m19.322678359s ago; threshold is 3m0s
Addresses:
InternalIP: 192.168.99.103
Hostname: minikube
Capacity:
cpu: 4
ephemeral-storage: 14777356Ki
hugepages-2Mi: 0
memory: 16419284Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 14777356Ki
hugepages-2Mi: 0
memory: 16419284Ki
pods: 110
System Info:
Kernel Version: 4.19.150
OS Image: Buildroot 2020.02.7
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.13
Kubelet Version: v1.19.4
Kube-Proxy Version: v1.19.4
Non-terminated Pods: (22 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default aerospike-aerospike-enterprise-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6m6s
default confluent-platform-cp-kafka-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6m8s
default confluent-platform-cp-kafka-1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6m6s
default confluent-platform-cp-kafka-2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5m51s
default confluent-platform-cp-kafka-connect-57b789f97-sc4nf 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6m8s
default confluent-platform-cp-zookeeper-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6m8s
default dnsutils 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5m59s
default redis-master-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6m3s
default redis-slave-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6m3s
default redis-slave-1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5m25s
default tellus-api-gateway-787f7574bd-s5pbm 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5m59s
default tellus-config-server-5f666967bc-s5dvq 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5m59s
default tellus-gateway-message-service-789766f9c8-6ws7w 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5m59s
default tellus-query-5954799f46-lgq9z 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5m58s
default tellus-web-64c6cc884b-smv2s 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5m58s
kube-system coredns-f9fd979d6-jclvd 100m (2%) 0 (0%) 70Mi (0%) 170Mi (1%) 7m18s
kube-system etcd-minikube 0 (0%) 0 (0%) 0 (0%) 0 (0%) 7m24s
kube-system kube-apiserver-minikube 250m (6%) 0 (0%) 0 (0%) 0 (0%) 7m24s
kube-system kube-controller-manager-minikube 200m (5%) 0 (0%) 0 (0%) 0 (0%) 7m24s
kube-system kube-proxy-pgh7p 0 (0%) 0 (0%) 0 (0%) 0 (0%) 7m18s
kube-system kube-scheduler-minikube 100m (2%) 0 (0%) 0 (0%) 0 (0%) 7m24s
kube-system storage-provisioner 0 (0%) 0 (0%) 0 (0%) 0 (0%) 7m24s
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 650m (16%) 0 (0%)
memory 70Mi (0%) 170Mi (1%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeHasSufficientMemory 7m33s (x5 over 7m34s) kubelet Node minikube status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 7m33s (x4 over 7m34s) kubelet Node minikube status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 7m33s (x4 over 7m34s) kubelet Node minikube status is now: NodeHasSufficientPID
Normal NodeHasNoDiskPressure 7m24s kubelet Node minikube status is now: NodeHasNoDiskPressure
Normal NodeAllocatableEnforced 7m24s kubelet Updated Node Allocatable limit across pods
Normal NodeHasSufficientMemory 7m24s kubelet Node minikube status is now: NodeHasSufficientMemory
Normal Starting 7m24s kubelet Starting kubelet.
Normal NodeHasSufficientPID 7m24s kubelet Node minikube status is now: NodeHasSufficientPID
Normal NodeReady 7m24s kubelet Node minikube status is now: NodeReady
Normal Starting 7m17s kube-proxy Starting kube-proxy.
Warning ContainerGCFailed 4m22s kubelet rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (17408575 vs. 16777216)
Normal NodeNotReady 80s kubelet Node minikube status is now: NodeNotReady
Warning ContainerGCFailed 19s (x4 over 3m21s) kubelet rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (31323462 vs. 16777216)