版本:在 GCE 中运行的Kubernetes v1.9.2
(kube-up.sh)
容器优化操作系统10032.88.0
症状:google-ip-forwarding-daemon.service
我们的 COS 节点在其日志中显示以下内容:
Mar 07 13:43:28 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.154.243.135'].
Mar 07 13:43:45 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.198.28.188'].
Mar 07 13:44:01 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.171.xxx'].
Mar 07 13:44:17 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'130.211.120.8'].
Mar 07 13:44:37 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.198.52.xxx'].
Mar 07 13:44:53 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.17.xx'].
Mar 07 13:45:10 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.155.181.xx'].
Mar 07 13:45:26 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.255.xxx'].
Mar 07 13:45:43 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.166.xxx'].
Mar 07 13:45:59 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] by adding None and removing [u'146.148.61.xx'].
Mar 07 13:46:19 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.197.104.xxx'].
Mar 07 13:46:41 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.198.172.xxx'].
Mar 07 13:46:58 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.198.162.xxx'].
Mar 07 13:47:14 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.155.131.xxx'].
Mar 07 13:47:34 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'130.211.238.xxx'] by adding None and removing [u'104.198.131.xxx'].
Mar 07 13:47:53 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'130.211.238.xxx'] by adding None and removing [u'104.154.72.xxx'].
Mar 07 13:48:10 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'130.211.238.xxx'] to [u'130.211.238.xxx'] by adding None and removing [u'104.154.20.xx'].
Mar 07 13:48:26 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'130.211.238.xxx'] to None by adding None and removing [u'130.211.238.xxx'].
发生这种情况时,正在运行的 POD 通常会继续运行,但有时会变为未知并重新安排。任何新调度的 pod 都会卡在 ContainerCreating 中,并出现以下错误:
Failed create pod sandbox.
&Error response from daemon: network none not found
除了重新启动实例外,我们还没有找到解决此症状的方法。