我是 Kubernetes 的新手。我使用 kubeadm 和 virtualbox 建立了一个包含 1 个 master 和 2 个 worker（worker1、worker2）的本地集群。我选择了 containerd 作为我的 Container Runtime。
我正面临一个让我发疯的网络问题。
我无法从 pod ping 任何外部地址，因为 DNS 无法解析
我使用以下内容来设置集群：
kubeadm init --apiserver-advertise-address=10.16.10.10 --apiserver-cert-extra-sans=10.16.10.10 --node-name=master0 --pod-network-cidr=10.244.0.0/16
交换和 SELinux 被禁用。
我用的是法兰绒。
[masterk8s@master0 .kube]$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master0 Ready control-plane,master 3h26m v1.23.1 10.16.10.10 <none> CentOS Linux 7 (Core) 3.10.0-1160.49.1.el7.x86_64 containerd://1.4.12
worker1 Ready <none> 169m v1.23.1 10.16.10.11 <none> CentOS Linux 7 (Core) 3.10.0-1160.49.1.el7.x86_64 containerd://1.4.12
worker2 Ready <none> 161m v1.23.1 10.16.10.12 <none> CentOS Linux 7 (Core) 3.10.0-1160.49.1.el7.x86_64 containerd://1.4.12
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default pod/dnsutils 1/1 Running 1 (59m ago) 119m 10.244.3.2 worker1 <none> <none>
default pod/nginx 1/1 Running 0 11s 10.244.4.2 worker2 <none> <none>
kube-system pod/coredns-64897985d-lnzs7 1/1 Running 0 126m 10.244.0.2 master0 <none> <none>
kube-system pod/coredns-64897985d-vfngl 1/1 Running 0 126m 10.244.0.3 master0 <none> <none>
kube-system pod/etcd-master0 1/1 Running 1 (125m ago) 126m 10.16.10.10 master0 <none> <none>
kube-system pod/kube-apiserver-master0 1/1 Running 1 (125m ago) 126m 10.16.10.10 master0 <none> <none>
kube-system pod/kube-controller-manager-master0 1/1 Running 1 (125m ago) 126m 10.16.10.10 master0 <none> <none>
kube-system pod/kube-flannel-ds-6g4dm 1/1 Running 0 81m 10.16.10.12 worker2 <none> <none>
kube-system pod/kube-flannel-ds-lvgpf 1/1 Running 0 89m 10.16.10.11 worker1 <none> <none>
kube-system pod/kube-flannel-ds-pkm4k 1/1 Running 1 (125m ago) 126m 10.16.10.10 master0 <none> <none>
kube-system pod/kube-proxy-8gnfx 1/1 Running 0 89m 10.16.10.11 worker1 <none> <none>
kube-system pod/kube-proxy-cbws6 1/1 Running 0 81m 10.16.10.12 worker2 <none> <none>
kube-system pod/kube-proxy-fxvm5 1/1 Running 1 (125m ago) 126m 10.16.10.10 master0 <none> <none>
kube-system pod/kube-scheduler-master0 1/1 Running 1 (125m ago) 126m 10.16.10.10 master0 <none> <none>
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 126m <none>
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 126m k8s-app=kube-dns
猫 /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
掌握：
[masterk8s@master0 .kube]$ ip r
default via 10.0.2.2 dev enp0s3
default via 10.16.10.1 dev enp0s9 proto static metric 102
10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15 metric 100
10.16.10.0/24 dev enp0s9 proto kernel scope link src 10.16.10.10 metric 102
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
10.244.3.0/24 via 10.244.3.0 dev flannel.1 onlink
10.244.4.0/24 via 10.244.4.0 dev flannel.1 onlink
192.168.56.0/24 dev enp0s8 proto kernel scope link src 192.168.56.100 metric 101
工人1：
[workerk8s@worker1 ~]$ ip r
default via 10.0.2.2 dev enp0s3 proto dhcp metric 100
default via 10.16.10.1 dev enp0s9 proto static metric 102
10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15 metric 100
10.16.10.0/24 dev enp0s9 proto kernel scope link src 10.16.10.11 metric 102
10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink
10.244.3.0/24 dev cni0 proto kernel scope link src 10.244.3.1
10.244.4.0/24 via 10.244.4.0 dev flannel.1 onlink
192.168.56.0/24 dev enp0s8 proto kernel scope link src 192.168.56.101 metric 101
我可以从 master 访问 kube-dns cluster-IP：
[masterk8s@master0 .kube]$ telnet 10.96.0.10 53
Trying 10.96.0.10...
Connected to 10.96.0.10.
Escape character is '^]'.
但不能来自工人：
[workerk8s@worker1 ~]$ telnet 10.96.0.10 53
Trying 10.96.0.10...
^C
我使用来自 kubernetes ( https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ ) 的 dnsutils pod 进行了一些测试：
（此 pod 已部署在 worker1 上，但 worker2 的问题相同）
[masterk8s@master0 .kube]$ kubectl exec -i -t dnsutils -- nslookup kubernetes.default
^C
command terminated with exit code 1
[masterk8s@master0 .kube]$ kubectl exec -i -t dnsutils -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local Home
nameserver 10.96.0.10
options ndots:5
节点之间有联系。但是不同节点上的 Pod 不能互相 ping 通。例子：
default pod/dnsutils 1/1 Running 1 (59m ago) 119m 10.244.3.2 worker1 <none> <none>
default pod/nginx 1/1 Running 0 11s 10.244.4.2 worker2 <none> <none>
10.244.3.2 只能从 worker1 访问，10.224.4.2 只能从 worker2 访问。
我的猜测是 kube-proxy 有问题，但不知道可能是什么。
我在 pod 日志中看不到任何错误。
有什么建议么？
谢谢
编辑：
解决了
Flannel 使用了错误的接口，因为我的节点有 3 个网络接口，我用 --iface 指定了正确的接口
name: kube-flannel
image: quay.io/coreos/flannel:v0.15.1
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=enp0s9
还意识到firewalld正在阻止对DNS的请求，并解决了添加问题（如何在不禁用firewalld（Kubernetes）的情况下使用Flannel）：
firewall-cmd --add-masquerade --permanent