kubernetes - iptables 模式下的 kube-proxy 无法正常工作

Question

我有

Kubernetes：v.1.1.1
iptables v1.4.21
内核：4.2.0-18-generic，Ubuntu 自带
网络通过在交换机上终止的 L2 VLAN 完成
没有云提供商

我所做的

我正在尝试使用 kube-proxy 的 iptables 模式。我用--proxy_mode=iptables参数启用了它。似乎缺少一些规则：

iptables -t nat -nvL

Chain PREROUTING (policy ACCEPT 8 packets, 459 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2116  120K KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 2 packets, 120 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  718 45203 KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain POSTROUTING (policy ACCEPT 5 packets, 339 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4d415351

Chain KUBE-NODEPORTS (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/docker-registry-fe:tcp */ tcp dpt:31195 MARK set 0x4d415351
    0     0 KUBE-SVC-XZFGDLM7GMJHZHOY  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/docker-registry-fe:tcp */ tcp dpt:31195
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* mngbox/jumpbox:ssh */ tcp dpt:30873 MARK set 0x4d415351
    0     0 KUBE-SVC-GLKZVFIDXOFHLJLC  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* mngbox/jumpbox:ssh */ tcp dpt:30873

Chain KUBE-SEP-5IXMK7UWPGVTWOJ7 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.160.8         0.0.0.0/0            /* mngbox/jumpbox:ssh */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* mngbox/jumpbox:ssh */ tcp to:10.116.160.8:22

Chain KUBE-SEP-BNPLX5HQYOZINWEQ (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.161.6         0.0.0.0/0            /* kube-system/monitoring-influxdb:api */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-influxdb:api */ tcp to:10.116.161.6:8086

Chain KUBE-SEP-CJMHKLXPTJLTE3OP (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.254.2         0.0.0.0/0            /* default/kubernetes: */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes: */ tcp to:10.116.254.2:6443


Chain KUBE-SEP-GSM3BZTEXEBWDXPN (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.160.7         0.0.0.0/0            /* kube-system/kube-dns:dns */ MARK set 0x4d415351
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:10.116.160.7:53


Chain KUBE-SEP-OAYOAJINXRPUQDA3 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.160.7         0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:10.116.160.7:53

Chain KUBE-SEP-PJJZDQNXDGWM7MU6 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.160.5         0.0.0.0/0            /* default/docker-registry-fe:tcp */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/docker-registry-fe:tcp */ tcp to:10.116.160.5:443

Chain KUBE-SEP-RWODGLKOVWXGOHUR (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.161.6         0.0.0.0/0            /* kube-system/monitoring-influxdb:http */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-influxdb:http */ tcp to:10.116.161.6:8083

Chain KUBE-SEP-WE3Z7KMHA6KPJWKK (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.161.6         0.0.0.0/0            /* kube-system/monitoring-grafana: */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-grafana: */ tcp to:10.116.161.6:8080

Chain KUBE-SEP-YBQVM4LA4YMMZIWH (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.161.3         0.0.0.0/0            /* kube-system/monitoring-heapster: */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-heapster: */ tcp to:10.116.161.3:8082

Chain KUBE-SEP-YMZS7BLP4Y6MWTX5 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.160.9         0.0.0.0/0            /* infra/docker-registry-backend:docker-registry-backend */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* infra/docker-registry-backend:docker-registry-backend */ tcp to:10.116.160.9:5000

Chain KUBE-SEP-ZDOOYAKDERKR43R3 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.160.10        0.0.0.0/0            /* default/kibana-logging: */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kibana-logging: */ tcp to:10.116.160.10:5601

Chain KUBE-SERVICES (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SVC-JRXTEHDDTAFMSEAS  tcp  --  *      *       0.0.0.0/0            10.116.0.48          /* kube-system/monitoring-grafana: cluster IP */ tcp dpt:80
    0     0 KUBE-SVC-CK6HVV5A27TDFNIA  tcp  --  *      *       0.0.0.0/0            10.116.0.188         /* kube-system/monitoring-influxdb:api cluster IP */ tcp dpt:8086
    0     0 KUBE-SVC-DKEW3YDJFV3YJLS2  tcp  --  *      *       0.0.0.0/0            10.116.0.6           /* infra/docker-registry-backend:docker-registry-backend cluster IP */ tcp dpt:5000
    0     0 KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  *      *       0.0.0.0/0            10.116.0.2           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
    0     0 KUBE-SVC-WEHLQ23XZWSA5ZX3  tcp  --  *      *       0.0.0.0/0            10.116.0.188         /* kube-system/monitoring-influxdb:http cluster IP */ tcp dpt:8083
    0     0 KUBE-SVC-XZFGDLM7GMJHZHOY  tcp  --  *      *       0.0.0.0/0            10.116.1.142         /* default/docker-registry-fe:tcp cluster IP */ tcp dpt:443
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 MARK set 0x4d415351
    0     0 KUBE-SVC-XZFGDLM7GMJHZHOY  tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
    0     0 KUBE-SVC-XZFGDLM7GMJHZHOY  tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 ADDRTYPE match dst-type LOCAL
    0     0 KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  *      *       0.0.0.0/0            10.116.0.2           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
    0     0 KUBE-SVC-7IHGTXJ4CF2KVXJZ  tcp  --  *      *       0.0.0.0/0            10.116.1.126         /* kube-system/monitoring-heapster: cluster IP */ tcp dpt:80
    0     0 KUBE-SVC-GLKZVFIDXOFHLJLC  tcp  --  *      *       0.0.0.0/0            10.116.1.175         /* mngbox/jumpbox:ssh cluster IP */ tcp dpt:2345
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 MARK set 0x4d415351
    0     0 KUBE-SVC-GLKZVFIDXOFHLJLC  tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
    0     0 KUBE-SVC-GLKZVFIDXOFHLJLC  tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 ADDRTYPE match dst-type LOCAL
    0     0 KUBE-SVC-6N4SJQIF3IX3FORG  tcp  --  *      *       0.0.0.0/0            10.116.0.1           /* default/kubernetes: cluster IP */ tcp dpt:443
    0     0 KUBE-SVC-B6ZEWWY2BII6JG2L  tcp  --  *      *       0.0.0.0/0            10.116.0.233         /* default/kibana-logging: cluster IP */ tcp dpt:8888
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* default/kibana-logging: external IP */ tcp dpt:8888 MARK set 0x4d415351
    0     0 KUBE-SVC-B6ZEWWY2BII6JG2L  tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* default/kibana-logging: external IP */ tcp dpt:8888 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
    0     0 KUBE-SVC-B6ZEWWY2BII6JG2L  tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* default/kibana-logging: external IP */ tcp dpt:8888 ADDRTYPE match dst-type LOCAL
    0     0 KUBE-NODEPORTS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-6N4SJQIF3IX3FORG (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-CJMHKLXPTJLTE3OP  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes: */

Chain KUBE-SVC-7IHGTXJ4CF2KVXJZ (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-YBQVM4LA4YMMZIWH  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-heapster: */

Chain KUBE-SVC-B6ZEWWY2BII6JG2L (3 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-ZDOOYAKDERKR43R3  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kibana-logging: */

Chain KUBE-SVC-CK6HVV5A27TDFNIA (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-BNPLX5HQYOZINWEQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-influxdb:api */

Chain KUBE-SVC-DKEW3YDJFV3YJLS2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-YMZS7BLP4Y6MWTX5  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* infra/docker-registry-backend:docker-registry-backend */

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-OAYOAJINXRPUQDA3  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */

Chain KUBE-SVC-GLKZVFIDXOFHLJLC (4 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-5IXMK7UWPGVTWOJ7  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* mngbox/jumpbox:ssh */

Chain KUBE-SVC-JRXTEHDDTAFMSEAS (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-WE3Z7KMHA6KPJWKK  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-grafana: */

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-GSM3BZTEXEBWDXPN  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */

Chain KUBE-SVC-WEHLQ23XZWSA5ZX3 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-RWODGLKOVWXGOHUR  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-influxdb:http */

Chain KUBE-SVC-XZFGDLM7GMJHZHOY (4 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-PJJZDQNXDGWM7MU6  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/docker-registry-fe:tcp */

当我请求服务 ip 时，在我的情况下它是 10.116.0.2 我收到一个错误

;; connection timed out; no servers could be reached

而当我向 10.116.160.7 服务器发出请求时，它工作正常。我可以看到流量根本没有指向 kube-proxy 规则，所以可能缺少一些东西。

我将非常感谢有关缺少规则的任何提示

编辑我用 thokin 要求的缺失信息更新了我的初始请求，他指出了调试 kube-proxy 的 iptables 规则的真正好方法，我可以通过以下方式确定我的问题：

for c in PREROUTING OUTPUT POSTROUTING; do iptables -t nat -I $c -d 10.116.160.7 -j LOG --log-prefix "DBG@$c: "; done
for c in PREROUTING OUTPUT POSTROUTING; do iptables -t nat -I $c -d 10.116.0.2 -j LOG --log-prefix "DBG@$c: "; done

然后我执行了以下命令： # nslookup kubernetes.default.svc.psc01.cluster 10.116.160.7 Server: 10.116.160.7 Address: 10.116.160.7#53

Name:   kubernetes.default.svc.psc01.cluster
Address: 10.116.0.1

# nslookup kubernetes.default.svc.psc01.cluster  10.116.0.2
;; connection timed out; no servers could be reached

结果，我得到了不同的“源”地址和传出接口：

[701768.263847] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=12436 PROTO=UDP SPT=54501 DPT=53 LEN=62 
[702620.454211] DBG@OUTPUT: IN= OUT=docker0 SRC=10.116.176.1 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=22733 PROTO=UDP SPT=28704 DPT=53 LEN=62 
[702620.454224] DBG@POSTROUTING: IN= OUT=docker0 SRC=10.116.176.1 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=22733 PROTO=UDP SPT=28704 DPT=53 LEN=62 
[702626.318258] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62 
[702626.318263] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62 
[702626.318266] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62 
[702626.318270] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62 
[702626.318284] DBG@POSTROUTING: IN= OUT=docker0 SRC=10.116.250.252 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62

所以，通过添加路线

ip route add 10.116.0.0/23 dev docker0

现在它工作正常！

score 5 · Accepted Answer

对于未来，结果iptables-save更容易阅读（无论如何对我来说）。

我看不出这里有什么遗漏。

KUBE-SERVICES捕获 10.116.0.2 端口 53/UDP 并将其传递给KUBE-SVC-TCOU7JCQXEZGVUNU

KUBE-SVC-TCOU7JCQXEZGVUNU只有一个端点，所以跳转到KUBE-SEP-GSM3BZTEXEBWDXPN

KUBE-SEP-GSM3BZTEXEBWDXPNDNAT 到 10.116.160.7 端口 53/UDP

如果您断言 10.116.160.7 有效而 10.116.0.2 无效，那确实很奇怪。这表明 iptables 规则根本没有触发。您是从节点本身还是从容器进行测试？

你用的是什么网络？L3（垫层？）法兰绒？OVS？还有什么？

什么云提供商（如果有）？

调试的第一步：运行：for c in PREROUTING OUTPUT; do iptables -t nat -I $c -d 10.116.0.2 -j LOG --log-prefix "DBG@$c: "; done

这会将 iptables 看到的任何数据包记录到您的服务 IP。现在看看dmesg。

kubernetes - iptables 模式下的 kube-proxy 无法正常工作

1 回答 1

Related

Reference