kubernetes - Kubernetes 服务通信问题 - Kubedns

Question

我有两个 pod 映射到两个服务，并使用笔记本电脑中的虚拟机 vm 运行。我有 kube dns 工作。一个 pod 是 web 服务，另一个是 mongodb。

webapp pod的规格如下

spec:
  containers:
    - resources:
        limits:
          cpu: 0.5
          .
          .
      name: wsemp
      ports:
        - containerPort: 8080
  #     name: wsemp
  #command: ["java","-Dspring.data.mongodb.uri=mongodb://192.168.6.103:30061/microservices", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
  command: ["java","-Dspring.data.mongodb.uri=mongodb://mongoservice/microservices", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

对应服务的规格

apiVersion: v1
kind: Service
metadata:
  labels:
    name: webappservice
  name: webappservice
spec:
  ports:
   - port: 8080
     nodePort: 30062
     targetPort: 8080
     protocol: TCP
  type: NodePort
  selector:
    name: webapp

Mongodb 吊舱规格

apiVersion: v1
kind: Pod
metadata:
  name: mongodb
  labels:
    name: mongodb
spec:
  containers:
    .
    .
  name: mongodb
  ports:
    - containerPort: 27017

MongoDB服务规范

apiVersion: v1
kind: Service
metadata:
  labels:
    name: mongodb
  name: mongoservice
spec:
  ports:
   - port: 27017
     nodePort: 30061
     targetPort: 27017
     protocol: TCP
  type: NodePort
  selector:
    name: mongodb

评论后更新服务中的目标港口

问题

webapp 启动时无法连接 mongoservice 端口并在启动时出现此错误

Exception in monitor thread while connecting to server mongoservice:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.connection.SocketStream.open(SocketStream.java:63) ~[mongodb-driver-core-3.2.2.jar!/:na]
at        com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:114) ~[mongodb-driver-core-3.2.2.jar!/:na]
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:128) ~[mongodb-driver-core-3.2.2.jar!/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_111]

描述 svc

kubectl describe svc mongoservice
Name:           mongoservice
Namespace:      default
Labels:         name=mongodb
Selector:       name=mongodb
Type:           NodePort
IP:         10.254.146.189
Port:           <unset> 27017/TCP
NodePort:       <unset> 30061/TCP
Endpoints:      172.17.99.2:27017
Session Affinity:   None
No events.

kubectl describe svc webappservice 
Name:           webappservice
Namespace:      default
Labels:         name=webappservice
Selector:       name=webapp
Type:           NodePort
IP:         10.254.112.121
Port:           <unset> 8080/TCP
NodePort:       <unset> 30062/TCP
Endpoints:      172.17.99.3:8080
Session Affinity:   None
No events.

调试

root@webapp:/# nslookup mongoservice
Server:     10.254.0.2
Address:    10.254.0.2#53

Non-authoritative answer:
Name:   mongoservice.default.svc.cluster.local
Address: 10.254.146.189

root@webapp:/# curl 10.254.146.189:27017
curl: (7) Failed to connect to 10.254.146.189 port 27017: Connection refused
root@webapp:/# curl mongoservice:27017
curl: (7) Failed to connect to mongoservice port 27017: Connection refused


sudo iptables-save | grep webapp

-A KUBE-NODEPORTS -p tcp -m comment --comment "default/webappservice:" -m tcp --dport 30062 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/webappservice:" -m tcp --dport 30062 -j KUBE-SVC-NQBDRRKQULANV7O3
-A KUBE-SEP-IE7EBTQCN7T6HXC4 -s 172.17.99.3/32 -m comment --comment "default/webappservice:" -j KUBE-MARK-MASQ
-A KUBE-SEP-IE7EBTQCN7T6HXC4 -p tcp -m comment --comment "default/webappservice:" -m tcp -j DNAT --to-destination 172.17.99.3:8080
-A KUBE-SERVICES -d 10.254.217.24/32 -p tcp -m comment --comment "default/webappservice: cluster IP" -m tcp --dport 8080 -j KUBE-SVC-NQBDRRKQULANV7O3
-A KUBE-SVC-NQBDRRKQULANV7O3 -m comment --comment "default/webappservice:" -j KUBE-SEP-IE7EBTQCN7T6HXC4
$ curl 10.254.217.24:8080
{"timestamp":1486678423757,"status":404,"error":"Not Found","message":"No message available","path":"/"}[osboxes@kube-node1 ~]$ 


sudo iptables-save | grep mongodb
[osboxes@osboxes ~]$ sudo iptables-save | grep mongo
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/mongoservice:" -m tcp --dport 30061 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/mongoservice:" -m tcp --dport 30061 -j KUBE-SVC-2HQWGC3WSIBZF7CN
-A KUBE-SEP-FVWOWAWXXVAVIQ5O -s 172.17.99.2/32 -m comment --comment "default/mongoservice:" -j KUBE-MARK-MASQ
-A KUBE-SEP-FVWOWAWXXVAVIQ5O -p tcp -m comment --comment "default/mongoservice:" -m tcp -j DNAT --to-destination 172.17.99.2:27017
-A KUBE-SERVICES -d 10.254.146.189/32 -p tcp -m comment --comment "default/mongoservice: cluster IP" -m tcp --dport 27017 -j KUBE-SVC-2HQWGC3WSIBZF7CN
-A KUBE-SVC-2HQWGC3WSIBZF7CN -m comment --comment "default/mongoservice:" -j KUBE-SEP-FVWOWAWXXVAVIQ5O
[osboxes@osboxes ~]$ sudo curl  10.254.146.189:8080
^C[osboxes@osboxes ~]$ sudo curl  10.254.146.189:27017

It looks like you are trying to access MongoDB over HTTP on the native driver port.


root@mongodb:/# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:27017           0.0.0.0:*               LISTEN     
tcp        0      0 172.17.99.2:60724       151.101.128.204:80      TIME_WAIT  
tcp        0      0 172.17.99.2:60728       151.101.128.204:80      TIME_WAIT

mongodb 容器在启动时没有错误。

尝试遵循https://kubernetes.io/docs/user-guide/debugging-services/#iptables中的步骤，卡在“尝试重新启动 kube-proxy 并将 -V 标志设置为 4”的部分，因为我不知道该怎么做。

我不是网络人，所以不知道如何以及需要分析什么。任何调试技巧都会有很大帮助。

谢谢。

score 1 · Accepted Answer

:)

作为旁注，请记住 curl 默认执行 HTTP 请求，但您尝试访问的主机中的端口 27017 未绑定到理解此类协议的应用程序。通常，在这些情况下您会使用 netcat：

nc -zv mongoservice 27017

这会报告来自此类主机的端口 27017 是否打开。

nc = 网猫
-z 扫描监听守护进程而不发送数据
-v 增加详细程度

关于您的 MongoDB 文件，您必须记住设置 targetPort 指令。正如关于 targetPort 的 Kubernetes 文档中所解释的：

This specification will create a Service which targets TCP port 80 on any Pod with the run: my-nginx label, and expose it on an abstracted Service port (targetPort: is the port the container accepts traffic on, port: is the abstracted Service port, which can be any port other pods use to access the Service). View service API object to see the list of supported fields in service definition.

因此，只需将其设置为 27017 以保持一致性。

遵循这些建议后，您不应遇到问题。保持良好的工作并尽可能多地学习！

score 0 · Accepted Answer

iptables 规则看起来不错，但不确定您的 kubernetes 中使用了哪种网络解决方案（flannel/calico）。您可以检查您是否可以从您的 web pod 访问 kube dns pod IP。

score 0 · Accepted Answer

谢谢。我对此有所了解，并且由于我使用的是 flannel 网络，因此 flannel 网络中的 pod 之间的通信存在问题。

特别是链接http://jayunit100.blogspot.com/2015/06/flannel-and-vagrant-heads-up.html中提到的这部分 FLANNEL_OPTIONS="--iface=eth1"

谢谢。

kubernetes - Kubernetes 服务通信问题 - Kubedns

3 回答 3

Related

Reference