3

我正在我的 K8S 集群上设置 NFS 服务器。我希望它充当外部实体的 NFS 服务器，即客户端将来自 K8S 集群外部，例如 VM。

Docker 镜像的端口要求是：

==================================================================
      SERVER STARTUP COMPLETE
==================================================================
----> list of enabled NFS protocol versions: 4.2, 4.1, 4
----> list of container exports:
---->   /exports *(rw,no_subtree_check)
----> list of container ports that should be exposed:
---->   111 (TCP and UDP)
---->   2049 (TCP and UDP)
---->   32765 (TCP and UDP)
---->   32767 (TCP and UDP)

所以我创建了一个 Debian Stretch docker 镜像。当我使用 运行它时docker run，我可以成功地从其他系统公开/exports和挂载它。

docker run -v /data:/exports -v /tmp/exports.txt:/etc/exports:ro \
--cap-add SYS_ADMIN -p 2049:2049 -p 111:111 -p 32765:32765 \
-p 32767:32767 8113b6abeac

上面的命令启动了我的 docker 容器，当我这样做时

mount.nfs4 <DOKCER_HOST_IP>:/exports /mount/

从另一个虚拟机，我可以成功挂载该卷。

所以直到这里的一切都是好的！

现在的任务是在 K8S 中部署它。

我的状态集定义是：

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: nfs-provisioner
spec:
  selector:
    matchLabels:
      app: nfs-provisioner
  serviceName: "nfs-provisioner"
  replicas: 1
  template:
    metadata:
      labels:
        app: nfs-provisioner
    spec:
      serviceAccount: nfs-provisioner
      terminationGracePeriodSeconds: 10
      imagePullSecrets:
      - name: artifactory
      containers:
        - name: nfs-provisioner
          image: repository.hybris.com:5005/test/nfs/nfs-server:1.2
          ports:
            - name: nfs
              containerPort: 2049
            - name: mountd
              containerPort: 20048
            - name: rpcbind
              containerPort: 111
            - name: rpcbind-udp
              containerPort: 111
              protocol: UDP
            - name: filenet
              containerPort: 32767
            - name: filenet-udp
              containerPort: 32767
              protocol: UDP
            - name: unknown
              containerPort: 32765
            - name: unknown-udp
              containerPort: 32765
              protocol: UDP
          securityContext:
            privileged: true
          env:
            - name: SERVICE_NAME
              value: nfs-provisioner
            - name: NFS_EXPORT_0
              value: '/exports *(rw,no_subtree_check)'
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: export-volume
              mountPath: /exports
      volumes:
        - name: export-volume
          hostPath:
            path: /var/tmp

如您所见，我已经指定了所有端口（TCP 和 UDP）

现在要将其暴露给外界而不仅仅是在集群内部，我的service.yaml文件减速是：

kind: Service
apiVersion: v1
metadata:
  name: nfs-provisioner
  labels:
    app: nfs-provisioner
spec:
  type: NodePort
  ports:
    - name: nfs
      port: 2049
    - name: mountd
      port: 20048
    - name: rpcbind
      port: 111
    - name: rpcbind-udp
      port: 111
      protocol: UDP
    - name: filenet
      port: 32767
    - name: filenet-udp
      port: 32767
      protocol: UDP
    - name: unknown
      port: 32765
    - name: unknown-udp
      port: 32765
      protocol: UDP
  selector:
    app: nfs-provisioner

这导致

kubectl get svc
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                                                      AGE
nfs-provisioner   NodePort    10.233.43.135   <none>        2049:30382/TCP,20048:31316/TCP,111:32720/TCP,111:32720/UDP,32767:30173/TCP,32767:30173/UDP,32765:31215/TCP,32765:31215/UDP   32m

现在我尝试/exports从 K8S 集群外部的另一个节点/VM 挂载。

我试过了

mount.nfs4 <K8S_Node_IP>:/exports /mount/

我试过了

mount.nfs4 -o port=<NodePort> <K8S_Node_IP>:/exports /mount/

我一次尝试了每个 NodePort。但它们都不起作用。我得到错误：

mount.nfs4 -o port=31316 <K8S_Node_IP>:/exports /mount/
mount.nfs4: mount to NFS server '<K8S_Node_IP>:/exports' failed: RPC Error: Unable to receive

我不确定这里可能是什么问题。我需要指定所有的nodePorts吗？如果是这样，我该怎么做？

4

1 回答 1

2

The issue here is that all the NodePorts are different as seen externally as from:

---->   111 (TCP and UDP)
---->   2049 (TCP and UDP)
---->   32765 (TCP and UDP)
---->   32767 (TCP and UDP)

You can try an L4 load balancer that exposes exactly those ports on a given IP address (internal or external) and forwards them to the nodePorts (which is what type=LoadBalancer does too).

Another option is to hard code the NodePorts in your services to match exactly the ones of the containers:

kind: Service
apiVersion: v1
metadata:
  name: nfs-provisioner
  labels:
    app: nfs-provisioner
spec:
  type: NodePort
  ports:
    - name: nfs
      port: 2049
      nodePort: 2049
    - name: mountd
      port: 20048
      nodePort: 20048
    - name: rpcbind
      port: 111
      nodePort: 111
    - name: rpcbind-udp
      port: 111
      nodePort: 111
      protocol: UDP
    - name: filenet
      port: 32767
      nodePort: 32767
    - name: filenet-udp
      port: 32767
      nodePort: 32767
      protocol: UDP
    - name: unknown
      port: 32765
      nodePort: 32765
    - name: unknown-udp
      port: 32765
      nodePort: 32765
      protocol: UDP
  selector:
    app: nfs-provisioner

You will have to change the nodePort range (--service-node-port-range) on the kubelet though. This is so that you can use 2049 and 111.

You can also change the ports that you NFS server listens on for 2049 (nfs) and 111 (portmapper) for example, that way you don't have to change --service-node-port-range

于 2018-11-21T20:51:55.933 回答