0

When using kubernetes(minikube) statefulset on local machine, EMQX Rules are persisting because same pod IP is being assigned to the emqx node, for example /opt/emqx/data/mnesia/emqx@172.17.0.9. Even if I delete the pod when the new pod starts, it gets assigned the same IP as before. Everything is working as it should.

But when I'm using aks(azure kubernetes) to deploy EMQX on aks cluster using azure files, pod IP is different everytime. For example if /opt/emqx/data/mnesia/emqx@10.1.1.10 is assigned to the EMQX node, then if I try to delete the pod then /opt/emqx/data/mnesia/emqx@10.1.1.11 might be assigned to it.

so, nothing is persisting.

Local code

---

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage5
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: emqx-pv5
spec:
  capacity:
    storage: 300Mi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage5
  local:
    path: /opt/emqx/data/mnesia
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - minikube

---

apiVersion: v1
kind: Service
metadata:
  name: emqx-headless
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app: emqx
  ports:
    - name: mqtt
      port: 1883
      protocol: TCP
      targetPort: 1883
    - name: mqttssl
      port: 8883
      protocol: TCP
      targetPort: 8883
    - name: mgmt
      port: 8081
      protocol: TCP
      targetPort: 8081
    - name: websocket
      port: 8083
      protocol: TCP
      targetPort: 8083
    - name: wss
      port: 8084
      protocol: TCP
      targetPort: 8084
    - name: dashboard
      port: 18083
      protocol: TCP
      targetPort: 18083

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: emqx-statefulset
  labels:
    app: emqx
spec:
  replicas: 1
  serviceName: emqx-headless
  selector:
    matchLabels:
      app: emqx
  template:
    metadata:
      labels:
        app: emqx
    spec:
      containers:
        - name: emqx
          image: emqx/emqx:4.2.7
          ports:
            - name: emqx-dashboard
              containerPort: 18083
            - name: ssl-port
              containerPort: 8883
            - name: emqx-port
              containerPort: 1883
            - name: ssl-dashboard
              containerPort: 18084
          env:
            - name: EMQX_LOADED_PLUGINS
              value: emqx_management,emqx_recon,emqx_retainer,emqx_dashboard,emqx_rule_engine,emqx_auth_username
            - name: EMQX_CLUSTER__DISCOVERY
              value: k8s
            - name: EMQX_NAME
              value: emqx
            - name: EMQX_CLUSTER__K8S__APISERVER
              value: https://kubernetes.default:443
            - name: EMQX_CLUSTER__K8S__SERVICE_NAME
              value: emqx
            - name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
              value: ip
            - name: EMQX_CLUSTER__K8S__APP_NAME
              value: emqx
            - name: EMQX_ALLOW_ANONYMOUS
              value: "false"
            - name: EMQX_LISTENER__SSL__EXTERNAL__MAX_CONNECTIONS
              value: "1024000"
            - name: EMQX_AUTH__USER__PASSWORD_HASH
              value: sha256
            - name: EMQX_AUTH__USER__1__USERNAME
              value: 
            - name: EMQX_AUTH__USER__1__PASSWORD
              value: 
            - name: EMQX_DASHBOARD__DEFAULT_USER__LOGIN
              value: 
            - name: EMQX_DASHBOARD__DEFAULT_USER__PASSWORD
              value:
            - name: EMQX_DASHBOARD__LISTENER__HTTPS
              value: "18084"
            - name: MQX_DASHBOARD__LISTENER__HTTPS__ACCEPTORS
              value: "4"
            - name: EMQX_DASHBOARD__LISTENER__HTTPS__MAX_CLIENTS
              value: "512"
          tty: true
          volumeMounts:
            - name: emqx-mnesia
              mountPath: "/opt/emqx/data/mnesia"

  volumeClaimTemplates:
    - metadata:
        name: emqx-mnesia
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "local-storage5"
        resources:
          requests:
            storage: 300Mi

Azure Kubernetes code

apiVersion: v1
kind: ServiceAccount
metadata:
  name: emqx
  namespace: emqx-test

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: emqx
subjects:
  - kind: ServiceAccount
    name: emqx
    namespace: emqx-test
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: emqx-files
provisioner: kubernetes.io/azure-file
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=0
  - gid=0
  - mfsymlinks
  - cache=strict
  - actimeo=30
parameters:
  skuName: Standard_LRS
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: emqx-pvc
  namespace: emqx-test
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: emqx-files
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: emqx
  namespace: emqx-test
spec:
  ports:
    - name: emqx-dashboard
      port: 80
      targetPort: 18083
      protocol: TCP
    - name: ssl-port
      port: 8883
      targetPort: ssl-port
      protocol: TCP
    - name: emqx-port
      port: 1883
      targetPort: emqx-port
      protocol: TCP
    - name: ssl-dashboard
      port: 443
      targetPort: 18084
      protocol: TCP
  selector:
    app: emqx
  type: LoadBalancer
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: emqx
  labels:
    app: emqx
  namespace: emqx-test
spec:
  serviceName: "emqx"
  selector:
    matchLabels:
      app: emqx
  replicas: 1
  template:
    metadata:
      labels:
        app: emqx
    spec:
      containers:
        - name: emqx
          image: emqx/emqx:4.2.7
          ports:
            - name: emqx-dashboard
              containerPort: 18083
            - name: ssl-port
              containerPort: 8883
            - name: emqx-port
              containerPort: 1883
            - name: ssl-dashboard
              containerPort: 18084
          env:
            - name: EMQX_LOADED_PLUGINS
              value: emqx_management,emqx_recon,emqx_retainer,emqx_dashboard,emqx_rule_engine,emqx_auth_username
            - name: EMQX_CLUSTER__DISCOVERY
              value: k8s
            - name: EMQX_NAME
              value: emqx
            - name: EMQX_CLUSTER__K8S__APISERVER
              value: https://kubernetes.default:443
            - name: EMQX_CLUSTER__K8S__NAMESPACE
              value: emqx-test
            - name: EMQX_CLUSTER__K8S__SERVICE_NAME
              value: emqx
            - name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
              value: ip
            - name: EMQX_CLUSTER__K8S__APP_NAME
              value: emqx
            - name: EMQX_ALLOW_ANONYMOUS
              value: "false"
            - name: EMQX_LISTENER__SSL__EXTERNAL__MAX_CONNECTIONS
              value: "1024000"
            - name: EMQX_AUTH__USER__PASSWORD_HASH
              value: sha256
            - name: EMQX_AUTH__USER__1__USERNAME
              value:
            - name: EMQX_AUTH__USER__1__PASSWORD
              value:
            - name: EMQX_DASHBOARD__DEFAULT_USER__LOGIN
              value:
            - name: EMQX_DASHBOARD__DEFAULT_USER__PASSWORD
              value:
            - name: EMQX_DASHBOARD__LISTENER__HTTPS
              value: "18084"
            - name: MQX_DASHBOARD__LISTENER__HTTPS__ACCEPTORS
              value: "4"
            - name: EMQX_DASHBOARD__LISTENER__HTTPS__MAX_CLIENTS
              value: "512"
          volumeMounts:
            - name: emqx-data
              mountPath: "/opt/emqx/data/mnesia"
          tty: true
      volumes:
        - name: emqx-data
          persistentVolumeClaim:
            claimName: emqx-pvc
4

1 回答 1

0

In k8s documentation on StatefulSet Basics you read:

The Pods' ordinals, hostnames, SRV records, and A record names have not changed, but the IP addresses associated with the Pods may have changed. In the cluster used for this tutorial, they have. This is why it is important not to configure other applications to connect to Pods in a StatefulSet by IP address.

This is expected and as you see and this behaviour is mentioned in documentation.

But why do you see different behavior on minikube and different on azure? IP addresses are assigned by CNIs. On minikube default CNI it the docker-bridge, and on azure its's Azure CNI, so it is up to the CNI what address is assigned.

It's best to always assume that you cannot rely on pod IP addresses to stay static. Use DNS for statefulsets and for other pods and services for communication and never use hardcoded pod ip addresses directly.

于 2021-02-09T09:39:27.667 回答