2

我正在尝试在 Kubernetes 上设置 EFK 堆栈。使用的 Elasticsearch 版本是 6.3.2。一切正常，直到我将探针配置放在部署 YAML 文件中。我收到如下错误。这导致 pod 被声明为不健康并最终重新启动，这似乎是错误的重新启动。

警告 Unhealthy 15s kubelet, aks-agentpool-23337112-0 Liveness probe failed: Get http://10.XXX.Y.ZZZ:9200/_cluster/health : dial tcp 10.XXX.Y.ZZZ:9200: connect: connection拒绝

我确实尝试使用来自不同容器的 telnet 到具有 IP 和端口的 elasticsearch pod，我成功了，但只有节点上的 kubelet 无法解析 pod 的 IP，导致探测失败。

以下是 Kubernetes Statefulset YAML 的 pod 规范的片段。对决议的任何帮助都会非常有帮助。在没有任何线索的情况下花了很多时间:(

PS：堆栈正在 AKS 群集上设置

      - name: es-data
        image: quay.io/pires/docker-elasticsearch-kubernetes:6.3.2
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: CLUSTER_NAME
          value: myesdb
        - name: NODE_MASTER
          value: "false"
        - name: NODE_INGEST
          value: "false"
        - name: HTTP_ENABLE
          value: "true"
        - name: NODE_DATA
          value: "true"
        - name: DISCOVERY_SERVICE
          value: "elasticsearch-discovery"
        - name: NETWORK_HOST
          value: "_eth0:ipv4_"          
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        - name: PROCESSORS
          valueFrom:
            resourceFieldRef:
              resource: limits.cpu
        resources:
          requests:
            cpu: 0.25
          limits:
            cpu: 1
        ports:
        - containerPort: 9200
          name: http
        - containerPort: 9300
          name: transport
        livenessProbe:
          httpGet:
            port: http
            path: /_cluster/health
          initialDelaySeconds: 40
          periodSeconds: 10
       readinessProbe:
         httpGet:
           path: /_cluster/health
           port: http
         initialDelaySeconds: 30
         timeoutSeconds: 10

没有探针，吊舱/容器运行得很好。期望在部署 YAML 上设置时探针应该可以正常工作，并且 POD 不应该重新启动。

4

3 回答 3

5

首先请使用检查日志

kubectl logs <pod name> -n <namespacename>

您必须首先运行 init 容器并更改卷权限。

您必须在 elasticsearch 容器启动之前运行整个配置，user : 1000您必须使用 init 容器更改卷权限。

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app : elasticsearch
    component: elasticsearch
    release: elasticsearch
  name: elasticsearch
spec:
  podManagementPolicy: Parallel
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app : elasticsearch
      component: elasticsearch
      release: elasticsearch
  serviceName: elasticsearch
  template:
    metadata:
      creationTimestamp: null
      labels:
        app : elasticsearch
        component: elasticsearch
        release: elasticsearch
    spec:
      containers:
      - env:
        - name: cluster.name
          value: <SET THIS>
        - name: discovery.type
          value: single-node
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        - name: bootstrap.memory_lock
          value: "false"
        image: elasticsearch:6.5.0
        imagePullPolicy: IfNotPresent
        name: elasticsearch
        ports:
        - containerPort: 9200
          name: http
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        resources:
          limits:
            cpu: 250m
            memory: 1Gi
          requests:
            cpu: 150m
            memory: 512Mi
        securityContext:
          privileged: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: elasticsearch-data
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sh
        - -c
        - chown -R 1000:1000 /usr/share/elasticsearch/data
        - sysctl -w vm.max_map_count=262144
        - chmod 777 /usr/share/elasticsearch/data
        - chomod 777 /usr/share/elasticsearch/data/node
        - chmod g+rwx /usr/share/elasticsearch/data
        - chgrp 1000 /usr/share/elasticsearch/data
        image: busybox:1.29.2
        imagePullPolicy: IfNotPresent
        name: set-dir-owner
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: elasticsearch-data
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 10
  updateStrategy:
    type: OnDelete
  volumeClaimTemplates:
  - metadata:
      creationTimestamp: null
      name: elasticsearch-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi

查看我的 yaml 配置，您可以使用。用于elasticsearch的单节点

于 2019-09-13T07:12:51.317 回答
5

问题是 ElasticSearch 本身有自己的健康状态（红色、黄色、绿色），您需要在配置中考虑到这一点。

这是我在自己的 ES 配置中发现的，基于官方 ES helm chart：

        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 40
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5

          exec:
            command:
              - sh
              - -c
              - |
                #!/usr/bin/env bash -e
                # If the node is starting up wait for the cluster to be green
                # Once it has started only check that the node itself is responding
                START_FILE=/tmp/.es_start_file

                http () {
                    local path="${1}"
                    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                      BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                    else
                      BASIC_AUTH=''
                    fi
                    curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
                }

                if [ -f "${START_FILE}" ]; then
                    echo 'Elasticsearch is already running, lets check the node is healthy'
                    http "/"
                else
                    echo 'Waiting for elasticsearch cluster to become green'
                    if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
                        touch ${START_FILE}
                        exit 0
                    else
                        echo 'Cluster is not yet green'
                        exit 1
                    fi
                fi
于 2019-09-13T07:15:20.163 回答
0

我的答案中概述的探针在出现时适用于 3 个节点发现Istio。如果livenessProbe不好，那么 k8s 将重新启动容器，甚至不允许正常启动。我使用内部弹性端口（用于节点到节点通信）来测试活跃度。这些端口使用 TCP。

      livenessProbe:
        tcpSocket:
          port: 9300
        initialDelaySeconds: 60 # it takes time from jvm process to start start up to point when discovery process starts
        timeoutSeconds: 10

          - name: discovery.zen.minimum_master_nodes
          value: "2"
          - name: discovery.zen.ping.unicast.hosts
          value: elastic
于 2019-09-27T09:19:22.023 回答