kubernetes - kube-apiserver pod 停留在 CreateContainerError 状态

Question

我使用 kubeadm 引导一个 kubernetes 集群。几个月不活动后，当我得到正在运行的 pod 时，我意识到 kube-apiserver 卡在 CreatecontainerError 中！

kubectl get pods -n kube-system
NAME                                    READY   STATUS                 RESTARTS   AGE
coredns-576cbf47c7-bcv8m                1/1     Running                435        175d
coredns-576cbf47c7-dwvmv                1/1     Running                435        175d
etcd-master                             1/1     Running                23         175d
kube-apiserver-master                   0/1     CreateContainerError   23         143m
kube-controller-manager-master          1/1     Running                27         175d
kube-proxy-2s9sx                        1/1     Running                23         175d
kube-proxy-rrp7m                        1/1     Running                20         127d
kube-scheduler-master                   1/1     Running                24         175d
kubernetes-dashboard-65c76f6c97-7cwwp   1/1     Running                34         169d
tiller-deploy-779784fbd6-cwrqn          1/1     Running                0          152m
weave-net-2g8s5                         2/2     Running                62         170d
weave-net-9r6cp                         2/2     Running                44         127d

我删除了这个 pod 以重新启动它，但仍然出现同样的问题。

更多信息 ：

$ kubectl get nodes
NAME      STATUS     ROLES    AGE    VERSION
master    Ready      master   175d   v1.12.1
worker    Ready      worker   175d   v1.12.1

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl describe pod kube-apiserver-master -n kube-system
Name:               kube-apiserver-master
Namespace:          kube-system
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               master/192.168.88.205
Start Time:         Wed, 07 Aug 2019 17:58:29 +0430
Labels:             component=kube-apiserver
                    tier=control-plane
Annotations:        kubernetes.io/config.hash: ce0f74ad5fcbf28c940c111df265f4c8
                    kubernetes.io/config.mirror: ce0f74ad5fcbf28c940c111df265f4c8
                    kubernetes.io/config.seen: 2019-08-07T17:58:28.178339939+04:30
                    kubernetes.io/config.source: file
                    scheduler.alpha.kubernetes.io/critical-pod: 
Status:             Running
IP:                 192.168.88.205
Containers:
  kube-apiserver:
    Container ID:  docker://3328849ad82745341717616f4ef6e951116fde376d19990610f670c30eb1e26f
    Image:         k8s.gcr.io/kube-apiserver:v1.12.1
    Image ID:      docker-pullable://k8s.gcr.io/kube-apiserver@sha256:52b9dae126b5a99675afb56416e9ae69239e012028668f7274e30ae16112bb1f
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-apiserver
      --authorization-mode=Node,RBAC
      --advertise-address=192.168.88.205
      --allow-privileged=true
      --client-ca-file=/etc/kubernetes/pki/ca.crt
      --enable-admission-plugins=NodeRestriction
      --enable-bootstrap-token-auth=true
      --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
      --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
      --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
      --etcd-servers=https://127.0.0.1:2379
      --insecure-port=0
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
      --requestheader-allowed-names=front-proxy-client
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
      --secure-port=6443
      --service-account-key-file=/etc/kubernetes/pki/sa.pub
      --service-cluster-ip-range=10.96.0.0/12
      --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
      --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    State:          Waiting
      Reason:       CreateContainerError
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Wed, 07 Aug 2019 17:58:30 +0430
      Finished:     Wed, 07 Aug 2019 13:28:11 +0430
    Ready:          False
    Restart Count:  23
    Requests:
      cpu:        250m
    Liveness:     http-get https://192.168.88.205:6443/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:  <none>
    Mounts:
      /etc/ca-certificates from etc-ca-certificates (ro)
      /etc/kubernetes/pki from k8s-certs (ro)
      /etc/ssl/certs from ca-certs (ro)
      /usr/local/share/ca-certificates from usr-local-share-ca-certificates (ro)
      /usr/share/ca-certificates from usr-share-ca-certificates (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  k8s-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/pki
    HostPathType:  DirectoryOrCreate
  ca-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:  DirectoryOrCreate
  usr-share-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/share/ca-certificates
    HostPathType:  DirectoryOrCreate
  usr-local-share-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/share/ca-certificates
    HostPathType:  DirectoryOrCreate
  etc-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ca-certificates
    HostPathType:  DirectoryOrCreate
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute
Events:            <none>

$ kubectl get pods kube-apiserver-master -n kube-system -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: ce0f74ad5fcbf28c940c111df265f4c8
    kubernetes.io/config.mirror: ce0f74ad5fcbf28c940c111df265f4c8
    kubernetes.io/config.seen: 2019-08-07T17:58:28.178339939+04:30
    kubernetes.io/config.source: file
    scheduler.alpha.kubernetes.io/critical-pod: ""
  creationTimestamp: 2019-08-13T08:33:18Z
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver-master
  namespace: kube-system
  resourceVersion: "19613877"
  selfLink: /api/v1/namespaces/kube-system/pods/kube-apiserver-master
  uid: 0032d68b-bda5-11e9-860c-000c292f9c9e
spec:
  containers:
  - command:
    - kube-apiserver
    - --authorization-mode=Node,RBAC
    - --advertise-address=192.168.88.205
    - --allow-privileged=true
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.12.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.88.205
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
  dnsPolicy: ClusterFirst
  hostNetwork: true
  nodeName: master
  priority: 2000000000
  priorityClassName: system-cluster-critical
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    operator: Exists
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2019-08-07T13:28:29Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2019-08-07T08:58:11Z
    message: 'containers with unready status: [kube-apiserver]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2019-08-07T08:58:11Z
    message: 'containers with unready status: [kube-apiserver]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: 2019-08-07T13:28:29Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://3328849ad82745341717616f4ef6e951116fde376d19990610f670c30eb1e26f
    image: k8s.gcr.io/kube-apiserver:v1.12.1
    imageID: docker-pullable://k8s.gcr.io/kube-apiserver@sha256:52b9dae126b5a99675afb56416e9ae69239e012028668f7274e30ae16112bb1f
    lastState:
      terminated:
        containerID: docker://3328849ad82745341717616f4ef6e951116fde376d19990610f670c30eb1e26f
        exitCode: 255
        finishedAt: 2019-08-07T08:58:11Z
        reason: Error
        startedAt: 2019-08-07T13:28:30Z
    name: kube-apiserver
    ready: false
    restartCount: 23
    state:
      waiting:
        message: 'Error response from daemon: Conflict. The container name "/k8s_kube-apiserver_kube-apiserver-master_kube-system_ce0f74ad5fcbf28c940c111df265f4c8_24"
          is already in use by container 14935b714aee924aa42295fa5d252c760264d24ee63ea74e67092ccc3fb2b530.
          You have to remove (or rename) that container to be able to reuse that name.'
        reason: CreateContainerError
  hostIP: 192.168.88.205
  phase: Running
  podIP: 192.168.88.205
  qosClass: Burstable
  startTime: 2019-08-07T13:28:29Z

如果需要任何其他信息，请告诉我。

我怎样才能让它正常运行？

Answer 1

来自 docker daemon 的此错误消息解释了该问题：

消息：'来自守护程序的错误响应：冲突。容器名称“/k8s_kube-apiserver_kube-apiserver-master_kube-system_ce0f74ad5fcbf28c940c111df265f4c8_24”已被容器 14935b714aee924aa42295fa5d252c760264d24ee63ea74e67092ccc3fb2b530 使用。您必须删除（或重命名）该容器才能重用该名称。原因：CreateContainerError

列出所有容器使用：

docker ps -a

您应该能够在列表容器中找到具有以下名称的容器：

/k8s_kube-apiserver_kube-apiserver-master_kube-system_ce0f74ad5fcbf28c940c111df265f4c8_24

或身份证：

14935b714aee924aa42295fa5d252c760264d24ee63ea74e67092ccc3fb2b530

然后你可以尝试通过运行来删除它：

docker rm "/k8s_kube-apiserver_kube-apiserver-master_kube-system_ce0f74ad5fcbf28c940c111df265f4c8_24"

或通过提供其 ID：

docker rm 14935b714aee924aa42295fa5d252c760264d24ee63ea74e67092ccc3fb2b530

如果删除还是有问题，添加-fflag强制删除：

docker rm -f 14935b714aee924aa42295fa5d252c760264d24ee63ea74e67092ccc3fb2b530

完成后，您可以尝试删除kube-apiserver-masterpod，以便重新创建它。