kubernetes - 运行 kubelet 时出现“x509：由未知机构签名的证书”

Question

我正在尝试在 CoreOS beta (1192.2.0) 上使用 kubelet 1.4.5 安装 kubernetes。

我正在使用来自https://github.com/coreos/coreos-kubernetes/tree/master/multi-node/generic的控制器和工作程序安装脚本的略微修改版本

所以总的来说，我使用以下 bash 脚本在 Gentoo Linux 上创建了许可证：

#!/bin/bash
export MASTER_HOST=coreos-2.tux-in.com
export K8S_SERVICE_IP=10.3.0.1
export WORKER_IP=10.79.218.3
export WORKER_FQDN=coreos-3.tux-in.com
openssl genrsa -out ca-key.pem 2048
openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca"
openssl genrsa -out apiserver-key.pem 2048
openssl req -new -key apiserver-key.pem -out apiserver.csr -subj "/CN=kube-apiserver" -config openssl.cnf
openssl x509 -req -in apiserver.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out apiserver.pem -days 365 -extensions v3_req -extfile openssl.cnf
openssl genrsa -out ${WORKER_FQDN}-worker-key.pem 2048
openssl req -new -key ${WORKER_FQDN}-worker-key.pem -out ${WORKER_FQDN}-worker.csr -subj "/CN=${WORKER_FQDN}" -config worker-openssl.cnf
openssl x509 -req -in ${WORKER_FQDN}-worker.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out ${WORKER_FQDN}-worker.pem -days 365 -extensions v3_req -extfile worker-openssl.cnf
openssl genrsa -out admin-key.pem 2048
openssl req -new -key admin-key.pem -out admin.csr -subj "/CN=kube-admin"
openssl x509 -req -in admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out admin.pem -days 365
echo done

这是openssl.cnf

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = coreos-2.tux-in.com
DNS.2 = coreos-3.tux-in.com
IP.1 = 10.3.0.1
IP.2 = 10.79.218.2
IP.3 = 10.79.218.3

这是我的worker-openssl.cnf

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
IP.1 = 10.79.218.3
DNS.1 = coreos-3.tux-in.com

我的控制器机器coreos-2.tux-in.com解析为 lan ip10.79.218.2

我的工作机器coreos-3.tux-in.com解析为 lan ip10.79.218.3

它创建了许可证就好了。但是当我使用它们并在主机上安装控制器脚本时，我看到当我运行时journalctl -xef -u kubelet，我注意到以下消息：

Nov 08 21:24:06 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:06.805868    2018 event.go:208] Unable to write event: 'x509: certificate signed by unknown authority' (may retry after sleeping)
Nov 08 21:24:06 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:06.950827    2018 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://coreos-2.tux-in.com:443/api/v1/services?resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:07 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:07.461042    2018 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://coreos-2.tux-in.com:443/api/v1/pods?fieldSelector=spec.nodeName%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:07 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:07.461340    2018 reflector.go:203] pkg/kubelet/kubelet.go:403: Failed to list *api.Node: Get https://coreos-2.tux-in.com:443/api/v1/nodes?fieldSelector=metadata.name%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.024366    2018 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://coreos-2.tux-in.com:443/api/v1/services?resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.171170    2018 eviction_manager.go:162] eviction manager: unexpected err: failed GetNode: node '10.79.218.2' not found
Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.543619    2018 reflector.go:203] pkg/kubelet/kubelet.go:403: Failed to list *api.Node: Get https://coreos-2.tux-in.com:443/api/v1/nodes?fieldSelector=metadata.name%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.543926    2018 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://coreos-2.tux-in.com:443/api/v1/pods?fieldSelector=spec.nodeName%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority

Answer 1

kubelet文档说--tls-cert-file标志需要在证书之后连接 CA。在你的情况下，它是apiserver.pem：

--tls-cert-file包含 HTTPS 的 x509 证书的文件。（CA 证书，如果有的话，连接在服务器证书之后）。如果未提供 --tls-cert-file 和 --tls-private-key-file ，则会为公共地址生成自签名证书和密钥，并保存到传递给 --cert-dir 的目录中。

如果我正确阅读了您的证书生成，apiserver.pem则不包含根 ca。

Answer 2

您的 OpenSSL 证书是“自签名”的：

openssl genrsa -out ca-key.pem 2048
openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca"

也就是说，您正在签署它们而不是受信任的证书颁发机构。只要您保持私钥安全，它应该是完全安全的。

如果您希望它由证书颁发机构签名，则需要生成 CSR（证书签名请求）。

https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs

Answer 3

一般来说，解决方案是创建另一个 etcd2 端口，该端口连接到每台机器的环回设备并在http而不是https. calico-policy-controller 请求不同 coreos 服务器的 etcd2 证书的更多信息

Answer 4

我在 CoreOS 1192.2.0 上使用 kubelet 和 rkt。

这是我用来在 worker 上启动 kubelet 的单元：

[Unit]
Description=Kubelet via Hyperkube ACI
Requires=k8s-assets.target
After=k8s-assets.target
[Service]
EnvironmentFile=/etc/proxy.env
Environment="RKT_OPTS=--volume=resolv,kind=host,source=/etc/resolv.conf --mount volume=resolv,target=/etc/resolv.conf --volume var-log,kind=host,source=/var/log --mount volume=var-log,target=/var/log"
Environment=KUBELET_VERSION=v1.4.0_coreos.0
ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
ExecStart=/usr/lib/coreos/kubelet-wrapper \
--api-servers=https://10.203.69.108 \
--register-node=true \
--allow-privileged=true \
--config=/etc/kubernetes/manifests    \
--hostname-override=node2.my.domain  \
--cluster_dns=10.3.0.10 \
--cluster_domain=cluster.local \
--kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml \
--tls-cert-file=/etc/kubernetes/ssl/worker.pem \
--tls-private-key-file=/etc/kubernetes/ssl/worker-key.pem
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target

重要的是

--api-servers必须指向主服务器的IP地址

--tls-cert-file必须指向工作人员证书公钥

--tls-private-key-file必须指向工作人员证书私钥

--kubeconfig必须指向一个有效的 kubeconfig 文件

这是我的 kubeconfig 文件（它包含已签署证书的 CA 的路径）：

apiVersion: v1
kind: Config
clusters:
- name: local
  cluster:
    certificate-authority: /etc/kubernetes/ssl/ca.pem
users:
- name: kubelet
  user:
    client-certificate: /etc/kubernetes/ssl/worker.pem
    client-key: /etc/kubernetes/ssl/worker-key.pem
contexts:
- context:
    cluster: local
    user: kubelet
  name: kubelet-context
current-context: kubelet-context

Answer 5

0. if your issue is :
: Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate"

1. look at your ca.crt 
openssl x509  -noout -text -in ca.crt, you will find below info :

 X509v3 Basic Constraints:
  CA:FLASE

X509v3 Basic Constraints means :
"Basic Constraints" identifies if the subject of certificates is a CA who is allowed to issue child certificates. For a certificate that can be used to sign certificates, the info is in some sense duplicated: X509v3 Basic Constraints: CA: TRUE --- Can sign certificates.

you should modify it to CA:TRUE through vi openssl.conf
[ v3_ca ]
basicConstraints = CA:true

Regenerate your crts.