1

我通过 kubeadm 设置了我的集群。在最后一步我 exec kubeadm init --config kubeadm.conf --v=5。我收到有关 clusterIp 值的错误。这是输出的一部分：

    I0220 00:16:27.625920   31630 clusterinfo.go:79] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace
I0220 00:16:27.947941   31630 kubeletfinalize.go:88] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem"
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I0220 00:16:27.949398   31630 kubeletfinalize.go:132] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation
[addons]: Migrating CoreDNS Corefile
I0220 00:16:28.447420   31630 dns.go:381] the CoreDNS configuration has been migrated and applied: .:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}
.
I0220 00:16:28.447465   31630 dns.go:382] the old migration has been saved in the CoreDNS ConfigMap under the name [Corefile-backup]
I0220 00:16:28.447486   31630 dns.go:383] The changes in the new CoreDNS Configuration are as follows:
Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.10.0.10": field is immutable
unable to create/update the DNS service
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.createDNSService
    /workspace/anago-v1.17.0-rc.2.10+70132b0f130acc/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns/dns.go:323
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.createCoreDNSAddon
    /workspace/anago-v1.17.0-rc.2.10+70132b0f130acc/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns/dns.go:305
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.coreDNSAddon

我的配置文件是这样的：

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 172.16.5.151
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: master02
#  taints:
#  - effect: NoSchedule
#    key: node-role.kubernetes.io/master
---
apiServer:
  certSANs:
    - "172.16.5.150"
    - "172.16.5.151"
    - "172.16.5.152"
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  external:
    endpoints:
    - "https://172.16.5.150:2379"
    - "https://172.16.5.151:2379"
    - "https://172.16.5.152:2379"
    caFile: /etc/k8s/pki/ca.pem
    certFile: /etc/k8s/pki/etcd.pem
    keyFile: /etc/k8s/pki/etcd.key

imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.17.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.10.0.0/16
  podSubnet: 192.168.0.0/16
scheduler: {}

我检查了 kubeadm 生成的 kube-apiserver.yaml。--service-cluster-ip-range=10.10.0.0/16 设置包含 10.10.0.10 你可以在下面看到：

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=172.16.5.151
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/k8s/pki/ca.pem
    - --etcd-certfile=/etc/k8s/pki/etcd.pem
    - --etcd-keyfile=/etc/k8s/pki/etcd.key
    - --etcd-servers=https://172.16.5.150:2379,https://172.16.5.151:2379,https://172.16.5.152:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.10.0.0/16
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.17.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 172.16.5.151
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/k8s/pki
      name: etcd-certs-0
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/k8s/pki
      type: DirectoryOrCreate
    name: etcd-certs-0
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
status: {}

正如你在上面看到的。所有 service-ip-range 已设置为 10.10.0.0/16。奇怪的是，当我执行“kubectl get svc”时，我得到的 kubernetes clusterip 是 10.96.0.1

[root@master02 manifests]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d3h

这意味着默认的服务 IP 范围是：10.96.0.0/16。我修改的内容不起作用。有谁知道如何自定义 service-ip-range 范围。以及如何解决我的问题？

4

2 回答 2

0

将此答案发布为社区 wiki 以扩展和解释根本原因。

何时kubeadm启动，并且我们不指定任何标志，$ kubeadm init它将kubeadm使用默认值创建集群。您可以签入Kubernetes 文档标志，这些标志可以在初始化期间指定并且是默认值。

--service-cidr字符串 默认值：“10.96.0.0/12” 为服务 VIP 使用替代的 IP 地址范围。

这就是为什么默认kubernetes服务10.96.0.1用作ClusterIP.

这里OP也想使用自己的配置。

--config字符串 kubeadm 配置文件的路径。

整个初始化工作流程可以在这里找到。

正如 Kubernetes 文档所解释的那样，Kubeadm 重置

尽最大努力还原由 kubeadm init 或 kubeadm join 所做的更改。

有时取决于我们的配置，一些配置保留在集群上。

此处提到了 OP 遇到的问题- 外部 etcd 清理

kubeadm reset如果使用外部 etcd，则不会删除任何 etcd 数据。这意味着如果您使用相同的 etcd 端点再次运行 kubeadm init，您将看到以前集群的状态。

关于不可变字段：Service “kube-dns” is invalid: spec.clusterIP: Invalid value: “10.10.0.10”: field is immutable. 在 Kubernetes 中，某些字段是安全的，以防止可能破坏集群工作的更改。

如果任何字段是immutable但我们必须更改它，则必须删除该对象并再次添加。

于 2020-02-21T14:51:11.540 回答
0

因为这个节点我之前是作为节点加入集群的，因为这个节点我之前是作为节点加入集群的。后来我用“kubeadm reset”命令重置了这个。重置之后，我作为master角色加入了集群。所以我在上面的问题中得到了错误。 错误是因为我reset之前clusterip的范围已经记录在etcd集群中了。并且“kubeadm reset”命令并没有清理etcd中的数据。所以新定义的clusterip与原来的冲突。所以解决方法是清理etcd中的数据，重新设置。（由于我搭建的集群是测试集群，所以直接清理了etcd，生产环境请注意）

于 2020-02-21T02:18:05.623 回答