1

我已经在 3 节点 k8s 集群(裸机)上使用 yaml 文件安装了 kong-ingress-controller(您可以在问题底部看到该文件),并且一切都已启动并运行:

$kubectl get pods --all-namespaces
NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
default       bar-deployment-64d5b5b5c4-99p4q             1/1     Running   0          12m
default       foo-deployment-877bf9974-xmpj6              1/1     Running   0          15m
kong          ingress-kong-5cd9db4db9-4cg4q               2/2     Running   0          79m
kube-system   calico-kube-controllers-5f6cfd688c-5njnn    1/1     Running   0          18h
kube-system   calico-node-5k9b6                           1/1     Running   0          18h
kube-system   calico-node-jbb7k                           1/1     Running   0          18h
kube-system   calico-node-mmmts                           1/1     Running   0          18h
kube-system   coredns-74ff55c5b-5q5fn                     1/1     Running   0          23h
kube-system   coredns-74ff55c5b-9bbbk                     1/1     Running   0          23h
kube-system   etcd-kubernetes-master                      1/1     Running   1          23h
kube-system   kube-apiserver-kubernetes-master            1/1     Running   1          23h
kube-system   kube-controller-manager-kubernetes-master   1/1     Running   1          23h
kube-system   kube-proxy-4h7hs                            1/1     Running   0          20h
kube-system   kube-proxy-sd6b2                            1/1     Running   0          20h
kube-system   kube-proxy-v9z8p                            1/1     Running   1          23h
kube-system   kube-scheduler-kubernetes-master            1/1     Running   1          23h

但问题就在这里

EXTERNAL_IPof待定kong-proxy service因此我无法从外部访问我的集群

$kubectl get services --all-namespaces
NAMESPACE     NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
default       bar-service               ClusterIP      10.103.49.102   <none>        5000/TCP                     15m
default       foo-service               ClusterIP      10.102.52.89    <none>        5000/TCP                     19m
default       kubernetes                ClusterIP      10.96.0.1       <none>        443/TCP                      23h
kong          kong-proxy                LoadBalancer   10.104.79.161   <pending>     80:31583/TCP,443:30053/TCP   82m
kong          kong-validation-webhook   ClusterIP      10.109.75.104   <none>        443/TCP                      82m
kube-system   kube-dns                  ClusterIP      10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP       23h

$ kubectl describe service kong-proxy -n kong

Name:                     kong-proxy
Namespace:                kong
Labels:                   <none>
Annotations:              service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
                          service.beta.kubernetes.io/aws-load-balancer-type: nlb
Selector:                 app=ingress-kong
Type:                     LoadBalancer
IP Families:              <none>
IP:                       10.104.79.161
IPs:                      10.104.79.161
Port:                     proxy  80/TCP
TargetPort:               8000/TCP
NodePort:                 proxy  31583/TCP
Endpoints:                192.168.74.69:8000
Port:                     proxy-ssl  443/TCP
TargetPort:               8443/TCP
NodePort:                 proxy-ssl  30053/TCP
Endpoints:                192.168.74.69:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

我的 k8s 版本是 1.20.1 ,我的 docker 版本是 19.3.10 。如果有人可以帮助我找到解决方案,那就太棒了

==============================================

kong-ingress-controller yaml 文件:

apiVersion: v1
kind: Namespace
metadata:
  name: kong
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: kongclusterplugins.configuration.konghq.com
spec:
  additionalPrinterColumns:
  - JSONPath: .plugin
    description: Name of the plugin
    name: Plugin-Type
    type: string
  - JSONPath: .metadata.creationTimestamp
    description: Age
    name: Age
    type: date
  - JSONPath: .disabled
    description: Indicates if the plugin is disabled
    name: Disabled
    priority: 1
    type: boolean
  - JSONPath: .config
    description: Configuration of the plugin
    name: Config
    priority: 1
    type: string
  group: configuration.konghq.com
  names:
    kind: KongClusterPlugin
    plural: kongclusterplugins
    shortNames:
    - kcp
  scope: Cluster
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      properties:
        config:
          type: object
        configFrom:
          properties:
            secretKeyRef:
              properties:
                key:
                  type: string
                name:
                  type: string
                namespace:
                  type: string
              required:
              - name
              - namespace
              - key
              type: object
          type: object
        disabled:
          type: boolean
        plugin:
          type: string
        protocols:
          items:
            enum:
            - http
            - https
            - grpc
            - grpcs
            - tcp
            - tls
            type: string
          type: array
        run_on:
          enum:
          - first
          - second
          - all
          type: string
      required:
      - plugin
  version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: kongconsumers.configuration.konghq.com
spec:
  additionalPrinterColumns:
  - JSONPath: .username
    description: Username of a Kong Consumer
    name: Username
    type: string
  - JSONPath: .metadata.creationTimestamp
    description: Age
    name: Age
    type: date
  group: configuration.konghq.com
  names:
    kind: KongConsumer
    plural: kongconsumers
    shortNames:
    - kc
  scope: Namespaced
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      properties:
        credentials:
          items:
            type: string
          type: array
        custom_id:
          type: string
        username:
          type: string
  version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: kongingresses.configuration.konghq.com
spec:
  group: configuration.konghq.com
  names:
    kind: KongIngress
    plural: kongingresses
    shortNames:
    - ki
  scope: Namespaced
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      properties:
        proxy:
          properties:
            connect_timeout:
              minimum: 0
              type: integer
            path:
              pattern: ^/.*$
              type: string
            protocol:
              enum:
              - http
              - https
              - grpc
              - grpcs
              - tcp
              - tls
              type: string
            read_timeout:
              minimum: 0
              type: integer
            retries:
              minimum: 0
              type: integer
            write_timeout:
              minimum: 0
              type: integer
          type: object
        route:
          properties:
            headers:
              additionalProperties:
                items:
                  type: string
                type: array
              type: object
            https_redirect_status_code:
              type: integer
            methods:
              items:
                type: string
              type: array
            path_handling:
              enum:
              - v0
              - v1
              type: string
            preserve_host:
              type: boolean
            protocols:
              items:
                enum:
                - http
                - https
                - grpc
                - grpcs
                - tcp
                - tls
                type: string
              type: array
            regex_priority:
              type: integer
            request_buffering:
              type: boolean
            response_buffering:
              type: boolean
            snis:
              items:
                type: string
              type: array
            strip_path:
              type: boolean
        upstream:
          properties:
            algorithm:
              enum:
              - round-robin
              - consistent-hashing
              - least-connections
              type: string
            hash_fallback:
              type: string
            hash_fallback_header:
              type: string
            hash_on:
              type: string
            hash_on_cookie:
              type: string
            hash_on_cookie_path:
              type: string
            hash_on_header:
              type: string
            healthchecks:
              properties:
                active:
                  properties:
                    concurrency:
                      minimum: 1
                      type: integer
                    healthy:
                      properties:
                        http_statuses:
                          items:
                            type: integer
                          type: array
                        interval:
                          minimum: 0
                          type: integer
                        successes:
                          minimum: 0
                          type: integer
                      type: object
                    http_path:
                      pattern: ^/.*$
                      type: string
                    timeout:
                      minimum: 0
                      type: integer
                    unhealthy:
                      properties:
                        http_failures:
                          minimum: 0
                          type: integer
                        http_statuses:
                          items:
                            type: integer
                          type: array
                        interval:
                          minimum: 0
                          type: integer
                        tcp_failures:
                          minimum: 0
                          type: integer
                        timeout:
                          minimum: 0
                          type: integer
                      type: object
                  type: object
                passive:
                  properties:
                    healthy:
                      properties:
                        http_statuses:
                          items:
                            type: integer
                          type: array
                        interval:
                          minimum: 0
                          type: integer
                        successes:
                          minimum: 0
                          type: integer
                      type: object
                    unhealthy:
                      properties:
                        http_failures:
                          minimum: 0
                          type: integer
                        http_statuses:
                          items:
                            type: integer
                          type: array
                        interval:
                          minimum: 0
                          type: integer
                        tcp_failures:
                          minimum: 0
                          type: integer
                        timeout:
                          minimum: 0
                          type: integer
                      type: object
                  type: object
                threshold:
                  type: integer
              type: object
            host_header:
              type: string
            slots:
              minimum: 10
              type: integer
          type: object
  version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: kongplugins.configuration.konghq.com
spec:
  additionalPrinterColumns:
  - JSONPath: .plugin
    description: Name of the plugin
    name: Plugin-Type
    type: string
  - JSONPath: .metadata.creationTimestamp
    description: Age
    name: Age
    type: date
  - JSONPath: .disabled
    description: Indicates if the plugin is disabled
    name: Disabled
    priority: 1
    type: boolean
  - JSONPath: .config
    description: Configuration of the plugin
    name: Config
    priority: 1
    type: string
  group: configuration.konghq.com
  names:
    kind: KongPlugin
    plural: kongplugins
    shortNames:
    - kp
  scope: Namespaced
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      properties:
        config:
          type: object
        configFrom:
          properties:
            secretKeyRef:
              properties:
                key:
                  type: string
                name:
                  type: string
              required:
              - name
              - key
              type: object
          type: object
        disabled:
          type: boolean
        plugin:
          type: string
        protocols:
          items:
            enum:
            - http
            - https
            - grpc
            - grpcs
            - tcp
            - tls
            type: string
          type: array
        run_on:
          enum:
          - first
          - second
          - all
          type: string
      required:
      - plugin
  version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tcpingresses.configuration.konghq.com
spec:
  additionalPrinterColumns:
  - JSONPath: .status.loadBalancer.ingress[*].ip
    description: Address of the load balancer
    name: Address
    type: string
  - JSONPath: .metadata.creationTimestamp
    description: Age
    name: Age
    type: date
  group: configuration.konghq.com
  names:
    kind: TCPIngress
    plural: tcpingresses
  scope: Namespaced
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      properties:
        apiVersion:
          type: string
        kind:
          type: string
        metadata:
          type: object
        spec:
          properties:
            rules:
              items:
                properties:
                  backend:
                    properties:
                      serviceName:
                        type: string
                      servicePort:
                        format: int32
                        type: integer
                    type: object
                  host:
                    type: string
                  port:
                    format: int32
                    type: integer
                type: object
              type: array
            tls:
              items:
                properties:
                  hosts:
                    items:
                      type: string
                    type: array
                  secretName:
                    type: string
                type: object
              type: array
          type: object
        status:
          type: object
  version: v1beta1
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kong-serviceaccount
  namespace: kong
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: kong-ingress-clusterrole
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - nodes
  - pods
  - secrets
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  - extensions
  - networking.internal.knative.dev
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - networking.k8s.io
  - extensions
  - networking.internal.knative.dev
  resources:
  - ingresses/status
  verbs:
  - update
- apiGroups:
  - configuration.konghq.com
  resources:
  - tcpingresses/status
  verbs:
  - update
- apiGroups:
  - configuration.konghq.com
  resources:
  - kongplugins
  - kongclusterplugins
  - kongcredentials
  - kongconsumers
  - kongingresses
  - tcpingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - create
  - get
  - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kong-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kong-ingress-clusterrole
subjects:
- kind: ServiceAccount
  name: kong-serviceaccount
  namespace: kong
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  name: kong-proxy
  namespace: kong
spec:
  ports:
  - name: proxy
    port: 80
    protocol: TCP
    targetPort: 8000
  - name: proxy-ssl
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    app: ingress-kong
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  name: kong-validation-webhook
  namespace: kong
spec:
  ports:
  - name: webhook
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app: ingress-kong
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ingress-kong
  name: ingress-kong
  namespace: kong
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-kong
  template:
    metadata:
      annotations:
        kuma.io/gateway: enabled
        prometheus.io/port: "8100"
        prometheus.io/scrape: "true"
        traffic.sidecar.istio.io/includeInboundPorts: ""
      labels:
        app: ingress-kong
    spec:
      containers:
      - env:
        - name: KONG_PROXY_LISTEN
          value: 0.0.0.0:8000, 0.0.0.0:8443 ssl http2
        - name: KONG_PORT_MAPS
          value: 80:8000, 443:8443
        - name: KONG_ADMIN_LISTEN
          value: 127.0.0.1:8444 ssl
        - name: KONG_STATUS_LISTEN
          value: 0.0.0.0:8100
        - name: KONG_DATABASE
          value: "off"
        - name: KONG_NGINX_WORKER_PROCESSES
          value: "2"
        - name: KONG_ADMIN_ACCESS_LOG
          value: /dev/stdout
        - name: KONG_ADMIN_ERROR_LOG
          value: /dev/stderr
        - name: KONG_PROXY_ERROR_LOG
          value: /dev/stderr
        image: kong:2.5
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - kong quit
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /status
            port: 8100
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: proxy
        ports:
        - containerPort: 8000
          name: proxy
          protocol: TCP
        - containerPort: 8443
          name: proxy-ssl
          protocol: TCP
        - containerPort: 8100
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /status
            port: 8100
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
      - env:
        - name: CONTROLLER_KONG_ADMIN_URL
          value: https://127.0.0.1:8444
        - name: CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY
          value: "true"
        - name: CONTROLLER_PUBLISH_SERVICE
          value: kong/kong-proxy
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: kong/kubernetes-ingress-controller:1.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: ingress-controller
        ports:
        - containerPort: 8080
          name: webhook
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
      serviceAccountName: kong-serviceaccount

4

2 回答 2

2

简短的回答是@iglen_在这个答案中所说的,但我决定解释解决方案。

当使用云提供商时LoadBalancer,服务的类型将由环境自动管理和配置(参见k8s 文档),但是在创建自己的裸机集群时,您需要添加服务来管理服务类型的IPs配置LoadBalancer。一种这样的服务是为此专门构建的Metal-LB 。

在安装 MetalLB 之前检查要求

在我们部署 MetalLB 之前,我们需要做一个步骤:

如果您在 IPVS 模式下使用 kube-proxy,从 Kubernetes v1.14.2 开始,您必须启用严格的 ARP 模式。

请注意,如果您使用 kube-router 作为服务代理,则不需要这个,因为它默认启用严格的 ARP。输入这个命令:

$ kubectl edit configmap -n kube-system kube-proxy

在打开的页面搜索模式中,在我的情况下,模式等于空字符串,所以我不需要更改任何内容,但如果模式设置ipvs为安装指南中所说的,则需要在下面设置此文件中的配置:

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
  strictARP: true

作为下一步,您需要运行以下命令:

$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/namespace.yaml
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/metallb.yaml
$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

运行上述命令后,我们有这个:

$ kubectl get all -n metallb-system
NAME                              READY   STATUS    RESTARTS   AGE
pod/controller-6b78sff7d9-2rv2f   1/1     Running   0          3m
pod/speaker-7bqev                 1/1     Running   0          3m
pod/speaker-txrg5                 1/1     Running   0          3m
pod/speaker-w7th5                 1/1     Running   0          3m

NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/speaker   3         3         3       3            3           kubernetes.io/os=linux   3m

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/controller   1/1     1            1           3m

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/controller-6b78sff7d9   1         1         1       3m

MetalLB 需要一些IPv4 addresses

$ ip a s
1: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
   [...]
    inet 10.240.1.59/24 brd 10.240.1.255 scope global dynamic noprefixroute ens160
       valid_lft 425669sec preferred_lft 421269sec
    [...]

ens160是我的控制平面网络接口,如您所见,它的 IP 范围是10.240.1.59/24,所以我将在该网络中分配一组 IP 地址:

$ sipcalc 10.240.1.59/24
-[ipv4 : 10.240.1.59/24] - 0

[CIDR]
Host address            - 10.240.1.59
Host address (decimal)  - 183500115
Host address (hex)      - AF0031B
Network address         - 10.240.1.0
Network mask            - 255.255.255.0
Network mask (bits)     - 24
Network mask (hex)      - FFFFF000
Broadcast address       - 10.240.1.255
Cisco wildcard          - 0.0.0.255
Addresses in network    - 256
Network range           - 10.240.1.0 - 10.240.1.255
Usable range            - 10.240.1.1 - 10.240.1.254

现在我将从中获取10 个IP 地址Usable range并将其分配给 MetalLB。configmap让我们为 MetalLB创建一个:

$ sudo nano metallb-cm.yaml

将以下配置粘贴到 metallb-cm.yaml 中:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.240.1.100-10.240.1.110

然后保存文件并运行以下命令:

$ kubectl create -f metallb-cm.yaml

现在让我们再次检查我们的服务:

$ kubectl get services --all-namespaces
NAMESPACE     NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
default       bar-service               ClusterIP      10.103.49.102   <none>        5000/TCP                     15m
default       foo-service               ClusterIP      10.102.52.89    <none>        5000/TCP                     19m
default       kubernetes                ClusterIP      10.96.0.1       <none>        443/TCP                      23h
kong          kong-proxy                LoadBalancer   10.104.79.161   10.240.1.100  80:31583/TCP,443:30053/TCP   82m
kong          kong-validation-webhook   ClusterIP      10.109.75.104   <none>        443/TCP                      82m
kube-system   kube-dns                  ClusterIP      10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP       23h

如您所见,类型的服务LoadBalancer现在有一个 IP 地址。

于 2021-09-17T08:00:15.703 回答
1

有同样的问题,经过几天寻找解决方案,我遇到了metallb,来自裸机上的nginx 入口安装

MetalLB 为不在支持的云提供商上运行的 Kubernetes 集群提供网络负载均衡器实现,有效地允许在任何集群中使用 LoadBalancer 服务

,从他们的文档中我得到了这个

Kubernetes 不为裸机集群提供网络负载均衡器(LoadBalancer 类型的服务)的实现。Kubernetes 附带的网络负载均衡器的实现都是调用各种 IaaS 平台(GCP、AWS、Azure ……)的胶水代码。如果您未在受支持的 IaaS 平台(GCP、AWS、Azure...)上运行,则 LoadBalancers 在创建时将无限期地保持在“待定”状态。

我没有完成安装,但我希望上面的解释能回答你关于外部 ip 待处理状态的问题

于 2021-09-14T12:32:42.233 回答