0

我们有一个部署在 Azure Kubernetes 中的反应节点 Web 应用程序。该应用程序当前部署在 AKS 中，Kubernetes 版本为 1.15，但我们希望将 Kubernetes 升级到 1.19。在将应用程序部署到 AKS 1.19 时，我们对多个 yaml 文件进行了必要的更改，以便应用程序在部署到 AKS 1.19 时兼容。在进行相关更改并将应用程序部署到 AKS 后，所有的 PODS - 后端、前端、traefik 都启动并运行，我们使用 csi 驱动程序将 Azure KEY VAULT 挂载到后端 POD，Traefik 是入口控制器。输出 - 将应用程序部署到 AKS 1.19 后 kubectl 获取 pod

backend-xxx-backend-85595564fd-gbf84                        1/1     Running     0          143m
backend-xxx-backend-85595564fd-gcg9b                        1/1     Running     0          138m
backend-xxx-backend-85595564fd-m65pm                        1/1     Running     0          141m
csi-secrets-store-provider-azure-1615448315-4db78                 1/1     Running     0          5d8h
csi-secrets-store-provider-azure-1615448315-54xw7                 1/1     Running     0          5d8h
csi-secrets-store-provider-azure-1615448315-secrets-store-47wrp   3/3     Running     0          5d8h
csi-secrets-store-provider-azure-1615448315-secrets-store-8z6ck   3/3     Running     0          5d8h
csi-secrets-store-provider-azure-1615448315-secrets-store-tc4jq   3/3     Running     0          5d8h
csi-secrets-store-provider-azure-1615448315-vk6wb                 1/1     Running     0          5d8h
frontend-xxx-frontend-677cc59f57-6m9sn                      1/1     Running     0          139m
frontend-xxx-frontend-677cc59f57-tfnx7                      1/1     Running     0          138m
frontend-xxx-traefik-5c7dd459df-8c8m7                             1/1     Running     0          101m
frontend-xxx-traefik-5c7dd459df-m8lvc                             1/1     Running     0          101m

但是，当我尝试使用 values.yaml 中提到的 URL 作为主机名从浏览器访问应用程序时，它返回 - 404 Page Not Found 错误，我们期望登录页面被加载。

入口 Yaml - 前端：

apiVersion: networking.k8s.io/v1  
kind: Ingress
metadata:
  name: {{ template "fullname" . }}-ingress
  annotations:
    {{- range $key, $value := .Values.ingress.annotations }}
      {{ $key }}: {{ $value | quote }}
    {{- end }}
spec:
  rules:
    {{- range $host := .Values.ingress.hosts }}
    - http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: {{ $fullname }}-svc
                port:
                  number: {{ $servicePort }}
    {{- end }}

Ingress Yaml - 后端

apiVersion: networking.k8s.io/v1  
kind: Ingress
metadata:
  name: {{ template "fullname" . }}-ingress
  annotations:
    {{- range $key, $value := .Values.ingress.annotations }}
      {{ $key }}: {{ $value | quote }}
    {{- end }}
spec:
  rules:
    {{- range $host := .Values.ingress.hosts }}
    - http:
        paths:
          - path: /api/
            pathType: ImplementationSpecific
            backend:
              service:
                name: {{ $fullname }}-svc
                port: 
                  number: {{ $servicePort }}
       {{- end }}


PS C > kubectl describe Endpoints
Name:         backend-xxx-backend-svc
Namespace:    default
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  <none>
Subsets:
  Addresses:          10.26.240.20,10.26.240.41,10.26.240.90
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    <unset>  8800  TCP

Events:  <none>


Name:         frontend-xxx-frontend-svc
Namespace:    default
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  <none>
Subsets:
  Addresses:          10.26.240.50,10.26.240.69
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    <unset>  80    TCP

Events:  <none>


Name:         frontend-xxx-traefik
Namespace:    default
Labels:       app=traefik
              app.kubernetes.io/managed-by=Helm
              chart=traefik-1.64.0
              heritage=Helm
              release=frontend-xxx
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-03-16T14:18:58Z
Subsets:
  Addresses:          10.26.240.54,10.26.240.82
  NotReadyAddresses:  <none>
  Ports:
    Name   Port  Protocol
    ----   ----  --------
    http   80    TCP
    https  8880  TCP

Events:  <none>


Name:         kubernetes
Namespace:    default
Labels:       endpointslice.kubernetes.io/skip-mirror=true
Annotations:  <none>
Subsets:
  Addresses:          52.139.10.83
  NotReadyAddresses:  <none>
  Ports:
    Name   Port  Protocol
    ----   ----  --------
    https  443   TCP

Events:  <none>


PS C > kubectl get svc -A
NAMESPACE     NAME                              TYPE           CLUSTER-IP        EXTERNAL-IP    PORT(S)                      AGE
default       backend-xxx-backend-svc     ClusterIP      192.168.244.160   <none>         80/TCP                       22h
default       eyvia-reportingservice-svc        ClusterIP      192.168.154.18    <none>         80/TCP                       22h
default       frontend-xxx-frontend-svc   ClusterIP      192.168.55.108    <none>         80/TCP                       22h
default       frontend-xxx-traefik              LoadBalancer   192.168.88.230    10.26.240.97   80:30469/TCP,443:30098/TCP   22h
default       kubernetes                        ClusterIP      192.168.0.1       <none>         443/TCP                      8d
kube-system   healthmodel-replicaset-service    ClusterIP      192.168.204.55    <none>         25227/TCP                    8d
kube-system   kube-dns                          ClusterIP      192.168.0.10      <none>         53/UDP,53/TCP                8d
kube-system   metrics-server                    ClusterIP      192.168.133.206   <none>         443/TCP                      8d

PS C> kubectl get ing -A
NAMESPACE   NAME                                  CLASS    HOSTS   ADDRESS   PORTS   AGE
default     backend-xxx-backend-ingress     <none>   *                 80      22h
default     frontend-xxx-frontend-ingress   <none>   *                 80      22h

PS C:> kubectl get ing -A -o yaml
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      ingress.kubernetes.io/custom-response-headers: 'server:'
      kubernetes.io/ingress.class: traefik
      meta.helm.sh/release-name: backend-prd
      meta.helm.sh/release-namespace: default
    creationTimestamp: "2021-03-17T12:22:10Z"
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
    managedFields:
    - apiVersion: networking.k8s.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:ingress.kubernetes.io/custom-response-headers: {}
            f:kubernetes.io/ingress.class: {}
            f:meta.helm.sh/release-name: {}
            f:meta.helm.sh/release-namespace: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/managed-by: {}
        f:spec:
          f:rules: {}
      manager: Go-http-client
      operation: Update
      time: "2021-03-17T12:22:10Z"
    name: backend-xxx-backend-ingress
    namespace: default
    resourceVersion: "1960069"
    selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/backend-xxx-backend-ingress
    uid: 20308a3e-76ad-4286-a91c-c05a7d7b94ba
  spec:
    rules:
    - http:
        paths:
        - backend:
            serviceName: backend-xxx-backend-svc
            servicePort: 80
          path: /api/
          pathType: Prefix
  status:
    loadBalancer: {}
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      certmanager.k8s.io/cluster-issuer: letsencrypt
      ingress.kubernetes.io/custom-response-headers: 'server:'
      kubernetes.io/ingress.class: traefik
      meta.helm.sh/release-name: frontend-prd
      meta.helm.sh/release-namespace: default
    creationTimestamp: "2021-03-17T12:22:27Z"
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
    managedFields:
    - apiVersion: networking.k8s.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:certmanager.k8s.io/cluster-issuer: {}
            f:ingress.kubernetes.io/custom-response-headers: {}
            f:kubernetes.io/ingress.class: {}
            f:meta.helm.sh/release-name: {}
            f:meta.helm.sh/release-namespace: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/managed-by: {}
        f:spec:
          f:rules: {}
      manager: Go-http-client
      operation: Update
      time: "2021-03-17T12:22:27Z"
    name: frontend-xxx-frontend-ingress
    namespace: default
    resourceVersion: "1960242"
    selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/frontend-xxx-frontend-ingress
    uid: 7f102b21-23d0-49cc-86b7-24a002050d04
  spec:
    rules:
    - http:
        paths:
        - backend:
            serviceName: frontend-xxx-frontend-svc
            servicePort: 80
          path: /
          pathType: Prefix
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""



PS C:> kubectl describe po frontend-xxx-traefik-5c7dd459df-8g94r
Name:               frontend-xxx-traefik-5c7dd459df-8g94r
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               aks-agentpool-27227329-vmss000002/10.26.240.66
Start Time:         Wed, 17 Mar 2021 17:52:27 +0530
Labels:             app=traefik
                    chart=traefik-1.64.0
                    heritage=Helm
                    pod-template-hash=5c7dd459df
                    release=frontend-xxx
Annotations:        checksum/config: 17af10aa3c1f384e3e5ec092c07352acc76bc89571785250128fd4e44a3e94ec
Status:             Running
IP:                 10.26.240.79
Controlled By:      ReplicaSet/frontend-xxx-traefik-5c7dd459df
Containers:
  frontend-xxx-traefik:
    Container ID:  containerd://e7c7966dec5283c5e36cc0089ee5b1ae1cb535bdac73d750045f518a51ba35c8
    Image:         traefik:1.7.9
    Image ID:      docker.io/library/traefik@sha256:6348d14975b683a2783cc8dbaa76efe5367a1d67bd3e9e2ee812da7da012224b
    Ports:         80/TCP, 8880/TCP, 443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      --configfile=/config/traefik.toml
    State:          Running
      Started:      Wed, 17 Mar 2021 17:52:29 +0530
    Ready:          True
    Restart Count:  0
    Liveness:       tcp-socket :80 delay=10s timeout=2s period=10s #success=1 #failure=3
    Readiness:      tcp-socket :80 delay=10s timeout=2s period=10s #success=1 #failure=1
    Environment:    <none>
    Mounts:
      /config from config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-h9kch (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      frontend-xxx-traefik
    Optional:  false
  default-token-h9kch:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-h9kch
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

我们在部署中使用 Traefik 作为入口控制器，当我看到 Traefik pod 的日志时，我只看到了这个 - 看起来日志级别设置为 info ：

E0317 16:00:04.641113       1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: v1.ServiceList.Items: []v1.Service: v1.Service.ObjectMeta: v1.ObjectMeta.readObjectFieldAsBytes: expect : after object field, but found p, error found in #10 byte of ...|:{},"k:{\"port\":80,|..., bigger context ...|anaged-by":{}}},"f:spec":{"f:ports":{".":{},"k:{\"port\":80,\"protocol\":\"TCP\"}":{".":{},"f:port":|...
E0317 16:00:05.644045       1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: v1.ServiceList.Items: []v1.Service: v1.Service.ObjectMeta: v1.ObjectMeta.readObjectFieldAsBytes: expect : after object field, but found p, error found in #10 byte of ...|:{},"k:{\"port\":80,|..., bigger context ...|anaged-by":{}}},"f:spec":{"f:ports":{".":{},"k:{\"port\":80,\"protocol\":\"TCP\"}":{".":{},"f:port":|...
E0317 16:00:06.646728       1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: v1.ServiceList.Items: []v1.Service: v1.Service.ObjectMeta: v1.ObjectMeta.readObjectFieldAsBytes: expect : after object field, but found p, error found in #10 byte of ...|:{},"k:{\"port\":80,|..., bigger context ...|anaged-by":{}}},"f:spec":{"f:ports":{".":{},"k:{\"port\":80,\"protocol\":\"TCP\"}":{".":{},"f:port":|...
E0317 16:00:07.649633       1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: v1.ServiceList.Items: []v1.Service: v1.Service.ObjectMeta: v1.ObjectMeta.readObjectFieldAsBytes: expect : after object field, but found p, error found in #10 byte of ...|:{},"k:{\"port\":80,|..., bigger context ...|anaged-by":{}}},"f:spec":{"f:ports":{".":{},"k:{\"port\":80,\"protocol\":\"TCP\"}":{".":{},"f:port":|...
E0317 16:00:08.652654       1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: v1.ServiceList.Items: []v1.Service: v1.Service.ObjectMeta: v1.ObjectMeta.readObjectFieldAsBytes: expect : after object field, but found p, error found in #10 byte of ...|:{},"k:{\"port\":80,|..., bigger context ...|anaged-by":{}}},"f:spec":{"f:ports":{".":{},"k:{\"port\":80,\"protocol\":\"TCP\"}":{".":{},"f:port":|...
E0317 16:00:09.654701       1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: v1.ServiceList.Items: []v1.Service: v1.Service.ObjectMeta: v1.ObjectMeta.readObjectFieldAsBytes: expect : after object field, but found p, error found in #10 byte of ...|:{},"k:{\"port\":80,|..., bigger context ...|anaged-by":{}}},"f:spec":{"f:ports":{".":{},"k:{\"port\":80,\"protocol\":\"TCP\"}":{".":{},"f:port":|...
E0317 16:00:10.656995       1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: v1.ServiceList.Items: []v1.Service: v1.Service.ObjectMeta: v1.ObjectMeta.readObjectFieldAsBytes: expect : after object field, but found p, error found in #10 byte of ...|:{},"k:{\"port\":80,|..., bigger context ...|anaged-by":{}}},"f:spec":{"f:ports":{".":{},"k:{\"port\":80,\"protocol\":\"TCP\"}":{".":{},"f:port":|...
E0317 16:00:11.659811       1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: v1.ServiceList.Items: []v1.Service: v1.Service.ObjectMeta: v1.ObjectMeta.readObjectFieldAsBytes: expect : after object field, but found p, error found in #10 byte of ...|:{},"k:{\"port\":80,|..., bigger context ...|anaged-by":{}}},"f:spec":{"f:ports":{".":{},"k:{\"port\":80,\"protocol\":\"TCP\"}":{".":{},"f:port":|...
E0317 16:00:12.662628       1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: v1.ServiceList.Items: []v1.Service: v1.Service.ObjectMeta: v1.ObjectMeta.readObjectFieldAsBytes: expect : after object field, but found p, error found in #10 byte of ...|:{},"k:{\"port\":80,|..., bigger context ...|anaged-by":{}}},"f:spec":{"f:ports":{".":{},"k:{\"port\":80,\"protocol\":\"TCP\"}":{".":{},"f:port":|...
E0317 16:00:13.665582       1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: v1.ServiceList.Items: []v1.Service: v1.Service.ObjectMeta: v1.ObjectMeta.readObjectFieldAsBytes: expect : after object field, but found p, error found in #10 byte of ...|:{},"k:{\"port\":80,|..., bigger context ...|anaged-by":{}}},"f:spec":{"f:ports":{".":{},"k:{\"port\":80,\"protocol\":\"TCP\"}":{".":{},"f:port":|...

关于我们缺少什么以及为什么我们最终找不到 404 页面的任何指针都将非常有帮助。

我们是否缺少任何配置。

4

1 回答 1

0

由于从描述中不清楚问题，我将建议以下项目进行调试。

  1. 跟踪 traefik 控制器 pod 的日志，以确保您在访问目标 URL 时获得流量。当心浏览器缓存，要么cURL使用新的隐身窗口进行浏览，要么进行浏览。来自所有控制器 pod 的尾日志，您可能有多个 pod 处理流量。如果您在控制器日志中看不到任何命中，则可能存在多个问题。检查您的入口控制器是否正确暴露。通过运行检查集群中运行的服务kubectl get svc -A. 查看 traefik 服务是否为 LB 类型。获取服务描述并在事件部分记录任何问题。如果未配置并且您看到错误，请删除流量图表并重新部署。如果仍然不正确，请从门户向 Azure 团队开具支持票证。但是，如果 LB 绑定正确。检查您是否有权访问运行 LB 的子网。也可能存在安装了多个入口控制器并且打错控制器的情况。在这种情况下，您还应该创建多个 LB。确保您达到了所需的 LB。

  2. 如果您在日志中看到错误，请按照错误进行操作。如果找不到带下划线的服务或 Pod 拒绝流量，它将显示上游错误。

  3. 入口控制器是通过 kubernetes 对象配置的反向代理。如果您使用基于路径的路由，请确保您的应用程序侦听路径。例如，后端流量被重定向到/api路径。如果应用程序侦听，/那么它将失败。还要检查 rewrite-target 和相关注释以进行正确配置。

  4. 如果您使用基于主机的路由，请确保您使用正确的主机名访问服务。获取 LB IP 的 DNS 名称，或者修改您hosts的文件，然后尝试 URL。

于 2021-03-22T14:10:28.830 回答