kubernetes - 用于外部服务的 Istio Egress MTL

Question

我目前（但未成功）尝试通过设置istio-egressgatewayMTL 来访问外部 K8s 集群服务。我正在遵循istio 文档中指定的说明，但没有按预期工作，而且我看不到我错在哪里。

环境

VMWare ESXi下的3个VM（1个master，2个Nodes）
操作系统：CentOS 7，内核5.2.10-1.el7来自 elrepo
k8s 网络：cilium -1.6.4
证书：cert-manager-0.12nowebhook
- ClusterIssuer CA 证书 = 从私有 PKI 导入
K8s 外部负载均衡器：metallb-0.8.3

kubectl并istio在K8s集群中使用

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:37:43Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

$ istioctl version
client version: 1.2.10
citadel version: 1.2.10
egressgateway version: 1.2.10
galley version: 1.2.10
ingressgateway version: 1.2.10
pilot version: 1.2.10
policy version: 1.2.10
sidecar-injector version: 1.2.10
telemetry version: 1.2.10

istio-egressgateway已按照istio 文档中的说明重新部署

这是应用.yaml几乎是示例的复制粘贴，稍作修改以通过其 IP 地址指向外部服务

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: nginx
  namespace: tools-istio
spec:
  hosts:
  - <my.hostname>
  ports:
  - number: 80
    name: http
    protocol: HTTP
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
  endpoints:
  - address: <private_ip_address>
    ports:
      https: 443
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
  namespace: tools-istio
spec:
  hosts:
  - <my.hostname>
  tls:
  - match:
    - port: 443
      sni_hosts:
      - <my.hostname>
    route:
    - destination:
        host: <my.hostname>
        port:
          number: 443
      weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway
  namespace: tools-istio
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - <my.hostname>
    tls:
      mode: MUTUAL
      serverCertificate: /etc/certs/cert-chain.pem
      privateKey: /etc/certs/key.pem
      caCertificates: /etc/certs/root-cert.pem
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-nginx
  namespace: tools-istio
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: nginx
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 443
        tls:
          mode: ISTIO_MUTUAL
          sni: <my.hostname>
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-nginx-through-egress-gateway
  namespace: tools-istio
spec:
  hosts:
  - <my.hostname>
  gateways:
  - istio-egressgateway
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: nginx
        port:
          number: 443
      weight: 100
  - match:
    - gateways:
      - istio-egressgateway
      port: 443
    route:
    - destination:
        host: <my.hostname>
        port:
          number: 443
      weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: originate-mtls-for-nginx
  namespace: tools-istio
spec:
  host: <my.hostname>
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: ISTIO_MUTUAL
        sni: <my.hostname>
        # I've also tried with
        # mode: MUTUAL
        # clientCertificate: /etc/nginx-external-tls/tls.crt
        # privateKey: /etc/nginx-external-tls/tls.key
        # caCertificates: /etc/nginx-external-tls/ca.crt

当通过我们向外部服务进行查询时，curl我们应该得到 HTML web 代码，但是，它尝试使用 HTTP，而不是 HTTPS 来执行查询，为什么？. istio-proxy从调试器 pod 应该使用 Citadel 证书将 HTTP 转换为 HTTPS

 $ kubectl -n tools-istio exec -it debugger-5844b6d674-4fvt9 -c debugger -- curl http://<my.hostname> -v
*   Trying <my.hostname>:80...
* TCP_NODELAY set
* Connected to <my.hostname> (<private_ip_address>) port 80 (#0)
> GET / HTTP/1.1
> Host: <my.hostname>
> User-Agent: curl/7.66.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< content-length: 91
< content-type: text/plain
< date: Wed, 08 Jan 2020 09:29:09 GMT
< server: envoy
<
* Connection #0 to host <my.hostname> intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure

验证时proxy-status我得到以下信息

$ istioctl proxy-status
NAME                                                   CDS                            LDS        EDS               RDS          PILOT                            VERSION
debugger-5844b6d674-4fvt9.tools-istio                  STALE (Never Acknowledged)     SYNCED     SYNCED (51%)      SYNCED       istio-pilot-5f45f6768c-fmvr7     1.2.10
istio-egressgateway-5ff889c5fd-jtz55.istio-system      SYNCED                         SYNCED     SYNCED (100%)     SYNCED       istio-pilot-5f45f6768c-fmvr7     1.2.10
istio-ingressgateway-779cfdd879-bb7w7.istio-system     STALE (Never Acknowledged)     SYNCED     SYNCED (96%)      NOT SENT     istio-pilot-5f45f6768c-fmvr7     1.2.10

根据官方istio文档，这可能意味着，要么是错误配置.yaml，要么是istio错误

检查端点时，一切看起来都很好

$ istioctl proxy-config endpoint debugger-5844b6d674-4fvt9.tools-istio | grep <my.hostname>
<private_ip_address>:80         HEALTHY     outbound|80||<my.hostname>
<private_ip_address>:443        HEALTHY     outbound|443||<my.hostname>

否则调试器 podistio-proxy日志会显示以下错误。但是外部证书不是 pod 级别的，而是重新编译的istio-egressgateway（这也没什么好玩的，我只是关注了 istio 文档），那么为什么 istio-proxy 正在寻找外部证书？

[2020-01-08 07:56:04.827][17][debug][init] [external/envoy/source/common/init/watcher_impl.cc:27] init manager Cluster outbound|443||afv.test.recouv destroyed
[2020-01-08 07:56:04.827][17][warning][config] [external/envoy/source/common/config/grpc_mux_subscription_impl.cc:73] gRPC config for type.googleapis.com/envoy.api.v2.Cluster rejected: Error adding/updating cluster(s) outbound|443||<my.hostname>: Invalid path: /etc/nginx-external-tls/tls.crt

Pilot 日志显示相同，istio-proxy调试器 pod 的容器一直在寻找外部服务 CA 证书，但该证书在egressgateway

$ kubectl -n istio-system logs istio-pilot-5f45f6768c-fmvr7 -c discovery | grep <my.hostname>

2020-01-08T08:48:54.536073Z     warn    ads     ADS:CDS: ACK ERROR 127.0.0.1:60012 sidecar~10.27.187.219~debugger-5844b6d674-4fvt9.tools-istio~tools-istio.svc.cluster.local-15433 (debugger-5844b6d674-4fvt9.tools-istio) version_info:"2020-01-08T07:50:00Z/110" node:<id:"sidecar~10.27.187.219~debugger-5844b6d674-4fvt9.tools-istio~tools-istio.svc.cluster.local" cluster:"debugger.tools-istio" metadata:<fields:<key:"CONFIG_NAMESPACE" value:<string_value:"tools-istio" > > fields:<key:"INCLUDE_INBOUND_PORTS" value:<string_value:"8080" > > fields:<key:"INTERCEPTION_MODE" value:<string_value:"REDIRECT" > > fields:<key:"ISTIO_META_INSTANCE_IPS" value:<string_value:"10.27.187.219,10.27.187.219,fe80::a07f:d0ff:feec:5e71" > > fields:<key:"ISTIO_PROXY_SHA" value:<string_value:"istio-proxy:f9c0feb10cc42277e97dd080a2e045f62d1739a9" > > fields:<key:"ISTIO_PROXY_VERSION" value:<string_value:"1.1.3" > > fields:<key:"ISTIO_VERSION" value:<string_value:"1.2.10" > > fields:<key:"POD_NAME" value:<string_value:"debugger-5844b6d674-4fvt9" > > fields:<key:"app" value:<string_value:"debugger" > > fields:<key:"istio" value:<string_value:"sidecar" > > fields:<key:"pod-template-hash" value:<string_value:"5844b6d674" > > fields:<key:"prometheus.io/path" value:<string_value:"/metrics" > > fields:<key:"prometheus.io/port" value:<string_value:"8080" > > fields:<key:"prometheus.io/scrape" value:<string_value:"true" > > fields:<key:"sidecar.istio.io/rewriteAppHTTPProbers" value:<string_value:"true" > > > locality:<> build_version:"f9c0feb10cc42277e97dd080a2e045f62d1739a9/1.11.0-dev/Modified/RELEASE/BoringSSL" > type_url:"type.googleapis.com/envoy.api.v2.Cluster" response_nonce:"59a1bb9a-aca6-47c0-b718-a86d7e392db0" error_detail:<code:13 message:"Error adding/updating cluster(s) outbound|443||afv.test.recouv: Invalid path: /etc/nginx-external-tls/tls.crt" >
  message: "Error adding/updating cluster(s) outbound|443||<my.hostname>: Invalid path: /etc/nginx-external-tls/tls.crt"

在istio-egressgateway一切接缝处都可以

$ kubectl -n istio-system logs istio-egressgateway-5ff889c5fd-jtz55
[2020-01-08 08:59:53.737][24][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:527] updating TLS cluster outbound_.80_._.<my.hostname>
[2020-01-08 08:59:53.737][24][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:975] membership update for TLS cluster outbound_.80_._.<my.hostname> added 1 removed 0
[2020-01-08 08:59:53.737][24][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:527] updating TLS cluster outbound_.443_._.<my.hostname>
[2020-01-08 08:59:53.737][24][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:975] membership update for TLS cluster outbound_.443_._.<my.hostname> added 1 removed 0

当然，证书应该在istio-egressgateway. 我使用 cert-manager 集群颁发者为外部服务提供证书，就像 AC 和它的秘密已经在集群上

# External AC
$ kubectl -n istio-system exec -it istio-egressgateway-5ff889c5fd-jtz55 -- ls /etc/cluster-issuer-tls
tls.crt  tls.key

# Client
$ kubectl -n istio-system exec -it istio-egressgateway-5ff889c5fd-jtz55 -- ls /etc/nginx-external-tls
ca.crt  tls.crt  tls.key

有没有人知道我错过了什么？这是配置错误还是错误？

任何帮助都将不胜感激，自从圣诞节假期之前我就一直坚持这一点

干杯

kubernetes - 用于外部服务的 Istio Egress MTL

0 回答 0

Related

Reference