0

我最近在 debian 10 上安装了 k3s，但在从 pod 内部连接到 Internet 时遇到问题。Ping、DNS、HTTP 可以正常工作，但 TLS 在握手时出现问题。我做了一个 tcpdump 并查看了 ClientHello，然后立即从服务器响应内部错误 SSL 错误 80。在运行集群的 VM 上一切正常，只是不在 Pod 内。

容器中的示例：

pacman -Syu
:: Synchronizing package databases...
error: failed retrieving file 'core.db' from mirror.pkgbuild.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'core.db' from mirror.rackspace.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'core.db' from mirror.leaseweb.net : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed to update core (download library error)
error: failed retrieving file 'extra.db' from mirror.pkgbuild.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'extra.db' from mirror.rackspace.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'extra.db' from mirror.leaseweb.net : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed to update extra (download library error)
error: failed retrieving file 'community.db' from mirror.pkgbuild.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'community.db' from mirror.rackspace.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'community.db' from mirror.leaseweb.net : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed to update community (download library error)
error: failed to synchronize all databases

curl -4 -v https://www.google.com
*   Trying 157.230.127.168:443...
* Connected to www.google.com (157.230.127.168) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

在虚拟机上：

curl -4 -v https://www.google.com
* Expire in 0 ms for 6 (transfer 0x556e74843fb0)
<snip>A lot of these expire messages</snip>
* Expire in 10 ms for 1 (transfer 0x556e74843fb0)
*   Trying 172.217.23.100...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x556e74843fb0)
* Connected to www.google.com (172.217.23.100) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=www.google.com
*  start date: Mar 11 15:00:19 2021 GMT
*  expire date: Jun  3 15:00:18 2021 GMT
*  subjectAltName: host "www.google.com" matched cert's "www.google.com"
*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x556e74843fb0)
> GET / HTTP/2
> Host: www.google.com
> User-Agent: curl/7.64.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200 
< date: Fri, 26 Mar 2021 01:32:52 GMT
< expires: -1
< cache-control: private, max-age=0
< content-type: text/html; charset=ISO-8859-1
< p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< server: gws
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< set-cookie: NID=212=0Nbn1MMUvMRUNa8iBWgekw-YrWof3yeeBb22v94ZrQ4KkMeVm5wouwkK9ElA353VmZCp_TOXah6KfC_KQX7S48W-IgXlQUz1z4ytqKnDzSXM7X40rLw8tBwMi4oH7eLyE4nGGAfsSlUne28lwMNLobMxYls54iUbgM4x9-kuMCA; expires=Sat, 25-Sep-2021 01:32:52 GMT; path=/; domain=.google.com; HttpOnly
< alt-svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
< accept-ranges: none
< vary: Accept-Encoding
<hmtl>...,</html>

有人知道如何调试或修复它吗？

更新

这里是来自 VM 和 pod 内部的 curl。我还将 Web 服务器修复为 Internet 上已知的工作服务器（Vercel 托管网站），因此我们调试同一台服务器。但它在 pod 内总是相同的错误。无论我联系哪个服务器。

kubectl run -i --tty --rm debug --image=archlinux --restart=Never -- bash

荚：

[root@debug /]# curl -v --tlsv1.2 --tls-max 1.2 https://wegmueller.it/ > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 157.230.127.168:443...
* Connected to wegmueller.it (157.230.127.168) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [229 bytes data]
* TLSv1.2 (IN), TLS alert, internal error (592):
{ [2 bytes data]
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
[root@debug /]#

虚拟机：

curl -v --tlsv1.2 --tls-max 1.2 https://wegmueller.it/ > /dev/null
* Expire in 0 ms for 6 (transfer 0x559d896cafb0)
* Expire in 1 ms for 1 (transfer 0x559d896cafb0)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Expire in 0 ms for 1 (transfer 0x559d896cafb0)
* Expire in 1 ms for 1 (transfer 0x559d896cafb0)
<snip a ton of expire messages>
* Expire in 50 ms for 1 (transfer 0x559d896cafb0)
*   Trying 76.76.21.21...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x559d896cafb0)
* Connected to wegmueller.it (76.76.21.21) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [223 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [106 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2458 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=wegmueller.it
*  start date: Mar 16 21:08:10 2021 GMT
*  expire date: Jun 14 21:08:10 2021 GMT
*  subjectAltName: host "wegmueller.it" matched cert's "wegmueller.it"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x559d896cafb0)
} [5 bytes data]
> GET / HTTP/2
> Host: wegmueller.it
> User-Agent: curl/7.64.0
> Accept: */*
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
} [5 bytes data]
< HTTP/2 200 
< date: Sun, 28 Mar 2021 15:24:59 GMT
< content-type: text/html; charset=utf-8
< content-disposition: inline; filename="index"
< cache-control: public, max-age=0, must-revalidate
< content-length: 3154
< access-control-allow-origin: *
< etag: W/"637eb49039bc1f6a7aa9e4d88a56b0708b722983f403c3c2f717371b2b25a472"
< accept-ranges: bytes
< x-vercel-cache: MISS
< age: 0
< server: Vercel
< x-vercel-id: cdg1::z6h9l-1616945098788-71384c5744be
< strict-transport-security: max-age=63072000
< 
{ [1112 bytes data]
100  3154  100  3154    0     0   3604      0 --:--:-- --:--:-- --:--:--  3600
* Connection #0 to host wegmueller.it left intact
4

1 回答 1

2
  • 错误：14094438：SSL 例程：ssl3_read_bytes:tlsv1 警报内部错误

openssl 错误代码包含 3 个部分。

  • 前 8 位是模块序列号。14(HEX) 表示序列号为 20 的 ssl 模块。
  • 接下来的 12 位是函数 SN。094(HEX) 是 SSL_SRP_CTX_init 函数，SN 为 148(DEC)。
  • 接下来的 12 位 438(HEX) 是错误原因 SSL_R_TLSV1_ALERT_INTERNAL_ERROR 和 1080(DEC)。

综上所述，代码 14094438 表示尝试启动 SSL 上下文时发生服务器错误。

但原因是什么？

客户端

客户端将 CLIENT HELLO 及其 tls 版本、密码套件等发送到服务器端。Pod 或 VM 上的 2 个 CLIENT HELLO 数据包有什么区别？这需要清楚。
在这里，您使用最新版本的 tls 1.3。如需更多信息，您可以降级客户端版本。curl 具有控制使用的 TLS 版本的选项。如果要指定使用 TLS 1.2 但不使用 1.1 或 1.3 等，则需要类似

curl --tlsv1.2 --tls-max 1.2 ...

服务器端

我还注意到www.google.com (157.230.127.168) 与www.google.com (172.217.23.100) 不同。服务器端的差异会导致这种情况吗？我们可以通过向同一个 IP 发送 curl 来确认。

我假设您和/或服务器使用了无与伦比的 openssl 库。但在得出结论之前，我们需要更多调试信息。

于 2021-03-26T08:42:49.277 回答