我有 Apache 2.2.24 作为 JDK 1.6.0_17 上运行的 2 个 Tomcat 7.0.35 实例的反向代理。Tomcat 运行 BlazeDS Web 应用程序,为 Adobe AIR 应用程序提供内容。
所以配置看起来像
Adobe AIR 应用程序---HTTPS----> Apache ---HTTPS---> Tomcat
由于我公司的基础设施标准,Apache 通过 SSL 连接到 Tomcat 并使用 HTTP 而不是 AJP。这是无法改变的。
Apache 配置如下所示:
<IfModule mod_ssl.c>
<VirtualHost xxx:461>
SSLEngine on
# Force to use explicit Ciphers first (see artf660612)
SSLHonorCipherOrder on
# Disable SSLv2
SSLProtocol -ALL -SSLv2 +SSLv3 +TLSv1
# Disable Weak Ciphers
SSLCipherSuite !aNULL:!eNULL:!EXPORT:!DSS:!DES:RC4-SHA:RC4-MD5:+HIGH:+MEDIUM:ALL
SSLCertificateKeyFile /vhosts/content/current/contentsys/ssl/ssl.key
SSLCertificateFile /vhosts/content/current/contentsys/ssl/ssl.cert
SSLCACertificateFile /apache22/conf/ssl.crt/ca-bundle.crt
<FilesMatch "\.(cgi|shtml|pl|phtml|php3?)$">
SSLOptions +StdEnvVars
</FilesMatch>
SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
# Add per-virtual host throttling
<IfModule mod_qos.c>
# Defines the maximum number of concurrent TCP connections for this server (virtual host).
QS_SrvMaxConn 50
</IfModule>
<IfModule mod_cband.c>
CBandScoreboard /var/run/http/apache22/cbandscore/dw12869
# ridiculously high bandwidth and requests per second as we are only
# concerned with the last parameter - concurrent requests
CBandSpeed 100M 1000 50
</IfModule>
<LocationMatch "/(WEB|META)-INF/">
AllowOverride None
order allow,deny
deny from all
</LocationMatch>
# Import global URL Rewriting Engine properties
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteOptions inherit
</IfModule>
## Workaround for Apache "Range:" HTTP header bug (CVE-2011-3192)
# Drop the Range header when more than 5 ranges.
SetEnvIf Range (?:,.*?){5,5} bad-range=1
RequestHeader unset Range env=bad-range
# We always drop Request-Range; as this is a legacy
# dating back to MSIE3 and Netscape 2 and 3.
RequestHeader unset Request-Range
## END OF: Workaround for Apache "Range:" HTTP header bug (CVE-2011-3192)
## Workaround for PHP CGI remote code disclosure and execution bug (CVE-2012-1823)
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{QUERY_STRING} ^(%2d|-)[^=]+$ [NC]
RewriteRule ^(.*) $1? [L]
</IfModule>
## END OF: Workaround for PHP CGI remote code disclosure and execution bug (CVE-2012-1823)
# mod_jk configuration
<IfDefine JK>
<IfModule mod_jk.c>
</IfModule>
</IfDefine>
<IfModule ct_apache_mod.c>
RewriteEngine on
RewriteRule ^/(healthcheck|web-info) - [S=1]
# restriction: if the HOST header is not on the allowed list, serve a redirect to canonical URL
</IfModule>
#
# BalancerConfiguration 11
#
<Proxy balancer://balancer11>
BalancerMember https://myserver1.com:19802 route=0 timeout=300
BalancerMember https://myserver2.com:19802 route=1 timeout=300
ProxySet stickysession=JSESSIONID|jsessionid
ProxySet scolonpathdelim=On
ProxySet lbmethod=byrequests
</Proxy>
<Location /myapp>
ProxyPassReverse balancer://balancer11/myapp
ProxyPassReverseCookiePath /myapp /myapp
ProxyPass balancer://balancer11/myapp
</Location>
# BEGIN Extra configuration verbatim from config file
# Cache ADK CSS and Image Assets
<Location /adk>
ExpiresActive On
ExpiresDefault "access plus 1 year"
</Location>
LoadModule deflate_module /apache22/modules/mod_deflate.so
<IfModule mod_deflate.c>
# Insert filter
SetOutputFilter DEFLATE
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png|bmp|icon|ico|tif|tiff)$ no-gzip dont-vary
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</IfModule>
# END Extra configuration verbatim from config file
</VirtualHost>
</IfModule>
Tomcat Server.xml 看起来像
<Connector protocol="org.apache.coyote.http11.Http11Protocol" port="19802"
maxThreads="200" connectionTimeout="20000" scheme="https" secure="true" SSLEnabled="true"
keystoreFile="/path/to/keystore" keystorePass="changeIt" keyAlias="web_ssl"
clientAuth="false" sslProtocol="TLS"/>
Tomcat 服务器每天上午 12.45 重新启动。从早上 8 点开始,我们有大约 20 个用户登录。flex 应用程序使用轮询,因此每 1 秒进行一次 HTTP 调用以获取数据。一天中的前几个小时一切正常。然后大约下午 3-4 点,我们开始在调用 1 个 Tomcat 节点时偶尔收到 502 Bad Gateway 错误。
[Fri Aug 16 20:01:19 2013] [info] [client xx.xx.xx.xxx] (70007)The timeout specified has expired: SSL input filter read failed.
[Fri Aug 16 20:01:19 2013] [error] [client aa.aa.aa.aaa] (70007)The timeout specified has expired: proxy: error reading status line from remote server myserver1.com:19802, referer: app:/Main.swf
[Fri Aug 16 20:01:19 2013] [error] [client aa.aa.aa.aaa] proxy: Error reading from remote server returned by /myapp/messagebroker/amfsecure, referer: app:/Main.swf
有时是以下
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] (104)Connection reset by peer: SSL input filter read failed.
[Fri Aug 16 20:17:30 2013] [error] (104)Connection reset by peer: proxy: prefetch request body failed to yy.yy.yy.yy:19802 (myserver.com) from xx.xx.xx.xx ()
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] (104)Connection reset by peer: SSL input filter read failed.
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] Connection closed to child 26 with standard shutdown (server mysite:443)
我们设置了粘性会话,因此尝试登录的新用户会被反复定向到 1 个节点,这可能是引发此错误的一个节点,因此他们在重启时也不断收到 502 超时。重新启动 Tomcat 节点可以解决问题,它会开始工作几个小时。但随后又再次出现。tomcat 本身可以直接访问而没有问题。需要注意的另一件事是,Apache 服务器还充当我不管理/控制的其他 Tomcat 的反向代理。
我一直在寻找其他面临此问题的人,并发现以下内容:
- 已知在 SSL 反向代理上使用 GZIP 分块存在问题 - https://issues.apache.org/bugzilla/show_bug.cgi?id=52595
- 人们指出,设置 force-proxy-request-1.0 和 proxy-nokeepalive 可以提供帮助 - https://serverfault.com/questions/206738/intermittent-error-when-using-mod-proxy-to-do-reverse-代理到肥皂服务
我们不能直接编辑 Apache 文件,因为它是一个受控环境,只能在底部添加配置选项。所以我添加了以下内容
SetEnvIf Request_URI "/myapp.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
我认为这会有所帮助,因为在 UAT 中使用 JMeter 进行负载测试看起来很有希望。下面的日志显示了添加上述指令后 1 个此类请求的输出。但是,我们仍然在生产中看到这个问题
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(46): proxy: BALANCER: canonicalising URL //balancer11/myapp/messagebroker/amfsecure
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_http.c(1771): proxy: start body send
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(280): proxy: BALANCER: Found value B8620734D4F2D92E9B0C507C16A5EF8A.1 for stickysession JSESSIONID|jsessionid
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(290): proxy: BALANCER: Found route 1
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(594): proxy: BALANCER (balancer://balancer11) worker (https://myserver1.com:19802) rewritten to https://myserver1.com:19802/myapp/messagebroker/amfsecure
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy.c(1028): Running scheme balancer handler (attempt 0)
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_http.c(2001): proxy: HTTP: serving URL https://myserver1.com:19802/myapp/messagebroker/amfsecure
[Wed Aug 21 15:57:07 2013] [debug] proxy_util.c(2013): proxy: HTTPS: has acquired connection for (myserver1.com)
[Wed Aug 21 15:57:07 2013] [debug] ssl_engine_io.c(1918): OpenSSL: I/O error, 5 bytes expected to read on BIO#cfa3578 [mem: d1f4dd8]
[Wed Aug 21 15:57:07 2013] [debug] proxy_util.c(2069): proxy: connecting https://myserver1.com:19802/myapp/messagebroker/amfsecure to myserver1.com:19802
[Wed Aug 21 15:57:07 2013] [debug] proxy_util.c(2195): proxy: connected /myapp/messagebroker/amfsecure to myserver1.com:19802
[Wed Aug 21 15:57:07 2013] [debug] mod_deflate.c(615): [client xx.xx.xx.xxx] Zlib: Compressed 0 to 2 : URL /myapp/messagebroker/amfsecure, referer: app:/Main.swf
[Wed Aug 21 15:57:07 2013] [debug] mod_headers.c(756): headers: ap_headers_output_filter()
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_http.c(1875): proxy: end body send
[Wed Aug 21 15:57:07 2013] [debug] proxy_util.c(2031): proxy: HTTPS: has released connection for (myserver1.com)
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(637): proxy_balancer_post_request for (balancer://balancer11)
[Wed Aug 21 15:57:07 2013] [info] [client xx.xx.xx.xxx] Connection closed to child 15 with unclean shutdown (server myserver:443)
用户报告的所有类似问题似乎都发生在早期版本的 Apache、Tomcat 等上,并且该错误早已得到修复。所以我真的不知道该尝试/做什么。任何指针将不胜感激。谢谢