2

我有 Apache 2.2.24 作为 JDK 1.6.0_17 上运行的 2 个 Tomcat 7.0.35 实例的反向代理。Tomcat 运行 BlazeDS Web 应用程序，为 Adob​​e AIR 应用程序提供内容。

所以配置看起来像

Adobe AIR 应用程序---HTTPS----> Apache ---HTTPS---> Tomcat

由于我公司的基础设施标准，Apache 通过 SSL 连接到 Tomcat 并使用 HTTP 而不是 AJP。这是无法改变的。

Apache 配置如下所示：

<IfModule mod_ssl.c>

<VirtualHost xxx:461>

  SSLEngine on

  # Force to use explicit Ciphers first (see artf660612)
  SSLHonorCipherOrder on
  # Disable SSLv2
  SSLProtocol -ALL -SSLv2 +SSLv3 +TLSv1

  # Disable Weak Ciphers
  SSLCipherSuite !aNULL:!eNULL:!EXPORT:!DSS:!DES:RC4-SHA:RC4-MD5:+HIGH:+MEDIUM:ALL
  SSLCertificateKeyFile /vhosts/content/current/contentsys/ssl/ssl.key
  SSLCertificateFile /vhosts/content/current/contentsys/ssl/ssl.cert
  SSLCACertificateFile /apache22/conf/ssl.crt/ca-bundle.crt
  <FilesMatch "\.(cgi|shtml|pl|phtml|php3?)$">
    SSLOptions +StdEnvVars
  </FilesMatch>

  SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0

  # Add per-virtual host throttling

  <IfModule mod_qos.c>
    # Defines the maximum number of concurrent TCP connections for this server (virtual host).
    QS_SrvMaxConn 50
  </IfModule>

  <IfModule mod_cband.c>
    CBandScoreboard /var/run/http/apache22/cbandscore/dw12869
    # ridiculously high bandwidth and requests per second as we are only
    # concerned with the last parameter - concurrent requests
    CBandSpeed 100M 1000 50
  </IfModule>

  <LocationMatch "/(WEB|META)-INF/">
    AllowOverride None
    order allow,deny
    deny from all
  </LocationMatch>

  # Import global URL Rewriting Engine properties
  <IfModule mod_rewrite.c>
    RewriteEngine on
    RewriteOptions inherit

  </IfModule>
  ## Workaround for Apache "Range:" HTTP header bug (CVE-2011-3192)

  # Drop the Range header when more than 5 ranges.
  SetEnvIf Range (?:,.*?){5,5} bad-range=1
  RequestHeader unset Range env=bad-range

  # We always drop Request-Range; as this is a legacy
  # dating back to MSIE3 and Netscape 2 and 3.
  RequestHeader unset Request-Range

  ## END OF: Workaround for Apache "Range:" HTTP header bug (CVE-2011-3192)
  ## Workaround for PHP CGI remote code disclosure and execution bug (CVE-2012-1823)

  <IfModule mod_rewrite.c>
    RewriteEngine On

    RewriteCond %{QUERY_STRING} ^(%2d|-)[^=]+$ [NC]
    RewriteRule ^(.*) $1? [L]
  </IfModule>

  ## END OF: Workaround for PHP CGI remote code disclosure and execution bug (CVE-2012-1823)   


  # mod_jk configuration 
  <IfDefine JK>
    <IfModule mod_jk.c>

    </IfModule>
  </IfDefine>

<IfModule ct_apache_mod.c>

    RewriteEngine on
    RewriteRule ^/(healthcheck|web-info) - [S=1]
    # restriction: if the HOST header is not on the allowed list, serve a redirect to canonical URL

</IfModule>

#
# BalancerConfiguration 11
#
<Proxy balancer://balancer11>
    BalancerMember https://myserver1.com:19802 route=0 timeout=300
    BalancerMember https://myserver2.com:19802 route=1 timeout=300
    ProxySet stickysession=JSESSIONID|jsessionid
    ProxySet scolonpathdelim=On
    ProxySet lbmethod=byrequests
</Proxy>
<Location /myapp>
    ProxyPassReverse balancer://balancer11/myapp
    ProxyPassReverseCookiePath /myapp /myapp
    ProxyPass balancer://balancer11/myapp
</Location>

# BEGIN Extra configuration verbatim from config file

# Cache ADK CSS and Image Assets
<Location /adk>
ExpiresActive On
ExpiresDefault "access plus 1 year"
</Location>

LoadModule deflate_module /apache22/modules/mod_deflate.so
<IfModule mod_deflate.c>
# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

# Don't compress images
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png|bmp|icon|ico|tif|tiff)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</IfModule>


# END   Extra configuration verbatim from config file
</VirtualHost>
</IfModule>

Tomcat Server.xml 看起来像

<Connector protocol="org.apache.coyote.http11.Http11Protocol" port="19802" 
    maxThreads="200" connectionTimeout="20000" scheme="https" secure="true" SSLEnabled="true" 
    keystoreFile="/path/to/keystore" keystorePass="changeIt" keyAlias="web_ssl" 
    clientAuth="false" sslProtocol="TLS"/>

Tomcat 服务器每天上午 12.45 重新启动。从早上 8 点开始，我们有大约 20 个用户登录。flex 应用程序使用轮询，因此每 1 秒进行一次 HTTP 调用以获取数据。一天中的前几个小时一切正常。然后大约下午 3-4 点，我们开始在调用 1 个 Tomcat 节点时偶尔收到 502 Bad Gateway 错误。

[Fri Aug 16 20:01:19 2013] [info] [client xx.xx.xx.xxx] (70007)The timeout specified has expired: SSL input filter read failed.
[Fri Aug 16 20:01:19 2013] [error] [client aa.aa.aa.aaa] (70007)The timeout specified has expired: proxy: error reading status line from remote server myserver1.com:19802, referer: app:/Main.swf
[Fri Aug 16 20:01:19 2013] [error] [client aa.aa.aa.aaa] proxy: Error reading from remote server returned by /myapp/messagebroker/amfsecure, referer: app:/Main.swf

有时是以下

[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] (104)Connection reset by peer: SSL input filter read failed.
[Fri Aug 16 20:17:30 2013] [error] (104)Connection reset by peer: proxy: prefetch request body failed to yy.yy.yy.yy:19802 (myserver.com) from xx.xx.xx.xx ()
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] (104)Connection reset by peer: SSL input filter read failed.
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] Connection closed to child 26 with standard shutdown (server mysite:443)

我们设置了粘性会话，因此尝试登录的新用户会被反复定向到 1 个节点，这可能是引发此错误的一个节点，因此他们在重启时也不断收到 502 超时。重新启动 Tomcat 节点可以解决问题，它会开始工作几个小时。但随后又再次出现。tomcat 本身可以直接访问而没有问题。需要注意的另一件事是，Apache 服务器还充当我不管理/控制的其他 Tomcat 的反向代理。

我一直在寻找其他面临此问题的人，并发现以下内容：

  1. 已知在 SSL 反向代理上使用 GZIP 分块存在问题 - https://issues.apache.org/bugzilla/show_bug.cgi?id=52595
  2. 人们指出，设置 force-proxy-request-1.0 和 proxy-nokeepalive 可以提供帮助 - https://serverfault.com/questions/206738/intermittent-error-when-using-mod-proxy-to-do-reverse-代理到肥皂服务

我们不能直接编辑 Apache 文件，因为它是一个受控环境，只能在底部添加配置选项。所以我添加了以下内容

SetEnvIf Request_URI "/myapp.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0

我认为这会有所帮助，因为在 UAT 中使用 JMeter 进行负载测试看起来很有希望。下面的日志显示了添加上述指令后 1 个此类请求的输出。但是，我们仍然在生产中看到这个问题

[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(46): proxy: BALANCER: canonicalising URL //balancer11/myapp/messagebroker/amfsecure
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_http.c(1771): proxy: start body send
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(280): proxy: BALANCER: Found value B8620734D4F2D92E9B0C507C16A5EF8A.1 for stickysession JSESSIONID|jsessionid
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(290): proxy: BALANCER: Found route 1
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(594): proxy: BALANCER (balancer://balancer11) worker (https://myserver1.com:19802) rewritten to https://myserver1.com:19802/myapp/messagebroker/amfsecure
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy.c(1028): Running scheme balancer handler (attempt 0)
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_http.c(2001): proxy: HTTP: serving URL https://myserver1.com:19802/myapp/messagebroker/amfsecure
[Wed Aug 21 15:57:07 2013] [debug] proxy_util.c(2013): proxy: HTTPS: has acquired connection for (myserver1.com)
[Wed Aug 21 15:57:07 2013] [debug] ssl_engine_io.c(1918): OpenSSL: I/O error, 5 bytes expected to read on BIO#cfa3578 [mem: d1f4dd8]
[Wed Aug 21 15:57:07 2013] [debug] proxy_util.c(2069): proxy: connecting https://myserver1.com:19802/myapp/messagebroker/amfsecure to myserver1.com:19802
[Wed Aug 21 15:57:07 2013] [debug] proxy_util.c(2195): proxy: connected /myapp/messagebroker/amfsecure to myserver1.com:19802
[Wed Aug 21 15:57:07 2013] [debug] mod_deflate.c(615): [client xx.xx.xx.xxx] Zlib: Compressed 0 to 2 : URL /myapp/messagebroker/amfsecure, referer: app:/Main.swf
[Wed Aug 21 15:57:07 2013] [debug] mod_headers.c(756): headers: ap_headers_output_filter()
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_http.c(1875): proxy: end body send
[Wed Aug 21 15:57:07 2013] [debug] proxy_util.c(2031): proxy: HTTPS: has released connection for (myserver1.com)
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(637): proxy_balancer_post_request for (balancer://balancer11)
[Wed Aug 21 15:57:07 2013] [info] [client xx.xx.xx.xxx] Connection closed to child 15 with unclean shutdown (server myserver:443)

用户报告的所有类似问题似乎都发生在早期版本的 Apache、Tomcat 等上，并且该错误早已得到修复。所以我真的不知道该尝试/做什么。任何指针将不胜感激。谢谢

4

0 回答 0