问题
我们在提供 HTTP Web 服务的 Weblogic 服务器 10.3.1 上部署了一个 java ee 应用程序。让我们称之为“服务器”。另一个 java 应用程序部署在另一台机器上的 tomcat 上,它将对服务器进行 Web 服务调用,我们称之为“客户端”。客户端和服务器应用程序都在使用 Axis。
客户端有可能会因异常而失败
"java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)",
1 分钟后,服务器应用程序日志显示:
org.apache.axis.Message ERROR -
java.io.IOException: java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99).
一天失败不超过5次。服务器上的总 Web 服务请求较少,例如数百。并且似乎在服务器长时间“休息”后往往会失败:每天早上的第一次调用可能会失败(我们晚上没有业务服务)。
异常堆栈是:
客户端:
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)
... 12 more
服务器端:
2013-03-06 08:37:29,491 org.apache.axis.Message ERROR - java.io.IOException:
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99)
at java.net.SocketOutputStream.write(SocketOutputStream.java:137)
at weblogic.servlet.internal.ChunkOutput.writeChunkNoTransfer(ChunkOutput.java:530)
at weblogic.servlet.internal.ChunkOutput.writeChunks(ChunkOutput.java:487)
at weblogic.servlet.internal.ChunkOutput.flush(ChunkOutput.java:382)
at weblogic.servlet.internal.CharsetChunkOutput.flush(CharsetChunkOutput.java:315)
at weblogic.servlet.internal.ChunkOutput$2.checkForFlush(ChunkOutput.java:580)
at weblogic.servlet.internal.CharsetChunkOutput.write(CharsetChunkOutput.java:222)
at weblogic.servlet.internal.ChunkOutputWrapper.write(ChunkOutputWrapper.java:146)
at weblogic.servlet.internal.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:138)
at org.apache.axis.utils.ByteArray.writeTo(ByteArray.java:375)
at org.apache.axis.SOAPPart.writeTo(SOAPPart.java:265)
at org.apache.axis.Message.writeTo(Message.java:539)
at org.apache.axis.transport.http.AxisServlet.sendResponse(AxisServlet.java:902)
at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:777)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:821)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:170)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:168)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:112)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
at com.ulic.ucia.framework.util.AppFilter.doFilter(AppFilter.java:48)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
at com.ebao.pub.framework.AppFilter.doFilter(AppFilter.java:101)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3588)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2200)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2106)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1428)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
当客户端出现故障时,我们检查应用日志和数据库,发现业务事务总是成功的。Weblogic 服务器以某种方式无法写入 HTTP 响应。Weblogic 服务器日志文件说:
BEA 101366 服务器无法在配置的超时值期间发送 HTTP 消息。套接字已关闭。
排序日志和一些分析
两台服务器都有ntp服务。我按时间对服务器日志和客户端日志进行了简化和排序,以获得更清晰的视图,
2013-03-06 08:36:18,755 Web Service Client began it's work
2013-03-06 08:36:18,758 Web Service Client make http calls and wait for response
2013-03-06 08:36:19,039 APP on WLS received request
2013-03-06 08:36:24,553 APP on WLS app log says transaction finished with a success.
2013-03-06 08:36:24,575 Web Service Client received java.net.SocketException: Connection reset”
2013-03-06 08:37:29,491 APP on WLS org.apache.axis.Message ERROR - java.io.IOException:java.net.SocketException: Socket closed
最值得注意的是,当服务器应用程序刚刚完成继续时,客户端在等待 6 秒后收到了连接重置。所以我认为它必须与服务器有关。当 weblogic 由于某种原因试图发送 http 响应时,服务器机器发送了一个“TCP RST”,但那是什么原因呢?据我所知,有两种情况(可能更多)“RST”将在 Java 应用程序中发送。Java 线程在没有关闭套接字的情况下结束,另一方面,tcp 堆栈将向套接字发送一个 'RST' 以指示错误 Tcp 堆栈在 'linger' 时间后未能发送所有数据。
现在我的大脑已经卡住了,不知道下一步该怎么做。任何建议将不胜感激。