I'm getting a VERY strange error I can't figure out in JBoss AS7. Basically the web application is a REST application that takes data in as a large POST, stores it in a database, and returns the stored data to the user in the response as XML using JAX-RS. This is repeated thousands of times. The data is stored, and the client gets the full response.
Other than the JBoss server logs getting spammed with the following warnings/errors, there are no problems. Both the web application and client do not report errors, and run fine.
Very quickly after starting the process I get the following warnings:
08:54:30,327 WARN [org.jboss.resteasy.core.SynchronousDispatcher] (http--0.0.0.0-8080-1) Failed executing POST /messages/ingestmessage/: org.jboss.resteasy.plugins.providers.jaxb.JAXBMarshalException: javax.xml.bind.MarshalException
- with linked exception:
[ClientAbortException: java.net.SocketException: Connection reset]
at org.jboss.resteasy.plugins.providers.jaxb.AbstractJAXBProvider.writeTo(AbstractJAXBProvider.java:148) [resteasy-jaxb-provider-2.3.2.Final.jar:]
at org.jboss.resteasy.core.interception.MessageBodyWriterContextImpl.proceed(MessageBodyWriterContextImpl.java:117) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.interceptors.encoding.GZIPEncodingInterceptor.write(GZIPEncodingInterceptor.java:100) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.interception.MessageBodyWriterContextImpl.proceed(MessageBodyWriterContextImpl.java:123) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.ServerResponse.writeTo(ServerResponse.java:250) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.writeJaxrsResponse(SynchronousDispatcher.java:585) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:506) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]
at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_25]
Caused by: javax.xml.bind.MarshalException
- with linked exception:
[ClientAbortException: java.net.SocketException: Connection reset]
at com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:328)
at com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:251)
at javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:74) [jboss-jaxb-api_2.2_spec-1.0.3.Final.jar:1.0.3.Final]
at org.jboss.resteasy.plugins.providers.jaxb.AbstractJAXBProvider.writeTo(AbstractJAXBProvider.java:144) [resteasy-jaxb-provider-2.3.2.Final.jar:]
... 25 more
Caused by: ClientAbortException: java.net.SocketException: Connection reset
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:426) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:415) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89) [jbossweb-7.0.13.Final.jar:]
at org.jboss.resteasy.util.CommitHeaderOutputStream.write(CommitHeaderOutputStream.java:52) [resteasy-jaxrs-2.3.2.Final.jar:]
at com.sun.xml.bind.v2.runtime.output.UTF8XmlOutput.write(UTF8XmlOutput.java:411)
at com.sun.xml.bind.v2.runtime.output.Encoded.write(Encoded.java:167)
at com.sun.xml.bind.v2.runtime.output.UTF8XmlOutput.writeName(UTF8XmlOutput.java:241)
at com.sun.xml.bind.v2.runtime.output.UTF8XmlOutput.endTag(UTF8XmlOutput.java:285)
at com.sun.xml.bind.v2.runtime.output.NamespaceContextImpl$Element.endElement(NamespaceContextImpl.java:504)
at com.sun.xml.bind.v2.runtime.XMLSerializer.endElement(XMLSerializer.java:318)
at com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:175)
at com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:159)
at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:356)
at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsXsiType(XMLSerializer.java:700)
at com.sun.xml.bind.v2.runtime.property.SingleElementNodeProperty.serializeBody(SingleElementNodeProperty.java:158)
at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:356)
at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsXsiType(XMLSerializer.java:700)
at com.sun.xml.bind.v2.runtime.property.ArrayElementNodeProperty.serializeItem(ArrayElementNodeProperty.java:69)
at com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:172)
at com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:159)
at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:356)
at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsXsiType(XMLSerializer.java:700)
at com.sun.xml.bind.v2.runtime.property.SingleElementNodeProperty.serializeBody(SingleElementNodeProperty.java:158)
at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:356)
at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsSoleContent(XMLSerializer.java:597)
at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(ClassBeanInfoImpl.java:338)
at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:498)
at com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:323)
... 28 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) [rt.jar:1.6.0_25]
at java.net.SocketOutputStream.write(SocketOutputStream.java:136) [rt.jar:1.6.0_25]
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:724) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:748) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:559) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.Response.doWrite(Response.java:594) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:398) [jbossweb-7.0.13.Final.jar:]
... 58 more
This is then followed by errors after several of the warnings...
08:54:34,044 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/cdsWebApp].[org.neoninc.cds.RestApplication]] (http--0.0.0.0-8080-7) Servlet.service() for servlet org.neoninc.cds.RestApplication threw exception: java.lang.IllegalStateException
at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:408) [jbossweb-7.0.13.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:71) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.handleFailure(SynchronousDispatcher.java:265) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.handleWriterException(SynchronousDispatcher.java:393) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.handleException(SynchronousDispatcher.java:219) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.handleWriteResponseException(SynchronousDispatcher.java:203) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:510) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]
at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_25]
From searching around, it seems this error can happen if the response is too large. I'd like to set the buffer size to a larger size, at least for this REST call.
Am I correct in thinking the buffer is the problem? The responses can be several hundred kilobytes (~650 kB). If so, how can I change the buffer size on the response in JBoss AS7? If not, what else could it be?
Any other ideas are welcome.
EDIT: More info. It looks like this might be a problem where the client cuts the conversation abruptly. I'm using JAX-RS to marshal out the xml to the client using
// Marshal the created message transition run to xml and return it
response = Response.status(201).entity(messageTransactionRunDomain).location(uriBuilder.build()).lastModified(new Date()).build();
The client is receiving the xml using
// Build the resource URL to get the first type . Also set the mime type.
long temptime = System.currentTimeMillis();
ClientResponse response =
webResource.type("application/vnd.neoninc.cds.processedmessage-v1.0+xml")
.post(ClientResponse.class, processedMsgIngest);
// Get the saved message transition run, so you can get the new id
if (response.getStatus() == 201) {
long endtime = System.currentTimeMillis();
log.debug("Time for " + readouts.size() + " readout values batching (seconds): " + (endtime-temptime)/1000.);
CDSRESTDataIngest.postValsTotalTime = CDSRESTDataIngest.postValsTotalTime + (endtime-temptime);
return true;
} else {
log.error("POST failed and returned a status of " + response.getStatus());
log.error(response.getEntity(String.class));
}
Where I'm confused, is this isn't a browser conversation. The client isn't hitting cancel or back, and is getting everything returned in the response...