2 Replies Latest reply on May 13, 2014 7:43 AM by jameswright

    Inconsistent Endpoint Delivery

    jameswright

      Has anyone had issues with endpoints become undeliverable? This is an intermittent issue that doesn't seem to affect all nodes necessarily. From the looks of it, and maybe I am reading it wrong, its failing before even being delivered to the endpoint in question.

       

      Whenever this happens I get this error:

      2014-05-12 11:02:05,713 [http-nio-0.0.0.0-9200-exec-13] [2023:jamesw:REGULAR] WARN org.apache.cxf.jaxrs.impl.WebApplicationExceptionMapper - javax.ws.rs.InternalServerErrorException: java.io.IOException: Stream closed

        at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:245)

        at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.handleMessage(JAXRSInInterceptor.java:100)

        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)

        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)

        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)

        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:223)

        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:203)

        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137)

        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:158)

        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:243)

        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:168)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)

        • Re: Inconsistent Endpoint Delivery
          jameswright

          I am seeing this, which may explain it:

           

          12 May 2014 12:03:46,279 [localhost-startStop-2] ERROR loader.WebappClassLoader - The web application [/] created a ThreadLocal with key of type [com.jivesoftware.cache.impl.TransactionalCache$3] (value [com.jivesoftware.cache.impl.TransactionalCache$3@b61f81e]) and a value of type [java.util.HashMap] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

          12 May 2014 12:03:46,279 [localhost-startStop-2] ERROR loader.WebappClassLoader - The web application [/] created a ThreadLocal with key of type [com.jivesoftware.cache.impl.TransactionalCache$3] (value [com.jivesoftware.cache.impl.TransactionalCache$3@59cb7f9a]) and a value of type [java.util.HashMap] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

          12 May 2014 12:03:46,279 [localhost-startStop-2] ERROR loader.WebappClassLoader - The web application [/] created a ThreadLocal with key of type [com.jivesoftware.cache.impl.TransactionalCache$1] (value [com.jivesoftware.cache.impl.TransactionalCache$1@22bb83ae]) and a value of type [java.util.HashMap] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

          12 May 2014 12:03:46,279 [localhost-startStop-2] ERROR loader.WebappClassLoader - The web application [/] created a ThreadLocal with key of type [com.jivesoftware.cache.impl.TransactionalCache$1] (value [com.jivesoftware.cache.impl.TransactionalCache$1@69c0cfc8]) and a value of type [java.util.HashMap] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

          12 May 2014 12:03:46,279 [localhost-startStop-2] ERROR loader.WebappClassLoader - The web application [/] created a ThreadLocal with key of type [com.jivesoftware.cache.impl.TransactionalCache$3] (value [com.jivesoftware.cache.impl.TransactionalCache$3@26f966b0]) and a value of type [java.util.HashMap] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

           

          Not exactly sure how to fix such an error. Or exactly what would cause it.

          • Re: Inconsistent Endpoint Delivery
            jameswright

            After working with Kevin Conaway I think we have isolated the cause of this error. After Kevin enabled TRACE logging on the org.apache.cxf package the additional cxf errors were displayed outlying that there was an ambiguous route.

             

            In this case, my service had the correct JAXRS annotations however the interface it was implementing did not. IE I had route annotation on both the service and, interface implemented by that service, and they did not match. This caused the router to be unsure which route to take. NOTE the: 'getEntities may get selected' followed by 'queryEntities .... has been selected'.

             

            getEntities was the method we wanted... on the other hand the route selected was queryEntities, which is an internal method that should not be invoked externally by the web service and does not contain a processable message body, hence the java.io.IOException: SteamClosed error.

             

            Here is the trace:

            2014-05-13 06:32:24,608 [http-nio-0.0.0.0-9200-exec-1] [3931:testdev:REGULAR] DEBUG org.apache.cxf.jaxrs.utils.JAXRSUtils - No method match, method name : createEntity, request path : /EntityMetadata, method @Path : /{type}, HTTP Method : GET, method HTTP Method : POST, ContentType : */*, method @Consumes : application/json,, Accept : text/html,application/xhtml+xml,image/webp,application/xml;q=0.9,*/*;q=0.8,, method @Produces : application/json,.

            2014-05-13 06:32:24,609 [http-nio-0.0.0.0-9200-exec-1] [3931:testdev:REGULAR] DEBUG org.apache.cxf.jaxrs.utils.JAXRSUtils - Resource operation getEntities may get selected

            2014-05-13 06:32:24,609 [http-nio-0.0.0.0-9200-exec-1] [3931:testdev:REGULAR] DEBUG org.apache.cxf.jaxrs.utils.JAXRSUtils - Resource operation queryEntities on the resource class com.jivesoftware.schneider.eu.support.api.EntityDetailsSecurityService has been selected

            2014-05-13 06:32:24,610 [http-nio-0.0.0.0-9200-exec-1] [3931:testdev:REGULAR] DEBUG org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor - Request path is: /entities/EntityMetadata

            2014-05-13 06:32:24,611 [http-nio-0.0.0.0-9200-exec-1] [3931:testdev:REGULAR] DEBUG org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor - Request HTTP method is: GET

            2014-05-13 06:32:24,612 [http-nio-0.0.0.0-9200-exec-1] [3931:testdev:REGULAR] DEBUG org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor - Request contentType is: */*

            2014-05-13 06:32:24,612 [http-nio-0.0.0.0-9200-exec-1] [3931:testdev:REGULAR] DEBUG org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor - Accept contentType is: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8

            2014-05-13 06:32:24,613 [http-nio-0.0.0.0-9200-exec-1] [3931:testdev:REGULAR] DEBUG org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor - Found operation: queryEntities

            2014-05-13 06:32:24,614 [http-nio-0.0.0.0-9200-exec-1] [3931:testdev:REGULAR] DEBUG org.apache.cxf.jaxrs.utils.JAXRSUtils - No Content-Type specified for HTTP GET

            2014-05-13 06:32:24,615 [http-nio-0.0.0.0-9200-exec-1] [3931:testdev:REGULAR] DEBUG org.apache.cxf.jaxrs.utils.JAXRSUtils - No Content-Type specified for HTTP GET

            2014-05-13 06:32:24,622 [http-nio-0.0.0.0-9200-exec-1] [3931:testdev:REGULAR] WARN org.apache.cxf.jaxrs.impl.WebApplicationExceptionMapper - javax.ws.rs.InternalServerErrorException: java.io.IOException: Stream closed

              at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:245)

              at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.handleMessage(JAXRSInInterceptor.java:100)

              at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)

              at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)

              at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)

             

            Jason Sharp Becky Dozier