Wildfly lack of responsiveness and channel closed
Hi, I'm writing here in search for help.
My application runs on wildfly (version 21) and after random time it stops responding to the clients.
The structure is:
- Administrative panel for normal administration (war file).
- Api project for the clients that they exchange messages via websocket with this api (another war file).
Behaviour:
Everything works until api server start to slow down with responsiveness until nothing works after an hour and only after then the administrative panel also stops working.
I've bought the fastthread pro plan and the ML indicates this problem:
37 threads were executing sun.nio.ch.EPoll.wait(java.base@11.0.23/Native Method). It can slow down transactions.
In the logs there is this recurring error:
- (default task-214) java.io.IOException: UT002002: Channel is closed
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) java.io.IOException: UT002002: Channel is closed
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.websockets.core.WebSocketChannel.send(WebSocketChannel.java:343)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.websockets.core.WebSockets.sendBlockingInternal(WebSockets.java:992)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.websockets.core.WebSockets.sendBlockingInternal(WebSockets.java:986)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.websockets.core.WebSockets.sendTextBlocking(WebSockets.java:200)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at io.undertow.websocket@2.2.4.Final//io.undertow.websockets.jsr.WebSocketSessionRemoteEndpoint$BasicWebSocketSessionRemoteEndpoint.sendObjectImpl(WebSocketSessionRemoteEndpoint.java:351)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at io.undertow.websocket@2.2.4.Final//io.undertow.websockets.jsr.WebSocketSessionRemoteEndpoint$BasicWebSocketSessionRemoteEndpoint.sendObject(WebSocketSessionRemoteEndpoint.java:340)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.masterchart.api.FrontController.postAttach(FrontController.java:265)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at java.base/java.lang.reflect.Method.invoke(Method.java:566)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1060)
2024-07-16 09:16:08,656 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:962)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:523)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.websocket@2.2.4.Final//io.undertow.websockets.jsr.JsrWebSocketFilter.doFilter(JsrWebSocketFilter.java:173)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at deployment.my-api-server.war//org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at org.wildfly.extension.undertow@22.0.1.Final//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at org.wildfly.extension.undertow@22.0.1.Final//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at org.wildfly.extension.undertow@22.0.1.Final//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
2024-07-16 09:16:08,657 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at org.wildfly.extension.undertow@22.0.1.Final//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at org.wildfly.extension.undertow@22.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at org.wildfly.extension.undertow@22.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at org.wildfly.extension.undertow@22.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at org.wildfly.extension.undertow@22.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at org.wildfly.extension.undertow@22.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at io.undertow.servlet@2.2.4.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at io.undertow.core@2.2.4.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:841)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at org.jboss.xnio@3.8.4.Final//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
2024-07-16 09:16:08,658 ERROR [stderr] (default task-214) at java.base/java.lang.Thread.run(Thread.java:829)
I really don't get why after random time the channel is in status closed, maybe its linked to the sun.nio.ch.EPoll.wait problem.
It appears only when the server starts to having issues, the server can work for hours without having issues.
The apis are written using spring boot.
If something has to be changed in this question, I will.
All advice welcome.
Thank you in advance.
Report URL - https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjQvMDcvMTUvNDY1NC50eHQtLTgtMzktNDg=
Edit your Comment