Hello Manoj,
Greetings!
There could be several reasons why application gets stuck or going unresponsive. Some of the reasons are:
• Garbage collection pauses
• Threads getting BLOCKED
• Network connectivity
• Load balancer routing issue
• Heavy CPU consumption of threads
• Operating System running with old patches
• Memory Leak
• DB not responding properly
• Kernel issues
• Backend slow downs
• Hypervisor/container orchestrator not allocating enough resources
:
:
So just thread dump is not enough to diagnose the problem. You have captured only thread dump, that too one snapshot of it. It's always a good practice to capture 3 thread dumps in a gap of 10 seconds between each one. Besides thread dumps you might have to capture other logs/artifacts to do thorough analysis.
You can use the open source yCrash script which will capture 360-degree application level artifacts (like GC logs, 3 snapshots of thread dumps, heap dumps) and system level artifacts (like top, top -H, netstat, vmstat, iostat, dmesg, diskusage, kernel parameters...). Once you have these data, either you can manually analyze them or upload it to yCrash tool, which will analyze all these artifacts and generate root cause analysis report. It has potential to indicate the root cause of the problem.
Also in this report it clearly states that,
1. There are 100 Garbage Collection(GC) Threads. This might be a high number of GC Threads. It may affect the application's performance. You may consider reducing GC threads.
2. Two threads are in STUCK state. Threads that are taken out of the pool and not returned after a specified time period are marked as STUCK threads. It may or may not be a problem. Study their stacktrace to understand it's behaviour below,
stackTrace: java.lang.Thread.State: RUNNABLE at df.JadePage.parseLIST(JadePage.java:1201) at df.JadePage.LoadPage(JadePage.java:397) at df.JadePage.LoadPage(JadePage.java:220) at df.JadePage.SetHTMLCode(JadePage.java:576) at df.JadePage.LoadPage(JadePage.java:361) at df.JadePage.SetHTMLCode(JadePage.java:576) at df.JadePage.LoadPage(JadePage.java:361) at df.Framework.Execute(Framework.java:297) at jsp_servlet.__server._jspService(__server.java:983) at weblogic.servlet.jsp.JspBase.service(JspBase.java:35) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:295) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:353) at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at ResponseHeaderFilter.doFilter(ResponseHeaderFilter.java:33) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3800) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3766) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344) at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197) at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203) at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71) at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2454) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2302) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2280) at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1721) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1681) at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272) at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352) at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337) at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57) at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41) at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:655) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420) at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
stackTrace: java.lang.Thread.State: RUNNABLE at df.JadePage.parseLIST(JadePage.java:1150) at df.JadePage.LoadPage(JadePage.java:397) at df.JadePage.LoadPage(JadePage.java:220) at df.JadePage.SetHTMLCode(JadePage.java:576) at df.JadePage.LoadPage(JadePage.java:361) at df.JadePage.SetHTMLCode(JadePage.java:576) at df.JadePage.LoadPage(JadePage.java:361) at df.Framework.Execute(Framework.java:297) at jsp_servlet.__server._jspService(__server.java:983) at weblogic.servlet.jsp.JspBase.service(JspBase.java:35) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:295) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:353) at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at ResponseHeaderFilter.doFilter(ResponseHeaderFilter.java:33) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3800) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3766) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344) at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197) at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203) at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71) at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2454) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2302) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2280) at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1721) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1681) at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272) at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352) at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337) at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57) at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41) at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:655) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420) at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
Edit your Comment