Corrective action plan to get rid out of high CPU & Memory utilisation issue

Hello Jignesh!

 Greetings.  I reviewed your report and share following observations:

a. http-0.0.0.0-8080-52 - High CPU consuming thread:

 http-0.0.0.0-8080-52 is reported to conusme high CPU ie 99.99%. However this thread's stack trace is given below:

java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:906)
- locked <0x00007fc158d02620> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:932)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None

 Actually this  thread is in WAITING state. WAITING threads don't consume CPU. I suspect you didn't capture 'thread -H -p {PID}'  and 'thread dump' at the same point in time. That's why we are seeing this disconnect. You can use this open source script, which captures 'thread -H -p {PID}'  and 'thread dump' concurrently and few more other artifacts. It might help you spot CPU consuming thread/code accurately.

 b. 43 BLOCKED Threads:

 I could see 43 threads to be in BLOCKED state. This might cause your application go unresponsive. Below is the transitive dependecy graph of BLOCKED threads from your report:

 If you notice all these 43 threads are BLOCKED by the 'http-0.0.0.0-8080-24' thread. Below is the stack of this thread:

http-0.0.0.0-8080-24
Stack Trace is:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:153)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at com.edb.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:146)
at com.edb.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:115)
at com.edb.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:197)
at java.io.InputStream.read(InputStream.java:101)
at com.edb.core.PGStream.ReceiveInteger4(PGStream.java:312)
at com.edb.core.PGStream.ReceiveTupleV3(PGStream.java:409)
at com.edb.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2548)
at com.edb.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:789)
- locked <0x00007fbe1b27c8f0> (a com.edb.core.v3.QueryExecutorImpl)
at com.edb.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:541)
at com.edb.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:459)
at com.edb.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:293)
at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)
at hisglobal.persistence.HelperMethodsDAO.executeQuery(HelperMethodsDAO.java:232)
at new_investigation.transactions.dao.Helper.InvestigationTemplateDataHelper.getCriteriaCode(InvestigationTemplateDataHelper.java:920)
at new_investigation.transactions.controller.Helper.TemplateProcessingUSE.processTheTestDocument(TemplateProcessingUSE.java:1478)
at new_investigation.transactions.controller.Helper.TemplateProcessingUSE.getResultEntryTemplatedocument(TemplateProcessingUSE.java:337)
- locked <0x00007fbdcbde9048> (a java.lang.Class for new_investigation.transactions.controller.Helper.TemplateProcessingUSE)
at new_investigation.transactions.controller.Helper.TemplateProcessingUSE.groupSelectedWorkOrders(TemplateProcessingUSE.java:230)
at new_investigation.transactions.controller.utl.InvResultEntryUTIL.showResultEntryPatDetails(InvResultEntryUTIL.java:549)
at new_investigation.transactions.controller.action.InvResultEntryACT.SHOWPATDETAILS(InvResultEntryACT.java:303)
at sun.reflect.GeneratedMethodAccessor8194.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.struts.actions.DispatchAction.dispatchMethod(DispatchAction.java:280)
at org.apache.struts.actions.DispatchAction.execute(DispatchAction.java:216)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
at hisglobal.presentation.HisRequestProcessor.process(HisRequestProcessor.java:18)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:324)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
at hissso.client.filter.HISSSOClientRequestFilter.doFilter(HISSSOClientRequestFilter.java:194)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
at org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285)
at org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:654)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:951)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00007fbdf462dd78> (a java.util.concurrent.locks.ReentrantLock$FairSync)

 From the stacktrace you can observe this thread is making following database call to 'new_investigation.transactions.dao.Helper.InvestigationTemplateDataHelper.getCriteriaCode(InvestigationTemplateDataHelper.java:920)' & this database call is taking forever to complete. Thus other threads are getting BLOCKED. You might want to investigate why this database call is taking such long time to complete. Your problem sounds very similar to this issue.