Hello Shankar Garlapati,
Greetings!
Inorder to diagnose CPU spike there is two approaches:
- Strategic/right approach
- Tactical approach
Strategic Approach:
Inorder to accurately point the lines of code causing the CPU spike, you need to analyze not only thread dumps but also 'top -H -p {PID}' command output, where {PID} is your Java application's process Id which is experiencing CPU spike. When you issue this ‘top’ command with given arguments, it will list all the threads running in the application and amount of CPU each one of the thread consume. Once you have both the data, you can identify high CPU consuming thread and lines of code they are executing.
You can use the open source yCrash script which will capture 360-degree application level artifacts (like GC logs, 3 snapshots of thread dumps, heap dumps) and system level artifacts (like top, top -H, netstat, vmstat, iostat, dmesg, diskusage, kernel parameters...). Once you have these data, either you can manually analyze them or upload it to yCrash tool. Tool analyzes all these dataset and generates an instant root cause analysis report pointing out exact line of code causing the CPU spike. Here is more detailed information on how to diagnose high CPU spike.
Tactical approach:
fastThread tool uses its heuristics algorithms and shows the potential (but not exact) threads are consuming high cpu in the section 'CPU consuming threads'. You want to refer to see the threads pointed in this 'CPU consuming threads' section.
In your case, I could see the following threads reported in ‘CPU consuming threads’ section to be potential suspects for CPU spike:
[ACTIVE] ExecuteThread: '40' for queue: 'weblogic.kernel.Default (self-tuning)'
nativeId: 18279
stackTrace: java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:378) - locked <0x000000061be25928> (a java.lang.Object) at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:175) at oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:571) at oracle.net.ns.NIOPacket.readHeader(NIOPacket.java:259) at oracle.net.ns.NIOPacket.readPacketFromSocketChannel(NIOPacket.java:191) at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:133) at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:106) at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:92) at oracle.net.ano.CryptoNIONSDataChannel.readDataFromSocketChannel(Unknown Source) at oracle.jdbc.driver.T4CMAREngineNIO.prepareForUnmarshall(T4CMAREngineNIO.java:765) at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:430) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:408) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:269) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:656) at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:271) at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:92) at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:808) at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1044) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1243) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3719) at oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1427) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3766) - locked <0x000000061be1ad60> (a oracle.jdbc.driver.T4CConnection) at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1168) at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:144) at sun.reflect.GeneratedMethodAccessor343.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:122) at org.hibernate.engine.jdbc.internal.proxy.AbstractProxyHandler.invoke(AbstractProxyHandler.java:81) at com.sun.proxy.$Proxy350.executeQuery(Unknown Source) at org.hibernate.loader.Loader.getResultSet(Loader.java:1962) at org.hibernate.loader.Loader.doQuery(Loader.java:829) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:289) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259) at org.hibernate.loader.Loader.loadEntity(Loader.java:2042) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:82) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:72) at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3710) at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:439) at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:420) at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:204) at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:251) at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:148) at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:992) at org.hibernate.internal.SessionImpl.access$1900(SessionImpl.java:170) at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2336) at org.hibernate.internal.SessionImpl.get(SessionImpl.java:888) at com.splwg.base.support.context.FrameworkSession.get(FrameworkSession.java:1262) at com.splwg.base.support.context.FrameworkSession.get(FrameworkSession.java:1241) at com.splwg.base.api.datatypes.SingleFieldId.getEntity(SingleFieldId.java:182) at com.splwg.base.support.api.GenericLocaleDependentBusinessEntity.fetchAbstractLanguageEntity(GenericLocaleDependentBusinessEntity.java:63) at com.splwg.base.domain.common.uiMap.UiMap_Per.fetchLanguageEntity(UiMap_Per.java:207) at com.splwg.base.domain.common.uiMap.UiMap_Per.fetchLanguageDescription(UiMap_Per.java:230) at com.splwg.base.domain.web.RetrieveUIMapService.read(RetrieveUIMapService.java:48) at com.splwg.base.support.pagemaintenance.AbstractPageMaintenance.readItem(AbstractPageMaintenance.java:91) at com.splwg.base.support.service.JavaPageReadService$1.execute(JavaPageReadService.java:31) at com.splwg.base.support.service.JavaPageReadService$1.execute(JavaPageReadService.java:26) at com.splwg.base.support.context.FlushDisabledExecutable.doWithFlushesDisabled(FlushDisabledExecutable.java:27) at com.splwg.base.support.service.JavaPageReadService.read(JavaPageReadService.java:35) at com.splwg.base.support.service.PageReadService.privateExecute(PageReadService.java:55) at com.splwg.base.support.service.PageReadService.execute(PageReadService.java:39) at com.splwg.base.api.service.ServiceDispatcher.lambda$readItemInSession$8(ServiceDispatcher.java:637) at com.splwg.base.api.service.ServiceDispatcher$$Lambda$129/1344922163.apply(Unknown Source) at com.splwg.base.api.service.ServiceDispatcherExecutionWrapper.privateRead(ServiceDispatcherExecutionWrapper.java:88) at com.splwg.base.api.service.ServiceDispatcherExecutionWrapper.read(ServiceDispatcherExecutionWrapper.java:83) at com.splwg.base.api.service.ServiceDispatcher.readItemInSession(ServiceDispatcher.java:637) at com.splwg.base.support.service.ServiceReplayExecuter$1.execute(ServiceReplayExecuter.java:156) at com.splwg.base.support.service.ServiceReplayExecuter$1.execute(ServiceReplayExecuter.java:153) at com.splwg.base.support.context.SessionExecutable.privateDoInReadOnlySession(SessionExecutable.java:164) at com.splwg.base.support.context.SessionExecutable.lambda$doInReadOnlySession$1(SessionExecutable.java:149) at com.splwg.base.support.context.SessionExecutable$$Lambda$109/1813898214.execute(Unknown Source) at com.splwg.base.support.context.SessionExecutableWrapper.executeInReadOnlySession(SessionExecutableWrapper.java:57) at com.splwg.base.support.context.SessionExecutableWrapper.doInReadOnlySession(SessionExecutableWrapper.java:36) at com.splwg.base.support.context.SessionExecutable.doInReadOnlySession(SessionExecutable.java:149) at com.splwg.base.support.service.ServiceReplayExecuter.execute(ServiceReplayExecuter.java:120) at com.splwg.base.support.service.ServiceReplayExecuter.executeInAppropriateSession(ServiceReplayExecuter.java:114) at com.splwg.base.support.service.ServiceReplayExecuter.executeWithReplay(ServiceReplayExecuter.java:64) at com.splwg.base.support.service.ServiceReplayExecuter.executeWithReplay(ServiceReplayExecuter.java:52) at com.splwg.base.api.service.ServiceDispatcher.readItem(ServiceDispatcher.java:570) at com.splwg.ejb.service.impl.ServiceBean$1.compute(ServiceBean.java:129) at com.splwg.ejb.service.impl.ServiceBean$1.compute(ServiceBean.java:124) at com.splwg.ejb.service.impl.AbstractServiceBeanInvocationHelper.computeWithTranslatedExceptions(AbstractServiceBeanInvocationHelper.java:25) at com.splwg.ejb.service.impl.ServiceBeanInvocationHelper.execute(ServiceBeanInvocationHelper.java:12) at com.splwg.ejb.service.impl.ServiceBean.readSystem(ServiceBean.java:134) at sun.reflect.GeneratedMethodAccessor707.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101) at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:101) at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52) at sun.reflect.GeneratedMethodAccessor695.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:94) at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy347.readSystem(Unknown Source) at com.splwg.ejb.service.impl.SPLServiceBean_quc567_ServiceImpl.__WL_invoke(Unknown Source) at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invokeInternal(SessionRemoteMethodInvoker.java:54) at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:21) at com.splwg.ejb.service.impl.SPLServiceBean_quc567_ServiceImpl.readSystem(Unknown Source) at com.splwg.ejb.service.impl.SPLServiceBean_quc567_ServiceImpl_WLSkel.invoke(Unknown Source) at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:260) at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:552) at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:313) at com.splwg.ejb.service.impl.SPLServiceBean_quc567_ServiceImpl_12214_WLStub.readSystem(Unknown Source) at sun.reflect.GeneratedMethodAccessor706.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at weblogic.ejb.container.internal.RemoteBusinessIntfProxy.invoke(RemoteBusinessIntfProxy.java:89) at com.sun.proxy.$Proxy335.readSystem(Unknown Source) at com.splwg.serviceclient.RemoteServiceDispatcher$1.executeService(RemoteServiceDispatcher.java:47) at com.splwg.serviceclient.RemoteServiceDispatcher$1.executeService(RemoteServiceDispatcher.java:44) at com.splwg.serviceclient.RemotePageServiceDispatcherHelper.doIt(RemotePageServiceDispatcherHelper.java:19) at com.splwg.serviceclient.RemoteServiceDispatcher.readSystem(RemoteServiceDispatcher.java:49) at com.splwg.base.web.userMap.UIMapRetrieverHelper.privateFetchUIMapDetails(UIMapRetrieverHelper.java:88) at com.splwg.base.web.userMap.UIMapRetrieverHelper.fetchMap(UIMapRetrieverHelper.java:72) at com.splwg.base.web.userMap.UIMapRetrieverHelper.fetchUIMap(UIMapRetrieverHelper.java:58) at com.splwg.base.web.userMap.IncludeProcessor.retrieveUIMapFragment(IncludeProcessor.java:34) at com.splwg.base.web.userMap.IncludeProcessor.processMatch(IncludeProcessor.java:70) at com.splwg.base.web.userMap.AbstractProcessor.processMatches(AbstractProcessor.java:25) at com.splwg.base.web.userMap.IncludeProcessor.expandIncludes(IncludeProcessor.java:53) at com.splwg.base.web.userMap.UserMapHTMLProcessor.privateAddStaticJavascript(UserMapHTMLProcessor.java:439) at com.splwg.base.web.userMap.UserMapHTMLProcessor.addStaticJavascript(UserMapHTMLProcessor.java:121) at com.splwg.base.web.userMap.ZoneXSLMapCache.addJavascriptMetadata(ZoneXSLMapCache.java:147) at com.splwg.base.web.userMap.ZoneXSLMapCache.computeMap(ZoneXSLMapCache.java:132) at com.splwg.base.web.userMap.ZoneXSLMapCache.retreiveMapFromCache(ZoneXSLMapCache.java:93) at com.splwg.base.web.userMap.ZoneXSLMapCache.privateGetMap(ZoneXSLMapCache.java:81) at com.splwg.base.web.userMap.ZoneXSLMapCache.getMap(ZoneXSLMapCache.java:44) at com.splwg.base.web.userMap.ZoneMapHelper.createMapFromSchema(ZoneMapHelper.java:333) at com.splwg.base.web.userMap.ZoneMapHelper.getMapFromMapZoneParm(ZoneMapHelper.java:366) at com.splwg.base.web.userMap.ZoneMapHelper.retrieveMap(ZoneMapHelper.java:159) at com.splwg.base.web.userMap.AbstractUiMapHelper.createUiMap(AbstractUiMapHelper.java:70) at com.splwg.base.web.userMap.ZoneMapHelper.createHtmlMap(ZoneMapHelper.java:113) at com.splwg.base.web.userMap.ZoneMap.getHtmlMapResult(ZoneMap.java:15) at com.splwg.base.web.userMap.AbstractUiMapServlet.doPost(AbstractUiMapServlet.java:46) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:295) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:353) at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at com.splwg.base.web.services.CSPSecurityFilter.doFilter(CSPSecurityFilter.java:103) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at com.splwg.base.web.services.CSPReportOnlySecurityFilter.doFilter(CSPReportOnlySecurityFilter.java:41) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at com.splwg.base.web.services.RequestContextFilter.doFilter(RequestContextFilter.java:64) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at com.splwg.base.web.utility.OJETMappingFilter.doFilter(OJETMappingFilter.java:95) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at com.splwg.base.web.utility.TopReplaceFilter.doFilter(TopReplaceFilter.java:48) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at com.splwg.base.web.utility.DoctypeReplaceFilter.doFilter(DoctypeReplaceFilter.java:36) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at com.splwg.base.web.services.SecurityFilter.forwardRequest(SecurityFilter.java:96) at com.splwg.base.web.services.SecurityFilter.doFilter(SecurityFilter.java:76) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at com.splwg.base.web.utility.SessionTimeOutFilter.doFilter(SessionTimeOutFilter.java:47) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at com.splwg.base.web.utility.CompressionFilter.doFilter(CompressionFilter.java:46) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at com.splwg.base.web.utility.StrictTransportSecurityFilter.doFilter(StrictTransportSecurityFilter.java:57) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at com.splwg.base.web.services.HttpMethodSecurityFilter.doFilter(HttpMethodSecurityFilter.java:47) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:92) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:248) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at oracle.jrf.servlet.ExtensibleGlobalFilter.doFilter(ExtensibleGlobalFilter.java:92) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3800) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3766) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344) at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197) at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203) at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71) at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2454) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2302) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2280) at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1710) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1670) at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272) at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352) at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337) at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57) at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41) at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:655) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420) at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
Thanks.
Edit your Comment