Why CPU Spike happened

Hi Vikas,

As per the attached report, threads are stuck in the "socketRead0" method and waiting for the response from the external service/resource. These threads are involved in performing I/O operations, particularly network-related I/O operations in your application. If a thread remains stuck for an extended period, it can result in high CPU usage. To resolve this issue, I recommend visiting this blog for further details.

Here is a stacktrace of the thread that is stuck:

OkHttp 100.64.0.1
PRIORITY : 5
THREAD ID : 0X00007FE3E85AE800
NATIVE ID : 0X1A80
NATIVE ID (DECIMAL) : 6784
STATE : RUNNABLE

stackTrace:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(java.base@11.0.18/Native Method)
at java.net.SocketInputStream.socketRead(java.base@11.0.18/SocketInputStream.java:115)
at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:168)
at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:140)
at sun.security.ssl.SSLSocketInputRecord.read(java.base@11.0.18/SSLSocketInputRecord.java:484)
at sun.security.ssl.SSLSocketInputRecord.readHeader(java.base@11.0.18/SSLSocketInputRecord.java:478)
at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(java.base@11.0.18/SSLSocketInputRecord.java:70)
at sun.security.ssl.SSLSocketImpl.readApplicationRecord(java.base@11.0.18/SSLSocketImpl.java:1455)
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(java.base@11.0.18/SSLSocketImpl.java:1066)
at okio.InputStreamSource.read(JvmOkio.kt:91)
at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:129)
at okio.RealBufferedSource.request(RealBufferedSource.kt:206)
at okio.RealBufferedSource.require(RealBufferedSource.kt:199)
at okhttp3.internal.http2.Http2Reader.nextFrame(Http2Reader.kt:89)
at okhttp3.internal.http2.Http2Connection$ReaderRunnable.invoke(Http2Connection.kt:618)
at okhttp3.internal.http2.Http2Connection$ReaderRunnable.invoke(Http2Connection.kt:609)
at okhttp3.internal.concurrent.TaskQueue$execute$1.runOnce(TaskQueue.kt:98)
at okhttp3.internal.concurrent.TaskRunner.runTask(TaskRunner.kt:116)
at okhttp3.internal.concurrent.TaskRunner.access$runTask(TaskRunner.kt:42)
at okhttp3.internal.concurrent.TaskRunner$runnable$1.run(TaskRunner.kt:65)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
Locked ownable synchronizers:
- <0x0000000627948098> (a java.util.concurrent.ThreadPoolExecutor$Worker)
- <0x00000006cce50440> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

Note - In some cases, the stuck thread may not cause high CPU usage directly but can still lead to degraded performance, unresponsiveness, or resource contention issues in the application.

Inorder to diagnose CPU spike there is two approaches:

1. Strategic/right approach
2. Tactical approach

Tactical approach:

 fastThread tool uses its heuristics algorithms and shows the potential (but not exact) threads are consuming high cpu in the section 'CPU consuming threads'.  You want to refer to see the threads pointed in this 'CPU consuming threads' section.

 In your case, I could following threads reported in ‘CPU consuming threads’ section to be potential suspects for CPU spike:

Below is the stack trace of the thread that is consuming high CPU:

java.lang.Thread.State: RUNNABLE
at java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(java.base@11.0.18/ThreadLocal.java:637)
at java.lang.ThreadLocal$ThreadLocalMap.remove(java.base@11.0.18/ThreadLocal.java:519)
at java.lang.ThreadLocal.remove(java.base@11.0.18/ThreadLocal.java:242)
at com.sun.xml.bind.v2.runtime.Coordinator.popCoordinator(Coordinator.java:135)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext.endElement(UnmarshallingContext.java:617)
at com.sun.xml.bind.v2.runtime.unmarshaller.InterningXmlVisitor.endElement(InterningXmlVisitor.java:81)
at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleEndElement(StAXStreamConnector.java:221)
at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:185)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:415)
at com.sun.xml.bind.v2.runtime.BridgeImpl.unmarshal(BridgeImpl.java:124)
at com.sun.xml.bind.api.Bridge.unmarshal(Bridge.java:237)
at com.sun.xml.ws.db.glassfish.BridgeWrapper.unmarshal(BridgeWrapper.java:272)
at com.sun.xml.ws.client.sei.ResponseBuilder$WrappedPartBuilder.readResponse(ResponseBuilder.java:134)
at com.sun.xml.ws.client.sei.ResponseBuilder.readWrappedResponse(ResponseBuilder.java:163)
at com.sun.xml.ws.client.sei.ResponseBuilder$RpcLit.readResponse(ResponseBuilder.java:729)
at com.sun.xml.ws.client.sei.StubHandler.readResponse(StubHandler.java:258)
at com.sun.xml.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:206)
at com.sun.xml.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:293)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:119)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:92)
at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:161)
at com.sun.proxy.$Proxy529.accountRead(Unknown Source)
at com.nokia.tpapps.servicemanager.server.current.subscriber.AccountWsEntityConfig.distributeObject(AccountWsEntityConfig.java:211)
at com.nokia.tpapps.servicemanager.broker.service.impl.current.subscriber.AccountBrokerServiceImpl.distributeObject(AccountBrokerServiceImpl.java:127)
at com.nokia.tpapps.servicemanager.trans.current.subscriber.SubscriberDataTrans.distribute(SubscriberDataTrans.java:744)
at com.nokia.tpapps.servicemanager.trans.current.subscriber.SubscriberDataTrans.distribute(SubscriberDataTrans.java:567)
at com.nokia.tpapps.servicemanager.trans.current.subscriber.AccountDataTrans.distribute(AccountDataTrans.java:986)
at com.nokia.tpapps.servicemanager.providers.subscriber.SubscriberGetProvider.execute(SubscriberGetProvider.java:135)
at com.nokia.tpapps.servicemanager.providers.subscriber.SubscriberGetProvider.execute(SubscriberGetProvider.java:242)
at com.nokia.tpapps.servicemanager.providers.subscriber.SubscriberGetProvider.execute(SubscriberGetProvider.java:54)
at com.nokia.tpapps.servicemanager.services.current.impl.subscriber.SubscriberServiceImpl.getEntityData(SubscriberServiceImpl.java:1533)
at com.nokia.tpapps.servicemanager.services.current.impl.subscriber.SubscriberServiceImpl.getEntity(SubscriberServiceImpl.java:3037)
at com.nokia.tpapps.servicemanager.services.current.impl.subscriber.SubscriberServiceImpl.getAccount(SubscriberServiceImpl.java:1058)
at jdk.internal.reflect.GeneratedMethodAccessor1436.invoke(Unknown Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.18/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.18/Method.java:566)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$1162/0x0000000801e43840.invoke(Unknown Source)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at com.eclipsesource.jaxrs.publisher.internal.ServletContainerBridge.service(ServletContainerBridge.java:76)
at org.apache.felix.http.base.internal.handler.ServletHandler.doHandle(ServletHandler.java:96)
at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:79)
at org.apache.felix.http.base.internal.dispatch.ServletPipeline.handle(ServletPipeline.java:42)
at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:49)
at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at com.nokia.tpapps.metrics.http.HTTPMetricsServiceImpl.doFilter(HTTPMetricsServiceImpl.java:144)
at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at com.nokia.tpapps.spscommon.filter.RequestBlockerInterceptor.doFilter(RequestBlockerInterceptor.java:83)
at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at com.nokia.tpapps.spscommon.filter.ReRateEDRFilter.doFilter(ReRateEDRFilter.java:180)
at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at com.nokia.tpapps.servicemanager.core.filter.ResponseFilter.doFilter(ResponseFilter.java:118)
at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at com.nokia.tpapps.spscommon.filter.ServiceProviderFilter.doFilter(ServiceProviderFilter.java:74)
at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at com.nokia.tpapps.metrics.http.HttpMetricFilter.doFilter(HttpMetricFilter.java:111)
at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at com.nokia.tpapps.spscommon.filter.MeAuditRequestFilter.doFilter(MeAuditRequestFilter.java:76)
at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at com.nokia.tpapps.spscommon.filter.LoggingRequestFilter.doFilter(LoggingRequestFilter.java:198)
at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at com.nokia.tpapps.spscommon.filter.ExternalTxHeaderFilter.doFilter(ExternalTxHeaderFilter.java:112)
at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at com.alcatel.tpapps.common.common.opentracing.OpenTracingRequestFilter.doFilter(OpenTracingRequestFilter.java:106)
at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at com.nokia.tpapps.spscommon.filter.GZIPRequestFilter.doFilter(GZIPRequestFilter.java:119)
at org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at org.apache.felix.http.base.internal.dispatch.FilterPipeline.dispatch(FilterPipeline.java:48)
at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:39)
at org.apache.felix.http.base.internal.DispatcherServlet.service(DispatcherServlet.java:67)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel$$Lambda$1270/0x0000000801f4d040.dispatch(Unknown Source)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at com.alcatel_lucent.as.service.jetty.common.connector.AbstractBufferEndPoint.lambda$fillable$0(AbstractBufferEndPoint.java:72)
at com.alcatel_lucent.as.service.jetty.common.connector.AbstractBufferEndPoint$$Lambda$1268/0x0000000801f4e440.run(Unknown Source)
at com.alcatel.as.service.concurrent.impl.Helper.runTask(Helper.java:57)
at com.alcatel.as.service.concurrent.impl.QueueExecutor.lambda$execute$5(QueueExecutor.java:211)
at com.alcatel.as.service.concurrent.impl.QueueExecutor$$Lambda$1002/0x0000000801a5d040.run(Unknown Source)
at com.alcatel.as.service.concurrent.impl.SerialQueue.run(SerialQueue.java:95)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
Locked ownable synchronizers:
- <0x000000067bc72db0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Strategic Approach:

 Inorder to accurately point the lines of code causing the CPU spike, you need to analyze not only thread dumps but also 'top -H -p {PID}' command output, where {PID} is your Java application's process Id which is experiencing CPU spike. When you issue this ‘top’ command with given arguments, it will list all the threads running in the application and amount of CPU each one of the thread consume. Once you have both the data, you can identify high CPU consuming thread and lines of code they are executing.

 You can use the open source yCrash script which will capture 360-degree application level artifacts (like GC logs, 3 snapshots of thread dumps, heap dumps) and system level artifacts (like top, top -H, netstat, vmstat, iostat, dmesg, diskusage, kernel parameters...). Once you have these data, either you can manually analyze them or upload it to yCrash tool. Tool analyzes all these dataset and generates an instant root cause analysis report pointing out exact line of code causing the CPU spike. Here is more detailed information on how to diagnose high CPU spike.