Profile Image
Bipin Tiwari

threads are consuming high cpu

Hi,

There is no change in infra or code . but suddenly we started receiving high CPU alerts and it sorted only after terminating the affected node. please advise the solution to fix this issue . from captured thread dumps I am not able to get specific input.

 

Report URL - https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjIvMDEvNS8tLWh5YnJpcy1zdG9yZWZyb250LWdyZWVuLTg2Yjc5NGJmOGMtZmttcG0uemlwLS03LTM5LTM5

  • highcpu

  • garbagecollection

  • topcommandoutput

  • highcpuconsuming

  • cpuspikes

Please Sign In or to post your comment or answer

Profile Image

Ankita

Hi Bipin,

 

fastThread tool uses its heuristics algorithms and shows the potential (but not exact) threads are consuming high cpu in the section 'CPU consuming threads'.  You want to refer to see the threads pointed in this 'CPU consuming threads' section.

 

 

+  Some times when garbage collection runs in background that can also cause the CPU to spike up. 

 

Inorder to accurately point the lines of code causing the CPU spike, you need to analyze not only thread dumps but also 'top -H -p {PID}' command output, where {PID} is your Java application's process Id which is experiencing CPU spike. When you issue this ‘top’ command with given arguments, it will list all the threads running in the application and amount of CPU each one of the thread consume. Once you have both the data, you can identify high CPU consuming thread and lines of code they are executing. You can either capture these artifacts and analyze them manually, or use the 14 day trial version of yCrash tool. yCrash tool automatically captures  application-level data (thread dump, heap dump, Garbage Collection log) and system-level data (netstat, vmstat, iostat, top, top -H, dmesg,...). It marries these two datasets and generates an instant root cause analysis report pointing out exact line of code causing the CPU spike. Here is more information on how to diagnose high CPU spike.

 

Profile Image

Umayal

Hi Bipin,

 

We can notice that 2 threads are looping on read0() method in sun.nio.ch.FileDispatcherImpl file. If threads loop infinitely, CPU consumption will start to spike up. Examine their stacktrace.

 

Following is the stack Trace:

stackTrace:
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.read0(java.base@11.0.4/Native Method)
at sun.nio.ch.FileDispatcherImpl.read(java.base@11.0.4/FileDispatcherImpl.java:48)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(java.base@11.0.4/IOUtil.java:276)
at sun.nio.ch.IOUtil.read(java.base@11.0.4/IOUtil.java:245)
at sun.nio.ch.FileChannelImpl.read(java.base@11.0.4/FileChannelImpl.java:223)
- locked <0x00000003962a9648> (a java.lang.Object)
at com.amazonaws.services.kinesis.producer.Daemon.readSome(Daemon.java:550)
at com.amazonaws.services.kinesis.producer.Daemon.receiveMessage(Daemon.java:246)
at com.amazonaws.services.kinesis.producer.Daemon.access$500(Daemon.java:63)
at com.amazonaws.services.kinesis.producer.Daemon$3.run(Daemon.java:301)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.4/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.4/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.4/Thread.java:834)

Profile Image

Ram Lakshmanan

Hello Bipin!

 

 I see two suspects:

 

a. There is lot of 'hybrisHTTP' threads active. They can cause CPU to spike.

 

b. As ankita as pointed out it could be related to GC activity as well.

 

Irrespective of both scenarios, you can use 14 day yCrash trial to identify the root cause of the problem.

Profile Image

Umayal

Hi Bipin,

 

I can also see that some threads are BLOCKED but this won't make the CPU to spike up. If threads are BLOCKED for a prolonged period, your application can become unresponsive. You can examine their stack trace.

 

 

 

 

 

 

Got something else on mind? Post Your Question

Not the answer you're looking for? Browse other questions tagged
  • highcpu

  • garbagecollection

  • topcommandoutput

  • highcpuconsuming

  • cpuspikes