JAVA springboot API showing sudden increase in CPU consumption

Hello Pooja!

 Greetings. Here are my initial thoughts:

1. Always CPU spikes are caused by threads that are in RUNNABLE state. Other thread states like WAITING, TIMED_WAITING, BLOCKED, NEW, TERMINATED will not cause CPU spike. 

2. It looks like you have captured when heap dump capture was in the process through the dynatrace one agent. You should have captured thread dumps before capturing heap dumps. When heap dump is captured, it freezes the JVM - thus we wouldn't what are the actual threads that are causing the problem.

3. Some times when garbage collection runs in background that can also cause the CPU to spike up. 

 Inorder to accurately point the lines of code causing the CPU spike, you need to analyze not only thread dumps but also 'top -H -p {PID}' command output, where {PID} is your Java application's process Id which is experiencing CPU spike. When you issue this ‘top’ command with given arguments, it will list all the threads running in the application and amount of CPU each one of the thread consume. Once you have both the data, you can identify high CPU consuming thread and lines of code they are executing. You can either capture these artifacts and analyze them manually, or use the 14 day trial version of yCrash tool. yCrash tool automatically captures  application-level data (thread dump, heap dump, Garbage Collection log) and system-level data (netstat, vmstat, iostat, top, top -H, dmesg,...). It marries these two datasets and generates an instant root cause analysis report pointing out exact line of code causing the CPU spike. Here is more information on how to diagnose high CPU spike.

Hi Ram,

Thanks for your reply.

The issue is that the service runs on Fargate in AWS so there is no way at moment I can connect to the task and manually capture thread dumps. The only feasible way is through Dynatrace and it gives the option of memory dumps only.

Hello Pooja!

 I see. If it is the case, can you capture thread dumps separately and few minutes later capture heap dumps and analyze them. Don't capture them together like it was done last earlier. 

 a. You can analyze the captured thread dumps in fastThread.

 b. You can use Eclipse MAT or HeapHero to analyze heap dumps. Here is a video clip that my assist you in analyzing heap dumps in Eclipse MAT.

 You can also check with amazon support whether there is a possibility to trigger shell script on demand in the fargate service? Let me also check whether there are any such options. 

Hi Ram, 

Thanks a lot for the pointers, I will check those today.

Meanwhile, what stands out as a hotspot wrt. CPU consumption is sun.security.rsa as shown in the snip below - 

When CPU spikes to 60% after 20 mins into the test, https-jsse-nio-*-exec threads consume the most CPU and this stacktrace shows the above methods in stacktrace.

Do you have any pointers around this?

Regards,

Hello Pooja!

 Screenshot of data that you have shared indicates CPU consumption is originating because of SSL handshake. But apparently SSL handshakes shouldn't consume this much CPU. Here is an excellent article, which summarizes that when Google switched their gmail service from http to https, it added the overhead of only 1% CPU. So I can think of only 2 reasons why SSL (i.e. https) to add so much CPU overhead:

a. Your connections are not persistent. Either your load testing tool or your application or AWS Fargate service is not keeping the HTTPS connections persistent. It's turning down the connections after every transaction. 

b. I have seen monitoring tools reporting wrong methods as CPU suspects.

Most likely it could #a in your case. Check your HTTPS connection settings. 

Thanks.

Hi Ram,

These are TCP connections to the database. We do have connection pooling enabled using HikariCP.

hikari.connectionTimeout=2000

hikari.initializationFailTimeout=0

hikari.maximumPoolSize=5

We have tried running the test with different PoolSize but CPU issue still persists.

Can it be related to any other configuration that is missing?

https://heaphero.io/my-heap-report.jsp?p=YXJjaGl2ZWQvMjAyMS8xMC8xLy0tZHVtcC5ocHJvZi03LTIwLTEuanNvbg==

Hi Ram,

Thanks for the reply. Application server is Tomcat and DB is Postgressql.

I was referring to DB connections as while looking at the hotspots, I found this stack where we can see that socketinputstream.socketread0 takes considerable network time, which doesn't look normal. Mostly I have seen this time under 1%.