Tomcat is utilizing high CPU sometimes and needs help in analyzing the thread dump

Hello Devidutta!

 Greetings. I have couple of observations on your thread dump:

1. Log4j - degrading your application's performance

 Above is the transitive dependency graph excerpt from your application's thread dump analysis report generated by fastThread. You can notice that Log4J code is blocking 17 threads. These 17 threads are actually processing your customer transactions, thus they will experience poor response time.

 Apparently your application is using 'Log4J' library for logging. 'Log4j' library has lot of racing condition/synchronization related bugs. Infact it has been acknowledged in several of their bugs: 

https://bz.apache.org/bugzilla/show_bug.cgi?id=57714

https://bz.apache.org/bugzilla/show_bug.cgi?id=50614

 These issues are NOT fixed in 'Log4j' library, rather they have fixed in 'Log4j2' library.  Infact 'Log4j' has reached end of life several years back (i.e. August 5, 2015). So if you upgrade to 'Log4j2' library, you can see good boost in your application's response time.

2. High CPU utilization 

 Inorder to accurately point the lines of code causing the CPU spike, you need to analyze not only thread dumps but also 'top -H -p {PID}' command output, where {PID} is your Java application's process Id which is experiencing 100% CPU spike. When you issue this command, it will list all the threads running in the application and amount of CPU each one of the thread consume. Once you have both the data, you can identify high CPU consuming thread and lines of code they are executing. You can either capture these artifacts and analyze them manually, or use the yCrash tool - which automatically captures  application-level data (thread dump, heap dump, Garbage Collection log) and system-level data (netstat, vmstat, iostat, top, top -H, dmesg,...). It marries these two datasets and generates an instant root cause analysis report. Here is more information on how to diagnose high CPU spike.