Profile Image
Lucas Tomasini

can you help me in detecting why the number of blocked threads are high?

Wonder if you can help me detect why the number of blocked threads are high in this dump. This is a dump of one of our instancia before it crashes.



Report URL - https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjEvMDMvOS8tLTAuMTQ0LjItaG90Zml4LTItMTAuNTAuNDguMTQyLTIwMjEwMzA4MTcwNzM3LmxvZy0tMjAtMzAtMjM=

  • blockedthreads

  • threaddump

Please Sign In or to post your comment or answer

Profile Image

Ram Lakshmanan

Hello Lucas!

 

 Greetings. Here are some quick observations, I would like to share.

 

a. Your application is using Log4j framework. Log4j framework has reached end of life on Aug' 2015. It has several performance problems. I could see 15 threads (originating from your tomcat container) stuck in Log4j logging. Log4j framework has several performance problems. Below are those 15 stuck threads stack trace. This may cause unresponsiveness in your application.

 

 

Here is a similar log4j problem faced by another customer. You should consider upgrading to Log4j2 framework (or something similar).

 

b. There is a good chance that log4j might be causing unresponsiveness in your application. But however it's hard to confirm with the data shared captured. You have captured only thread dump, that too one snapshot of it. It's always a good practice to capture 3 thread dumps in a gap of 10 seconds between each one. Besides thread dumps you might have to capture other logs to do thorough analysis. You may consider using tools like yCrash, which would captures application level logs (like GC logs, 3 snapshots of thread dumps, heap dumps) and system level logs (like top, top -H, netstat, vmstat, iostat, dmesg, diskusage...) and does complete analysis to identify the root cause of the problem. 

Profile Image

Lucas Tomasini

Hi Ram,

Thank you for your response!

I did upgrade to Log4j2 because of the performance issues you mentioned. As you can see in the stack trace, the group is org.apache.logging.log4j, which corresponds to Log4j2.


However the problem is still present. When the traffic increases, the instances crash.

Profile Image

Ram Lakshmanan

Hello Lucas!


 You are right, you are using Log4j2. I stand corrected. It isn't clear why 15+ threads are stuck in Log4j2 code. Could it be memory related issue? Are you seeing OutOfMemoryError in the logs?


 Running yCrash will capture 360-degree accurate data. It will help to debug the problem more effectively.  

Profile Image

Ram Lakshmanan

Hello Lucas!


 I had little bit more closer look at the 15 threads which are in BLOCKED state. Below is the portion of it's stacktrace. Actually your threads are getting blocked in TOMCAT server classloading level. 


 What version of Tomcat are you running? Here is a bug in tomcat https://bz.apache.org/bugzilla/show_bug.cgi?id=56530 - talking about the problem you are experiencing. It looks like this issue has been fixed since Tomcat 8.0.18. 
 


java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1212) - waiting to lock <0x00000004e2c5f440> (a java.lang.Object) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1180) at org.apache.logging.log4j.core.impl.ThrowableProxyHelper.loadClass(ThrowableProxyHelper.java:209) at org.apache.logging.log4j.core.impl.ThrowableProxyHelper.toExtendedStackTrace(ThrowableProxyHelper.java:114) at org.apache.logging.log4j.core.impl.ThrowableProxy.(ThrowableProxy.java:109) at org.apache.logging.log4j.core.impl.ThrowableProxy.(ThrowableProxy.java:93) at org.apache.logging.log4j.core.impl.Log4jLogEvent.getThrownProxy(Log4jLogEvent.java:629) at org.apache.logging.log4j.core.impl.Log4jLogEvent.serialize(Log4jLogEvent.java:742) at org.apache.logging.log4j.core.impl.Log4jLogEvent.createMemento(Log4jLogEvent.java:796) at org.apache.logging.log4j.core.appender.AsyncAppender.append(AsyncAppender.java:162) at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156) at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129) at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120) at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84) at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543) at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502) at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485) at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460) at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82) at org.apache.logging.log4j.core.Logger.log(Logger.java:161) at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2198) at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2152) at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2135) at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2011) at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1983) at org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:750) at 

Got something else on mind? Post Your Question

Not the answer you're looking for? Browse other questions tagged
  • blockedthreads

  • threaddump