Application server doesn't serve request due to blocked thread

Hello Alfredo!

 Greetings. I could see the potential reason for your application to go unresponsive. Below is the excerpt from your thread dump report: 

This transitive dependency graph is showing that 'pool-5-thread-1' is BLOCKING 25 threads from progressing forward. This could potentially make your application go unresponsive. Below is the top part of the 'pool-5-thread-1' stacktrace:

pool-5-thread-1
Stack Trace is:
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked <0x00000007d7db2a48> (a java.io.IOException)
at java.lang.Throwable.<init>(Throwable.java:265)
at java.lang.Exception.<init>(Exception.java:66)
at java.io.IOException.<init>(IOException.java:58)
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:345)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:95)
- locked <0x00000006c0c03180> (a java.io.BufferedOutputStream)
at java.io.PrintStream.write(PrintStream.java:449)
- locked <0x00000006c0c03160> (a java.io.PrintStream)
at org.jboss.logmanager.handlers.UncloseableOutputStream.write(UncloseableOutputStream.java:39)
at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
at org.jboss.logmanager.handlers.UninterruptibleOutputStream.write(UninterruptibleOutputStream.java:59)
at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
at org.jboss.logmanager.handlers.UninterruptibleOutputStream.write(UninterruptibleOutputStream.java:87)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
- locked <0x00000006c0c03120> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129)
- locked <0x00000006c0c03120> (a java.io.OutputStreamWriter)
at java.io.BufferedWriter.write(BufferedWriter.java:230)
- locked <0x00000006c0c03120> (a java.io.OutputStreamWriter)
at java.io.Writer.write(Writer.java:157)
at org.jboss.logmanager.handlers.WriterHandler.doPublish(WriterHandler.java:68)
- locked <0x00000006c0c030e0> (a java.lang.Object)
at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:79)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:296)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:304)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:304)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:304)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:304)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:304)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:304)
at org.jboss.logmanager.Logger.logRaw(Logger.java:721)
at org.apache.log4j.Category.callAppenders(Category.java:67)
at org.apache.log4j.Category.forcedLog(Category.java:119)
at org.apache.log4j.Category.info(Category.java:227)

:

:

Basically this stacktrace tells that your application is using log4j which is internally using JBOSS logmanager to write the log messages to the file. Unfortunately JBOSS logmanager is unable to write the contents to file and it's getting stuck. Before it got stuck it acquired 8 locks . Due to which 25 other threads are strandarded.

From the stacktrace it isn't clear to me why logmanager isn't able to the log message to the file. But here are the potential reasons:

a. Server ran out of diskspace

b. There is some file/disk contention

c. There is some OS level issue going that is preventing the writing

d. File lock taken away

e. File permissions changed

You may want to investigate from this prespective and identify the root cause.

Thanks.

Hi Ram,

thank you very much for your suggestion. We are trying to understand why it happens.

Alfredo

Hi Ram,

i resolved the problem. Looking at the stacktrace I noticed that the blocking thread had always the same stacktrace.

Analyzing the involved classes  I discovered that the problem was linked to a concurrence issue caused by a process in execution in both the application servers .

Thanks a lot for your support .

Alfredo

There was the same process running on two different servers trying to execute the same update .

We disactivated the execution on one of the two servers and the thread deadlock didn't verifiy anymore. 

We assume that the two processes were going to randomly execute the same transaction at the same time that it failed causing the threads to hang.