heap memory is 9GB

Hi atel,

As per my observations, there are 453 threads in the WAITING state and they all have the same stack trace. Here is the stack trace of one thread:

stackTrace:
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000058188d808> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None

There could be several reasons why the application is going unresponsive. Some of the reasons are:

• Garbage collection pauses
• Threads getting BLOCKED
• Network connectivity
• Load balancer routing issue
• Heavy CPU consumption of threads
• Operating System running with old patches
• Memory Leak
• DB not responding properly

:

:

 So just thread dump is not enough to diagnose the problem. Besides thread dumps, you might have to capture other logs/artifacts to do a thorough analysis.

 You can use the open source yCrash script which will capture 360-degree application level artifacts (like GC logs, 3 snapshots of thread dumps, heap dumps) and system level artifacts (like top, top -H, netstat, vmstat, iostat, dmesg, diskusage, kernel parameters...). Once you have these data, either you can manually analyze them or upload it to yCrash tool, which will analyze all these artifacts and generate one unified root cause analysis marrying all these artifacts. It can indicate the root cause of the problem.

Hello Atel!

 Greetings. All the 453 threads that are in WAITING state are originating from 'default-task' thread pool. It looks like once threads are created in this pool they are not exiting (i.e., die down) even when there is no work. Can you check the thread pool configuration of 'default-task'. Is minimum thread pool size set? Is it set to a lower value?