What could be the reason many threads waiting?

Hello Santosh!

 Your application has only 188 threads in total. In that 188 threads, 80 threads are in WAITING state. There are couple of reasons why these many threads are in WAITING state.

1. ForJoingPool thread pool

In this 80 WAITING threads, 38 of them have below stack trace:

java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.12/Native Method)
- parking to wait for <0x000000055aee9388> (a java.util.concurrent.ForkJoinPool)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.12/LockSupport.java:194)
at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.12/ForkJoinPool.java:1628)
at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.12/ForkJoinWorkerThread.java:183)
Locked ownable synchronizers:
- None

All these threads are originating from ForkJoinPool-4 and ForkJoinPool-5 thread pool. I suspect you have kept your min thread pool size to be very high for these pools. That might be causing the threads to go in to WAITING state and not do anything. If you can lower the min thread pool size, idle threads would exit from the pool. 

2. Curator-PathChildrenCache

 In this 80 WAITING threads, 11 of them have below stack trace:

java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.12/Native Method)
- parking to wait for <0x00000005564135e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.12/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.12/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.12/LinkedBlockingQueue.java:433)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.12/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.12/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.12/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.12/Thread.java:829)
Locked ownable synchronizers:
- None

 All of these threads are originating from Curator-PathChildrenCache thread pool. 

I suspect you have kept your min thread pool size to be very high for this pool. That might be causing the threads to go in to WAITING state and not do anything. If you can lower the min thread pool size, idle threads would exit from the pool. 

 Fixing above two changes should lower your WAITING threads count in your application.