Why so many threads are in waiting state and they dont come out ever?

Hello Surabhi!

 Greetings. Your application is having lot of waiting threads because of following reasons:

a. c3p0 thread pool mis-configuration?

Below are your major thread groups:

 If you notice you 'C3P0PooledConnectionPoolManager' thread group is your largest group and it has 66 threads. Basically this is the thread group for c3p0 - JDBC Connection pool. In this thread group 55 threads have the following stacktrace:

stackTrace:
java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
- waiting on <2bad3cbe> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
Locked ownable synchronizers:
- None

These threads aren't doing much, they are just waiting. Your c3p0 thread pool configuration doesn't sound right. You might want to check the c3p0 thread pool configuration. May be you want to lower the minimum thread pool count for this thread pool.

b. JMX connections not closed properly?

 There are 21 JMX threads in TIMED_WAITING state with following stacktrace:

java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
- waiting on <1db29a2f> (a [I)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)
at java.lang.Thread.run(Thread.java:750)
Locked ownable synchronizers:
- None

 Apparently this type of issue will surface if JMX connections aren't closed explicity. Whenever you open JMX connection, you need to close it. Example:

JMXConnector connector = JMXConnectorFactory.connect(url);
//...
connector.close();

c. Minimum thread pool not configured?

 In your application there are thread groups by this name: 'StatusStorerModule' and 'AuditEventStorerModule'. Most of the threads in this thread group have the following stacktrace:

stackTrace:
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <5a443681> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2044)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Locked ownable synchronizers:
- None

 It indicates these threads aren't doing anything and just waiting for new jobs. What is the minimum thread pool size for these thread pools? You might want to configure lower value for minimum thread pool size, so that when thread pool is inactive, threads will die down from this pool.