Rising pool threads in TomCat

Hello JJ!

 Majority of your threads are in TIMED_WAITING state and most of them tend to have this sort of stack trace:

java.lang.Thread.State: TIMED_WAITING
prio=5 blockedtime=4 blockedcount=3 waitedtime=180048694 waitedcount=3957
at java.base@11.0.9/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11.0.9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@11.0.9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at java.base@11.0.9/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at java.base@11.0.9/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at java.base@11.0.9/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11.0.9/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11.0.9/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.9/java.lang.Thread.run(Thread.java:834)

Strong Suspect - 'Not shutting down executor':

 Your application is using internal thread pool (i.e. ThreadPoolExecutor). And it looks like you are creating new Thread Pools for very frequently (may be even every request) and not shutting down the thread pool explcitly after usage. This is obvious based on the thread names in your thread dump:

 Above is the excerpt from your thread dump analsyis report generated by fastThread showing the thread names. If you see pool counter is getting incremented but thread count remains to be 1. It tells, you are creating multiple thread pools, but each thread pool  just has 1 thread in it. After usage, you are not shutting down the executor. Inorder to shutdown the executor, you need to invoke the 'Shutdown()' API. 

 We have solved similar problem in another application and documented our findings here. Take a look at it, you might be also running in to same issue. 

 NOTE: Invoking 'shutdown()' API should resolve your immediate problem. However it's not a recommended practise to create multiple thread pools with each having 1 thread in it. You may consider creating 1 thread pool with multiple threads in it.

Thanks for the reply.

The problem I have is *how* to call the shutdown method in our environment? We are using ColdFusion 2018, which is built on top of Java. We can access Java methods/objects, but I do not know how to access the Executor and call it, and when?

To start with I created a Java Object like this:

<cfset xs = createobject("java", "java.util.concurrent.ExecutorService")>

when I "dump" that object, I see all of its methods:

However, if I try to run shutdown() at that point I get the error:

"The shutdown method was not found. Either there are no methods with the specified method name and argument types or the shutdown method is overloaded with argument types that ColdFusion cannot decipher reliably. ColdFusion found 0 methods that match the provided arguments. If this is a Java object and you verified that the method exists, use the javacast function to reduce ambiguity."

Stuck on that.

Btw you also said " You may consider creating 1 thread pool with multiple threads in it."

How do we do that? In configuration?

Hi Ram. Thanks for your insights. It seems we cannot just "inject" a call to the shutdown() call. This has to be done at the "Engine" level of ColdFusion. Do you know how can we see the thread pools using a tool like jConsole using JMX/Mbeans? Is there a way do you know?

Hi Ram, we don't create the thread(s) using our own code, we are just using Tomcat. This is an internal process.