Profile Image
kushagra

My threads are increasing over time. Im using JAVA quartz scheduler.

My threads are increasing every time a job is triggered in a Quartz scheduler. There are 4 jobs that run every hour. After every job there is an increase in the number of threads

 

Report URL - https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjIvMDUvMzAvLS10aHJlYWRkdW1wLm91dC0tMTgtMjMtNDE=

  • quartz-scheduler

  • threads-increase

  • executor-memory-leak

  • executor

  • executor-not-closed

  • threadsincreasing

  • javaquartzscheduler

  • notclosingexecutor

  • nettylibrary

  • cluster-n-worker-thread

  • threadshavingsamestacktrace

Please Sign In or to post your comment or answer

Profile Image

Charlie Arehart

Are you sure there's such a direct connection between the quartz jobs and the increasing threads? In your thread dump/report, the high count of threads is nearly all about "cluster" threads that are related to "netty", not quartz. Do you recognize your use of either netty or its clustering? Do you believe there's a connection between those and quartz? Are the URLs called from the quartz jobs ones that run against this same jvm?

 

And when you say the threads are rising per the running of quartz jobs, which thread type is increasing? We can't tell without seeing two such thread dumps/reports in a row (with an indication of the time between between them).

 

FWIW, quartz does indeed have a threadpool (which defaults to not being limited), but again that doesn't seem to be what's rising here. If you're interested in more on that, see the

org.quartz.threadPool.threadCount property in your quartz.properties file (the default name for that file, which you should find alongside the quartz jar used in your app). For more on all that, see the docs: 

http://www.quartz-scheduler.org/documentation/quartz-2.3.0/configuration/index.html

 

Of course, someone else may see things differently than I do. I'm offering this answer if it might help.

Profile Image

kushagra

Thanks Charlie for an early response. I will try to upload to successive thread dumps . I will also look into the quartz document and come back to you. Once again Thanks !

Profile Image

Ram Lakshmanan

Hello Kushagra!

 

 Greetings. I reviewed your thread dump analysis report. Here are my observations:

 

a. Not closing executor (i.e. thread pool):

 My strong suspect is your application isn't closing the thread pool (i.e. Executor) after using it. Something similar to this problem. #b and #c listed below are my reasons for this suspicision. 

 

b. 85% of threads having same stacktrace

Your application has 2918 threads, out of which 2472 threads have identical stacktrace.

 

Here is that stacktrace:

 

java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait(Native Method)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:194)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:219)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None

 

If you notice these threads are orignating from netty library. Besides that these threads aren't doing nothing. This doesn't sound right. Do you know which part of your application is using this netty library? May be you need to take a close look at the application code which is spawning these many threads.

 

c. Thread pool by name 'cluster-n-worker-thread'

 If you notice your application has created 130 thread pools by name: 

 

cluster1-nio-worker
cluster2-nio-worker
cluster3-nio-worker
cluster4-nio-worker
cluster5-nio-worker
:
:
cluster130-nio-worker

 

 Each thread pool has exact 22 threads in each of them. All most of all the threads in this pool has the stacktrace pointed in #b.

 

 

Profile Image

kushagra

Thanks Ram for an early response. I will definitely look into it and try to take successive thread dumps to clear the picture more. Once again Thankyou !

Got something else on mind? Post Your Question

Not the answer you're looking for? Browse other questions tagged
  • quartz-scheduler

  • threads-increase

  • executor-memory-leak

  • executor

  • executor-not-closed

  • threadsincreasing

  • javaquartzscheduler

  • notclosingexecutor

  • nettylibrary

  • cluster-n-worker-thread

  • threadshavingsamestacktrace