I see the Report URL did not paste, adding it here once again
Blocked thread vs waiting thread
Hello the JVM Heap dump that i analysed i see 70% of threads of the application are either timed_waiting or blocked, what is causing this . Do i need my developer to fine tune the code to enable this ? THe thread dump is from an application that is deployed across 6 node weblogic cluster on 12.2.x
Report URL - Sorry, unable to generate URL
View(s)
11- 0Like(s)
- 0Dislike(s)
- 0Saved
0
Share(s)
Facebook
Twitter
Email
Copy Link
Ram Lakshmanan
Hello Shivraj!
Greetings.
In your application there are 67 threads in BLOCKED state. Here are few things that I want to higlight:
a. HystrixTimer thread pool - threads BLOCKED
There are 20 threads in 'HystrixTimer' thread pool. All of them are in BLOCKED state. Basically they are waiting for new tasks submitted to this thread pool for execution. I will not be worried about it. Below is their stacktrace:
stackTrace: -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x1fe972cb0 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java/util/concurrent/DelayQueue.take(DelayQueue.java:160) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:967) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:927) at java/lang/Thread.run(Thread.java:682) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace
b. Hystrix thread pool - threads BLOCKED
You also have 'Hystrix' thread pool which has 20 threads in it. All of those threads are also in BLOCKED state. They are waiting for new jobs to submitted to begin their execution. I will not be worried about it. Below is their stacktrace:
stackTrace: -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x226e44ac8 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:967) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:927) at java/lang/Thread.run(Thread.java:682) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace
However one concern in above two thread pools is: why threads aren't dieing down when there is no work. May be you need to configure min thread pool count to a lower value for this 'Hystrix' and 'HystrixTimer' thread pool. Threads aren't cheap resource and they should be handled with care.
Other than that, I don't see any major problems in the thread pool. Are you facing any issues in your application?
Shivraj M
@Ram Appreciate your analysis & help here. So the issue started since we migrated from Weblogic 12.1.x to 12.2.x & Jrockit 1.6 to Jrockit 1.8, since then a lot of performance issues and having to have the managed instances restart once certain thread count is reached to avoid downtime for now. So wondering what could be causing, Developer confirms they compiled their code to work with Jrockit 1.8 but the issue persists. We need to get to root cause to be able to determine the impact and start fixing this recurring thread pool filling up and not allowing to process transactions until released by a restart. Does this report help in any other way to determine and help fix ? Can i look at resetting the time-out values for long running idle threads ? does it look like the threads are not optimized?
Ram Lakshmanan
Hello Shivraj!
>Can i look at resetting the time-out values for long running idle threads ? does it look like the threads are not optimized?
We don't have sufficient data to come to this conclusion yet.
>Does this report help in any other way to determine and help fix ?
Not really. There could be several reasons why your application is have performance hiccups. Some of the reasons are:
- Garbage collection pauses
- Threads getting BLOCKED
- Network connectivity
- Load balancer routing issue
- Heavy CPU consumption of threads
- Operating System running with old patches
- Memory Leak
- DB not responding properly
:
:
So just thread dump is not enough to diagnose the problem. You have captured only thread dump, that too one snapshot of it. It's always a good practice to capture 3 thread dumps in a gap of 10 seconds between each one. Besides thread dumps you might have to capture other logs/artifacts to do thorough analysis.
You can use the open source yCrash script which will capture 360-degree application level artifacts (like GC logs, 3 snapshots of thread dumps, heap dumps) and system level artifacts (like top, top -H, netstat, vmstat, iostat, dmesg, diskusage, kernel parameters...). Once you have these data, either you can manually analyze them or upload it to yCrash tool, which will analyze all these artifacts and generate root cause analysis report. It has potential to indicate the root cause of the problem.
Also once you have captured the data, you are welcome to schedule a free 30 minute consultation!
Edit your Comment