Hello Anil,
In your GC report, the throughput is very very low. The application is required to be profiled.
You can refer this link for more information.
https://answers.ycrash.io/question/please-let-us-know-your-opinion?q=1201
continious gc threads using 99% cpu
Hello,
we are observing continious GC Thread using 90% cpu
25632 root 20 0 61.0g 43.5g 225432 R 99.9 69.3 34:25.64 GC Thread#0
25660 root 20 0 61.0g 43.5g 225432 R 99.9 69.3 34:25.27 GC Thread#5
25662 root 20 0 61.0g 43.5g 225432 R 99.9 69.3 34:24.02 GC Thread#7
25658 root 20 0 61.0g 43.5g 225432 R 99.7 69.3 34:23.89 GC Thread#3
25661 root 20 0 61.0g 43.5g 225432 R 99.7 69.3 34:25.05 GC Thread#6
25656 root 20 0 61.0g 43.5g 225432 R 99.3 69.3 34:24.30 GC Thread#1
25657 root 20 0 61.0g 43.5g 225432 R 99.3 69.3 34:24.48 GC Thread#2
25659 root 20 0 61.0g 43.5g 225432 R 99.3 69.3 34:24.06 GC Thread#4
25630 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.00 java
25631 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:21.76 azl-pegamdqa05
25633 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.11 G1 Main Marker
25634 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 1:37.67 G1 Conc#0
25635 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:08.73 G1 Refine#0
25636 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.26 G1 Young RemSet
25637 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:29.29 VM Thread
25638 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.03 Reference Handl
25639 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.06 Finalizer
25640 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.00 Signal Dispatch
25641 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.43 Service Thread
25642 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 2:42.32 C2 CompilerThre
25643 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:27.56 C1 CompilerThre
25644 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:01.98 Sweeper thread
25645 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.00 Common-Cleaner
25648 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.03 FileHandlerLogF
25649 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.00 idle-timeout-ta
25650 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.03 req-rsp-timeout
25652 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.03 HTTP-Dispatcher
25653 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:02.39 VM Periodic Tas
25654 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.01 AsyncFileHandle
25663 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 1:37.66 G1 Conc#1
25664 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.05 Tomcat JDBC Poo
25701 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:01.77 Log4j2-TF-7-Asy
25732 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.00 retry-executor-
25734 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.02 Coarse-Ticker
25751 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:05.38 G1 Refine#1
25757 root 20 0 61.0g 43.5g 225432 S 0.0 69.3 0:00.17 Service-registr
Report URL - https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjQvMDkvMTEvZ2MubG9nLS00LTIyLTUx&channel=WEB
View(s)
20- 0Like(s)
- 0Dislike(s)
- 0Saved
0
Share(s)
Facebook
Twitter
Email
Copy Link
Ram Lakshmanan
Hello Anil!
If your GC threads are continuously at 99%, it's a clear indication that your application has reached it's memory limits and GC threads are unable to reclaim it. You should capture GC log and study it's behaviour. You will observe either one of the patterns in the GC log analysis:
1. Consecutive Full GCs due to spike in traffic volume
2. Memory leak in your application
Let's discuss them:
1. Consecutive Full GCs due to spike in traffic volume
When the application’s traffic volume increases more than JVM can handle, this Consecutive full GC pattern will become pervasive. In the above figure, please refer to the black arrow mark in the graph. From 12:02pm to 12:30 pm on Oct’ 06, Full GCs (i.e., ‘red triangle’) are consecutively running; however, heap usage isn’t dropping during that time frame. It indicates that traffic volume spiked up in the application during that time frame, thus the application started to generate more objects, and Garbage Collection couldn’t keep up with the object creation rate. Thus, GC events started to run consecutively. Please note that when a GC event runs, it has two side effects:
a. CPU consumption will go high (as GC does an enormous amount of computation).
b. Entire application will be paused; no customers will get response.
Thus, during this time frame, 12:02pm to 12:30pm on Oct’ 06, since GC events are consecutively running, application’s CPU consumption would have been skyrocketing and customers wouldn’t be getting back any response. When faced with this pattern, you can utilize the solutions provided in this blog to address GC pattern issues effectively.
Here is the real-world GC log analysis report, which depicts this ‘Consecutive Full GC’ pattern.
2. Memory Leak in your application
This is a ‘classic pattern’ that you will see whenever the application suffers from memory problems. In the above figure, please observe the black arrow mark in the graph. You can notice that Full GC (i.e., ‘red triangle’) events are continuously running. This pattern is similar to the previous ‘Consecutive Full GC’ pattern, with one sharp difference. In the ‘Consecutive Full GC’ pattern, application would recover from repeated Full GC runs and return back to normal functioning state, once traffic volume dies down. However, if the application runs into a memory leak, it wouldn’t recover, even if traffic dies. The only way to recover the application is to restart the application. If the application is in this state, you can use tools like yCrash, HeapHero, Eclipse MAT to diagnose memory leak. Here is a more detailed post on how to diagnose Memory leak.
Here is the real-world GC log analysis report, which depicts this ‘Memory Leak’ pattern.
Edit your Comment