Hi Ankit,
If you see the Heap Before GC graph, you will see the memory reached its limit (red triangle) and dropped down to 15MB when the full GC happened everytime.
In the report, I could see your application's throughput value is 72.098% and Max Pause GC Time is 3 min 37 sec. It shows that your application is spending more time on GC activities instead of spending time on customer transactions.
To Reduce the long GC pauses, you can consider the following solution:
1) In the Object Stats section, you will see one property called Avg. creation rate. This property will tell us how many objects our application is creating per second. In your report, the application is creating objects 924.28 MB/sec.
If the application's object creation rate is high then the garbage collection rate will also be high and it will increase GC pause time as well. You may consider optimizing your application code in order to reduce the pause time. You can use HeapHero, JProfiler, YourKit, JVisualVM tools to optimize the object creation rates.
2) You can set maximum pause time by using the -XX:MaxGCPauseMillis G1 GC argument. The default value is 200ms.
3. You can set a number of threads used during the parallel phase of garbage collection. This can be by using -XX:ParallelGCThreads argument.
4) You can use -XX:ConcGCThreads argument to set a number of threads concurrent garbage collector will use.
5) If multiple processes are running on the server where your application is running then, there is a possibility that your application might be crunched for CPU & I/O cycles. Under such circumstances also application's throughput will degrade.
I could also see a couple of reasons behind triggering GC. Here are some of the reasons :
1) G1 Evaluation Pause - When this happens G1 GC doesn't have enough memory and it can't expand as it is already on its max. 18 min 38 sec 28 ms of GC pause time was triggered by this event.
Solutions -
- Increase the value of the -XX:G1ReservePercent argument. The default value is 10%. It means the G1 GC will try to keep 10% of memory free always. When you try to increase this value, GC will be triggered earlier, preventing the Evacuation pauses.
2) Full GC - Allocation Failure - 17 min 35 sec 90 ms of GC pause is triggered by this event. This could happen because either heap fragmentation or application is creating too many objects that can't be reclaimed quickly.
Solution -
- Even there is enough space in the heap, full GC can occur due to the lack of contiguous space. This can happen because of a lot of humongous objects present in the memory. A potential solution to solve this problem is to increase the heap region size by using the option -XX:G1HeapRegionSize to decrease the amount of memory wasted by humongous objects.
3) Evacuation Failure - This is similar to the 1st point. 14 sec 170 ms of GC pause time was triggered by this event. This could happen because there are not free regions to promote to the old generation because the heap cannot expand more since it is reached a limit.
Solution -
- You can eliminate all heap-related arguments. Evacuation failure might happen because of over-tuning. you can keep only -Xms, -Xmx, and -XX:MaxGCPauseMillis. Remove any additional heap sizing such as -Xmn, -XX:NewSize, -XX:MaxNewSize, -XX:SurvivorRatio, etc.
4) G1 Humongous Allocation - 1 sec 300 ms of GC pause time is triggered by this event. Any object that is more than half a region size is considered a “Humongous object”. If the regions contain humongous objects, space between the last humongous object in the region and the end of the region will be unused. If there are multiple such humongous objects, this unused space can cause the heap to become fragmented.
Solution -
- You can use the -XX:G1HeapRegionSize argument as I mentioned in the 2nd point.
5) Metadata GC Threshold - 140 ms of GC pause time is triggered by the Metadata GC Threshold event. This type of GC event is triggered either because configured metaspace size is too small than the actual requirement or there is a classloader leak (very unlikely, but possible).
Solution -
- You may consider setting -XX:MetaspaceSize to a higher value. If this property is not present already please configure it.
6) If you are running on Java 8 update 20 and above, you may consider passing -XX:+UseStringDeduplication to your application. It will remove duplicate strings in your application and has the potential to improve the overall application's performance.
Edit your Comment