Hi arcow,
Below are my observations and recommendations:
a) Safepoint Duration: 2 times application threads were stopped for more than 5 seconds. you can find more detail on this article.
b) G1 Evacuation Pause event : Due to 'G1 Evacuation Pause' your application is pausing for 33+ minutes. This very high. Our ML algorithm, provides following suggestion to improve the 'G1 Evacuation pause' time:
Solution:
1. Evacuation failure might happen because of over tuning. So eliminate all the memory related properties and keep only min and max heap and a realistic pause time goal (i.e. Use only -Xms, -Xmx and a pause time goal -XX:MaxGCPauseMillis). Remove any additional heap sizing such as -Xmn, -XX:NewSize, -XX:MaxNewSize, -XX:SurvivorRatio, etc.
2. If the problem still persists then increase JVM heap size (i.e. -Xmx).
3. If you can't increase the heap size and if you notice that the marking cycle is not starting early enough to reclaim the old generation then reduce -XX:InitiatingHeapOccupancyPercent. The default value is 45%. Reducing the value will start the marking cycle earlier. On the other hand, if the marking cycle is starting early and not reclaiming, increase the -XX:InitiatingHeapOccupancyPercent threshold above the default value.
4. You can also increase the value of the '-XX:ConcGCThreads' argument to increase the number of parallel marking threads. Increasing the concurrent marking threads will make garbage collection run fast.
5. Increase the value of the '-XX:G1ReservePercent' argument. Default value is 10%. It means the G1 garbage collector will try to keep 10% of memory free always. When you try to increase this value, GC will be triggered earlier, preventing the Evacuation pauses. Note: G1 GC caps this value at 50%.
c) G1 Humongous Allocation event : Due to 'G1 Humongous Allocation' your application is pausing for 3 secs. Frequent humongous allocations can cause couple of performance issues:
1. If the regions contain humongous objects, space between the last humongous object in the region and the end of the region will be unused. If there are multiple such humongous objects, this unused space can cause the heap to become fragmented.
2. Until Java 1.8u40 reclamation of humongous regions were only done during full GC events. Where as in the newer JVMs, clearing humongous objects are done in cleanup phase.
Our ML algorithm, provides following suggestion to improve the 'G1 Humongous Allocation':
Solution:
You can increase the G1 region size so that allocations would not exceed 50% limit. By default region size is calculated during startup based on the heap size. It can be overriden by specifying '-XX:G1HeapRegionSize' property. Region size must be between 1 and 32 megabytes and has to be a power of two. Note: Increasing region size is sensitive change as it will reduce the number of regions. So before increasing new region size, do thorough testing.
d) As you are using G1 GC algorithm you may consider passing -XX:+UseStringDeduplication to your application. It will remove duplicate strings in your application and has potential to improve overall application's performance. You can learn more about this property in this article.
e) It looks like your application's Initial heap size and Max heap size is set to different values. Setting initial and Max heap size to same value has potential to improve your application's availability, performance and startup time. See details here.
f) -XX:+UseGCLogFileRotation argument is not recommended to be passed. It can have undesirable effects. To learn more about it's effects and alternative solution refer to refer here.
Edit your Comment