JDK8 G1GC Heap > 32GB < 64GB

Hello Stephen - my friend!

 Good to hear from you. Interesting question(s).

 Her are my thoughts around it.

 1. Here is a GC log analysis report of a JVM which has ~117GB heap size. This JVM's GC throughput is 93.473%, however avg GC pause time is 195ms and Max pause time is 660ms. It's using G1 GC algorithm. GC pause times are within your expectation. However definetly there is room for optimization in this example.

 2. Here is a post that published sometime back, where I share my thoughts/experiences around large heap size vs small heap size JVMs.

 3. I am working on new post, which summarizes the GC optimization we did for an organization. This organization was having 30GB heap. Young Gen was set at 15GB. It was running with CMS algorithm. When we reduced the young generation size to 1GB, it resulted in significant GC performance improvement. Let me share that story with you through email for now. However it was CMS algorithgm and your's is G1 GC (which has more levers). 

 Good luck and keep us posted about your progress. Looking forward to see the outcome. Thanks. 

Hi Ram,

Noted, thank you for the response. I'd be happy to share my experiment.

32GB JVM 40 hour lifetime

https://gceasy.io/my-gc-report.jsp?p=YXJjaGl2ZWQvMjAyMi8wMi8yMy8tLTMyR0JHMUdDLS0xMS01Ny0yMA==&channel=API

48GB JVM 40 hour lifetime

https://gceasy.io/my-gc-report.jsp?p=YXJjaGl2ZWQvMjAyMi8wMi8yMy8tLTQ4R0JHMUdDLS0xMS01Ny00&channel=API

I have seen positive results so far and I'd like to share it. Kindly check the GC times, graphs, and the micrometrics when comparing both.

1. FGC: None

2. YGC Count: 32GB: 392, 48GB 298

As expected, reduction in YGC for the 48GB heap. Mixed collections were around the same.

3. Avg GC Pause Time: 32GB:208ms, 48GB: 303ms

Only increase in 93ms for the 48GB heap

4. Created bytes: 32GB: 4.88 tb, 48GB: 6.69 tb

Created bytes increased by 37%. This means the additional 16GB allocated is not wasted, our application is fully utilizing all the space. Checking the young gen peak it also shows that young gen is hitting its maximum allocation of 60%.

5. Total Pause Time: 32GB: 2 min 4 sec 590 ms, 48GB: 2 min 18 sec 370 ms.

Total pause time only 14s more for the 48GB heap, acceptable for our application.

6. Throughput: 32GB 99.914, 48GB 99.905

Reduction by 0.009, acceptable for our application

7. Longest Pause Time: Hard to tell from the graph as the longest GC in both instances were during the JVM startup. But all subsequent YGC were under 600s as the Avg GC Pause Time would tell above.

8. Avg creation rate: 32GB 35.16 mb/sec, 48GB 48.39 mb/sec

Increase by 37% for the 48GB heap. This is very good as we can now process 37% more transactions.

9. GC Causes

32GB:

- G1 Evacuation Pause: 438

- G1 Humongous Allocation: 47

- GCLocker Initiated GC: 10

48GB:

- G1 Evacuation Pause: 336

- G1 Humongous Allocation: 33 

- GCLocker Initiated GC: 13

All areas are reduced with the 48GB heap. Our application is creating large objects so the G1RegionSize was set to 32MB for the 48GB heap, giving total of 1536 regions. Due to the bigger region size we saw less Humongous Allocations

As expected, larger heap would allow for more room in young gen and S0/S1, reducing the evacuation pause events.

The aim of my study is the opposite of your study. My aim is to eliminate all allocation/evacuation failures by giving eden/s0/s1 enough space, allowing objects to live its natural life and die in young generation without being promoted to old gen. This approach saves old generation for to-space exhaustion issues when young generation is completely filled, old generation will have lots of space to accommodate. It also shows that we are able to process a lot more customer transactions (37% increase) while only adding 97ms of average GC time. We are able to significantly reduce the chances of FGCs because old gen utilization is low since we now have a very low promotion rate.

I will continue to share my findings as we progress.

I took a look at the 117GB heap report, let me share my thoughts.

In my opinion it is very inefficient. Young Gen is capped at around 4GB, leaving 113GB for old gen. The heap is poorly utilized. Given the insane object creation rate of 1.03gb/s, this means young gen will fill in 4 seconds leading to a lot of promotions to old gen which is evident from the graphs. High promotion rate is the opposite of what we want as collections in old gen (mixed) are longer that young gen. Too many mixed gc and ygc would take place in a short period of time. While they may be short because of small young gen, the frequency will be high due to the creation rate leading to overall high total gc pause times effectively reducing throughput. Maybe this approach is one that is adopted but my study goes in the opposite direction.

Regards,

Stephen  

Hello Stephen!

 1. Highly encouraging results. Great job my friend. Despite increasing heap size, there is no noticeable difference in the GC throughput and avg. GC pause time. Also long GC pause difference can be ignored because it doesn't happen that frequently. 

 2. Since there is a 37% increase in the object creation rate, I strongly suspect two things:

a. Overall application's transaction throughput should improved.

b. There is also good chance for overall application transactions response time to improve. 

 Can you please confirm whether it's the case? If so this heap size increase should be totally justified. It should be easy to  convince your management and fellow team members on this change.

 3. I also noticed these two changes made in your JVM arguments (which might be fine):

32GB: -XX:G1MaxNewSizePercent=50
48GB: Removed

32GB: -XX:MaxGCPauseMillis=300
48GB: -XX:MaxGCPauseMillis=500

 4. I don't see 'metaspace' graph to be reported. It indicates GC never ran in metaspace. You may also consider lowering your metaspace size from 4GB to 1GB. Because 4GB mioght be lot for most applications.

Hi Ram,

1. Thank you for confirming my findings! Noted on the GC pauses for the very few long ones.

2. I cannot confirm if more customer transactions are getting processed because I do not have access to that data, but looking at the 37% increase all I can do is make this assumption. I will check with my application team next week. Currently the tuning only applied to 1 JVM. Next week I will roll out to all of our JVMs, so then we have a better comparison. I will compare 1 week before tuning and 1 week after tuning.

3. Yes these changes were intentional.

32GB: -XX:G1MaxNewSizePercent=50
48GB: Removed. 

I had to cap it at 50% for the 32GB heap because I was running into a danger zone where old gen was filling too fast cause allocation and evacuation failures. Capping at 50% gave old gen a space of 50%. 

Since I increased the heap to 48GB, I allowed young gen to grow to the maximum default limit of 60% since the overall young gen size is much bigger: 29GB vs 17GB, we should see less usage of old gen which is exactly what we see in the graphs (compare peak old gen usage)

32GB: -XX:MaxGCPauseMillis=300
48GB: -XX:MaxGCPauseMillis=500

This was also intentional, I did do an iteration on the 48GB heap with 300ms. There is an important truth here, XX:MaxGCPauseMillis is a soft goal but it determines a lot of the JVM decisions. One important one is the size of young gen. The JVM will initiate a YGC earlier on in its young gen growth as it calculates the average time it would take, and compares it with the XX:MaxGCPauseMillis. It would try to meet this goal, effectively limiting the size of young gen in order to meet the XX:MaxGCPauseMillis goal. When I set to 300 or 200 for the 48GB, young gen was being limited severly, similar to how the 117GB usage looked. It doesnt mean that my GC average would be 500ms, as we can see it is barely 300ms. But it does have a big influence on the YGC decisions which can really limit the YG size. Hope I was able to explain this properly.

4. Noted on the metaspace, I also made this observation. I am planning to reduce to 1GB this weekend. Once startup of JVM is successful, I think should be okay. My servers only got 64gb so running a 48GB JVM is a bit risky when we consider meta + out of heap usage. Reducing meta will give me a bit more head room.

Regards,

Stephen 

Hi Team,

Updating my findings for 3 day run of the JVMs.

32GB

https://gceasy.io/my-gc-report.jsp?p=YXJjaGl2ZWQvMjAyMi8wMi8yNS8tLTMyR0JHMUdDLS0xLTktMjQ=&channel=API

48GB

https://gceasy.io/my-gc-report.jsp?p=YXJjaGl2ZWQvMjAyMi8wMi8yNS8tLTQ4R0JHMUdDLS0xLTktMjg=&channel=API

My next study is to perform a test with 3 options: 16GB, 32GB, 48GB. I will share my findings in a few days.

Looking forward to your thoughts on the best option of the 3!

Regards,

Stephen