Profile Image
lokesh khandelwal

why does GC happens way before 70% occupancy is reached

We have allocated 248GB of RAM, and -XX:InitiatingHeapOccupancyPercent=70, 

 

Below is my JVM options set.

 

set "JAVA_OPTS=-Xms248g -Xmx248g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:NewRatio=2 -XX:InitiatingHeapOccupancyPercent=70 -Xlog:gc*:verbose_gc.log:time"



Report URL - https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDYvMjIvLS12ZXJib3NlX2djLmxvZy0tNi00Ny0zNw==&channel=WEB

 

Pls guide me how do i improve throughput.

  • gcthroughput

  • jvmoptions

  • initiatingheapoccupancypercent

Please Sign In or to post your comment or answer

Profile Image

Mahesh

Hi Lokesh,

 

Here are my observations:

  1. If you see the "Heap before GC" and "Interactive Graphs" section, you will see full GC was triggered when your heap size reached around 253 mb.



  2. In order to increase the throughput, you will need to reduce the GC pause time. Because if your application is suffering from the long GC pause then it will also degrade your throughput.

    Currently, your application is spending 44% of its time on GC activities and around 165 GC events took more than 5.0 seconds.

    Your application is creating 422.95 MB of objects per second. This could be one of the reasons your application is having long GC pauses.

    If your application's object creation rate is very high, then to keep with it, the garbage collection rate will also be very high. A high garbage collection rate will increase the GC pause time as well. Thus, optimizing the application to create fewer objects is THE EFFECTIVE strategy to reduce long GC pauses. This might be a time-consuming exercise, but it is 100% worth doing. In order to optimize object creation rate in the application, you can consider using java profilers like JProfiler , YourKit , JVisualVM...). These profilers will report:

    + What are the objects that created?
    + What is the rate at which these objects are created?
    + What is the amount of space they are occupying in memory?
    + Who is creating them?

Profile Image

lokesh khandelwal

Thanks Mahesh for responding, i was actually confused with interactive graph measure, it is using heap size in mb( is it bits or bytes), if it is bytes then i think it is triggering correctly.

for the Object creation part, actually we are doing batch processing which involves processing of millions of transactions, we do not want to scale out hence we are testing for optimum hardware and configuration for scale up type of scenario.

The objects created is high because we have approx 260 threads running in parallel each processing thousands of records. Mycurrent bottle neck is GC which as pointed by you is taking 40+% of time, need suggestion for this as my object creation is high(they are part of calculation and are optimised in multiple rounds) still they are short lived, how best i can configure to have maximum throughput. 

 

Profile Image

Ankita

Hello Lokesh!

 

Your GC throughput is suffering a lot. Here are my initial thoughts:
 

a. You have high number of threads, which is creating high number of objects. Can you lower your thread count?
 
b. Your heap size is quite high (248GB). High heaps tend to consume high pause time
 
c. You can tune your G1 GC settings further. Your arguments doesn't sound right.
 
d. We have seen wonderful GC pauses time with Z GC (which is available from JDK 11) for such large heap size.
 

If it's of interest, I can setup a free 30 minute consultation with our architect (who developed GCeasy). He is based in Pacific Time zone. Thanks.

Profile Image

Mahesh

Yes, it is in bytes. It would be difficult to pinpoint the exact root cause without understanding the application. I would like to recommend you to take a free consultation with the GCeasy architect as @Ankita mentioned in the above comment. You can also reach out at team@tier1app.com.

Profile Image

lokesh khandelwal

Thanks Mahesh and Ankita, i have further tuned GC properties and i am able to achieve throughput of 90% with higher threads and higher volume, you can see the report in below location

https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDYvMjQvLS12ZXJib3NlX2djLmxvZy0tNS0zMC0yNA==&channel=WEB

 

the catch was i need to keep more part for RAM for OS, so i left 1/4 of RAM for OS and found good result. below is my JVM config for 256 GB ram VM.

set "JAVA_OPTS=-Xms200g -Xmx200g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:NewRatio=1  -XX:InitiatingHeapOccupancyPercent=70 -Xlog:gc*:verbose_gc.log:time"

 

Pls suggest if there is further scope of improvement on this.

 

I am now planing to test similar volume with ZGC.

 

Profile Image

Ram Lakshmanan

Hello Lokesh!

 

 That is impressive. Earlier it looks like your BATCH application was running for ~4 hours. Now it was able to complete with in 1 hour. Sounds like a significant improvement. Is it because today's workload on the batch application low? OR you increased the thread count significantly? If so by how much?

 

 Here are some suggestions:

 

1. You might consider removing -XX:NewRatio=1. This property will set the Young Gen size explicitly. In G1 GC algorithm, it's advisable not to set any of the generation size.

 

 2. You might consider removing -XX:MaxGCPauseMillis=200, since this is a batch application, you don't want to set aggressive GC pause time goal like 200ms. This will make Garbage collector run more aggresively. This could be one of the reason for degradation in GC Throughput percentage. Besides that by setting the Max GC pause time to 200ms, Garbage collector isn't giving you that pause time.

 

Experimenting with Z GC is a great idea. Pursue it. 

Load More Comments

Got something else on mind? Post Your Question

Not the answer you're looking for? Browse other questions tagged

  • gcthroughput

  • jvmoptions

  • initiatingheapoccupancypercent

Edit your question

Separate with comma, space or ENTER key. Special characters are not allowed in tag.

Edit your Answer

Edit your Comment