G1GC performance tuning

While you await a reply from the ycrash folks, I'll offer some thoughts based on experience, not reviewing anything more than what you offered in your note.

I don't think fiddling with jvm knobs is the answer. Gc's are a reflection of heap being used. Usually it's a simple matter of picking the right heap size for your app--especially for a job, doing essentially "one thing".

Before considering RAISING the Heap, there's always the possibility that it's spending time doing Gc's simply because you made the Heap much higher than needed and the jvm lazily used it.  So first, have you run it with far less heap and had oom heap errors? 

Second, you haven't said how long this job runs, or what jvm version it is. Both can influence how to analyze things. (Sorry if those are indicated somewhere that I missed them. On a phone, once we're answering, we can't easily see the original note. 

Third, if it does need more heap, it may run better with even just 1gb more (seriously), or it may need 5g more, or 10 or 20. No none can say, only testing will determine what the app needs.

Finally, another way to go about resolving things is to lower the Heap requirements of the app. That can be harder, of course. (For some, doubling the Heap and running it may work and be trivial. Others may not have that available memory. Or doubling may not be enough.)

In that case, you'd need to find out WHAT in the app is a) using so much heap, and b) holding on to it so that it can't be gc'ed. Again, that is harder and also needs to be assessed at points within the processing of the app, as things could change dramatically over time during processing.

Hello BhagyaShri!

 In the metrics you have shared, I could see your application's GC throughput is reported as 59.6%. This is a very poor throughput. It indicates your application is spending 59.6% in processing customer transactions and remaining 40.4% of time in doing Garbage Collection. It means in 1 day, your application is spending close to 10 hours in garbage collection. 

 This kind of poor throughput can happen only under two circumstances:

a. Your application is suffering from memory leak

b. Your application is suffering from consecutive Full GC problem

 I recommend you to read 'Pattern #4 Consecutive Full GC pattern' and 'Pattern #5 Memory Leak Pattern' in this blog post. For these two situations, just adjusting JVM arguments that you have mentioned will not work. Then what is the solution:

a. Solution to Memory Leak:

You can use root cause analysis tools like yCrash - which automatically captures application-level data (thread dump, heap dump, Garbage Collection log) and system-level data (netstat, vmstat, iostat, top, top -H, dmesg, kernel parameters…). Besides capturing the data automatically, it marries these two datasets and generates an instant root cause analysis report. Below is the report generated by the yCrash tool when the above sample program is executed:

Fig: yCrash tool pointing out the root cause of OutOfMemoryError

You can notice the yCrash tool precisely pointing out the root cause of memory leak. You can register here to get a 14 day trial of yCrash.

b. Addressing Consecutive Full GCs

 Full GCs Consecutively run because of lack of memory. May be your application's traffic volume has grown and your application isn't able to cope up with currently allocated memory. This can be addressed by increasing -Xmx. For more details refer here