Profile Image
Christian Nüssgens

Stop-The-World Class Unloading (several sceonds)

Hi there,

 

we have a strange Problem with one of our webapps (which was not observed in other webapps / tomcat installations)

We are running a webapp in a Tomcat-Servlet container (Java 1.8.0_302 OpenJDK 1.8.0_302-8u302-b08-1~deb9u1-b08, Apache Tomcat/8.5.54 Debian)

 

From time to time we have really long gc/remark stop-the-world pauses (as you can see form the shared report). In this case 14 seconds.

 

We are using the G1GC and did already enable -XX:+TraceClassUnloading for troubleshooting purposes. But we still do not have a clue why this is happening. The classes being unloaded are mostly related to reflection and Lambdas (here is an short representative excerpt):

catalina.out.2.gz:[Unloading class sun.reflect.GeneratedConstructorAccessor7 0x00000001001e6428]
catalina.out.2.gz:[Unloading class sun.reflect.GeneratedConstructorAccessor6 0x00000001001a4028]
catalina.out.2.gz:[Unloading class sun.reflect.GeneratedConstructorAccessor5 0x00000001001a3428]
catalina.out.2.gz:[Unloading class sun.reflect.GeneratedConstructorAccessor4 0x00000001001a2428]
catalina.out.2.gz:[Unloading class sun.reflect.GeneratedConstructorAccessor3 0x00000001001a1828]
catalina.out.2.gz:[Unloading class sun.reflect.GeneratedConstructorAccessor2 0x000000010015ec28]
catalina.out.2.gz:[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor5 0x000000010015e028]
catalina.out.2.gz:[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor4 0x000000010015dc28]
catalina.out.2.gz:[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor3 0x000000010015d828]
catalina.out.2.gz:[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor2 0x000000010015d428]
catalina.out.2.gz:[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1 0x000000010015d028]
catalina.out.2.gz:[Unloading class sun.reflect.GeneratedConstructorAccessor1 0x00000001000e9028]
catalina.out.2.gz:[Unloading class java.lang.invoke.LambdaForm$BMH/1109371569 0x00000001000cd028]
catalina.out.2.gz:[Unloading class java.lang.invoke.LambdaForm$BMH/670700378 0x00000001000ccc28]
catalina.out.2.gz:[Unloading class java.lang.invoke.LambdaForm$BMH/812265671 0x00000001000cc428]
catalina.out.2.gz:[Unloading class java.lang.invoke.LambdaForm$BMH/1650967483 0x00000001000c2028]
catalina.out.2.gz:[Unloading class java.lang.invoke.LambdaForm$BMH/2128227771 0x00000001000ba028]

But in this specific webapp - were we experiencing this issue - we are not doing more reflective operations than in other webapps too (were we are not experiencing such issues)

 

While googleing the issue I stumpled over the VM argument -Xnoclassgc but I'm not sure if it's safe to activate such flag (or run into other problems such as OOM)

 

Do you have any idea or advice?

Is it safe - or at least not a too bad idea - to enable -Xnoclassgc?

Or can we enable another Parameter (/set another VM-option) to investigate this issue further?

 

Kind regards

Christian

 

Report URL - https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMTIvMjEvLS10b21jYXQtc2VydmVyLWdjMjAyMS0xMi0xNV8xNC0yNC0zNS5sb2cuZ3otLTktMjMtMTY=&channel=WEB

  • probleminwebapps

  • tomcatinstallations

  • vmargument

  • xnoclassgc

Please Sign In or to post your comment or answer

Profile Image

Mahesh

Hi Christian,

 

Here is my observation - 

 

1) As per this article, the remark phase also does reference processing. If your application has a huge number of soft/weak references then it could increase the time of the remark phase. You can pass the "-XX:+PrintReferenceGC" VM argument for more details.

 

2) You can also pass the "-XX:ConcGCThreads" VM argument to increase the number of parallel marking threads. Increasing the concurrent marking threads will make garbage collection run fast.

 

3) You may want to increase your java heap size and if you can't increase the heap size and if you notice that the marking cycle is not starting early enough to reclaim the old generation then reduce "-XX:InitiatingHeapOccupancyPercent". The default value is 45%. Reducing the value will start the marking cycle earlier. On the other hand, if the marking cycle is starting early and not reclaiming, increase the "-XX:InitiatingHeapOccupancyPercent" threshold above the default value.

 

4) "-Xnoclassgc" - As per my understanding this argument is not recommended because it may cause out of memory errors.

Profile Image

Christian Nüssgens

Thanks for your fast reply.

 

In addition here is the plain output of the gc log with the GC remark

2021-12-20T11:28:34.914+0100: 421438.951: [GC remark 2021-12-20T11:28:34.915+0100: 421438.952: [Finalize Marking, 0.0369695 secs] 2021-12-20T11:28:34.952+0100: 421438.989: [GC ref-proc, 0.0207319 secs] 2021-12-20T11:28:34.972+0100: 421439.010: [Unloading, 13.8785201 secs], 14.3193429 secs]
 [Times: user=0.14 sys=0.24, real=14.39 secs]

If I understand correctly its "just" the Unloading taking that long.
Is it still possible that this is caused by reference processing?

Profile Image

Mahesh

Hi Christian,

 

As I can see in your comment that "real" time is greater than "user" and "sys" time then it might be due to the following reasons:

+ Lack of CPU

+ Heavy I/O activities

 

For more details please visit this blog.

Profile Image

Ram Lakshmanan

Hello Christian!

 

 Your application has been running for 5+ days. In all these 5 days, only twice GC pause has exceeded 10 seconds:

 

2021-12-19T10:08:34.128: 10.14 seconds
2021-12-20T11:28:34.914: 14.39 seconds

 

 Under the both occassions "real" time has been greater than "user' and "sys" time.  As mahesh had pointed out, you may want to check out this post to learn more about it.

 

 Here are the steps from here:

 

a.  If you are using any monitoring tools, you may want to check CPU or disk (i.e. I/O) activity has high during the above mentioned timestamps. 

b. If #a isn't the case, you may consider increasing GC thread count in the application. More GC parallelization can also reduce the GC pause time.

 

 Keep us posted about your findings.

Got something else on mind? Post Your Question

Not the answer you're looking for? Browse other questions tagged

  • probleminwebapps

  • tomcatinstallations

  • vmargument

  • xnoclassgc

Edit your question

Separate with comma, space or ENTER key. Special characters are not allowed in tag.

Edit your Answer

Edit your Comment