Profile Image
sahil.kalamkar

There are frequent FULL GCs and eventual container kills

Hi, I am facing an issue of my YARN containers being killed continuosly, the reason being Physical memory exceeded. Naturally I captured heap dumps, upon investigation, turned out at the time of Physical memory exceeding, a lot of large objects were unreachable. Indicating, they would have been collected by the GC in the next run, but either the container gets killed immediately after a full GC(Heap memory crossing 3000MB), or after a while. I checked for anomaly in Native memory allocation, but could not find out of normal space allocation, I checked the same since physical memory combines other aspects apart from JVM heap memory.

What can be a solution? 
One of the solutions coming to my mind is not allow my OLD gen grow beyond a certain point (say 90%), as soon as it reaches that threshold, ParallelGC should trigger Full GC.


Report URL - https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjUvMDMvMzEvbG93b25iZ2MubG9nLS0xMi01OC0yOA==&channel=WEB

    Please Sign In or to post your comment or answer

    Profile Image

    Charlie Arehart

    Sahil, you have not said how much memory is on the box (or VM or container) that is running this jvm/app. Is it around 5gb? It seems simply that you're exceeding that, thus the Linux OOM killer is killing the process/jvm (as "the largest user of memory").

     

    Perhaps you already "knew" that (but do let us know the OS memory size, in case it may help us). And perhaps you're just wondering "I set my max heap well within that, so how is it exceeding that?"

     

    Note that your report shows you using 1gb of metaspace. You could certainly set a maxmetaspace for the jvm...but that would merely cause the jvm itself to crash with its own oom:metaspace error.

     

    So assuming you can't raise the total os memory (or you merely prefer not to, or "just want to understand where the memory use is"), it seems the focus should turn to why the metaspace is growing so much. Usually that's about the tracking of meta info about class loading, and it's being filled due to excessive class loading. Sometimes there may be configurable aspects of your app (in your case, yarn) that might influence that, where you could get the app to lower the number of classes it's loading. I know nothing of yarn to suggest that.

     

    Maybe someone else will, or perhaps the ycrash team will see something else to suggest for you, or to ask if you. I'm just a fellow user and server troubleshooter offering my thoughts if they may help. 

    Profile Image

    sahil.kalamkar

    Hey thanks Charlie, 

    is running beyond physical memory limits. Current usage: 4.0 GB of 4 GB physical memory used; 5.6 GB of 16.4 GB virtual memory used. Killing container.\nDump of the process-tree for <cid> :\n\t|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE\n\t|- 1345266 1344628 1344628 1344628 (java) 150349 5676 5998563328 1052048 

    This is the error

    Another question I have is, how is metaspace memory decided, because even though I see 1GB allocated , I see only 0.1GB in peak usage.

    Profile Image

    Charlie Arehart

    Given what you've said, you could try adding :

    - XX:MaxMetaspaceSize=250mb

     

    That will limit the metaspace to that size. You might also lower the xmx to 3gb, to limit that as well, so that both are well within that 4g on the box. Again, though, perhaps the ycrash folks or others will have better recommendations. 

    Got something else on mind? Post Your Question

    Edit your question

    Separate with comma, space or ENTER key. Special characters are not allowed in tag.

    Edit your Answer

    Edit your Comment