JVM is restarting at different times for a few applications running on the same server. Please advise

Hello Travis!

 Based on your description, I here are few potential suspects to intermittent restarts:

a. Operating System/Kernel is terminating the process

b. Application is crashed due to internal errors

c. Application is running in to memory issues (May be OutOfMemoryErrors)

 I looked in to the thread dump you have attached. It looks normal. However to diagnose these suspects, just thread dump isn't enough. Infact it's also not clear whether you captured this thread dump from the JVM before it died or during healthy time frame. 

 Let's see how to diagnose above suspects:

a. Operating System/Kernel is terminating the process

 If kernel would have terminated the Java process, it would be reported in the kernel logs. Can you issue the below command on your server. Below command will print the kernel logs: 

 dmesg -T

 Once you have the kernel logs, please share it here. We can see the details.

b. Application is crashed due to internal errors

 When Java application crashes it generates hs_err_pid file. Can you search for this file in your server and see what is reported there? YOu can share this file here or you can upload it in fastThread tool. It can also analyze it.

c. Application is running in to memory issues (May be OutOfMemoryErrors)

 Do you have GC log enabled on your JVM? If not consider enabling it. It will not add any overhead to your application. It will show wealth of information about the application's memory performance. 

 Equipped with above information we can diagnose the problem effectively. OR you are also welcome to use the open source yCrash script, which will capture above mentioned and more logs to diagnose the problem more effectively.

Hey Ram, Thanks for your immediate response concerning this problem. I wanted to give you feedback on your recommendations: I made the adjustments as you suggested. It was determined that the hypervisor was killing the VM-guest consuming large memory, as you suggested and confirmed in the kernel buffer logs (dmesg -T). This was a tough one to figure out and appreciate all your input to help isolate the issue. Thanks!

-Travis