sudden spike in server CPU from less than 10% to 65%

Hello Pooja!

 Greetings. You should have captured thread dump when the CPU spike is happening and not before or after. It looks like your are using dynatrace for monitoring. In both the dumps, you have sent I could see dynatrace capturing heap dump from your application. Capturing heap dump is a CPU/resource intensive operation. Is heap dump triggerred based on your knowledge? Even if it's the case why do you want to take it two times.  Here is the stack trace of the dynatrace thread which is capturing the heap dump in your application:

'oneagent memorydump' (Dynatrace thread):

at com.sun.management.internal.HotSpotDiagnostic.dumpHeap0(Native Method)
at com.sun.management.internal.HotSpotDiagnostic.dumpHeap(HotSpotDiagnostic.java:61)
Local Variable: java.lang.String#195031
Local Variable: com.sun.management.internal.HotSpotDiagnostic$$Lambda$1415#1
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:566)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at jdk.internal.reflect.GeneratedMethodAccessor163.invoke(<unknown string>)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:566)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)
Local Variable: com.sun.jmx.mbeanserver.ConvertingMethod#146
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
Local Variable: com.sun.jmx.mbeanserver.PerInterface$MethodAndSig#41
Local Variable: java.util.Collections$SingletonList#31
Local Variable: com.sun.management.internal.HotSpotDiagnostic#1
Local Variable: com.sun.jmx.mbeanserver.PerInterface#19
Local Variable: java.util.Collections$1#1
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
Local Variable: com.sun.jmx.mbeanserver.MXBeanSupport#4
at javax.management.StandardMBean.invoke(StandardMBean.java:405)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809)
Local Variable: javax.management.StandardMBean#3
Local Variable: com.sun.jmx.interceptor.DefaultMBeanServerInterceptor#1
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
Local Variable: java.lang.String#27352
Local Variable: java.lang.String[]#39657
at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(MXBeanProxy.java:150)
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:167)
Local Variable: javax.management.ObjectName#289
Local Variable: com.sun.jmx.mbeanserver.JmxMBeanServer#1
Local Variable: com.sun.jmx.mbeanserver.ConvertingMethod#190
Local Variable: com.sun.jmx.mbeanserver.MXBeanLookup#1
Local Variable: com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler#1
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:258)
Local Variable: java.lang.reflect.Method#68898
Local Variable: javax.management.MBeanServerInvocationHandler#1
Local Variable: java.lang.Object[]#109951
Local Variable: com.sun.jmx.mbeanserver.MXBeanProxy#1
at com.sun.proxy.$Proxy299.dumpHeap(<unknown string>)
Local Variable: com.sun.proxy.$Proxy299#1
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:566)
at com.compuware.apm.agent.util.memorydump.SunHeapDumpHandler.dumpHeap(SunHeapDumpHandler.java:19)
Local Variable: com.compuware.apm.agent.util.memorydump.SunHeapDumpHandler#1
Local Variable: java.lang.reflect.Method#86914
at com.compuware.apm.agent.util.memorydump.HeapDumpWriter.createHeapDump(HeapDumpWriter.java:61)
Local Variable: java.lang.String#232666
Local Variable: java.io.File#26
Local Variable: java.io.File#25
at com.compuware.apm.agent.util.memorydump.HeapDumpWriter.run(HeapDumpWriter.java:32)
Local Variable: java.lang.String#36949
Local Variable: com.compuware.apm.agent.util.memorydump.HeapDumpWriter#1
at java.lang.Thread.run(Thread.java:834)

Strategic/Right approach to diagnose CPU spike:

 Inorder to accurately point the lines of code causing the CPU spike, you need to analyze not only thread dumps but also 'top -H -p {PID}' command output, where {PID} is your Java application's process Id which is experiencing CPU spike. When you issue this ‘top’ command with given arguments, it will list all the threads running in the application and amount of CPU each one of the thread consume. Once you have both the data, you can identify high CPU consuming thread and lines of code they are executing. You can either capture these artifacts and analyze them manually, or use the 14 day trial version of yCrash tool. yCrash tool automatically captures  application-level data (thread dump, heap dump, Garbage Collection log) and system-level data (netstat, vmstat, iostat, top, top -H, dmesg,...). It marries these two datasets and generates an instant root cause analysis report pointing out exact line of code causing the CPU spike. Here is more information on how to diagnose high CPU spike.