Profile Image
Seokwon Park

Can anybody explain my prob prblem?

2023-04-19 23:35:51

Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0.18+9-LTS-jvmci-21.3-b27 mixed mode, sharing):

Threads class SMR info:

_java_thread_list=0x00007fe750030a50, length=48, elements={

0x00007fe79828f000, 0x00007fe798291000, 0x00007fe798296800, 0x00007fe798299000,

0x00007fe79829b000, 0x00007fe79829d000, 0x00007fe79829f000, 0x00007fe7982b4000,

0x00007fe7983a2800, 0x00007fe798afc000, 0x00007fe798afe000, 0x00007fe798b01000,

0x00007fe750001800, 0x00007fe750003800, 0x00007fe75c00a000, 0x00007fe75c00c000,

0x00007fe799311800, 0x00007fe7994e5800, 0x00007fe79933a800, 0x00007fe799520800,

0x00007fe799a22000, 0x00007fe7999c1800, 0x00007fe799a52800, 0x00007fe799b0c800,

0x00007fe740011800, 0x00007fe75000b800, 0x00007fe799435800, 0x00007fe7991e7000,

0x00007fe7990ab000, 0x00007fe7990ac000, 0x00007fe7990b1000, 0x00007fe799a8b800,

0x00007fe799a8d800, 0x00007fe798023000, 0x00007fe74400b000, 0x00007fe74c2c3000,

0x00007fe758006000, 0x00007fe75803d000, 0x00007fe774030000, 0x00007fe75c03c000,

0x00007fe758007800, 0x00007fe7489b5800, 0x00007fe74003a800, 0x00005652102f9800,

0x00007fe740037800, 0x00007fe74c6d3000, 0x00007fe74c28a000, 0x00007fe750037800

}

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=3179.48ms elapsed=610041.45s tid=0x00007fe79828f000 nid=0x51 waiting on condition  [0x00007fe77d17b000]

   java.lang.Thread.State: RUNNABLE

at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.18/Native Method)

at java.lang.ref.Reference.processPendingReferences(java.base@11.0.18/Reference.java:241)

at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.18/Reference.java:213)

   Locked ownable synchronizers:

- None

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=4182.69ms elapsed=610041.45s tid=0x00007fe798291000 nid=0x52 in Object.wait()  [0x00007fe77d07a000]

   java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(java.base@11.0.18/Native Method)

- waiting on <no object reference available>

at java.lang.ref.ReferenceQueue.remove(java.base@11.0.18/ReferenceQueue.java:155)

- waiting to re-lock in wait() <0x00000000c01fe958> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(java.base@11.0.18/ReferenceQueue.java:176)

at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.18/Finalizer.java:170)

   Locked ownable synchronizers:

- None

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.58ms elapsed=610041.40s tid=0x00007fe798296800 nid=0x53 runnable  [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

- None

"Service Thread" #5 daemon prio=9 os_prio=0 cpu=0.14ms elapsed=610041.39s tid=0x00007fe798299000 nid=0x54 runnable  [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

- None

"JVMCI-native CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=239387.45ms elapsed=610041.38s tid=0x00007fe79829b000 nid=0x55 runnable  [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   No compile task

   Locked ownable synchronizers:

- None

"C1 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=20282.88ms elapsed=610041.38s tid=0x00007fe79829d000 nid=0x56 waiting on condition  [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   No compile task

   Locked ownable synchronizers:

- None

"Sweeper thread" #8 daemon prio=9 os_prio=0 cpu=638.23ms elapsed=610041.36s tid=0x00007fe79829f000 nid=0x57 runnable  [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

- None

"Common-Cleaner" #9 daemon prio=8 os_prio=0 cpu=449.27ms elapsed=610041.26s tid=0x00007fe7982b4000 nid=0x58 in Object.wait()  [0x00007fe77c661000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(java.base@11.0.18/Native Method)

- waiting on <no object reference available>

at java.lang.ref.ReferenceQueue.remove(java.base@11.0.18/ReferenceQueue.java:155)

- waiting to re-lock in wait() <0x00000000c00487c0> (a java.lang.ref.ReferenceQueue$Lock)

at jdk.internal.ref.CleanerImpl.run(java.base@11.0.18/CleanerImpl.java:148)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

at jdk.internal.misc.InnocuousThread.run(java.base@11.0.18/InnocuousThread.java:134)

   Locked ownable synchronizers:

- None

"Heartbeater" #10 daemon prio=5 os_prio=0 cpu=22.82ms elapsed=610038.25s tid=0x00007fe7983a2800 nid=0x6a waiting on condition  [0x00007fe76dffe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c01d02d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1182)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"AsyncAppender-Worker-async-file-appender" #13 daemon prio=5 os_prio=0 cpu=0.14ms elapsed=609999.87s tid=0x00007fe798afc000 nid=0x76 waiting on condition  [0x00007fe76c4a6000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c15b25d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ArrayBlockingQueue.take(java.base@11.0.18/ArrayBlockingQueue.java:417)

at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)

   Locked ownable synchronizers:

- None

"AsyncAppender-Worker-async-file-appender" #14 daemon prio=5 os_prio=0 cpu=10700.79ms elapsed=609999.84s tid=0x00007fe798afe000 nid=0x77 waiting on condition  [0x00007fe76c3a5000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c16e3860> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ArrayBlockingQueue.take(java.base@11.0.18/ArrayBlockingQueue.java:417)

at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)

   Locked ownable synchronizers:

- None

"AsyncAppender-Worker-async-oci-log-file-appender" #15 daemon prio=5 os_prio=0 cpu=9336.36ms elapsed=609999.82s tid=0x00007fe798b01000 nid=0x78 waiting on condition  [0x00007fe76c2a4000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c16e3ac0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ArrayBlockingQueue.take(java.base@11.0.18/ArrayBlockingQueue.java:417)

at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)

   Locked ownable synchronizers:

- None

"logback-1" #16 daemon prio=5 os_prio=0 cpu=13081.77ms elapsed=609999.44s tid=0x00007fe750001800 nid=0x79 waiting on condition  [0x00007fe76c1a3000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c16e3d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1170)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"logback-2" #17 daemon prio=5 os_prio=0 cpu=13232.64ms elapsed=609999.44s tid=0x00007fe750003800 nid=0x7a waiting on condition  [0x00007fdf3cdfe000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c16e3d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1170)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"logback-3" #18 daemon prio=5 os_prio=0 cpu=16948.09ms elapsed=609999.41s tid=0x00007fe75c00a000 nid=0x7b waiting on condition  [0x00007fdf3ccfd000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c16e3d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1170)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"logback-4" #19 daemon prio=5 os_prio=0 cpu=14484.44ms elapsed=609999.39s tid=0x00007fe75c00c000 nid=0x7c waiting on condition  [0x00007fdf3cbfc000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c16e3d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1170)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"Tomcat JDBC Pool Cleaner[1442407170:1681337394306]" #21 daemon prio=5 os_prio=0 cpu=23561.06ms elapsed=609957.26s tid=0x00007fe799311800 nid=0x8b waiting for monitor entry  [0x00007fdf3c4f8000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:1978)

- waiting to lock <0x00000000c63df968> (a oracle.jdbc.driver.T4CConnection)

at org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:416)

at org.apache.tomcat.jdbc.pool.PooledConnection.release(PooledConnection.java:652)

at org.apache.tomcat.jdbc.pool.ConnectionPool.release(ConnectionPool.java:636)

at org.apache.tomcat.jdbc.pool.ConnectionPool.abandon(ConnectionPool.java:590)

at org.apache.tomcat.jdbc.pool.ConnectionPool.checkAbandoned(ConnectionPool.java:1088)

at org.apache.tomcat.jdbc.pool.ConnectionPool$PoolCleaner.run(ConnectionPool.java:1590)

at java.util.TimerThread.mainLoop(java.base@11.0.18/Timer.java:556)

at java.util.TimerThread.run(java.base@11.0.18/Timer.java:506)

   Locked ownable synchronizers:

- <0x00000000c3744640> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)

"Timer-0" #22 daemon prio=5 os_prio=0 cpu=0.22ms elapsed=609956.97s tid=0x00007fe7994e5800 nid=0x8c in Object.wait()  [0x00007fdf3abd6000]

   java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(java.base@11.0.18/Native Method)

- waiting on <0x00000000c1ef51c8> (a java.util.TaskQueue)

at java.lang.Object.wait(java.base@11.0.18/Object.java:328)

at java.util.TimerThread.mainLoop(java.base@11.0.18/Timer.java:527)

- waiting to re-lock in wait() <0x00000000c1ef51c8> (a java.util.TaskQueue)

at java.util.TimerThread.run(java.base@11.0.18/Timer.java:506)

   Locked ownable synchronizers:

- None

"oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser" #23 daemon prio=4 os_prio=1 cpu=109.37ms elapsed=609956.96s tid=0x00007fe79933a800 nid=0x8d in Object.wait()  [0x00007fdf3aad5000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(java.base@11.0.18/Native Method)

- waiting on <no object reference available>

at oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:329)

- waiting to re-lock in wait() <0x00000000c1ef51e0> (a oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser)

   Locked ownable synchronizers:

- None

"InterruptTimer" #24 daemon prio=5 os_prio=0 cpu=1536.18ms elapsed=609956.13s tid=0x00007fe799520800 nid=0x8e in Object.wait()  [0x00007fdf3a9d4000]

   java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(java.base@11.0.18/Native Method)

- waiting on <no object reference available>

at java.lang.Object.wait(java.base@11.0.18/Object.java:328)

at java.util.TimerThread.mainLoop(java.base@11.0.18/Timer.java:527)

- waiting to re-lock in wait() <0x00000000c1ef5558> (a java.util.TaskQueue)

at java.util.TimerThread.run(java.base@11.0.18/Timer.java:506)

   Locked ownable synchronizers:

- None

"Metrics-Worker-1" #27 prio=5 os_prio=0 cpu=86254.02ms elapsed=609950.89s tid=0x00007fe799a22000 nid=0x90 waiting on condition  [0x00007fdf397d2000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(java.base@11.0.18/Native Method)

at java.lang.Thread.sleep(java.base@11.0.18/Thread.java:339)

at java.util.concurrent.TimeUnit.sleep(java.base@11.0.18/TimeUnit.java:446)

at com.oracle.pic.telemetry.commons.metrics.Metrics$Worker.run(Metrics.java:608)

   Locked ownable synchronizers:

- None

"FileSystemWatchService" #30 daemon prio=5 os_prio=0 cpu=58889.69ms elapsed=609950.88s tid=0x00007fe7999c1800 nid=0x91 runnable  [0x00007fdf396d1000]

   java.lang.Thread.State: RUNNABLE

at sun.nio.fs.LinuxWatchService.poll(java.base@11.0.18/Native Method)

at sun.nio.fs.LinuxWatchService$Poller.run(java.base@11.0.18/LinuxWatchService.java:316)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"GCLog-Monitor-Thread" #29 daemon prio=5 os_prio=0 cpu=575081.45ms elapsed=609950.87s tid=0x00007fe799a52800 nid=0x92 waiting on condition  [0x00007fdf395d0000]

   java.lang.Thread.State: TIMED_WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c1ef5ca8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)

at java.util.concurrent.LinkedBlockingDeque.pollFirst(java.base@11.0.18/LinkedBlockingDeque.java:513)

at java.util.concurrent.LinkedBlockingDeque.poll(java.base@11.0.18/LinkedBlockingDeque.java:675)

at sun.nio.fs.AbstractWatchService.poll(java.base@11.0.18/AbstractWatchService.java:108)

at com.oracle.pic.telemetry.commons.G1GCLogMonitor$Worker.run(G1GCLogMonitor.java:135)

   Locked ownable synchronizers:

- None

"Timer-1" #32 prio=5 os_prio=0 cpu=6.83ms elapsed=609950.48s tid=0x00007fe799b0c800 nid=0x93 in Object.wait()  [0x00007fdf392cf000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(java.base@11.0.18/Native Method)

- waiting on <no object reference available>

at java.util.TimerThread.mainLoop(java.base@11.0.18/Timer.java:553)

- waiting to re-lock in wait() <0x00000000c1ef5f80> (a java.util.TaskQueue)

at java.util.TimerThread.run(java.base@11.0.18/Timer.java:506)

   Locked ownable synchronizers:

- None

"pool-5-thread-1" #33 prio=5 os_prio=0 cpu=2302978.58ms elapsed=609950.38s tid=0x00007fe740011800 nid=0x94 waiting on condition  [0x00007fdf391ce000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c144c4f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.18/LinkedBlockingQueue.java:433)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"ForkJoinPool.commonPool-worker-3" #34 daemon prio=5 os_prio=0 cpu=2351.97ms elapsed=609950.37s tid=0x00007fe75000b800 nid=0x95 waiting on condition  [0x00007fdf390cd000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c122e460> (a java.util.concurrent.ForkJoinPool)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.18/ForkJoinPool.java:1628)

at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.18/ForkJoinWorkerThread.java:183)

   Locked ownable synchronizers:

- None

"AsyncAppender-Worker-async-oci-access-log-file-appender" #35 daemon prio=5 os_prio=0 cpu=9341.77ms elapsed=609947.31s tid=0x00007fe799435800 nid=0x98 waiting on condition  [0x00007fdf38dcc000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c30317e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ArrayBlockingQueue.take(java.base@11.0.18/ArrayBlockingQueue.java:417)

at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)

   Locked ownable synchronizers:

- None

"metrics-scheduled-metric-reporter-1-thread-1" #38 daemon prio=5 os_prio=0 cpu=24162.30ms elapsed=609947.22s tid=0x00007fe7991e7000 nid=0x9a waiting on condition  [0x00007fdf38bca000]

   java.lang.Thread.State: TIMED_WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c15be738> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1182)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"ServiceHostReporter" #39 daemon prio=5 os_prio=0 cpu=8163.45ms elapsed=609947.19s tid=0x00007fe7990ab000 nid=0x9b in Object.wait()  [0x00007fdf38ac9000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(java.base@11.0.18/Native Method)

- waiting on <no object reference available>

at java.util.TimerThread.mainLoop(java.base@11.0.18/Timer.java:553)

- waiting to re-lock in wait() <0x00000000c3031bc8> (a java.util.TaskQueue)

at java.util.TimerThread.run(java.base@11.0.18/Timer.java:506)

   Locked ownable synchronizers:

- None

"AsyncEmitter-Worker-Thread" #26 daemon prio=5 os_prio=0 cpu=2470691.37ms elapsed=609947.19s tid=0x00007fe7990ac000 nid=0x9c waiting on condition  [0x00007fdf389c8000]

   java.lang.Thread.State: TIMED_WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c147ed08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)

at java.util.concurrent.ArrayBlockingQueue.poll(java.base@11.0.18/ArrayBlockingQueue.java:432)

at com.oracle.pic.commons.metrics.emitters.AsyncEmitter$Worker.run(AsyncEmitter.java:209)

   Locked ownable synchronizers:

- None

"pool-6-thread-1" #40 prio=5 os_prio=0 cpu=154017.30ms elapsed=609946.95s tid=0x00007fe7990b1000 nid=0x9d waiting on condition  [0x00007fdf388c7000]

   java.lang.Thread.State: TIMED_WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c20694c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1182)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"dw-42-acceptor-0@337c9d35-application@52997ccc{SSL, (ssl, http/1.1)}{0.0.0.0:24000}" #42 prio=3 os_prio=2 cpu=581.73ms elapsed=609946.06s tid=0x00007fe799a8b800 nid=0x9f runnable  [0x00007fdf384c5000]

   java.lang.Thread.State: RUNNABLE

at sun.nio.ch.ServerSocketChannelImpl.accept0(java.base@11.0.18/Native Method)

at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@11.0.18/ServerSocketChannelImpl.java:533)

at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@11.0.18/ServerSocketChannelImpl.java:285)

at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)

at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:704)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- <0x00000000c303c108> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"dw-admin-43-acceptor-0@433c1548-admin@30114f16{HTTP/1.1, (http/1.1)}{0.0.0.0:19001}" #43 prio=3 os_prio=2 cpu=9252.20ms elapsed=609946.05s tid=0x00007fe799a8d800 nid=0xa0 runnable  [0x00007fdf383c4000]

   java.lang.Thread.State: RUNNABLE

at sun.nio.ch.ServerSocketChannelImpl.accept0(java.base@11.0.18/Native Method)

at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@11.0.18/ServerSocketChannelImpl.java:533)

at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@11.0.18/ServerSocketChannelImpl.java:285)

at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)

at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:704)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- <0x00000000c303bd98> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"DestroyJavaVM" #44 prio=5 os_prio=0 cpu=14269.11ms elapsed=609945.94s tid=0x00007fe798023000 nid=0x3b waiting on condition  [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

- None

"Scheduler-456552476-1" #46 prio=5 os_prio=0 cpu=1846.84ms elapsed=609933.65s tid=0x00007fe74400b000 nid=0xa3 waiting on condition  [0x00007fe73d0b2000]

   java.lang.Thread.State: TIMED_WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c303b8f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1182)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"pool-9-thread-1" #48 daemon prio=5 os_prio=0 cpu=260226186.78ms elapsed=609933.53s tid=0x00007fe74c2c3000 nid=0xa5 runnable  [0x00007fe73ceaf000]

   java.lang.Thread.State: RUNNABLE

at sun.security.ssl.TransportContext.isOutboundDone(java.base@11.0.18/TransportContext.java:487)

at sun.security.ssl.SSLEngineImpl.isOutboundDone(java.base@11.0.18/SSLEngineImpl.java:768)

- locked <0x00000000c63e2c48> (a sun.security.ssl.SSLEngineImpl)

at oracle.net.nt.SSLSocketChannel.shutdown(SSLSocketChannel.java:374)

- locked <0x00000000c63e28f8> (a oracle.net.nt.SSLSocketChannel)

at oracle.net.nt.SSLSocketChannel.write(SSLSocketChannel.java:242)

at oracle.net.ns.NIOPacket.writeToSocketChannel(NIOPacket.java:302)

at oracle.net.ns.NIONSDataChannel.writeDataToSocketChannel(NIONSDataChannel.java:173)

at oracle.net.ns.NIONSDataChannel.writeDataToSocketChannel(NIONSDataChannel.java:124)

at oracle.jdbc.driver.T4CMAREngineNIO.flush(T4CMAREngineNIO.java:727)

at oracle.jdbc.driver.T4CMAREngineNIO.prepareForUnmarshall(T4CMAREngineNIO.java:733)

at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:413)

at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485)

at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)

at oracle.jdbc.driver.T4C7Ocommoncall.doOCOMMIT(T4C7Ocommoncall.java:72)

at oracle.jdbc.driver.T4CConnection.doCommit(T4CConnection.java:961)

- eliminated <0x00000000c63df968> (a oracle.jdbc.driver.T4CConnection)

at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:1937)

- locked <0x00000000c63df968> (a oracle.jdbc.driver.T4CConnection)

at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:1942)

at jdk.internal.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.18/DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(java.base@11.0.18/Method.java:566)

at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:131)

at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109)

at org.apache.tomcat.jdbc.pool.interceptor.AbstractCreateStatementInterceptor.invoke(AbstractCreateStatementInterceptor.java:79)

at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109)

at org.apache.tomcat.jdbc.pool.DisposableConnectionFacade.invoke(DisposableConnectionFacade.java:81)

at com.sun.proxy.$Proxy81.commit(Unknown Source)

at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.commit(AbstractLogicalConnectionImplementor.java:86)

at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:282)

at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:101)

at io.dropwizard.hibernate.SessionFactoryHealthCheck.lambda$check$0(SessionFactoryHealthCheck.java:56)

at io.dropwizard.hibernate.SessionFactoryHealthCheck$$Lambda$795/0x0000000840b15c40.call(Unknown Source)

at java.util.concurrent.FutureTask.run(java.base@11.0.18/FutureTask.java:264)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1128)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- <0x00000000c303d778> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"logback-5" #49 daemon prio=5 os_prio=0 cpu=16066.34ms elapsed=609933.41s tid=0x00007fe758006000 nid=0xa6 waiting on condition  [0x00007fe73cbaf000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c16e3d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1170)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"logback-6" #50 daemon prio=5 os_prio=0 cpu=13968.25ms elapsed=609933.42s tid=0x00007fe75803d000 nid=0xa7 waiting on condition  [0x00007fe73caae000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c16e3d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1170)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"Scheduler-192232101-1" #57 prio=5 os_prio=0 cpu=176.94ms elapsed=609627.28s tid=0x00007fe774030000 nid=0xc4 waiting on condition  [0x00007fe73c5ab000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c303dbc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1170)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"logback-7" #123 daemon prio=5 os_prio=0 cpu=11778.28ms elapsed=606946.47s tid=0x00007fe75c03c000 nid=0x1c8 waiting on condition  [0x00007fdf387c6000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c16e3d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1170)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"logback-8" #124 daemon prio=5 os_prio=0 cpu=14760.24ms elapsed=606946.43s tid=0x00007fe758007800 nid=0x1c9 waiting on condition  [0x00007fe73b8a8000]

   java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c16e3d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1170)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"pool-10-thread-1" #7170 daemon prio=5 os_prio=0 cpu=7581.34ms elapsed=325765.82s tid=0x00007fe7489b5800 nid=0x6e08 waiting on condition  [0x00007fe73b3a3000]

   java.lang.Thread.State: TIMED_WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c15c0440> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)

at java.util.concurrent.ArrayBlockingQueue.poll(java.base@11.0.18/ArrayBlockingQueue.java:432)

at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1053)

at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)

at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"dw-11931" #11931 prio=5 os_prio=0 cpu=2.97ms elapsed=20233.80s tid=0x00007fe74003a800 nid=0x77dc runnable  [0x00007fe73c9ad000]

   java.lang.Thread.State: RUNNABLE

at sun.nio.ch.EPoll.wait(java.base@11.0.18/Native Method)

at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.18/EPollSelectorImpl.java:120)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.18/SelectorImpl.java:124)

- locked <0x00000000c303de60> (a sun.nio.ch.Util$2)

- locked <0x00000000c303dcf0> (a sun.nio.ch.EPollSelectorImpl)

at sun.nio.ch.SelectorImpl.select(java.base@11.0.18/SelectorImpl.java:141)

at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183)

at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:190)

at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)

at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"dw-admin-12255" #12255 prio=5 os_prio=0 cpu=10.02ms elapsed=457.55s tid=0x00005652102f9800 nid=0x1f2 waiting on condition  [0x00007fe73af9f000]

   java.lang.Thread.State: TIMED_WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c303f938> (a java.util.concurrent.SynchronousQueue$TransferStack)

at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)

at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@11.0.18/SynchronousQueue.java:462)

at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@11.0.18/SynchronousQueue.java:361)

at java.util.concurrent.SynchronousQueue.poll(java.base@11.0.18/SynchronousQueue.java:937)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:324)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:399)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"dw-admin-12257" #12257 prio=5 os_prio=0 cpu=8.02ms elapsed=322.38s tid=0x00007fe740037800 nid=0x202 waiting on condition  [0x00007fe73c3a9000]

   java.lang.Thread.State: TIMED_WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c2071500> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)

at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"dw-admin-12260" #12260 prio=5 os_prio=0 cpu=2.59ms elapsed=120.92s tid=0x00007fe74c6d3000 nid=0x221 runnable  [0x00007fe73cfb1000]

   java.lang.Thread.State: RUNNABLE

at sun.nio.ch.EPoll.wait(java.base@11.0.18/Native Method)

at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.18/EPollSelectorImpl.java:120)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.18/SelectorImpl.java:124)

- locked <0x00000000c303fd58> (a sun.nio.ch.Util$2)

- locked <0x00000000c303faf8> (a sun.nio.ch.EPollSelectorImpl)

at sun.nio.ch.SelectorImpl.select(java.base@11.0.18/SelectorImpl.java:141)

at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183)

at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:190)

at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)

at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"dw-admin-12261" #12261 prio=5 os_prio=0 cpu=1.36ms elapsed=52.25s tid=0x00007fe74c28a000 nid=0x22b waiting on condition  [0x00007fe73b4a4000]

   java.lang.Thread.State: TIMED_WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)

- parking to wait for  <0x00000000c2071500> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)

at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018)

at java.lang.Thread.run(java.base@11.0.18/Thread.java:834)

   Locked ownable synchronizers:

- None

"Attach Listener" #12262 daemon prio=9 os_prio=0 cpu=0.69ms elapsed=0.48s tid=0x00007fe750037800 nid=0x245 waiting on condition  [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

- None

"VM Thread" os_prio=0 cpu=551760.15ms elapsed=610041.47s tid=0x00007fe79828c000 nid=0x50 runnable 

"GC Thread#0" os_prio=0 cpu=800261.63ms elapsed=610042.11s tid=0x00007fe798038000 nid=0x3e runnable 

"GC Thread#1" os_prio=0 cpu=798600.10ms elapsed=610033.44s tid=0x00007fe764001000 nid=0x6f runnable 

"G1 Main Marker" os_prio=0 cpu=14.34ms elapsed=610042.10s tid=0x00007fe798054000 nid=0x40 runnable 

"G1 Conc#0" os_prio=0 cpu=1352.96ms elapsed=610042.10s tid=0x00007fe798056000 nid=0x42 runnable 

"G1 Refine#0" os_prio=0 cpu=108.89ms elapsed=610042.08s tid=0x00007fe7981f6800 nid=0x45 runnable 

"G1 Refine#1" os_prio=0 cpu=0.93ms elapsed=610033.31s tid=0x00007fe768001000 nid=0x70 runnable 

"G1 Young RemSet Sampling" os_prio=0 cpu=163381.17ms elapsed=610042.07s tid=0x00007fe7981f8000 nid=0x46 runnable 

"VM Periodic Task Thread" os_prio=0 cpu=285968.06ms elapsed=610040.87s tid=0x00007fe7982fb800 nid=0x69 waiting on condition 

JNI global refs: 17, weak refs: 0

 

Report URL - https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjMvMDQvMjgvanN0YWNrLW9jMTAucnRmLS0yMS01Ni0xNg==

  • poolmanagement

  • threaddump

Please Sign In or to post your comment or answer

Profile Image

Ram Lakshmanan

Hello Seokwon!

 

 I reviewed your thread dump. There is only 57 threads in it. There are no warning signals in them. What is the issue you are facing? 

 

There could be several reasons for an application to suffer from performance hiccups. Some of the reasons are:

  • Garbage collection pauses
  • Threads getting BLOCKED
  • Network connectivity
  • Load balancer routing issue
  • Heavy CPU consumption of threads
  • Operating System running with old patches
  • Memory Leak
  • DB not responding properly

:

:

 

 So just thread dump is not enough to diagnose the problem. You have captured only thread dump, that too one snapshot of it. It's always a good practice to capture 3 thread dumps in a gap of 10 seconds between each one. Besides thread dumps you might have to capture other logs/artifacts to do thorough analysis.

 

 You can use the open source yCrash script which will capture 360-degree application level artifacts (like GC logs, 3 snapshots of thread dumps, heap dumps) and system level artifacts (like top, top -H, netstat, vmstat, iostat, dmesg, diskusage, kernel parameters...). Once you have these data, either you can manually analyze them or upload it to yCrash tool, which will analyze all these artifacts and generate root cause analysis report. It has potential to indicate the root cause of the problem.

 

Profile Image

Seokwon Park

I am trying to change the pool mangement. 

Profile Image

Ram Lakshmanan

Hello Seokwon! Can you provide more details. When you say 'Pool Management' what do you mean?

 

- Do you want to increase the threads?

- Do you want to lower the threads?

- Reduce the idle threads?

- Are you trying to manage your application's Executor pool?

- Are you using to manage 3rd party pool (i.e. application server thread pool)?

 

More details needed to give proper direction. 

Profile Image

Seokwon Park

Pool managment mean "JDBC pool connection management". The below is current configuration. Can you provide you opinion on this please?

 

dataSourceFactory {
driverClass: oracle.jdbc.OracleDriver
url: "jdbc:oracle:thin:@(description= xxxxx)"
user: ***
initialSize: 2
minIdleTime: 5m
minSize: 2
maxSize: 10
maxWaitForConnection: 5s
validationQuery: null
validationQueryTimeout: 5s
validationInterval: 60s
removeAbandoned: true
removeAbandonedTimeout: 120s
maxConnectionAge: 7200s
checkConnectionOnConnect: true
checkConnectionOnBorrow: true
checkConnectionOnReturn: false
checkConnectionWhileIdle: false
logAbandonedConnections: true
properties: {
"hibernate.dialect": "org.hibernate.dialect.Oracle12cDialect"
"hibernate.show_sql": "true"
"hibernate.generate_statistics": "true"
"hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS": "2000"
"current_session_context_class": "thread"
"hibernate.hbm2ddl.auto": "validate"
"oracle.jdbc.fanEnabled": "true"
"oracle.jdbc.implicitStatementCacheSize": "100"
"oracle.jdbc.defaultConnectionValidation": "SOCKET"
"secretName": "adw-occ-password"
"connectionType": "LOW"
"compartmentID": "ocid1.compartment.oc1..aaaaaaaa2f63cf5lvb6imsbkzxnrklrybc7fogkpdeob5yzoxj4mvb4uugwa"
}

 

Got something else on mind? Post Your Question

Not the answer you're looking for? Browse other questions tagged
  • poolmanagement

  • threaddump