what might be the root cause of so many threads in waiting state

Hi Santosh,

In your fastThread report, I could see that you have uploaded 2 dumps. In "Dump1", total 55 threads are waiting for the response from the external service and in "Dump2", total 54 threads are waiting for the response from the external service.

Dump1 -

1) 44 threads are waiting for the response from the ActiveMQ service. You are using Apache ActiveMQ in your application for messaging service. Can you try increasing a timeout for ActiveMQ in your code? Here is an identical stack trace of these threads - 

ActiveMQ Transport: tcp://amq-a/172.27.204.123:61616@44346
PRIORITY :5

THREAD ID :0X00007F110C00D000

NATIVE ID :0X1C42

NATIVE ID (DECIMAL) :7234

STATE :RUNNABLE

stackTrace:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(java.base@11.0.11/Native Method)
at java.net.SocketInputStream.socketRead(java.base@11.0.11/SocketInputStream.java:115)
at java.net.SocketInputStream.read(java.base@11.0.11/SocketInputStream.java:168)
at java.net.SocketInputStream.read(java.base@11.0.11/SocketInputStream.java:140)
at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634)
at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59)
at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619)
at java.io.DataInputStream.readInt(java.base@11.0.11/DataInputStream.java:392)
...

2) 10 threads are waiting for the response from the Oracle JDBC driver. As per my understanding, while connecting to your oracle database the connection got stuck and never timeout. In order to fix this issue, you may consider setting the timeout while creating a connection. Here is an identical stack trace of these threads - 

ForkJoinPool-5-worker-37
PRIORITY :5

THREAD ID :0X00007F111C209000

NATIVE ID :0X1CE7

NATIVE ID (DECIMAL) :7399

STATE :RUNNABLE

stackTrace:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(java.base@11.0.11/Native Method)
at java.net.SocketInputStream.socketRead(java.base@11.0.11/SocketInputStream.java:115)
at java.net.SocketInputStream.read(java.base@11.0.11/SocketInputStream.java:168)
at java.net.SocketInputStream.read(java.base@11.0.11/SocketInputStream.java:140)
at oracle.net.ns.Packet.receive(Packet.java:311)
at oracle.net.ns.DataPacket.receive(DataPacket.java:105)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:249)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:171)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:89)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
at oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225)
at oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1066)
at oracle.jdbc.driver.OracleStatement.fetchMoreRows(OracleStatement.java:3716)
at oracle.jdbc.driver.InsensitiveScrollableResultSet.fetchMoreRows(InsensitiveScrollableResultSet.java:1015)
at oracle.jdbc.driver.InsensitiveScrollableResultSet.absoluteInternal(InsensitiveScrollableResultSet.java:979)
at oracle.jdbc.driver.InsensitiveScrollableResultSet.next(InsensitiveScrollableResultSet.java:579)
- locked <0x00000005fadd5ee8> (a oracle.jdbc.driver.T4CConnection)
at com.zaxxer.hikari.pool.HikariProxyResultSet.next(HikariProxyResultSet.java)
at org.hibernate.loader.Loader.getRowsFromResultSet(Loader.java:1035)
at org.hibernate.loader.Loader.processResultSet(Loader.java:990)
...

JDK's java.net layer uses SocketInputStream.socketRead0() API to read and receive the data from the external application. Sometimes your application might not get a response from the external applications. In that case, your application thread will be stuck in this java.net.SocketInputStream.socketRead0() API.

Dump 2 -

1) 45 threads are waiting for the response from the ActiveMQ service. You can check point #1 mentioned above in Dump1.

2) 9 threads are waiting for the response from the Oracle JDBC driver. Please check point #2 mentioned above in Dump1.