threads are stuck in socketRead0(). Please advise

Hi Pratik,

Below are the problems that I can see in your thread dump:

+ 2 threads are BLOCKED on line #93 of weblogic.socket.PosixSocketMuxer file in processSockets() method. If threads are BLOCKED for a prolonged period, your application may become unresponsive.

+ 14 threads are stuck waiting for response from external system. It can slow down transactions.

+ 1 thread is BLOCKED on rollback() method in oracle.jdbc.driver.PhysicalConnection file.

+ 3 threads are blocked because of weblogic.socket.PosixSocketMuxer. In order to examine refer to the transitive graph section in the thread report. 

+ If you observe the above screenshot ExecuteThread: '3' for queue:... is blocking the 3 threads. To understand this graph click on the link on the right top of this section.

+ If you observe the above screenshot oracle.jdbc.driver.T4CConnection is blocking one thread. Examine the stack trace of this thread. 

Note: To see the identified problems and recommended solutions by our machine learning (ML) algorithms, please subscribe to our plan.

Hi Pratik,

As @Sainath mentioned, 14 threads are stuck in socketRead0() of the java.net.SocketInputStream file because they are waiting for a response from the external service. Find a few thread stack traces taken from your report below - 

1) From the below stack trace, It looks like threads are executing the DB query and reading DB response from the network socket. Maybe your query is taking a long time. You may consider setting a timeout. The Oracle JDBC thin driver has a property called "oracle.jdbc.ReadTimeout" which can be set to a number of milliseconds you're willing to wait.

stackTrace:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Packet.java:300)
at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:884)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)
- locked <0x00000006d5064098> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493)
at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:135)
:
:

2) I think in your application, DiameterMessageService is connecting to an external service and that is taking a long time to return the response.

stackTrace:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x00000007193f2958> (a java.io.BufferedInputStream)
at java.io.DataInputStream.read(DataInputStream.java:132)
at amdocs.jtes.diameter.impl.service.DiameterMessageService.readDiameterBytes(DiameterMessageService.java:483)
at amdocs.jtes.manager.DiameterResponseHandler.run(DiameterResponseHandler.java:71)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)

3) Here are some other stack trace that you can investigate - 

stackTrace:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at amdocs.adjust.fnd.network.impl.AbstractNetworkConnection.readInputMessageToByteBuffer(AbstractNetworkConnection.java:443)
at amdocs.adjust.fnd.network.impl.AbstractNetworkConnection.readMessage(AbstractNetworkConnection.java:309)
at amdocs.adjust.fnd.network.responses.ResponseHandler.handleResponseMessage(ResponseHandler.java:106)
at amdocs.adjust.fnd.network.responses.ResponseHandler.run(ResponseHandler.java:70)

stackTrace:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked <0x00000006cf6a1048> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at amdocs.epi.lock.util.SocketReceiver.readMessage(SocketReceiver.java:718)
at amdocs.epi.lock.client.ClientSocketReceiver.handleMessage(ClientSocketReceiver.java:144)
at amdocs.epi.lock.util.SocketReceiver.run(SocketReceiver.java:218)
at java.lang.Thread.run(Thread.java:662)
at amdocs.epi.session.EpiThread.run(EpiThread.java:204)

For more details visit 

https://blog.fastthread.io/2018/09/02/threads-stuck-in-java-net-socketinputstream-socketread0/