High thread count - apache axis client

Hello Ritesh,

First of all, there are more than 3350+ threads created in your application. This is a high thread count. This can result in Out Of Memory error. Unable to create new native thread. 

Also found below problems in the thread report that you attached:

1) 1365 threads were executing java.net.SocketInputStream.socketRead0(Native Method). This may slow down the transactions. Here is the stack trace of one thread:

stackTrace:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x0000000761db9a88> (a java.io.BufferedInputStream)
at org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:583)
at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:143)
at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
at org.apache.axis.client.Call.invoke(Call.java:2767)
at org.apache.axis.client.Call.invoke(Call.java:2443)
at org.apache.axis.client.Call.invoke(Call.java:2366)
at org.apache.axis.client.Call.invoke(Call.java:1812)
at com.newgen.ondemandpdfgeneration.OnDemandPDFGenerationSoap11BindingStub.generatePDFUsingXML(OnDemandPDFGenerationSoap11BindingStub.java:219)
at com.newgen.lic.ondemandpdfgeneration.client.GeneratePDFThread.run(GeneratePDFThread.java:72)
Locked ownable synchronizers:
- None

This stack trace shows the program that is currently running in a "RUNNABLE" state. It appears that the program is making a network call over a socket to a remote server, specifically using the Axis web services framework to invoke a SOAP-based web service method named "generatePDFUsingXML".

The program seems to be blocked at the read operation performed by the BufferedInputStream, which is currently locked. The thread is waiting for data to be available to read from the socket, which is likely being processed by the remote server.

Here are the potential solutions for this:

1. Instrument timeout settings:

You can set the “readTimeout” property in the HTTP Transport Policy Set for the Web Service client or set “timeout” on the org.apache.axis2.context.MessageContext in the application code.

2) Work with remote application

Sometimes it could be quite possible that transactions might be slowing down because of performance problems in the remote application. In those circumstances, you need to bring it to remote application’s awareness of the slow down and work with them to fix the problem.

3) Non-blocking HTTP client

You can also consider using non-blocking HTTP client libraries like Grizzly or Netty which do not have blocking operations to hang a thread. But this solution is more a strategic solution, which involves code changes & thorough testing.

Note, this is a comprehensive list but maybe not be a complete list of potential solutions. If you have additional solutions and timeout settings that you would like to add to this blog, please drop us a note in the below feedback section. We will be glad to update this blog with your recommendation(s).