Unable to figure out the class that is causing new Executor service to spawn and thus, lead to extra threads in application.

Hi Satin,

Sorry for late reply. If you see the the Identical stack trace table, you will see 12 threads are stuck and waiting for the response from other external services. You can click on the stacktrace link to see a more detail.

Below is the stacktrace :

at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.aerospike.client.cluster.Connection.readFully(Connection.java:97)
at com.aerospike.client.command.DeleteCommand.parseResult(DeleteCommand.java:63)
at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:57)
at com.aerospike.client.AerospikeClient.delete(AerospikeClient.java:425)
at com.snapdeal.aerospike.client.services.impl.AerospikeClientServiceImpl.deleteRecord(AerospikeClientServiceImpl.java:147)
at com.snapdeal.aerospike.manager.impl.CommunicationCandidateManagerImpl.deleteCandidate(CommunicationCandidateManagerImpl.java:238)
at com.snapdeal.comsender.postprocessor.PostProcessorImpl.execute(PostProcessorImpl.java:81)
at com.snapdeal.comsender.processor.impl.CommunicationProcessorImpl.execute(CommunicationProcessorImpl.java:191)
at com.snapdeal.comsender.processor.impl.CommunicationProcessorImpl.execute(CommunicationProcessorImpl.java:52)
at com.snapdeal.comsender.service.workers.scheduleComms.ScheduleWorker.run(ScheduleWorker.java:42)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

+ I think you may consider setting the timeout value using the setQueryTimeout() API.

This API will set the number of seconds the JDBC driver will wait for getting results from database. If the limit is exceeded, SQLTimeoutException is thrown. JDBC driver applies this limit to the execute, executeQuery() and executeUpdate() methods. By default, there is no limit on the amount of time allowed for a running statement to complete.

Hello Satin!

 Greetings.

 Your application has total 348 threads. FastThread report has the 'Thread Group' section. Below is the excerpt of this section from your report:

You can notice in this section, 199 threads are in the 'pool' thread group. With in this pool thread group, I see threads with following names:

pool-Transactional-thread

pool-new-scheduled

pool-kafkaConsumer

pool-TriggerExtractor

But most of these threads are originating from 'pool-Transactional-thread'. Below are the stacktrace of the some of the threads in this pool. They all seem to be executing 'com.snapdeal.comsender' class. So I believe with this you should be able to figure out the originating executor.

pool-Transactional-thread-5453

PRIORITY : 5

THREAD ID : 0X00007F22D0042800

NATIVE ID : 0X4356

NATIVE ID (DECIMAL) : 17238

STATE : RUNNABLE

stackTrace:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:961)
- locked <0x00000005ad3d6500> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:918)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000005ab178f98> (a sun.security.ssl.AppInputStream)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at org.apache.http.impl.BHttpConnectionBase.fillInputBuffer(BHttpConnectionBase.java:346)
at org.apache.http.impl.BHttpConnectionBase.awaitInput(BHttpConnectionBase.java:356)
at org.apache.http.impl.DefaultBHttpClientConnection.isResponseAvailable(DefaultBHttpClientConnection.java:132)
at org.apache.http.impl.conn.CPoolProxy.isResponseAvailable(CPoolProxy.java:152)
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:218)
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:63)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:122)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1286)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1102)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:759)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:733)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:715)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:675)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:657)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:521)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1807)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1665)
at com.snapdeal.ce.core.fileAccessor.S3Manager.putStreamText(S3Manager.java:119)
at com.snapdeal.comsender.helpers.CommunicationAuditHelper.saveCommunicationPayloadToS3(CommunicationAuditHelper.java:47)
at com.snapdeal.comsender.processor.SingleCommunicationProcessorWithDPLogging.sendCommunication(SingleCommunicationProcessorWithDPLogging.java:318)
at com.snapdeal.comsender.processor.impl.TransactionalCommunicationProcessorImpl.execute(TransactionalCommunicationProcessorImpl.java:176)
at com.snapdeal.comsender.processor.impl.TransactionalCommunicationProcessorImpl.execute(TransactionalCommunicationProcessorImpl.java:56)
at com.snapdeal.comsender.service.packet.impl.PacketProcessingServiceImpl.lambda$processTranCandidateRequest$28(PacketProcessingServiceImpl.java:74)
at com.snapdeal.comsender.service.packet.impl.PacketProcessingServiceImpl$$Lambda$991/507240845.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

pool-Transactional-thread-5450

PRIORITY : 5

THREAD ID : 0X00007F22D0040800

NATIVE ID : 0X4345

NATIVE ID (DECIMAL) : 17221

STATE : RUNNABLE

stackTrace:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.aerospike.client.cluster.Connection.readFully(Connection.java:97)
at com.aerospike.client.command.WriteCommand.parseResult(WriteCommand.java:67)
at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:57)
at com.aerospike.client.AerospikeClient.put(AerospikeClient.java:339)
at com.snapdeal.aerospike.client.services.impl.AerospikeClientServiceImpl.putRecord(AerospikeClientServiceImpl.java:164)
at com.snapdeal.aerospike.manager.impl.IdempotencyCappingManagerImpl.setCapping(IdempotencyCappingManagerImpl.java:36)
at com.snapdeal.comsender.service.capping.impl.CommunicationCappingService.setIdempotentCappingData(CommunicationCappingService.java:189)
at com.snapdeal.comsender.service.capping.impl.CommunicationCappingService.setAllCapping(CommunicationCappingService.java:157)
at com.snapdeal.comsender.processor.impl.TransactionalCommunicationProcessorImpl.execute(TransactionalCommunicationProcessorImpl.java:251)
at com.snapdeal.comsender.processor.impl.TransactionalCommunicationProcessorImpl.execute(TransactionalCommunicationProcessorImpl.java:56)
at com.snapdeal.comsender.service.packet.impl.PacketProcessingServiceImpl.lambda$processTranCandidateRequest$28(PacketProcessingServiceImpl.java:74)
at com.snapdeal.comsender.service.packet.impl.PacketProcessingServiceImpl$$Lambda$991/507240845.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)