Why so many blocked threads appeared and suddenly resolved?

Hello! 

 Greetings. This problem seems to be stemming because of inefficient code in Atlassian SDK. You have uplaoded 6 thread dumps. But if you notice starting from dump 4, 5 and 6, number of BLOCKED threads count start to grow rapidly. In Dump #4, there were 104 BLOCKED threads, in Dump #5 there were 189 BLOCKED threads, in Dump #189 there are 237 BLOCKED threads

 All of these threads are BLOCKED because of atlassian SDK. It looks like atlassian SDK is invoking 'java.lang.System.getProperty()' method to read the system properties on every call (instead of caching them).

'java.lang.System.getProperty()' underlyingly uses 'java.util.Hashtable.get()' method. This method is a synchronized method. It means only one thread can invoke this method at a time. If any other threads tries to invoke this method they will be put in to BLOCKED state. Unfortunately atlassian JIRA SDK is invoking this '

java.lang.System.getProperty()' every single time. Due to which application is becoming unresponsive.

 Here is the transitive dependency graph indiacating the threads that are getting BLOCKED because of this inefficient programming.

 Here with I am attaching the stacktrace of few atlassian JIRA code who is invoking this '

java.lang.System.getProperty()' repeatedly.

ForkJoinPool-777-worker-2
Stack Trace is:
java.lang.Thread.State: RUNNABLE
at java.util.Hashtable.get(Hashtable.java:362)
- locked <0x0000000080f5e118> (a java.util.Properties)
at java.util.Properties.getProperty(Properties.java:969)
at java.util.Properties.getProperty(Properties.java:988)
at java.lang.System.getProperty(System.java:756)
at net.java.ao.atlassian.ConverterUtils.enforceLength(ConverterUtils.java:16)
at net.java.ao.atlassian.ConverterUtils.checkLength(ConverterUtils.java:9)
at net.java.ao.atlassian.AtlassianFieldNameConverter.getName(AtlassianFieldNameConverter.java:46)
at net.java.ao.Common.getValueFieldsNames(Common.java:421)
at net.java.ao.EntityManager.find(EntityManager.java:649)
at net.java.ao.EntityManager.find(EntityManager.java:621)
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.find(EntityManagedActiveObjects.java:140)
at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.find(TenantAwareActiveObjects.java:296)

Camel Thread #6 - backboneThreadPool
Stack Trace is:
java.lang.Thread.State: RUNNABLE
at java.util.Hashtable.get(Hashtable.java:362)
- locked <0x0000000080f5e118> (a java.util.Properties)
at java.util.Properties.getProperty(Properties.java:969)
at java.util.Properties.getProperty(Properties.java:988)
at java.lang.System.getProperty(System.java:756)
at net.java.ao.atlassian.ConverterUtils.enforceLength(ConverterUtils.java:16)
at net.java.ao.atlassian.ConverterUtils.checkLength(ConverterUtils.java:9)
at net.java.ao.atlassian.AtlassianFieldNameConverter.getName(AtlassianFieldNameConverter.java:46)
at net.java.ao.Common.getPrimaryKeyField(Common.java:257)
at net.java.ao.EntityProxy.fetchOneToMany(EntityProxy.java:308)
at net.java.ao.EntityProxy.invoke(EntityProxy.java:174)

http-nio-8080-exec-105 url: /browse/QG-12191,

Stack Trace is:
java.lang.Thread.State: RUNNABLE
at java.util.Hashtable.get(Hashtable.java:362)
- locked <0x0000000080f5e118> (a java.util.Properties)
at java.util.Properties.getProperty(Properties.java:969)
at java.util.Properties.getProperty(Properties.java:988)
at java.lang.System.getProperty(System.java:756)
at org.ofbiz.core.entity.EntityFindOptions.<init>(EntityFindOptions.java:124)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.forEach(SelectQueryImpl.java:221)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.asList(SelectQueryImpl.java:164)
at com.atlassian.jira.issue.link.DefaultIssueLinkManager.getLinks(DefaultIssueLinkManager.java:343)
at com.atlassian.jira.issue.link.DefaultIssueLinkManager.lambda$new$0(DefaultIssueLinkManager.java:80)
at com.atlassian.jira.issue.link.DefaultIssueLinkManager$$Lambda$338/1649014422.load(Unknown Source)
at com.atlassian.jira.cache.request.RequestCacheImpl.get(RequestCacheImpl.java:58)
at com.atlassian.jira.issue.link.DefaultIssueLinkManager$$Lambda$3854/1365889448.apply(Unknown Source)

So what version of atlassian JIRA SDK are you using? Can you check whether they have come up with new version of the library? Can you check whether they have fixed this issue in their newer version?

Hello!

 I went ahead and filed this as a ticket in the atlassian community: 

Application unresponsive, BLOCKED Threads (atlassian.com)