Application unresponsive and webcontainer blocking threads

Hello Ved!

 Based on the thread dump, I could see that your application should have become unresponsive. I could see following are the potential reasons for your application to go unresponsive.

Reason 1: 'Webcontainer 883' thread blocking 89 threads

Above is the transitive dependency graph that is showing 'Webcontainer: 888' thread to be blocking 89 threads. Below is the top part of this thread's stacktrace:

at java/io/UnixFileSystem.getBooleanAttributes0(Native Method)
at java/io/UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:253(Compiled Code))
at java/io/File.exists(File.java:830(Compiled Code))
at sun/misc/URLClassPath$FileLoader.getResource(URLClassPath.java:1349(Compiled Code))
at sun/misc/URLClassPath.getResource(URLClassPath.java:338(Compiled Code))
at java/net/URLClassLoader$ClassFinder.run(URLClassLoader.java:1180(Compiled Code))
at java/security/AccessController.doPrivileged(AccessController.java:686)
at java/net/URLClassLoader.findClass(URLClassLoader.java:602(Compiled Code))
at java/lang/ClassLoader.loadClassHelper(ClassLoader.java:844(Compiled Code))
at java/lang/ClassLoader.loadClass(ClassLoader.java:823(Compiled Code))
at sun/misc/Launcher$AppClassLoader.loadClass(Launcher.java:325(Compiled Code))
at java/lang/ClassLoader.loadClass(ClassLoader.java:803(Compiled Code))
at org/eclipse/osgi/internal/loader/BundleLoader.findClassInternal(BundleLoader.java:502(Compiled Code))
at org/eclipse/osgi/internal/loader/BundleLoader.findClass(BundleLoader.java:422(Compiled Code))
at org/eclipse/osgi/internal/loader/BundleLoader.findClass(BundleLoader.java:410(Compiled Code))
at org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader.loadClass(DefaultClassLoader.java:107(Compiled Code))
at java/lang/ClassLoader.loadClassHelper(ClassLoader.java:836(Compiled Code))
at java/lang/ClassLoader.loadClass(ClassLoader.java:823(Compiled Code))
at com/ibm/ws/bootstrap/ExtClassLoader.loadClass(ExtClassLoader.java:135(Compiled Code))
at java/lang/ClassLoader.loadClass(ClassLoader.java:803(Compiled Code))
at java/lang/Class.forNameImpl(Native Method)
at java/lang/Class.forName(Class.java:273(Compiled Code))

:

:

Basically this thread acquired the lock and got stuck in the '

java/io/UnixFileSystem.getBooleanAttributes0(Native Method)'. I suspect there is something going on at the Disk/OS level. Can you check whether is sufficient disk space, whether there is proper read/write permissions,...

Reason 2: 'WebContainer : 641' thread is blocking 37 threads 

Above is the transitive dependency graph of 'WebContainer : 641' thread which is blocking 37 threads. Below is the top portion of the 'WebContainer : 641' thread's stacktrace:

at sun/misc/Unsafe.park(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:186(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:847(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:881(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1210(Compiled Code))
at java/util/concurrent/locks/ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:954(Compiled Code))
at org/terracotta/offheapstore/AbstractLockedOffHeapHashMap.computeIfPresentWithMetadata(AbstractLockedOffHeapHashMap.java:602(Compiled Code))
at org/terracotta/offheapstore/concurrent/AbstractConcurrentOffHeapMap.computeIfPresentWithMetadata(AbstractConcurrentOffHeapMap.java:785(Compiled Code))
at org/ehcache/impl/internal/store/disk/EhcachePersistentConcurrentOffHeapClockCache.computeIfPresentAndPin(EhcachePersistentConcurrentOffHeapClockCache.java:198(Compiled Code))
at org/ehcache/impl/internal/store/offheap/AbstractOffHeapStore.getAndFault(AbstractOffHeapStore.java:869(Compiled Code))
at org/ehcache/impl/internal/store/tiering/TieredStore$3.apply(TieredStore.java:100(Compiled Code))
at org/ehcache/impl/internal/store/tiering/TieredStore$3.apply(TieredStore.java:96(Compiled Code))
at org/ehcache/impl/internal/store/heap/OnHeapStore$15.apply(OnHeapStore.java:726(Compiled Code))
at org/ehcache/impl/internal/store/heap/OnHeapStore$15.apply(OnHeapStore.java:723(Compiled Code))
at org/ehcache/impl/internal/store/heap/OnHeapStore$Fault.get(OnHeapStore.java:1037(Compiled Code))
at org/ehcache/impl/internal/store/heap/OnHeapStore$Fault.access$1000(OnHeapStore.java:1010(Compiled Code))
at org/ehcache/impl/internal/store/heap/OnHeapStore.resolveFault(OnHeapStore.java:772(Compiled Code))
at org/ehcache/impl/internal/store/heap/OnHeapStore.getOrComputeIfAbsent(OnHeapStore.java:732(Compiled Code))
at org/ehcache/impl/internal/store/tiering/TieredStore.get(TieredStore.java:96(Compiled Code))
at org/ehcache/core/Ehcache.get(Ehcache.java:169(Compiled Code))
at org/ehcache/jsr107/Eh107Cache.get(Eh107Cache.java:94(Compiled Code))
at org/springframework/cache/jcache/JCacheCache.lookup(JCacheCache.java:75(Compiled Code))
at org/springframework/cache/support/AbstractValueAdaptingCache.get(AbstractValueAdaptingCache.java:56(Compiled Code))
at com/backbase/portal/foundation/business/caching/impl/GlobalModelCacheManagerImpl.getByKey(GlobalModelCacheManagerImpl.java:400)
at com/backbase/portal/foundation/business/caching/impl/GlobalModelCacheManagerImpl.getGlobalModelCacheEntity(GlobalModelCacheManagerImpl.java:75)

:

:

Basically this thread is getting stuck in ehcache. Can you check whether you are running on the latest version of ehcache library? May be consider upgrading this library.

Reason 3: Lots of threads are throwing Errors/Exceptions

Few threads are throwing Exceptions/Errors. This shouldn't be the problem, however would like to bring it to your visibility.

  'WebContainer : 307'  thread is throwing an error! Examine it's  stack trace.
  'WebContainer : 124'  thread is throwing an error! Examine it's  stack trace.
  'WebContainer : 436'  thread is throwing an error! Examine it's  stack trace.
  'WebContainer : 394'  thread is throwing an error! Examine it's  stack trace.
  'WebContainer : 58'  thread is throwing an error! Examine it's  stack trace.
  'WebContainer : 360'  thread is throwing an error! Examine it's  stack trace.
  'WebContainer : 422'  thread is throwing an error! Examine it's  stack trace.
  'WebContainer : 330'  thread is throwing an error! Examine it's  stack trace.
  'WebContainer : 399'  thread is throwing an error! Examine it's  stack trace.

Looks like several of these threads are throwing exceptions in the Google library.

at java/lang/J9VMInternals.getStackTrace(Native Method)
at java/lang/Throwable.getInternalStackTrace(Throwable.java:268(Compiled Code))
at java/lang/Throwable.getStackTrace(Throwable.java:196(Compiled Code))
at com/google/inject/internal/util/$SourceProvider.get(SourceProvider.java:60(Compiled Code))
at com/google/inject/spi/Elements$RecordingBinder.getSource(Elements.java:340(Compiled Code))
at com/google/inject/spi/Elements$RecordingBinder.bind(Elements.java:254)
at com/google/inject/spi/Elements$RecordingBinder.bind(Elements.java:124(Compiled Code))