Profile Image
manasa.a156

What is causing RUNNABLE thread count to grow up

We are seeing below error despite there are no unusal spike in cpu utilization and heap memory

java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

 

Report URL - https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjMvMTEvMTMvdGhyZWFkLWR1bXAtMTEtMTAtMDMudHh0LS0yMS0zMS0y

  • neo4j

  • neo4j-java-driver

  • outofmemoryerror

Please Sign In or to post your comment or answer

Profile Image

Ram Lakshmanan

Hello Manasa!

 

 Greetings. By looking at your thread dump, problem seems very obvious. Your application has 10,607 threads. In this 10,348 threads all have the same stacktrace (i.e. >97% of threads have the same stack trace). Below is the stacktrace of those 10,348 threads:

 

java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPoll.wait(java.base@11.0.20/Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.20/EPollSelectorImpl.java:120)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.20/SelectorImpl.java:124)
- locked <0x000000009dc4d420> (a org.neo4j.driver.internal.shaded.io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000009dc4d3c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(java.base@11.0.20/SelectorImpl.java:141)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:810)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(java.base@11.0.20/Thread.java:829)

 

 If you notice all these threads are originating from neo4j graph database driver. So it's a clear indication problem in originating from ne04j driver. Here are the next steps, I will do:

 

a. Check whether you are running on the latest version of ne04j driver library?

b. Whether there are any issues going ton the ne04j Database. Sometimes when there is an issues on the ne04j database (like lack of diskspace, multiple threads can be spanned by the driver.

c. Did you make any recent code changes or configuration changes on how you are invoking ne04j?

 

 

 

Got something else on mind? Post Your Question

Not the answer you're looking for? Browse other questions tagged
  • neo4j

  • neo4j-java-driver

  • outofmemoryerror