Server responding slow

Hello Nadeem Mohammed!

There could be several reasons why server responding slow. Some of the reasons are:

• Garbage collection pauses
• Threads getting BLOCKED
• Network connectivity
• Load balancer routing issue
• Heavy CPU consumption of threads
• Operating System running with old patches
• Memory Leak
• DB not responding properly
• Kernel issues
• Backend slow downs
• Hypervisor/container orchestrator not allocating enough resources

:

:

 So just thread dump is not enough to diagnose the problem. You have captured only thread dump, that too one snapshot of it. It's always a good practice to capture 3 thread dumps in a gap of 10 seconds between each one. Besides thread dumps you might have to capture other logs/artifacts to do thorough analysis.

 You can use the open source yCrash script which will capture 360-degree application level artifacts (like GC logs, 3 snapshots of thread dumps, heap dumps) and system level artifacts (like top, top -H, netstat, vmstat, iostat, dmesg, diskusage, kernel parameters...). Once you have these data, either you can manually analyze them or upload it to yCrash tool, which will analyze all these artifacts and generate root cause analysis report. It has potential to indicate the root cause of the problem.

Hello Nadeem!

 I took a closer look at your report. I could see a major problem going on in your application, that could cause the slow down in your servers.

Weblogic Logging is BLOCKING threads

 Below is the excerpt from your thread dump report. You can see that fastThread tool is pointing out that 

line #81 of com.bea.logging.RotatingFileStreamHandler file in publish() method is blocking 12 threads. Behind every thread there is a customer waiting. Thus those customers wouldn't be getting any response. 

Below is the stacktrace of the thread which is Blocking the 12 threads.

java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
- locked <0x000000079afc5ef0> (a java.io.BufferedOutputStream)
at com.bea.logging.RotatingFileOutputStream.flush(RotatingFileOutputStream.java:257)
- locked <0x00000006c1920db8> (a weblogic.logging.FileStreamHandler)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
- locked <0x00000006c1921658> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at java.util.logging.StreamHandler.flush(StreamHandler.java:259)
- eliminated <0x00000006c1920db8> (a weblogic.logging.FileStreamHandler)
at com.bea.logging.RotatingFileStreamHandler.publish(RotatingFileStreamHandler.java:88)
- locked <0x00000006c1920db8> (a weblogic.logging.FileStreamHandler)
at java.util.logging.Logger.log(Logger.java:738)
at com.bea.logging.BaseLogger.log(BaseLogger.java:66)
at weblogic.logging.WLLogger.log(WLLogger.java:35)
at weblogic.management.logging.DomainLogHandlerImpl.publishLogEntries(DomainLogHandlerImpl.java:137)
at weblogic.management.logging.DomainLogHandlerImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:645)
at weblogic.rmi.internal.BasicServerRef$2.run(BasicServerRef.java:534)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:386)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:531)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:138)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:655)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)

You can notice from the stack trace that application is trying to do logging, however thread is unable to write in to the log file. This can happen because of following reasons:

a. Lack of file permissions on the log file

b. Multiple processes are trying to write in to the same log file

c. Lack of disk space on the server in which your appilication is running

d. Running on a older version of the logging library

e. Upgraded to recent version of the logging library (which introduced this problem).