Profile Image
Uday Kumar

Response time are very high due to blocked threads

https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjMvMDEvNS90aHJlYWR0YWtlNS50eHQtLTEyLTE0LTIx&


Report URL - https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjMvMDEvNS90aHJlYWR0YWtlNS50eHQtLTEyLTE0LTIx

  • high response time

  • blocked threads

  • synchronized code block

  • Spring Framework

  • Class loading problem

  • classloader api

  • jackson library

Please Sign In or to post your comment or answer

Profile Image

Mahesh

Hi Uday,

 

In the attached fastThread report I could see the below 2 threads are stuck in 

 java.lang.ClassLoader.loadClass() method. Before getting stuck both threads acquired a lock due to that 37 threads are blocked. Here are the stuck threads: http-nio-8099-exec-71 and http-nio-8099-exec-91

 

I am not sure about the root cause but both threads are stuck while loading a class dynamically. Here is a source code of java.lang.ClassLoader.loadClass() method:

@Override
    public Class<?> loadClass(String name, boolean resolve)
            throws ClassNotFoundException {
        synchronized (getClassLoadingLock(name)) {
            Class<?> result = findExistingLoadedClass(name);
            result = (result != null) ? result : doLoadClass(name);
            if (result == null) {
                throw new ClassNotFoundException(name);
            }
            return resolveIfNecessary(result, resolve);
        }
    }

 

In the above code snippet you can see the synchronized code block in a method, so many threads must block there waiting to obtain the object's monitor lock.

Profile Image

sainath

Hello Uday Kumar,

 

The reason for the threads being in a blocked state in your application is due to the line

java.lang.ClassLoader.loadClass(). In your Spring Framework try to load the class using ClassLoader#loadClass(). Because of that, there are many threads are in the blocked state. Here is the stack trace of http-nio-8099-exec-2 the thread:

 

java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ClassLoader.loadClass(java.base@11.0.17/ClassLoader.java:569)
- waiting to lock <0x0000000709804ea0> (a java.lang.Object)
at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:151)
at java.lang.ClassLoader.loadClass(java.base@11.0.17/ClassLoader.java:522)
at java.lang.Class.forName0(java.base@11.0.17/Native Method)
at java.lang.Class.forName(java.base@11.0.17/Class.java:398)
at org.springframework.util.ClassUtils.forName(ClassUtils.java:292)
at org.springframework.util.ClassUtils.isPresent(ClassUtils.java:354)
at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.registerWellKnownModulesIfAvailable(Jackson2ObjectMapperBuilder.java:840)
at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.configure(Jackson2ObjectMapperBuilder.java:691)
at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.build(Jackson2ObjectMapperBuilder.java:674)
at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.<init>(MappingJackson2HttpMessageConverter.java:59)
at org.springframework.web.client.RestTemplate.<init>(RestTemplate.java:178)
at com.fnp.commons.rest.RestConnector.<init>(RestConnector.java:36)
at com.fnp.dooda.rest.CMSServiceClient.getWidgetConfiguration(CMSServiceClient.java:24)
at com.fnp.dooda.service.support.WidgetServiceSupport.getWidgetData(WidgetServiceSupport.java:55)
at com.fnp.dooda.webapp.controller.WidgetController.widget(WidgetController.java:46)
at jdk.internal.reflect.GeneratedMethodAccessor141.invoke(Unknown Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.17/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.17/Method.java:566)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at com.fnp.authcommonfilter.filter.TokenAuthenticationFilter.doFilterInternal(TokenAuthenticationFilter.java:127)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at com.fnp.authcommonfilter.filter.TokenAuthenticationFilter.doFilterInternal(TokenAuthenticationFilter.java:127)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at com.fnp.commons.web.filter.XSSFilter.doFilter(XSSFilter.java:27)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at com.fnp.commons.web.filter.RequestContextFilter.doFilter(RequestContextFilter.java:36)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at com.fnp.authcommonfilter.filter.TokenAuthenticationFilter.doFilterInternal(TokenAuthenticationFilter.java:127)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.cloud.sleuth.instrument.web.servlet.TracingFilter.doFilter(TracingFilter.java:68)
at org.springframework.cloud.sleuth.autoconfig.instrument.web.TraceWebServletConfiguration$LazyTracingFilter.doFilter(TraceWebServletConfiguration.java:131)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.springframework.cloud.sleuth.instrument.web.tomcat.TraceValve.invoke(TraceValve.java:90)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x000000071ac0db70> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(java.base@11.0.17/Thread.java:829)

If many threads are in the blocked state then eventually performance gets impacted. 

 

To understand why threads enter into a BLOCKED state when invoking the ‘ClassLoader.loadClass()’ method, we will have to look at its source code. Below is the source code excerpt of the ClassLoader.loadClass() method. If you would like to see the complete source code of java.lang.ClassLoader, you may refer here:

 

protected Class<?> loadClass(String name, boolean resolve)
        throws ClassNotFoundException
    {
        synchronized (getClassLoadingLock(name)) {
            // First, check if the class has already been loaded
            Class<?> c = findLoadedClass(name);
            if (c == null) {
                long t0 = System.nanoTime();
                try {
                    if (parent != null) {
                        c = parent.loadClass(name, false);
                    } else {
                        c = findBootstrapClassOrNull(name);
                    }
                    :
                    :

In the above lines of the source code, you will see the usage of ‘synchronized’ code block. When a block of code is synchronized, only one thread will be allowed to enter that block. In your case, many threads are trying to access ‘ClassLoader.loadClass()’ concurrently. Only one thread will be allowed to enter into the synchronized code block, the remaining threads will be put into a BLOCKED state. 

 

Here are potential solutions:

a. Try to see whether you can invoke ‘ClassLoader.loadClass()’ API during application startup time instead of run-time.

 

b. If your application is loading the same class again & again at runtime, then try to load the class only once. After that point, cache the class and re-use it, as shown in the above example.

 

c. Check whether frameworks have given any fixes in their latest version, if so upgrade to the latest version.

 

Profile Image
Ram Lakshmanan
Hello Uday!


 


 Greetings.


 


 Your REST application is facing the exact Class loading problem that is summarized in this post.


 


 Your application is using Spring Framework, which is internally using Jackson library to serialized/deserialize JSON. During this serialization/deserialization you are running in to the class loading issue as summarized in the above post.


 


 What version of Spring Framework, jackson library are you using in your application? May be you are using a old version? It could be quite possible for this classloading issue has been given in the newer version of these libraries.


 
 
						                	

						            	

						      
						            	
						                    
							                    
						                        
						                  
									

									
								

							

						

						
	                    
	                

	                

	                    

	                        Got something else on mind?
	                        
	                    		Post Your Question
	                    	
	                    

	                

	                 
	                

	                    
Not the answer you're looking for? Browse other questions tagged
	                        
    
	                        
  • high response time
  • blocked threads
  • synchronized code block
  • Spring Framework
  • Class loading problem
  • classloader api
  • jackson library
    • 
		                    

		                

	                 

	                 
	            

	            





	        

	    

	

	
	
	



	 
    
	
	

        

        	
            
Edit your question

            
			

				

					
				

			

			

				
				

					
					

						
						
					

				

			

			

				
				

					

						

							
						

						
						
						

		                    
		                    

		                    	

									

										
									  

									  
									

								

								
Separate with comma, space or ENTER key. Special characters are not allowed in tag.

								
							

				         	
						

		                    
		                    
		                

					

				

			

		

    


	
	
	



	 
    

	

        

        	
            
Edit your Answer

            
			

				

					
				

			

			

				

					

						

							
						

						
							
						

		                    
		                    
		                

					

				

			

		

    


	
	
	



	 
    

	

        

        	
            
Edit your Comment