首页 > 其他分享 >分布式服务雪崩

分布式服务雪崩

时间：2023-11-30 13:45:41浏览次数：48  
标签：java springframework 分布式服务 util 雪崩 apache org ApplicationFilterChain

分布式服务雪崩

简介

公司分布式服务链路，底层强依赖数美第三方风控，校验用户是否黑产，随着并发量上升，数美接口返回时间逐渐增加，导致风控服务很多线程等待中，其他调用风控服务的上游服务线程也等待，然后上游的上游。。。，一层层全阻塞。

模拟：

新建springboot服务

tomcat配置：

server:
  tomcat:
    uri-encoding: UTF-8
    #最大链接数
    max-connections: 1000
    #最大等待队列长度
    accept-count: 200
    #请求头最大长度kb
    max-http-header-size: 1048576
    #请请求体最大长度kb
    #max-http-post-size: 2097152
    threads:
      #最大线程数
      max: 150
      #最小线程数
      min-spare: 10
  #服务http端口
  port: 8080

同一个服务，分端口部署，模拟上游调用下游

下游接口：

	@RequestMapping(value = "/test", method = {RequestMethod.GET})
	public ResultBase test() throws InterruptedException {
		Thread.sleep(30000);
		return ResultBase.getSuccessResult(ResultDTO.of().setResult(null));
	}

使用RestTemplate调用下游接口

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.ClientHttpRequestFactory;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.web.client.RestTemplate;


@Configuration
public class RestTemplateConfig {
	@Bean
	public RestTemplate restTemplate(ClientHttpRequestFactory factory) {
		return new RestTemplate(factory);

	}

	@Bean
	public ClientHttpRequestFactory simpleClientHttpRequestFactory() {
		SimpleClientHttpRequestFactory factory = new SimpleClientHttpRequestFactory();
		factory.setConnectTimeout(15000);
		factory.setReadTimeout(5000);
		return factory;
	}
}

使用abtest进行压测

ab -n 60000 -c 300 http://127.0.0.1:8080/rest

使用同步线程调用下游

	@RequestMapping(value = "/rest", method = {RequestMethod.GET})
	public ResultBase rest() {
		restTemplate.getForObject("http://127.0.0.1:8090/test", ResultBase.class);
		return ResultBase.getSuccessResult(ResultDTO.of().setResult(null));
	}

调用前：

image-20231129181738198

掉用开始：

image-20231129181615344

部分进入WAITING

image-20231129181640413

使用异步线程调用下游

	@RequestMapping(value = "/rest", method = {RequestMethod.GET})
	public ResultBase rest() {
		CompletableFuture.supplyAsync(() -> {
			//长时间的计算任务
			restTemplate.getForObject("http://127.0.0.1:8090/test", ResultBase.class);
			return null;
		});
		return ResultBase.getSuccessResult(ResultDTO.of().setResult(null));
	}

开始调用前：

image-20231129174340278

进行中：

image-20231129174552153

可以看到，15个forkjoinpool

image-20231129174649024

差不多150个请求在WAITING

[arthas@20636]$ thread 228
"http-nio-8080-exec-170" Id=228 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@bd6cb20 owned by "http-nio-8080-exec-268" Id=326
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@bd6cb20
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    at ch.qos.logback.core.OutputStreamAppender.writeBytes(OutputStreamAppender.java:197)
    at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:231)
    at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
    at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
    at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
    at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
    at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
    at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
    at ch.qos.logback.classic.Logger.filterAndLog_1(Logger.java:398)
    at ch.qos.logback.classic.Logger.info(Logger.java:583)
    at com.shanjige.chatgpt.chatgptweb.aop.ControllerAspect.aroundAdvice(ControllerAspect.java:80)
    at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
    at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692)
    at com.shanjige.chatgpt.chatgptweb.web.IndexController$$EnhancerBySpringCGLIB$$a0081f82.rest(<generated>)
    at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:626)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:97)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

image-20231129173949265

差不多150个请求在TIMED_WAITING

[arthas@20636]$ thread 319
"http-nio-8080-exec-261" Id=319 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e4f12a6
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e4f12a6
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:90)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

结束后：

image-20231129174003462

比开始时，多了15个forkjoinpool

使用线程池调用下游

	private ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(20, 20, 200, TimeUnit.SECONDS,
			new LinkedBlockingDeque<>(300));

	@RequestMapping(value = "/rest", method = {RequestMethod.GET})
	public ResultBase rest() {
		threadPoolExecutor.execute(()-> restTemplate.getForObject("http://127.0.0.1:8090/test", ResultBase.class));
		return ResultBase.getSuccessResult(ResultDTO.of().setResult(null));
	}

image-20231129170832340

可以看到，线程池设置的为20，线程池满了阻塞以后，差不多150个请求线程全都WAITING

参考

曹工说Tomcat：200个http-nio-8080-exec线程全都被第三方服务拖住了，这可如何是好（上：线程模型解析）

标签：java,springframework,分布式服务,util,雪崩,apache,org,ApplicationFilterChain
From： https://www.cnblogs.com/hongdada/p/17867100.html

相关文章

  • Redis缓存雪崩、击穿、穿透解释及解决方法，缓存预热，布隆过滤器 ，互斥锁
    Redis缓存雪崩、击穿、穿透解释及解决方法，缓存预热，布隆过滤器，互斥锁......
  • Redis 缓存系统常见问题及解决方案（缓存击穿,缓存穿透,缓存雪崩）
    1、缓存穿透缓存穿透指当用户在Redis缓存系统执行一条无效查询时，这条无效查询将穿透Redis缓存系统并向MySQL数据库请求数据，而MySQL数据库也获取不到数据。黑客可以利用缓存穿透原理，恶意执行大量无效查询，这将会对MySQL数据库的访问造成很大的压力解决方法：1、缓存......
  • 缓存雪崩/击穿/穿透
    缓存雪崩大量缓存同时过期，就叫缓存雪崩。缓存中有大量数据同时过期，导致大量请求缓存缺失redis实例宕机措施设置过期时间时，随机小范围打散服务降级缓存命令率下降到警告值或者数据库负载突然增大时，很可能发生了缓存雪崩。可以通过服务降级措施，来保证核心接口能正......
  • 如何解决缓存穿透、缓存击穿、缓存雪崩
    SpringBoot在缓存方面也提供了一些优秀的解决方案，帮助我们解决缓存穿透、缓存击穿、缓存雪崩等问题。一、Redis缓存Redis是一个高性能的键值对存储数据库，也是一个基于内存的数据结构存储系统，同时也支持持久化数据存储。Redis提供了丰富的数据结构，包括字符串、哈希、列表、集合、有......
  • 分布式服务
    1.RPC2.API网关3.服务注册与发现4.负载均衡5.系统监控6.容器化7.ServiceMesh8.微服务......
  • redis缓存更新策略，缓存穿透，缓存雪崩，缓存击穿。封装redis工具类
    (redis缓存)缓存是存储数据的临时地方，一般读写性能高1.给商铺添加缓存思路：在对应的serviceImpl里写逻辑@OverridepublicResultqueryById(Longid){Stringkey=CACHE_SHOP_KEY+id;//1.从redis查询商铺缓存StringshopJSON=stringRedisTemplate.opsF......
  • 什么是缓存雪崩
    在同一时间大量请求同一个接口，接口就会不断的查询数据库，就会导致缓存的雪崩； 解决办法：在基础时间的基础上，再加上一个随机的过期时间比如10-15s； ps：不要使用newRandom生成随机数，因为大量的随机数使用newRandom可能会导致生成同一个数值；......
  • Redis中的缓存雪崩、缓存击穿、缓存穿透问题
    1.什么是缓存雪崩当我们提到缓存系统中的问题，缓存雪崩是一个经常被讨论的话题。缓存雪崩是指在某一时刻发生大量的缓存失效，导致瞬间大量的请求直接打到了数据库，可能会导致数据库瞬间压力过大甚至宕机。尤其在高并发的系统中，这种情况会导致连锁反应，整个系统可能会崩溃。1.1缓存......
  • 看完包你搞懂Redis缓存穿透、击穿和雪崩！！！说到做到
    缓存穿透缓存穿透是指当用户对Redis发出无效或者不存在的数据信息操作时，这条数据在Redis中不存在，Redis就会在MySQL数据库中查询，可时无效的信息在mysql数据库中也不存在，就会造成Redis一直查询MySQL，对MySQL造成极大压力解决方式方式一：返回缓存空值这种方式有点像“以牙还牙”，对......
  • 分布式服务的接口幂等如何设计
    接口幂等性就是用户对同一接口发起了一次或多次请求之后，对数据的影响是不变的，不会因为多次请求而产生不同的结果     ......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99