Log4j漏洞修复
修复参考文档:https://www.cert.org.cn/publish/main/9/2021/20211215154225883558274/20211215154225883558274_.html
1. log4j是什么
- Apache Log4j是一个基于Java的日志记录组件
- 文档:https://logging.apache.org/log4j/2.x/
2. log4j漏洞是什么
Log4j2组件在处理程序日志记录时存在JNDI注入缺陷,未经授权的攻击者利用该漏洞,可向目标服务器发送精心构造的恶意数据,触发Log4j2组件解析缺陷,实现目标服务器的任意代码执行,获得目标服务器权限(用户输入一段字符串,log4j打印这段字符串的时候会作为代码去执行)
- 受影响的版本: 2.0 => 2.14版本(log4j使用的版本号落入此区间则需要进行升级到最新版本 -- 2.17)
- 文档:https://www.docker.com/blog/apache-log4j-2-cve-2021-44228/
3. 如何排查系统中使用了log4j
- Java构建的程序组件 → 到docker官网查询对应镜像是否经过log4j漏洞扫描 → githup上组件源码是否存在log4j依赖并检查log4j依赖版本 → 进入容器执行
find / -name log4j*
查询log4j的jar包依赖
4. log4j修复方案
- 升级log4j的版本 -- 2.17.1(更新Java程序的依赖版本)
- k8s组件版本升级
- 容器: 修改pod的image到最新版本(查询组件是否修复log4j漏洞)
5. 当前云平台受影响的组件
kubectl get pods -A|grep -E "jenkins|elasticsearch|sonarqube"
- 持续集成
- Jenkins
镜像地址:https://hub.docker.com/r/jenkins/jenkins/tags
# 当前镜像版本是:system/jenkins:lts (2.733版本)
-
Jenkins中未使用log4j作为日志组件,不需要进行升级处理
-
sonarqube
1.镜像地址:https://hub.docker.com/_/sonarqube?tab=tags
# 当前镜像版本是:system/steamer-sonarqube:6.7
-
经查询 sonarqube中 ES使用的log4j版本是:2.9.1,common使用的是:2.8.2 => 需要进行升级
-
监控
- Elasticsearch
- 文档:
- Elasticsearch
kubectl describe pod steamer-elasticsearch-1-0 -n kube-system
# 检查发现是版本是:system/elasticsearch:6.4.2
# 去hub.docker.com官网查询是否经过 log4j漏洞扫描
# 推荐更新镜像为:6.8.23
# log4j通过jar包引入
[root@demo elasticsearch]# find / |grep log4j
/usr/share/elasticsearch/config/log4j2.properties
/usr/share/elasticsearch/lib/log4j-1.2-api-2.11.1.jar
/usr/share/elasticsearch/lib/log4j-api-2.11.1.jar
/usr/share/elasticsearch/lib/log4j-core-2.11.1.jar
/usr/share/elasticsearch/modules/x-pack-security/log4j-slf4j-impl-2.11.1.jar
- 经查询目前使用的log4j版本为:2.11.1 => 需要进行升级
6. 修复方案
- 更新组件中log4j版本
- 升级组件版本
7. Docker login 登录私有Harbor仓库
docker login --username=admin https://hub.tiduyun.com:5000
# https://hub.tiduyun.com:5000 是私有仓库的 IP/域名:端口
# 指定账号 --username
- 配置从私有仓库拉取数据
# /etc/docker/daemon.josn
{
"insecure-registries": ["https://192.168.0.133:5000"],
"registry-mirrors": ["https://192.168.0.133:5000"]
}
8. 构建新的镜像
- es.dockerfile
FROM system/elasticsearch:6.4.2
COPY log4j-jar/log4j-1.2-api-2.17.1.jar /usr/share/elasticsearch/lib/log4j-1.2-api-2.11.1.jar
COPY log4j-jar/log4j-api-2.17.1.jar /usr/share/elasticsearch/lib/log4j-api-2.11.1.jar
COPY log4j-jar/log4j-core-2.17.1.jar /usr/share/elasticsearch/lib/log4j-core-2.11.1.jar
COPY log4j-jar/log4j-slf4j-impl-2.17.1.jar /usr/share/elasticsearch/modules/x-pack-security/log4j-slf4j-impl-2.11.1.jar
- sonarqube.dockerfile
FROM system/steamer-sonarqube:6.7
COPY log4j-jar/log4j-1.2-api-2.17.1.jar /opt/sonarqube/elasticsearch/lib/log4j-1.2-api-2.9.1.jar
COPY log4j-jar/log4j-api-2.17.1.jar /opt/sonarqube/elasticsearch/lib/log4j-api-2.9.1.jar
COPY log4j-jar/log4j-core-2.17.1.jar /opt/sonarqube/elasticsearch/lib/log4j-core-2.9.1.jar
COPY log4j-jar/log4j-api-2.17.1.jar /opt/sonarqube/lib/common/log4j-api-2.8.2.jar
COPY log4j-jar/log4j-core-2.17.1.jar /opt/sonarqube/lib/common/log4j-core-2.8.2.jar
COPY log4j-jar/log4j-to-slf4j-2.17.1.jar /opt/sonarqube/lib/common/log4j-to-slf4j-2.8.2.jar
COPY log4j-jar/log4j-over-slf4j-1.7.30.jar /opt/sonarqube/lib/server/log4j-over-slf4j-1.7.25.jar
下载链接:https://dlcdn.apache.org/logging/log4j/2.17.1/apache-log4j-2.17.1-bin.tar.gz
- 构建命令
sudo docker build -t="system/elasticsearch:6.4.3" -f ./es.dockerfile . --no-cache
sudo docker tag system/elasticsearch:6.4.3 192.168.0.133:5000/system/elasticsearch:6.4.3
sudo docker push 192.168.0.133:5000/system/elasticsearch:6.4.3
sudo docker build -t="system/steamer-sonarqube:6.7.2" -f ./sonarqube.dockerfile . --no-cache
# 重新对镜像打标签并将其推送到私有harbor仓库
sudo docker tag system/steamer-sonarqube:6.7.2 192.168.0.133:5000/system/steamer-sonarqube:6.7.2
sudo docker push 192.168.0.133:5000/system/steamer-sonarqube:6.7.2
因为sonarqube中版本跨度比较大,运行镜像时报错
9. 应用新的镜像
- 查询对应pod的控制器
- 编辑对象pod的控制器,修改image指定的镜像
- 删除当前运行的pod,触发新的pod创建
- 查询日志查看启动是否正常
10 ES更新为6.8.23
a. 拉取官方镜像并将镜像推送到私有仓库harbor
# 1.从docker官网拉取镜像
sudo docker pull elasticsearch:6.8.23
# 2.重新打tag
sudo docker tag elasticsearch:6.8.23 192.168.0.133:5000/system/elasticsearch:6.8.23
# 3. 推送镜像到本地私有仓库
sudo docker push 192.168.0.133:5000/system/elasticsearch:6.8.23
b. 修改ES所有pod对应的控制器中镜像为6.8.23镜像
# 查找pod资源
kubectl get pods -A|grep elasticsearch
# 查找资源的控制器
kubectl describe pod/steamer-elasticsearch-1-0 -n kube-system|grep -i controll
# 使用edit编译资源控制器中的image镜像指定6.8.23版本
kubectl edit statefulset steamer-elasticsearch-1 -n kube-system
c. 删除pod让其重新创建
# 删除ES对应的pod
kubectl delete pod steamer-elasticsearch-1-0 -n kube-system
# 删除日志收集组件
kubectl get pods -A|grep file
kubectl delete pod filebeat-gh8dd filebeat-jhphf filebeat-m8sh6 -n kube-system
d. 检查
# 查看filebeat日志是否报错
kubectl logs filebeat-bc4vt -n kube-system
# 查询ES是否有报错
kubectl logs steamer-elasticsearch-1-0 -n kube-system -c steamer-elasticsearch-1
# 查询ES是否正常写入
# 进入ES容器
kubectl exec -it steamer-elasticsearch-1-0 -n kube-system -c steamer-elasticsearch-1
# 执行ES查询api
# 1. 查询对应索引是否创建
curl -X GET 'http://localhost:9201/_cat/indices?v'
# 2. 查询索引下是否有写入,如果索引下文档数量是递增的,则有filebeat的写入
curl 192.168.0.133:9200/filebeat-2022.01.25/_count
e. 出现的错误 -- Unrecognized VM option 'UseConcMarkSweepGC'
Unrecognized VM option 'UseConcMarkSweepGC'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
# 原因是:VM option出现问题,当前6.8.23依赖的jdk版本不支持该参数,需要修改
# 可以在本地通过docker方式运行镜像,将容器中的jvm.options复制出来,放入并修改云平台中的configmap
- 6.8.23
Java options官方文档:https://docs.oracle.com/en/java/javase/14/docs/specs/man/java.html
# unit可以k m g, 如 Xms500m Xmx2g
-Xms1g # jvm启动时分配的内存,类似k8s中的request
-Xmx1g # java程序在运行时使用的内存上限
# 8-13指的是 java 8到13版本
8-13:-XX:+UseConcMarkSweepGC # 使用GC类型为CMS,GC过程可以应用程序过程并发执行,需要消耗更多CPU资源
8-13:-XX:CMSInitiatingOccupancyFraction=75 # 内存占用75%时执行GC
8-13:-XX:+UseCMSInitiatingOccupancyOnly # 仅用于设置回收的阈值,如果不指定则第一次GC使用Fraction值,后面GC自动调整阈值
14-:-XX:+UseG1GC # 使用G1GC替换CMSGC
14-:-XX:G1ReservePercent=25 # 保留内存的百分比,避免GC失败转为full GC
14-:-XX:InitiatingHeapOccupancyPercent=30 # 堆内存触发GC的阈值
# Java应用程序使用的变量使用-D开头
-Des.networkaddress.cache.ttl=60 # ES缓存DNS解析时间
-Des.networkaddress.cache.negative.ttl=10 # ES缓存失败的DNS解析时间
-XX:+AlwaysPreTouch # 在服务申请内存时是分配真实物理内存而不是虚拟内存
-Xss1m # Xss设置线程栈大小
-Djava.awt.headless=true # 无头模式,文档:https://www.oracle.com/technical-resources/articles/javase/headless.html
-Dfile.encoding=UTF-8 # 文件的编码为utf-8
-Djna.nosys=true # 允许Java程序更容易地使用本机库,而不需要JNI或其他本机代码
-XX:-OmitStackTraceInFastThrow # 省略异常栈信息从而快速抛出
14-:-XX:+ShowCodeDetailsInExceptionMessages # 在异常信息中显示代码详情
-Dio.netty.noUnsafe=true # netty在申请direct内存时先判断是否能使用unsafe模式
-Dio.netty.noKeySetOptimization=true # 开启netty反射优化
-Dio.netty.recycler.maxCapacityPerThread=0 # netty对象池大小
# log4j配置
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j2.formatMsgNoLookups=true
-Djava.io.tmpdir=${ES_TMPDIR} # 临时目录
# 调试参数
-XX:+HeapDumpOnOutOfMemoryError # JVM发生OOM时,自动生成DUMP文件
-XX:HeapDumpPath=data # 参数表示生成DUMP文件的路径,也可以指定文件名称
-XX:ErrorFile=logs/hs_err_pid%p.log # 错误日志文件
# GC参数
8:-XX:+PrintGCDetails # 打印GC详情
8:-XX:+PrintGCDateStamps # 打印GC日期时间戳
8:-XX:+PrintTenuringDistribution # JVM 在每次新生代GC时,打印出幸存区中对象的年龄分布
8:-XX:+PrintGCApplicationStoppedTime # 打印GC的STW时间
8:-Xloggc:logs/gc.log # GC日志保存文件
8:-XX:+UseGCLogFileRotation # GC日志滚动
8:-XX:NumberOfGCLogFiles=32 # GC滚动日志个数
8:-XX:GCLogFileSize=64m # GC滚动日志大小,必须大于
# Java 9 GC参数
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
9-:-Djava.locale.providers=COMPAT # Java9兼容模式
10-:-XX:UseAVX=2 # 指定AVX指令集版本
- 原 6.4.2
-Xms2g
-Xmx2g
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+AlwaysPreTouch
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Djava.io.tmpdir=${ES_TMPDIR}
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=data
-XX:ErrorFile=logs/hs_err_pid%p.log
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
9-:-Djava.locale.providers=COMPAT
f. configmap资源
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-es-config
namespace: kube-system
data:
elasticsearch1.yml: |-
xpack.security.enabled: false
http.cors.enabled: true
http.cors.allow-origin: "*"
cluster.name: prometheus-elasticsearch
node.name: node-singleton
node.master: true
http.host: 127.0.0.1
http.port: 9200
transport.host: 192.168.0.133
transport.tcp.port: 9300
indices.memory.index_buffer_size: 15%
thread_pool.bulk.queue_size: 1024
jvm.options: |-
-Xms2g
-Xmx2g
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30
-Des.networkaddress.cache.ttl=60
-Des.networkaddress.cache.negative.ttl=10
-XX:+AlwaysPreTouch
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
14-:-XX:+ShowCodeDetailsInExceptionMessages
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j2.formatMsgNoLookups=true
-Djava.io.tmpdir=${ES_TMPDIR}
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=data
-XX:ErrorFile=logs/hs_err_pid%p.log
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
9-:-Djava.locale.providers=COMPAT
11. 去除nginx代理
a. 创建statefulset资源对象
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: steamer-elasticsearch
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: Reconcile
app: steamer-elasticsearch
kubernetes.io/cluster-service: "true"
name: steamer-elasticsearch
version: 6.8.23
spec:
podManagementPolicy: OrderedReady
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: steamer-elasticsearch
name: steamer-elasticsearch
version: 6.8.23
serviceName: steamer-elasticsearch
updateStrategy:
type: OnDelete
template:
metadata:
labels:
app: steamer-elasticsearch
kubernetes.io/cluster-service: "true"
name: steamer-elasticsearch
version: 6.8.23
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: security
operator: In
values:
- linux
topologyKey: beta.kubernetes.io/os
weight: 100
containers:
- env:
- name: NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: steamer_log_stdout
value: "True"
image: system/elasticsearch:6.8.23
imagePullPolicy: IfNotPresent
name: steamer-elasticsearch
ports:
- containerPort: 9200
hostPort: 9200
name: es-http
- containerPort: 9300
hostPort: 9300
name: es-tcp
resources:
limits:
cpu: "2"
requests:
cpu: 100m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /usr/share/elasticsearch/data
name: elasticsearch-data
- mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
name: config
subPath: elasticsearch.yml
- mountPath: /usr/share/elasticsearch/config/jvm.options
name: config1
subPath: jvm.options
dnsPolicy: ClusterFirst
hostNetwork: true
nodeSelector:
kubernetes.io/hostname: 192.168.0.133
zone: master
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: elasticsearch-logging
serviceAccountName: elasticsearch-logging
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /srv/steamer/elasticsearch/data
type: ""
name: elasticsearch-data
- configMap:
defaultMode: 420
items:
- key: elasticsearch1.yml
path: elasticsearch.yml
name: prometheus-es-config
name: config
- configMap:
defaultMode: 420
items:
- key: jvm.options
path: jvm.options
name: prometheus-es-config
name: config1
- 去除掉Nginx容器与Nginx需要的volume定义
- 通过hostPort向主机暴露ES的端口 9200与9300
b. 应用statefulset资源对象
kubectl apply -f es-6.8-remove-nginx.yaml
c. 可能的错误
error: error validating "es-6.8-remove-nginx.yaml": error validating data: [ValidationError(StatefulSet.spec.selector): unknown field "app" in io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector, ValidationError(StatefulSet.spec.selector): unknown field "name" in io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector, ValidationError(StatefulSet.spec.selector): unknown field "version" in io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector]; if you choose to ignore these errors, turn validation off with --validate=false
# 在template中的selector中没有指定标签的匹配方式,需要指定matchLables
The StatefulSet "steamer-elasticsearch-1" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'replicas', and 'updateStrategy' are forbidden
# 因为在应该同名且相同命名空间中的statefulset时,默认只允许修改 replicas replicas updateStrategy字段,如果需要修改则先删除原来的资源,然后重新应用
e. 检查
# 1. 检查es对应的statefulset资源是否创建并且状态
kubectl get statefulset -A |grep elas
# 2. 检查对应的pod是否创建
kubectl get pods -A|grep elas
# 3. 检查是否可以通过节点主机访问ES
curl 192.168.0.133:9200/_cat/indices?v
# 可能错误:远程主机拒绝访问,原因是ES未指定绑定网口,需要修改ES配置文件
# 4. 检查是否有数据写入
curl 192.168.0.133:9200/filebeat-2022.01.25/_count
# 如果返回中的count参数是递增的,则能正常写入
f. 修改ES配置资源
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-es-config
namespace: kube-system
data:
elasticsearch1.yml: |-
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
http.cors.allow-headers: Authorization
http.cors.enabled: true
http.cors.allow-origin: "*"
cluster.name: prometheus-elasticsearch
node.name: node-singleton
node.master: true
http.host: 0.0.0.0
http.bind_host: 0.0.0.0
http.port: 9200
transport.host: 0.0.0.0
transport.bind_host: 0.0.0.0
transport.tcp.port: 9300
indices.memory.index_buffer_size: 15%
thread_pool.bulk.queue_size: 1024
jvm.options: |-
-Xms2g
-Xmx2g
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30
-Des.networkaddress.cache.ttl=60
-Des.networkaddress.cache.negative.ttl=10
-XX:+AlwaysPreTouch
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
14-:-XX:+ShowCodeDetailsInExceptionMessages
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j2.formatMsgNoLookups=true
-Djava.io.tmpdir=${ES_TMPDIR}
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=data
-XX:ErrorFile=logs/hs_err_pid%p.log
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
9-:-Djava.locale.providers=COMPAT
- bind_host 绑定主机上所有网口 0.0.0.0
g. 配置权限认证
# 1. 进入ESrongq
kubectl exec -it steamer-elasticsearch-0 -n kube-system /bin/bash
# 2. 交互方式设置命令密码
# 依次设置 elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user 用户的密码
elasticsearch-setup-passwords interactive
# 3. 检查
curl localhost:9200/_cat/indices
# 会发现报以下错误
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/_cat/indices]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [/_cat/indices]","header":{"WWW-Authenticate":"Basic realm=..."}}
# 4. 带认证额访问
curl -u elastic:<password> localhost:9200:9200/_cat/indices
# 能正常返回并会多出一个索引:.security-6
# 5. 退出容器,检查是否通过主机端口访问
curl -u elastic:<password> 192.168.0.133:9200/filebeat-2022.01.25/_count
# 会发现count计算器会递增,说明filebeat能正常写入数据
12. ES源码升级log4j版本后构建镜像包
a. 下载源码并切换指定分支
git clone https://github.com/elastic/elasticsearch
# 系统需要安装git工具 -- 版本控制工具
git branch -v
# -v查询本地仓库缓存的分支,-r查询远程分支,-a显示所有分支
git checkout <tag>
# 切换到指定分支
# 若git方式无法拉取代码,可以下载对应版本的tar.gz包
标签:修复,jar,system,漏洞,XX,elasticsearch,true,log4j
From: https://www.cnblogs.com/2bjiujiu/p/17985862