欢迎关注我的公众号:
目前刚开始写一个月,一共写了18篇原创文章,文章目录如下:
istio多集群探秘,部署了50次多集群后我得出的结论
istio防故障利器,你知道几个,istio新手不要读,太难!
不懂envoyfilter也敢说精通istio系列-http-rbac-不要只会用AuthorizationPolicy配置权限
不懂envoyfilter也敢说精通istio系列-02-http-corsFilter-不要只会vs
不懂envoyfilter也敢说精通istio系列-03-http-csrf filter-再也不用再代码里写csrf逻辑了
不懂envoyfilter也敢说精通istio系列http-jwt_authn-不要只会RequestAuthorization
不懂envoyfilter也敢说精通istio系列-05-fault-filter-故障注入不止是vs
不懂envoyfilter也敢说精通istio系列-06-http-match-配置路由不只是vs
不懂envoyfilter也敢说精通istio系列-07-负载均衡配置不止是dr
不懂envoyfilter也敢说精通istio系列-08-连接池和断路器
不懂envoyfilter也敢说精通istio系列-09-http-route filter
不懂envoyfilter也敢说精通istio系列-network filter-redis proxy
不懂envoyfilter也敢说精通istio系列-network filter-HttpConnectionManager
不懂envoyfilter也敢说精通istio系列-ratelimit-istio ratelimit完全手册
学习目标
什么是ServiceEntry
使用服务条目资源(Service Entries)可以将条目添加到 Istio 内部维护的服务注册表中。添加服务条目后,Envoy 代理可以将流量发送到该服务,就好像该服务条目是网格中的服务一样。通过配置服务条目,可以管理在网格外部运行的服务的流量。
此外,可以配置虚拟服务和目标规则,以更精细的方式控制到服务条目的流量,就像为网格中的其他任何服务配置流量一样。
资源详解
Field | Type | Description | Required |
| | The hosts associated with the ServiceEntry. Could be a DNS name with wildcard prefix.The hosts field is used to select matching hosts in VirtualServices and DestinationRules.For HTTP traffic the HTTP Host/Authority header will be matched against the hosts field.For HTTPs or TLS traffic containing Server Name Indication (SNI), the SNI value will be matched against the hosts field.NOTE 1: When resolution is set to type DNS and no endpoints are specified, the host field will be used as the DNS name of the endpoint to route traffic to.NOTE 2: If the hostname matches with the name of a service from another service registry such as Kubernetes that also supplies its own set of endpoints, the ServiceEntry will be treated as a decorator of the existing Kubernetes service. Properties in the service entry will be added to the Kubernetes service if applicable. Currently, the only the following additional properties will be considered by | Yes |
| | The virtual IP addresses associated with the service. Could be CIDR prefix. For HTTP traffic, generated route configurations will include http route domains for both the | No |
| | The ports associated with the external service. If the Endpoints are Unix domain socket addresses, there must be exactly one port. | Yes |
| | Specify whether the service should be considered external to the mesh or part of the mesh. | No |
| | Service discovery mode for the hosts. Care must be taken when setting the resolution mode to NONE for a TCP port without accompanying IP addresses. In such cases, traffic to any IP on said port will be allowed (i.e. | Yes |
| | One or more endpoints associated with the service. Only one of | No |
| | Applicable only for MESH_INTERNAL services. Only one of | No |
| | A list of namespaces to which this service is exported. Exporting a service allows it to be used by sidecars, gateways and virtual services defined in other namespaces. This feature provides a mechanism for service owners and mesh administrators to control the visibility of services across namespace boundaries.If no namespaces are specified then the service is exported to all namespaces by default.The value “.” is reserved and defines an export to the same namespace that the service is declared in. Similarly the value “*” is reserved and defines an export to all namespaces.For a Kubernetes Service, the equivalent effect can be achieved by setting the annotation “networking.istio.io/exportTo” to a comma-separated list of namespace names. | No |
| | If specified, the proxy will verify that the server certificate’s subject alternate name matches one of the specified values.NOTE: When using the workloadEntry with workloadSelectors, the service account specified in the workloadEntry will also be used to derive the additional subject alternate names that should be verified. | No |
exportTo
1当前名称空间
1部署sleep
kubectl apply -f samples/sleep/sleep.yaml -n istio
2修改默认访问策略
mesh下面
outboundTrafficPolicy:
mode: REGISTRY_ONLY
重启pod istiod使之生效
2应用serviceentry
serviceentries/se-baidu-dot.yaml
kubectl apply -f se-baidu-dot.yaml -n istio
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: baidu
spec:
exportTo:
- "."
hosts:
- "www.baidu.com"
ports:
- number: 80
name: http
protocol: HTTP
location: MESH_EXTERNAL
resolution: DNS
2名称空间
serviceentries/se-baidu-namespace.yaml
kubectl apply -f se-baidu-namespace.yaml -n istio
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: baidu
spec:
exportTo:
- "istio-system"
hosts:
- "www.baidu.com"
ports:
- number: 80
name: http
protocol: HTTP
location: MESH_EXTERNAL
resolution: DNS
修改名称空间为istio,再测试
3 所有名称空间
serviceentries/se-baidu-star.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: baidu
spec:
exportTo:
- "*"
hosts:
- "www.baidu.com"
ports:
- number: 80
name: http
protocol: HTTP
location: MESH_EXTERNAL
resolution: DNS
hosts
serviceentries/se-baidu-hosts.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: baidu
spec:
hosts:
- "www.baidu.com"
- "www.csdn.net"
ports:
- number: 80
name: http
protocol: HTTP
location: MESH_EXTERNAL
resolution: DNS
resolution
DNS
serviceentries/se-baidu-resolution-dns.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: baidu
spec:
hosts:
- "www.baidu.com"
ports:
- number: 80
name: http
protocol: HTTP
location: MESH_EXTERNAL
resolution: DNS
STATIC
mongodb-se-resolution-static.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: mongodb-se
spec:
hosts:
- mymongodb.demo
addresses:
- "192.168.198.158/32"
ports:
- number: 27017
name: mongodb
protocol: MONGO
location: MESH_EXTERNAL
resolution: STATIC
endpoints:
- address: 192.168.198.154
se-baidu-resolution-static.yaml
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: baidu
spec:
hosts:
- "www.baidu.com"
ports:
- number: 80
name: http
protocol: HTTP
location: MESH_EXTERNAL
resolution: STATIC
endpoints:
- address: 36.152.44.96
NONE
se-baidu-resolution-none.yaml
配置静态dns
kubectl edit cm coredns -n kube-system
hosts { 192.168.198.158 mymongodb.demo 36.152.44.96 www.baidu.com fallthrough }
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: baidu
spec:
hosts:
- www.baidu.com
location: MESH_EXTERNAL
ports:
- number: 80
name: http
protocol: HTTP
resolution: NONE
进入pod访问
kubectl exec -it client-bcd749854-dnkml -n istio -- /bin/sh
wget www.baidu.com
vs dr se联合使用
1部署mongodb
yum install mongodb-org
配置mongodb远程访问
bind 0.0.0.0
启动mongod
systemctl start mongod
2创建se
mongodb-se-resolution-static-multi-ep.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: mongodb-se
spec:
hosts:
- mymongodb.demo
addresses:
- "192.168.198.158/32"
ports:
- number: 27017
name: mongodb
protocol: MONGO
location: MESH_EXTERNAL
resolution: STATIC
endpoints:
- address: 192.168.198.154
- address: 192.168.198.155
3创建vs
vs-mongodb.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: vs-mongodb
spec:
hosts:
- "mymongodb.demo"
tcp:
- route:
- destination:
host: mymongodb.demo
4创建dr
dr-mongodb-random.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: mymongodb
spec:
host: mymongodb.demo
trafficPolicy:
loadBalancer:
simple: RANDOM
5设置coredns静态dns
kubectl get cm -n kube-system coredns -o yaml
hosts { 192.168.198.158 mymongodb.demo fallthrough }
6进入mongodb pod
kubectl exec -it mongodb-v1-64d4666575-6n2dq -n istio -- /bin/bash
7访问
mongo --host mymongodb.demo
或
mongo --host 192.168.198.158
location
Name | Description |
| Signifies that the service is external to the mesh. Typically used to indicate external services consumed through APIs. |
| Signifies that the service is part of the mesh. Typically used to indicate services added explicitly as part of expanding the service mesh to include unmanaged infrastructure (e.g., VMs added to a Kubernetes based service mesh). |
MESH_EXTERNAL
serviceentries/se-baidu-star.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: baidu
spec:
exportTo:
- "*"
hosts:
- "www.baidu.com"
ports:
- number: 80
name: http
protocol: HTTP
location: MESH_EXTERNAL
resolution: DNS
MESH_INTERNAL
se-details-location-internal.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: details-se
spec:
hosts:
- details.bookinfo.com
location: MESH_INTERNAL
ports:
- number: 9080
name: http
protocol: HTTP
resolution: STATIC
workloadSelector:
labels:
app: details
添加静态路由
hosts { 192.168.198.158 mymongodb.demo 36.152.44.96 www.baidu.com 10.68.190.94 details.bookinfo.com fallthrough }
删除client pod
kubectl delete pod client-bcd749854-dnkml -n istio
进入pod
kubectl exec -it client-bcd749854-hs2s7 -n istio -- /bin/sh
wget details.bookinfo.com:9080/details/0
addresses
se-details-adresses.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: details-se
spec:
hosts:
- details.bookinfo.com
addresses:
- 192.168.198.177/32
- 192.168.198.178/32
location: MESH_INTERNAL
ports:
- number: 9080
name: http
protocol: HTTP
resolution: STATIC
workloadSelector:
labels:
app: details
两个address第一个不生效,最后一个生效,改为一个address再试
ports
http端口:
serviceentries/se-baidu-star.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: baidu
spec:
exportTo:
- "*"
hosts:
- "www.baidu.com"
ports:
- number: 80
name: http
protocol: HTTP
location: MESH_EXTERNAL
resolution: DNS
443端口
se-baidu-ports-https.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: baidu
spec:
exportTo:
- "*"
hosts:
- "www.baidu.com"
ports:
- number: 443
name: https
protocol: HTTPS
location: MESH_EXTERNAL
resolution: DNS
se-jd-ports-https.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: jd-api
spec:
hosts:
- api.jd.com
ports:
- number: 443
name: https
protocol: HTTPS
resolution: DNS
kubectl exec -it sleep-557747455f-wqtls -n istio -- /bin/sh
curl 百度一下,你就知道
curl 多快好省,购物上京东!
使用egress
se-cnn.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: cnn
spec:
hosts:
- edition.cnn.com
ports:
- number: 80
name: http-port
protocol: HTTP
- number: 443
name: https
protocol: HTTPS
resolution: DNS
cnn-egressgateway.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: cnn-egressgateway
spec:
selector:
istio: egressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- edition.cnn.com
dr-egressgateway-cnn.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: dr-egressgateway-cnn
spec:
host: istio-egressgateway.istio-system.svc.cluster.local
subsets:
- name: cnn
vs-cnn.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: direct-cnn-through-egress-gateway
spec:
hosts:
- edition.cnn.com
gateways:
- istio-egressgateway
- mesh
http:
- match:
- gateways:
- mesh
port: 80
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
subset: cnn
port:
number: 80
weight: 100
- match:
- gateways:
- istio-egressgateway
port: 80
route:
- destination:
host: edition.cnn.com
port:
number: 80
weight: 100
curl http://edition.cnn.com/politics -I
查看egress日志
kubectl logs istio-egressgateway-bd6d77495-vmhvg -n istio-system -f
endpoints
Field | Type | Description | Required |
| | Address associated with the network endpoint without the port. Domain names can be used if and only if the resolution is set to DNS, and must be fully-qualified without wildcards. Use the form unix:///absolute/path/to/socket for Unix domain socket endpoints. | Yes |
| | Set of ports associated with the endpoint. If the port map is specified, it must be a map of servicePortName to this endpoint’s port, such that traffic to the service port will be forwarded to the endpoint port that maps to the service’s portName. If omitted, and the targetPort is specified as part of the service’s port specification, traffic to the service port will be forwarded to one of the endpoints on the specified | No |
| | One or more labels associated with the endpoint. | No |
| | Network enables Istio to group endpoints resident in the same L3 domain/network. All endpoints in the same network are assumed to be directly reachable from one another. When endpoints in different networks cannot reach each other directly, an Istio Gateway can be used to establish connectivity (usually using the | No |
| | The locality associated with the endpoint. A locality corresponds to a failure domain (e.g., country/region/zone). Arbitrary failure domain hierarchies can be represented by separating each encapsulating failure domain by /. For example, the locality of an an endpoint in US, in US-East-1 region, within availability zone az-1, in data center rack r11 can be represented as us/us-east-1/az-1/r11. Istio will configure the sidecar to route to endpoints within the same locality as the sidecar. If none of the endpoints in the locality are available, endpoints parent locality (but within the same network ID) will be chosen. For example, if there are two endpoints in same network (networkID “n1”), say e1 with locality us/us-east-1/az-1/r11 and e2 with locality us/us-east-1/az-2/r12, a sidecar from us/us-east-1/az-1/r11 locality will prefer e1 from the same locality over e2 from a different locality. Endpoint e2 could be the IP associated with a gateway (that bridges networks n1 and n2), or the IP associated with a standard service endpoint. | No |
| | The load balancing weight associated with the endpoint. Endpoints with higher weights will receive proportionally higher traffic. | No |
| | The service account associated with the workload if a sidecar is present in the workload. The service account must be present in the same namespace as the configuration ( WorkloadEntry or a ServiceEntry) |
https://istio.io/latest/docs/reference/config/networking/workload-entry/#WorkloadEntry
address
mongodb-se-resolution-static-multi-ep.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: mongodb-se
spec:
hosts:
- mymongodb.demo
addresses:
- "192.168.198.158/32"
ports:
- number: 27017
name: mongodb
protocol: MONGO
location: MESH_EXTERNAL
resolution: STATIC
endpoints:
- address: 192.168.198.154
- address: 192.168.198.155
labels
1创建se
endpoints/se-mongodb-labels.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: mongodb-se
spec:
hosts:
- mymongodb.demo
addresses:
- "192.168.198.158/32"
ports:
- number: 27017
name: mongodb
protocol: MONGO
location: MESH_EXTERNAL
resolution: STATIC
endpoints:
- address: 192.168.198.154
labels:
version: v1
- address: 192.168.198.155
labels:
version: v2
2创建vs
endpoints/vs-mongodb-v1.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: vs-mongodb
spec:
hosts:
- "mymongodb.demo"
tcp:
- route:
- destination:
host: mymongodb.demo
subset: v1
3创建dr
endpoints/dr-mongodb.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: mymongodb
spec:
host: mymongodb.demo
trafficPolicy:
loadBalancer:
simple: RANDOM
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
4访问
kubectl exec -it mongodb-v1-64d4666575-6n2dq -n istio -- /bin/bash
mongo --host mymongodb.demo
结果都路由到v1版本
locality
region/zone/subzone
distribute
[root@master01 kube]# kubectl get node --show-labels
NAME STATUS ROLES AGE VERSION LABELS
192.168.198.154 Ready master 22d v1.20.5 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=192.168.198.154,kubernetes.io/os=linux,kubernetes.io/role=master,topology.istio.io/subzone=sz01,topology.kubernetes.io/region=us-central1,topology.kubernetes.io/zone=z1
192.168.198.155 Ready master 22d v1.20.5 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=192.168.198.155,kubernetes.io/os=linux,kubernetes.io/role=master,topology.istio.io/subzone=sz02,topology.kubernetes.io/region=us-central2,topology.kubernetes.io/zone=z2
192.168.198.156 Ready node 22d v1.20.5 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=192.168.198.156,kubernetes.io/os=linux,kubernetes.io/role=node,topology.istio.io/subzone=sz03,topology.kubernetes.io/region=us-central3,topology.kubernetes.io/zone=z3
endpoints/se-mongodb-locality.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: mongodb-se
spec:
hosts:
- mymongodb.demo
addresses:
- "192.168.198.158/32"
ports:
- number: 27017
name: mongodb
protocol: MONGO
location: MESH_EXTERNAL
resolution: STATIC
endpoints:
- address: 192.168.198.154
locality: "us-central1/z1/sz01"
labels:
version: v1
- address: 192.168.198.155
labels:
version: v2
locality: "us-central2/z2/sz02"
topology.kubernetes.io/region=us-central1
topology.kubernetes.io/zone=z1
topology.istio.io/subzone=sz01
endpoints/dr-mongodb-locality.yaml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: dr-mongodb
spec:
host: mymongodb.demo
trafficPolicy:
loadBalancer:
localityLbSetting:
enabled: true
distribute:
- from: "us-central1/z1/*"
to:
#"us-central3/z3/*": 100
"us-central2/z2/*": 100
#"us-central1/z1/*": 100
outlierDetection:
consecutive5xxErrors: 1
interval: 5m
baseEjectionTime: 15m
endpoints/vs-mongodb-locality.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: vs-mongodb
spec:
hosts:
- "mymongodb.demo"
tcp:
- route:
- destination:
host: mymongodb.demo
kubectl exec -it mongodb-v1-64d4666575-hl6br -n istio -- /bin/bash
mongo --host 192.168.198.158
failover
endpoints/dr-mongodb-locality-failover.yaml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: dr-mongodb
spec:
host: mymongodb.demo
trafficPolicy:
loadBalancer:
localityLbSetting:
enabled: true
failover:
- from: us-central1/z1/sz01
to: us-central2/z2/sz02
- from: us-central2/z2/sz02
to: us-central1/z1/sz01
outlierDetection:
consecutive5xxErrors: 1
interval: 1s
baseEjectionTime: 15m
network
endpoints/se-mongodb-network.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: mongodb-se
spec:
hosts:
- mymongodb.demo
addresses:
- "192.168.198.158/32"
ports:
- number: 27017
name: mongodb
protocol: MONGO
location: MESH_EXTERNAL
resolution: STATIC
endpoints:
- address: 192.168.198.154
network: n1
- address: 192.168.198.155
不成功
weight
endpoints/se-mongodb-weight.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: mongodb-se
spec:
hosts:
- mymongodb.demo
addresses:
- "192.168.198.158/32"
ports:
- number: 27017
name: mongodb
protocol: MONGO
location: MESH_EXTERNAL
resolution: STATIC
endpoints:
- address: 192.168.198.154
weight: 10
- address: 192.168.198.155
weight: 90
serviceAccount
endpoints/se-mongodb-serviceaccount.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: mongodb-se
spec:
hosts:
- mymongodb.demo
addresses:
- "192.168.198.158/32"
ports:
- number: 27017
name: mongodb
protocol: MONGO
location: MESH_EXTERNAL
resolution: STATIC
endpoints:
- address: 192.168.198.154
serviceAccount: mongov1
- address: 192.168.198.155
serviceAccount: mongov2
不知道起什么作用
ports
endpoints/se-mongodb-endpoint-ports.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: mongodb-se
spec:
hosts:
- mymongodb.demo
addresses:
- "192.168.198.158/32"
ports:
- number: 27019
name: mongodb
protocol: MONGO
location: MESH_EXTERNAL
resolution: STATIC
endpoints:
- address: 192.168.198.154
ports:
mongodb: 27017
- address: 192.168.198.155
ports:
mongodb: 27017
mongo --host mymongodb.demo --port 27019
subjectAltNames
在default部署details2
details2-deploy.yaml
se-details-subject-alt-names.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: details-se
spec:
hosts:
- details.default.com
addresses:
- 192.168.198.159
location: MESH_INTERNAL
ports:
- number: 9080
name: http
protocol: HTTP
resolution: STATIC
subjectAltNames:
- "aa"
workloadSelector:
labels:
app: default-details
不知道有什么作用
workloadSelector
在default部署details2
details2-deploy.yaml
apiVersion: v1
kind: Service
metadata:
name: details
labels:
app: details
service: details
spec:
ports:
- port: 9080
name: http
selector:
app: details
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: bookinfo-details
labels:
account: details
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: details-v1
labels:
app: default-details
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: default-details
version: v1
template:
metadata:
labels:
app: default-details
version: v1
spec:
serviceAccountName: bookinfo-details
containers:
- name: details
image: docker.io/istio/examples-bookinfo-details-v1:1.16.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
securityContext:
runAsUser: 1000
se-details-workloadSelector.yaml
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: details-se
spec:
hosts:
- details.default.com
addresses:
- 192.168.198.159
location: MESH_INTERNAL
ports:
- number: 9080
name: http
protocol: HTTP
resolution: STATIC
workloadSelector:
labels:
app: default-details
kubectl apply -f se-details-workloadSelector.yaml
标签:ServiceEntry,name,mongodb,istio,192.168,详解,hosts,io From: https://blog.51cto.com/u_11979904/5948167