四层代理service
# 功能：为一组pods上的应用程序公开网络服务的抽象方法，并为这一组pod提供相同的DNS名字，从而实现负载均衡。
# 产生背景：
正常的K8S环境都是集群部署，运行在pod中的某个服务可以在集群内通过POD IP访问
[root@master-worker-node-1 service]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test 1/1 Running 0 20s 10.244.31.42 only-worker-node-3 <none> <none>
[root@master-worker-node-1 service]# curl 10.244.31.42
the version is 1.1.0, this is a nginx server in container, and the ip is 10.244.31.42/32
能够访问的原因是现场集群部署过程中使用了ipip类型的calico网络，etcd会记录每个node上的pod subnet信息，主机上的tunl0网卡直接将流量发送给pod。
[root@master-worker-node-1 service]# ip add show tunl0
5: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
inet 10.244.123.128/32 scope global tunl0
valid_lft forever preferred_lft forever
[root@master-worker-node-1 service]# tcpdump -i tunl0 -p icmp -nnvvv
dropped privs to tcpdump
tcpdump: listening on tunl0, link-type RAW (Raw IP), capture size 262144 bytes
21:30:57.712908 IP (tos 0x0, ttl 64, id 8740, offset 0, flags [DF], proto ICMP (1), length 84)
10.244.123.128 > 10.244.31.42: ICMP echo request, id 5, seq 117, length 64
21:30:57.714115 IP (tos 0x0, ttl 63, id 19327, offset 0, flags [none], proto ICMP (1), length 84)
10.244.31.42 > 10.244.123.128: ICMP echo reply, id 5, seq 117, length 64
但是这类通常我们不会使用这类自主式pod运行业务pod，因为主机关机或者误删除pod，都将导致业务不可访问。哪怕是通过控制器控制pod数量，但pod的删除、重建将导致的ip地址变化同样是服务不可访问。
因此，将pod上的应用程序公开网络服务需要解决两个问题：1、即使pod地址发生变化，业务仍不受影响；2、多个pod提供相同应用服务时，如何进行四成负载。
# K8S集群中的三个IP
# 1、node IP：集群节点的物理网络
[root@master-worker-node-1 service]# ip add show ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 52:54:00:53:50:5f brd ff:ff:ff:ff:ff:ff
inet 192.168.122.89/24 brd 192.168.122.255 scope global noprefixroute ens3
valid_lft forever preferred_lft forever
inet 192.168.122.253/24 scope global secondary ens3
valid_lft forever preferred_lft forever
inet6 fe80::df8:5c2e:6181:c92b/64 scope link noprefixroute
valid_lft forever preferred_lft forever
# 2、pod IP：pod地址
[root@master-worker-node-1 service]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test 1/1 Running 0 17m 10.244.31.42 only-worker-node-3 <none> <none>
# 3、cluster IP：集群IP，或者叫service IP，仅存在iptables或者ipvs规则中
[root@master-worker-node-1 service]# kubectl get service -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
gray-release-service ClusterIP 10.105.60.46 <none> 30080/TCP 30h version=gray-release
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 16d <none>
Service type: NodePort、Cluster IP，集群内集群外的区别，语法区别不大
# service基本语法
[root@master-worker-node-1 service]# cat first-service.yaml
kind: Service
apiVersion: v1
metadata:
name: test-service
spec:
selector:
func: test-service
type: NodePort # ClusterIP、ExternalName、NodePort、LoadBalancer
ports:
- name: test
port: 31111 # port that will be exposed by this service.
protocol: TCP # TCP UDP SCTP
nodePort: 32333 # The port on each node on which this service is exposed when type is NodePort or LoadBalancer. Usually assigned by the system.
targetPort: 80 # Number or name of the port to access on the pods targeted by the service.
# 通过deployment创建pod
[root@master-worker-node-1 service]# kubectl get deployment -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
nginx-deploy 3/3 3 3 5m24s nginx-test nginx:stable-alpine-perl func=test-service
[root@master-worker-node-1 service]# kubectl get pods --show-labels -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS
nginx-deploy-76cc5c5b-kjwxn 1/1 Running 0 3m51s 10.244.31.44 only-worker-node-3 <none> <none> func=test-service,pod-template-hash=76cc5c5b
nginx-deploy-76cc5c5b-q6dt7 1/1 Running 0 3m51s 10.244.31.46 only-worker-node-3 <none> <none> func=test-service,pod-template-hash=76cc5c5b
nginx-deploy-76cc5c5b-rzj96 1/1 Running 0 3m51s 10.244.54.50 only-worker-node-4 <none> <none> func=test-service,pod-template-hash=76cc5c5b
# 创建service
[root@master-worker-node-1 service]# kubectl get service test-service -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
test-service NodePort 10.105.161.8 <none> 31111:31123/TCP 2m54s func=test-service
# 创建service的时候会同步创建一个同名的endpoints资源
[root@master-worker-node-1 service]# kubectl get endpoints test-service
NAME ENDPOINTS AGE
test-service 10.244.31.44:80,10.244.31.46:80,10.244.54.50:80 3h26m
# service和pod自动关联
[root@master-worker-node-1 service]# kubectl describe service test-service
Name: test-service
Namespace: default
Labels: <none>
Annotations: <none>
Selector: func=test-service
Type: NodePort # 通过访问每个节点的物理网络都可以访问到pod内的应用资源
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.105.161.8 # service的地址
IPs: 10.105.161.8
Port: test 31111/TCP # 将service的31111端口映射到内部pod的80端口
TargetPort: 80/TCP # 表示内部pod暴露的端口
NodePort: test 31123/TCP # 将service的31111端口和每个集群中的物理节点31123端口绑定
Endpoints: 10.244.31.44:80,10.244.31.46:80,10.244.54.50:80 # 标签选择器对应上的pod ip
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
# 访问service IP的地址+端口可以访问内部资源
[root@master-worker-node-1 service]# curl 10.105.161.8:31111
the version is 1.1.0, this is a nginx server in container, and the ip is 10.244.54.50/32
[root@master-worker-node-1 service]# curl 10.105.161.8:31111
the version is 1.1.0, this is a nginx server in container, and the ip is 10.244.31.46/32
[root@master-worker-node-1 service]# curl 10.105.161.8:31111
the version is 1.1.0, this is a nginx server in container, and the ip is 10.244.31.44/32
# 在集群中任意一个节点上，访问其物理网络的31123端口也能访问到pod内部资源
[root@master-worker-node-2 ~]# ip add show ens3 | grep 192
inet 192.168.122.106/24 brd 192.168.122.255 scope global noprefixroute ens3
[root@master-worker-node-2 ~]# curl 192.168.122.106:31123
the version is 1.1.0, this is a nginx server in container, and the ip is 10.244.54.50/32
[root@master-worker-node-1 service]# ip add show ens3 | grep 192
inet 192.168.122.89/24 brd 192.168.122.255 scope global noprefixroute ens3
inet 192.168.122.253/24 scope global secondary ens3
[root@master-worker-node-1 service]# curl 192.168.122.89:31123
the version is 1.1.0, this is a nginx server in container, and the ip is 10.244.31.46/32
# 上述环境中，能访问的原因与ipvs规则有关
[root@master-worker-node-1 manifests]# kubectl get configmaps -n kube-system -o yaml | grep -E "ipvs|Proxy"
ipvs:
kind: KubeProxyConfiguration
mode: ipvs
# 每一个节点上都有ipvs规则
[root@master-worker-node-1 manifests]# ipvsadm -Ln | grep 31123 -A 3
TCP 172.17.0.1:31123 rr
-> 10.244.31.44:80 Masq 1 0 0 # rr表示轮询
-> 10.244.31.46:80 Masq 1 0 0
-> 10.244.54.50:80 Masq 1 0 0
TCP 192.168.122.89:31123 rr
-> 10.244.31.44:80 Masq 1 0 0
-> 10.244.31.46:80 Masq 1 0 0
-> 10.244.54.50:80 Masq 1 0 0
TCP 192.168.122.253:31123 rr
-> 10.244.31.44:80 Masq 1 0 0
-> 10.244.31.46:80 Masq 1 0 0
-> 10.244.54.50:80 Masq 1 0 0
--
TCP 10.244.123.128:31123 rr
-> 10.244.31.44:80 Masq 1 0 0
-> 10.244.31.46:80 Masq 1 0 0
-> 10.244.54.50:80 Masq 1 0 0
Service type: ExternalName
# 背景：默认情况下，访问同名空间下的service仅需要访问service-name就行，而一个命名空间下的pod想要其他命名空间下的service，需要使用完整域名才行。
# 完整域名: service-name.namespace-name.svc.cluster.local
[root@master-worker-node-1 ~]# kubectl get svc -o wide -A | grep -E "NAME|service"
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default service-default ClusterIP 10.109.121.131 <none> 30000/TCP 7h36m func=test-service
ns-1 service-ns-1 ClusterIP 10.103.47.187 <none> 30000/TCP 7h25m func=test-service
ns-2 service-ns-2 ClusterIP 10.103.149.111 <none> 30000/TCP 7h24m func=test-service
[root@master-worker-node-1 ~]# kubectl exec -it -n ns-1 centos -- bash
[root@centos /]# cat /etc/resolv.conf
search ns-1.svc.cluster.local svc.cluster.local cluster.local # 域名
nameserver 10.96.0.10
options ndots:5
[root@centos /]# curl service-ns-1:30000
the version is 1.1.0, this is a nginx server in container, and the ip is 10.244.31.55/32
[root@centos /]# curl service-ns-2:30000
curl: (6) Could not resolve host: service-ns-2
[root@centos /]# curl service-ns-2.ns-2.svc.cluster.local:30000
the version is 1.1.0, this is a nginx server in container, and the ip is 10.244.54.57/32
[root@centos /]# host service-ns-1
service-ns-1.ns-1.svc.cluster.local has address 10.103.47.187
[root@centos /]# host service-ns-2
Host service-ns-2 not found: 3(NXDOMAIN)
[root@centos /]# curl 10.103.109.121.131:30000
curl: (6) Could not resolve host: 10.103.109.121.131
[root@centos /]# curl 10.109.121.131:30000
the version is 1.1.0, this is a nginx server in container, and the ip is 10.244.31.52/32
# 上面的例子看到，虽然访问短域名不行，但是可以通过完整域名和clusterip进行访问，那为什么还需要external name呢？
# 我理解这就是因为1、service IP可能因为集群环境的重启的发生变化，而且地址记忆没有域名清晰；2、带service-name和namespace-name的域名不好记忆，可以通过命名别名一样完成配置。方便维护
# 在上面的例子中，ns-1中的POD想要访问ns-2中的service，就可以将ns-2的完整域名配置一个软连接，external Name
# externalName配置文件
[root@master-worker-node-1 service]# cat external-name.yaml
apiVersion: v1
kind: Service
metadata:
name: test-external-name
namespace: ns-1
spec:
type: ExternalName
externalName: service-ns-2.ns-2.svc.cluster.local
[root@master-worker-node-1 service]# kubectl apply -f external-name.yaml
service/test-external-name created
[root@master-worker-node-1 service]# kubectl get service -o wide -A | grep service
default service-default ClusterIP 10.109.121.131 <none> 30000/TCP 8h func=test-service
ns-1 service-ns-1 ClusterIP 10.103.47.187 <none> 30000/TCP 7h49m func=test-service
ns-1 test-external-name ExternalName <none> service-ns-2.ns-2.svc.cluster.local <none> 6m10s <none>
ns-2 service-ns-2 ClusterIP 10.103.149.111 <none> 30000/TCP 7h49m func=test-service
[root@master-worker-node-1 service]# kubectl exec -it centos -n ns-1 -- bash
[root@centos /]# host test-external-name
test-external-name.ns-1.svc.cluster.local is an alias for service-ns-2.ns-2.svc.cluster.local.
service-ns-2.ns-2.svc.cluster.local has address 10.103.149.111
[root@centos /]# ping test-external-name -c 2
PING service-ns-2.ns-2.svc.cluster.local (10.103.149.111) 56(84) bytes of data.
64 bytes from service-ns-2.ns-2.svc.cluster.local (10.103.149.111): icmp_seq=1 ttl=64 time=0.129 ms
64 bytes from service-ns-2.ns-2.svc.cluster.local (10.103.149.111): icmp_seq=2 ttl=64 time=0.201 ms
--- service-ns-2.ns-2.svc.cluster.local ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.129/0.165/0.201/0.036 ms
[root@centos /]# curl test-external-name:30000
the version is 1.1.0, this is a nginx server in container, and the ip is 10.244.31.56/32
# 感觉像是访问自己命名空间下的资源一样。
小结
1、k8s中有三种ip，node IP、Pod IP和Cluster IP；
2、service的类型有四种：
cluster ip：只能在集群内访问；means a service will only be accessible inside the cluster, via the cluster IP
nodePort: 可以在集群外访问，也可以在集群内访问。means a service will be exposed on one port of every node, in addition to 'ClusterIP' type.
ExternalName：means a service consists of only a reference to an external name that kubedns or equivalent will return as a CNAME record, with no exposing or proxying of any pods involved.
LoadBalancer: means a service will be exposed via an external load balancer (if the cloud provider supports it), in addition to 'NodePort' type.
3、k8s中的namespace没有网络隔离功能，不能的ns只能算是不同的项目，不是内核态的namespace。因此，不同ns下的pod默认是互通的，pod也能访问其他ns下的资源。但是访问需要使用IP地址或者完整域名。此时为了方便，可以创建externalName的链接来实现跨namespace的互访。