本地k8s集群pod和办公网互通并实现域名解析
背景
在微服务场景下，研发团队在进行开发联调测试过程中，需要连接到k8s集群的注册中心中等中间件并和集群内的其他业务服务pod互相通信。
网络基本情况
| 网络类型 | ip段 |
|---|---|
| 办公网络 | 172.16.0.0/16 |
| Pod网络 | 10.233.64.0/18 |
| Service网络 | 10.233.0.0/18 |
实施方案
- 添加一台主机，打上污点，禁止调度，专门用于路由转发
# 开启该主机的路由转发功能
[root@nfs bin]# sysctl -a |grep ip_forward
net.ipv4.ip_forward = 1
# 如果该值不为1则需修改
vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
# 生效，如需稳妥可以重启服务器
sysctl -p
# 查看node状态
[root@master ~]# kubectl describe node node4
Name: node4
Roles: worker
...
Taints: <none>
Unschedulable: false
...
# 打污点
kubectl taint nodes node4 node-role.kubernetes.io/master=:NoSchedule
# 再次查看node状态
[root@master ~]# kubectl describe node node4
Name: node4
Roles: worker
...
node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: 172.16.50.146/23
projectcalico.org/IPv4IPIPTunnelAddr: 10.233.105.0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 23 Aug 2023 11:20:08 +0800
Taints: node-role.kubernetes.io/master:NoSchedule
Unschedulable: false
...
# node4配置iptables进行转发
iptables -t nat -A POSTROUTING -s 172.16.0.0/16 -d 10.233.64.0/18 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 172.16.0.0/16 -d 10.233.0.0/18 -j MASQUERADE
# 检查是iptable链情况
[root@node4 bin]# iptables -t nat -L -n --line-numbers | grep -A 10 "Chain POSTROUTING"
Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 cali-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 /* cali:O3lYWMrLQYEMJtB5 */
2 KUBE-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
3 MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
4 MASQUERADE all -- 172.16.0.0/16 10.233.64.0/18
5 MASQUERADE all -- 172.16.0.0/16 10.233.0.0/18
# 如有错误可以删除
iptables -t nat -D POSTROUTING 3
- 三层交换机配置路由,将pod和service下一跳指向到node3节点
# 添加路由
sys
ip route-static 10.233.0.0 18 172.16.50.146
ip route-static 10.233.64.0 18 172.16.50.146
# 查看路由
display ip routing-table
...
Destination/Mask Proto Pre Cost NextHop Interface
0.0.0.0/0 Static 60 0 172.16.0.1 ...
0.0.0.0/32 Direct 0 0 127.0.0.1 ...
10.233.0.0/18 Static 60 0 172.16.50.146 ... # pod路由
10.233.64.0/18 Static 60 0 172.16.50.146 ... # service路由
# 保存配置
save
# 本地电脑测试连通性
xxxdeiMac:~ xxx$ ping 10.233.0.3
PING 10.233.0.3 (10.233.0.3): 56 data bytes
64 bytes from 10.233.0.3: icmp_seq=0 ttl=63 time=0.460 ms
64 bytes from 10.233.0.3: icmp_seq=1 ttl=63 time=0.481 ms
64 bytes from 10.233.0.3: icmp_seq=2 ttl=63 time=0.453 ms
64 bytes from 10.233.0.3: icmp_seq=3 ttl=63 time=0.464 ms
- 集群内域名解析集群外使用
这里只需要拿到coredns的service IP，配置到个人电脑上即可以进行正常解析。如果公司内部有搭建dns服务器，可以直接将cluster.local forword到coredns service ip上，也能实现。
kubectl get svc coredns -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
coredns ClusterIP 10.233.0.3 <none> 53/UDP,53/TCP,9153/TCP 279d
# 测试解析
iMac:~ xxx$ dig A default-http-backend.kubesphere-controls-system.svc.cluster.local
; <<>> DiG 9.10.6 <<>> A default-http-backend.kubesphere-controls-system.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14176
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;default-http-backend.kubesphere-controls-system.svc.cluster.local. IN A
;; ANSWER SECTION:
default-http-backend.kubesphere-controls-system.svc.cluster.local. 30 IN A 10.233.21.176 # 此处可以看到，已经解析成功
;; Query time: 1 msec
;; SERVER: 10.233.0.3#53(10.233.0.3)
;; WHEN: Thu Aug 24 18:03:27 CST 2023
;; MSG SIZE rcvd: 175