kubernetes的搭建（一）

标签：systemd service kubernetes firewalld etc master root 搭建

集群的搭建

集群的类型

kubunetes的集群类型大致上分为两类：一主多从和多主多从。

一主多从：一台master节点和多台node节点，搭建简单，但是有单机故障的风险，适用于测试环境
多主多从：多台master节点和多台node节点，搭建麻烦，安全性高，适用于生产环境

为了测试简单，本次搭建的是：一主两从

安装方式

kubernetes有多种部署方式，目前主流的方式有如下：

minikube: 一个用于快速搭建单节点kubernetes的工具
kubeadm：一个用于快速搭建kubernetes集群的工具
二进制包：从官网下载每个组件的二进制包，依次去安装，次方式对与理解kubernetes组件更加有效

安装前准备

主机规划：

作用	ip	操作系统	配置
master	192.168.109.100	centos7 基础设施服务器	2 cpu 2G内存 50G硬盘
node1	192.168.109.101	centos7 基础设施服务器	2 cpu 2G内存 50G硬盘
node2	192.168.109.102	centos7 基础设施服务器	2 cpu 2G内存 50G硬盘

搭建基础环境

配置网卡ip和yum源

查看系统版本

[root@master ~]# cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core)
[root@node1 ~]# cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core) 
[root@node2 ~]# cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core)

配置域名解析

master

[root@master ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.109.100 master
192.168.109.101 node1
192.168.109.102 node2

node1

[root@node1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.109.100 master
192.168.109.101 node1
192.168.109.102 node2

node2

[root@node2 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.109.100 master
192.168.109.101 node1
192.168.109.102 node2

配置时间同步
启动chronyd服务并且设置开机自启

master

[root@master ~]# systemctl start  chronyd
[root@master ~]# systemctl enable chronyd

node1

[root@node1 ~]# systemctl start chronyd
[root@node1 ~]# systemctl enable chronyd

node2

[root@node2 ~]# systemctl start chronyd
[root@node2 ~]# 
[root@node2 ~]# systemctl enable chronyd

禁用iptable和firewalld服务
1.关闭firewalld

master

[root@master ~]# systemctl enable chronyd
[root@master ~]# systemctl stop firewalld.service   
[root@master ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: inactive (dead) since 三 2024-05-01 14:52:19 CST; 51s ago
Docs: man:firewalld(1)
Process: 717 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)
Main PID: 717 (code=exited, status=0/SUCCESS)

5月 01 13:51:29 master systemd[1]: Starting firewalld - dynamic fi....
月 01 13:51:30 master systemd[1]: Started firewalld - dynamic fir....
月 01 14:52:16 master systemd[1]: Stopping firewalld - dynamic fi....
5月 01 14:52:19 master systemd[1]: Stopped firewalld - dynamic fir....
Hint: Some lines were ellipsized, use -l to show in full.
[root@master ~]# systemctl disable firewalld.service 
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

node1

[root@node1 ~]# systemctl stop firewalld.service 
[root@node1 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: inactive (dead) since 三 2024-05-01 14:52:20 CST; 51s ago
 Docs: man:firewalld(1)
Process: 724 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)
Main PID: 724 (code=exited, status=0/SUCCESS)

5月 01 13:51:37 master systemd[1]: Starting firewalld - dynamic firewall daemon...
5月 01 13:51:37 master systemd[1]: Started firewalld - dynamic firewall daemon.
5月 01 14:52:16 node1 systemd[1]: Stopping firewalld - dynamic firewall daemon...
5月 01 14:52:20 node1 systemd[1]: Stopped firewalld - dynamic firewall daemon.
[root@node1 ~]# systemctl disable firewalld.service 
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

node2

[root@node2 ~]# systemctl stop firewalld.service 
[root@node2 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: inactive (dead) since 三 2024-05-01 14:52:19 CST; 52s ago
 Docs: man:firewalld(1)
Process: 724 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)
Main PID: 724 (code=exited, status=0/SUCCESS)

5月 01 13:51:40 master systemd[1]: Starting firewalld - dynamic firewall daemon...
5月 01 13:51:41 master systemd[1]: Started firewalld - dynamic firewall daemon.
5月 01 14:52:16 node2 systemd[1]: Stopping firewalld - dynamic firewall daemon...
5月 01 14:52:19 node2 systemd[1]: Stopped firewalld - dynamic firewall daemon.
[root@node2 ~]# systemctl disable firewalld.service 
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

关闭iptables
当前虚拟机并没有iptables服务，所以不用关闭
比如：

[root@master ~]# systemctl stop  iptables
Failed to stop iptables.service: Unit iptables.service not loaded.

禁用selinux

master

[root@master ~]# sed -i 's/'SELINUX=enforcing'/'SELINUX=disabled/g'' /etc/selinux/config

node1

[root@node1 ~]# sed -i 's/'SELINUX=enforcing'/'SELINUX=disabled/g'' /etc/selinux/config

node2

[root@node2 ~]# sed -i 's/'SELINUX=enforcing'/'SELINUX=disabled/g'' /etc/selinux/config

禁止使用swap分区
kubernetes不支持swap分区，所以需要禁止使用swap分区
直接将/etc/fstab文件中的swap相关配置注释掉

master

[root@master ~]# cat /etc/fstab 
#
# /etc/fstab
# Created by anaconda on Wed May  1 21:37:04 2024
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=d97390d4-c671-411c-9371-015002de02a5 /boot                   xfs     defaults        0 0
#/dev/mapper/centos-swap swap                    swap    defaults        0 0
[root@master ~]#

node1

[root@node1 ~]# cat /etc/fstab 
#
# /etc/fstab
# Created by anaconda on Wed May  1 21:37:04 2024
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=d97390d4-c671-411c-9371-015002de02a5 /boot                   xfs     defaults        0 0
#/dev/mapper/centos-swap swap                    swap    defaults        0 0

node2

[root@node2 ~]# cat /etc/fstab 
#
# /etc/fstab
# Created by anaconda on Wed May  1 21:37:04 2024
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=d97390d4-c671-411c-9371-015002de02a5 /boot                   xfs     defaults        0 0
#/dev/mapper/centos-swap swap                    swap    defaults        0 0

修改linux内核参数,添加网桥过滤和地址转发功能

master

[root@master ~]# cat /etc/  sysctl.d/kubernetes.conf 
net.bridge. bridge-nf-call-ip6tables     = 1
net.bridge. bridge-nf-call-iptables =    1
net.ipv4.ip_forward = 1
[root@master ~]#

node1

[root@node1 ~]# cat /etc/   sysctl.d/kubernetes.conf 
net.bridge. bridge-nf-call-ip6tables     = 1
net.bridge. bridge-nf-call-iptables =    1
net.ipv4.ip_forward = 1

node2

[root@node2 ~]# cat /etc/   sysctl.d/kubernetes.conf 
net.bridge. bridge-nf-call-ip6tables     = 1
net.bridge. bridge-nf-call-iptables =    1
net.ipv4.ip_forward = 1

重新加载配置和加载网桥过滤模块

master

[root@master ~]#       sysctl -p
[root@master ~]#       modprobe br_netfilter

node1

[root@node1 ~]#        sysctl -p
[root@node1 ~]#        modprobe           br_netfilter

node2

[root@node2 ~]#        sysctl -p
[root@node2 ~]#        modprobe           br_netfilter

查看是否加载成功,使用如下命令
```
lsmod |grep br_netfil
```

配置ipvs功能
在kubernetes中service有两种代理模型，一种是基于iptables的，一种是基于ipvs的。两者比较的话，ipvs的性能，明显要高一些，但是如果要是用它，需要手动载入ipvs模块

安装ipset和ipvsadmin （如果没有指定虚拟机，三台同样的操作）

[root@master ~]# yum -y     install ipset ipvsadmin

添加需要加载的模块写入脚本文件

[root@master ~]# cat /etc/  sysconfig/modules/ipvs.   modules 
#!/bin/bash  
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe --     nf_conntrack_ipv4

设置脚本文件可执行权限

[root@master ~]# chmod  +x /etc/sysconfig/modules/   ipvs.modules

执行脚本文件

[root@master ~]# /bin/  bash /etc/sysconfig/  modules/ipvs.modules

查看是否加载成功

[root@master ~]# lsmod  |   grep -e ip_vs -e   nf_conntrack_ipv4
nf_conntrack_ipv4       15053  0 
nf_defrag_ipv4          12729  1 nf_conntrack_ipv4
ip_vs_sh                12688  0 
ip_vs_wrr               12697  0 
ip_vs_rr                12600  0 
ip_vs                   141432  6 ip_vs_rr,   ip_vs_sh,ip_vs_wrr
nf_conntrack            133053  2 ip_vs,  nf_conntrack_ipv4
libcrc32c               12644  3 xfs,ip_vs,  nf_conntrack

重启虚拟机

reboot

检查swap分区和selinux

[root@master ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           1982         100        1752           9         130        1726
Swap:             0           0           0
[root@master ~]# getenforce 
Disabled
[root@master ~]#

环境搭建（二）

安装docker

切换镜像源

  [root@master yum.repos.d]# wget docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

查看当前镜像源中支持的docker版本

yum list docker-ce --showduplicates

安装指定版本的docker
ps: 安装指定版本的docker必须使用--setopt=obsoletes=1选项，否则yum会自动安装更高版本
```
[root@master yum.repos.d]   # yum install  --setopt=obsoletes=0     docker-ce-18.06.2.ce-3. el7 -y
```

添加一个配置文件,设置docker开机自启
docker在默认情况下使用cgroupdiver为cgroupfs，而kubernetes推荐使用systemd来代替cgroupfs，所以需要修改配置文件

[root@master ~]# mkdir /    etc/docker
[root@master ~]# vim /etc/  docker/daemon.json
[root@master ~]#    systemctl restart docker.  service 
[root@master ~]#    systemctl enable docker
Created symlink from /etc/  systemd/system/multi-user.    target.wants/docker.    service to /usr/lib/    systemd/system/docker.  service.
[root@master ~]# docker     --version
Docker version 18.06.   2-ce, build 6d37f41
[root@master ~]#

安装kubernetes

配置yum源

[root@master yum.repos.d]   # cat k8s.repo 
[kubernetes]
name=kubernetes
baseurl=http://mirrors.   aliyun.com/kubernetes/ yum/repos/   kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.    aliyun.com/kubernetes/  yum/doc/yum-key.gpg
       http://mirrors.    aliyun.com/ kubernetes/yum/  doc/  rpm-package-key.  gpg
 //yum仓库的配置文件一定要顶格写

安装kubelet、kubeadm、kubectl

[root@master yum.repos.d]# yum -y install --setopt=obsoletes=0 kubeadm-1.17.4-0 kubelet-1.17.4-0 kubectl-1.17.4-0 -y

配置kubelet的cgroup

[root@master yum.repos.d]# cat /etc/sysconfig/kubelet 
KUBELET_CGROUP_ARGS="--cgroup-drive=systemd"
KUBE_PROXT_MODE="ipvs"
//手动指定使用ipvs代理

设置kubelet开机自启

[root@master yum.repos.d]# systemctl enable kubelet.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@master yum.repos.d]#  //后面配置集群启动的时候，会自动启动kubelet，不需要手动启动

准备集群镜像

#在安装集群之前，必须提前准备好集群需要的镜像，所需镜像可以使用·kubeadm config images list·命令查看
# 下载镜像
# 次镜像在k8s的仓库中，由于网络问题，无法连接，所以尝试使用aliyun仓库中的镜像。然后改名成k8s的镜像，最后在删除阿里云的镜像
images=(
    kube-apiserver:v1.17.4
    kube-controller-manager:v1.17.4
    kube-scheduler:v1.17.4
    kube-proxy:v1.17.4
    pause:3.1
    etcd:3.4.3-0
    coredns:1.6.5
)

for imagename in ${images[@]}; do
      docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imagename
      docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imagename   k8s.gcr.io/$imagename
      docker  rmi  registry.cn-hangzhou.aliyuncs.com/google_containers/$imagename
done
docker images

docker images 查看出来7个镜像

[root@master yum.repos.d]# docker images  
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                v1.17.4             6dec7cfde1e5        4 years ago         116MB
k8s.gcr.io/kube-apiserver            v1.17.4             2e1ba57fe95a        4 years ago         171MB
k8s.gcr.io/kube-controller-manager   v1.17.4             7f997fcf3e94        4 years ago         161MB
k8s.gcr.io/kube-scheduler            v1.17.4             5db16c1c7aff        4 years ago         94.4MB
k8s.gcr.io/coredns                   1.6.5               70f311871ae1        4 years ago         41.6MB
k8s.gcr.io/etcd                      3.4.3-0             303ce5db0e90        4 years ago         288MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        6 years ago         742kB
[root@master yum.repos.d]#

从这里开始只使用master节点

集群初始化

初始化，只需要在master上执行

kubeadm  init \
    --kubernetes-version=v1.17.4 \
    --pod-network-cidr=10.24 4.0.0/16 \
    --service-cidr=10.96.0.0/12 \
    --apiserver-advertise-address=192.168.200.128

看到successful表示成功

如果遇到报错，需要再次执行初始化，删除掉之前生成的重复文件，关闭服务。再次init执行
配置kubectl

 wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
 kubectl apply -f  kube-flannel.yml
 （kubectl delete -f kube-flannel.yml    删除）

- 测试（nginx）

kubectl create deployment nginx --image=nginx:1.14-alpine
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod
kubectl get service
curl http://192.168.109.101:31775

标签：systemd,service,kubernetes,firewalld,etc,master,root,搭建
From： https://www.cnblogs.com/humlogs/p/18169073