一、集群信息 192.168.1.5 vm5 master1 etcd 192.168.1.6 vm6 master2 etcd 192.168.1.7 vm7 master3 etcd 192.168.1.8 vm8 node01 证书说明: 二、初始化 # 关闭防火墙 systemctl stop firewalld systemctl disable firewalld # 关闭selinux sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久 setenforce 0 # 临时 # 关闭swap swapoff -a # 临时 sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久 # 根据规划设置主机名 hostnamectl set-hostname <hostname> # 在master添加hosts cat >> /etc/hosts << EOF 192.168.1.5 vm5 master1 192.168.1.6 vm6 master2 192.168.1.7 vm7 master3 192.168.1.8 vm8 node01 EOF # 将桥接的IPv4流量传递到iptables的链 cat > /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system # 生效 # 时间同步 yum install ntpdate -y ntpdate time.windows.com 二、部署ETCD集群 Etcd 是一个分布式键值存储系统,Kubernetes使用Etcd进行数据存储,所以先准备一个Etcd数据库,为解决Etcd单点故障,应采用集群方式部署,这里使用3台组建集群,可容忍1台机器故障,当然,你也可以使用5台组建集群,可容忍2台机器故障。 2.1 准备cfssl证书生成工具 cfssl是一个开源的证书管理工具,使用json文件生成证书,相比openssl更方便使用。 找任意一台服务器操作,这里用Master节点。 wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo 2.2 生成Etcd证书 1. 自签证书颁发机构(CA) 创建工作目录: 自签CA: cd /root/k8s/cert/etcd [root@vm5 etcd]# cat > ca-config.json << EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "www": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF [root@vm5 etcd]# cat > ca-csr.json << EOF { "CN": "etcd CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing" } ] } EOF 生成证书: [root@vm5 etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca - [root@vm5 etcd]# ls *pem ca-key.pem ca.pem 2. 使用自签CA签发Etcd HTTPS证书 创建证书申请文件: [root@vm5 etcd]# cat > server-csr.json << EOF { "CN": "etcd", "hosts": [ "192.168.1.5", "192.168.1.6", "192.168.1.7" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing" } ] } EOF 注:上述文件hosts字段中IP为所有etcd节点的集群内部通信IP,一个都不能少!为了方便后期扩容可以多写几个预留的IP。 生成证书: [root@vm5 etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server [root@vm5 etcd]# ls server*pem server-key.pem server.pem [root@vm5 etcd]# ls ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem server.csr server-csr.json server-key.pem server.pem 2.3、部署Etcd集群 1.下载etcd二进制文件 下载地址:https://github.com/etcd-io/etcd/releases/ cd /root/k8s/etcd;wget https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz 创建工作目录并解压二进制包 [root@vm5 ~]# mkdir -p /opt/etcd/{bin,cfg,ssl} [root@vm5 ~]# tar zxvf etcd-v3.4.9-linux-amd64.tar.gz [root@vm5 ~]# cp etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/ 2.创建etcd配置文件 [root@vm5 ~]# cat > /opt/etcd/cfg/etcd.conf << EOF #[Member] ETCD_NAME="etcd-1" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.1.5:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.1.5:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.5:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.5:2379" ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.1.5:2380,etcd-2=https://192.168.1.6:2380,etcd-3=https://192.168.1.7:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" EOF 说明: ETCD_NAME:节点名称,集群中唯一 ETCD_DATA_DIR:数据目录 ETCD_LISTEN_PEER_URLS:集群通信监听地址 ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址 ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址 ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址 ETCD_INITIAL_CLUSTER:集群节点地址 ETCD_INITIAL_CLUSTER_TOKEN:集群Token ETCD_INITIAL_CLUSTER_STATE:加入集群的当前状态,new是新集群,existing表示加入已有集群事实上 3.systemd管理etcd [root@vm5 ~]# cat > /usr/lib/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=/opt/etcd/cfg/etcd.conf ExecStart=/opt/etcd/bin/etcd \ --cert-file=/opt/etcd/ssl/server.pem \ --key-file=/opt/etcd/ssl/server-key.pem \ --peer-cert-file=/opt/etcd/ssl/server.pem \ --peer-key-file=/opt/etcd/ssl/server-key.pem \ --trusted-ca-file=/opt/etcd/ssl/ca.pem \ --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \ --logger=zap Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF 4.拷贝刚才生成的证书 把刚才生成的证书拷贝到配置文件中的路径: [root@vm5 ~]# cp /root/k8s/cert/etcd/ca*pem /opt/etcd/ssl/ [root@vm5 ~]# cp /root/k8s/cert/etcd/server*pem /opt/etcd/ssl/ 5.启动并设置开机启动 [root@vm5 ~]# systemctl daemon-reload [root@vm5 ~]# systemctl start etcd [root@vm5 ~]# systemctl enable etcd 6.将上面master1所有生成的文件拷贝到master2和master3 [root@vm5 ~]# scp -r /opt/etcd/ master2:/opt/ [root@vm5 ~]# scp /usr/lib/systemd/system/etcd.service master2:/usr/lib/systemd/system/ [root@vm5 ~]# scp -r /opt/etcd/ master3:/opt/ [root@vm5 ~]# scp /usr/lib/systemd/system/etcd.service master3:/usr/lib/systemd/system/ 然后在节点2和节点3分别修改etcd.conf配置文件中的节点名称和当前服务器IP: [root@vm6 ~]# vi /opt/etcd/cfg/etcd.conf #[Member] ETCD_NAME="etcd-2" # 修改此处,节点2改为etcd-2,节点3改为etcd-3 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.1.6:2380" # 修改此处为当前服务器IP ETCD_LISTEN_CLIENT_URLS="https://192.168.1.6:2379" # 修改此处为当前服务器IP #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.6:2380" # 修改此处为当前服务器IP ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.6:2379" # 修改此处为当前服务器IP ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.1.5:2380,etcd-2=https://192.168.1.6:2380,etcd-3=https://192.168.1.7:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new 最后启动etcd并设置开机启动,同上: systemctl daemon-reload systemctl start etcd systemctl enable etcd 7.查看集群状态 [root@vm5 etcd]# ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.1.5:2379,https://192.168.1.6:2379,https://192.168.1.7:2379" endpoint health https://192.168.1.6:2379 is healthy: successfully committed proposal: took = 22.316606ms https://192.168.1.5:2379 is healthy: successfully committed proposal: took = 22.851606ms https://192.168.1.7:2379 is healthy: successfully committed proposal: took = 29.681209ms [root@vm6 etcd]# ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.1.5:2379,https://192.168.1.6:2379,https://192.168.1.7:2379" endpoint health https://192.168.1.7:2379 is healthy: successfully committed proposal: took = 22.986606ms https://192.168.1.6:2379 is healthy: successfully committed proposal: took = 21.665307ms https://192.168.1.5:2379 is healthy: successfully committed proposal: took = 27.629108ms 三、安装docker 下载地址:https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz 3.1 下载解压二进制包 以下在所有master节点、worker操作。采用二进制安装: cd /root/k8s/ wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz tar -xf docker-19.03.9.tgz cp -rf docker/* /usr/bin 3.2 systemd管理docker [root@vm5 ~]# cat > /usr/lib/systemd/system/docker.service << EOF [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service Wants=network-online.target [Service] Type=notify ExecStart=/usr/bin/dockerd ExecReload=/bin/kill -s HUP $MAINPID LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TimeoutStartSec=0 Delegate=yes KillMode=process Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target EOF 3.3 创建配置文件 [root@vm5 ~]# mkdir /etc/docker [root@vm5 ~]# cat > /etc/docker/daemon.json << EOF { "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"] } EOF 阿里云镜像加速器 3.4 启动并设置开机启动 systemctl daemon-reload systemctl start docker systemctl enable docker 四、部署Master 4.1 生成kube-apiserver证书(在任意一个节点操作) 1.自签证书颁发机构(CA) [root@vm5 ~]# cd /root/k8s/cert/k8s [root@vm5 k8s]# cat > ca-config.json << EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF [root@vm5 k8s]# cat > ca-csr.json << EOF { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing", "O": "k8s", "OU": "System" } ] } EOF 生成证书: [root@master1 k8s]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca - [root@master1 k8s]# ls /root/k8s/cert/k8s/*pem ca-key.pem ca.pem 2.使用自签CA签发kube-apiserver HTTPS证书 创建证书申请文件: [root@master1 k8s]# cat > server-csr.json << EOF { "CN": "kubernetes", "hosts": [ "192.0.0.1", "127.0.0.1", "192.168.0.1", "192.168.1.5", "192.168.1.6", "192.168.1.7", "192.168.1.100", "192.168.1.8", "192.168.1.9", "192.168.1.10", "192.168.1.11", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF 注:上述文件hosts字段中IP为所有Master/LB/VIP的IP,为了方便后期扩容可以多写几个预留的IP。 生成证书: [root@vm5 k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server [root@vm5 k8s]# ls server*pem server-key.pem server.pem 所有证书: [root@vm5 k8s]# tree /root/k8s/cert/ /root/k8s/cert/ ├── etcd │ ├── ca-config.json │ ├── ca.csr │ ├── ca-csr.json │ ├── ca-key.pem │ ├── ca.pem │ ├── server.csr │ ├── server-csr.json │ ├── server-key.pem │ └── server.pem └── k8s ├── ca-config.json ├── ca.csr ├── ca-csr.json ├── ca-key.pem ├── ca.pem ├── server.csr ├── server-csr.json ├── server-key.pem └── server.pem 4.2 下载安装kubernetes二进制包 https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG 只需下载kubernetes-server-linux-amd64.tar.gz, 4.3 解压二进制包 [root@vm5 ~]# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} [root@vm5 ~]# tar zxvf kubernetes-server-linux-amd64.tar.gz [root@vm5 ~]# cd kubernetes/server/bin [root@vm5 bin]# cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin [root@vm5 bin]# cp kubectl /usr/bin/ 4.4 部署kube-apiserver 1.创建apiserver配置文件 cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF KUBE_APISERVER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --etcd-servers=https://192.168.1.5:2379,https://192.168.1.6:2379,https://192.168.1.7:2379 \\ --bind-address=192.168.1.5 \\ --secure-port=6443 \\ --advertise-address=192.168.1.5 \\ --allow-privileged=true \\ --service-cluster-ip-range=192.0.0.0/24 \\ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\ --authorization-mode=RBAC,Node \\ --enable-bootstrap-token-auth=true \\ --token-auth-file=/opt/kubernetes/cfg/token.csv \\ --service-node-port-range=30000-32767 \\ --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\ --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\ --tls-cert-file=/opt/kubernetes/ssl/server.pem \\ --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\ --client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-signing-key-file=/opt/kubernetes/ssl/server-key.pem \\ --etcd-cafile=/opt/etcd/ssl/ca.pem \\ --etcd-certfile=/opt/etcd/ssl/server.pem \\ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \\ --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \\ --proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \\ --requestheader-allowed-names=kubernetes \\ --requestheader-extra-headers-prefix=X-Remote-Extra- \\ --requestheader-group-headers=X-Remote-Group \\ --requestheader-username-headers=X-Remote-User \\ --enable-aggregator-routing=true \\ --audit-log-maxage=30 \\ --audit-log-maxbackup=3 \\ --audit-log-maxsize=100 \\ --audit-log-path=/opt/kubernetes/logs/k8s-audit.log" EOF --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ # 1.20以上版本必须有此参数 --service-account-issuer=https://kubernetes.default.svc.cluster.local \ # 1.20以上版本必须有此参数 KUBE_APISERVER_OPTS="--logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --etcd-servers=https://192.168.1.5:2379,https://192.168.1.6:2379,https://192.168.1.7:2379 --bind-address=192.168.1.5 --secure-port=6443 --advertise-address=192.168.1.5 --allow-privileged=true --service-cluster-ip-range=192.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --enable-bootstrap-token-auth=true --token-auth-file=/opt/kubernetes/cfg/token.csv --service-node-port-range=30000-32767 --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem --tls-cert-file=/opt/kubernetes/ssl/server.pem --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem --client-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-signing-key-file=/opt/kubernetes/ssl/server-key.pem --etcd-cafile=/opt/etcd/ssl/ca.pem --etcd-certfile=/opt/etcd/ssl/server.pem --etcd-keyfile=/opt/etcd/ssl/server-key.pem --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem --proxy-client-cert-file=/opt/kubernetes/ssl/server.pem --proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem --requestheader-allowed-names=kubernetes --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --enable-aggregator-routing=true --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/opt/kubernetes/logs/k8s-audit.log" 说明: –logtostderr:启用日志 —v:日志等级 –log-dir:日志目录 –etcd-servers:etcd集群地址 –bind-address:监听地址,每个Master节点都不一样,注意修改 –secure-port:https安全端口 –advertise-address:集群通告地址 –allow-privileged:启用授权 –service-cluster-ip-range:Service虚拟IP地址段 –enable-admission-plugins:准入控制模块 –authorization-mode:认证授权,启用RBAC授权和节点自管理 –enable-bootstrap-token-auth:启用TLS bootstrap机制 –token-auth-file:bootstrap token文件 –service-node-port-range:Service nodeport类型默认分配端口范围 –kubelet-client-xxx:apiserver访问kubelet客户端证书 –tls-xxx-file:apiserver https证书 –etcd-xxxfile:连接Etcd集群证书 –audit-log-xxx:审计日志 2.拷贝刚才生成的apiserver证书 把刚才生成的证书拷贝到配置文件中的路径: [root@vm5 ~]# cp /root/k8s/cert/k8s/ca*pem /root/k8s/cert/k8s/server*pem /opt/kubernetes/ssl/ 3.启用 TLS Bootstrapping 机制 TLS Bootstraping:Master apiserver启用TLS认证后,Node节点kubelet和kube-proxy要与kube-apiserver进行通信,必须使用CA签发的有效证书才可以,当Node节点很多时,这种客户端证书颁发需要大量工作,同样也会增加集群扩展复杂度。为了简化流程,Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书,kubelet会以一个低权限用户自动向apiserver申请证书,kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式,目前主要用于kubelet,kube-proxy还是由我们统一颁发一个证书。 创建上述配置文件中token文件: [root@vm5 bin]# head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 1d920817861356e8066a1396900f18d4 [root@vm5 bin]# cat > /opt/kubernetes/cfg/token.csv << EOF 1d920817861356e8066a1396900f18d4,kubelet-bootstrap,10001,"system:node-bootstrapper" EOF 格式:token,用户名,UID,用户组 4.systemd管理apiserver cat > /usr/lib/systemd/system/kube-apiserver.service << EOF [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF 5.启动并设置开机启动 [root@vm5 ~]# systemctl daemon-reload [root@vm5 ~]# systemctl start kube-apiserver [root@vm5 ~]# systemctl enable kube-apiserver 测试apiserver: [root@vm5 ~]# curl --insecure https://192.168.1.5:6443/ { "kind": "Status", "apiVersion": "v1", "metadata": { }, "status": "Failure", "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"", "reason": "Forbidden", "details": { }, "code": 403 } https://blog.csdn.net/eagle89/article/details/123347509
标签:k8s1.20,二进制,ca,192.168,etc,json,etcd,安装,csr From: https://www.cnblogs.com/skyzy/p/17035816.html