微服务网关 APISIX 群集配置指南

1.前言

APISIX 通过 ETCD 存储数据，利用 ETCD 本身的功能实现群集管理和控制。有关 APISIX 的安装配置详情参考“ APISIX 网关 CentOS 7 环境安装配置”。

前面文章描述的是单个 ETCD 节点的配置操作，本文通过配置 ETCD 3节点群集的方式实现 APISIX 的群集操作。

1.1.环境

操作系统：CentOS7，APISIX：2.10.1 ，Dashboard：2.9，ETCD：3.5

1.2.补充内容

ApiSix 的前置相关组件安装参考“ ​​​​nstall-dependencies.md​​​​”

curl​​​​​ https://raw.githubusercontent.com/apache/apisix/master/utils/install-dependencies.sh ​​​​​​-sL | bash -

更详细的安装配置可以参考“ ​​​​how-to-build.md​​​​”

2.环境配置

2.1.配置主机

在三台机器上分别执行以下操作完成主机名称设置

cat >> /etc/hosts <<EOF

192.168.0.109 etcd1

192.168.0.110 etcd2

192.168.0.111 etcd3

EOF

​2.2.安全配置

关闭系统的 Selinux 功能和操作系统防火墙

systemctl stop firewalld

systemctl disable firewalld

setenforce 0

vim /etc/selinux/config

SELINUX=disabled

3.TLS 密钥和证书

使用TLS证书对通信进行加密，并开启即与CA根证书签名的双向认证

3.1.建立证书目录和数据目录（递归创建目录  -p）

mkdir -p /etc/ssl/etcd/ssl/

mkdir -p /var/lib/etcd

3.2.创建根 CA 证书

可以通过 openssl 和 cfssl 来创建证书，这里通过 openssl 的方式创建

cd /etc/ssl/etcd/ssl

openssl genrsa -out ca.key 2048

openssl req -x509 -new -nodes -key ca.key -subj "/CN=apisix" -days 3650 -out ca.crt

​3.3.创建 ETCD 证书和私钥

3.3.1、创建配置文件

cat > ./etcd-ca.conf <<EOF

[ req ]

default_bits = 2048

prompt = no

default_md = rsa

req_extensions = req_ext

distinguished_name = dn

[ dn ]

C = CN

ST = GuangDong

L = DongGuan

O = etcd

OU = LingFang

CN = etcd

[ req_ext ]

subjectAltName = @alt_names

[ alt_names ]

DNS.1 = localhost

DNS.2 = etcd1

DNS.3 = etcd2

DNS.4 = etcd3

IP.1 = 127.0.0.1

IP.2 = 192.168.0.109

IP.3 = 192.168.0.110

IP.4 = 192.168.0.111

[ v3_ext ]

authorityKeyIdentifier=keyid,issuer:always

basicConstraints=CA:FALSE

keyUsage=keyEncipherment,dataEncipherment

extendedKeyUsage=serverAuth,clientAuth

subjectAltName=@alt_names

EOF

3.3.2、生成密钥

openssl genrsa -out etcd.key 2048

3.3.3、生成证书签发请求(certificate signing request)

openssl req -new -key etcd.key -out etcd.csr -config etcd-ca.conf

3.3.4、生成证书

openssl x509 -req -in etcd.csr -CA /etc/ssl/etcd/ca.crt -CAkey /etc/ssl/etcd/ca.key  -CAcreateserial -out etcd.crt -days 3650 -extensions v3_ext -extfile etcd-ca.conf

3.3.5、验证证书

openssl verify -CAfile ca.crt etcd.crt

openssl x509 -text -noout -in /etc/ssl/etcd/ssl/etcd.crt

4.安装 ETCD

4.1.安装证书

将CA证书ca.crt，etcd证书etcd.crt和秘钥etcd.key，拷贝到各节点的/etc/ssl/etcd/ssl/目录中

4.2.下载并安装 ETCD

将其中的 etcd 和 etcdctl 两个可执行文件复制到各节点的/usr/local/bin目录

tar zxvf etcd-v3.5.1-linux-amd64.tar.gz

cp ./etcd-v3.5.1-linux-amd64/{etcd,etcdctl} /usr/local/bin/

4.3.以 systemd 形式管理 etcd 服务

1、创建 ETCD 配置文件

mkdir -p /etc/etcd

cat > /etc/etcd/etcd.conf <<EOF

# [Member Flags]

# ETCD_ELECTION_TIMEOUT=1000

# ETCD_HEARTBEAT_INTERVAL=100

# 指定etcd的数据目录

ETCD_NAME=etcd1

ETCD_DATA_DIR=/var/lib/etcd/

# [Cluster Flags]

# ETCD_AUTO_COMPACTION_RETENTIO:N=0

ETCD_INITIAL_CLUSTER_STATE=new

ETCD_ADVERTISE_CLIENT_URLS=https://192.168.0.109:2379

ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.0.109:2380

ETCD_LISTEN_CLIENT_URLS=https://192.168.0.109:2379,https://127.0.0.1:2379

ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster

ETCD_LISTEN_PEER_URLS=https://192.168.0.109:2380

ETCD_INITIAL_CLUSTER=etcd1=https://192.168.0.109:2380,etcd2=https://192.168.0.110:2380,etcd3=https://192.168.0.111:2380

# [Proxy Flags]

ETCD_PROXY=off

# [Security flags]

# ETCD_CLIENT_CERT_AUTH=

# ETCD_PEER_CLIENT_CERT_AUTH=

# 指定etcd的公钥证书和私钥

ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.crt

ETCD_CERT_FILE=/etc/ssl/etcd/ssl/etcd.crt

ETCD_KEY_FILE=/etc/ssl/etcd/ssl/etcd.key

# 指定etcd的Peers通信的公钥证书和私钥

ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.crt

ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/etcd.crt

ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/etcd.key

#ETCD_PEER_CLIENT_CERT_AUTH=true

#ETCD_CLIENT_CERT_AUTH=true

# [Profiling flags]

# ETCD_METRICS={{ etcd_metrics }}

EOF

2、创建 systemd 服务启用文件

cat > /usr/lib/systemd/system/etcd.service <<EOF

[Unit]

Description=etcd server

After=network.target

After=network-online.target

Wants=network-online.target

[Service]

Type=notify

WorkingDirectory=/var/lib/etcd/

EnvironmentFile=-/etc/etcd/etcd.conf

ExecStart=/usr/local/bin/etcd

NotifyAccess=all

Restart=always

RestartSec=5s

LimitNOFILE=40000

[Install]

WantedBy=multi-user.target

EOF

其中 ETCD2 和 ETCD3 主机配置文件中的内容需要对应修改

4.4.启动 ETCD

只有一个节点启动etcd服务后，etcd会去访问其他两个节点的2380端口，如果一定时间内无法访问，则会服务启动失败。所以应当迅速启动至少两个节点，服务才会成功启动。

systemctl daemon-reload

systemctl enable etcd

systemctl start etcd

systemctl status etcd

4.5.检查集群是否健康

etcdctl --cacert=/etc/ssl/etcd/ssl/ca.crt --cert=/etc/ssl/etcd/ssl/etcd.crt  \

   --key=/etc/ssl/etcd/ssl/etcd.key \

   --endpoints="https://192.168.0.109:2379,https://192.168.0.110:2379,https://192.168.0.111:2379" \

   endpoint health

curl --cacert /etc/ssl/etcd/ssl/ca.crt -i https://192.168.0.109:2379/version \

   --cert /etc/ssl/etcd/ssl/etcd.crt --key /etc/ssl/etcd/ssl/etcd.key

etcdctl --cacert=/etc/ssl/etcd/ssl/ca.crt --cert=/etc/ssl/etcd/ssl/etcd.crt  \

   --key=/etc/ssl/etcd/ssl/etcd.key --endpoints="https://192.168.0.110:2379" \

   get / --prefix --keys-only

openssl s_client -connect 192.168.0.109:2379 -key /etc/ssl/etcd/etcd.key  \

-cert /etc/ssl/etcd/etcd.crt -CAfile /etc/ssl/etcd/ssl/ca.crt

4.6.ETCD 参考指南

​​​​ https://jimmysong.io/kubernetes-handbook/practice/etcd-cluster-installation.html ​​​​

5.APISix 配置

主要涉及配置文件和 ETCD 连接部分的内容

5.1.config.yaml

apisix:

 admin_key:

   - name: admin

     key: 3cea9e6d29e70e98c09d29ca98bd452a43d057bb3e6a748f3d835492

     role: admin

 port_admin: 9180

 node_listen:

       - 80

 allow_admin:

   - 127.0.0.0/24

 ssl:

   enable: true

   listen:

           - 443

   ssl_trusted_certificate: /etc/ssl/etcd/ssl/ca.crt

   #ssl_protocols: SSLv3 TLSv1.2 TLSv1.3

etcd:

 host:

   - "https://127.0.0.1:2379"

   - "https://192.168.0.109:2379"

   - "https://192.168.0.110:2379"

   - "https://192.168.0.111:2379"

 prefix: /apisix

 timeout: 30

 resync_delay: 5

 health_check_timeout: 10

 tls:

   cert: /etc/ssl/etcd/ssl/etcd.crt

   key: /etc/ssl/etcd/ssl/etcd.key

   verify: true

5.2.conf.yaml

conf:

 listen:

   # host: 127.0.0.1      

   port: 9000

 # ssl:

 #   host: 127.0.0.1     # the address on which the `Manager API` should listen for HTTPS.

                         # The default value is 0.0.0.0, if want to specify, please enable it.

 #   port: 9001            # The port on which the `Manager API` should listen for HTTPS.

 #   cert: "/tmp/cert/example.crt" # Path of your SSL cert.

 #   key:  "/tmp/cert/example.key"  # Path of your SSL key.

 allow_list:             # If we don't set any IP list, then any IP access is allowed by default.

   - 127.0.0.1           # The rules are checked in sequence until the first match is found.

   - ::1

   - 0.0.0.0/0

 etcd:

   endpoints:

     - 127.0.0.1:2379

     - 192.168.0.109:2379

     - 192.168.0.110:2379

     - 192.168.0.111:2379

   mtls:

     key_file: "/etc/ssl/etcd/ssl/etcd.key"          # Path of your self-signed client side key

     cert_file: "/etc/ssl/etcd/ssl/etcd.crt"         # Path of your self-signed client side cert

     ca_file: "/etc/ssl/etcd/ssl/ca.crt"           # Path of your self-signed ca cert, the CA is used to sign callers' certificates

   # prefix: /apisix       # apisix config's prefix in etcd, /apisix by default

 log:

   error_log:

     level: warn       # supports levels, lower to higher: debug, info, warn, error, panic, fatal

     file_path:

       logs/error.log  # supports relative path, absolute path, standard output

   access_log:

     file_path:

       logs/access.log  # supports relative path, absolute path, standard output

 max_cpu: 0              

authentication:

 secret:

   secret              # secret for jwt token generation.

 expire_time: 3600     # jwt token expire time, in second

 users:                # yamllint enable rule:comments-indentation

   - username: admin   # username and password for login `manager api`

     password: admin

   - username: user

     password: user

plugins:                          # plugin list (sorted in alphabetical order)

 - api-breaker

 - authz-keycloak

 - basic-auth

 - batch-requests

 - consumer-restriction

 - cors

 # - dubbo-proxy

 - echo

 # - error-log-logger

 # - example-plugin

 - fault-injection

 - grpc-transcode

 - hmac-auth

 - http-logger

 - ip-restriction

 - jwt-auth

 - kafka-logger

 - key-auth

 - limit-conn

 - limit-count

 - limit-req

 # - log-rotate

 # - node-status

 - openid-connect

 - prometheus

 - proxy-cache

 - proxy-mirror

 - proxy-rewrite

 - redirect

 - referer-restriction

 - request-id

 - request-validation

 - response-rewrite

 - serverless-post-function

 - serverless-pre-function

 # - skywalking

 - sls-logger

 - syslog

 - tcp-logger

 - udp-logger

 - uri-blocker

 - wolf-rbac

 - zipkin

 - server-info

 - traffic-split

6.防火墙配置

受 ApiSix 在匹配 ETCD 群集 mtls 的限制，ETCD 群集不采用证书配置模式，考虑安全通过系统自带的防火墙功能实现对源 IP 地址对目标端口的访问控制。

systemctl start firewalld

firewall-cmd --add-port=80/tcp --permanent

firewall-cmd --add-port=443/tcp --permanent

firewall-cmd --add-port=9000/tcp --permanent

firewall-cmd --add-port=22/tcp --permanent

firewall-cmd --list-rich-rules

firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.0.111" port port="2379" protocol=tcp accept'

firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.0.111" port port="2380" protocol=tcp accept'

firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.0.109" port port="2379" protocol=tcp accept'

firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.0.109" port port="2380" protocol=tcp accept'