如何快速搭建k8s的kubemark性能测试环境

Kubemark学习文档

搭建k8s集群

搭建细节参考:https://blog.51cto.com/u_12040959/5043212​

什么是kubemark

kubemark是K8S官方给出的性能测试工具，能够利用相当小的资源，模拟出一个大规模K8S集群。其主要架构如图所示：需要一个外部 K8S集群(external cluster，需要带有worker节点)以及一套完整的kubemark master控制面(可以是单节点也可以是多节点)，即另外一个 K8S集群(kubemark集群)，但是该只有master节点，没有worker节点。我们需要在external cluster中部署运行hollow pod，这些pod会主动向kubemark集群中的master注册，成为kubemark集群中的 hollow node(虚拟节点)。然后我们就可以在kubemark集群中进行性能测试。虽然与真实集群的稍微有点误差，不过可以代表真实集群的数据。​

总之需要两个集群，一个external cluster集群，需要工作节点，一个kubemark集群只需要master节点。​

搭建过程

3.1拉取镜像

在external cluster集群手动拉取kubemark镜像。​

官网地址：​​https://hub.docker.com/​​ 搜索kubemark​

找到合适的镜像 例：docker pull ss104301/kubemark​

3.2创建secret

将kubemark k8s的config文件(任意kubemark k8s主节点的/root/.kube/config)拷贝到当前路径(/home/hdf)​

kubectl create ns kubemark​

kubectl create secret generic kubeconfig --type=Opaque --namespace=kubemark --from-file=kubelet.kubecnotallow=config --from-file=kubeproxy.kubecnotallow=config ​

3.3 创建ReplicationController

镜像都需要提前手动下载，以及替换成自己需要的镜像​

apiVersion: v1​

kind: ReplicationController​

metadata:​

name: hollow-node​

namespace: kubemark​

labels:​

name: hollow-node​

spec:​

replicas: 3​

selector:​

name: hollow-node​

template:​

metadata:​

labels:​

name: hollow-node​

spec:​

initContainers:​

- name: init-inotify-limit​

image: busybox:1.32​

imagePullPolicy: IfNotPresent​

command: ['sysctl', '-w', 'fs.inotify.max_user_instances=1000']​

securityContext:​

privileged: true​

volumes:​

- name: kubeconfig-volume​

secret:​

secretName: kubeconfig​

- name: logs-volume​

hostPath:​

path: /var/log​

- name: no-serviceaccount-access-to-real-master​

emptyDir: {}​

containers:​

- name: hollow-kubelet​

image: staging-k8s.gcr.io/kubemark:v1.20.10​

imagePullPolicy: IfNotPresent​

ports:​

- containerPort: 4194​

- containerPort: 10250​

- containerPort: 10255​

env:​

- name: NODE_NAME​

valueFrom:​

fieldRef:​

fieldPath: metadata.name​

command: [​

"/kubemark",​

"--morph=kubelet",​

"--name=$(NODE_NAME)",​

"--kubecnotallow=/kubeconfig/kubelet.kubeconfig",​

"--log-file=/var/log/kubelet-$(NODE_NAME).log",​

"--logtostderr=false"​

]​

volumeMounts:​

- name: kubeconfig-volume​

mountPath: /kubeconfig​

readOnly: true​

- name: logs-volume​

mountPath: /var/log​

resources:​

requests:​

cpu: 50m​

memory: 100M​

securityContext:​

privileged: true​

- name: hollow-proxy​

image: staging-k8s.gcr.io/kubemark:v1.20.10​

imagePullPolicy: IfNotPresent​

env:​

- name: NODE_NAME​

valueFrom:​

fieldRef:​

fieldPath: metadata.name​

command: [​

"/kubemark",​

"--morph=proxy",​

"--name=$(NODE_NAME)",​

"--kubecnotallow=/kubeconfig/kubeproxy.kubeconfig",​

"--log-file=/var/log/kubeproxy-$(NODE_NAME).log",​

"--logtostderr=false"​

]​

volumeMounts:​

- name: kubeconfig-volume​

mountPath: /kubeconfig​

readOnly: true​

- name: logs-volume​

mountPath: /var/log​

resources:​

requests:​

cpu: 50m​

memory: 100M​

3.4效果图

External集群​

Kubemark集群​

基于perf-test测试性能

4.1安装go基础环境

官网下载对于系统的golang安装包，然后解压到/usr/local目录即可使用​

cd /usr/local​

wget ​​https://golang.google.cn/dl/go1.17.9.linux-amd64.tar.gz​​ ​

tar -zxvf go1.17.9.linux-amd64.tar.g​

vi /etc/profile​

export PATH=$PATH:/usr/local/go/bin​

source /etc/profile​

4.2下载perf-tests源码

官网地址：https://github.com/kubernetes/perf-tests/​

cd perf-tests/​

go build -o clusterloader './cmd/'​

# kube config for kubernetes api​

KUBE_CONFIG=${HOME}/.kube/config​

# Provider setting​

# Supported provider for xiaomi: local, kubemark, lvm-local, lvm-kubemark​

PROVIDER='kubemark'​

# SSH config for metrics' collection​

KUBE_SSH_KEY_PATH=$HOME/.ssh/id_rsa​

MASTER_SSH_IP=10.142.43.51​

MASTER_SSH_USER_NAME=root​

# Clusterloader2 testing strategy config paths​

# It supports setting up multiple test strategy. Each testing strategy is individual and serial.​

TEST_CONFIG='configs/examples/density/config.yaml'​

# Clusterloader2 testing override config paths​

# It supports setting up multiple override config files. All of override config files will be applied to each testing strategy.​

# OVERRIDE_CONFIG='testing/density/override/200-nodes.yaml'​

# Log config​

REPORT_DIR='./reports'​

LOG_FILE='logs/tmp.log'​

4.3运行clusterloader2

$CLUSTERLOADER_BIN --kubecnotallow=$KUBE_CONFIG \​

--provider=$PROVIDER \​

--masterip=$MASTER_SSH_IP --mastername=$MASTER_SSH_USER_NAME \​

--testcnotallow=$TEST_CONFIG \​

--report-dir=$REPORT_DIR \​

--alsologtostderr 2>&1 | tee $LOG_FILE​

# --testoverrides="${OVERRIDE_CONFIG:-}" \​

4.4测试结果

整个性能测试过程以及部分测试结果会打印到stdout中，如果按本文所述方法运行clusterloader2，打印结果会保存一份至${LOG_FILE}。​

调度吞吐量结果:​

Jan 9 13:55:47.876: INFO: E2E startup time for 10500 pods: 28m40.333662404s​

Jan 9 13:55:47.876: INFO: Throughput (pods/s) during cluster saturation phase: 6.1034675​

^[[1mSTEP^[[0m: Printing Pod to Node allocation data​

Jan 9 13:55:47.884: INFO: Density Pods: 10500 out of 10500 created, 10500 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady​

Pod启动e2e耗时结果:​

Jan 9 14:09:31.598: INFO: perc50: 1614, perc90: 2241, perc99: 2541​

Jan 9 14:09:31.598: INFO: Approx throughput: 7486.052881923156 pods/min​

report文件夹​

详细的性能测试结果clusterloader2收集在report文件夹下，如果按本文方法运行，则结果存放在{$REPORT_DIR}中。因为涉及到的性能测试结果非常多，本文不在此一一列举。目前Kubernetes性能分析最主要的性能指标如下:​

APIServer Restful API响应时间 - APIResponsiveness​

Pod 启动总耗时 - PodStartupLatency​

Scheduler调度性能 - SchedulingThroughput, SchedulingMetrics​

ETCD指标 - EtcdMetrics​