k8s.HPA.使用自定义指标Pod扩容

k8s.HPA.使用自定义指标Pod扩容

环境 env :
kubernetes v1.22
metrics-server 0.6.1
prometheus v2.36.1
prometheus-adapter.v0.10.0

概述

弹性伸缩

• 1、资源层弹性, 如增减k8s节点
• 2、调度层弹性, 如增减pod副本数量(HPA)、增减pod负载占比(VPA)

HPA概念

• HPA全称Horizontal Pod Autoscaling 即pod水平自动扩展
• HPA定期轮询获取监控指标,计算pod数量,自动增减pod数量
• pod数量算法: 期望副本数 = ceil[当前副本数 * (当前指标 / 期望指标)]

HPA的API版本

# 获取hpa版本
kubectl api-versions | grep autoscal
autoscaling/v1 #只支持cpu指标
autoscaling/v2beta1 #cpu,内存,自定义指标
autoscaling/v2beta2 #cpu,内存,自定义指标,外部指标
autoscaling/v2 #1.23版本开始有,v2beta2将被废弃

HPA Core metrics(核心指标)

• 根据cpu,内存使用量、使用率指标使用HPA
• 使用率根据requests计算(resources.requests[cpu|memory])
• 需要部署metrics-server提供指标数据(安装对应支持的k8s版本)
  https://github.com/kubernetes-sigs/metrics-server

# top命令正常则metrics-server已安装
kubectl top nodes

# 安装metrics-server 0.6.1
# kubectl apply -f https://gitee.com/alivv/elvin-demo/raw/master/k8s/metrics-server.v0.6.1.yml

创建HPA实例

# 创建deployemnt和hpa demo
kubectl apply -f https://gitee.com/alivv/elvin-demo/raw/master/k8s/hpa-nginx-demo.yml

# 查看hpa
kubectl get hpa
kubectl describe hpa hpa-nginx-demo

# 查看pod数量
kubectl get deploy hpa-nginx-demo

# 删除demo
kubectl delete -f https://gitee.com/alivv/elvin-demo/raw/master/k8s/hpa-nginx-demo.yml

HPA Custom Metrics(自定义指标)

• 需要部署Prometheus和Prometheus Adapter
  Prometheus(采集) + Prometheus Adapter(查询接口) -> 自定义指标
• prometheus-adapter
  https://github.com/kubernetes-sigs/prometheus-adapter

prometheus-adpater部署

# prometheus部署 略
# prometheus-adpater部署, By Elvin
#下载修改prometheus地址后部署
wget https://gitee.com/alivv/elvin-demo/raw/master/k8s/prometheus-adapter.v0.10.0.yml
# 替换成自己的prometheus地址
prometheus_url=http://prometheus.monitor.svc:9090
sed -i "s@http://prometheus.monitor.svc:9090@${prometheus_url}@" prometheus-adapter.v0.10.0.yml

kubectl apply -f prometheus-adapter.v0.10.0.yml

# 查看接口和自定义指标
kubectl get --raw "/apis/custom.metrics.k8s.io" | jq .     
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

自定义指标实例如下

apiVersion: v1
kind: ConfigMap
metadata:
  name: adapter-config
  namespace: custom-metrics
data:
  config.yaml: |
    rules:
  
    - seriesQuery: '{__name__=~"container_memory_working_set_bytes|container_spec_memory_limit_bytes",container!="POD",namespace!="",pod!=""}'
      resources:
        overrides:
          namespace:
            resource: namespace
          pod:
            resource: pod
      name:
        matches: ^(.*)
        #指标名称
        as: "memory_limit_usage"
        #prometheus查询语句
      metricsQuery: round(sum (container_memory_working_set_bytes{container!="POD",namespace!="",pod!=""}) by (namespace, pod) / sum (container_spec_memory_limit_bytes{container!="POD",namespace!="",pod!=""}) by (namespace, pod) * sum (avg_over_time(kube_pod_status_ready{condition="true"}[1m])) by (namespace, pod) * 100) 

简要说明

• memory_limit_usage #内存使用率
  container_memory_working_set_bytes / container_spec_memory_limit_bytes
  java程序占用内存大,监控和扩容使用的限制值

• pod处于ready状态前1分钟不参与计算
  避免pod刚启动时的cpu、内存波动造成hpa判断不准,
  使用avg_over_time求前1分钟的平均值,
  pod状态not ready时kube_pod_status_ready值为0

HPA实例

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-nginx-demo
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hpa-nginx-demo
  minReplicas: 1
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          #自定义监控指标名称
          name: memory_limit_usage
        target:
          type: AverageValue
          #平均值大于等于90时
          averageValue: 90

  #autoscaling/v2beta2开始支持扩缩策略
  behavior:
    scaleUp:
      #扩容前等待s 默认0s
      stabilizationWindowSeconds: 30
      policies:
      - type: Percent
        value: 100
        #每15s最大扩容当前1倍数量Pod
        periodSeconds: 15
      - type: Pods
        value: 4
        #每15s最大允许扩容4个Pod
        periodSeconds: 15
      #使用以上两种扩容策略中算出来扩容Pod数量最大的
      selectPolicy: Max
    scaleDown:
      #缩容等待 默认300
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 100 # 允许全部缩掉
        periodSeconds: 15

根据不同业务场景调节HPA扩缩容灵敏度

参考

#Pod 水平自动扩缩
https://kubernetes.io/zh-cn/docs/tasks/run-application/horizontal-pod-autoscale/

#部署HPA实现高可用和成本控制
https://zzq23.blog.csdn.net/article/details/108982724

#HPA基于CPU、内存和自定义指标自动扩缩容
https://blog.csdn.net/fly910905/article/details/105375822/

#Kubernetes自定义监控指标 Prometheus Adapter
https://www.cnblogs.com/zhangmingcheng/p/15773348.html

#根据不同业务场景调节HPA扩缩容灵敏度
https://cloud.tencent.com/document/product/457/50660

#HPA自动伸缩常见问题
https://help.aliyun.com/document_detail/181491.html