Kubernetes集群通过Helm部署skywalking及测试

• 1.前言
• 2.skywalking组件
• 3.Helm部署步骤
  • • 3.1安装包下载
    • 3.2修改配置
    • 3.3helm安装
    • 3.4访问方式
• 4.制作skywalking-agent-sidecar镜像
• 5.在deployment中应用skywalking-agent

1.前言

本文主要介绍Kubernetes中如何用Helm3部署skywalking，并对pod应用进行链路追踪。

2.skywalking组件

skywalking的几个组成部分：

• skywalking-oap-server：后端服务
• skywalking-ui：ui前端
• skywalking-es-init： 初始化es集群数据使用
• elasticsearch：存储skywalking的数据指标

3.Helm部署步骤

skywalking这里使用的版本是8.7版，elasticsearch用的是外部服务的单节点elasticsearch7.1
注意：多个skywalking共用1个elasticsearch，会冲突
可参考官方链接：https://github.com/apache/skywalking-kubernetes

3.1安装包下载

这里使用helm来部署skywalking

# 克隆skywalking-kubernetes到本地
export REPO=chart
git clone https://github.com/apache/skywalking-kubernetes
cd skywalking-kubernetes
helm repo add elastic https://helm.elastic.co

# 更新chart
helm dep up ${REPO}/skywalking
helm repo add skywalking https://apache.jfrog.io/artifactory/skywalking-helm

# 新建namespace
kubectl create namespace skywalking

3.2修改配置

修改values-my-es.yaml文件（修改elasticsearch连接地址，8.7版本的skywalking-oap镜像tag为8.7.0-es7）

[root@k8s-worker-node1 ~]# cd skywalking-kubernetes/chart/skywalking/
[root@k8s-worker-node1 ~/skywalking-kubernetes/chart/skywalking]# ll
total 92
drwxr-xr-x 2 root root  4096 Nov 23 17:54 charts
-rw-r--r-- 1 root root  1559 Nov 23 14:28 Chart.yaml
-rw-r--r-- 1 root root   877 Nov 23 14:28 OWNERS
-rw-r--r-- 1 root root 45982 Nov 23 14:28 README.md
drwxr-xr-x 2 root root  4096 Nov 23 14:28 templates
-rw-r--r-- 1 root root  1026 Nov 23 14:28 values-es6.yaml
-rw-r--r-- 1 root root  1027 Nov 25 13:47 values-es7.yaml
-rw-r--r-- 1 root root  1287 Nov 24 11:12 values-my-es.yaml
-rw-r--r-- 1 root root  1280 Nov 24 10:27 values-my-es.yaml.bak
-rw-r--r-- 1 root root 12032 Nov 24 11:13 values.yaml

# 因为我们用的是现有的elasticsearch，所以这里修改的文件是values-my-es.yaml，建议可以提前备份一份
[root@k8s-worker-node1 ~/skywalking-kubernetes/chart/skywalking]# cat values-my-es.yaml
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Default values for skywalking.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

oap:
  image:
    tag: 8.7.0-es7
  storageType: elasticsearch7

ui:
  image:
    tag: 8.7.0

elasticsearch:
  enabled: false
  config:               # For users of an existing elasticsearch cluster,takes effect when `elasticsearch.enabled` is false
    host: 172.16.108.119
    port:
      http: 9200
    user: "elastic"         # [optional]
    password: "*******"     # [optional]

3.3helm安装

# helm安装
helm install skywalking skywalking/skywalking -n skywalking --values /root/skywalking-kubernetes/chart/skywalking/values-my-es.yaml

3.4访问方式

# 通过nodeport暴露服务端口到集群外，选一个段内没有使用到的端口
kubectl edit svc skywalking-ui -n skywalking

[root@k8s-worker-node1 ~/skywalking-kubernetes/chart/skywalking]# kubectl get svc skywalking-ui -n skywalking -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: skywalking
    meta.helm.sh/release-namespace: skywalking
  labels:
    app: skywalking
    app.kubernetes.io/managed-by: Helm
    chart: skywalking-4.3.0
    component: ui
    heritage: Helm
    release: skywalking
  name: skywalking-ui
  namespace: skywalking
spec:
  clusterIP: 10.43.123.47
  clusterIPs:
  - 10.43.123.47
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - nodePort: 30001
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: skywalking
    component: ui
    release: skywalking
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

此时，就可以使用nodeIP:nodePort的方式访问skywalking的UI界面了。

4.制作skywalking-agent-sidecar镜像

# 下载skywalking-agent
wget https://archive.apache.org/dist/skywalking/8.7.0/apache-skywalking-apm-es7-8.7.0.tar.gz

# 解压
tar zxf apache-skywalking-apm-es7-8.7.0.tar.gz 

# 编写Dockerfile
[root@devops03 ~]# cat Dockerfile 
FROM busybox:latest
LABEL maintainer="even"
COPY apache-skywalking-apm-bin-es7/agent/ /usr/skywalking/agent/ 

# 封装镜像
docker build -t skywalking-agent-sidecar:8.7.0-el7 .
Sending build context to Docker daemon  993.9MB
Step 1/3 : FROM busybox:latest
latest: Pulling from library/busybox
405fecb6a2fa: Pull complete 
Digest: sha256:fcd85228d7a25feb59f101ac3a955d27c80df4ad824d65f5757a954831450185
Status: Downloaded newer image for busybox:latest
 ---> 9d5226e6ce3f
Step 2/3 : LABEL maintainer="even"
 ---> Running in 5930707a4a2c
Removing intermediate container 5930707a4a2c
 ---> f911940de7f1
Step 3/3 : COPY apache-skywalking-apm-bin-es7/agent/ /usr/skywalking/agent/
 ---> b74813f035e7
Successfully built b74813f035e7
Successfully tagged skywalking-agent-sidecar:8.7.0-el7

# 修改tag
docker tag skywalking-agent-sidecar:8.7.0-el7 harbor.xxx.xxx/library/skywalking-agent-sidecar:8.7.0-el7

# 推到harbor
[root@devops03 ~]# docker push harbor.xxx.xxx/library/skywalking-agent-sidecar:8.7.0-el7 
The push refers to repository [harbor.xxx.xxx/library/skywalking-agent-sidecar]
6efbbcc0e641: Pushed 
40cf597a9181: Pushed 
8.7.0-el7: digest: sha256:d9d842c53c848cfb81f2033b4d0ac4cd73fa39a25b199ba2c96529a9e2506d2d size: 739

5.在deployment中应用skywalking-agent

可以通过sidecar的方式将skywalking-agent挂载到需要监控的业务deployment中，主要是两部分需要改动，一是通过initContainers初始化加载skywalking-agent的镜像，二是通过volumes把agent文件挂载到业务容器中。

[root@k8s-worker-node1 ~]# cat deploy-skywalking.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: springboot-demo
  namespace: skywalking
  labels:
    app: springboot-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: springboot-demo
  template:
    metadata:
      labels:
        app: springboot-demo
    spec:
      # 构建初始化镜像(通过初始化镜像的方式集成SkyWalking Agent)
      initContainers:
        - image: harbor.xxx.xxx/library/skywalking-agent-sidecar:8.7.0-el7
          name: skywalking-agent-sidecar
          imagePullPolicy: IfNotPresent
          command: [ "sh" ]
          args:
            [
                "-c",
                "cp -R /usr/skywalking/agent/* /skywalking/agent",
            ]
          volumeMounts:
            - mountPath: /skywalking/agent
              name: skywalking-agent
      containers:
        - name: springboot-demo
          # image: ${ORIGIN_REPO}/spring-boot-skywalking-demo:${IMAGE_TAG}
          # 这里用的是随便找到的一个springboot项目的镜像
          image: xjk27400861/springbootapp:42
          imagePullPolicy: IfNotPresent
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: BUILD_TAG
              value: ${BUILD_TAG}
            - name: NAMESPACE
              value: skywalking
            #这里通过JAVA_TOOL_OPTIONS，而不是JAVA_OPTS可以实现不通过将agent命令加入到java应用jvm参数而实现agent的集成
            - name: JAVA_TOOL_OPTIONS
              value: -javaagent:/usr/skywalking/agent/skywalking-agent.jar
            - name: SW_AGENT_NAME
              value: springboot-demo
            - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
              # FQDN: servicename.namespacename.svc.cluster.local
              value: skywalking-oap.skywalking.svc:11800
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 200m
              memory: 512Mi
          volumeMounts:
            - mountPath: /usr/skywalking/agent
              name: skywalking-agent
      volumes:
        - name: skywalking-agent
          emptyDir: { }

---
apiVersion: v1
kind: Service
metadata:
  name: springboot-demo
  namespace: skywalking
  labels:
    app: springboot-demo
spec:
  ports:
    - name: port
      port: 80
      protocol: TCP
      targetPort: 8080
  selector:
    app: springboot-demo
  type: ClusterIP

参考链接：
1.https://segmentfault.com/a/1190000039878680/en
2.https://www.cnblogs.com/lina-2159/p/16190984.html