loki采集k8s日志

前言

• loki 是轻量、易用的日志聚合系统。如果你的k8s集群规模并不大，推荐使用grafana+loki的方案来做微服务日志的采集；

Loki组成

• loki架构很简单，主要由3部分组成：

• loki：服务端，负责存储日志和处理查询；
• promtail：采集端，负责采集日志发送给loki；
• grafana：负责采集日志的展示；

promtail.yml

kubectl apply -f promtail.yml  

注意： 使用前，请删掉promtail.yml中的中文注释

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-configmap
data:
  promtail-config.yml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    positions:
      filename: /tmp/positions.yaml

    clients:
    - url: https://{YOUR_LOKI_ENDPOINT}/loki/api/v1/push  #输入自己的loki地址

    scrape_configs:
    - job_name: containers
      static_configs:
      - targets:
        - localhost
        labels:
          log_from: static_pods
          __path__: /var/log/pods/*/*/*.log
      pipeline_stages:
      - docker: {}
      - match:
          selector: '{log_from="static_pods"}'
          stages:
          - regex:
              source: filename
              expression: "(?:pods)/(?P<namespace>\\S+?)_(?P<pod>\\S+)-\\S+?-\\S+?\\S+?/(?P<container>\\S+?)/"
          - labels:
              namespace:
              pod:
              container:
      - match:
          selector: '{namespace!~"(default|kube-system|kube-systemen)"}'   #输入自己的命名空间
          action: drop
          drop_counter_reason: no_use
          
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
  namespace: kube-systemen
spec:
  selector:
    matchLabels:
      name: promtail
  template:
    metadata:
      labels:
        name: promtail
    spec:
      nodeSelector:
        promtail: promtail
      containers:
        - name: promtail
          image: grafana/promtail:2.4.2
          imagePullPolicy: Always
          args:
            - -config.file=/mnt/config/promtail-config.yml
            - -client.external-labels=hostip=$(NODE_IP),hostname=$(NODE_NAME),env=STM
          ports:
            - containerPort: 9080
          volumeMounts:
            - mountPath: /opt/docker/containers
              name: containers-volume
            - mountPath: /var/lib/docker/containers
              name: containers-volumes
            - mountPath: /var/log/pods
              name: pods-volume
            - mountPath: /mnt/config
              name: config-volume
          env: 
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: 'spec.nodeName'
          - name: NODE_IP
            valueFrom:
              fieldRef:
                fieldPath: 'spec.nodeIP'
                
      volumes:
        - name: containers-volume
          hostPath:
            path: /opt/docker/containers
        - name: containers-volumes
          hostPath:
            path: /var/lib/docker/containers
        - name: pods-volume
          hostPath:
            path: /var/log/pods
        - name: config-volume
          configMap:
            name: promtail-configmap
            items: 
              - key: promtail-config.yml
                path: promtail-config.yml
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          errect: NoSchedule

官方文档：https://grafana.com/docs/loki/latest/send-data/promtail/installation/