搭建https的es+kibana(7.9.1)

背景：elasticsearch7需要开启https才可以创建报警，因此就需要搭建https的elasticsearch

参考官方网站：https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-elasticsearch.html

第一步，创建crd

kubectl create -f https://download.elastic.co/downloads/eck/2.12.1/crds.yaml

第二步，安装operator的RDBC

kubectl apply -f https://download.elastic.co/downloads/eck/2.12.1/operator.yaml

第三步，确保operator的pod成功启动

[root@localhost ~]# kubectl -n elastic-system get pods 
NAME                 READY   STATUS    RESTARTS      AGE
elastic-operator-0   1/1     Running   6 (20h ago)   20h

第四步，创建es集群

[root@localhost es]# cat es.yaml.bak 
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: eck-cluster
  namespace: elk
spec:
  version: 7.9.1
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.ml: false
      xpack.graph.enabled: false
      xpack.ml.enabled: false
      xpack.watcher.enabled: false
      xpack.monitoring.collection.enabled: true
      xpack.security.enabled: true
      xpack.security.http.ssl.enabled: true
      xpack.security.authc:
          anonymous:
            username: anonymous
            roles: superuser
            authz_exception: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: standard
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms512m -Xmx512m -Des.allow_insecure_settings=true
          - name: READINESS_PROBE_PROTOCOL
            value: https
          resources:
            requests:
              memory: 2Gi
            limits:
              memory: 4Gi

第五步，确保es正常启动，我这里只创建了一个节点，因此是yellow，启动3个就是green了

[root@localhost es]# kubectl -n elk get es 
NAME          HEALTH   NODES   VERSION   PHASE   AGE
eck-cluster   yellow   1       7.9.1     Ready   61m

第六步，创建kibana，这里要注意的是红色字体的名字要对应kubectl -n elk get es显示的名字

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: eck-cluster
  namespace: elk
spec:
  version: 7.9.1
  count: 1
  config: 
    #elasticsearch.hosts: ["https://eck-cluster-es-http.elk.svc:9200"]
    elasticsearch.username: "elastic"
    elasticsearch.password: "xxxxx"
  elasticsearchRef:
    name: eck-cluster

第七步，确保es和kibana都正常启动

[root@localhost es]# kubectl -n elk get pods 
NAME                              READY   STATUS    RESTARTS   AGE
eck-cluster-es-default-0          1/1     Running   0          44m
eck-cluster-kb-7d75dd6758-g2rcp   1/1     Running   0          57m

第八步，开启端口转发，访问kibana  UI

kubectl -n elk port-forward --address 0.0.0.0 svc/eck-cluster-kb-http 5601:5601

第九步，可以看到kibana的报警已经可以使用了

补充，如果需要filebeat连接es发送日志的话，也需要https协议，红色字体是开启https协议的内容具体配置如下：

filebeat.inputs:
- type: log
  processors:
  paths:
    - "/mnt/log/logstash.log"
  fields:
    app: xxx
    index: xxxx
    group: xxx
    namespaces: xxx
    clusterName: "${CLUSTER_NAME:}"
  #没有新日志采集后多长时间关闭文件句柄，默认5分钟，设置成1分钟，加快文件句柄关闭；
  close_inactive: 1m
  #传输了3h后荏没有传输完成的话就强行关闭文件句柄；
  close_timeout: 3h
  #这个配置项也应该配置上，默认值是0表示不清理，不清理的意思是采集过的文件描述在registry文件里永不清理，在运行一段时间后，registry会变大，可能会带来问题。
  clean_inactive: 72h
  #设置了clean_inactive后就需要设置ignore_older，且要保证ignore_older < clean_inactive
  ignore_older: 70h
  #multiline.pattern: ^20[0-9]{2}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}
  #multiline.negate: true
  #multiline.match: after
  # 和福林确认日志只需要处理1毫秒内的多行输出
  #multiline.timeout: 50ms
  # 只保留40行日志
  #multiline.max_lines: 40
  exclude_files: ['debug']
  json.message_key: message
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
- type: log
  processors:
  paths:
    - "/mnt/log/requestlog.log"
  fields:
    app: xxx
    index: xxx
    group: xxx
    namespaces: xxx
    clusterName: "${CLUSTER_NAME:}"
  close_inactive: 1m
  close_timeout: 3h
  clean_inactive: 72h
  ignore_older: 70h
  exclude_files: ['debug']
  json.message_key: message
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
- type: log
  processors:
  paths:
    - "/mnt/log/gc.log"
  fields:
    app: xxxx
    index: gc
    group: xxx
    namespaces: xxxx
    clusterName: "${CLUSTER_NAME:}"
  close_inactive: 1m
  close_timeout: 3h
  clean_inactive: 72h
  ignore_older: 70h
  exclude_files: ['debug']
output.elasticsearch:
  enabled: true
  hosts: ["eck-cluster-es-http.elastic-system.svc:9200"]
  protocol: "https"
  username: "elastic-system-eck-cluster-kibana-user"
  password: "wYPuZ0719ix52U408M6prQXx"
  ssl:
    certificate_authorities: ["/usr/share/filebeat/ca.pem"]
    erification_mode: "certificate"
  index: "%{[fields.index]}-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template.name: "%{[fields.index]}"
setup.template.pattern: "%{[fields.index]}-*"
setup.template.enabled: false
setup.template.overwrite: true