开源可观测性平台Signoz【日志采集篇】

标签：otel Signoz 开源 signoz docker collector 日志 containers

转载说明：如果您喜欢这篇文章并打算转载它，请私信作者取得授权。原创不易，请文明转载，谢谢。

在开源可观测性平台Signoz系列【开篇】中，介绍了signoz的基础理论知识、安装部署及一些初始化配置。本文则记录signoz怎么采集日志，包括docker容器日志和主机日志

1. 收集容器日志

1.1 收集signoz本机docker容器日志

signoz在启动后，默认就会收集本机所有docker容器的日志，无需另行配置。如果登录signoz后，发现日志模块为空：原因可能是本机docker安装目录不在默认的/var/lib/下。在signoz的docker-compose.yml文件中（signoz-0.16.2/deploy/docker/clickhouse-setup/docker-compose.yml）文件中otel-collector模块，第201行默认的配置是：

- /var/lib/docker/containers:/var/lib/docker/containers:ro

如果当前服务器的docker安装目录并不在/var/lib/docker下，而是在别的路径如/home/docker，默认配置就无法收集到日志。同时可以看到默认的挂载路径/var/lib/docker/containers目录下是空文件，所以signoz无法采集到docker容器的日志。

同时clickhouse-setup_otel-collector_1容器的日志会有如下报错：

2023-03-20T09:14:21.855Z warn fileconsumer/file.go:61 no files match the configured include patterns {"kind": "receiver", "name": "filelog/dockercontainers", "pipeline": "logs", "component": "fileconsumer", "include": ["/var/lib/docker/containers//.log"], "exclude": []}

修改方法： 修改signoz-0.16.2/deploy/docker/clickhouse-setup/docker-compose.yml第201行为：

- /home/docker/containers:/var/lib/docker/containers:ro

然后重启otel-collector即可，如果是新的signoz，直接重启整个signoz也可。

1.2 收集其余主机docker容器日志

收集非signoz服务器的docker日志就需要安装客户端了，将服务器的docker目录挂载到客户端容器里，然后上报到signoz主机，原理类似filebeat采集日志。

1）创建指标收集配置文件otel-collector-config.yaml

[root@test otel]# cat otel-collector-config.yaml
receivers:
  filelog/containers:
    include: [  "/var/lib/docker/containers/*/*.log" ]
    #exclude: [ "/var/lib/docker/containers/*/<container_id>.log" ] #排除某个容器日志
    start_at: end
    include_file_path: true
    include_file_name: false
    operators:
    - type: json_parser
      id: parser-docker
      output: extract_metadata_from_filepath
      timestamp:
        parse_from: attributes.time
        layout: '%Y-%m-%dT%H:%M:%S.%LZ'
    - type: regex_parser
      id: extract_metadata_from_filepath
      regex: '^.*containers/(?P<container_id>[^_]+)/.*log$'
      parse_from: attributes["log.file.path"]
      output: parse_body
    - type: move
      id: parse_body
      from: attributes.log
      to: body
      output: add_source
    - type: add
      id: add_source
      field: resource["source"]
      value: "docker"
    - type: remove
      id: time
      field: attributes.time
processors:
  batch:
    send_batch_size: 10000
    send_batch_max_size: 11000
    timeout: 10s
exporters:
  otlp/log:
    endpoint: http://${signoz IP或域名}:4317  #如果部署signoz未修改端口映射，则默认是4317 
    tls:
      insecure: true
service:
  pipelines:
    logs:
      receivers: [filelog/containers]
      processors: [batch]
      exporters: [ otlp/log ]
[root@test otel]#

2）编写docker-compose.yml，用docker-compose方式启动

[root@test otel]# cat docker-compose.yaml 
version: "3"
services:
  otel-collector:
    image: signoz/signoz-otel-collector:0.66.5
    command: ["--config=/etc/otel-collector-config.yaml"]
    user: root # required for reading docker container logs
    container_name: signoz-host-otel-collector
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
      - /home/docker/containers:/var/lib/docker/containers:ro  #挂载本机正确的docker路径
    restart: on-failure
[root@test otel]#

3）启动signoz-otel-collector

docker-compose -f docker-compose.yml up -d

如果没装docker-compose，也可以直接docker方式启动otel-collector：

docker run -d --name signoz-host-otel-collector \
  --user root \
  -v /var/lib/docker/containers:/var/lib/docker/containers:ro \
  -v ./otel-collector-config.yaml:/etc/otel/config.yaml \
  signoz/signoz-otel-collector:0.66.5

4）启动signoz-host-otel-collector后，到signoz查看效果

主机日志如下： signoz日志如下：

2. 收集主机日志

以nginx为例，记录signoz收集非docker容器的服务器指定日志文件。

1）修改1.2中创建的otel-collector-config.yaml，加入filelog内容

[root@test otel]# cat otel-collector-config.yaml 
receivers:
  filelog/containers:
    include: [  "/var/lib/docker/containers/*/*.log" ]
    start_at: end
    include_file_path: true
    include_file_name: false
    operators:
    - type: json_parser
      id: parser-docker
      output: extract_metadata_from_filepath
      timestamp:
        parse_from: attributes.time
        layout: '%Y-%m-%dT%H:%M:%S.%LZ'
    - type: regex_parser
      id: extract_metadata_from_filepath
      regex: '^.*containers/(?P<container_id>[^_]+)/.*log$'
      parse_from: attributes["log.file.path"]
      output: parse_body
    - type: move
      id: parse_body
      from: attributes.log
      to: body
      output: add_source
    - type: add
      id: add_source
      field: resource["source"]
      value: "docker"
    - type: remove
      id: time
      field: attributes.time
 #####新增内容-1开始##########     
  filelog:
    include: [ "/var/log/nginx/*.log" ]  #本机nginx日志路径
    start_at: beginning
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d,%H:%M:%S %z'
      - type: move
        from: attributes.message
        to: body
      - type: remove
        field: attributes.time
 #####新增内容-1结束##########  
processors:
  batch:
    send_batch_size: 10000
    send_batch_max_size: 11000
    timeout: 10s
exporters:
  otlp/log:
    endpoint: http://${IP of signoz}:4317  #如果部署signoz未修改端口映射，则默认是4317 
    tls:
      insecure: true
service:
  pipelines:
    logs:
      receivers: [filelog/containers,filelog]  #新增内容-2：添加filelog
      processors: [batch]
      exporters: [ otlp/log ]
[root@test otel]#

2）重启 signoz-host-otel-collector

docker-compose -f docker-compose.yaml restart

3）到signoz查看日志：虽然功能齐全，signoz在日志搜索与查看方面不是很好用，相比ELK，要逊色一些。

3. 日志保留时长设置

日志采集保存后，通常会设置数据保留时间。在signoz中设置日志保留时长很简单，在界面就可以完成。

登录signoz进入Settings—General，即可设置和修改Metrics、Traces、Logs的数据保留时间：

点击查看系列文章：开源可观测性平台Signoz系列【开篇】

标签：otel,Signoz,开源,signoz,docker,collector,日志,containers
From： https://blog.51cto.com/u_10950710/6363279