LLM 日志采集

@[TOC](Logstash+MQ 日志采集)

需求场景

logback+MQ+Logstash

采集多台服务（下文用生产端代替）的日志数据，汇总到一台服务器（下文用消费端代替）中。

实现

1.RabbitMQ 将日志消息发布到mq，消费端获取消息。 2.Logstash 收集日志数据。放在消费端，接收mq消息，文件形式输出到本地。

服务环境

1.​​windows环境​​ 2 .mq（​​环境模拟​​）

生产端配置

1.在logback.xml中配置MQ

1.引入依赖

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-rabbit</artifactId>
</dependency>

2.logback.xml 修改 ​​以一个为例，分别在项目中配置即可​​

<appender name="RABBITMQ" class="org.springframework.amqp.rabbit.logback.AmqpAppender">
        <layout>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %X{requestId}] %-5level %logger{50} - %msg%n</pattern>
        </layout>
        <host>192.168.1.209</host>
        <port>5672</port>
        <username>guest</username>
        <password>guest</password>
        <declareExchange>true</declareExchange>
        <exchangeType>direct</exchangeType>
        <exchangeName>app.log</exchangeName>
        <routingKeyPattern>info</routingKeyPattern>
        <generateId>true</generateId>
        <charset>UTF-8</charset>
        <durable>true</durable>
        <deliveryMode>NON_PERSISTENT</deliveryMode>
        <autoDelete>false</autoDelete>
    </appender>

root中配置appender-ref

<root level="info">
        <appender-ref ref="RABBITMQ" />
    </root>

消费端配置

1.下载erlang，rabbitmq 安装自行百度，一步到底，可视化需要安装插件，百度一大堆 2.下载logstash运行

logstash.conf 配置

​​以下以收集两个服务的日志为例​​

input {
 
  rabbitmq {
       type =>"app"
       durable => true
       exchange => "app.log"
       exchange_type => "direct"
       key => "info"
       host => "192.168.1.209"
       port => 5672
       user => "guest"
       password => "guest"
       queue => "log_queue"
       auto_delete => false
  }
  
    rabbitmq {
       type =>"fmis"
       durable => true
       exchange => "fmis.log"
       exchange_type => "direct"
       key => "info"
       host => "192.168.1.209"
       port => 5672
       user => "guest"
       password => "guest"
       queue => "log_queue"
       auto_delete => false
  }
 
}
 

output {
 
   if [type] == "app" {
    file {
      path => "E:/test-log/app-%{+YYYY-MM-dd}.log"
      codec => multiline {
            pattern => "^\d"
            negate => true
            what => "previous"
        }
 
    }
  }
   if [type] == "fmis" {
    file {
      path => "E:/test-log/fmis-%{+YYYY-MM-dd}.log"
      codec => multiline {
            pattern => "^\d"
            negate => true
            what => "previous"
        }
 
    }
  }
}

配置好后，启动rabitmq和logstash

效果

访问 192.168.1.209:15672

2.查看本地