首页 > 其他分享 >一文教会你如何使用 iLogtail SPL 处理日志

一文教会你如何使用 iLogtail SPL 处理日志

时间：2024-10-25 18:10:09浏览次数：1  
标签：__ Type content iLogtail SPL time 日志 true processor

作者：阿柄

随着流式处理的发展，出现了越来越多的工具和语言，使得数据处理变得更加高效、灵活和易用。在此背景下，SLS 推出了 SPL(SLS Processing Language) 语法，以此统一查询、端上处理、数据加工等的语法，保证了数据处理的灵活性。iLogtail 作为日志、时序数据采集器，在 2.0 版本中，全面支持了 SPL 。本文对处理插件进行了梳理，介绍了如何编写 SPL 语句，从插件处理模式迁移到 2.0 版本的 SPL 处理模式，帮助用户实现更加灵活的端上数据处理。

SPL

iLogtail 一共支持 3 种处理模式。

  • 原生插件模式： 由 C++ 实现的原生插件，性能最强。
  • 拓展插件模式： 由 Go 实现的拓展插件，提供了丰富的生态，足够灵活。
  • SPL 模式： 随着 iLogtail 2.0 在 C++ 处理插件中支持了 SPL 的处理能力，对数据处理能力带来了很大的提升，兼顾性能与灵活性。用户只需要编写 SPL 语句，即可以利用 SPL 的计算能力，完成对数据的处理。SPL 语法可以参考：https://help.aliyun.com/zh/sls/user-guide/spl-syntax/

总的来说，iLogtail 2.0 + SPL 主要有以下的优势：

  1. 统一数据处理语言： 对于同样一种格式的数据，用户可以在不同场景中使用同一种语言进行处理，提高了数据处理的效率
  2. 查询处理更高效： SPL 对弱结构化数据友好，同时 SPL 主要算子由 C++ 实现，接近 iLogtail 1.X 版本的原生性能
  3. 丰富的工具和函数： SPL 提供了丰富的内置函数和算子，用户可以更加灵活地进行组合
  4. 简单易学： SPL 属于一种低代码语言，用户可以快速上手，日志搜索、处理一气呵成

接下来，本文将介绍如何用灵活的 SPL 语句，实现其他两种处理模式相同的处理能力。

原生插件对比

正则解析

根据正则提取提取字段。输入 Nginx 格式：

127.0.0.1 - - [07/Jul/2022:10:43:30 +0800] "POST /PutData?Category=YunOsAccountOpLog" 0.024 18204 200 37 "-" "aliyun-sdk-java"

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths: 
      - /workspaces/ilogtal/debug/simple.log
processors:
  - Type: processor_parse_regex_native
    SourceKey: content
    Regex: ([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\\"]*)\" \"([^\\"]*)\"
    Keys:
      - ip
      - time
      - method
      - url
      - request_time
      - request_length
      - status
      - length
      - ref_url
      - browser
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\\"]*)\" \"([^\\"]*)\"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser
      | project-away content
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30",
    "method": "POST",
    "url": "/PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713184059"
}

分隔符解析

根据分隔符分隔提取字段，输入：

127.0.0.1,07/Jul/2022:10:43:30 +0800,POST,PutData Category=YunOsAccountOpLog,0.024,18204,200,37,-,aliyun-sdk-java

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_parse_delimiter_native
    SourceKey: content
    Separator: ","
    Quote: '"'
    Keys:
      - ip
      - time
      - method
      - url
      - request_time
      - request_length
      - status
      - length
      - ref_url
      - browser
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-csv content as ip, time, method, url, request_time, request_length, status, length, ref_url, browser
      | project-away content
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30 +0800",
    "method": "POST",
    "url": "PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713231487"
}

Json 解析

解析 json 格式日志，输入：

{"url": "POST /PutData?Category=YunOsAccountOpLog HTTP/1.1","ip": "10.200.98.220",    "user-agent": "aliyun-sdk-java",    "request": "{"status":"200","latency":"18204"}",    "time": "07/Jul/2022:10:30:28",    "__time__": "1713237315"}

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_parse_json_native
    SourceKey: content
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | project-away content
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{    "url": "POST /PutData?Category=YunOsAccountOpLog HTTP/1.1",
    "ip": "10.200.98.220",
    "user-agent": "aliyun-sdk-java",
    "request": "{\"status\":\"200\",\"latency\":\"18204\"}",
    "time": "07/Jul/2022:10:30:28",
    "__time__": "1713237315"
}

正则解析+时间解析

根据正则表达式解析字段，并将其中的一个字段解析成日志时间，输入：

127.0.0.1,07/Jul/2022:10:43:30 +0800,POST,PutData Category=YunOsAccountOpLog,0.024,18204,200,37,-,aliyun-sdk-java

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_parse_regex_native
    SourceKey: content
    Regex: ([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\\"]*)\" \"([^\\"]*)\"
    Keys:
      - ip
      - time
      - method
      - url
      - request_time
      - request_length
      - status
      - length
      - ref_url
      - browser
  - Type: processor_parse_timestamp_native
    SourceKey: time
    SourceFormat: '%Y-%m-%dT%H:%M:%S'
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      * 
      | parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+)\] \"(\w+) ([^\\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\\"]*)\" \"([^\\"]*)\"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser
      | extend ts=date_parse(time, '%Y-%m-%d %H:%i:%S')
      | extend __time__=cast(to_unixtime(ts) as INTEGER)
      | project-away ts
      | project-away content
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30 +0800",
    "method": "POST",
    "url": "PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713231487"
}

正则解析+过滤

根据正则表达式解析字段，并根据解析后的字段值过滤日志。输入：

127.0.0.1 - - [07/Jul/2022:10:43:30 +0800] "POST /PutData?Category=YunOsAccountOpLog" 0.024 18204 200 37 "-" "aliyun-sdk-java"
127.0.0.1 - - [07/Jul/2022:10:44:30 +0800] "Get /PutData?Category=YunOsAccountOpLog" 0.024 18204 200 37 "-" "aliyun-sdk-java"

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_parse_regex_native
    SourceKey: content
    Regex: ([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\\"]*)\" \"([^\\"]*)\"
    Keys:
      - ip
      - time
      - method
      - url
      - request_time
      - request_length
      - status
      - length
      - ref_url
      - browser
  - Type: processor_filter_regex_native
    FilterKey:
      - method
      - status
    FilterRegex:
      - ^(POST|PUT)$
      - ^200$
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\\"]*)\" \"([^\\"]*)\"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser
      | project-away content
      | where regexp_like(method, '^(POST|PUT)$') and regexp_like(status, '^200$')
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30",
    "method": "POST",
    "url": "/PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713238839"
}

脱敏

将日志中的敏感信息脱敏。输入：

{"account":"1812213231432969","password":"04a23f38"}

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_desensitize_native
    SourceKey: content
    Method: const
    ReplacingString: "******"
    ContentPatternBeforeReplacedString: 'password":"'
    ReplacedContentPattern: '[^"]+'
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-regexp content, 'password":"(\S+)"' as password
      | extend content=replace(content, password, '******')
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{
    "content": "{\"account\":\"1812213231432969\",\"password\":\"******\"}",
    "__time__": "1713239305"
}

拓展插件对比

添加字段

在输出结果中添加字段，输入：

this is a test log

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_add_fields
    Fields:
      service: A
    IgnoreIfExist: false
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend service='A'
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{
    "content": "this is a test log",
    "service": "A",
    "__time__": "1713240293"
}

Json 解析+丢弃字段

解析 json 并删除解析后的指定字段。输入：

{"key1": 123456, "key2": "abcd"}

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_parse_json_native
    SourceKey: content
  - Type: processor_drop
    DropKeys: 
      - key1
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | project-away content
      | project-away key1
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{    "key2": "abcd",
    "__time__": "1713245944"
}

Json 解析+重命名字段

解析 json 并重命名解析后的字段。输入：

{"key1": 123456, "key2": "abcd"}

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_parse_json_native
    SourceKey: content
  - Type: processor_rename
    SourceKeys:
      - key1
    DestKeys:
      - new_key1
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | project-away content
      | project-rename new_key1=key1
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{
    "new_key1": "123456",
    "key2": "abcd",
    "__time__": "1713249130"
}

Json 解析+过滤日志

解析 json 并根据字段条件过滤日志。输入：

{"ip": "10.**.**.**", "method": "POST", "browser": "aliyun-sdk-java"}
{"ip": "10.**.**.**", "method": "POST", "browser": "chrome"}
{"ip": "192.168.**.**", "method": "POST", "browser": "aliyun-sls-ilogtail"}

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_parse_json_native
    SourceKey: content
  - Type: processor_filter_regex
    Include:
      ip: "10\\..*"
      method: POST
    Exclude:
      browser: "aliyun.*"
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | project-away content
      | where regexp_like(ip, '10\..*') and regexp_like(method, 'POST') and not regexp_like(browser, 'aliyun.*')
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{
    "ip": "10.**.**.**",
    "method": "POST",
    "browser": "chrome",
    "__time__": "1713246645"
}

Json 解析+字段值映射处理

解析 json 并根据字段值的不同，映射为不同的值。输入：

{"_ip_":"192.168.0.1","Index":"900000003"}
{"_ip_":"255.255.255.255","Index":"3"}

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_parse_json_native
    SourceKey: content
  - Type: processor_dict_map
    MapDict:
      "127.0.0.1": "LocalHost-LocalHost"
      "192.168.0.1": "default login"
    SourceKey: "_ip_"
    DestKey: "_processed_ip_"
    Mode: "overwrite"
    HandleMissing": true
    Missing: "Not Detected"
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | project-away content
      | extend _processed_ip_= 
      CASE 
        WHEN _ip_ = '127.0.0.1' THEN 'LocalHost-LocalHost' 
        WHEN _ip_ = '192.168.0.1' THEN 'default login' 
        ELSE 'Not Detected'
      END
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{
    "_ip_": "192.168.0.1",
    "Index": "900000003",
    "_processed_ip_": "default login",
    "__time__": "1713259557"
}

字符串替换

替换日志中的指定字符串。输入：

hello,how old are you? nice to meet you

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_string_replace
    SourceKey: content
    Method: const
    Match: "how old are you?"
    ReplaceString: ""
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend content=replace(content, 'how old are you?', '')
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{    "content": "hello, nice to meet you",
    "__time__": "1713260499"
}

数据编码与解码

Base64

对日志进行 Base64 加密。输入：

this is a test log

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_base64_encoding
    SourceKey: content
    NewKey: content1
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend content1=to_base64(cast(content as varbinary))
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{    "content": "this is a test log",
    "content1": "dGhpcyBpcyBhIHRlc3QgbG9n",
    "__time__": "1713318724"
}

MD5

对日志进行 MD5 加密。输入：

hello,how old are you? nice to meet you

原有插件：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_string_replace
    SourceKey: content
    Method: const
    Match: "how old are you?"
    ReplaceString: ""
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend content1=lower(to_hex(md5(cast(content as varbinary))))
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{
    "content": "this is a test log",
    "content1": "4f3c93e010f366eca78e00dc1ed08984",
    "__time__": "1713319673"
}

新增能力项

数学计算

输入：4。

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend val = cast(content as double)
      | extend power_test = power(val, 2)
      | extend round_test = round(val)
      | extend sqrt_test = sqrt(val)
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{
    "content": "4",
    "power_test": 16.0,
    "round_test": 4.0,
    "sqrt_test": 2.0,
    "val": 4.0,
    "__time__": "1713319673"
}

URL 计算

URL 编码解码

输入：

https://homenew.console.aliyun.com/home/dashboard/ProductAndService

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend encoded = url_encode(content)
      | extend decoded = url_decode(encoded)
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{
    "content": "https://homenew.console.aliyun.com/home/dashboard/ProductAndService",
    "decoded": "https://homenew.console.aliyun.com/home/dashboard/ProductAndService",
    "encoded": "https%3A%2F%2Fhomenew.console.aliyun.com%2Fhome%2Fdashboard%2FProductAndService",
    "__time__": "1713319673"
}

URL 提取

输入：

https://sls.console.aliyun.com:443/lognext/project/dashboard-all/logsearch/nginx-demo?accounttraceid=d6241a173f88471c91d3405cda010ff5ghdw

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend host = url_extract_host(content)
      | extend query = url_extract_query(content)
      | extend path = url_extract_path(content) 
      | extend protocol = url_extract_protocol(content) 
      | extend port = url_extract_port(content) 
      | extend param = url_extract_parameter(content, 'accounttraceid')
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{
    "content": "https://sls.console.aliyun.com:443/lognext/project/dashboard-all/logsearch/nginx-demo?accounttraceid=d6241a173f88471c91d3405cda010ff5ghdw",
    "host": "sls.console.aliyun.com",
    "param": "d6241a173f88471c91d3405cda010ff5ghdw",
    "path": "/lognext/project/dashboard-all/logsearch/nginx-demo",
    "port": "443",
    "protocol": "https",
    "query": "accounttraceid=d6241a173f88471c91d3405cda010ff5ghdw",
    "__time__": "1713319673"
}

比较&逻辑运算符

输入：

{"num1": 199, "num2": 10, "num3": 9}

SPL：

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | extend compare_result = cast(num1 as double) > cast(num2 as double) AND cast(num2 as double) > cast(num3 as double)
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

输出：

{    "compare_result": "true",
    "content": "{\"num1\": 199, \"num2\": 10, \"num3\": 9}",
    "num1": "199",
    "num2": "10",
    "num3": "9",
    "__time__": "1713319673"
}

其他

更多能力请参考：https://help.aliyun.com/zh/sls/user-guide/function-overview

欢迎大家补充更多 iLogtail SPL 实践案例！

标签：__,Type,content,iLogtail,SPL,time,日志,true,processor
From： https://www.cnblogs.com/alisystemsoftware/p/18503068

相关文章

  • 麒麟操作系统中的系统日志如何帮助你了解电脑运行情况
    ​麒麟操作系统中的系统日志是了解电脑运行状况的关键工具：1.识别并定位系统问题；2.监测系统运行趋势；3.审计系统安全状况；4.优化电脑性能；5.理解应用程序行为。了解日志的基础知识是向高效问题解决迈出的第一步。1.识别并定位系统问题麒麟操作系统的日志记录包含了系统启动、运行......
  • Spring中使用MDC和traceId实现日志链路追踪
    前言在系统出现问题时，我们常需要对日志进行分析，而如果日志打印时没有明确的标识，且日志内容不同线程很多时，很难找出哪段日志是我们需要的。针对这一问题，我们可以考虑使用MDC来实现日志链路追踪，迅速找到所需要的日志信息。当然，这也十分适合当下流行的微服务，特别是上下游节点有多个......
  • JSON日志处理 | 基于SparkSql实现
    目录0 主要JSON处理函数1JSON处理函数使用2案例分析3小结0 主要JSON处理函数get_json_object:提取单个JSON字段json_tuple:同时提取多个JSON字段from_json:JSON字符串转结构化数据to_json:结构化数据转JSON字符串schema_of_json:推断JSONs......
  • 使用mysqlbinlog 备份 binlog日志文件
    使用mysqlbinlog备份二进制日志文件默认情况下，mysqlbinlog读取二进制日志文件并以文本格式显示其内容。这使您能够更轻松地检查文件中的事件并重新执行它们（例如，通过将输出用作mysql的输入）。mysqlbinlog可以直接从本地文件系统读取日志文件，或者，--read-from-remote-server它可......
  • SpringBoot入门到精通（十三）日志：别小看它，否则吃亏的是自己！学会你也可以设计架构
    别小看他，当你面对的时候，就会知道，多么痛的领悟！如何在SpringBoot中使用Logback记录详细的日志？整合LogBack，Log4J...等，是不是很多方法！但需要注意，我讲的可能和你是一样的，但也是不一样的。常见日志级别：高---低排列TRACE：描述：最详细的日志级别，通常用于开发和调试阶段......
  • 2024/10/24日 日志 --》关于Mybatis的学习笔记整理 - 环境与性质
    步入了Mybatis的学习之中，以下为其相关内容的细化笔记整理点击查看代码--MyBatis--·MyBatis是一款优秀的持久层框架，用于简化JDBC开发--·官网：https://mybatis.net.cn/ --持久层：--·负责将数据保存到数据库的那一层代码--JavaEE三层架构：表现层、业务层、持久层分......
  • Linux 定时清理日志
    在Linux系统中，你可以使用cron作业来定时运行一个清理日志的脚本。以下是一个简单的示例，包括一个用于清理日志的脚本和一个用于设置定时任务的cron作业。1.创建清理日志的脚本首先，创建一个脚本文件，例如cleanup_logs.sh，并赋予其可执行权限。这个脚本将删除指定目录下超过......
  • linux硬盘日志清理
    日志文件清理#!/bin/bash#定义需要检查的目录directories=("/home/logs/archived"\"/data/logs")#获取当前硬盘分区的使用情况df_output=$(df--output=pcent/data|tail-n1|tr-d'%')#获取当前时间current_time=$(date"+%Y-%m-%d%H:......
  • 如何将rust日志输出到android终端
    本博客所有文章除特别声明外，均采用CCBY-NC-SA4.0许可协议。转载请注明来自唯你背景在Rust中，使用println!打印日志时，输出实际上是发送到标准输出（stdout），而AndroidLogcat专门用于处理和显示应用程序的日志信息，此环境下标准输出实现被重新定义。这意味着Rust日志输出不......
  • 一个著名的日志系统是怎么设计出来的
    原文：一个著名的日志系统是怎么设计出来的1前言Java帝国在诞生之初就提供了集合、线程、IO、网络等常用功能，从C和C++领地那里吸引了大量程序员过来加盟，但是却有意无意地忽略了一个重要的功能：输出日志。对于这一点，IO大臣其实非常清楚，日志是个很重要的东西，因为程序运行起......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99