首页 > 其他分享 >graylog 索引模版处理

graylog 索引模版处理

时间:2022-10-31 20:22:10浏览次数:132  
标签:index templateName text 索引 graylog template 模版 type

graylog 默认分词只支持对应几个固定的字段,如果需要自定义索引信息,就可以使用模版能力,默认包含了一个graylog-internal,order 为-1 但是我们可以扩展

默认索引信息

  • 查询信息
<wiz_code_mirror>              
GET <endpoint>/_template/graylog-internal?pretty'
   

效果

<wiz_code_mirror>              
{
    "graylog-internal": {
        "order": -1,
        "index_patterns": [
            "graylog_*"
        ],
        "settings": {
            "index": {
                "analysis": {
                    "analyzer": {
                        "analyzer_keyword": {
                            "filter": "lowercase",
                            "tokenizer": "keyword"
                        }
                    }
                }
            }
        },
        "mappings": {
            "_source": {
                "enabled": true
            },
            "dynamic_templates": [
                {
                    "internal_fields": {
                        "mapping": {
                            "type": "keyword"
                        },
                        "match_mapping_type": "string",
                        "match": "gl2_*"
                    }
                },
                {
                    "store_generic": {
                        "mapping": {
                            "type": "keyword"
                        },
                        "match_mapping_type": "string"
                    }
                }
            ],
            "properties": {
                "gl2_processing_timestamp": {
                    "format": "uuuu-MM-dd HH:mm:ss.SSS",
                    "type": "date"
                },
                "gl2_accounted_message_size": {
                    "type": "long"
                },
                "gl2_receive_timestamp": {
                    "format": "uuuu-MM-dd HH:mm:ss.SSS",
                    "type": "date"
                },
                "full_message": {
                    "fielddata": false,
                    "analyzer": "standard",
                    "type": "text"
                },
                "streams": {
                    "type": "keyword"
                },
                "source": {
                    "fielddata": true,
                    "analyzer": "analyzer_keyword",
                    "type": "text"
                },
                "message": {
                    "fielddata": false,
                    "analyzer": "standard",
                    "type": "text"
                },
                "timestamp": {
                    "format": "uuuu-MM-dd HH:mm:ss.SSS",
                    "type": "date"
                }
            }
        },
        "aliases": {}
    }
}

调整

  • 模版内容
{
  "template": "graylog_*",
  "index_patterns": ["*"],
  "mappings": {
    "properties": {
      "http_method": {
        "type": "keyword"
      },
      "http_response_code": {
        "type": "long"
      },
      "ingest_time": {
        "type": "date",
        "format": "strict_date_time"
      },
      "took_ms": {
        "type": "long"
      },
      "response_body": {
        "type": "text"
      },
      "request_body": {
        "type": "text"
      },
      "request": {
        "type": "text"
      },
      "http_user_agent": {
        "type": "text"
      }
    }
  }
}
  • 配置
PUT /_template/graylog-custom-mapping?pretty
  • 查看效果
GET /_template/graylog-custom-mapping?pretty

内容

{
  "graylog-custom-mapping": {
    "order": 0,
    "index_patterns": [
      "*"
    ],
    "settings": {},
    "mappings": {
      "properties": {
        "request": {
          "type": "text"
        },
        "http_method": {
          "type": "keyword"
        },
        "ingest_time": {
          "format": "strict_date_time",
          "type": "date"
        },
        "request_body": {
          "type": "text"
        },
        "took_ms": {
          "type": "long"
        },
        "response_body": {
          "type": "text"
        },
        "http_response_code": {
          "type": "long"
        },
        "http_user_agent": {
          "type": "text"
        }
      }
    },
    "aliases": {}
  }
}

代码处理

graylog2-server/src/main/java/org/graylog2/indexer/indices/Indices.java

  • Indices.java
public void ensureIndexTemplate(IndexSet indexSet) {
      final IndexSetConfig indexSetConfig = indexSet.getConfig();
      final String templateName = indexSetConfig.indexTemplateName();
      try {
          final Map<String, Object> template = buildTemplate(indexSet, indexSetConfig);
          if (indicesAdapter.ensureIndexTemplate(templateName, template)) {
              LOG.info("Successfully ensured index template {}", templateName);
          } else {
              LOG.warn("Failed to create index template {}", templateName);
          }
      } catch (IgnoreIndexTemplate e) {
          LOG.warn(e.getMessage());
          if (e.isFailOnMissingTemplate() && !indicesAdapter.indexTemplateExists(templateName)) {
              throw new IndexTemplateNotFoundException(f("No index template with name '%s' (type - '%s') found in Elasticsearch",
                      templateName, indexSetConfig.indexTemplateType().orElse(null)));
          }
      }
  }
  • 不同es 适配
    比如es7 IndicesAdapterES7.java
 
@Override
  public boolean ensureIndexTemplate(String templateName, Map<String, Object> template) {
      final PutIndexTemplateRequest request = new PutIndexTemplateRequest(templateName)
              .source(template);
 
      final AcknowledgedResponse result = client.execute((c, requestOptions) -> c.indices().putTemplate(request, requestOptions),
              "Unable to create index template " + templateName);
 
      return result.isAcknowledged();
  }

具体内部处理实际上是基于了sysjob,相关job 如下

 

 

es 索引模型

  • 写路径

 

 

  • 读路径

 

 

说明

graylog 对于es 索引的管理还是比较方便的,充分利用了es 的能力,实现了比较强大的日志检索

参考资料

https://docs.graylog.org/docs/elasticsearch
https://docs.graylog.org/docs/index-model
https://docs.graylog.org/docs/query-language
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-templates.html
https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog2/indexer/indices/Indices.java
https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog2/indexer/indices/IndicesAdapter.java

标签:index,templateName,text,索引,graylog,template,模版,type
From: https://www.cnblogs.com/rongfengliang/p/16845628.html

相关文章

  • 索引和联合索引
    (1)对一张表来说,如果有一个复合索引on  (col1,col2),就没有必要同时建立一个单索引oncol1。(2)如果查询条件需要,可以在已有单索引 oncol1的情况下,添加复合索引on  (c......
  • tf.gather,取指定维度多个索引的数据
    tensorflow和numpy在数据处理上语法相似但又不完全一样,比如在numpy中想取指定维度的多个指定索引所指向的数据时,直接用一个列表保存索引就能直接取,比如:#b的shape为[2,3,2......
  • 总结一下使用索引的一些建议
    在区分度高的字段上建立索引可以有效的使用索引,区分度太低,无法有效的利用索引,可能需要扫描所有数据页,此时和不使用索引差不多联合索引注意最左匹配原则:必须按照从左......
  • MYSQL索引
    索引的优点索引大大减少了服务器需要扫描的数据量索引可以帮助服务器避免排序和临时表索引可以将随机I/O变成顺序I/O索引只要帮助存储引擎快速查找到记录,带来......
  • 索引接口汇总整理
    ##Neo4j###索引-Btreeindex-Rangeindex-Pointindex-Lookupindex-FullTextindex-Textindex其中Btree为当前默认索引,在neo4j的文档中rangeindex与pointin......
  • SQL之 数据库表字段约束与索引
    第三范式MySQL四种字段约束主键约束非空约束唯一约束创建索引添加和删除索引......
  • k8s资源配置清单模版
    常用配置kind:ServiceapiVersion:v1metadata:name:fastapiapplabels:app:fastapispec:selector:app:fastapiports:-protocol:TCP......
  • 搜索引擎关键字
    目录:基本内容实例浏览器快捷键Google搜索技巧1.基本内容双引号(英文状态下)""代表全匹配搜索,表示包含括号中出现的所有词,连顺序也要完全匹配。减号......
  • PyTorch: 张量的拼接、切分、索引
    本文已收录于Pytorch系列专栏:​​Pytorch入门与实践​​专栏旨在详解Pytorch,精炼地总结重点,面向入门学习者,掌握Pytorch框架,为数据分析,机器学习及深度学习的代码能力打下......
  • 3 栈帧 递归 类成员 静态字段 常量 静态函数 属性 构造函数 析构函数 this readonly
    好记性不如烂笔头目录好记性不如烂笔头栈帧递归=深入了解类==1类成员2成员修饰符的顺序3静态字段4从类的外部访问静态成员4.1静态成员的生存期5静态函数成员6其他......