现象:偶发性某天应用服务日志数据丢失,查看es无数据
架构:filebeat--->logstash--->es
logstash 上报错现象:
2023-10-13T13:05:14,161][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"srgp-2023.10.13", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x3b1e2406>], :response=>{"index"=>{"_index"=>"srgp-2023.10.13", "_type"=>"_doc", "_id"=>"X41sJ4sBPtPNo0ugp5is", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [logdate] of type [date] in document with id 'X41sJ4sBPtPNo0ugp5is'. Preview of field's value: '2023-10-13 13:05:13,009'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2023-10-13 13:05:13,009] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}
[2023-10-13T13:05:14,161][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"srgp-2023.10.13", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x529a2802>], :response=>{"index"=>{"_index"=>"srgp-2023.10.13", "_type"=>"_doc", "_id"=>"9MlsJ4sBGw4njSrKpxWs", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [logdate] of type [date] in document with id '9MlsJ4sBGw4njSrKpxWs'. Preview of field's value: '2023-10-13 13:05:13,007'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2023-10-13 13:05:13,007] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}}}}
应用日志:
2023-10-13 09:51:14,710 INFO [AYQqkUTKRJufCZpBDujhww] [00534e185d57476498936]xxxxxxxxx
2023-10-13 09:51:14,750 INFO [AYQqkUTKRJufCZpBDujhww] xxxxxx
咨询ChatGpt:
GPT的解决方法也不行,其实上面的配置我的logstash也有,还是报这个错。
出处: ## https://www.cnblogs.com/liucx/
解决方法:
经过排查,logstash获取日志时,会以日志的第一条记录作为es自动创建的映射字段,logdate的类型变为时间类型了,导致无法写入到es,后修改这个索引的模板解决,强制mapping中的logdate设置为text类型
解决啦
标签:index,13,写入,parse,date,type,logstash,es From: https://www.cnblogs.com/liucx/p/17762760.html