首页 > 其他分享 >[ES] ES问题汇总

[ES] ES问题汇总

时间：2024-09-11 18:16:08浏览次数：11  
标签：java dli.233 h0 flink 汇总 问题 r28 1.12 ES

Q: 写入失败，字段数超出1000个的限制

问题描述

...
2024-08-26 10:37:06,775 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - Sink: 设备历史状态写入ES (2/2)#5 (0c9d2bb8575b51dced4ba167a09ec08a) switched from RUNNING to FAILED.
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] in index [device_status_snapshot_history_202408] has been exceeded]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1907) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1884) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1641) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1568) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
	at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:985) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
	at com.xxx.sink.DeviceStatusEsHistorySinkFunction.invoke(DeviceStatusEsHistorySinkFunction.java:87) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
	at com.xxx.sink.DeviceStatusEsHistorySinkFunction.invoke(DeviceStatusEsHistorySinkFunction.java:25) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
	at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:57) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:191) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
	at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:204) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
	at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:174) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:412) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:199) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:634) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:598) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:765) [flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:580) [flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_372]
	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://10.37.19.116:9200], URI [/device_status_snapshot_history_202408/_doc?timeout=1m], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [device_status_snapshot_history_202408] has been exceeded"}],"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [device_status_snapshot_history_202408] has been exceeded"},"status":400}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:318) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:262) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1628) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1568) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
		at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:985) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
		at com.xxx.sink.DeviceStatusEsHistorySinkFunction.invoke(DeviceStatusEsHistorySinkFunction.java:87) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
		at com.xxx.sink.DeviceStatusEsHistorySinkFunction.invoke(DeviceStatusEsHistorySinkFunction.java:25) ~[blob_p-ba3774ed58ac44003b7e38716877c41d2a9a31f5-2f4e02b4169d9934c84e916e8cbb11b7:?]
		at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:57) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
		at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:191) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
		at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:204) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
		at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:174) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
		at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
		at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:412) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
		at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:199) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
		at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:634) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
		at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:598) ~[flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
		at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:765) [flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
		at org.apache.flink.runtime.taskmanager.Task.run(Task.java:580) [flink-dist_2.11-1.12.2-h0.cbu.dli.233.r28.jar:1.12.2-h0.cbu.dli.233.r28]
		at java.lang.Thread.run(Thread.java:750) [?:1.8.0_372]
2024-08-26 10:37:06,776 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - Call stack:
    at java.lang.Thread.getStackTrace(Thread.java:1564)
    at org.apache.flink.runtime.taskmanager.Task.transitionState(Task.java:1065)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:837)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:580)
    at java.lang.Thread.run(Thread.java:750)
...

问题分析&解决方法

方法1：放大字段数的限制

  • 设置索引的最大字段数为 2000
PUT {myIndexName}/_settings
{
	"index.mapping.total_fields.limit": 2000
}

设置成功的响应:

{
  "acknowledged" : true
}

查看是否设置成功

GET {myIndexName}/_settings

X 参考文献

X 参考文献

标签：java,dli.233,h0,flink,汇总,问题,r28,1.12,ES
From： https://www.cnblogs.com/johnnyzen/p/18408675

相关文章

  • Rest-assured框架详解
    Rest-assured框架官网官网url: https://rest-assured.io/一、接口测试介绍-什么是接口测试本质上基于某种协议，发送请求给服务器，服务器返回响应数据，对响应数据进行分析，判断和我们的预期是否一致，从而验证功能是否正确。-为什么做接口测试更早发现问题，降低研发成本减少回归......
  • mysql 连接失败：message from server: "Host '192.168.xx.xxx' is not allowed to conn
    前言mysql连接失败：messagefromserver:"Host'192.168.xx.xxx'isnotallowedtoconnecttothisMySQLserver"解决错误信息表明你尝试从IP地址192.168.xx.xxx连接到MySQL服务器，但是该IP地址没有被授权连接权限。为了解决这个问题，你需要确保你的MySQL用户权限......
  • Leetcode 2453. Destroy Sequential Targets | rust 实现
    题解问题描述给定一个整数数组nums和一个整数space，我们需要找到一个目标值，使得该目标值在nums中的出现次数最多。如果有多个目标值出现次数相同，则返回最小的目标值。解题思路哈希表统计：使用哈希表map来统计每个seed%space的出现次数，题干中给出的等式等价为nums[n......
  • 1.Kubernetes简介
    ......
  • 为什么说Scrum是有效的？(Why does Scrum work)
    在《敏捷规模化实践集》一书中，JimWang老师为此书序言"WhydoesScrumwork"。书中围绕"从团队到企业，我们可否自下而上地规模化敏捷方法；大规模敏捷需要做系统的改变；规模化敏捷技术与框架概览......”展开。在此书中，我们将一起探索如何为更庞大的敏捷框架铺设基石。这些基石的实践......
  • web 开发（5）- HttpRequest对象
    json发送数据接收数据defjson(request):body=request.body#print(body)body_str=body.decode()print(body_str)print(type(body_str))importjsonbody_dict=json.loads(body_str)print(body_dict)returnHttpResponse('json')请求代码SessionSession......
  • Ethereum学习笔记 ---- 使用 Remix 调试功能理解 bytes 在 memory 中的布局
    目录编写合约编译、部署、调用合约调试交易1.调用函数bytesInMemory()，分析bytes的MemoryLayout执行RETUR前的最后一刻，stack快照如下执行完毕时刻的MemoryLayout对MemoryLayout的分析2.调用函数bytesArrayInMemmory(4)，分析bytes[]的MemoryLayout执行RETUR前......
  • 嵌套集合模型(Nested set model)介绍
    嵌套集合模型(Nestedsetmodel)介绍pilishen /更新于5年前本文翻译自维基百科Nestedsetmodel 此文档是 nestedset-无限分类正确姿势的扩展阅读 nestedsetmodel(嵌套集合模型)是一种在关系型数据库中表示nestedsets（嵌套集合） 的特殊技术。[nestedsets]通常......
  • 电脑键盘功能基础知识汇总
    电脑键盘是我们平时使用电脑时最常用的输入设备之一，了解键盘的基本功能，不见能提升工作效率，还能让我们在使用电脑时更加得心应手。今天要和大家探讨一下电脑键盘的基础功能，并介绍一些小技巧，帮助大家更好的利用这个工具。在深入了解键盘基本功能之前，我们先一起认识一下键盘的组成部分......
  • 电脑键盘功能基础知识汇总
    电脑键盘是我们平时使用电脑时最常用的输入设备之一，了解键盘的基本功能，不见能提升工作效率，还能让我们在使用电脑时更加得心应手。今天要和大家探讨一下电脑键盘的基础功能，并介绍一些小技巧，帮助大家更好的利用这个工具。在深入了解键盘基本功能之前，我们先一起认识一下键盘的组成部分......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99