一、题目
在集群中,数据首先分布在data_hot节点,rollover 设置max_age:3d, max_docs:5,max_size:50gb, 优先级为100。
max_age:15s, forcemarge 段合并,数据迁移到data_warm节点, 副本数为0,优先级为50
max_age:30s, 数据从data_warm迁移到data_cold节点
max_age:60s, 数据删除
运用ilm,创建index template名字为task_ilm_template,匹配模式task_ilm-*,并指定ilm 策略,创建索引task_ilm_00001注意分布情况
二、思考
索引生命周期管理ILM是对索引数据管理的一种自动化方式,主要是通过policy+索引模版结合使用。
通过创建ILM,在policy中可以指定多个阶段,生命周期阶段包括:
-
Hot: The index is actively being updated and queried.
-
Warm: The index is no longer being updated but is still being queried.
-
Cold: The index is no longer being updated and is queried infrequently. The information still needs to be searchable, but it’s okay if those queries are slower.
-
Frozen: The index is no longer being updated and is queried rarely. The information still needs to be searchable, but it’s okay if those queries are extremely slow.
-
Delete: The index is no longer needed and can safely be removed.
每个阶段可以制定不同的动作action,action主要有以下:
-
Rollover: Creates a new write index when the current one reaches a certain size, number of docs, or age.
-
Shrink: Reduces the number of primary shards in an index.
-
Force merge: Triggers a force merge to reduce the number of segments in an index’s shards.
-
Delete: Permanently remove an index, including all of its data and metadata.
通过创建索引模版并配置ILM策略,可以在使用模版创建索引后,对数据起到一个自动化生命周期管理的目的。
三、解题
Step 1、前置工作修改集群
1)集群节点角色配置
- node-1:data_hot, data_content
- node-2:data_warm, data_content
- node-3:data_cold, data_content
2)索引生命周期刷新间隔
对于生产环境非必要配置,主要是为了索引生命周期刷新轮询间隔为1s,在验证时候能快速看到效果。
PUT _cluster/settings
{
"persistent": {
"indices.lifecycle.poll_interval": "1s"
}
}
Step 2、创建ILM策略
可以通过命令行和图形化配置两种方式
【命令行方式】
PUT _ilm/policy/task_test_policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"set_priority": {
"priority": 100
},
"rollover": {
"max_size": "50gb",
"max_primary_shard_size": "50gb",
"max_age": "3d",
"max_docs": 5
}
}
},
"warm": {
"min_age": "15s",
"actions": {
"forcemerge": {
"max_num_segments": 1,
"index_codec": "best_compression"
},
"set_priority": {
"priority": 50
},
"allocate": {
"number_of_replicas": 0
}
}
},
"cold": {
"min_age": "30s",
"actions": {
"readonly": {}
}
},
"delete": {
"min_age": "60s",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}
【图形化方式】
打开kibana在左侧菜单中找到management展开点击stack management
在stack management中找到index lifecycle polices菜单,点击右侧的create ploicy的按钮
配置hot phase
配置 warm phase
配置 cold phase
配置 delete phase,需要在 cold phase中点击 删除 按钮,会增加一个delete phase
Step 3、创建索引模版配置ILM策略
注意:在创建索引模版的时候要指定lifecycle,也就是刚刚创建的ILM策略。此外,还要指定rollover滚动操作别名,缺少别名创建的index也不能正常执行ILM策略。
【命令行方式】
PUT _index_template/task_ilm_template
{
"settings": {
"index": {
"lifecycle": {
"name": "task_test_policy",
"rollover_alias": "task_ilm"
},
"routing": {
"allocation": {
"include": {
"_tier_preference": null
}
}
},
"number_of_shards": "5",
"number_of_replicas": "1"
}
},
"mappings": {
"properties": {
"age": {
"type": "integer"
},
"name": {
"type": "text"
}
}
},
"aliases": {}
}
【图像界面方式】
在左侧菜单中点击data,在右侧主页面中选择index template 标签页,并点击create template按钮
创建索引模版操作如下
此处需要注意:
1)匹配模式最后通配符要使用“-”中杠
2)需要指定配置,重点需要配置数据节点
创建模版中所需的字段及类型
最后点击Create template按钮
Step 4、创建索引验证
注意这里有两个点
1)需要指定索引的别名,并且可以基于别名进行写入操作
2)数据的批量写入操作是基于别名的
# DELETE task_ilm-000001
PUT task_ilm-000001
{
"aliases": {
"task_ilm": {
"is_write_index":true
}
}
}
POST task_ilm/_bulk
{"create":{"_id":1}}
{"name":"key 1","age":1}
{"create":{"_id":2}}
{"name":"key 2","age":2}
{"create":{"_id":3}}
{"name":"key 3","age":3}
{"create":{"_id":4}}
{"name":"key 4","age":4}
{"create":{"_id":5}}
{"name":"key 5","age":5}
POST task_ilm/_bulk
{"create":{"_id":6}}
{"name":"key 6","age":6}
创建后索引获取索引配置
在执行写入第6条数据的时候,数据写入到task_ilm-0002索引,前5条数据在task_ilm-0001索引, 在执行超过15s后,task_ilm-0001索引从node-1的hot节点转移到node-2的warm节点
在执行超过30s后,task_ilm-0001索引从node-2的warm节点转移到node-2的code节点
在执行超过60s后,task_ilm-0001索引node3中删除
四、总结
这个一个综合性比较强的考点,考点内容很多包括ILM,包括索引模版等内容。
整体过程还是比较复杂的,这个需要多练习。
参考资料