Prometheus 运维工具 Promtool （四）TSDB 功能

标签：__ name 运维 -- flink taskmanager Prometheus pod Promtool

Prometheus 运维工具 Promtool （四）TSDB 功能_Promtool

Promtool 在 TSDB 方面一个有 6 个子命令，分别用来进行写性能测试、分析、列出其中的块、dump、从 OpenMetric 导入数据块、为新的记录规则创建数据块，接下来我们依次看一下。

这大概是第一篇写 Promtool TSDB 方面的文章了。在网络上都没有找到相关资料。

写性能测试

Promtool 可以对 Prometheus 进行写的性能测试，命令参数如下：

[root@Erdong-Test ~]# ./promtool tsdb bench write --help
usage: promtool tsdb bench write [<flags>] [<file>]

Run a write performance benchmark.

Flags:
  -h, --help                 Show context-sensitive help (also try --help-long and --help-man).
      --version              Show application version.
      --enable-feature= ...  Comma separated feature names to enable (only PromQL related). See
                             https://prometheus.io/docs/prometheus/latest/feature_flags/ for the options and more
                             details.
      --out="benchout"       Set the output path.
      --metrics=10000        Number of metrics to read.
      --scrapes=3000         Number of scrapes to simulate.

Args:
  [<file>]  Input file with samples data, default is (../../tsdb/testdata/20kseries.json).

首先从官网下载测试所需要的数据 20kseries.json 文件，这个文件在源码仓库的 /tsdb/testdata/ 下，可以通过这个命令来下载

wget https://raw.githubusercontent.com/prometheus/prometheus/main/tsdb/testdata/20kseries.json

下载好以后，指定 metrics 和 scrapes 两个参数进行测试

[root@Erdong-Test ~]# ./promtool tsdb bench write --metrics=10000 --scrapes=3000 ./20kseries.json
level=info ts=2022-07-27T13:21:25.546626055Z caller=head.go:493 msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2022-07-27T13:21:25.546734166Z caller=head.go:536 msg="On-disk memory mappable chunks replay completed" duration=11.815µs
level=info ts=2022-07-27T13:21:25.546766084Z caller=head.go:542 msg="Replaying WAL, this may take a while"
level=info ts=2022-07-27T13:21:25.547101874Z caller=head.go:613 msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2022-07-27T13:21:25.547131383Z caller=head.go:619 msg="WAL replay completed" checkpoint_replay_duration=38.26µs wal_replay_duration=315.491µs total_replay_duration=409.177µs
level=info ts=2022-07-27T13:21:25.549132675Z caller=db.go:1467 msg="Compactions disabled"
>> start stage=readData
>> completed stage=readData duration=145.973395ms
>> start stage=ingestScrapes
ingestion completed
>> completed stage=ingestScrapes duration=3.628682202s
 > total samples: 30000000
 > samples/sec: 8.267435217071414e+06
>> start stage=stopStorage
>> completed stage=stopStorage duration=1.522008202s

这个测试会在当前目录生成一个 benchout 的文件夹，里边是测试生成的文件。这个路径也可以通过参数来进行修改，生成到其他地方或者其他名称。

简单来看，这次读数据消耗了 145ms ，存储数据消耗了 1.5s 。

关于这个结果解读，我们后续在进行专门的讲解。

TSDB 分析

Promtool 可以对 Prometheus 的数据块进行分析，分析 churn 、label 对的基数和压缩效率，命令参数如下：

[root@Erdong-Test ~]# ./promtool tsdb analyze --help
usage: promtool tsdb analyze [<flags>] [<db path>] [<block id>]

Analyze churn, label pair cardinality and compaction efficiency.

Flags:
  -h, --help                 Show context-sensitive help (also try --help-long and --help-man).
      --version              Show application version.
      --enable-feature= ...  Comma separated feature names to enable (only PromQL related). See
                             https://prometheus.io/docs/prometheus/latest/feature_flags/ for the options and more
                             details.
      --limit=20             How many items to show in each list.
      --extended             Run extended analysis.

Args:
  [<db path>]   Database path (default is data/).
  [<block id>]  Block to analyze (default is the last block).

在分析的时候可以指定每个列表最多显示多少个项，必须要指定数据的 data 目录，block id 可以指定也可以不指定，不指定的话会分析最新的那个 block 。执行命令以后可以看到如下输出。

[root@Erdong-Test ~]# ./promtool tsdb analyze ./prometheus/data/
Block ID: 01G8ZXKAHRGF7BV0FTQYZ18FH5
Duration: 59m55.713s
Series: 606677
Label names: 26
Postings (unique label pairs): 13190
Postings entries (total label pairs): 7819776

Label pairs most involved in churning:
329487 workload_user_cattle_io_workloadselector=deployment-appops-pushgateway-flink
329487 pod_template_hash=6d695f75cf
329487 kubernetes_namespace=appops
62797 kubernetes_pod_name=pushgateway-flink-6d695f75cf-2xhfh
58524 kubernetes_pod_name=pushgateway-flink-6d695f75cf-8fq58
52495 kubernetes_pod_name=pushgateway-flink-6d695f75cf-d76n7
51544 kubernetes_pod_name=pushgateway-flink-6d695f75cf-z4bjb
22074 kubernetes_pod_name=pushgateway-flink-6d695f75cf-9jgdk
21516 kubernetes_pod_name=pushgateway-flink-6d695f75cf-96hpq
20426 kubernetes_pod_name=pushgateway-flink-6d695f75cf-9twr2
20047 kubernetes_pod_name=pushgateway-flink-6d695f75cf-ngzwz
11819 kubernetes_pod_name=pushgateway-flink-6d695f75cf-8hv9z
8242 kubernetes_pod_name=pushgateway-flink-6d695f75cf-pddrc
3783 __name__=flink_taskmanager_Status_JVM_Memory_Heap_Used
3783 __name__=flink_taskmanager_job_task_checkpointAlignmentTime
3783 __name__=flink_taskmanager_Status_JVM_ClassLoader_ClassesLoaded
3783 __name__=flink_taskmanager_Status_JVM_Threads_Count
3783 __name__=flink_taskmanager_Status_Shuffle_Netty_UsedMemorySegments
3783 __name__=flink_taskmanager_Status_Flink_Memory_Managed_Used
3783 __name__=flink_taskmanager_job_task_Shuffle_Netty_Output_Buffers_outputQueueLength

Label names most involved in churning:
329487 kubernetes_namespace
329487 kubernetes_pod_name
329487 workload_user_cattle_io_workloadselector
329487 __name__
329487 pod_template_hash
329487 job
314352 instance
314050 host
314050 tm_id
185403 task_attempt_num
185403 task_name
185403 subtask_index
185403 job_name
185403 task_id
185403 task_attempt_id
185403 job_id
15134 operator_name
15134 operator_id
66 method
55 quantile

Most common label pairs:
606565 kubernetes_namespace=appops
606565 workload_user_cattle_io_workloadselector=deployment-pushgateway-flink
606565 pod_template_hash=6d695f75cf
87063 kubernetes_pod_name=pushgateway-flink-6d695f75cf-ngzwz
87063 kubernetes_pod_name=pushgateway-flink-6d695f75cf-9jgdk
87062 kubernetes_pod_name=pushgateway-flink-6d695f75cf-96hpq
87062 kubernetes_pod_name=pushgateway-flink-6d695f75cf-9twr2
67920 kubernetes_pod_name=pushgateway-flink-6d695f75cf-2xhfh
62700 kubernetes_pod_name=pushgateway-flink-6d695f75cf-8fq58
54174 kubernetes_pod_name=pushgateway-flink-6d695f75cf-d76n7
53304 kubernetes_pod_name=pushgateway-flink-6d695f75cf-z4bjb
11892 kubernetes_pod_name=pushgateway-flink-6d695f75cf-8hv9z
8325 kubernetes_pod_name=pushgateway-flink-6d695f75cf-pddrc
6965 __name__=flink_taskmanager_job_task_Shuffle_Netty_Input_Buffers_inputFloatingBuffersUsage
6965 __name__=flink_taskmanager_job_task_numBuffersOut
6965 __name__=flink_taskmanager_job_task_isBackPressured
6965 __name__=flink_taskmanager_Status_JVM_ClassLoader_ClassesLoaded
6965 __name__=flink_taskmanager_job_task_numBuffersOutPerSecond
6965 __name__=push_time_seconds
6965 __name__=flink_taskmanager_job_task_buffers_inPoolUsage

Label names with highest cumulative label value length:
50890 task_name
42890 tm_id
34890 operator_id
34890 task_id
34890 task_attempt_id
34890 job_id
21890 job_name
18928 job
18890 operator_name
13890 host
5972 __name__
3890 task_attempt_num
3890 subtask_index
3100 instance
340 kubernetes_pod_name
40 revision
38 handler
35 workload_user_cattle_io_workloadselector
19 quantile
16 method

Highest cardinality labels:
1012 instance
1002 job
1000 operator_id
1000 task_id
1000 task_attempt_num
1000 host
1000 task_name
1000 tm_id
1000 job_id
1000 job_name
1000 task_attempt_id
1000 subtask_index
1000 operator_name
137 __name__
10 kubernetes_pod_name
7 handler
7 quantile
4 method
3 code
2 version

Highest cardinality metric names:
6965 my_batch_job_duration_seconds
6965 flink_taskmanager_Status_Flink_Memory_Managed_Used
6965 flink_taskmanager_Status_JVM_CPU_Load
6965 flink_taskmanager_Status_JVM_CPU_Time
6965 flink_taskmanager_Status_JVM_ClassLoader_ClassesLoaded
6965 flink_taskmanager_Status_JVM_ClassLoader_ClassesUnloaded
6965 flink_taskmanager_Status_JVM_GarbageCollector_PS_MarkSweep_Count
6965 flink_taskmanager_Status_JVM_GarbageCollector_PS_MarkSweep_Time
6965 flink_taskmanager_Status_JVM_GarbageCollector_PS_Scavenge_Count
6965 flink_taskmanager_Status_JVM_GarbageCollector_PS_Scavenge_Time
6965 flink_taskmanager_Status_JVM_Memory_Direct_Count
6965 flink_taskmanager_Status_JVM_Memory_Direct_MemoryUsed
6965 flink_taskmanager_Status_JVM_Memory_Direct_TotalCapacity
6965 flink_taskmanager_Status_JVM_Memory_Heap_Committed
6965 flink_taskmanager_Status_JVM_Memory_Heap_Max
6965 flink_taskmanager_Status_JVM_Memory_Heap_Used
6965 flink_taskmanager_Status_JVM_Memory_Mapped_Count
6965 flink_taskmanager_Status_JVM_Memory_Mapped_MemoryUsed
6965 flink_taskmanager_Status_JVM_Memory_Mapped_TotalCapacity
6965 flink_taskmanager_Status_JVM_Memory_Metaspace_Committed

列出 TSDB 数据块

使用 Promtool 还可以列出当前 Prometheus 的所有数据块。命令的参数如下：

[root@Erdong-Test ~]# ./promtool tsdb list --help
usage: promtool tsdb list [<flags>] [<db path>]

List tsdb blocks.

Flags:
  -h, --help                 Show context-sensitive help (also try --help-long and --help-man).
      --version              Show application version.
      --enable-feature= ...  Comma separated feature names to enable (only PromQL related). See
                             https://prometheus.io/docs/prometheus/latest/feature_flags/ for the options and more
                             details.
  -r, --human-readable       Print human readable values.

Args:
  [<db path>]  Database path (default is data/).

执行这个命令我们来看一下效果，建议加上 -r 参数

[root@Erdong-Test ~]# ./promtool tsdb list -r ./prometheus-pushgateway/data/
BLOCK ULID                  MIN TIME                       MAX TIME                       DURATION     NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE
01G8XB6J42S6FTDMAPYV30CTR3  2022-07-26 12:00:03 +0000 UTC  2022-07-26 13:00:00 +0000 UTC  59m56.018s   78421200     435860       435412       92MiB217KiB450B
01G8XEMDR374M816NZ2NMNS5AQ  2022-07-26 13:00:03 +0000 UTC  2022-07-26 14:00:00 +0000 UTC  59m56.018s   78421200     435860       435412       92MiB217KiB825B
01G8XJ29C2JVV8PEQFR748C1QR  2022-07-26 14:00:03 +0000 UTC  2022-07-26 15:00:00 +0000 UTC  59m56.018s   78421200     435860       435412       94MiB125KiB158B
01G8XNG502K46CF1960R1FYEAA  2022-07-26 15:00:03 +0000 UTC  2022-07-26 16:00:00 +0000 UTC  59m56.018s   78421200     435860       435412       92MiB388KiB448B
01G8XRY0M2CV8J323VWM393JTZ  2022-07-26 16:00:03 +0000 UTC  2022-07-26 17:00:00 +0000 UTC  59m56.018s   78421200     435860       435412       93MiB978KiB543B
01G8XWBW82NGHBPP6QPZMJ6XAM  2022-07-26 17:00:03 +0000 UTC  2022-07-26 18:00:00 +0000 UTC  59m56.018s   78421200     435860       435412       93MiB979KiB590B
01G8XZSQW2WY7SG70Z40CW2CZV  2022-07-26 18:00:03 +0000 UTC  2022-07-26 19:00:00 +0000 UTC  59m56.018s   78421200     435860       435412       94MiB719KiB630B

通过这个命令我们可以看到 Block ID、数据存储的开始时间和结束时间，一共存储了多长时间段数据，Sample 的数量，Chunk 的数量，Serie 的数量、以及 Block 的大小。

Dump

Promtool 还行可以进行 dump，只不过是从 TSDB 中 dump Sample 数据出来。

[root@Erdong-Test ~]# ./promtool tsdb dump --help
usage: promtool tsdb dump [<flags>] [<db path>]

Dump samples from a TSDB.

Flags:
  -h, --help                 Show context-sensitive help (also try --help-long and --help-man).
      --version              Show application version.
      --enable-feature= ...  Comma separated feature names to enable (only PromQL related). See
                             https://prometheus.io/docs/prometheus/latest/feature_flags/ for the options and more
                             details.
      --min-time=-9223372036854775808
                             Minimum timestamp to dump.
      --max-time=9223372036854775807
                             Maximum timestamp to dump.

Args:
  [<db path>]  Database path (default is data/).

对于这个命令一定要指定数据目录 data 的路径，另外最大最小时间也指定一下，否则会 dump 数据库中所有的 Sample 数据。

我们执行一下这个命令

[root@Erdong-Test ~]# ./promtool tsdb dump --min-time=1658927217000 --max-time=1658930818000 ../prometheus-pushgateway/data/

这个命令 dump 出来的数据好像只会在屏幕输出，并不会输出到文件，dump 的时候记得人为导入到某个文件。

从 OpenMetric 导入数据块

Promtool 工具还可以从 OpenMetrics 输入导入 sample 数据并生成 TSDB 数据块。这个命令的使用场景是在不同的监控系统或者时序数据库之间迁移数据使用的，先将对应的数据转换成 OpenMetric 格式，然后将 OpenMetric 格式的数据导入到 Prometheus 的 TSDB 数据库中。

这个命令的参数如下:

[root@Erdong-Test ~]# ./promtool tsdb create-blocks-from openmetrics --help
usage: promtool tsdb create-blocks-from openmetrics <input file> [<output directory>]

Import samples from OpenMetrics input and produce TSDB blocks. Please refer to the storage docs for more details.

Flags:
  -h, --help                 Show context-sensitive help (also try --help-long and --help-man).
      --version              Show application version.
      --enable-feature= ...  Comma separated feature names to enable (only PromQL related). See
                             https://prometheus.io/docs/prometheus/latest/feature_flags/ for the options and more
                             details.
  -r, --human-readable       Print human readable values.
  -q, --quiet                Do not print created blocks.

Args:
  <input file>          OpenMetrics file to read samples from.
  [<output directory>]  Output directory for generated blocks.

这个我没有对应的实验环境，参数提示让参考存储文档了解更多细节，就是这个文档 https://prometheus.io/docs/prometheus/latest/storage/ 。大家自行学习一下，我有环境了在写文章。

为新的记录规则创建数据块

当创建新的记录规则以后，这个新的记录规则是没有历史数据的。记录规则所产生的数据只在创建时开始存储到数据库中，Promtool 可以创建记录规则的历史数据。

[root@Erdong-Test ~]# ./promtool tsdb create-blocks-from rules --help
usage: promtool tsdb create-blocks-from rules --start=START [<flags>] <rule-files>...

Create blocks of data for new recording rules.

Flags:
  -h, --help                 Show context-sensitive help (also try --help-long and --help-man).
      --version              Show application version.
      --enable-feature= ...  Comma separated feature names to enable (only PromQL related). See
                             https://prometheus.io/docs/prometheus/latest/feature_flags/ for the options and
                             more details.
  -r, --human-readable       Print human readable values.
  -q, --quiet                Do not print created blocks.
      --url=http://localhost:9090
                             The URL for the Prometheus API with the data where the rule will be backfilled
                             from.
      --start=START          The time to start backfilling the new rule from. Must be a RFC3339 formatted date
                             or Unix timestamp. Required.
      --end=END              If an end time is provided, all recording rules in the rule files provided will
                             be backfilled to the end time. Default will backfill up to 3 hours ago. Must be a
                             RFC3339 formatted date or Unix timestamp.
      --output-dir="data/"   Output directory for generated blocks.
      --eval（232, 232, 232); background-color: rgb(249, 249, 249);">[root@Erdong-Test ~]# ./promtool tsdb create-blocks-from rules \
    --start 1617079873 \
    --end 1617097873 \
    --url http://mypromserver.com:9090 \
    rules.yaml rules2.yaml

命令 promtool tsdb create-blocks-from rules 的输出结果为一个目录，该目录中包含记录规则文件中所有规则的历史规则数据块。默认情况下，输出目录为data/。为了利用这些新的块数据，必须将这些块移动到正在运行的 Prometheus 实例数据目录下，然后在 Prometheus 实例启动的时候将参数 --storage.tsdb.allow-overlapping-blocks 设为启用。之后 Prometheus 会在在下一次压缩运行时，将新块与现有块合并。这样记录规则文件生成的历史数据就可以在 Prometheus 中进行查询了。

创建记录规则的历史数据有一些限制，

如果多次执行生成历史数据，且开始时间和结束时间重叠，则每次运行记录规则生成历史数据时都会创建包含相同数据的块。
记录规则文件中的所有规则都将被评估。
如果在记录规则文件中设置了时间间隔，该时间间隔优先于 tsdb create-blocks-from rules 命令中的--eval（并将依赖数据移动到 Prometheus 服务器数据目录，以便从 Prometheus API 访问它)。

总结

4 篇的一个小的系列已经完成。第 4 篇在写的过程中，除了官方文档，没有其他资料可以参考，如果有错误的地方请联系我修改。

标签：__,name,运维,--,flink,taskmanager,Prometheus,pod,Promtool
From： https://blog.51cto.com/erdong/5780729