Kafka 日志保留策略(Log Retention Policy)
前言
一两周前测试kafka,创建了topic: data-time
,发布了一部分数据,测试kafka的发布和订阅均正常。
一两周后,也就是现在,再次取订阅topic: data-time
。发现无法订阅了,准备的说是 :订阅失效了。
为了弄清楚前因后果,记录一下...
一、kafka配置文档(server.properties)
摘选 server.properties 的日志保留策略内容
############################# Log Retention Policy #############################
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
按照注释内容理解:kafka的日志保留策略分成两部分,其一是时间角度,其二是空间角度。这两种策略是互相独立的,互不影响的,可以同时存在。
二、时间角度
log.retention.hours=168
kafka默认日志文件保留时长是 168 hours (即7 days),当日志文件从创建时间开始,一旦超过设置的日志保留时长,日志文件可能会被删除
个人理解:kafka后台线程会每隔一段时间(默认5 mins)去检查一下日志文件是否可以删除,当满足删除条件(时长)时,就可以将日志文件删除。
时间参数:
1️⃣ log.retention.hours
2️⃣ log.retention.minutes
3️⃣ log.retention.ms
当log.retention.ms=-1,则日志保留时长是没有限制的(即永久保留)
时间参数优先级:log.retention.ms > log.retention.minutes > log.retention.hours
三、空间角度
log.retention.bytes=1073741824
注意:在kafka配置文档中,这个配置条件是被注释的。
kafka默认日志文件保留大小是 1073741824 bytes (即1 Gb),当日志文件从创建开始,一旦超过设置的日志保留大小,日志文件可能会被删除
个人理解:kafka后台线程会每隔一段时间(默认5 mins)去检查一下日志文件是否可以删除,当满足删除条件(大小)时,就可以将日志文件删除。