个人博客地址：使用RSyslog将Nginx Access Log写入Kafka | 一张假钞的真实世界
环境说明
- CentOS Linux release 7.3.1611
- kafka_2.12-0.10.2.2
- nginx/1.12.2
- rsyslog-8.24.0-34.el7.x86_64.rpm
创建测试Topic
$ ./kafka-topics.sh --zookeeper 192.168.72.25:2181/kafka --create --topic develop-test-topic --partitions 10 --replication-factor 3RSyslog安装
一般系统自带RSyslog服务无需另外安装。但是因为数据需要通过RSyslog的omkafka模块写入到Kafka，而omkafka在RSyslog的v8.7.0+版本才支持，所以要看当前系统中RSyslog的版本是否需要升级。使用以下命令查看：
# rsyslogd -v
rsyslogd 7.4.7, compiled with:
FEATURE_REGEXP: Yes
FEATURE_LARGEFILE: No
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
Runtime Instrumentation (slow code): No
uuid support: Yes
See http://www.rsyslog.com for more information.执行以下命令安装：
# sudo yum install rsyslog安装依赖关系如下：
添加omkafka模块
# yum install rsyslog-kafkaRSyslog Client Nginx配置
注意，Nginx 1.7.1之后才支持syslog的方式处理日志。具体配置项参见官网文档Logging to syslog。
Nginx配置主要是日志格式和Access Log配置项：
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
server {
listen 11000;
location / {
proxy_pass http://10.16.0.144:11000;
access_log syslog:server=localhost,facility=local7,tag=nginx11000Root,severity=info main;
}
}RSyslog Server端配置
RSyslog的主配置文件/etc/rsyslog.conf，其中会包含引入/etc/rsyslog.d下扩展名为conf的配置文件。
修改配置文件/etc/rsyslog.conf将下面两行前面的注释去掉：
$ ModLoad imudp
$ UDPServerRun 514在/etc/rsyslog.d目录下创建rsyslog_nginx_kafka_cluster.conf，配置内容如下：
module(load="imudp")
input(type="imudp" port="514")
# nginx access log ==> rsyslog server(local) ==> kafka
module(load="omkafka")
template(name="nginx-11000-root" type="string" string="%msg%")
if $inputname == "imudp" then {
if ($programname == "nginx11000Root") then
action(type="omkafka"
template="nginx-11000-root"
broker=["192.168.72.10:9092","192.168.72.20:9092","192.168.72.25:9092","192.168.72.26:9092","192.168.72.27:9092","192.168.72.48:9092","192.168.72.55:9092","192.168.72.80:9092","192.168.72.81:9092","192.168.72.97:9092"]
topic="develop-test-topic"
partitions.auto="on"
confParam=[
"socket.keepalive.enable=true"
]
)
}
:rawmsg, contains, "nginx11000Root" ~联调测试
启动RSyslog服务：
# service rsyslog start
Redirecting to /bin/systemctl start rsyslog.service遇到的问题
syslog tag 只能包含字母和数字
# nginx -t
nginx: [emerg] syslog "tag" only allows alphanumeric characters and underscore in /etc/nginx/conf.d/jx-11000-jenkins149-36-144.conf:7
nginx: configuration file /etc/nginx/nginx.conf test failed‘omkafka’ is unknown
RSyslog中没有包含omkafka模块，需要另外安装。查看/var/log/messages日志信息会有以下提示：
# tail -f messages
Mar 15 15:13:40 192-168-72-29 systemd: Started System Logging Service.
Mar 15 15:13:40 192-168-72-29 rsyslogd: could not load module '/usr/lib64/rsyslog/omkafka.so', dlopen: /usr/lib64/rsyslog/omkafka.so: cannot open shared object file: No such file or directory [v8.24.0-34.el7 try http://www.rsyslog.com/e/2066 ]
Mar 15 15:13:40 192-168-72-29 rsyslogd: could not load module '/usr/lib64/rsyslog/omkafka.so', dlopen: /usr/lib64/rsyslog/omkafka.so: cannot open shared object file: No such file or directory [v8.24.0-34.el7 try http://www.rsyslog.com/e/2066 ]
Mar 15 15:13:40 192-168-72-29 rsyslogd: module name 'omkafka' is unknown [v8.24.0-34.el7 try http://www.rsyslog.com/e/2209 ]
Mar 15 15:13:40 192-168-72-29 rsyslogd: module name 'omkafka' is unknown [v8.24.0-34.el7 try http://www.rsyslog.com/e/2209 ]CentOS 6.5升级Rsyslog
CentOS 6.5自带的RSyslog版本是rsyslogd 5.8.10。按照以下方式安装新版本：
# cd /etc/yum.repos.d/
# wget http://rpms.adiscon.com/v8-stable/rsyslog.repo
# yum install rsyslog