使用 Tengine 作为反向代理服务器，发现 Tengine 日志中接口请求过慢，需要绘制 grafana 展示详细的接口情况 ------------------------------------------------------------------------------------------------------------------------------------
解决思路#
-
第一版解决思路，修改 Tengine 中 access log 日志格式，转行为 json 写入 elk 中，grafana 从 elk 中获取日志信息绘图
- access log 日志信息不全，grafana 绘图有些数据无法获取
- access log 中 request_time 字段格式不对，无法进行聚合以及求 AVG，计算接口平均耗时等
- 其他的一些问题，比如 我们 elk 中有 Tengine 和 k8s nginx 日志，获取数据从所以所有的 日志中获取，而我们只希望从 Tengine 日志获取
- 基于上述的原因放弃了第一版解决思路
-
access log 日志格式
# nginx.conf
log_format main_new escape=json '{ "@timestamp": "$time_iso8601", '
'"remote_addr": "$remote_addr",'
'"remote_port": "$remote_port",'
'"remote_user": "$remote_user",'
'"scheme": "$scheme",'
'"server_protocol": "$server_protocol",'
'"http_host": "$http_host",'
'"request_uri": "$request_uri",'
'"request_method": "$request_method",'
'"request_time": "$request_time",'
'"request_length": "$request_length",'
'"response_status": "$status",'
'"http_cookie": "$http_cookie",'
'"body_bytes_sent": "$body_bytes_sent",'
'"http_referer": "$http_referer",'
'"http_user_agent": "$http_user_agent",'
'"http_x_forwarded_for": "$http_x_forwarded_for",'
'"upstream_addr": "$upstream_addr",'
'"upstream_status": "$upstream_status",'
'"upstream_connect_time": "$upstream_connect_time",'
'"upstream_response_time": "$upstream_response_time"}';
# default.conf
access_log /var/log/nginx/prod-access.log main_new;
- 第二版解决思路，使用 nginx-module-vts 模块暴露接口代理情况，并且此模块自带 exporter 对接 prometheus
调整 Tengine 安装，编译进 nginx-module-vts 模块#
修改 Dockerfile 重新打镜像#
- 修改项
--add-module=modules/ngx_http_upstream_session_sticky_module \
--add-module=modules/nginx-module-vts-0.2.2 \ # 新增
&& curl -L "https://github.com/openresty/headers-more-nginx-module/archive/v0.33.tar.gz" -o more.tar.gz \
&& curl -L "https://github.com/vozlt/nginx-module-vts/archive/refs/tags/v0.2.2.tar.gz" -o more2.tar.gz \ # 新增
&& tar -zxC /usr/src/tengine-$TENGINE_VERSION/modules -f more.tar.gz \
&& tar -zxC /usr/src/tengine-$TENGINE_VERSION/modules -f more2.tar.gz \ # 新增
&& rm more.tar.gz \
&& rm more2.tar.gz \ # 新增
- 完整的 Dockerfile 文件
FROM alpine:3.17
ENV TENGINE_VERSION 2.4.0
# nginx: https://git.io/vSIyj
RUN rm -rf /var/cache/apk/* && \
rm -rf /tmp/*
ENV CONFIG "\
--prefix=/etc/nginx \
--sbin-path=/usr/sbin/nginx \
--modules-path=/usr/lib/nginx/modules \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--http-client-body-temp-path=/var/cache/nginx/client_temp \
--http-proxy-temp-path=/var/cache/nginx/proxy_temp \
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
--http-scgi-temp-path=/var/cache/nginx/scgi_temp \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_realip_module \
--with-http_addition_module \
--with-http_sub_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_mp4_module \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_random_index_module \
--with-http_secure_link_module \
--with-http_stub_status_module \
--with-http_auth_request_module \
--with-http_xslt_module=dynamic \
--with-http_image_filter_module=dynamic \
--with-http_geoip_module=dynamic \
--with-threads \
--with-stream \
--with-stream_ssl_module \
--with-stream_ssl_preread_module \
--with-stream_realip_module \
--with-stream_geoip_module=dynamic \
--with-http_slice_module \
--with-mail \
--with-mail_ssl_module \
--with-compat \
--with-file-aio \
--with-http_v2_module \
--add-module=modules/ngx_http_upstream_check_module \
--add-module=modules/headers-more-nginx-module-0.33 \
--add-module=modules/ngx_http_upstream_session_sticky_module \
--add-module=modules/nginx-module-vts-0.2.2 \
"
RUN addgroup -S nginx \
&& adduser -D -S -h /var/cache/nginx -s /sbin/nologin -G nginx nginx \
&& adduser -u 82 -D -S -G www-data www-data \
&& apk add --no-cache --virtual .build-deps \
gcc \
libc-dev \
make \
openssl-dev \
pcre-dev \
zlib-dev \
linux-headers \
curl \
libxslt-dev \
gd-dev \
geoip-dev \
&& curl -L "https://github.com/alibaba/tengine/archive/$TENGINE_VERSION.tar.gz" -o tengine.tar.gz \
&& mkdir -p /usr/src \
&& tar -zxC /usr/src -f tengine.tar.gz \
&& rm tengine.tar.gz \
&& cd /usr/src/tengine-$TENGINE_VERSION \
&& curl -L "https://github.com/openresty/headers-more-nginx-module/archive/v0.33.tar.gz" -o more.tar.gz \
&& curl -L "https://github.com/vozlt/nginx-module-vts/archive/refs/tags/v0.2.2.tar.gz" -o more2.tar.gz \
&& tar -zxC /usr/src/tengine-$TENGINE_VERSION/modules -f more.tar.gz \
&& tar -zxC /usr/src/tengine-$TENGINE_VERSION/modules -f more2.tar.gz \
&& rm more.tar.gz \
&& rm more2.tar.gz \
&& ls -l /usr/src/tengine-$TENGINE_VERSION/modules \
&& ./configure $CONFIG --with-debug \
&& make -j$(getconf _NPROCESSORS_ONLN) \
&& mv objs/nginx objs/nginx-debug \
&& mv objs/ngx_http_xslt_filter_module.so objs/ngx_http_xslt_filter_module-debug.so \
&& mv objs/ngx_http_image_filter_module.so objs/ngx_http_image_filter_module-debug.so \
&& mv objs/ngx_http_geoip_module.so objs/ngx_http_geoip_module-debug.so \
&& mv objs/ngx_stream_geoip_module.so objs/ngx_stream_geoip_module-debug.so \
&& ./configure $CONFIG \
&& make -j$(getconf _NPROCESSORS_ONLN) \
&& make install \
&& rm -rf /etc/nginx/html/ \
&& mkdir /etc/nginx/conf.d/ \
&& mkdir -p /usr/share/nginx/html/ \
&& install -m644 html/index.html /usr/share/nginx/html/ \
&& install -m644 html/50x.html /usr/share/nginx/html/ \
&& install -m755 objs/nginx-debug /usr/sbin/nginx-debug \
&& install -m755 objs/ngx_http_xslt_filter_module-debug.so /usr/lib/nginx/modules/ngx_http_xslt_filter_module-debug.so \
&& install -m755 objs/ngx_http_image_filter_module-debug.so /usr/lib/nginx/modules/ngx_http_image_filter_module-debug.so \
&& install -m755 objs/ngx_http_geoip_module-debug.so /usr/lib/nginx/modules/ngx_http_geoip_module-debug.so \
&& install -m755 objs/ngx_stream_geoip_module-debug.so /usr/lib/nginx/modules/ngx_stream_geoip_module-debug.so \
&& ln -s ../../usr/lib/nginx/modules /etc/nginx/modules \
&& strip /usr/sbin/nginx* \
&& strip /usr/lib/nginx/modules/*.so \
&& rm -rf /usr/src/tengine-$NGINX_VERSION \
\
# Bring in gettext so we can get `envsubst`, then throw
# the rest away. To do this, we need to install `gettext`
# then move `envsubst` out of the way so `gettext` can
# be deleted completely, then move `envsubst` back.
&& apk add --no-cache --virtual .gettext gettext \
&& mv /usr/bin/envsubst /tmp/ \
\
&& runDeps="$( \
scanelf --needed --nobanner --format '%n#p' /usr/sbin/nginx /usr/lib/nginx/modules/*.so /tmp/envsubst \
| tr ',' '\n' \
| sort -u \
| awk 'system("[ -e /usr/local/lib/" $1 " ]") == 0 { next } { print "so:" $1 }' \
)" \
&& apk add --no-cache --virtual .nginx-rundeps $runDeps \
&& apk del .build-deps \
&& apk del .gettext \
&& mv /tmp/envsubst /usr/local/bin/ \
\
# Bring in tzdata so users could set the timezones through the environment
# variables
&& apk add --no-cache tzdata \
# Bring in curl and ca-certificates to make registering on DNS SD easier
&& apk add --no-cache curl ca-certificates \
# forward request and error logs to docker log collector
&& ln -sf /dev/stdout /var/log/nginx/access.log \
&& ln -sf /dev/stderr /var/log/nginx/error.log
COPY nginx.conf /etc/nginx/nginx.conf
EXPOSE 80 443
STOPSIGNAL SIGTERM
CMD ["nginx", "-g", "daemon off;"]
- 制作
Docker镜像
cd tengine
docker build -t harbor.evescn.com/devops/tengine:2.4.0-nginx-module-vts .
docker push harbor.evescn.com/devops/tengine:2.4.0-nginx-module-vts
nginx.conf文件
user nginx;
# This number should be, at maximum, the number of CPU cores on your system.
worker_processes auto;
error_log /var/log/nginx/error.log error;
pid /var/run/nginx.pid;
events {
# The effective method, used on Linux 2.6+, optmized to serve many clients with each thread.
use epoll;
# Determines how many clients will be served by each worker process.
worker_connections 4000;
# Accept as many connections as possible, after nginx gets notification about a new connection.
multi_accept on;
}
http {
#################
# 开启vts #
#################
vhost_traffic_status_zone;
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;}
include /etc/nginx/mime.types;
default_type application/octet-stream;
#long time
check_shm_size 5M;
# Allow the server to close the connection after a client stops responding.
reset_timedout_connection on;
client_header_timeout 15;
# Send the client a "request timed out" if the body is not loaded by this time.
client_body_timeout 10;
# If the client stops reading data, free up the stale client connection after this much time.
send_timeout 15;
# Timeout for keep-alive connections. Server will close connections after this time.
keepalive_timeout 30;
# Number of requests a client can make over the keep-alive connection.
keepalive_requests 30;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
#################
# 新增日志格式 #
#################
log_format main_new escape=json '{ "@timestamp": "$time_iso8601", '
'"remote_addr": "$remote_addr",'
'"remote_port": "$remote_port",'
'"remote_user": "$remote_user",'
'"scheme": "$scheme",'
'"server_protocol": "$server_protocol",'
'"http_host": "$http_host",'
'"request_uri": "$request_uri",'
'"request_method": "$request_method",'
'"request_time": "$request_time",'
'"request_length": "$request_length",'
'"response_status": "$status",'
'"http_cookie": "$http_cookie",'
'"body_bytes_sent": "$body_bytes_sent",'
'"http_referer": "$http_referer",'
'"http_user_agent": "$http_user_agent",'
'"http_x_forwarded_for": "$http_x_forwarded_for",'
'"upstream_addr": "$upstream_addr",'
'"upstream_status": "$upstream_status",'
'"upstream_connect_time": "$upstream_connect_time",'
'"upstream_response_time": "$upstream_response_time"}';
client_body_buffer_size 128k;
client_max_body_size 40m;
proxy_read_timeout 180s;
# Compression.
gzip on;
gzip_min_length 10240;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml;
gzip_disable "msie6";
# Sendfile copies data between one FD and other from within the kernel.
sendfile on;
# Don't buffer data-sends (disable Nagle algorithm).
tcp_nodelay on;
# Causes nginx to attempt to send its HTTP response head in one packet, instead of using partial frames.
tcp_nopush on;
# Hide web server information
server_tokens off;
server_info off;
server_tag off;
# redirect server error pages to the static page
error_page 404 /404.html;
error_page 500 502 503 504 /50x.html;
include /etc/nginx/*.d/*.conf;
#######################
# 暴露 vts 端口和路径 #
#######################
server {
listen 9112;
location /vts {
vhost_traffic_status_display;
vhost_traffic_status_display_format html;
}
}
}
- 启动服务
# cat docker-compose.yml
version: '3'
services:
rabbitmq:
image: harbor.evescn.com/devops/tengine:2.4.0-nginx-module-vts
hostname: tengine
container_name: tengine
ports:
- "80:80"
- "443:443"
# 新增监控页面接口
- "9112:9112"
volumes:
# 配置文件目录
- /data/tengine/conf.d/:/etc/nginx/conf.d/
# ssl 证书文件
- /data/tengine/certs/:/etc/nginx/certs/
# 日志文件
- /data/tengine/logs/:/var/log/nginx/
restart: always
# docker-compose up -d
# docker-compose ls
NAME STATUS CONFIG FILES
tengine running(1) /data/docker-compose/tengine/docker-compose.yml
访问监控页面测试#
web页面
http://IP:9112/vts/
exporter页面
http://IP:9112/vts/format/prometheus
Grafana 绘图#
- 中文（简体）
- 中文（繁体）
- 丹麦语
- 乌克兰语
- 乌尔都语
- 亚美尼亚语
- 俄语
- 保加利亚语
- 克罗地亚语
- 冰岛语
- 加泰罗尼亚语
- 匈牙利语
- 卡纳达语
- 印地语
- 印尼语
- 古吉拉特语
- 哈萨克语
- 土耳其语
- 威尔士语
- 孟加拉语
- 尼泊尔语
- 布尔语(南非荷兰语)
- 希伯来语
- 希腊语
- 库尔德语
- 德语
- 意大利语
- 拉脱维亚语
- 挪威语
- 捷克语
- 斯洛伐克语
- 斯洛文尼亚语
- 旁遮普语
- 日语
- 普什图语
- 毛利语
- 法语
- 波兰语
- 波斯语
- 泰卢固语
- 泰米尔语
- 泰语
- 海地克里奥尔语
- 爱沙尼亚语
- 瑞典语
- 立陶宛语
- 缅甸语
- 罗马尼亚语
- 老挝语
- 芬兰语
- 英语
- 荷兰语
- 萨摩亚语
- 葡萄牙语
- 西班牙语
- 越南语
- 阿塞拜疆语
- 阿姆哈拉语
- 阿尔巴尼亚语
- 阿拉伯语
- 韩语
- 马尔加什语
- 马拉地语
- 马拉雅拉姆语
- 马来语
- 马耳他语
- 高棉语