首页 > 其他分享 >gitlab 502问题解决

gitlab 502问题解决

时间:2024-02-04 12:11:08浏览次数:27  
标签:run log gitlab pid server file 解决 502

问题现象:

 

Whoops, GitLab is taking too much time to respond. Try refreshing the page, or going back and attempting the action again. Please contact your GitLab administrator if this problem persists.   问题定位分析: 一、查看系统资源使用情况 磁盘满了

gitlab 默认启用 Prometheus,数据存储时长是 15天,经观察磁盘占用较大(我这边平均每天1G)

如果磁盘紧张的情况下可以修改数据保留时长,或直接关闭 Prometheus 监控,修改配置文件的方法如下:

配置文件位置 gitlab/config/gitlab.rb

1、修改保留天数,下面的配置内容默认全部是注释掉的,可以去掉前面的 # ,然后将15d修改为对应所需的数值即可。

prometheus['flags'] = {
'storage.tsdb.path' => "/var/opt/gitlab/prometheus/data",
# 'storage.tsdb.retention.time' => "15d",
'storage.tsdb.retention.time' => "2d",
'config.file' => "/var/opt/gitlab/prometheus/prometheus.yml"

2、直接关闭该功能,将配置文件中的 # prometheus['enable'] = true 取消注释后修改为 prometheus['enable'] = false

# prometheus['enable'] = true
prometheus['enable'] = false
最后重启 gitlab 即可,多余的文件会被自动删除。

 

二、还是502,进一步排查

root@gitlab:/# gitlab-ctl status
run: alertmanager: (pid 347) 15s; run: log: (pid 343) 15s
run: gitaly: (pid 474) 14s; run: log: (pid 329) 15s
run: gitlab-exporter: (pid 356) 15s; run: log: (pid 354) 15s
run: gitlab-kas: (pid 346) 15s; run: log: (pid 338) 15s
run: gitlab-workhorse: (pid 345) 15s; run: log: (pid 336) 15s
run: grafana: (pid 342) 15s; run: log: (pid 333) 15s
run: logrotate: (pid 353) 15s; run: log: (pid 349) 15s
run: nginx: (pid 352) 15s; run: log: (pid 348) 15s
run: postgres-exporter: (pid 330) 15s; run: log: (pid 324) 15s
down: postgresql: 1s, normally up, want up; run: log: (pid 332) 15s
run: prometheus: (pid 355) 15s; run: log: (pid 351) 15s
run: puma: (pid 602) 2s; run: log: (pid 328) 15s
run: redis: (pid 331) 15s; run: log: (pid 325) 15s
run: redis-exporter: (pid 341) 15s; run: log: (pid 334) 15s
run: sidekiq: (pid 344) 15s; run: log: (pid 335) 15s
run: sshd: (pid 37) 26s; run: log: (pid 36) 26s
root@gitlab:/# root@gitlab:~# docker exec -it gitlab bash
root@gitlab:/# gitlab-ctl tail postgresql

 

postgresql没有正常,查看日志

root@gitlab:/# gitlab-ctl tail postgresql
==> /var/log/gitlab/postgresql/state <==

==> /var/log/gitlab/postgresql/current <==
2024-02-04_03:30:08.79426 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:08.79427 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:09.79872 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:09.79874 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:10.80367 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:10.80368 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:11.80861 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:11.80862 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:12.81352 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:12.81353 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:13.81798 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:13.81800 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:14.82294 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:14.82296 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:15.82814 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:15.82817 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:16.83341 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:16.83343 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
^CTraceback (most recent call last):

之前的服务器启动崩溃:如果之前的 PostgreSQL 服务器启动过程中发生了崩溃,可能会留下一个空的锁文件,这会阻止新的 PostgreSQL 实例启动。

查找并删除空的锁文件:在 PostgreSQL 的数据目录中查找名为 postmaster.pid 的文件,该文件是用于跟踪 PostgreSQL 进程的锁文件。如果发现该文件为空,则可以尝试删除它。

cd /var/opt/gitlab/postgresql/data/

rm -fr postmaster.pid 

启动 postgresql服务

root@gitlab:~# gitlab-ctl start postgresql
ok: run: postgresql: (pid 767) 25s

检测服务,服务正常,服务恢复

root@gitlab:~# gitlab-ctl status
run: alertmanager: (pid 343) 91s; run: log: (pid 341) 91s
run: gitaly: (pid 518) 88s; run: log: (pid 315) 91s
run: gitlab-exporter: (pid 332) 91s; run: log: (pid 322) 91s
run: gitlab-kas: (pid 334) 91s; run: log: (pid 325) 91s
run: gitlab-workhorse: (pid 342) 91s; run: log: (pid 338) 91s
run: grafana: (pid 330) 91s; run: log: (pid 319) 91s
run: logrotate: (pid 340) 91s; run: log: (pid 336) 91s
run: nginx: (pid 329) 91s; run: log: (pid 320) 91s
run: postgres-exporter: (pid 324) 91s; run: log: (pid 316) 91s
run: postgresql: (pid 767) 39s; run: log: (pid 318) 91s
run: prometheus: (pid 333) 91s; run: log: (pid 326) 91s
run: puma: (pid 761) 41s; run: log: (pid 317) 91s
run: redis: (pid 335) 91s; run: log: (pid 313) 91s
run: redis-exporter: (pid 331) 91s; run: log: (pid 321) 91s
run: sidekiq: (pid 756) 41s; run: log: (pid 337) 91s
run: sshd: (pid 29) 101s; run: log: (pid 28) 101s
root@gitlab:~#

 

 

 

标签:run,log,gitlab,pid,server,file,解决,502
From: https://www.cnblogs.com/gaoyanbing/p/18005951

相关文章

  • 解决缓存与数据库同步下的同步锁问题之分段锁
    契子  在实际业务会我们会使用第三方的缓存例如:Reids、Memcache等;但是,并且我们在查询使用缓存时都得尽可能的保证缓存的一致性,在读取时得保证尽可能的保证缓存拿到的是数据库的最新数据,那么在实现的逻辑上一般都为这样:1、请求线程先读取缓存实现2、如果缓存没有数据的话触发......
  • Gitlab Prometheus 磁盘空间占用
    gitlab默认启用Prometheus,数据存储时长是15天,经观察磁盘占用较大(我这边平均每天1G)如果磁盘紧张的情况下可以修改数据保留时长,或直接关闭Prometheus监控,修改配置文件的方法如下:配置文件位置gitlab/config/gitlab.rb1、修改保留天数,下面的配置内容默认全部是注释掉的,可以去......
  • 关于 AJAX 请求跨域问题在 Vue 项目中的解决方式
    0.前言关于域,sry刚刚新建文件夹,写好了就换过来;此文为88岁高龄、入门级前端初心者专用笔记;暂时只关心AJAX请求受同源策略的影响及在Vue项目中的解决方式捏;1.必要性1.0你需要知道(1)协议、域名、端口都相同,才为同源;(2)浏览器报跨域错误,并不是服务器没有返回,而......
  • 财务数据处理问题及解决方案分享
    一、平台介绍财务自营计费主要承接京东自营数据在整个供应链中由C端转B端的功能实现,在整个供应链中属于靠后的阶段了,系统主要功能是计费和向B端的汇总。二、问题描述近年来自营计费数据量大增,有百亿+的数据量,一天中汇总占据了一半的数据库资源。1、每天从单表千万W+中定位几万......
  • Linux服务器升级GLIBC失败导致shell不可用的问题解决经历
    转自https://blog.csdn.net/u010549608/article/details/126281354?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522170696599716800182728626%2522%252C%2522scm%2522%253A%252220140713.130102334..%2522%257D&request_id=170696599716800182728626&biz_i......
  • spring声明式事务(@Transactional)开发常犯的几个错误及解决办法
    目前JAVA的微服务项目基本都是SSM结构(即:springCloud+springMVC+Mybatis),而其中Mybatis事务的管理也是交由spring来管理,大部份都是使用声明式事务(@Transactional)来进行事务一致性的管理,然后在实际日常开发过程中,发现很多开发同学都用错了spring声明式事务(@Transactional)或者说使用......
  • VSCode项目中安装npm依赖包失败解决方案
    解决VScode提示:无法将“node”“npm”项识别为cmdlet、函数、脚本文件或可运行程序的名称。请检查名称的拼写,如果包括路径,请确保路径正确,然后再试一次。此方法用于解决使用vscode打开项目文件后,使用npminstall命令安装node_modules依赖包失败的问题方法一:创建新终端窗口;......
  • volatile源码解析【解决可见性(依据happened-befor)有序性(依据内存屏障)】
    @TOC转自极客时间解决内存可见性问题volatile实现原理-源码分析......
  • 一篇文章解决你的无线AP选型难题:从入门到精通
    无线网络覆盖项目中,无线AP的合理选型和部署非常重要。今天给大家安排。这篇文章,给你总结了6类典型的无线组网场所,针对每种场景的特点,给出相应的设备选型和部署的方案,同时整理了一些部署无线AP过程中容易忽略的细节,希望能给你日后组网提供一些帮助。01不同类型的AP,不同的使用环境不......
  • 12个RAG常见痛点及解决方案
    Barnett等人的论文《SevenFailurePointsWhenEngineeringaRetrievalAugmentedGenerationSystem》介绍了RAG的七个痛点,我们将其延申扩展再补充开发RAG流程中常遇到的另外五个常见问题。并且将深入研究这些RAG痛点的解决方案,这样我们能够更好地在日常的RAG开发中避免和解决......