gitlab 升级

背景

现在的gitlab版本是13.0.1,目标是 14.9.5 因为，这个版本及以后支持禅道集成。

升级过程

1. 先备份数据

gitlab-rake gitlab:backup:create  
#有两个文件需要手工备份
#gitlab.rb路径：/etc/gitlab/gitlab.rb
#gitlab-secrets.json路径：/etc/gitlab/gitlab-secrets.json
#备份文件名称为1681869286_2023_04_19_13.0.1_gitlab_backup.tar
#运行备份数据到命令，备份文件默认在/var/opt/gitlab/backups ，如果没有请查找/etc/gitlab/gitlab.rb backup相关配置 容器的话也可以在对应的目录下找到 如 /usr/local/src/lyy/gitlab/data/backups

2. 先升级到最近的主版本，之后再依次进行升级，在docker hub找到对应版本的镜像 然后修改docker-compose.yaml文件

image: 'gitlab/gitlab-ce:13.0.14-ce.0'

3. 然后执行升级命令

docker-compose down && docker-compose up -d  

4. 启动成功，确认正常进行下一个版本更新,版本顺序

docker pull gitlab/gitlab-ce:13.0.14-ce.0
docker pull gitlab/gitlab-ce:13.8.8-ce.0
docker pull gitlab/gitlab-ce:13.12.15-ce.0
docker pull gitlab/gitlab-ce:14.0.12-ce.0
docker pull gitlab/gitlab-ce:14.3.6-ce.0
docker pull gitlab/gitlab-ce:14.9.5-ce.0
docker pull gitlab/gitlab-ce:14.10.5-ce.0
docker pull gitlab/gitlab-ce:15.0.5-ce.0
docker pull gitlab/gitlab-ce:15.1.6-ce.0

遇到的问题

1. 按照直觉上来就升级到目录版本，无法启动，提示请按照流程升级。
   
2. 权限问题 执行如下命令

docker exec -it gitlab update-permissions

3. 问题如下

gitlab | ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR:  relation "services" does not exist
gitlab | LINE 8:  WHERE a.attrelid = '"services"'::regclass

首先参考链接5，执行了如下语句

gitlab-rake gitlab:check SANITIZE=true 
# 不能登录了 error 500
gitlab-rake db:migrate:status 

然后参考链接6 使用gitlab-psql执行了修改sql语句

gitlab-psql
ALTER TABLE Integrations RENAME TO Services;  

4. 问题如下

gitlab | ==> /var/log/gitlab/gitlab-exporter/current <==
gitlab | 2023-04-19_06:22:49.46623 E, [2023-04-19T06:22:49.466177 #796] ERROR -- : Error connecting to the database: ERROR:  column ci_builds.commands does not exist
gitlab | 2023-04-19_06:22:49.46624 LINE 30:     ci_builds.commands  

执行参考原来的升能，进行了小版本的替换，换成了14.1.8-ce.0。官方给的是此版本修复了这个问题。

5. 问题如下

gitlab | ==> /var/log/gitlab/gitlab-rails/production.log <==
gitlab |   
gitlab | ActionView::Template::Error (PG::UndefinedTable: ERROR:  relation "integrations" does not exist
gitlab | LINE 8:  WHERE a.attrelid = '"integrations"'::regclass  

受第四个问题的影响，和查了相关的资料，就是把之前改过的表再手动改回来

gitlab-psql
ALTER TABLE Services RENAME TO Integrations; 

6. 问题如下

StandardError: An error has occurred, all later migrations canceled:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
  {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"push_event_payloads", :column_name=>"event_id", :job_arguments=>[["event_id"], ["event_id_convert_to_bigint"]]}

参考链接8,9 执行如下语句

# 此命令是报错提示要执行的
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]']
sudo gitlab-rake db:migrate
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart  

参考文章

1. https://docs.gitlab.com/ee/update/
2. https://www.cnblogs.com/faberbeta/p/16341131.html
3. https://www.cnblogs.com/faberbeta/p/16341131.html
4. https://www.cnblogs.com/colder/p/16650365.html
5. https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6352
6. https://stackoverflow.com/questions/71927588/gitlab-ce-500-error-after-upgrade-downgrade
7. https://docs.gitlab.com/ee/update/plan_your_upgrade.html
8. https://forum.gitlab.com/t/upgrading-gitlab-from-14-0-to-14-3-2-fails-due-to-migration-issues/59479/8
9. https://docs.gitlab.com/ee/update/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
10. https://gitlab.com/gitlab-org/gitlab/-/issues/26845