Fixing gitlab 12.0 after OS updates
Unfortunately I’m currently stuck with gitlab 12.0.x as I want to use mysql, my server is still 32bit and only has 8G of RAM for all services it’s running. I do plan to move gitlab to a docker container on a x86_64 server but at this time I’m stuck with what I have.
Now most of the ruby packages for gitlab are installed for the gitlab user, however some are shared system and even then some of the local ones are linked against system libs if they’ve a native component.
On occasion gitlab won’t work correctly and needs a recompile of components. The below runs as the git user after stopping gitlab. There’s some slight changes to the official docs to ensure the ruby components are rebuilt and not reusing the potentially broken already installed ones.
I’m not sure if the backup would work prior to doing this but it’s advisable to try before stopping the service.
Let’s get to it.
Backup - as git user.
$ bundle exec rake gitlab:backup:create RAILS_ENV=production
Stop the service - probably best as root
$ service gitlab stop
Reinstall all the gitlab components - as git user.
$ cd /home/git/gitlab-shell/
$ bin/compile
$ cd /home/git/gitlab-workhorse
$ make clean
$ make
$ cd /home/git/gitaly/
$ make clean
$ cd /home/git/gitaly/ruby && bundle install --redownload --deployment
$ cd ..
$ make
$ cd /home/git/gitlab-pages/
$ make
$ cd /home/git/gitlab
$ bundle install --redownload --deployment --without development test aws kerberos
$ bundle clean
$ bundle exec rake db:migrate RAILS_ENV=production
$ bundle exec rake gettext:compile RAILS_ENV=production
$ bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile RAILS_ENV=production NODE_ENV=production NO_SOURCEMAPS=true
$ bundle exec rake cache:clear RAILS_ENV=production
Start the gitlab service back up - again probably as root
service gitlab start
Check gitlab is ok - as git user
$ cd /home/git/gitlab
$ bundle exec rake gitlab:env:info RAILS_ENV=production
System information
System: Gentoo 2.6
Current User: git
Using RVM: no
Ruby Version: 2.5.7p206
Gem Version: 3.1.2
Bundler Version:2.1.4
Rake Version: 12.3.2
Redis Version: 5.0.7
Git Version: 2.24.1
Sidekiq Version:5.2.7
Go Version: go1.12.15 linux/386
GitLab information
Version: 12.0.12
Revision: ce125b970e4
Directory: /home/git/gitlab
DB Adapter: MySQL
DB Version: 10.2.29-MariaDB-log
URL: https://git.stevewilson.co.uk
HTTP Clone URL: https://git.stevewilson.co.uk/some-group/some-project.git
SSH Clone URL: git@git.stevewilson.co.uk:some-group/some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 9.3.0
Repository storage paths:
- default: /home/git/repositories
GitLab Shell path: /home/git/gitlab-shell
Git: /usr/bin/git
$ cd /home/git/gitlab
$ bundle exec rake gitlab:check RAILS_ENV=production
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0)
Running /home/git/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /home/git/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... yes
Init script up-to-date? ... yes
Projects have namespace: ...
( redacted multiple - "Group / Project ... yes" )
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.5.7)
Git version >= 2.21.0 ? ... yes (2.24.1)
Git user has default SSH configuration? ... yes
Active users: ... 2
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Opening the website login and other functionality should be working again.
Rolling the above commands run as the git user into a single script with error handling gives me this. Please ensure you try to take a backup and stop the service before running and then start after. This includes the fix for gpgme to use system libraries
#!/bin/bash -x
# © - Steve Wilson 2019-
# taken from https://phaq.phunsites.net/2010/11/22/trap-errors-exit-codes-and-line-numbers-within-a-bash-script/
# trap handler: print location of last error and process it further
#
function my_trap_handler()
{
MYSELF="$0" # equals to my script name
LASTLINE="$1" # argument 1: last line of error occurence
LASTERR="$2" # argument 2: error code of last command
echo "${MYSELF}: line ${LASTLINE}: exit status of last command: ${LASTERR}"
# do additional processing: send email or SNMP trap, write result to database, etc.
}
# trap commands with non-zero exit code
#
trap 'my_trap_handler ${LINENO} $?' ERR
# lets have some common sense with the shell
# ref: https://coderwall.com/p/fkfaqq/safer-bash-scripts-with-set-euxo-pipefail
# removed -x option to remove debuging
set -euox pipefail
#rm -rfv /home/git/gitlab/vendor/ /home/git/gitaly/ruby/vendor/ || exit 1
cd /home/git/gitlab-shell/ || exit 1
bin/compile || exit 1
cd /home/git/gitlab-workhorse || exit 1
make clean || exit 1
make || exit 1
cd /home/git/gitaly/ || exit 1
make clean || exit 1
cd ruby/ || exit 1
bundle install --redownload --deployment || exit 1
cd .. || exit 1
make || exit 1
cd /home/git/gitlab-pages/ || exit 1
make || exit 1
cd /home/git/gitlab || exit 1
if ! ( grep -q 'BUNDLE_BUILD__GPGME: "--use-system-libraries"' .bundle/config ) ; then
echo 'BUNDLE_BUILD__GPGME: "--use-system-libraries"' >> .bundle/config
fi
bundle install --redownload --deployment --without development test aws kerberos || exit 1
bundle clean || exit 1
bundle exec rake db:migrate RAILS_ENV=production || exit 1
bundle exec rake gettext:compile RAILS_ENV=production || exit 1
bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile RAILS_ENV=production NODE_ENV=production NO_SOURCEMAPS=true || exit 1
bundle exec rake cache:clear RAILS_ENV=production || exit 1