Coding with Titans

so breaking things happens constantly, but never on purpose

Recover GitLab after PostgreSQL 9.6.2 failed upgrade

Recently I described, how to keep in sync with GitLab latest releases on the Raspberry Pi. And of course it lead me to a problem after unsuccessful update to version 8.17-rc3. This release has the PostgreSQL 9.6.2 embedded and apparently everything got broken in the middle of the upgrade process. After all my tries to revert to previous GitLab version, lots of time wasted (yes, each call to apt-get upgrade or install specified GitLab version was taking several hours!), I still stayed with totally unresponsive instance.

Then again I came back to basics.

  • I rebooted the device

    sudo shutdown –r now

  • Installed latest GitLab-CE

    sudo apt-get upgrade

  • Asked it to revert PostgreSQL to previous version

    sudo gitlab-ctl revert-pg-upgrade

  • At this stage, it was showing some strange errors about inability to connect to psql service via TCP/IP, that was pretty strange. So I checked the status.

    sudo gitlab-ctl status

run: gitlab-workhorse: (pid 3099) 7362s; run: log: (pid 604) 1487541047s
run: logrotate: (pid 11405) 160s; run: log: (pid 602) 1487541047s
run: nginx: (pid 1917) 7686s; run: log: (pid 608) 1487541047s
**down: postgresql: 1s**
run: redis: (pid 1930) 7685s; run: log: (pid 603) 1487541047s
run: sidekiq: (pid 3079) 7364s; run: log: (pid 607) 1487541047s
run: unicorn: (pid 3257) 7282s; run: log: (pid 606) 1487541047s
  • And then logs!

    sudo gitlab-ctl tail postgresql

  • And it was continuously repeating an error, while parsing line 210 of the configuration file, what was all the time stopping the service.

    sudo vi /var/opt/gitlab/postgresql/data/postgresql.con

  • Since it was about replication, I commented it out and service started normally.

  • Then manually upgraded PostgreSQL

    sudo gitlab-ctl pg-upgrade

  • This time all went OK.