Security Server Cluster Ubuntu 14.04 to 18.04 Upgrade.

Owned by Petteri Kivimäki
Last updated: Apr 25, 2019

This document describes the steps required for upgrading a Security Server cluster from Ubuntu 14.04 LTS  to Ubuntu 18.04 LTS in-place. Please read carefully through the whole document before starting the upgrade process. It is assumed that the reader is familiar with the Ubuntu Linux distribution and has experience of Ubuntu release upgrades.

Upgrading a system that uses a hardware security module has not been tested. Please verify that the HSM is compatible with Ubuntu 18.04 and  check the HSM module documentation.

The upgrade process is based on

with some additional steps due to PostgreSQL database version upgrade from 9.3 to 10. Please review both documents before continuing.

Upgrade process

  • Ensure that the X-Road software on all nodes is at version 6.20.0/1 or 6.21.0 (N.B.! all nodes must have the same version), and all Ubuntu packages are updated.
    • If necessary, update the cluster, following instructions in [1], section 7.
  • Pause the database and configuration synchronization on the slave nodes ([1], section 7.2.1)

Upgrading the master node

  • Set the master node to maintenance mode or manually disable it from the external load balancer.
  • Upgrade the master node software, using the upgrade process described in [2]

    • At the "Upgrade the database" step, upgrade both database clusters* (main and serverconf) to version 10
      (*) a PostgreSQL "cluster" is a collection of databases served by one postgres instance, should not be confused with a security server cluster

      Example
      $ sudo pg_lsclusters
Ver Cluster 	Port Status Owner    Data directory               		Log file
9.3 main    	5432 online postgres /var/lib/postgresql/9.3/main 		/var/log/postgresql/postgresql-9.3-main.log
9.3 serverconf 	5433 online postgres /var/lib/postgresql/9.3/serverconf	/var/log/postgresql/postgresql-9.3-serverconf.log
9.5 main    	5434 online postgres /var/lib/postgresql/9.5/main 		/var/log/postgresql/postgresql-9.5-main.log
10  main    	5435 online postgres /var/lib/postgresql/10/main  		/var/log/postgresql/postgresql-10-main.log

$ sudo pg_dropcluster --stop 9.5 main
$ sudo pg_dropcluster --stop 10 main

$ sudo pg_upgradecluster --method=upgrade --link 9.3 main
$ sudo pg_upgradecluster --method=upgrade --link 9.3 serverconf
    • After upgrading the databases, continue the upgrade process as described in [2]

  • If the master node was disabled manually from the external load balancer, verify that the master node is working and enable it in the load balancer

Upgrading the slave nodes

After successfully upgrading the master, slave nodes can be upgraded one by one.
(Alternatively, install new Ubuntu 18.04 slave node(s) as described in [1])

  • Gracefully disable the slave node from the load balancer, either manually or using the health check maintenance mode (see [1], section 7.2.3)
  • Upgrade the slave node software following /wiki/spaces/XRDDEV/pages/13178021 until the Upgrade the database step
  • Back up the file /var/lib/postgresql/9.3/serverconf/recovery.conf
    • e.g sudo cp /var/lib/postgresql/9.3/serverconf/recovery.conf /tmp/

  • Upgrade only the 9.3 main database and drop the other databases (including serverconf)

    Since the serverconf database is small, dropping and recreating it a straightforward option.
    See https://www.postgresql.org/docs/10/pgupgrade.html for more information and other options.

    Example
    $ sudo pg_lsclusters
Ver Cluster 	Port Status Owner    Data directory               		Log file
9.3 main    	5432 online postgres /var/lib/postgresql/9.3/main 		/var/log/postgresql/postgresql-9.3-main.log
9.3 serverconf 	5433 online postgres /var/lib/postgresql/9.3/serverconf	/var/log/postgresql/postgresql-9.3-serverconf.log
9.5 main    	5434 online postgres /var/lib/postgresql/9.5/main 		/var/log/postgresql/postgresql-9.5-main.log
10  main    	5435 online postgres /var/lib/postgresql/10/main  		/var/log/postgresql/postgresql-10-main.log

$ sudo pg_dropcluster --stop 9.5 main
$ sudo pg_dropcluster --stop 10 main
$ sudo pg_upgradecluster --method=upgrade --link 9.3 main

  • Recreate the serverconf database

    sudo pg_dropcluster --stop 9.3 serverconf
sudo pg_createcluster -p 5433 10 serverconf

    Follow the instructions in configuring the slave instance for replication in [1] to set up the database replication.

    • Note that you can find the <master> and <nodename> information in the recovery.conf -file

  • Set up the configuration synchronization using systemd, as described in [1], section 5.2.1.
  • Remove the old configuration synchronization:
    sudo rm /etc/cron.d/xroad-state-sync /etc/init/xroad-sync.conf /etc/init/xroad-sync-wait.conf

  • Update the X-Road package repository to point to the Ubuntu 18.04 packages

    sudo apt-add-repository -r "deb https://artifactory.niis.org/xroad-release-deb trusty-current main"
sudo apt-add-repository "deb https://artifactory.niis.org/xroad-release-deb bionic-current main"

    (alternatively, edit /etc/apt/sources.list directly)

  • Make sure that the database replication is up to date. The following should return t:

    Example
    $ sudo -iu postgres psql -p 5433 -tc 'select pg_last_wal_replay_lsn() = pg_last_wal_receive_lsn()'
 t

  • Upgrade the packages on the slave node to the Ubuntu 18.04 version.
    sudo apt update && sudo apt full-upgrade

  • Enable the shared configuration synchronization on the slave node:
    sudo rm /var/tmp/xroad/sync-disabled
    service xroad-sync start

  • restart the X-Road services and wait until the slave node is healthy.

  • See [1], section 6 for instructions about verifying that the upgrade was successful.
  • After the node is healthy, enable the slave node in the load balancer if you manually disabled it.

Troubleshooting