Security Server Cluster Ubuntu 18.04 to 20.04 Upgrade.

Owned by Petteri Kivimäki
Last updated: Jan 30, 2021

This document describes the steps required for upgrading a Security Server cluster from Ubuntu 18.04 LTS  to Ubuntu 20.04 LTS in-place. Please read carefully through the whole document before starting the upgrade process. It is assumed that the reader is familiar with the Ubuntu Linux distribution and has experience of Ubuntu release upgrades.

Upgrading a system that uses a hardware security module has not been tested. Please verify that the HSM is compatible with Ubuntu 20.04 and  check the HSM module documentation.

The upgrade process is based on

with some additional steps due to PostgreSQL database version upgrade from 10 to 12. Please review both documents before continuing.

Upgrade process

  • Ensure that the X-Road software on all nodes is at version 6.25.0, and all Ubuntu packages are updated.
    • If necessary, update the cluster, following instructions in [1], section 7.
  • Pause the database and configuration synchronization on the slave nodes ([1], section 7.2.1)

Upgrading the master node

  • Set the master node to maintenance mode or manually disable it from the external load balancer.
  • Upgrade the master node software, using the upgrade process described in [2]

    • At the "Upgrade the database" step, upgrade both database clusters* (main and serverconf) to version 12
      (*) a PostgreSQL "cluster" is a collection of databases served by one postgres instance, should not be confused with a security server cluster

      Example
      $ sudo pg_lsclusters
Ver Cluster      Port Status Owner    Data directory                     Log file
10  main         5432 online postgres /var/lib/postgresql/10/main        /var/log/postgresql/postgresql-10-main.log
10  serverconf   5433 online postgres /var/lib/postgresql/10/serverconf  /var/log/postgresql/postgresql-10-serverconf.log
12  main         5434 online postgres /var/lib/postgresql/12/main        /var/log/postgresql/postgresql-12-main.log

$ sudo pg_dropcluster --stop 12 main
$ sudo pg_upgradecluster --method=upgrade --link 10 main
$ sudo pg_upgradecluster --method=upgrade --link 10 serverconf
    • After upgrading the databases, continue the upgrade process as described in [2]

  • If the master node was disabled manually from the external load balancer, verify that the master node is working and enable it in the load balancer

Upgrading the slave nodes

After successfully upgrading the master, slave nodes can be upgraded one by one.
(Alternatively, install new Ubuntu 20.04 slave node(s) as described in [1])

  • Gracefully disable the slave node from the load balancer, either manually or using the health check maintenance mode (see [1], section 7.2.3)
  • Upgrade the slave node software following [2] until the Upgrade the database step
  • Back up the file /var/lib/postgresql/10/serverconf/recovery.conf
    • e.g sudo cp /var/lib/postgresql/10/serverconf/recovery.conf /tmp/

  • Upgrade only the 10 main database and drop the other databases (including serverconf)

    Since the serverconf database is small, dropping and recreating it a straightforward option.
    See https://www.postgresql.org/docs/12/pgupgrade.html for more information and other options.

    Example
    $ sudo pg_lsclusters
Ver Cluster     Port Status Owner    Data directory                     Log file
10  main        5432 online postgres /var/lib/postgresql/10/main        /var/log/postgresql/postgresql-10-main.log
10  serverconf  5433 online postgres /var/lib/postgresql/10/serverconf  /var/log/postgresql/postgresql-10-serverconf.log
12  main        5434 online postgres /var/lib/postgresql/12/main        /var/log/postgresql/postgresql-12-main.log

$ sudo pg_dropcluster --stop 12 main
$ sudo pg_upgradecluster --method=upgrade --link 10 main

  • Recreate the serverconf database

    sudo pg_dropcluster --stop 10 serverconf
sudo pg_createcluster -p 5433 12 serverconf

    Follow the instructions in configuring the slave instance for replication in [1] to set up the database replication.

    • Note that you can find the <master> and <nodename> information in the backed up recovery.conf -file, but the replication configuration is a bit different on PostgreSQL 12.

  • Update the X-Road package repository to point to the Ubuntu 20.04 packages

    sudo apt-add-repository -r "deb https://artifactory.niis.org/xroad-release-deb bionic-current main"
sudo apt-add-repository "deb https://artifactory.niis.org/xroad-release-deb focal-current main"

    (alternatively, edit /etc/apt/sources.list directly)

  • Make sure that the database replication is up to date. The following should return t:

    Example
    $ sudo -iu postgres psql -p 5433 -tc 'select pg_last_wal_replay_lsn() = pg_last_wal_receive_lsn()'
 t

  • Upgrade the packages on the slave node to the Ubuntu 20.04 version.
    sudo apt update && sudo apt full-upgrade

  • Enable the shared configuration synchronization on the slave node:
    sudo rm /var/tmp/xroad/sync-disabled
    service xroad-sync start

  • restart the X-Road services and wait until the slave node is healthy.

  • See [1], section 6 for instructions about verifying that the upgrade was successful.
  • After the node is healthy, enable the slave node in the load balancer if you manually disabled it.

Troubleshooting