Security Server Cluster Ubuntu 20.04 to 22.04 Upgrade.

Owned by Eneli Reimets
Last updated: Nov 16, 2022
3 min read

This document describes the steps required for upgrading a Security Server cluster from Ubuntu 20.04 LTS  to Ubuntu 22.04 LTS in-place. Please read carefully through the whole document before starting the upgrade process. It is assumed that the reader is familiar with the Ubuntu Linux distribution and has experience of Ubuntu release upgrades.

Upgrading a system that uses a hardware security module has not been tested. Please verify that the HSM is compatible with Ubuntu 22.04 and check the HSM module documentation.

The upgrade process is based on

with some additional steps due to PostgreSQL database version upgrade from 12 to 14. Please review both documents before continuing.

Upgrade process

  • Ensure that the X-Road software on all nodes is at version 7.2.0, and all Ubuntu packages are updated.

    • If necessary, update the cluster, following instructions in [1], section 7.

  • Pause the database and configuration synchronization on the slave nodes ([1], section 7.2.1)

Upgrading the master node

  • Set the master node to maintenance mode or manually disable it from the external load balancer.

  • Upgrade the master node software, using the upgrade process described in [2]

    • At the “Upgrade the database” step, upgrade both database clusters* (main and serverconf) to version 14
      (*) a PostgreSQL "cluster" is a collection of databases served by one postgres instance, should not be confused with a security server cluster

      $ sudo pg_lsclusters Ver Cluster      Port Status Owner    Data directory                     Log file 12  main         5432 online postgres /var/lib/postgresql/12/main        /var/log/postgresql/postgresql-12-main.log 12  serverconf   5433 online postgres /var/lib/postgresql/12/serverconf  /var/log/postgresql/postgresql-12-serverconf.log 14  main         5434 online postgres /var/lib/postgresql/14/main        /var/log/postgresql/postgresql-14-main.log $ sudo pg_dropcluster --stop 14 main $ sudo pg_upgradecluster --method=upgrade --link 12 main $ sudo pg_upgradecluster --method=upgrade --link 12 serverconf

    • After upgrading the databases, continue the upgrade process as described in [2]

  • If the master node was disabled manually from the external load balancer, verify that the master node is working and enable it in the load balancer

Upgrading the slave nodes

After successfully upgrading the master, slave nodes can be upgraded one by one.
(Alternatively, install new Ubuntu 22.04 slave node(s) as described in [1])

  • Gracefully disable the slave node from the load balancer, either manually or using the health check maintenance mode (see [1], section 7.2.3)

  • Upgrade the slave node software following [2] until the “Upgrade the database step”

  • Back up the file /var/lib/postgresql/10/serverconf/recovery.conf

    • sudo cp /var/lib/postgresql/10/serverconf/recovery.conf /tmp/

  • Upgrade only the 12 main database and drop the other databases (including serverconf)

    Since the serverconf database is small, dropping and recreating it a straightforward option.
    See https://www.postgresql.org/docs/14/pgupgrade.html for more information and other options.

    $ sudo pg_lsclusters Ver Cluster     Port Status Owner    Data directory                     Log file 12  main        5432 online postgres /var/lib/postgresql/12/main        /var/log/postgresql/postgresql-12-main.log 12  serverconf  5433 online postgres /var/lib/postgresql/12/serverconf  /var/log/postgresql/postgresql-12-serverconf.log 14  main        5434 online postgres /var/lib/postgresql/14/main        /var/log/postgresql/postgresql-14-main.log $ sudo pg_dropcluster --stop 14 main $ sudo pg_upgradecluster --method=upgrade --link 12 main

  • Recreate the serverconf database

    Follow the instructions in configuring the slave instance for replication in [1] to set up the database replication.

    • Note that you can find the <master> and <nodename> information in the backed up recovery.conf -file, but the replication configuration is a bit different on PostgreSQL 14.

  • Update the X-Road package repository to point to the Ubuntu 22.04 packages

    (alternatively, edit /etc/apt/sources.list directly)

  • Make sure that the database replication is up to date. The following should return t:

  • Upgrade the packages on the slave node to the Ubuntu 22.04 version.

  • Enable the shared configuration synchronization on the slave node:

  • The configuration synchronization can be forced, if necessary:

  • Restart the X-Road services and wait until the slave node is healthy.

  • See [1], section 6 for instructions about verifying that the upgrade was successful.

  • After verifying that the database upgrade was successful, drop the old main database, see [2] section “Drop old database and obsolete packages”

  • After the node is healthy, enable the slave node in the load balancer if you manually disabled it.

Troubleshooting

  • If there is a full backup available, one can restore it and start over.

  • If there is a configuration backup available from the master node, one can

    • Recreate the master node using the backup, as described in

    • Add new slave nodes, as described in [1].