This document describes the steps required for upgrading a Security Server cluster from Ubuntu 20.04 LTS to Ubuntu 22.04 LTS in-place. Please read carefully through the whole document before starting the upgrade process. It is assumed that the reader is familiar with the Ubuntu Linux distribution and has experience of Ubuntu release upgrades.
Upgrading a system that uses a hardware security module has not been tested. Please verify that the HSM is compatible with Ubuntu 22.04 and check the HSM module documentation.
The upgrade process is based on
[1] https://github.com/nordic-institute/X-Road/blob/develop/doc/Manuals/LoadBalancing/ig-xlb_x-road_external_load_balancer_installation_guide.md (especially sections 4, 5, and 7) and
with some additional steps due to PostgreSQL database version upgrade from 12 to 14. Please review both documents before continuing.
Ensure that the X-Road software on all nodes is at version 7.2.0, and all Ubuntu packages are updated.
If necessary, update the cluster, following instructions in [1], section 7.
Pause the database and configuration synchronization on the slave nodes ([1], section 7.2.1)
Set the master node to maintenance mode or manually disable it from the external load balancer.
See [1] section 7.2.2, steps 1 and 2.
Upgrade the master node software, using the upgrade process described in [2]
At the “Upgrade the database” step, upgrade both database clusters* (main and serverconf) to version 14
(*) a PostgreSQL "cluster" is a collection of databases served by one postgres instance, should not be confused with a security server cluster
$ sudo pg_lsclusters Ver Cluster Port Status Owner Data directory Log file 12 main 5432 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log 12 serverconf 5433 online postgres /var/lib/postgresql/12/serverconf /var/log/postgresql/postgresql-12-serverconf.log 14 main 5434 online postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log $ sudo pg_dropcluster --stop 14 main $ sudo pg_upgradecluster --method=upgrade --link 12 main $ sudo pg_upgradecluster --method=upgrade --link 12 serverconf |
After upgrading the databases, continue the upgrade process as described in [2]
If the master node was disabled manually from the external load balancer, verify that the master node is working and enable it in the load balancer
See [1], section 7.2.2 for details
After successfully upgrading the master, slave nodes can be upgraded one by one.
(Alternatively, install new Ubuntu 22.04 slave node(s) as described in [1])
Gracefully disable the slave node from the load balancer, either manually or using the health check maintenance mode (see [1], section 7.2.3)
Upgrade the slave node software following [2] until the “Upgrade the database step”
Back up the file /var/lib/postgresql/10/serverconf/recovery.conf
sudo cp /var/lib/postgresql/10/serverconf/recovery.conf /tmp/ |
Upgrade only the 12 main database and drop the other databases (including serverconf)
Since the serverconf database is small, dropping and recreating it a straightforward option.
See https://www.postgresql.org/docs/14/pgupgrade.html for more information and other options.
$ sudo pg_lsclusters Ver Cluster Port Status Owner Data directory Log file 12 main 5432 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log 12 serverconf 5433 online postgres /var/lib/postgresql/12/serverconf /var/log/postgresql/postgresql-12-serverconf.log 14 main 5434 online postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log $ sudo pg_dropcluster --stop 14 main $ sudo pg_upgradecluster --method=upgrade --link 12 main |
Recreate the serverconf database
sudo pg_dropcluster --stop 12 serverconf sudo pg_createcluster -p 5433 14 serverconf |
Follow the instructions in configuring the slave instance for replication in [1] to set up the database replication.
Note that you can find the <master> and <nodename> information in the backed up recovery.conf -file, but the replication configuration is a bit different on PostgreSQL 14.
Update the X-Road package repository to point to the Ubuntu 22.04 packages
sudo apt-add-repository -r "deb https://artifactory.niis.org/xroad-release-deb focal-current main" sudo apt-add-repository "deb https://artifactory.niis.org/xroad-release-deb jammy-current main" |
(alternatively, edit /etc/apt/sources.list directly)
Make sure that the database replication is up to date. The following should return t:
$ sudo -iu postgres psql -p 5433 -tc 'select pg_last_wal_replay_lsn() = pg_last_wal_receive_lsn()' t |
Upgrade the packages on the slave node to the Ubuntu 22.04 version.
sudo apt update && sudo apt full-upgrade |
Enable the shared configuration synchronization on the slave node:
$ sudo rm /var/tmp/xroad/sync-disabled |
The configuration synchronization can be forced, if necessary:
sudo service xroad-sync start |
Restart the X-Road services and wait until the slave node is healthy.
See [1], section 6 for instructions about verifying that the upgrade was successful.
After verifying that the database upgrade was successful, drop the old main database, see [2] section “Drop old database and obsolete packages”
After the node is healthy, enable the slave node in the load balancer if you manually disabled it.
If there is a full backup available, one can restore it and start over.
If there is a configuration backup available from the master node, one can
Recreate the master node using the backup, as described in How to Upgrade Security Server to Ubuntu 22.04 Using a Configuration Backup?
Add new slave nodes, as described in [1].