Security Server Cluster Ubuntu 22.04 to 24.04 Upgrade.

This document describes the steps required for upgrading a Security Server cluster from Ubuntu 22.04 LTS  to Ubuntu 24.04 LTS in-place. Please read carefully through the whole document before starting the upgrade process. It is assumed that the reader is familiar with the Ubuntu Linux distribution and has experience of Ubuntu release upgrades.

Upgrading a system that uses a hardware security module has not been tested. Please verify that the HSM is compatible with Ubuntu 24.04 and check the HSM module documentation.

The upgrade process is based on

• [1] External Load Balancer Installation Guide (especially sections 4, 5, and 7) and

• [2]Security Server Ubuntu 22.04 to 24.04 In-place Upgrade.

with some additional steps due to PostgreSQL database version upgrade from 14 to 16. Please review both documents before continuing.

Upgrade process

• Ensure that the X-Road software on all nodes is at version 7.5.0, and all Ubuntu packages are updated.

  • If necessary, update the cluster, following instructions in [1], section 7.

• Pause the database and configuration synchronization on the secondary nodes ([1], section 7.2.1)

Upgrading the primary node

• Set the primary node to maintenance mode or manually disable it from the external load balancer.

  • See [1] section 7.2.2, steps 1 and 2.

• Upgrade the primary node software, using the upgrade process described in [2]

  • At the “Upgrade the database” step, upgrade both database clusters* (main and serverconf) to version 16
    ^{(*) a PostgreSQL "cluster" is a collection of databases served by one postgres instance, should not be confused with a security server cluster}

    $ sudo pg_lsclusters Ver Cluster Port Status Owner Data directory Log file 14 main 5432 online postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log 14 serverconf 5433 online postgres /var/lib/postgresql/14/serverconf /var/log/postgresql/postgresql-14-serverconf.log 16 main 5434 online postgres /var/lib/postgresql/16/main /var/log/postgresql/postgresql-16-main.log $ sudo pg_dropcluster --stop 16 main $ sudo pg_upgradecluster --method=upgrade --link 14 main $ sudo pg_upgradecluster --method=upgrade --link 14 serverconf

  • After upgrading the databases, continue the upgrade process as described in [2]

• If the primary node was disabled manually from the external load balancer, verify that the primary node is working and enable it in the load balancer

  • See [1], section 7.2.2 for details

Upgrading the secondary nodes

After successfully upgrading the primary, secondary nodes can be upgraded one by one.
(Alternatively, install new Ubuntu 24.04 secondary node(s) as described in [1])

• Gracefully disable the secondary node from the load balancer, either manually or using the health check maintenance mode (see [1], section 7.2.3)

• Upgrade the secondary node software following [2] until the “Upgrade the database step”

• Back up the file /etc/postgresql/14/serverconf/postgresql.conf

  • sudo cp /etc/postgresql/14/serverconf/postgresql.conf /tmp/

• Upgrade only the 14 main database and drop the other databases (including serverconf)

  Since the serverconf database is small, dropping and recreating it a straightforward option.
  See pg_upgrade for more information and other options.

  $ sudo pg_lsclusters Ver Cluster Port Status Owner Data directory Log file 14 main 5432 online postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log 14 serverconf 5433 online,recovery postgres /var/lib/postgresql/14/serverconf /var/log/postgresql/postgresql-14-serverconf.log 16 main 5434 online postgres /var/lib/postgresql/16/main /var/log/postgresql/postgresql-16-main.log $ sudo pg_dropcluster --stop 16 main $ sudo pg_dropcluster --stop 14 serverconf $ sudo pg_upgradecluster --method=upgrade --link 14 main

• Recreate the serverconf database

  Follow the instructions in configuring the secondary instance for replication in [1] to set up the database replication.

  • Note that you can find the <primary> and <nodename> information in the backed up postgresql.conf file.

• Update the X-Road package repository to point to the Ubuntu 24.04 packages

  Alternatively, find the repository definition in /etc/apt/sources.list.d/xroad.list and edit it directly. More about Debian source lists.

• Make sure that the database replication is up to date. The following should return t:

• Upgrade the packages on the secondary node to the Ubuntu 24.04 version.

• Enable the shared configuration synchronization on the secondary node:

• The configuration synchronization can be forced, if necessary:

• Enable and restart the X-Road services and wait until the secondary node is healthy.

• See [1], section 6 for instructions about verifying that the upgrade was successful.

• After verifying that the database upgrade was successful, drop the old main database, see [2] section “Drop old database and obsolete packages”

• After the node is healthy, enable the secondary node in the load balancer if you manually disabled it.

Troubleshooting

• If there is a full backup available, one can restore it and start over.

• If there is a configuration backup available from the primary node, one can

  • Recreate the primary node using the backup, as described in How to Upgrade Security Server to Ubuntu 24.04 Using a Configuration Backup?

  • Add new secondary nodes, as described in [1].