This document describes the steps required for upgrading a Security Server cluster from Ubuntu 18.04 LTS  to Ubuntu 20.04 LTS in-place. Please read carefully through the whole document before starting the upgrade process. It is assumed that the reader is familiar with the Ubuntu Linux distribution and has experience of Ubuntu release upgrades.

Upgrading a system that uses a hardware security module has not been tested. Please verify that the HSM is compatible with Ubuntu 20.04 and  check the HSM module documentation.

The upgrade process is based on

• [1] External Load Balancer Installation Guide (especially sections 4, 5, and 7) and
• [2] Security Server Ubuntu 18.04 to 20.04 In-place Upgrade

with some additional steps due to PostgreSQL database version upgrade from 10 to 12. Please review both documents before continuing.

Upgrade process

• Ensure that the X-Road software on all nodes is at version 6.25.0, and all Ubuntu packages are updated.
  
  • If necessary, update the cluster, following instructions in [1], section 7.
• Pause the database and configuration synchronization on the slave nodes ([1], section 7.2.1)

Upgrading the master node

• Set the master node to maintenance mode or manually disable it from the external load balancer.
  • See [1] section 7.2.2, steps 1 a
• Upgrade the master node software, using the upgrade process described in [2]
  

  • At the "Upgrade the database" step, upgrade both database clusters* (main and serverconf) to version 12
    ^{(*) a PostgreSQL "cluster" is a collection of databases served by one postgres instance, should not be confused with a security server cluster}

    $ sudo pg_lsclusters Ver Cluster Port Status Owner Data directory Log file 10 main 5432 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log 10 serverconf 5433 online postgres /var/lib/postgresql/10/serverconf /var/log/postgresql/postgresql-10-serverconf.log 12 main 5434 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log $ sudo pg_dropcluster --stop 12 main $ sudo pg_upgradecluster --method=upgrade --link 10 main $ sudo pg_upgradecluster --method=upgrade --link 10 serverconf

    
    

  • After upgrading the databases, continue the upgrade process as described in [2]
    
    
• If the master node was disabled manually from the external load balancer, verify that the master node is working and enable it in the load balancer
  • (see [1], section 7.2.2 for details)

Upgrading the slave nodes

After successfully upgrading the master, slave nodes can be upgraded one by one.
(Alternatively, install new Ubuntu 20.04 slave node(s) as described in [1])

• Gracefully disable the slave node from the load balancer, either manually or using the health check maintenance mode (see [1], section 7.2.3)
• Upgrade the slave node software following [2] until the Upgrade the database step
• Back up the file /var/lib/postgresql/10/serverconf/recovery.conf
  • e.g sudo cp /var/lib/postgresql/10/serverconf/recovery.conf /tmp/

• Upgrade only the 10 main database and drop the other databases (including serverconf)

  Since the serverconf database is small, dropping and recreating it a straightforward option. See https://www.postgresql.org/docs/12/pgupgrade.html for more information and other options.

  
  

  $ sudo pg_lsclusters Ver Cluster Port Status Owner Data directory Log file 10 main 5432 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log 10 serverconf 5433 online postgres /var/lib/postgresql/10/serverconf /var/log/postgresql/postgresql-10-serverconf.log 12 main 5434 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log $ sudo pg_dropcluster --stop 12 main $ sudo pg_upgradecluster --method=upgrade --link 10 main

  
  

• Recreate the serverconf database

  sudo pg_dropcluster --stop 10 serverconf sudo pg_createcluster -p 5433 12 serverconf

  Follow the instructions in configuring the slave instance for replication in [1] to set up the database replication.

  • Note that you can find the <master> and <nodename> information in the backed up recovery.conf -file, but the replication configuration is a bit different on PostgreSQL 12.

• Update the X-Road package repository to point to the Ubuntu 20.04 packages

  sudo apt-add-repository -r "deb https://artifactory.niis.org/xroad-release-deb bionic-current main" sudo apt-add-repository "deb https://artifactory.niis.org/xroad-release-deb focal-current main"

  (alternatively, edit /etc/apt/sources.list directly)

• Make sure that the database replication is up to date. The following should return t:

  $ sudo -iu postgres psql -p 5433 -tc 'select pg_last_wal_replay_lsn() = pg_last_wal_receive_lsn()' t

  
  

• Upgrade the packages on the slave node to the Ubuntu 20.04 version.
  sudo apt update && sudo apt full-upgrade

• Enable the shared configuration synchronization on the slave node:
sudo rm /var/tmp/xroad/sync-disabled
service xroad-sync start

• restart the X-Road services and wait until the slave node is healthy.

• See [1], section 6 for instructions about verifying that the upgrade was successful.
• After the node is healthy, enable the slave node in the load balancer if you manually disabled it.

Troubleshooting

• If there is a full backup available, one can restore it and start over.
• If there is a configuration backup available from the master node, one can
  • Recreate the master node using the backup, as described in Upgrading Security Server to Ubuntu 20.04 Using a Configuration Backup
  • Add new slave nodes, as described in [1].

Related articles

Related articles appear here based on the labels you select. Click to edit the macro and add or change labels.