Security Server Cluster Ubuntu 14.04 to 18.04 Upgrade.
- Petteri Kivimäki
This document describes the steps required for upgrading a Security Server cluster from Ubuntu 14.04 LTS to Ubuntu 18.04 LTS in-place. Please read carefully through the whole document before starting the upgrade process. It is assumed that the reader is familiar with the Ubuntu Linux distribution and has experience of Ubuntu release upgrades.
Upgrading a system that uses a hardware security module has not been tested. Please verify that the HSM is compatible with Ubuntu 18.04 and check the HSM module documentation.
The upgrade process is based on
- [1] External Load Balancer Installation Guide (especially sections 4, 5, and 7) and
- [2] Security Server Ubuntu 14.04 to 18.04 In-place Upgrade
with some additional steps due to PostgreSQL database version upgrade from 9.3 to 10. Please review both documents before continuing.
Upgrade process
- Ensure that the X-Road software on all nodes is at version 6.20.0/1 or 6.21.0 (N.B.! all nodes must have the same version), and all Ubuntu packages are updated.
- If necessary, update the cluster, following instructions in [1], section 7.
- Pause the database and configuration synchronization on the slave nodes ([1], section 7.2.1)
Upgrading the master node
- Set the master node to maintenance mode or manually disable it from the external load balancer.
- See [1] section 7.2.2, steps 1 and 2.
- Upgrade the master node software, using the upgrade process described in [2]
At the "Upgrade the database" step, upgrade both database clusters* (main and serverconf) to version 10
(*) a PostgreSQL "cluster" is a collection of databases served by one postgres instance, should not be confused with a security server clusterExample$ sudo pg_lsclusters Ver Cluster Port Status Owner Data directory Log file 9.3 main 5432 online postgres /var/lib/postgresql/9.3/main /var/log/postgresql/postgresql-9.3-main.log 9.3 serverconf 5433 online postgres /var/lib/postgresql/9.3/serverconf /var/log/postgresql/postgresql-9.3-serverconf.log 9.5 main 5434 online postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log 10 main 5435 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log $ sudo pg_dropcluster --stop 9.5 main $ sudo pg_dropcluster --stop 10 main $ sudo pg_upgradecluster --method=upgrade --link 9.3 main $ sudo pg_upgradecluster --method=upgrade --link 9.3 serverconf
- After upgrading the databases, continue the upgrade process as described in [2]
- If the master node was disabled manually from the external load balancer, verify that the master node is working and enable it in the load balancer
- (see [1], section 7.2.2 for details)
Upgrading the slave nodes
After successfully upgrading the master, slave nodes can be upgraded one by one.
(Alternatively, install new Ubuntu 18.04 slave node(s) as described in [1])
- Gracefully disable the slave node from the load balancer, either manually or using the health check maintenance mode (see [1], section 7.2.3)
- Upgrade the slave node software following /wiki/spaces/XRDDEV/pages/13178021 until the Upgrade the database step
- Back up the file
/var/lib/postgresql/9.3/serverconf/recovery.confe.g sudo cp /var/lib/postgresql/9.3/serverconf/recovery.conf /tmp/
Upgrade only the 9.3 main database and drop the other databases (including serverconf)
Since the serverconf database is small, dropping and recreating it a straightforward option.
See https://www.postgresql.org/docs/10/pgupgrade.html for more information and other options.Example$ sudo pg_lsclusters Ver Cluster Port Status Owner Data directory Log file 9.3 main 5432 online postgres /var/lib/postgresql/9.3/main /var/log/postgresql/postgresql-9.3-main.log 9.3 serverconf 5433 online postgres /var/lib/postgresql/9.3/serverconf /var/log/postgresql/postgresql-9.3-serverconf.log 9.5 main 5434 online postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log 10 main 5435 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log $ sudo pg_dropcluster --stop 9.5 main $ sudo pg_dropcluster --stop 10 main $ sudo pg_upgradecluster --method=upgrade --link 9.3 main
Recreate the serverconf database
sudo pg_dropcluster --stop 9.3 serverconf sudo pg_createcluster -p 5433 10 serverconf
Follow the instructions in configuring the slave instance for replication in [1] to set up the database replication.
Note that you can find the <master> and <nodename> information in the recovery.conf -file
- Set up the configuration synchronization using systemd, as described in [1], section 5.2.1.
- Remove the old configuration synchronization:
sudo rm /etc/cron.d/xroad-state-sync /etc/init/xroad-sync.conf /etc/init/xroad-sync-wait.conf Update the X-Road package repository to point to the Ubuntu 18.04 packages
sudo apt-add-repository -r "deb https://artifactory.niis.org/xroad-release-deb trusty-current main" sudo apt-add-repository "deb https://artifactory.niis.org/xroad-release-deb bionic-current main"
(alternatively, edit /etc/apt/sources.list directly)
Make sure that the database replication is up to date. The following should return
t:Example$ sudo -iu postgres psql -p 5433 -tc 'select pg_last_wal_replay_lsn() = pg_last_wal_receive_lsn()' t
Upgrade the packages on the slave node to the Ubuntu 18.04 version.
sudo apt update && sudo apt full-upgradeEnable the shared configuration synchronization on the slave node:
sudo rm /var/tmp/xroad/sync-disabled
service xroad-sync startrestart the X-Road services and wait until the slave node is healthy.
- See [1], section 6 for instructions about verifying that the upgrade was successful.
- After the node is healthy, enable the slave node in the load balancer if you manually disabled it.
Troubleshooting
- If there is a full backup available, one can restore it and start over.
- If there is a configuration backup available from the master node, one can
- Recreate the master node using the backup, as described in How to Upgrade Security Server to Ubuntu 18.04 Using a Configuration Backup?
- Add new slave nodes, as described in [1].
Related articles