MISP2 Ubuntu 18.04 to 22.04 upgrade

Owned by Raido Kaju
Apr 06, 2023

This document describes the steps required for upgrading a MISP2 host from Ubuntu 18.04 LTS host to Ubuntu 22.04 LTS. Please read carefully through the whole document before starting the upgrade process. It is assumed that the reader is familiar with the Ubuntu Linux distribution and has experience with Ubuntu release upgrades.

With the upgrade to Ubuntu 22.04 LTS, PostgreSQL is updated from version 10 to 14. Additionally, the Tomcat version is updated from 8 to 9.

NIIS does not support the WSO2 component for MISP2 and, as such, does not have Ubuntu 22.04 packages available.


Preparations

  • Review the Ubuntu release notes and upgrade instructions:
  • Review the Ubuntu release notes and upgrade instructions:

  • Update all packages to the latest versions:
    apt update && apt full-upgrade

  • Ensure that the MISP2 software version is at least 2.8.0

  • If you have adoptopenjdk-8 installed, make sure you remove it. Otherwise, it will cause problems when upgrading the MISP2 packages at a later stage:
    • This can be checked with the following command: java -version
    • sudo apt remove adoptopenjdk-8-hotspot-jre
  • Create a copy of the /etc/apache2/sites-enabled/ssl.conf file

  • Recommended: Ensure you have an up-to-date server and database backup (or virtual machine snapshot).

  • Make a backup of your configurations:

    • You should back up the /var/lib/tomcat8/webapps/misp2 folder

      • For example with tar: tar -czvf misp2-backup.tar.gz /var/lib/tomcat8/webapps/misp2
    • You should back up the /etc/apache2/sites-enabled/ssl.conf file.

    • This makes it possible to restore the server configuration if the upgrade fails.

  • Stop the tomcat8 server and prevent it from starting automatically at boot
    • sudo systemctl stop tomcat8
    • sudo systemctl disable tomcat8
  • Pre-create the tomcat9 webapps directory so the MISP2 installer can pick it up during the upgrade.
    • mkdir -p /var/lib/tomcat9/webapps
  • Copy the misp2 folder to tomcat9 so the MISP2 installer can pick it up during the upgrade.
    • cp -r /var/lib/tomcat8/webapps/misp2 /var/lib/tomcat9/webapps/

  • Optionally, take a backup of the message log database. Without a backup, a failure in the upgrade process may cause data loss.

    • See https://www.postgresql.org/docs/10/backup.html
    • The default schema name is misp2db as used in the example command below. For your specific schema name, please refer to the database credentials in the /var/lib/tomcat8/webapps/misp2/WEB-INF/classes/config.cfg file.
    • Using pg_dump is a straightforward option, but it can be problematic if the database is extremely large:
      sudo -iu postgres pg_dump -d misp2db -F c -f <dump_file>

Upgrading Ubuntu 18.04 to 20.04

  • Install update-manager-core if it is not already installed.
  • Make sure the Prompt line in /etc/update-manager/release-upgrades is set to 'lts'
  • Launch the upgrade tool with the command sudo do-release-upgrade
  • Follow the on-screen instructions.
  • When the upgrade is finished, reboot when prompted.
  • Due to the dependencies defined by MISP2 packages, the PostgreSQL installation is not automatically updated. Please do so manually by running the following command:

    • sudo apt install postgresql-12

  • After that, a new empty database instance should have been created called 12 main. In the following steps, we will remove the empty instance and upgrade the MISP2 database from version 10 to 12.

  • List the database instances with pg_lsclusters, and drop the extra ones
    Do not remove the version 10 (main) running on port 5432
    sudo pg_dropcluster --stop <version, e.g. 12> main

    Example
    $ sudo pg_lsclusters
Ver Cluster Port Status Owner    Data directory               Log file
10  main    5432 online postgres /var/lib/postgresql/10/main  /var/log/postgresql/postgresql-10-main.log
12  main    5433 online postgres /var/lib/postgresql/12/main  /var/log/postgresql/postgresql-12-main.log
 
$ sudo pg_dropcluster --stop 12 main

  • Upgrade the old database to version 12.

    Ubuntu's pg_upgradecluster helper script by default dumps the old database, which can require a lot of time and free disk space if the database is large. The --method=upgrade uses pg_upgrade instead and the --link option avoids copying the data files.
    (warning) Using --link requires that the new and old database are in the same filesystem. See https://www.postgresql.org/docs/12/pgupgrade.html for details and recovery instructions.

    sudo pg_upgradecluster --method=upgrade --link 10 main

    Example
    $ sudo pg_upgradecluster --method=upgrade --link 10 main

$ sudo pg_lsclusters
Ver Cluster Port Status Owner    Data directory               Log file
10  main    5433 down   postgres /var/lib/postgresql/10/main  /var/log/postgresql/postgresql-10-main.log
12  main    5432 online postgres /var/lib/postgresql/12/main  /var/log/postgresql/postgresql-12-main.log

  • Drop the old database:


    • sudo pg_dropcluster 10 main

Upgrading Ubuntu 20.04 to 22.04

  • Install update-manager-core if it is not already installed.
  • Make sure the Prompt line in /etc/update-manager/release-upgrades is set to 'lts'
  • Launch the upgrade tool with the command sudo do-release-upgrade
  • Follow the on-screen instructions.
    • When asked about the /etc/apache2/mods-available/jk.conf file, keep the local version.
  • When the upgrade is finished, reboot when prompted.
  • Due to the dependencies defined by MISP2 packages, the PostgreSQL installation is not automatically updated. Please do so manually by running the following command:

    • sudo apt install postgresql-14

  • After that, a new empty database instance should have been created called 12 main. In the following steps, we will remove the empty instance and upgrade the MISP2 database from version 12 to 14.

  • List the database instances with pg_lsclusters, and drop the extra ones
    Do not remove the version 12 (main) running on port 5432
    sudo pg_dropcluster --stop <version, e.g. 14> main

    Example
    $ sudo pg_lsclusters
Ver Cluster Port Status Owner    Data directory               Log file
12  main    5432 online postgres /var/lib/postgresql/12/main  /var/log/postgresql/postgresql-12-main.log
14  main    5433 online postgres /var/lib/postgresql/14/main  /var/log/postgresql/postgresql-14-main.log
 
$ sudo pg_dropcluster --stop 14 main

  • Upgrade the old database to version 14.

    Ubuntu's pg_upgradecluster helper script by default dumps the old database, which can require a lot of time and free disk space if the database is large. The --method=upgrade uses pg_upgrade instead and the --link option avoids copying the data files.
    (warning) Using --link requires that the new and old database are in the same filesystem. See https://www.postgresql.org/docs/12/https://www.postgresql.org/docs/14/pgupgrade.htmlpgupgrade.html for details and recovery instructions.

    sudo pg_upgradecluster --method=upgrade --link 12 main

    Example
    $ sudo pg_upgradecluster --method=upgrade --link 12 main

$ sudo pg_lsclusters
Ver Cluster Port Status Owner    Data directory               Log file
12  main    5433 down   postgres /var/lib/postgresql/12/main  /var/log/postgresql/postgresql-12-main.log
14  main    5432 online postgres /var/lib/postgresql/14/main  /var/log/postgresql/postgresql-14-main.log

  • Drop the old database:

    • sudo pg_dropcluster 12 main

Upgrading the MISP2 software

  • Re-enable the MISP2 source repository in the /etc/apt/sources.list file. Remember to also replace focal with jammy in the line.
  • Upgrade the MISP2 packages using:
    • sudo apt update
    • sudo apt upgrade
  • Due to a bug in Ubuntu 22.04 when running tomcat9 with JAVA8, please also run the following commands as a workaround:
  • Restore the /etc/apache2/sites-enabled/ssl.conf file from your copy
  • Restart apache2 with:
    • sudo systemctl restart apache2
  • Clean up old packages with the following commands:
    • sudo apt purge postgresql-10 postgresql-12
    • sudo apt autoremove

Troubleshooting

  • If recovery from an upgrade failure is not possible and there is a backup/snapshot available, one can restore it and start over.