How to Store the Security Server Message Log Archives in the Google Cloud Storage?

Security Server’s business log is stored in the message log database which contains all the messages processed by the Security Server. Each message is time-stamped and signed which makes it possible to verify the message content afterwards. By default, time-stamped messages are archived from the database to disk every six hours. Time-stamped and archived messages are kept in the message log database for 30 days until they are removed automatically.

The archived log records are not automatically transferred to an external host or log storage for long-term archiving purposes. It is the administrator’s responsibility to configure the transfer of the logs to a long-term storage.

Long-term Archival

Google Cloud Storage can be used as a long-term storage for the Security Server logs. Google Cloud Storage is unified object storage to store data in Google's cloud. Google Cloud Storage always encrypts data both in transit and at rest. For additional security, customer-managed or customer-supplied encryption key can be used to encrypt data at rest.

To store and transfer the archived message log records in a Cloud Storage bucket, follow the steps below.

• Create a Cloud Storage bucket.
  • Select a default storage class based on the retention period of the archived log files.
  • Select Uniform access control mode.
  • Based on the security requirements, select Google-managed (default) or customer-managed encryption key.
    
    • In case a customer-managed encryption key is used, the key ring containing the key must be located in the same location (region / multi-region) where the Cloud Storage bucket is located.
  • Additionally, it's possible to set a retention policy that specifies the minimum duration that the objects are protected from deletion or modification after they're uploaded.
  • N.B.! Only users who are authorized to access message log archives should be granted access to the bucket. See available IAM roles.
• Once the bucket is created, add a lifecycle policy that automatically updates storage class or deletes objects that have exceeded certain age, e.g. delete objects older than 90 days. Lifecycle policies can be used to automatically delete logs that have exceeded their retention period.
• The service account used by the Security Server(s) must have Storage Object Creator IAM role for the bucket.
  • The role allows the Security Server(s) to create objects. Does not give permission to view, delete, or overwrite objects.
  • Multiple Security Servers can share the same bucket and they are not able access each others message log archives.
• Install (if not already installed) gsutil tool on the Security Server host.
• Create a bash script (e.g. /usr/share/xroad/scripts/archive-logs.sh) on the Security Server host and grant xroad user sufficient rights to execute the script. Below there's an example script that moves archived message log files from Security Server to Cloud Storage bucket, and deletes all the successfully moved files from the Security Server.
  

  • N.B.! Replace <MY_BUCKET> and <SECURITY_SERVER_CODE> with the names of your bucket and Security Server code.

  • In case multiple Security Servers use the same bucket, it's recommended to use server code as an object name prefix so that log files from different Security Servers are separated.

#!/bin/bash

/snap/bin/gsutil mv /var/lib/xroad/*.zip gs://<MY_BUCKET>/<SECURITY_SERVER_CODE> >/dev/null 2>&1

exit 0

• Override the configuration parameter archive-transfer-command (create or edit the file "/etc/xroad/conf.d/local.ini") to set up a transferring script. The script is executed every time when the archival process is run. The archiving schedule can be customized using the archive-interval configuration parameter.
  
  • The script uses Google's gsutil tool to move the archived log records from the Security Server to the Cloud Storage bucket. Once a file is successfully uploaded to the bucket, gsutil removes the file from the Security Server.

[message-log]
archive-transfer-command=/usr/share/xroad/scripts/archive-logs.sh

• Restart the xroad-proxy process after updating the archive-transfer-command configuration parameter value.

Related articles

• Page:
  How to Interpret Security Server Proxy Error Messages?
• Page:
  How to Set Up a Local Test Environment Using Docker Compose?
• Page:
  Security Server Cluster RHEL 8 (or RHEL 7) to RHEL 9 Upgrade
• Page:
  How to Upgrade Security Server to RHEL 9 Using a Configuration Backup?
• Page:
  How to Access the Security Server Postgres Database?