How to Store the Security Server Message Log Archives in the Azure Blob Storage?
Security Server’s business log is stored in the message log database which contains all the messages processed by the Security Server. Each message is time-stamped and signed which makes it possible to verify the message content afterwards. By default, time-stamped messages are archived from the database to disk every six hours. Time-stamped and archived messages are kept in the message log database for 30 days until they are removed automatically.
More information about the message log can be found at: https://github.com/ria-ee/X-Road/blob/develop/doc/Manuals/ug-ss_x-road_6_security_server_user_guide.md#11-message-log
The archived log records are not automatically transferred to an external host or log storage for long-term archiving purposes. It is the administrator’s responsibility to configure the transfer of the logs to a long-term storage.
Long-term Archival
Azure Blob Storage can be used as a long-term storage for the Security Server logs. Azure Blob storage is Microsoft's object storage solution for the cloud. Azure Blob Storage always encrypts data both in transit and at rest. For additional security, customer-managed or customer-provided encryption key can be used to encrypt data at rest.
More information about the Azure Blob Storage can be found at: https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-overview
To store and transfer the archived message log records in a Blob Storage container, follow the steps below.
- Create a storage account.
- Account kind: StorageV2 (general purpose v2)
- Select the storage account properties (e.g. location, replication, default storage class etc.) based on your requirements.
- Create a blob storage container under the storage account.
- It is recommended to create a container per Security Server.
- N.B.! Only users who are authorized to access message log archives should be granted access to the container. See available default roles.
- Enable a system-assigned managed identity on the Security Server host VM.
- Grant the Storage Blob Data Contributor role to the managed identity - the scope of the role must be the container.
- The role must be granted for a specific container, not for the whole storage account.
- The built-in Storage Blob Data Contributor role allows read, write and delete access to the container. It's recommended to create a custom role that allows only write access to the container. However, creating custom roles requires Azure AD Premium P1 or P2.
- Go to the container Settings → Access Policy and create a time-based retention policy for the container.
- The objects (=Security Server log archives) cannot be deleted or updated during the retention period.
- Go to the storage account configuration → Blob service → Lifecycle Management and create lifecycle rules that automatically update storage class or delete objects that have exceeded certain age, e.g. delete objects older than 90 days. Lifecycle policies can be used to automatically delete logs that have exceeded their retention period.
- Install (if not already installed) AzCopy tool on the Security Server host.
- Create a bash script (e.g. /usr/share/xroad/scripts/archive-logs.sh) on the Security Server host and grant xroad user sufficient rights to execute the script. Below there's an example script that transfers archived message log files from Security Server to blob storage, and deletes all the successfully transferred files from the Security Server.
N.B.! Replace
<STORAGE_ACCOUNT>
and <CONTAINER> with the names of your storage account and container (line 11).
The below script is just an example and therefore, it is not recommended to use it for production purposes.
#!/bin/bash # Log in by using the system-assigned identity of a VM LOGIN=$(azcopy login --identity) if [[ "$LOGIN" != *"Login with identity succeeded"* ]]; then echo $LOGIN exit 1 fi # Copy X-Road message log archives to the specified STORAGE_ACCOUNT and CONTAINER OUTPUT=$(azcopy copy "/var/lib/xroad/*.zip" 'https://<STORAGE_ACCOUNT>.blob.core.windows.net/<CONTAINER>') # Get the ID of the operation ID=$(echo $OUTPUT | grep -Po '(\w+-\w+-\w+-\w+-\w+)' | head -1) # Get the list of files that were successfully copied FILES=$(azcopy jobs show $ID --with-status=Success) for fn in $FILES; do if [[ "$fn" =~ ^\/var\/lib\/xroad/mlog- ]]; then # Remove file from Security Server rm "$fn" fi done exit 0
- Override the configuration parameter archive-transfer-command (create or edit the file "/etc/xroad/conf.d/local.ini") to set up a transferring script. The script is executed every time when the archival process is run. The archiving schedule can be customized using the archive-interval configuration parameter.
- The script uses Azure's azcopy tool to move the archived log records from the Security Server to the Blob Storage container. Once a file is successfully uploaded to the blob storage, the script removes the file from the Security Server.
[message-log] archive-transfer-command=/usr/share/xroad/scripts/archive-logs.sh
- Restart the xroad-proxy process after updating the archive-transfer-command configuration parameter value.
These instructions can be used to transfer archived log records to Azure Blob Storage from Security Servers hosted on Azure. In case, the Security Server is hosted on another cloud or on-premise, another authorization method (other than system-assigned managed identity) must be used. More information about the alternatives.
Related articles