In this guide, we’ll look at how you can persist the Systemd Journals logs on your Server. Preserving the System Journal can be useful for troubleshooting your services when things keep breaking. At the end of this article, you should be able to configure your system journal to preserve the record of events when a server is rebooted.

<img alt="" data-ezsrc="https://kirelos.com/wp-content/uploads/2020/02/echo/preserve-systemd-journals-persistent.png" data-ez ezimgfmt="rs rscb4 src ng ngcb4 srcset" src="data:image/svg xml,”>

The System journals Basics

Systemd, by default, stores the system journals in the /run/log/journal directory. As discussed in our Understanding the Linux File System Hierarchy everything in /run directory is cleared and contents are recreated on reboot. This means the journals are cleared when the system reboots.

We can adjust the configuration settings of the systemd-journald service in the /etc/systemd/journald.conf file to make the journals persist across reboot. Open the file to view its contents:

$ sudo vim /etc/systemd/journald.conf

These are the common options which can be configured – Most will be commented out in most systems for you to adjust accordingly.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K

Configuring Persistent System Journals

To configure systemd-journald service to preserve system journals persistently across reboot, you need to set Storage to persistent:

Other values that can be set for Storage parameter are:

  • persistent: Stores journals in the/var/log/journaldirectory which persists across reboots. Directory is created if doesn’t exist
  • volatile: Stores journals in the volatile/run/log/journaldirectory. This doesn’t persist system reboots.
  • auto: The rsyslog will determine whether to use persistent or volatile storage. If the/var/log/journaldirectory exists, then rsyslog uses persistent storage, otherwise it uses volatile storage.

For persistent journals storage, set it to:

[Journal]
Storage=persistent

Once the changes are committed, restart systemd-journald service to bring the configuration changes into effect.

sudo systemctl restart systemd-journald

Confirm the service is restarted and running:

$ systemctl status systemd-journald
● systemd-journald.service - Journal Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled)
   Active: active (running) since Sun 2020-02-09 10:59:36 UTC; 8s ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 5299 (systemd-journal)
   Status: "Processing requests..."
   CGroup: /system.slice/systemd-journald.service
           └─5299 /usr/lib/systemd/systemd-journald

Feb 09 10:59:36 cent7.novalocal systemd-journal[5299]: Permanent journal is using 8.0M (max allowed 1.9G, trying to leave 2.9G free of 16.9G a…it 1.9G).
Feb 09 10:59:36 cent7.novalocal systemd-journal[5299]: Permanent journal is using 8.0M (max allowed 1.9G, trying to leave 2.9G free of 16.9G a…it 1.9G).
Feb 09 10:59:36 cent7.novalocal systemd-journal[5299]: Time spent on flushing to /var is 68.506ms for 7962 entries.
Feb 09 10:59:36 cent7.novalocal systemd-journal[5299]: Journal started
Hint: Some lines were ellipsized, use -l to show in full.

The /var/log/journal directory should be created.

$ ls /var/log/journal
65113b1a0d2f6087d515e6a8cd0ee7ef

The subdirectories under /var/log/journal have hexadecimal characters in their long names and contain*.journalfiles. 

$ ls /var/log/journal/65113b1a0d2f6087d515e6a8cd0ee7ef/
system.journal

The*.journalfiles are the binary files that store the structured and indexed journal entries.

Adjusting Maximum storage usage for Journals

You can set the maximum size of the persistent journal by uncommenting and changing the following: 

SystemMaxUse=500M

The default size limit is set to a value of 10% of the size of the underlying file system but capped at 4 GiB.

sudo systemctl restart systemd-journald

More guides:

Understanding the Linux File System Hierarchy

How To Create Hard Links and Soft (Symbolic) Links in Linux

, , , ,
Show Comments