Debugging Oversized Systemd Journal Logs

Background

Our machines have been frequently triggering disk capacity alerts recently, and we found that the following directories were consuming a significant amount of disk space:

sudo du -sh ./* --exclude ./proc | sort -rh  | head -n 5

/run/log/journal is the systemd log directory, used to store systemd journal files. You can use the journalctl tool to manage and clean up these log files. Below is how to clean up systemd logs.

Solution

1. List log entries:

To view the available log entries, use the following command:

journalctl

This will display the currently available systemd logs.

2. Configure log retention policy:

Systemd uses a log retention policy to manage how long log files are kept. You can configure the retention policy by editing the /etc/systemd/journald.conf file. Look for the SystemKeepFree, SystemMaxUse, and RuntimeMaxUse options, then set them according to your needs. For example:

SystemMaxUse=100M

This configures the maximum disk space used by systemd journal logs to 100MB. You can adjust this value to suit your requirements.

3. Clean up logs:

Once you have set the retention policy, you can run the following command to clean up the logs:

sudo journalctl --vacuum-time=1h

This will delete logs older than the specified time, keeping only the most recent ones.

You can change the value of the **`--vacuum-time`** flag to define how long logs should be retained.

4. Verify the cleaned-up log size:

To verify the size of the log files after cleanup, run the following command again:

journalctl

This will display the currently available log information, including the portion that has been cleaned up.

Tips

Note that cleaning up systemd logs requires sufficient privileges. Make sure you have the necessary permissions to perform these operations, or use the sudo command to execute them with administrator privileges. Cleaning up log files may free up disk space, but it may also result in the loss of some older log information. Therefore, choose your cleanup strategy carefully based on your needs and available storage.