Rotating log files with Logrotate

Logs are an important part of running your own server. They give you insight into what's happening on your system, help you diagnose problems, find bad actors, and more. They also fill up disk space pretty quickly, especially on busy servers.

In this lesson you will learn about Logrotate, a Linux utility to manage automatic rotation, archiving, compression, retention, and even mailing of log files.

This lesson is brought to you by Cherry Servers: a wide range of dedicated and virtual servers at a great price, with fast delivery and outstanding 24/7 human support.

This lesson is brought to you by DigitalOcean: If you want fast VPS provisioning and clear, no-surprise pricing for your next WordPress box, DigitalOcean is a solid pick.

Logrotate

The logrotate utility is pre-installed and available on most modern Linux systems, including Ubuntu and Debian. If not, you can usually get it from an official package:

sudo apt install logrotate

On some systems, it is also pre-configured to rotate existing system log files, so that's one thing you don't really have to worry about.

However, knowing a little bit about how logrotate works and how it's configured allows you to better understand your system, customize it if needed, and apply similar configurations to other logging you come across in the future.

The wp-content/debug.log is a great example from the WordPress space. I've seen many sites fill this log up to gigabytes of data. Even on some managed platforms this file is rarely rotated or cleaned up, and when people realize how big it has grown, their first instinct is usually to simply truncate or delete it, losing potentially valuable log data.

We'll add a logrotate configuration for the WordPress debug log in a future module after we've actually installed WordPress, but in this lesson we'll look at some existing configs.

How logrotate works

The logrotate program is invoked on a timer. It parses through its configuration files and starts looking at various log files to see if they need to be rotated.

A rotation usually happens by renaming the existing log file and creating a new one in its place. There is a slightly less efficient way available too, for some applications that may freak out if their log file is renamed.

Rotation propagates to all previous rotations of the same log file too, often adding compression to the mix, with the oldest file eventually ending up on the chopping block. Eventually you may end up with the following file structure of the authentication log, for example:

auth.log
auth.log.1
auth.log.2.gz
auth.log.3.gz

Here the auth.log is the currently active log file, and any authentication issues will be logged there. The auth.log.1, auth.log.2.gz, etc. are the results of multiple log rotations.

Searching compressed files

When looking for information in log files it is very common to have to go back further than the last few rotations, so you may end up having to search in the compressed files.

The zgrep command is great for this, and allows you to search multiple files as well. Here's a quick example:

sudo zgrep 1.2.3.4 /var/log/auth.log.*.gz

It'll also work with non-compressed files, so you can safely combine searches across both. You can also use vim to open a .gz file and search there if needed.

Configuration

Logrotate has a main configuration in /etc/logrotate.conf which sets some defaults for all other configs. The rest of the configuration is split between multiple files in the /etc/logrotate.d/ directory, where you can also drop your own configs when needed.

By default most Linux distributions will have existing configs for system logs. For example in Ubuntu 24.04 you'll see an /etc/logrotate.d/rsyslog file with rotation set up for the system log, auth.log, mail.log and others.

When installing software, especially from official packages, new logrotate configurations are usually added if that software writes to any new logs. For example, we installed fail2ban in a previous lesson, so you should see a /etc/logrotate.d/fail2ban file with a rotation config for /var/log/fail2ban.log.

When working with custom builds, or building from source, it will be up to us to figure out if this software needs its logs rotated.

Logging may also depend on the configuration of specific applications. The wp-content/debug.log is a good example. Nginx can be configured to log things to a different location, and so can PHP, both with a server-level config, and also using the ini_set() function inside a script, or writing directly to a file, which is what WooCommerce does by default.

When you come across any such configurations, it's important to understand where the log files are being written, ensure they are not publicly accessible (we'll talk more about that in future modules), and ensure they are rotated.

Example

Have a look through some of the existing configuration files in the /etc/logrotate.d directory and note that some files may be used to configure multiple log targets. Chances are you already have a rotation configured for your /var/log/auth.log in /etc/logrotate.d/rsyslog, but if you don't, let's create one from scratch.

In the spirit of "all configs under version control", let's create a new file in our configs repo, and symlink it to the logrotate.d directory:

cd /configs
sudo touch misc/logrotate.conf
sudo ln -s /config/misc/logrotate.conf /etc/logrotate.d/custom

Let's use the following contents for this new configuration file:

/var/log/auth.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Here's a quick explanation of the config directives I used:

• daily rotates this log file every day
• rotate 30 keeps 30 files (so 30 days) before removing
• compress uses gzip to compress old log files
• delaycompress starts compression from the second rotated file (.1 remains uncompressed for easier access)
• missingok don't panic if the file is missing
• notifempty skip rotating if the file is empty
• postrotate runs a command after rotation

Note that this configuration extends some of the default directives set in my /etc/logrotate.conf file in Ubuntu 24.04:

• su root adm drops privileges to the specified user/group
• create ensures that the new file is created in place of the rotated one

The postrotate directive is perhaps the most confusing one. The command that you need to run after a rotation will depend on what's typically accessing the log file, and whether it needs to know about the rotation. Some programs (like Nginx) keep an open file descriptor to the log file, so you'll need to signal them (typically a HUP or USR1) to make sure they close and re-open that descriptor, to avoid continuing to write to the rotated file.

In other cases, like a wp-content/debug.log you won't need a postrotate at all, because PHP workers don't keep any descriptors open once the request has finished processing.

All the available configuration directives can be found in the manual.

Testing and debugging

You can test a full rotation of log files using:

sudo logrotate -f /etc/logrotate.conf

Or a specific configuration, using:

sudo logrotate -f /etc/logrotate.d/custom

You can use the --verbose flag for verbose output, or the --debug flag, which ensures no changes are actually made to the logs or the logrotate state file. This is useful to ensure your directives are doing exactly what you expect, especially when working through compliance requirements.