Rate limiting with Nginx

Rate limiting is a common technique to prevent overloading the server with too many requests. This can help protect our site availability when under stress from crawlers, bots, scrapers, AI agents and malicious actors.

In this lesson we'll look at rate-limiting PHP execution using the ngx_http_limit_req_module in Nginx.

How rate limiting works

Nginx uses a "leaky bucket" algorithm for every rate limiting zone we define. Each zone allows us to limit requests based on a key, which can be an arbitrary variable, such as the user's IP address.

For example, here is a rate limit zone called ip_flood:

limit_req_zone $binary_remote_addr zone=ip_flood:10m rate=10r/s;

This zone is keyed by the user's IP address in binary format, and allocates a 10 megabyte shared memory zone, enough to store about 8k unique keys on 64-bit platforms. The maximum request rate is set to 10 per second.

The zone is typically defined in the http {} context of the Nginx configuration, then used in a server {} or location {} context, for example:

location ~ \.php$ {
    limit_req zone=ip_flood nodelay;
    # ...
}

The limit_req directive uses our ip_flood zone for rate limiting, discarding any requests above the limit due to the nodelay flag. Given its context, this rate limit applies to PHP processing only, so images and other static assets will not be limited.

It's important to understand that this specific configuration allows us to process 10 requests spaced out evenly in a single second. In other words, only one request every 100 milliseconds. Two requests in the same 100ms will hit the rate limit.

This behavior is typically okay when protecting against bots and crawlers, however it could be unexpected for typical user behavior through a web browser, where requests are usually sent in bursts, with longer times between each burst. This is where the burst option may help:

limit_req zone=ip_flood burst=9 nodelay;

This configuration allocates 9 slots for burst capacity, which means up to 10 requests will be accepted during the first 100ms. The overall request rate will still not exceed 10 per second and the burst capacity will be refilled at the rate of one slot per 100ms. This is a much better fit for regular browser behavior.

Using Nginx maps for rate limiting

When working with multiple rate limit zones for various different conditions, maintaining multiple location blocks each with its own set of limit_req directives quickly becomes messy. Luckily, the key argument for limit_req_zone accepts any variable, not just the user's IP, and if that key happens to be empty, the rate limiting is skipped for that request.

This allows us to use Nginx maps, for example:

map $uri $limit_php_login {
    default ''; # do not rate limit
    /wp-login.php $binary_remote_addr;
    /xmlrpc.php $binary_remote_addr;
}

This map defines a $limit_php_login variable, which can now be referenced as the key in a rate limiting zone:

limit_req_zone $limit_php_login zone=php_login:10m rate=30r/m;

And then used in the same PHP location block:

location ~ \.php$ {
    limit_req zone=ip_flood burst=9 nodelay;
    limit_req zone=php_login nodelay;
    # ...
}

Even though this limit appears to be set for all PHP requests, the map ensures that it is applied to only the necessary URLs, when $limit_php_login is not empty.

When a location block has multiple limit_req directives defined, they are all applied in order, and the request will fail if any of the assigned zones needs to be limited.

What to rate limit

It's worth noting that a typical WordPress site will do more than one request in many cases, for example favicon.ico, load-scripts.php and load-styles.php will all attempt to hit PHP. The WordPress admin heartbeat is admin-ajax.php based, and any REST routes are also served by PHP. WooCommerce and other plugins can make plenty of AJAX and REST API requests on the front-end as well.