How to Configure Nginx Rate Limit and Whitelist

Web servers often need to handle traffic spikes and protect against abuse. Nginx Rate Limit lets you control how many requests the server processes within a given time frame. This tutorial covers configuring Nginx Rate Limit for server protection.

Requirements

Before starting, make sure you have:

• A web server running Nginx
• Basic knowledge of Nginx configuration files (nginx.conf)
• Administrative access to change the server’s configuration

Obtaining the Real Client IP

Getting the real client IP address matters when you want to apply rate-limiting rules accurately. By default, Nginx gets client IP information from the connecting TCP socket. If your server sits behind a reverse proxy or load balancer, you must configure Nginx to extract the real client IP from the headers.

Add the following configuration directive within the http block of your nginx.conf file:

http {
  real_ip_header X-Forwarded-For;
  set_real_ip_from <ip_address_of_proxy>;
}

Replace <ip_address_of_proxy> with the actual IP address of your reverse proxy or load balancer.

Managing Security Concerns

Getting the real client IP makes rate-limiting configuration simpler, but it introduces security risks. Attackers can forge or modify headers to bypass rate-limiting rules by pretending to be a different IP. To reduce this risk, enable trusted proxy validation.

Add the following directive inside the server block of your nginx.conf file:

server {
  ...
  set_real_ip_from <ip_address_of_proxy>;<optional_list_of_trusted_proxies>;
  real_ip_header X-Real-IP;
  real_ip_recursive on;
  ...
}

Replace <ip_address_of_proxy> with the actual IP address of your reverse proxy or load balancer. You can also include additional trusted proxies separated by semicolons (;).

Let’s go deeper on the set_real_ip_from directive.

Understanding the set_real_ip_from Directive

The set_real_ip_from directive lets you specify which IP addresses Nginx should trust for real client IP information. This matters when using Nginx behind a reverse proxy or load balancer, since it ensures Nginx identifies the actual client IP address for rate limiting and logging.

By default, Nginx gets client IP information from the connecting TCP socket, which may be the IP of the proxy or load balancer rather than the actual client. The set_real_ip_from directive tells Nginx to replace the client IP with the one specified in the headers from the proxy server.

Syntax

The syntax for set_real_ip_from is:

set_real_ip_from <ip_address_or_block> | <variable_value>;

• ip_address_or_block>: A single IP address or an IP address block using CIDR notation.
• variable_value>: A variable name containing the IP address or addresses.

Multiple IP Addresses

You can specify multiple IP addresses or blocks by separating them with a semicolon (;). For example:

set_real_ip_from 192.168.1.1;
set_real_ip_from 10.0.0.0/16;

Nginx will trust the client IP information from any IP address within the specified range.

Trailing Semicolon

If you include multiple set_real_ip_from directives, each line must end with a semicolon except for the last one. The trailing semicolon indicates that additional IP addresses or blocks will follow. The final line should not have a trailing semicolon.

set_real_ip_from 192.168.1.1;
set_real_ip_from 10.0.0.0/16;  # No trailing semicolon here

Variable Value

Instead of specifying the IP address or block directly, you can use a variable to store the IP address and retrieve it dynamically. For instance:

map $http_x_custom_header $proxy_ip {
  default 192.168.0.1;
}
set_real_ip_from $proxy_ip;

Here, the value of $proxy_ip comes from the X-Custom-Header HTTP header sent by the proxy server.

Order of Execution

Nginx processes set_real_ip_from directives from top to bottom. If multiple directives overlap with the same IP or block, only the last one takes effect.

set_real_ip_from 192.168.1.0/24;
set_real_ip_from 192.168.1.100;

In this example, only requests where the client IP matches 192.168.1.100 will have their IP replaced, even if they fall within the 192.168.1.0/24 range.

Summary

The set_real_ip_from directive is an important configuration option in Nginx when dealing with reverse proxies or load balancers. By specifying which IP addresses or blocks Nginx should trust to extract the real client IP, you get accurate identification for rate limiting, logging, and other purposes.

Understanding how to use set_real_ip_from and its various syntax options lets you configure Nginx correctly for your specific environment, allowing precise management of incoming requests based on real client IP addresses.

Should You Turn real_ip_recursive On or Off?

When configuring your NGINX server, you might consider using real_ip_recursive. This directive processes proxy servers recursively. It can help in some situations, but it is not necessary or ideal for every setup. Here are some factors to consider when deciding whether to turn real_ip_recursive on or off.

Understanding Real IP Recursive

Before getting into specifics, here is a simple explanation of what real_ip_recursive does.

The real IP module in Nginx finds the original client IP address and stores it in a variable like HTTP_X_REAL_IP when using more than one load balancer. Applications can use this when knowing the client’s IP is useful.

The real_ip_recursive directive is part of this module and is off by default. When enabled, it tells NGINX to replace the client address with the one from the latest non-trusted intermediate proxy.

When to Use Real IP Recursive

Turning this option on helps if you have chained proxies before requests reach your server. If each proxy adds its IP to the X-Forwarded-For field and real_ip_recursive is off, only the first (leftmost) non-trusted IP will be treated as the client’s real address, which could still be an intermediate proxy.

If you enable real_ip_recursive, it processes all intermediate hops until it finds a non-trusted IP further down in the X-Forwarded-For field. That becomes the client’s real address.

So if you are dealing with multiple chained proxies, setting real_ip_recursive may better fit your needs since it lets nginx recursively parse X-Forwarded-For headers for a trusted client IP.

When Not To Use Real IP Recursive

If your setup does not include multiple proxies (reversed or chained), having real_ip_recursive on may be excessive and unnecessary.

Remember that enabling this feature without understanding appropriate use cases and configurations of trusted proxies could lead to problems. You could end up trusting an unsecured or incorrect source IP address.

Recap

Whether you should turn real_ip_recursive on or off depends on your configuration and needs:

• Turn On: If you are working with multiple chained proxy networks where intermediate hops exist between the client browser and nginx server.
• Turn off: For simpler applications without multi-level proxy networks.

As with any server configuration decision, test carefully before deploying. Reading NGINX’s official documentation can help with confusing directives like real_ip_recursive.

Configuring Nginx Rate Limit

This tutorial walks through configuring rate limits in Nginx. Rate limiting controls the requests a client can make within a specified period.

Step 1: Define Rate Limit Parameters

To define the rate limit, specify three parameters:

• $limit_req_zone: This directive sets the shared memory zone for rate limiting. Define it in the http block of the Nginx configuration file.

http {
    limit_req_zone $binary_remote_addr zone=mylimit:10m rate=1r/s;
}

This creates a shared memory zone called “mylimit” with 10 megabytes of storage and a limit of 1 request per second per client IP address.

• $limit_req: This directive applies rate limiting within a specific location block. Set the desired limit and define what happens when a request exceeds it.

location /api {
    limit_req zone=mylimit burst=5 nodelay;
    ...
}

This applies rate limiting to the /api location. The burst parameter sets how many requests can exceed the rate limit before further requests get delayed. The nodelay parameter ensures additional requests beyond the burst limit are not delayed.

Step 2: Customize Error Messages (Optional)

If a client exceeds the rate limit, Nginx returns a 503 error by default. You can customize this error message to provide more useful feedback to users.

error_page 503 @ratelimit;

location @ratelimit {
    return 429 "Too Many Requests";
    # You can also use an HTML page or redirect to another endpoint as per your requirements
}

This defines an error_page directive to catch the 503 error. Within the @ratelimit location, it returns a 429 status code and the message “Too Many Requests”. You can modify this response as needed.

Step 3: Test and Monitor

After configuring the rate limit, test and monitor its effectiveness. Send requests to your server and check if the rate-limiting rules are being enforced correctly.

Use tools like curl or load testing software to simulate different traffic scenarios and verify the rate limits work as expected.

Defining a Whitelist for VPN Users

Sometimes you may want to exclude certain IP addresses or IP ranges from rate-limiting rules. For example, if your organization uses a VPN service that assigns the same egress IP to multiple users, you can create a whitelist to exempt them from rate limiting.

Add the following configuration inside the http block of your nginx.conf file:

http {

  geo $whilelist {
        default 0;
	      187.120.188.233 1;
        50.19.172.14/32 1;
        2a03:eec0:1415::/48 1;
    }

    map $whilelist $limit {
        0 $binary_remote_addr;
        1 "";
    }

    limit_req_zone $limit zone=global-zone:10m rate=10r/s;
    limit_req zone=global-zone burst=5 nodelay;
    limit_req_status 429;

  ...
}

Replace IPs with the IP range assigned to your VPN service. Use the $limit variable in rate limiting rules to exclude VPN users from restrictions.

Important

The configuration above only works if you are using $real_ip_header and $set_real_ip_from.

The ngx_http_realip_module gets the real client from a header like X-Forwarded-For and replaces the value on $binary_remote_addr.

This means if you are not using $real_ip_header and $set_real_ip_from, the $binary_remote_addr will have the wrong IP.

Also, if you are using a WAF like Imperva, you must define all IPs from Imperva as trusted sources to look for X-Forwarded-For and use that to get the client IP. Because the WAF sits in front of your application, it holds the X-Forwarded-For header with the client IP.

Besides WAF, if you have a Load Balancer, you must also specify the IP range or IP from the Load Balancer as a trusted source (set_real_ip_from) to define the Client IP (real_ip).

Second and Most Important

limit_req_zone $http_x_forwarded_for zone=global-zone:10m rate=10r/s;

This line sets a request rate limit using the limit_req_zone directive in the nginx configuration file.

The limit request zone is determined by $http_x_forwarded_for, which usually contains the client’s IP address when the request was forwarded to nginx via a reverse proxy. Here is the security issue.

If an attacker knows you are using $http_x_forwarded_for to set your rate limit, they could spoof this value with any IP address they choose. By changing their IP address for every request, they can bypass the rate limitations and send more requests per second than the defined limit of 10 requests per second (10r/s).

The zone=global-zone:10m part declares a shared memory zone to keep track of all incoming requests, holding up to 10MB of data (enough for about 160,000 IP addresses). If an attacker can spoof their IP address consistently, they could fill up the allocated memory zone and prevent legitimate users from accessing your web resources.

Additionally, if you have multiple proxies before reaching nginx, $http_x_forwarded_for may contain a list of IP addresses. By default, without parsing, this directive might handle it incorrectly and take only the first or last value, which could be easily spoofed.

For a more secure setup, use limitations on values that malicious actors cannot manipulate, such as $binary_remote_addr, which uses the direct remote address even if requests pass through multiple proxies. Trust each proxy involved in forwarding HTTP requests to Nginx and configure them securely.

To make your configuration more secure and reduce the risks of IP spoofing when using $http_x_forwarded_for, use the real_ip_header and set_real_ip_from directives instead.

Check official nginx documentation and consult with security professionals when configuring these parameters.

Define Rate Limit by API Key

Besides limiting by client IP address, Nginx can limit rate by using an API key. This method gives more precise management of API usage and makes tracking usage patterns easier.

Use the limit_req_zone directive with the $http_api_key variable in your nginx configuration file:

http {
    limit_req_zone $http_api_key zone=api_zone:10m rate=1r/s;
}

This creates a shared memory zone called api_zone with a limit of 1 request per second per API key.

Next, use the limit_req directive within a specific location block to apply the rate limit:

location /api {
    limit_req zone=api_zone burst=5 nodelay;
    ...
}

This implements rate limiting for the /api location using the api_zone shared memory zone. The burst parameter sets the maximum number of requests allowed beyond the rate limit before further requests get delayed. The nodelay parameter ensures additional requests beyond the burst limit are not delayed.

Clients must include an API key in their requests to use this approach. Provide the API key to authorized clients and track their usage to ensure they do not exceed their allotted rate limit.

Conclusion

For defense in depth beyond Nginx, AWS WAF rules add rate-limiting and geo-blocking at the CDN layer. Protecting APIs? Cognito JWT authorizers handle authentication before traffic reaches your server.

By configuring Nginx Rate Limit, you can protect your server from high-volume traffic and potential DDoS attacks. Get the real client IP by configuring Nginx to extract it from headers, and enable trusted proxy validation to protect against header forgery. Use whitelists to exempt specific IP addresses or ranges from rate limiting.

Monitor your server’s performance and adjust rate-limiting settings to balance security and user experience. Stay alert for emerging threats and regularly update your web applications to maintain protection.

Additionally, rate limiting based on API keys gives more granular control over API usage and lets you track usage patterns. Monitor your server’s performance and adjust rate-limiting settings to balance security and user experience.