How to Add File Caching in Nginx (and Actually Check It Is Working)

If you are serving the same files again and again from your backend, say PDF documents, images, or any static download, then every single request is travelling all the way to your application server and your storage layer, even when nothing has changed. This is wasteful. The file is the same, but your backend is doing the same work over and over.

Nginx can fix this without touching a single line of your application code. You put Nginx in front, let it keep a copy of each file on its own disk, and from then on it serves the repeated requests itself. Your backend and your storage (in our case MinIO) stop seeing the repeats entirely.

In this post, I will walk you through the whole thing, the way I actually set it up on one of our servers. We will cover the config, one important trap around range requests that can silently corrupt your downloads, and most importantly, how to check that the cache is really working and how much disk it is eating up.

The Basic Idea

Let us keep it simple first. When someone requests a file, say /api/v1/files/abc-123 here is what normally happens and what happens after caching.

Without caching, every request goes the full distance:

Browser  →  Nginx  →  Backend  →  MinIO

With caching, only the first request goes that far. Nginx keeps a copy, and after that:

Request 1:  Browser  →  Nginx  →  Backend  →  MinIO   (Nginx saves a copy)
Request 2:  Browser  →  Nginx                          (served from disk, backend is idle)

That second request never disturbs your backend at all. On a site where the same documents get downloaded hundreds of times, this is a huge saving for basically zero effort.

Step 1: Declare the Cache Storage

The first thing Nginx needs is a place to store the cached files and a bit of memory to track them. This goes at the top of your http {} block. It must sit here, not inside a server or location block; otherwise Nginx will refuse to start.

http {
    # This tells Nginx where to keep cached files and how much memory
    # to use for tracking them.
    proxy_cache_path /var/cache/nginx/files
                     levels=1:2
                     keys_zone=files_cache:100m
                     max_size=50g
                     inactive=30d
                     use_temp_path=off;

    # ... rest of your config ...
}

Let me explain each part in plain language, because these names are not obvious.

levels=1:2 Just tells Nginx to spread the cached files across nested subfolders instead of dumping everything in one giant directory. This keeps the filesystem fast.

keys_zone=files_cache:100m Gives your cache a name (files_cache, we will use this name again later) and sets aside 100 MB of RAM to hold the keys, which is roughly enough for 8 lakh files. Note that this 100 MB is only for the tracking data, not the files themselves.

max_size=50g is the ceiling for actual disk usage. Once the cache grows past 50 GB, Nginx starts throwing out the least recently used files.

inactive=30d means that if a file has not been accessed for 30 days, Nginx removes it, even if plenty of space is left.

use_temp_path=off tells Nginx to write directly into the cache folder instead of writing to a temp location first and then copying. This avoids an unnecessary copy across the disk.

Step 2: Add the Caching Location Block

Now we tell Nginx to actually cache the file route. This goes inside your server block. If you already have a general /api/v1/ location, put this more specific one just above it.

# Cache the file download route.
# This is more specific than /api/v1/ so Nginx picks it for file paths.
location /api/v1/files/ {
    proxy_pass http://127.0.0.1:8000/api/v1/files/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # Pass the Range header through so partial requests still work,
    # and include it in the cache key (this is the important bit,
    # explained below).
    proxy_set_header  Range $http_range;
    proxy_cache       files_cache;
    proxy_cache_key   "$uri$http_range";
    proxy_cache_valid 200 206 30d;

    # If many people request the same cold file at once, let only
    # one request fetch it while the others wait. Stops a stampede.
    proxy_cache_lock  on;

    # Keep serving the old cached copy while refreshing, or if the
    # backend has a hiccup.
    proxy_cache_use_stale updating error timeout http_500 http_502 http_503;

    # This adds a header so we can see HIT or MISS while testing.
    add_header X-Cache-Status $upstream_cache_status always;
}

Notice proxy_cache files_cache refers back to the name we gave in step one. And proxy_cache_valid 200 206 30d says: cache both normal responses (200) and partial responses (206) for 30 days.

The Range Request Trap (Please Do Not Skip This)

This one caught me out, and it is worth understanding, because when it goes wrong, it goes wrong silently.

PDF viewers in the browser are clever. They often do not download the whole file at once. Instead, they ask for just a part of it, like this:

GET /api/v1/files/abc-123
Range: bytes=0-1023          (just give me the first 1 KB)

The backend replies with 206 Partial Content and only those bytes. Now here is the problem. By default, Nginx builds its cache key only from the URL. So Nginx would store that tiny 1 KB partial piece under the key for abc-123. Then when the next user asks for the full file, Nginx happily hands them the cached 1 KB, thinking that is the complete file. The download is corrupted, and there is no error anywhere. This is called cache poisoning.

The fix is exactly what we did above. We added $http_range into the cache key:

proxy_cache_key "$uri$http_range";

Now each byte range gets stored as its own separate entry, and a partial request can never pollute the full file entry. Small line, big difference.

Step 3: Create the Folder and Set the Right Owner

Before you reload Nginx, the cache folder must exist, and Nginx must be allowed to write to it.

sudo mkdir -p /var/cache/nginx/files

Now, which user owns it? This trips up a lot of people. On some systems Nginx runs as nginx, on Debian and Ubuntu usually www-data, and on some builds it runs as nobody. Do not guess. Check which user your Nginx workers actually run as:

ps aux | grep '[n]ginx: worker'

Look at the first column of the output. On my server it showed nobody, so I used:

sudo chown -R nobody:nogroup /var/cache/nginx/files

If yours shows www-data, then use www-data:www-data instead, and so on. Match whatever your workers are running as; otherwise, Nginx cannot write to the folder and caching silently does nothing.

Step 4: Test the Config and Reload

Always test before reloading. This checks for syntax mistakes so you do not take the site down.

sudo nginx -t && sudo systemctl reload nginx

If it says the test is successful, you are live.

How to Check Cache Hits and Misses

This is the part everyone wants. How do you actually know it is working and not just sitting there doing nothing?

Because we added the X-Cache-Status header, we can just look at the response headers. Run this twice on a real file:

curl -sI https://your-domain.gov.np/api/v1/files/<some-uuid> | grep -i x-cache

The first time, the file is not cached yet, so you will see:

X-Cache-Status: MISS

Run the same command a second time, and now it should show:

X-Cache-Status: HIT

That HIT is your proof. The second request was served entirely by Nginx from its own disk, and your backend never even heard about it. You may also occasionally see LOCK (a request waiting for another one to finish fetching) or UPDATING (serving an old copy while refreshing), both of which are normal and healthy.

How to See the Cached Files and Disk Space Used

The cache folder stays empty until a file is actually requested. It fills up lazily, only when needed. Once some files have been requested, you can inspect the cache.

To list everything that has been cached:

sudo find /var/cache/nginx/files -type f

To count how many files are cached right now:

sudo find /var/cache/nginx/files -type f | wc -l

To see how much disk space the whole cache is occupying, this is the one you will use most:

sudo du -sh /var/cache/nginx/files

That gives you a single clean number like 2.3G, which is the total size of everything cached so far.

One thing to note: these cached files are not plain PDFs. Nginx adds its own little header at the top of each file containing the key, timestamps, and status before the actual file content. If you are curious, you can peek at one and see the key it was stored under:

sudo head -c 500 /var/cache/nginx/files/<some-file> | strings

Look for the KEY: line. It shows exactly what got cached, including the range, which is a nice way to confirm your range keying is working as intended.

One Warning About Deletes

There is one thing to keep in mind. Because Nginx holds its own copy, if you delete a file from your backend and storage, Nginx will happily keep serving the cached copy until it expires, which in our config is up to 30 days.

For most public documents this is fine. But if you have a delete route and it matters that a deleted file disappears immediately, then the plain open source Nginx has no built in purge. You would need to add the ngx_cache_purge module and have your delete handler tell Nginx to remove that specific key. If you do not want to go that far, a simpler compromise is to lower proxy_cache_valid to a shorter time so deleted files fall out of the cache sooner.

Wrapping Up

That is the whole thing. To recap what we did:

We declared a cache storage area in the http block, added a caching location for our file route, made sure to include the Range header in the cache key so partial requests do not corrupt full files, created the folder with the correct owner, and then verified everything using the X-Cache-Status header and a couple of simple shell commands to check disk usage.

The best part is there was no change to the application at all. Nginx sits quietly in front, absorbs all the repeat traffic, and your backend and storage get to breathe. For a bit of config, it is one of the highest-value things you can do for a file-heavy site.