Handling Redirects with Varnish and Nginx

serversetup

[ NOTE: I am now using Nginx for everything, i.e. not using Varnish anymore, and getting the same or better results. ]

I run Varnish here on the site, with Nginx as the backend. I’ve written before about my overall setup, and how to improve site performance, so I won’t go into it here.

Here I want to cover a subtlety of putting varnish in front of nginx (or any other web server, really) with respect to redirects. Redirects occur when the client makes a request to the server that cannot (or should not) be served in the manner asked, e.g. asking for /resource instead of /resource/. The trailing slash matters, as those to URLs look the same to humans but are not the same to the web server.

Web servers such as Apache and Nginx handle this naturally by sending clients a redirect for the latter when you ask for the former, i.e. you ask for /resource, you get sent back a redirect (HTTP code 301 or 301) saying that you should be asking for /resource/.

Simple enough. This all happens transparently to the client, as the initial request, the redirect response, and the second request happen so quickly that most people don’t even notice the turnaround.

But things can get a bit weird when you have two web daemons fielding requests, like when you have Varnish in front of Nginx. The problem is that the two daemons have to be listening on different ports. Varnish and Nginx can’t both be on the same exact IP address and port combination (TCP/IP stacks don’t like that), so you have to perform some sort of networking trickery to get them to work.

One common way of doing it is to have Varnish listen out front on port 80, and to have Nginx listen in back on 8080, or 81, or whatever. Then you tell Varnish what port Nginx is on and everything works great.

Until someone asks for /resource.

When that happens, Varnish asks Nginx for the resource and gets a redirect, only the redirect isn’t to a usable URL — it’s to the backend URL. So instead of being sent to site.com/resource/ like you would be if Varnish weren’t there, you get sent to site.com:8080/resource/, which your users shouldn’t be able to connect to (you do have a firewall on your server, right?).

Unsupervised Learning — Security, Tech, and AI in 10 minutes…

Get a weekly breakdown of what's happening in security and tech—and why it matters.

But there’s a simple solution.

TL;DR: Have Varnish listen on 1.2.3.4:80 (your external IP), and have Nginx listen on localhost:80. This way both daemons are convinced they’re authoritative, so when Nginx responds with redirects the won’t be mangled with some alternative port in the URL that external users can’t reach.

Here’s what the localhost config looks like in Nginx: 

    server {
        listen  localhost:80;
        server_name  localhost;
    ...

Now all your redirects should look as if Nginx is sitting in front, and you should be good to go. Hope this helps.

[ Note: Don’t try to handle this in Varnish itself; this is a backend issue and should be handled by properly situating your daemons, not by rewrite hand-waving on the acceleration server. ]

::

Related posts: