MaxCDN Blog

Accept-Encoding, It’s Vary important.

February 4, 2013

One of the best things about running BootstrapCDN are the new things I’ve learned about web performance. Today, I’d like to share a few insights gained while resolving an issue originally brought up by a peer. The concern was to include the Vary: Accept-Encoding header — it’s not on every server response, but it should be. Here’s why.

When browsers make a request, they include HTTP headers for the server to decide what to send back (Is this a mobile client? Can it handle compressed content? Does it need a certain language?).

That’s great for direct access, but modern networks use intermediate caches and CDNs. And there’s the problem: how does the cache use headers to decide what to send back? How can it replicate the server’s decision-making logic?

Vary to the rescue. The Vary header describes what information “uniquely” identifies a request — caches should only be used if the incoming request matches the Vary information in the cache.

For example, if a server sends the Vary: User-Agent header, intermediate caches will store a separate cache entry for each User-Agent they see (every OS + browser combination, yikes). This behavior was an issue for me in support (we’re hiring!), because we saw origin servers getting hammered as each user-agent requested new content and sidestepped the cache. After some research, I figured out why this happened (turn off Vary: User-Agent), but the header left a bad taste in my mouth.

Well, the BootstrapCDN issue came up, so I decided to give Vary another look. I went to one of my favorite WebPerf education sites and found this article that explains gzip & Accept-Encoding.  After following the communication graphs, I thought “If the Origin Server, CDN and Browser support gzip-encoding, why is an extra header needed?”

There had to be a good reason, so I asked SO.  Two people I respect highly, and had the pleasure of working with at NetDNA (again, we’re hiring!), answered the question:

“If for some reason the client has an uncompressed version of the file in its cache, it will know not to subsequently request a compressed version of it again and instead to just use the uncompressed file from the cache.” – Kyle Rush 

“Like Kyle said… just replace “client” with a “upstream proxy” (isp, corporate network, etc). So you have the risk of serving uncompressed version to end user that supports gzip, and vice versa.” – Sajal Kayan

Imagine two clients: an old browser without compression, and a modern one with it. If they both request the same page, then depending on who sent the request first, the compressed or uncompressed version would be stored in the CDN. Now the problems start: the old browser could ask for a regular “index.html” and get the cached, compressed version (random junk data), or the new browser could get the cached, uncompressed version and try to “unzip” it. Bad news, either way.

The fix is for the origin server to send back Vary: Accept-Encoding. Now the intermediate CDNs will keep separate cache entries (one for Accept-encoding: gzip, another if you didn’t send the header). These days you’re unlikely to have clients without compression, but why risk cache mixups?

Origin servers should include Vary: Accept-Encoding, and here’s how:


<IfModule mod_headers.c>
  <FilesMatch ".(js|css|xml|gz|html)$">
    Header append Vary: Accept-Encoding


gzip_vary on


    <remove name="Vary"></remove>
    <add name="Vary" value="Accept-Encoding"></add>

For those of you reading that are our clients, a pull zone using our CDN will pass the “Vary” header from your origin server – so feel free to set that up! If you don’t have good control of your origin server, you can easily add the header using our EdgeRules™ product.

Happy Encoding ;)

  • sajalkayan

    Nice post. If the content is “varied” based on any request header, that should be declared in the Vary response header.

    A related post :

    Thats for setting up IIS correctly to work with few CDNs, but is also relevant for keeping upstream proxies(ISP, corp firewalls) in mind.

  • Marcelo Sulca Nieto

    Gracias por tu ayuda estuve buscando esto
    muchas gracias very good

  • Freddywang

    I think MaxCDN ignores this at the moment. I have js/css resources that return both gzip and non gzipped content depending whether browser send it Accept-Encoding: gzip. But MaxCDN always return me non-gzipped content despite I am using browser that support gzip compression. A huge bummer.

    • jdorfman

      @freddywang:disqus please email me the urls so I can debug. jdorfman at maxcdn dot com

  • jdorfman
  • Joshua Mervine

    Fantastic post! Great info!

  • belidomain

    Very good info, but how much percentage it could help our web performance? thanks.

  • Josh Muccio

    Hey Justin! Thanks for the great post BTW. Seriously cool stuff you all are doing. My site runs on Nginx and I’ve confirmed that I have “gzip_vary on” enabled in my config file. It looks like the external assets of the page are still not using the Accept-Encoding header. Do I have to do this manually (in html) for each external asset that I load?

  • Randy Hudson

    Any half-decent “intermediate cache” should treat content encoding no different than transfer encoding. It is a fundamental HTTP header that the cache should understand. If it has a cached version that is gzipped, and some client asks for an uncompressed version, it should serve the gzipped version, decompressing it on the fly. Similarly, if it doesn’t have the content cached, it could request it from the origin server, but modifying the request such that the hop between the cache and origin server can be compressed, even though the response back to the client won’t be.

  • Maxime Jobin

    Great resource about how vary is important and how it can be handled. Thank you.

  • the1900

    wow thanks a lot I didn`t know about vary field on http header before

  • Secured Zip

    Worked for us after deploying nginx with pagespeed along the side of it. This is actually set by default for Nginx with gzip_vary on, so if you don’t know how to place it or set it to “on”, just nano or vim to /etc/nginx/nginx.conf