Using Varnish with ESI for a REST API

Over the last years Varnish has proven to be a great reverse proxy for dynamic websites. On top of great performance it offers features like ESI, allowing you to efficiently compose content with different caching characteristics. For a good introduction take a look at this presentation:

For a project I’m currently working on we are implementing a REST API. Performance for this API is critical, so amongst some other solutions Varnish is used. This was done with minimal effort, as the API already had the correct caching headers. There was a noticeable improvement, however I had the feeling this could be improved much more.
I realised I can basically compare our REST API to any website, it uses HTTP in the same way. The main diffence is the content being JSON instead of HTML. Just like most websites, our API composes content in many different ways, for instance ‘standalone’ resouces, collections of resources or embedded resources.
In a website this would be a clear use case for ESI, so why not use it in our API?

Why use ESI in a REST API

In our API, and probably any REST API, two features are heaviliy used:

If we use ‘standard’ caching, we have several issues:

  • We have to generate the same resource lots of times, for each time it’s embedded or included in a collection
  • The same resource ends up in a huge number of cache objects, making purging a headache at the least
  • Lots of duplicate/similar data in cache, the exact same resource might be taking up valuable space in thousands of cache objects

By using ESI to include the resources into collections or as an embedded resource we instantly solve these issues:

  • The resource is only generated once, and ESI-included in all collections / embeds
  • It ends up in just one cache object, saving lots of cache space
  • Because it’s just in one cache object, you can easily purge it using one purge call to the resource URI. If you have another resource that embeds this updated resource, it will get the updated version ‘injected’ automatically using ESI.
  • On top of that, you can also easily vary the cache lifetime of each resource.

Here are two diagrams, displaying the difference for a simplified book API:



This is the  ‘standard’ caching scenario. Several things to notice:

  • Several items are generated multiple times (for instance ‘Category X’)
  • Several items are in cache multiple times (‘Category X’ and ‘Author A’)
  • Purging is very complex, one update results in multiple purges. Apart from this being inefficient and hurting your cache hitrate, how do you even know which purges to do?
  • What would happen if ‘category X’ is updated? All cache would be invalidated and all (9) items would have to be regenerated!

With ESI

With ESI

Now the alternative scenario, using ESI:

  • Each item is generated only once
  • Each item is in cache only once
  • Purging is very simple, one purge for each item
  • The complexity is now in composing the cache items into combined responses. But this is managed by Varnish, you only need to provide the correct ESI includes in the book items. This can be done at the time you generate the book item, and know what other items you want to embed. You just add the ESI include tag and you’re done.
    Varnish will keep a record of the references, and correctly processes purges with a partial update. In the previous scenario you had to keep track of all references, just to know what to purge for updates.
  • Compare this with the category update of the previous scenario. This time it results in just one purge, and only one item to regenerate!

How to implement

Now that we know the benefits of using ESI, how can we implement it? It’s similar to any ESI implementation (see for an intro) but some extra steps are needed to get it working correctly for JSON output.

First of all, by default Varnish will not parse any non-HTML content for ESI statements. Even if you set do_esi  to true. JSON will also not be parsed. This can be changed with a Varnish runtime parameter in your Varnish startup configuration file:

Now ESI should be parsed in JSON. Please be aware that this setting enables the ESI parsing for any type of files if do_esi  is set to true in your Varnish configuration (VCL). So you need to be extra careful with that setting, to prevent expensive parses of big files.

Secondly, you need to include ESI include tags in your JSON correctly. I’m using the PHP json_encode function to generate json output efficiently. But there is an issue in adding ESI tags in your content, consider this example:

This will return the following:

However, the quotes around the ESI tag are an issue. The ESI include wil return a JSON object, that will be placed inside the quotes. So, we need to get rid of the quotes. There are various options like manual JSON encoding, regex, I’m currently using a simple approach that performs well with placeholders:

This results in the correct output:

The JSON object returned by the ESI tag will now be placed in this JSON correctly.
The implementation is simplified for this example, in reality this is a helper function that also takes care of route generation. But it demonstrates the issue and a possible solution. If you don’t use a helper and just want to remove the quotes this could also be done using a regex. As far as I know it’s not possible to include a ‘raw’ value in json_encode, that would be ideal for this case.


So, as you can see ESI can be used in an API relatively easily, with some great advantages in return!

Be aware though that it’s not a drop in solution that will work for all scenarios. For instance if a resource gets deleted, any cached object that references this resource using ESI needs attention. There are various ways to handle this, it really depends on your use case. As always, you need to be very careful with caching and think things through!

Bas de Nooijer is a freelance developer with a strong PHP background (Zend Certified Engineer). Besides PHP he also specializes in Varnish and Solr, also for non-PHP projects. He created the opensource PHP Solr client “Solarium“.

Dutch Web Alliance