photo credit: York Autograss via photopin (license)

How to get Turpentine right for Magento

If you got a Magento shop and you’ve boosted the performance of it by using Turpentine (the Varnish plugin for Magento) you’ve already experienced the massive speed and power it provides.
If you’re Turpentine configuration in your local.xml  file is misconfigured, chances are that under heavy load, with many concurrent visitors, you’re Magento site may experience very long load times, and even in some cases 502 gateway time-out errors.
So what gives?

How Turpentine works

To figure out how Turpentine works you can add the following line on top of your index.php :

And the following to the bottom:

What this line does, is each time index.php  is loaded (on every Magento request) it logs how long the page took to render and stores this in var/log/counter.log . When the complete page is cached (without ESI blocks) the content of the file will look similar to this after the initial request to the homepage (with an empty Varnish cache):

When everything is correct, each request after this won’t add a line to this file. This way we can be sure that everything comes from the Varnish cache and Magento isn’t loaded on every request.

Turpentine and ESI blocks

Turpentine also offers the possibility to make use of ESI blocks in Magento. If you don’t know what ESI blocks are or how they work, I recommend reading this article first.
When you make use of ESI blocks in Magento (and Turpentine already has some ESI blocks configured by default in turpentine_esi.xml ), you notice that for each ESI block an individual request is made to Magento. Take a look at your counter.log -file for example:

As you see, this single request to index.php  made 2 additional requests under water to load the individual ESI blocks. Therefore, the load time of the first request will take somewhat more than 2 seconds, but all requests that come after that are blazing fast (if you look at the TTL, it’s set to 36000).

Configuring ESI blocks in Magento

As the Turpentine documentation states, Turpentine offers the method setEsiOptions()  for every block in Magento. If you’ve applied this method to some blocks in your local.xml -file, you may already have tinkered with this. But as with many things in programming: with great power comes great responsibility, and caching is also one of them.

Common mistake #1: use a TTL of 0

You may think for some parts in your shop: “This block is customer specific so it doesn’t need to be cached.”. This is a very common thought for stuff like a shopping cart or the ‘Hello Mr. X’-message on top of your site.

However it’s very, very bad practice. Setting a TTL to 0 is telling Turpentine that for each time this ESI block is loaded, fetch a new one from the server, slowing the whole page down. Remember the counter.log -file we mentioned before? This will fill up on every page, while you think that Varnish is working great.
And it’s hard to detect this flaw on a development environment with a single user, because you still might get fast responses from the server. But on a production server with 100s of concurrent users at any given time this results in a very huge load on the server, each page request.
The best way to solve this is by just giving it a higher TTL (or omit it to use the default TTL) and register the events on which the cache should be flushed:

This caches the cart with a proper TTL and only flushes it when needed.

Common mistake #2: Session blocks on the initial page load

When you’ve applied the first patch you’ll notice that after the first initial page load all other loads are blazing fast. Expect 40ms load times or less. However, the initial page load still takes a while since ESI blocks that require user-specific data (like the shopping cart for example) still need to be loaded the first time.
And this is also something that you won’t detect on your local development environment, but when you stress-test your shop with a tool like Siege for example and you shoot 100 concurrent requests at once at your shops, you’ll see that the initial load can go from 1-2 second to 10-20 seconds or even more. This is of course unacceptable.
The most easy way to fix this, is to load user-specific assets with AJAX. This way, the page gets loaded blazing fast on initial load, and stuff like login information or the shopping cart gets loaded on the background. You can set this up in your own implementation of Magento, or use Turpentine’s out-of-the-box implementation of this by setting the method -tag in your setEsiOptions() -action:

In conclusion

When you’re configuring your Magento store with Turpentine you can expect great performance gains. But you also have to be careful on how you set it all up and stress-test it under heavy load. That way you can be sure that your site keeps performing under these conditions.
Keeping the 2 common mistakes in mind that are described above are a good way to start. If you have more tips, mistakes or additions to this article you can add them in the comments below this article. I’m always eager to learn and improve.

Visitors give this article an average rating of 4.8 out of 5.

How would you rate this article?

2 thoughts on “How to get Turpentine right for Magento”

  1. Jan Fervers says:

    Thanks a lot Giel! A good look at the most important thing, the base. This ttl 0 thing is one of the most mistakes and it’s very common. You’ll find a lot of threads and wrong recommendations if you’re search for this phrase.

    Best regards

  2. Martin says:

    Hi there, do you have experience with turpentine and varnish 4.0.

Leave a Reply