What kind of performance improvement will I see using Evolved Caching?

Using the extension you should immediately see a very significant improvement to page loading times. Not only this, but there will also be a very significant reduction in server resources required to run the site meaning you can reliably serve pages to many more people, much faster. If you don't see this then have a look at the What will Evolved Caching improve on my store? question at the bottom of the main documentation page. It is most likely that you need to do one or both of:

  1. Upgrade your hosting
  2. Reduce the number of JS, CSS and image assets per page

Exact performance figures will vary according to server and the amount of content on page, but a general rule is that the larger and more complex a page, the bigger the saving will be. You will also see the best results for a site running on good hosting, with JS and CSS merging enabled, and with as few images as possible per page.

Benchmarks

We put together some benchmarks to give you an idea of the kind of saving you can expect, you can find them below. It's important to note that some other caching solutions may say they have better performance than Evolved Caching in certain areas, but here you will find fair, realistic, achievable performance improvement figures which you can consistently expect to see on any store with an equivalent hardware setup. All of the the benchmarking and load testing was performed on servers with very modest specifications at the lower end of what it is realistic to expect Magento to run on. Some of the tests were also performed on live stores, so these are real world, reproducible performance figures - not just the theoretical best case results from tests performed in a highly controlled and selective environment.

This first set of benchmarks list page load time for a single request. As discussed above, Evolved Caching first loads generic, non user specific HTML, and then loads any dynamic content via an AJAX call. These benchmarks show load times for the home page, a product page and a category page. The first image shows the load time without Evolved Caching, the second image the time taken for Evolved Caching to load the generic, non user specific HTML, and the third image the time taken to complete the AJAX request for dynamic content.

Note that the Waiting figure is the important one here - this is the time it takes for the server to process the request and start sending data to the browser. The time it takes for the browser to then fully load the page depends heavily on the speed of the users connection. Evolved Caching reduces the time it takes for the user to start seeing content render in their browser, no caching solution (Varnish included) will help in reducing the time it takes for the page to finish loading. This is defined by the number of JS, CSS and images assets on the page - the less assets there are, the quicker this process will be.

Lets look at the home page first.

Home Page

HTML generation time without Evolved Caching

HTML generation time using Evolved Caching

AJAX dynamic content request time using Evolved Caching

So you can see that without Evolved Caching, the server takes 1560ms to generate the page HTML, but with Evolved Caching it takes just an incredible 9ms. This means that the server processing time for the request is just 0.58% of what it would take without Evolved Caching, and this is all the time it takes to generate the entire pages HTML and send it to the browser.

Once the browser has loaded this lightening fast page (at which point the user can view and use the page), an AJAX request is sent for the dynamic, user specific content. This in itself only takes the server 179ms to process, making the total server processing time for the entire page, including dynamic content, only 188ms - just 12% of the time it would take without Evolved Caching.

Now lets look at a product page.

Product Page

HTML generation time without Evolved Caching

HTML generation time using Evolved Caching

AJAX dynamic content request time using Evolved Caching

With the product page containing more data, and being more complex than the home page, we see an even greater saving here using Evolved Caching. The server would normally take 2650ms to generate the page HTML, but with Evolved Caching it takes only 10ms - just 0.38% of the time.

Total server processing time is only 194ms to include the AJAX request for dynamic content - just 7.32% of the time it would otherwise take.

Finally lets look at a category page.

Category Page

HTML generation time without Evolved Caching

HTML generation time using Evolved Caching

AJAX dynamic content request time using Evolved Caching

Again the category page also shows a massive performance boost. The server would normally take 2300ms to generate page HTML, but takes only 9ms with Evolved Caching - just 0.4% of the time. Total server processing time is only 192ms to include the AJAX call - just 8.35% of the time it would otherwise take.

But what about when the server is under load?

It's all very well having great load times when you are talking about a single request, but what about when the site is under heavy load with multiple users browsing at the same time? Well, Siege is a great tool for testing the performance of a server under load and allows you to simulate any number of simultaneous users browsing the site. After the tool has completed the test, it gives you various statistics like availability, transaction rate and transaction time. We used Siege to benchmark performance of a server under load both with and without Evolved Caching, here are the results.

Siege testing 20 concurrent users without Evolved Caching

Siege testing 20 concurrent users with Evolved Caching

As you can see in both cases 20 concurrent users are handled fine by the server with availability being 100%. However looking at the other statistics you can see even with just 20 concurrent users Evolved Caching serves much more data, much quicker with the transaction rate being far more than double, and the response time dropping from 1.09 to just 0.04 seconds.

When looking at 50 concurrent users we see an even greater improvement in performance.

Siege testing 50 concurrent users without Evolved Caching

Siege testing 50 concurrent users with Evolved Caching

Again the server is able to handle the number of requests in both cases with availability at 100%. However the other statistics shows the huge boost in performance Evolved Caching offers under load. Transaction rate jumps from just 8.35 to a massive 45.05 transactions a second, with response time dropping from 4.09 to just 0.08 seconds. Other statistics show a similarly substantial improvement in performance.

Lets step the tests up to 100 concurrent users.

Siege testing 100 concurrent users without Evolved Caching

Siege testing 100 concurrent users with Evolved Caching

Under load Evolved Caching really starts to show it's value. You can see that with no Evolved Caching, the server is now unable to fulfill all of the requests, and achieves only 85% availability, while with Evolved Caching continues to easily serve all 100 requests. The statistics speak even more clearly under greater load with the response time staying at just 0.13, compared to 7.60 seconds, and transaction rate rising to an impressive 80, compared to 9.22 transactions a second.

Finally lets see how 200 concurrent users are handled.

Siege testing 200 concurrent users without Evolved Caching

Siege testing 200 concurrent users with Evolved Caching

This time the server really is unable to handle the load achieving just 54.5% availability, but Evolved Caching continues to handle and serve all 200 requests maintaining 100% availability. Under this load all the server can handle normally is 10.39 transactions a second, but with Evolved Caching it can now handle more than 13 times the number of transactions at 138.89. The response time sees a similar improvement going from a clearly unacceptable 8.62 seconds to a still lightening fast 0.24 seconds.

That's pretty much as far as the server could go without caching, but we thought we would carry on the testing with Evolved Caching enabled and see just how far the performance scaled. Using a very standard Apache configuration we found Evolved Caching to just carry on giving with performance reaching an incredible 420 transactions a second, and a response time of just 0.49 seconds, whilst still maintaining 100% availability. That's over 40 times the number of transactions the server can otherwise handle. Your server hardware will do so much more with this extension installed.

I need consistent performance over time

Well that's exactly what Evolved Caching will give you, and here is the data to prove it! For this test we used BrowserMob which is a tool used to load test your site with real browsers over an extended period of time. We ran this test on a real, functioning store, with the natural peaks and troughs in performance you see under the load of users. The store is actually running on a low end VPS hosting spec, just 500MB of memory - not only does the extension perform incredibly, it might even put money back in your pocket by saving on hosting costs!

We ran four tests in total, two without Evolved Caching enabled, and two with. We configured BrowserMob to test the site for an hour each time, and load it with fifteen simultaneous browsers resulting in thousands of transactions over the duration of each test. The results speak for themselves:

First test with Evolved Caching disabled
Second test with Evolved Caching disabled

First test with Evolved Caching enabled
Second test with Evolved Caching enabled

It is plain to see not only the difference in performance Evolved Caching brings, but also the incredible consistency of results and clear levelling out of server performance. Compared with Siege, 15 simultaneous users is rather modest and of course we already know from those results that Evolved Caching scales to give bigger improvements under bigger loads, but even under this comparatively light load we see a huge increase in performance between the sets of results.

How does performance compare to Varnish?

In our benchmarks comparing the time taken for Evolved Caching and Varnish to generate full page HTML, we were getting results within 5ms of each other - so performance is near identical between the two.