Recently I posted a question on linkedin “Performance Specialists” group regarding soak(/endurance) test and whether it is required, if team wants to achieve daily deployment into production. I have to say it was an interesting discussion with different views. That discussion lead me to write this post about my own thought on the topic. Before we get to that, let’s first define what Endurance test is and its objectives.
What is Endurance testing?
Endurance or Soak testing is a type of load test that normally runs for extended period of time (few hours to days) while the system under test is subjected to a production like load(or anticipated load) to understand/validate its performance characteristics.
Why conduct Endurance test?
Endurance test is conducted to answer different kinds of questions such as:
- Does my application performance remain consistence or degrade over time? For example, your application might have a resource leak such as memory or connection leak which manifests slowly over time and impacts your system.
- Is there any interference that was not accounted for and could potentially impact system performance? Example application backups/VM backups/batch jobs/third party jobs running at different time of day that might not have been accounted for in other tests but do impact system performance.
- Are there any slow manifesting problems that have not been detected by other types of test? Other then the resource leaks, you could also detect problems which are due to sheer volume of data. Example of such an issue is full table scan on the database. As the data grows, the database queries start to slow down because they are doing full table scan. Similarly, running out disk space because too much data is being written to log file which in turn causes the application to crash.
Now let’s get back to the question of whether endurance testing is required or not, if you want to achieve daily deployment.
My personal view is that there is no simple answer to conducting endurance test and for how long to achieve daily deployments. Would you want to conduct an endurance test that lasts for 12 hours, if you are making a text change? Probably, NOT. What about if there is a rewrite of a function that makes call to database and third party api? Probably, YES.
In the end it will all come down to the level of risk that team/stakeholders are willing to take to deploy code into production, standby their decision and how quickly they can mitigate performance issues in production without impacting brand image, sales, user experience and so forth.
However, I don’t believe Endurance testing is dead. There is a place for it in the continuous deployment world. It just needs a little bit more thinking and planning (different approach may be), if you want to achieve daily deployments. You can run soak test overnight, analyze and share results in the morning with rest of the team or conduct it over the weekend. Another approach could be that the team reviews which changes they believe are of low risk and therefore best candidate for deployment during the week without requiring an endurance test. While high risk changes undergo endurance testing over the weekend before deploying into production, the following week.
Finally there needs to be right monitoring and alerting tools in place to identity performance related issues (be it in production or non prod). Any issue identified in production also need to be fed back to performance engineering team. This will help them improve their performance testing process.