When does test automation become worthwhile?
Factors for evaluating the cost-effectiveness of test automation
Although many projects are repetitive or similar, the content is usually different – not least due to the technologies or products used.
If you Google this question and try to find an answer online, you'll often find the statement that it has to be worthwhile. In other words, the investment in test automation should show a positive return on investment. This calculation can take into account factors such as total labor costs, the number of test stages (e.g., DEV, SIT, UAT), the number of affected applications or technologies used, the number of redundant test cases, the time spent on manual test execution, the number of releases per year, and many others. Calculating these factors can be very complex and pose a significant challenge. Some figures are difficult to ascertain, and obtaining the necessary information—for certain factors—to make an accurate calculation can be time-consuming or, in the worst case, prove to be a "mission impossible."
Too many key performance indicators (KPIs) increase complexity
How can one specifically determine whether test automation is worthwhile?
First, let's look at the logical factors:
Generally, test automation is worthwhile for tests that are performed frequently. Having to repeat these tests as a tester can become tedious, and attention spans can decrease dramatically over time. This increases the rate of careless errors, which in turn significantly reduces test quality and increases the risk of overlooking bugs. An automation robot, on the other hand, is not affected by fatigue and has no problem repeating a test countless times. The quality standard remains consistent even after a million executions.
Time is a crucial factor in testing. Manual testers (humans) perform tests during normal office hours, resulting in "test interruptions" during the testing phase. Robots, on the other hand, don't need breaks, no end of the workday, no social interaction, no weekends, no coffee breaks, and certainly no smoke breaks (assuming the robot is a diesel model). Of course, 24/7 testing would also be possible with manual testers. This would "only" require testers distributed globally to cover every time zone. The entire project then "only" needs to be coordinated by someone, and the 24-hour dilemma would be solved. It's worth noting, to name just a few examples, that language barriers, different cultures and work styles, as well as working hours, must be taken into account during this coordination. Another challenge can be the global distribution and operation of the software, as well as establishing a uniform global standard.
A robot doesn't get bored of always performing the same tests and doesn't tire
An important factor that must also be considered in our question is the cost of a manual tester. In other words, what does a manual tester cost me, and how many testers do I need to complete my test set within the given timeframe? Of course, one could break down these costs further and, for example, analyze the cost-benefit ratio. However, that would make illustrating our question considerably more complex.
Conversely, what would it cost me to automate this test set? This would require including licensing costs, the costs of an automation specialist or expert, and any maintenance costs for system adjustments.
Therefore, we can conclude that the initial costs of test automation are higher than those of a manual setup. However, these costs decrease over time, as maintenance will be less demanding than manual test execution. The development of an automation system should be carefully planned and strategically designed to address future challenges. In other words, if the automation methodology is well-conceived and future-proof, future maintenance will be simpler, faster, and more cost-effective.
The initial costs for a test automation setup are higher, but decrease rapidly over time
Together with our partner, pom+ Consulting AG, we had the opportunity to directly investigate the above question at a large Swiss financial services provider and were able to deliver a well-founded analysis.
By breaking down the complexity, we were able to find a comprehensive answer. Our approach and methodology were straightforward. As a first step, we gathered the relevant information (including the technologies of the system to be tested, existing test cases, test environments, current testing procedures, and annual releases, among other things) and, together with the client, identified and defined the evaluation criteria. This definition provided a foundation for conducting an independent analysis. As the next logical step, we examined the defined test set to determine whether implementing test automation was technically worthwhile. Finally, we considered the economic perspective to assess the financing implications.
Use a simple calculation tool to help you and get an overview
To get a first impression of whether test automation is worthwhile for your company, you can download the Infometis Automation Calculator 2.0 and play around with the numbers in Excel.
Sounds exciting?
Of course, our team is happy to provide you with a neutral evaluation.
