After software delivery, the testers execute the automated test cases. This should run completely automatically and require minimal effort. Yet, you get the impression that work is still being done on the test execution days later. What exactly are the testers doing, and should it really take this long? How can this time be reduced?
The good news is that the costs are measurable and explainable. It's possible to reduce the causes of these costs or, ideally, eliminate them entirely. Below you'll find a customer example that demonstrates which measures have helped.
There are various ways to measure the stability of automated test cases. We present two common metrics and attempt to categorize your automation into different levels. Regular application of these metrics will allow you to measure whether the implemented measures improve the stability of your test automation.
Metric 1: False-Negative Rate
You run your automated system tests (preferably overnight) and check the next morning how many of them passed successfully. What percentage did you achieve? 60%, or perhaps 80%, or even 90%?
Why didn't all test cases pass successfully? You likely discovered some bugs or encountered changes to the software requested by the business team. This explains some of the failed test cases. The remaining number of failed test cases, divided by the total number of test cases, is what we call thefalse negative rate. Here are a few reasons why test cases might fail even though neither a bug nor a business change is the cause:
- Technical changes
- The technical ID of individual fields has changed.
- Test data
- Test data was not processed correctly.
- Quality of automated test cases
- Test cases were either incomplete or implemented poorly.
- There are dependencies between test cases and the execution order was incorrect.
- Changes were made to test case templates that affected more test cases than intended.
- A test case uses static waits, and the speed of the test environment is not identical for every execution.
- Interference from the test environment/infrastructure
- The test environment was either unreachable or only partially accessible during the test execution.
- The executing test agent failed or delivered incorrect results.
- Batch processes (for example, end-of-day processing) ran during the test phase.
We can reduce these disruptive factors or, in the best case, eliminate them completely.
How many of these test cases do you own proportionally? The Tricentis test report attempts a classification:
These numbers sound very high. We only consider an automated test set stable when it's in the top quartile. Are you there yet? Congratulations! If not, we'd be happy to help you get there.
Metric 2: Execution Rate
You execute the complete, automated system test set. Afterwards, the failed test cases are analyzed and bugs are logged. Once this process is complete, we stop the clock. How many working hours or days did you spend? And how many test cases do you have? The number of test cases divided by the manual effort required to execute them is what we call the execution rate. It also measures how much manual effort you need for follow-up after test execution.
You can use this metric to check for improvements over time. The execution rate depends on many factors and is only partially comparable between different projects. It can also be used to demonstrate the value of test automation.
What conclusions can be drawn from the execution rate?
A high execution rate tends to mean:
- Stable test automation (low false negative rate)
- Efficient testing team
- High software quality
- Test cases could exhibit deep test coverage
- The effort required for manual follow-up is minimal.
- The test environment is fairly stable.
A customer example
We implemented these two metrics for a client at the beginning of 2022. The client has approximately 1200 automated regression test cases and uses the banking software Avaloq with the automation tool Tosca. At the end of 2021, we made several changes, namely:
- Tosca upgrade to version 14.2
- Test data creation via database entry instead of the user interface (with our product: TAMI ). Additionally: Switch to the new test data system TDS (instead of TDM).
- Conversion of some Tosca Classic modules to TBox modules (using our module migration service).
Admittedly, that's a lot of changes at once. However, all the changes were necessary to reduce technical debt and take automation to the next level in the long term, for example, with faster test execution. We then conducted two test runs for the upcoming release after the software changes.
Test run 1:
False negative rate: 38%. Execution rate: 80 test cases per workday. Oops, that needs improvement! We've made the following changes:
- Test data
- The test data was not set up correctly. The issuing country of the identification was missing. This led to errors during test execution and has been corrected.
- Some test cases did not use synthetic test data, and the customer configuration has changed since the last execution. We have switched these test cases to synthetic test data.
- Not enough test data was generated. The number has been increased for the next test run.
- Quality of automated test cases
- The test cases needed to be better aligned with the newly created test data.
- Some Classic modules cannot be executed via distributed execution and must be migrated to TBox modules (e.g., the module ‹TC String Operations›)
- There were dependencies between test cases, and the execution order was incorrect. The dependencies were reduced or resolved.
- Infrastructure
- Test agents did not execute the test cases correctly because drive access was lost. This drive had to be remapped, which is now part of the test preparation.
- Not all test cases were executed via distributed execution, as was mistaken. The unexecuted test cases have been added to the relevant list.
- The test environment was temporarily unavailable or only partially available.
Test run 2:
False negative rate: 17%. Execution rate: 100 test cases per workday. Better, but there's still room for improvement.
- Test data
- Adjustments to test data creation. The test customer's account managers were not always added correctly. The creation process has been optimized.
- Quality of automated test cases
- Some Classic modules cannot be executed via distributed execution and must be migrated to TBox modules (this is a lengthy process).
- Due to the creation of new test cases, test case templates were modified, which unfortunately incorrectly changed the execution of existing test cases. This had to be corrected.
- Infrastructure
- The suitability engine did not run correctly during the test. We will check this beforehand next time.
- The test environment was temporarily unavailable or only partially available.
- The application's login screen was unavailable for a few seconds. We have implemented recovery scenarios so that another login attempt will be made after a short time before the test is aborted.
Our work here is not yet finished. We will continue to work on it and further stabilize the test cases. We aim to achieve a false negative rate of less than 5%.
And one more thing:
Quality assurance is not just about test automation and should be integrated into the entire project process. We would be happy to provide you with a test assessment to show you where your test organization stands.
Learn more about Test Assessment
After a three-hour workshop, we'll send you a customized report with suggestions for next steps. We recommend repeating this assessment every 3–6 months to measure progress and adjust the focus topics accordingly. And it gets even better: We offer the first test assessment free of charge for new clients!
