What exactly belongs in a regression set?

Is this functionality still relevant? Haven't we already covered this in another test?

We want our regression analysis to achieve the highest possible test coverage and be largely automated. Reality often looks quite different: thousands of manual test cases in the regression set, and you're certain that at least 25% of them are unnecessary – but which ones? Why have so many tests accumulated? Using a practical example, we'd like to illustrate which test cases belong in a regression analysis and which don't. Correct selection is a key element for a clean, effective regression set.

This practical example is also suitable for training specialists who have had little contact with testing.

Contents

1. Example specification
2. Derivation of the system tests
3. Creation of the regression set
4. Maintenance of the regression set
5. Summary

Example specification

A company has launched a digitization project. The goal is to create an app for customers that can perform the following tasks:

• The app displays the current address
• The app allows you to make address changes

The following details should also be noted:

• The app can be used by legal entities and natural persons
• The app may only be used by customers aged 16 and over. Downloading the app will be blocked for younger customers
• If an invalid new address is entered, an error message will appear
• Required fields for natural persons are: First name, last name, street, postal code, city, country
• Required fields for legal entities are: Company name, street, postal code, city, country
• After changing your address, you can choose whether you want to receive confirmation by email or letter

Derivation of the system tests

The first two test levels, unit tests and integration tests, are usually performed by developers or technical testers. These tests assess the functionality of individual modules and the interaction of interdependent components. System tests, on the other hand, verify compliance with the specifications while the entire system is available. It is crucial that system tests have high test coverage to identify software errors before deployment to production. Short system tests are advantageous for easier identification and localization of potential errors. The following system tests would be suitable for the specification mentioned above:

1. Natural person. User is under 16 years old (download blocked)
2. A natural person can display their current residential address
3. Natural person enters invalid address, e.g., the entered postal code does not exist (error message)
4. A natural person enters a valid address and selects confirmation via email
5. A natural person enters a valid address and selects confirmation by mail
6. Natural person. Verification of mandatory fields. Entry of an incomplete address and verification of error messages
7. A legal entity can display its current company headquarters
8. Legal entity enters invalid address, e.g., entered postal code does not exist (error message)
9. The legal entity enters a valid address and selects confirmation via email
10. The legal entity enters a valid address and selects confirmation by mail
11. Legal entity: Verification of required fields. Entry of an incomplete address and verification of error messages

Creation of the regression set

After the software goes live, the system tests are converted into regression tests. The number of test cases for regression is to be significantly reduced. The goal of regression is to identify undesirable side effects of software changes. It is not about retesting the entire specification/software. This also applies if test automation is planned, because even automated tests require regular maintenance and are therefore not "free" to execute. The regression tests focus on the essentials. Combining several system tests can also be advantageous to save execution time. The error density in regression tests is usually much lower than in system tests. We have selected the most important system tests and combined them where appropriate. This results in the following regression set:

1. A natural person (over 16 years old) can view their current address, then enter a valid, new address and select confirmation by mail
2. The legal entity displays the current address, then enters a valid, new address and selects confirmation via email
3. A legal entity enters an invalid address, e.g., the entered postal code does not exist (error message). The street field is then left blank (error message).

Maintenance of the regression set

What happens if the app is further developed in a future release? The regression set requires regular maintenance. We assume the specification will be extended to include the following point:

• It is possible to activate the address change only at a future date ("future address")

This results in several additional system tests, which we will not list here. Instead of increasing the regression set after the release, the existing test set can be adapted to also include the future address check:

1. A natural person (over 16 years old) displays their current address, then enters a valid, new future address and selects confirmation by mail.
2. The legal entity displays its current address, then enters a valid, new address (change effective immediately) and selects confirmation via email.
3. A legal entity enters an invalid address, e.g., the entered postal code does not exist (error message). The street field is then left blank (error message).

It is also possible that test cases can be deleted from the regression set after a software change if they are no longer relevant due to the software change (example: if address changes are only possible via app, there is no longer a need for tests to test this via website).

Summary

• There are several test levels. Only a small proportion of the system tests are used in the regression analysis.
• Even if testing is to be automated, the regression set should be kept small. Automated tests are not "free" and require maintenance.
• System tests can be combined for regression to save execution time.
• The regression set must be maintained after each release.

What does your regression set look like? If you would like a consultation, please feel free to us .

‍