The European AI Act

Requirements for test data and testing

Since August 1, 2024, the EU AI Act – the world's first law governing artificial intelligence – has been in force at the international level. This law aims to build trust in AI systems, provide companies with clear rules for their development and use, and simultaneously protect society from potential risks. It establishes an important foundation for the safe use and further development of AI systems.

The law follows a risk-based approach and distinguishes four risk levels:

• Unacceptable risks: AI systems that threaten safety, rights, or livelihoods (e.g., social scoring) are prohibited. This includes technologies that could potentially be used in a discriminatory or harmful way.
• High risk: AI systems used in sensitive areas such as healthcare or law enforcement are subject to strict regulations. Examples include medical diagnostic tools or facial recognition systems in public spaces.
• Limited risk: Systems with moderate risk (e.g., chatbots) require transparency. Users must be informed that they are interacting with AI.
• Minimal or no risk: Systems like AI in video games have low or no requirements, as they do not affect sensitive areas or rights.

Who is covered by the AI ​​law?

Although the AI ​​Act is an EU law, it also affects Switzerland. This is particularly relevant for:

• Companies: For example, Swiss companies that offer or use AI services in the EU, such as software developers and technology providers.
• Service providers: Organizations that operate AI-based services, for example in the financial or healthcare sectors.
• Authorities: Institutions that use AI systems to increase efficiency or support decision-making.
• Traders and importers: Companies that source AI systems from the EU and distribute them in Switzerland.
• Developers of AI tools: This also includes those who adapt AI models to specific use cases. Swiss companies that offer AI services in the EU or use AI systems from the EU must comply with the regulations.

What does this mean for testing?

Naturally, we wondered what the EU AI Act means for software quality assurance and testing. We've compiled the most important points in this blog.

Especially for high-risk systems, there are clear regulations that directly impact testing and are now legally mandated. This means that specific tests must be performed to ensure that all legal requirements are met. These include validating functionalities, identifying and minimizing risks, and documenting all test results for conformity assessment.

Testing as an essential part of the development process

It must be demonstrably ensured that all requirements have been implemented appropriately and tested and verified within the framework of a risk management system.

This activity must be implemented as a continuous, iterative process throughout the entire lifecycle of the AI ​​system. Test strategies must explicitly contribute to ensuring that the intended purpose is fulfilled. Testing activities must be embedded throughout the entire development process.

Test data as part of the conformity assessment

Training, validation, and test data for AI models must meet clear quality and governance criteria. These include:

• Clear purposes for data collection, especially personal data
• Measures to avoid bias and data gaps, as well as the use of "fair data"
• Use of representative test data
• Strict protection measures for personal data

For AI systems that are not trained with data, these points apply exclusively to the test data.

The role of synthetic test data

Synthetic test and training data play a central role in protecting personal data and preventing bias. In particular, the AI ​​Act mandates the use of synthetic or anonymized data when dealing with personal data.

The AI ​​law therefore requires the use of synthetic or anonymized data, particularly in the case of personal data, and defines certain exceptions for the processing of real personal data.

AI systems often produce large amounts of synthetic data that are difficult to distinguish from real data. Such data must be clearly labeled.

Ensuring quality management

The implementation of a quality management system is a mandatory requirement for suppliers. Technical documentation is required, which should include, among other things, the following points:

• A detailed description of the test methods used
• Documentation of all test results and their interpretation
• Evidence for risk assessment and its treatment
• Measures to ensure data integrity and fairness
• Information on test environments and datasets used

These elements help to meet legal requirements and ensure transparency throughout the entire process.

Testing under real-world conditions

The AI ​​law uses the concept of real-world testing, which entails additional obligations. It is therefore important to recognize when such testing is taking place. Key characteristics include:

• The test simulates real usage scenarios or is conducted in the production environment
• The data used is representative of reality and the actual application of the AI ​​system

For high-risk AI systems, numerous regulations apply to testing activities under real-world conditions outside of AI real-world laboratories. The most important ones are summarized below:

• A plan exists for such testing activities
• The submitted plan was (tacitly) approved by the relevant Member State
• The tests are registered with the EU
• There is an EU branch or a legal representative in the EU
• Data transfer complies with the safeguards under Union law
• The tests are limited in time to their necessity
• Vulnerable people are adequately protected
• Personal data may only be processed with consent and will be deleted after the test
• The testing process is supervised by specialized personnel
• Predictions, recommendations, or decisions made by the AI ​​system can be reversed

Building AI testing skills

Would you like to learn more about testing AI systems? Our training and consulting services will help you to confidently implement the legal requirements.

‍

‍

‍