Using LLMs (ChatGPT) for test case automation
How Artificial Intelligence paves the way for automated JavaScript generation
Benefits for companies and insights from practice
In the dynamic world of software development, test automation is essential to ensure the quality and reliability of applications. Many DevOps teams rely on Behavior Driven Development (BDD) to precisely define acceptance criteria and test cases, ensuring that the software meets expectations. The advent of Large Language Models (LLM) opens up new possibilities in test automation.
At Infometis, we've set ourselves the goal of evaluating innovative use cases for artificial intelligence in software development through our AI learning group, in order to demonstrate its advantages to our customers. In this article, we show how LLMs (Learning Lifecycle Management) simplifies the process from user stories and requirements through BDD specifications to the automatic generation of JavaScript. You can also read about the benefits this technology offers companies and the insights we gained from implementing this use case in this article.
Addressing real-world problems
In our last blog post (Infometis AI Learning Group) from August 20, 2024, on the 5-Step AI Transformation Playbook, we illustrated the steps that need to be considered from ideation to the implementation of an AI use case. Based on the premise of not only understanding AI in theory, but also actively integrating it into everyday business, we at Infometis pursue a bottom-up approach. This means that the use cases come directly from our employees, based on concrete challenges from their daily work. This is also the case here: The presented use case was developed from a real problem posed by an Infometis employee.
The use case can be summarized in three steps:
Step 1: Acceptance criteria and test cases in BDD (Given-When-Then format) should be created from the requirement (user story / prose text)
Step 2: The test cases in Given-When-Then format are transformed into a Cucumber Feature File
Step 3: The JavaScript for test automation is created from the Cucumber Feature File
After gathering requirements and assessing data availability and quality, we evaluated various Large Language Models. From the initial proofs of concept, we were already able to draw positive results in the automatic generation of JavaScript for test case automation.
Infometis CustomGPT “From BDD to JavaScript”
To solve the problem, a custom GPT (From BDD to JavaScript) was configured in this proof of concept. Specifications regarding data sources, structure, language, and context were defined. Using these structured specifications, the LLM was able to generate a functional result.
Practical example: Searching for a training course on the Infometis website
Let's assume we have the following user story:
User Story (Prompt)
- As a user, I would like to use the Infometis search bar on the Infometis website to search for a training course.
- When the user searches for the search string "ChatGPT Prompt Engineering", they want to see the search results for that search string
Output CustomGPT
//
Thank you for your request! I will now create the requested test case in Given-When-Then format, convert it into a Cucumber Feature File, and finally generate the corresponding JavaScript code for test automation.
1. Test case in Given-When-Then format
Title: Search for a training course using the search string "ChatGPT Prompt Engineering"
Given: the user is on the Infometis website
When: the user enters the search string "ChatGPT Prompt Engineering" into the search bar and starts the search
Then: the user should see the search results for "ChatGPT Prompt Engineering".
2. Cucumber Feature File
3. JavaScript for test automation
Explanations about JavaScript:
//
Advantages of using LLMs to generate JavaScript
Using large language models to generate JavaScript for test automation offers numerous advantages that make the development process more efficient and effective. Key advantages include:
1. Automated conversion of user stories into BDD specifications: LLMs can convert user stories and requirements into clearly structured BDD specifications in the "Given-When-Then" format. This significantly simplifies the transition from requirements analysis to test implementation.
2. Automated test generation: After converting user stories into BDD specifications, LLMs can directly convert these specifications into JavaScript, significantly reducing the manual effort required to create tests.
3. Time saving: The automatic generation of BDD specifications and tests saves valuable development time that can be better used for other tasks, such as developing new features or improving the user experience.
4. Consistent code quality: The use of LLMs ensures that the generated code follows a uniform standard, which improves maintainability and readability.
5. Scalability:
- LLMs allow teams to quickly scale to a large number of tests without requiring additional manual effort.
- With an add-on, requirements can be passed directly from Jira to an LLM, making the workflow seamless and efficient.
Potential for improvement: The result, specifically the JavaScript, of the CustomGPT can be used for test automation, but may need to be optimized in one or two places.
The advantages of LLMs: The advantages of Large Language Models lie in their ability to analyze and output texts, language patterns, and code, thus offering enormous potential. To solve this problem, both the analytical capabilities (searching, extracting, classifying, and sorting) and the output capabilities (generating, summarizing, rewriting, and translating) of LLMs are applied.
Would you like to try it yourself?
We'll accompany you on your AI journey. Discover the right service for you on our website – we're happy to support you with your next project!
