.png)
Contents
- Changes in the Gartner Magic Quadrant for Software Test Automation (November 2019)
- Where is the market headed?
- Strengths/Weaknesses of the three leaders (SmartBear, Tricentis, Eggplant)
- Conclusion and Outlook
What does that mean?
This Magic Quadrant report focuses on a subcategory of software testing tools: software test automation. According to Gartner, these tools must be able to design, develop, maintain, manage, and execute functional tests. Typically, this should work not only on the web but also on mobile, desktop, in the cloud, and on embedded systems. This was thoroughly tested, particularly against the user interface and API.
…without functioning test automation, every DevOps toolchain breaks down – Serge Wolf
Continuous testing is the buzzword, and without functioning test automation, every DevOps toolchain breaks down. This is becoming increasingly challenging because the applications being developed today are so-called "multi-experience applications." This means that, in addition to a modern web interface, mobile channels (apps, wearables), desktops, as well as voice assistants or chatbots must be tested simultaneously. However, this forces providers of such automation solutions to continuously evolve. At the same time, it is becoming increasingly difficult for a niche solution, such as testim.io, which focuses on AI web automation, to ever establish itself in this crucial quadrant.
This has also led to the three leaders maintaining their position in their quadrant by expanding their portfolios through acquisitions (e.g., flood.io, Zephyr, NCC) to ensure they have a solution ready for all market changes.
The solutions and vendors in this quadrant are therefore very diverse. Gartner itself now recognizes this and concludes that this will likely be the last "Magic Quadrant for Software Test Automation." This is because the field has become too broad and too complex. I quote:
Gartner anticipates that this research will be the final Magic Quadrant for Software Test Automation, as it no longer provides the correct perspective for understanding and evaluating the vendor landscape. – Gartner
But what exactly has happened in the Magic Quadrant? What are the Leaders, the Challengers, and the Visionaries doing?
Eggplant
Eggplant software has been in the Leaders quadrant since last year and has particularly distinguished itself in AI testing. While Eggplant doesn't yet have a strong presence in Switzerland, the company and its software are worth a closer look. They also achieved a top-right ranking in Forrester's "Omnichannel Functional Test Automation Tools, Q3 2018."
.jpeg)
Eggplant is headquartered in London and was founded in 2008. According to the company, they have over 650 corporate clients.
Strengthen
Test Augmentation
Eggplant has a strong focus on using datasets and models to help the tester determine where and how much to test.
Multi-OS
Eggplant's solutions also run on MacOS and many other technologies.
Customer Survey
Eggplant consistently received very good ratings in independent customer surveys.
Weaken
API
Compared to the other leaders, Eggplant is clearly lagging behind.
Functional growth
Eggplant's solutions grow more slowly than those of their competitors, both in breadth and depth.
Pricing
Reference customers have expressed negative opinions regarding pricing and negotiation flexibility.
SmartBear
SmartBear is currently experiencing rapid growth and has become a collection of excellent individual solutions. Recently, two well-known solutions, Cucumber and Bitbar, were added. SmartBear now offers over 14 products.
According to the company, SmartBear has over 6.5 million users and over 22,000 customers. Wikipedia states that SmartBear has over 300 employees, but this number is likely outdated.
Strengthen
Width
For modern applications, SmartBear has a top solution ready for almost everything.
API
SmartBear is the clear market leader here. Their SoapUI solution is part of the standard arsenal, especially among software developers.
OpenSource
SmartBear has been supporting the open-source community in automated software testing since its inception. Products like Swagger and SoapUI are particularly noteworthy in this regard.
Weaken
Test Case Design
Topics such as test design or test case generation are almost entirely absent from the portfolio, but are currently important tools for the modern software tester.
Configuration/UX/Licenses
Customers complain about the cumbersome configuration and lack of user-friendliness. Furthermore, the license management is unclear.
Product duplicates
SmartBear recently acquired many companies. The overlap in functionality among their products is sometimes immense.
Tricentis
Tricentis is the only one of the three Leaders to have been in the Leaders quadrant for five years. In 2019 alone, Tricentis acquired three companies (IntelliCorp, LiveCompare, and TestProject).
Tricentis has over 1,600 customers and, particularly in Switzerland, certainly the largest footprint of the three Leaders. Gartner recommends taking a closer look at Tricentis if you are looking for a best-in-class, model-based testing approach that supports a wide range of technologies and applications.
Strengthen
Model-Based
Tricentis Tosca is the only solution that is particularly well-suited for business users.
Partnerships
Tricentis has successfully established strong partnerships with renowned global and relevant local expert firms*. This ecosystem helps the market build effective solutions with Tricentis.
*Author's note: Gartner was probably thinking specifically of infometis .
Customer Experience
Customers rate Tosca as very good in modern, legacy, and packaged applications
Weaken
Product integration
The numerous acquisitions have hampered integration into the portfolio. This means that solutions like qTest take longer to integrate seamlessly with other Tricentis solutions.
Software developer
Developers often ignore Tricentis Tosca because the solution does not integrate well into the existing development suite or because Tosca is simply not technical/hip enough for this group of people.
Repository
A few reference customers have expressed negative opinions regarding the maintainability of large repositories.
Conclusion
The software test automation market is currently worth $2.8 billion and is growing at approximately 14.3% per year. Many customers now use multiple solutions. This is simply because the range of technologies used is too broad for a single tool. A single-vendor strategy seems to be outdated.
The entire market will shift significantly towards SaaS and services in the future. Whoever leads the way here, especially with AI and ML, will win the race.
Automatte Now!
If you would like to know more about this topic or need specific support, please contact us today!
