Idea Science’s Head of Testing, Dave Melgaard, discusses the current state of artificial intelligence in test automation.
The allure of AI-driven test automation to automatically update test scripts as systems evolve is compelling. This is especially true for organisations who have experienced the ongoing burden of maintaining test automation scripts. Many leading test tool vendors claim to have AI-driven test automation solutions to help address this issue, but it’s important to know what’s real and what’s not.
In mobile and web-based testing, the capability now exists for AI-driven test automation to learn an application and dynamically adapt the automated tests as the application changes. This is achieved via machine learning (ML) and intelligent object detection, to recognise UI elements and identify controls based on visual clues. For example, control type (input, button, dropdown, table, etc.), label name, position, or the colour of a control on screen. This approach leverages similar technology to Tesla’s self-driving cars, which can adapt to changing road conditions.
Intelligent Object Detection
The need for speed
Until recently, UI test automation tools were only capable of processing approximately 1.8 frames per second – much less than the 24 frames per second that humans can process. However, using intelligent object recognition, tools such as Tricentis Tosca Vision AI can process up to 40 frames per second. This is of particular benefit to organisations where application delivery speed is a key driver, and also in CI/CD environments where testing needs to be accelerated even further. Automated tests can also be built based on wireframes, or even hand drawn mock-ups, to enable ‘extreme shift-left testing’.
Many newer test automation tools now also use natural language processing. This provides additional efficiency and ease of use benefits, such as no-code/low-code automation, model-based testing and the use of plain English statements to create scripts. Additionally, conducting AI-driven risk-based testing further reduces human effort and increases speed.
Current limitations
Whilst recent improvements in AI-driven test automation look promising, there’s still plenty of hype, which requires healthy scepticism. According to Tricentis, Vision AI has been ‘trained’ on 9 million controls and over 12 million examples, but that doesn’t mean the AI won’t make mistakes – especially when faced with scenarios that it’s never encountered before. The vendor suggests that the product learns from its mistakes as a human would do, but quantifying exactly how that learning impacts the quality of software delivery is still unknown.
AI-built test scripts can also create errors, causing false positives and/or interrupt test execution. The combination of false positives, script errors, and bloated test suites are still potential issues that need to be addressed.
API testing with AI-driven test automation is also currently problematic. While the AI can analyse web service calls and define tests to verify an API, it still struggles with business logic and defining flows for complex, end-to-end scenarios.
Where to from here?
The 2020-21 World Quality Report shows AI adoption is steadily increasing, with 93% of respondents claiming it’s the strongest growth area of their test activities. This is mirrored by leading software test tool vendors, who are investing heavily in AI-driven test automation to meet this demand.
The COVID pandemic has also heightened interest in AI/ML-based digital innovation as the need for organisations to increase efficiency and innovate grows, along with the increased risk and complexity of software development. Test and development teams are expected to maintain quality whilst their budgets are under pressure, so it’s understandable that people are excited about this new technology and its potential.
With system complexity increasing and release cycle times decreasing, it’s clear that AI-driven test automation will play a key role in helping organisations maintain quality at speed.
The usage of AI-driven test automation is likely to increase as the technology evolves and its business benefits become clearer. Depending on your particular use cases, this technology can increase speed and efficiency, however, further refinement is still required for these products to reach their full potential.
It’s also important to note that AI-driven test automation does not make your test automation team redundant, but frees them from having to automate basic tasks and allows them to focus on more complex end-to-end business processes and backend validation. It does however, raise the interesting topic of Using AI to test AI. We’ll cover this in a future Idea Science article.
Want to optimise your investment in test automation? Idea Science can make it happen!
