The State of Quality 2022 by Katalon

In today’s rapidly changing world, organizations have to look both inward and outward for improvements in their software quality (QA) practices. They need to collect and analyze information and data generated from their practices to determine their weaknesses and strengths in delivering software quality at speed. At the same time, they must look for the current status and trends of their respective industries.

Based on a survey of 3,000+ respondents, this report captures a snapshot of the current state of software quality, presents insights, and predicts trends in QA techniques, practices, and tools. As our study surveyed software engineers, QA engineers, analysts, and managers, the findings reflect the opinions and voices of people who are directly involved in ensuring the quality of their software.

Findings

1. Although it is the most popular QA technique, automated system testing is not as widely used as people would expect.

Software teams today employ a combination of QA techniques during software development. Apart from manual testing, which is performed by virtually all software teams, automated system testing is the most widely used technique among all 11 techniques surveyed.

Still, just over a half (51%) of respondents applied automation for system and integration testing, which is rather low given the fact that it has become crucial for Agile and DevOps practices.

2. Monitoring and testing in production emerged as an answer for balancing between quality and speed.

To achieve quality at speed, teams need to find the right balance between ensuring the quality of software and reducing the time needed to perform QA activities. They cannot sacrifice valuable time to make error-free software.

This survey found that monitoring and testing in production is the third most popular QA technique which was applied by 38% of respondents teams.

This finding suggests that software teams embrace the possibility of errors in production and use the technique as a way to strike a balance between quality and speed.

3. Automated testing and code review/inspection are the two most effective QA techniques.

Teams tend to use a QA technique when they find that it is effective. This survey showed that the two most widely used QA techniques are also the most effective ones. They are test automation (for integration, system, and acceptance testing) and code review/inspection, which were recognized by 65% and 54% of respondents, respectively.

4. Frequent requirement change and the lack of time to ensure quality are the most cited challenges in delivering high-quality software.

Like our surveys in recent years, this survey found the same two most common challenges in achieving high-quality software: frequent requirement changes reported by 46% of respondents and the lack of time to ensure quality by 39%.

Frequent changes in requirements make the application of automation difficult as teams have to spend time revising their automation tests to reflect the changes. With a lack of time, teams tend to perform ad hoc manual testing instead of investing resources in more effective QA techniques such as test automation.

5. Professionals are satisfied with their software quality upon delivery, but they are not equally satisfied with their QA practices and tools.

When it comes to applying test automation, many respondents were unhappy with their automation tools and practices, so they had to resort to manual testing instead.

6. The popularity of free open-source automation tools continues to decline as commercial tools are more powerful.

Tools play a key role in the success of test automation practices. Since 2018, our surveys have tracked the popularity of common tools for test automation.

One trend is clear: Selenium is the most widely used tool, but its popularity has declined sharply over the last four years, going from 86% in 2018 to 54% in 2020, and 37% this year.

This survey also shows that tools and frameworks such as Katalon and Perfecto have become more powerful. Commercial tools such as these eliminate the need to build automation frameworks based on Selenium and Appium for test automation.

7. Test automation is increasingly applied to more testing activities.

Another trend found in this survey is that teams applied automation to more testing activities. While a majority of respondents applied automation to regression testing (53%), they also applied automation to analyzing test results (38%), generating test data (36%), and performing performance tests (34%).

It is possible that software teams have more mature and powerful automation tools to perform more testing activities automatically.

8. The most common challenges in applying automation tools continue to persist.

This year’s survey found that the most common challenges with automation tools remain the same as those reported in 2020.

They include:

These challenges are not necessarily essential as automation practices and tools have the potential to eliminate the need for advanced skills required for automation and to cope with changing requirements.

9. Most teams have achieved a high ROI with test automation.

Return on investment (ROI) is an important metric to measure the benefits of test automation. This survey asked software professionals to rate their ROI in terms of the cost/ time-saving of their investment in test automation.

10. AI for test automation is still in its infancy

Artificial intelligence (AI) offers great potential for test automation. It can reduce or even eliminate the need for human involvement in time-consuming activities such as test design and generation, data preparation, test execution, verification, and test maintenance.

Still, our survey found that less than 50% of respondents use some AI capabilities for their test automation. They mainly used AI for generating test cases and scripts, test data generation, defect detection, and test prioritization and selection. This finding suggests that organizations, especially automation tool vendors, continue to improve the adoption of AI for test automation.

Download the full report here!