How can AI be utilised in software testing?

Artificial intelligence will continue to play an increasingly important role in all areas of software development, including testing. In this blog series, we will explore the possibilities of artificial intelligence in the field of testing, starting with the basics and outlining possible use cases.

Artificial intelligence is becoming ubiquitous—whether you want it or not. As a user of digital services, I have been utilising AI for years, often without even realising it. With the advent of heavily marketed services like ChatGPT and Copilot, AI has become a prominent presence in my own and the general public’s consciousness, and I see no end to this phenomenon.

AI is playing an increasingly important role in professional use. AI is already a reality in software testing. AI features are being added to testing tools. As a tester, I can utilise general services like ChatGPT to enhance my work efficiency.

The first part of this series of articles on AI briefly reviews the main principles of AI and how it could be used specifically in software testing – already today.

Generative AI and machine learning

Artificial intelligence that is useful in software testing can be roughly divided into two types: generative AI and traditional machine learning.

Generative AI applications, such as ChatGPT, produce text, images, or other digital content based on user requests. Generative AI is based on an LLM, or large language model, which is a huge mass of data used to train AI.

Machine learning, on the other hand, aims to identify patterns in the data fed into it and use this information to arrive at a predetermined and limited outcome. Predictive AI and machine vision fall under this category. An example of this is the artificial intelligence that unlocks your phone, which has been trained to perform facial recognition as accurately as possible – but only for that purpose.

How do AIs work?

Simply put, a machine learning system improves when it is taught. “That was right, that was wrong—again! Again!” Just like the Teletubbies.

Broader AI, on the other hand, learns by finding meaning in vast amounts of data, and teaching it requires access to thatdata. Self-learning is referred to as deep learning.

When utilizing AI, the difference between these types of AIs relates to how much computing power is required to use them. As a limited machine learning model gradually becomes more efficient, it ultimately does not require a huge amount of processing power.

LLM-based AI, on the other hand, becomes a more powerful machine the more it consumes, but at the same time it learns to understand human communication better and better.

How can artificial intelligence be used in testing?

What role does AI play in software testing amid the flood of possibilities, applications, and services? What do I need to do to make my testing AI-driven?

AI features are increasingly being integrated into testing tools. However, there is no need to wait for new developments – even ChatGPT can already accomplish a great deal.

More important than the tools themselves is thinking about the benefits that AI can bring to your work. How can it make your work better, more efficient, faster, and easier?

Here are a few thoughts on the use cases for which AI can be adapted in testing.

Generating test data

Libraries that create more natural and diverse content and values have long been used in test automation. For example, many people are familiar with the faker library. It is often sufficient to use similar tools on limited test data. However, problems can arise if the data format is not text or if the data offers many variations, such as optional elements or fields. Instead of a single file, it may be necessary to generate an entire dataset or database. Artificial intelligence can offer a more effective solution to these situations.

Test generation

One of the most important steps in testing is identifying what needs to be tested. The most important thing is to test in a meaningful way, i.e., critical parts and parts where errors can be expected due to their complexity or development history. AI can act as an assistant that can suggest areas to test. It can also create test cases, or at least test case templates, if given a use case. Unit testing should not be forgotten either: if AI is given source code to look at, it can create almost ready-made unit tests. Coding tools offer AI add-ons, such as Copilot, which accelerates test writing by generating nearly ready-made unit tests for individual functions.

Test maintenance

The weakness of both manual and automated tests is that they do not always keep up with changes. This may be due to changes in specifications that have been forgotten to be included in test cases or documentation. Or a technical change has been made to the user interface or interface that was changed in unit tests, but not in system tests. The tester encounters this when the test fails. When the cause is identified as a change, there is a need to update the test cases that are affected by the change. The number of test cases to be updated can be large. In the worst case, the update effort can lead to most of the testers’ time being wasted on updating tests. With AI, updating is faster.

Log analysis

The tester’s job is to find errors. For the tester to write an effective bug report, they often need to identify the relevant section in the log file so that the error can be pinpointed in the code. This is why tools such as Splunk or Cloudwatch searches are used to try to find the relevant log. Artificial intelligence, especially an LLM-type of AI, can provide a better tool for this, as there is no need to guess the spelling of the search term.

Another important area in testing is identifying abnormal situations. Artificial intelligence can be taught to recognize unexpected situations from log files. This enables the detection of production problems and enhances test coverage.

Generating summaries

Testers must familiarize themselves with a wide variety of documentation. It’s part of the job. But sometimes there is a lot of documentation. And not just the system’s own documentation, but also regulations and definitions related to its operation. AI can conveniently summarise those documents. The summary can then be used to determine whether the document contains information that is essential for testing.

However, this is not the only use case for summaries. Testing itself produces documents such as test cases, results, and error reports. When reporting on the progress of testing, the aim is to summarise the situation in words and graphs. An essential part of test reporting that requires professional skill is assessing the situation behind the numbers. However, this part may only be a couple of sentences in the report, while the rest of the content either takes time to produce or is left to JIRA’s “ticket status” graphs, for example. AI can be used in reporting to create that “other content” as a summary of the documentation produced by the tests.

Avoid these pitfalls!

The use of AI also involves problems and even dangers. These include, for example, biases inherent in the training data. If there is a built-in bias in the source data, AI will inevitably repeat it.

Artificial intelligence can also be trained incorrectly, either intentionally or unintentionally. It is not uncommon for a text found online to contain incorrect or outdated information, which the AI will then repeat once trained. It cannot be ruled out that AI models have been deliberately taught to ‘lie’.

The use of AI also raises concerns about privacy. Can customer data be used to train artificial intelligence, and is this use compliant with the GDPR? What about the possibility of trade secrets being revealed if you ask questions in ChatGPT?

We will return to these questions in future episodes of the series. Thank you for your interest – tests be careful out there!