Review of TestCraft, An AI-Powered Test Generation Tool – 30 Days of AI in Testing (Day 6)

The 30 Day of AI in Testing is a challenge setup by the Ministry of Testing. My hope is that by taking part in the challenge, and completing some of the tasks, I can improve my understanding of AI and improve my software testing practices by learning more about how AI can be applied to testing.

Here is a link to the Ministry of Testing event page where you can find out more about the event:
30 Days of AI in Testing event page.

In my previous blog post, I completed days 1 – 3 of the challenge by exploring what I actually wanted to achieve from the challenge, reading some articles and blog posts that can provide me some basic understanding of AI in Testing, and identified some tools that could be used to improve various software testing activities.

Day 6 – Identifying Tools To Support A Testing Need

For Day 6 of the challenge, we were asked to identify a testing need and then analyse and research testing tools that can support that need. For Day 3 of the challenge, I identified some testing needs and some tools which provided a good starting point.

The testing need I decided to focus on was Test Case Generation however I wanted to focus on test idea generation rather than automated test case generation where the code was generated for you.

TestCraft

I have been looking at other AI-powered tools, however I deciedd to start with TestCraft because it was freely available as a Chrome Extension (it is also available on other browsers including Edge, Firefox and Safari). It could be added to the browser in less than a minute. No signing up for an account or downloading was required.

To use TestCraft, you simply pick an element and then ask it to generate test ideas. You’ll then be given a list of test ideas for that element that you’ve picked. You can then select specific tests and generate automated test code. You can also select an element and opt to check for accessibility.

Pros:

• You can focus test ideas on specific elements on a page, rather than entire pages. It means the test ideas are relevant to just the specific feature under test, and you don’t have to delete loads of tests that are not relevant.
  • This is particularly useful for accessibility testing, as most accessibilty tools tend to focus on the entire page which can provide a very cluttered report.
• The test ideas and automation code can be easily copied over to use in other documents, and can be edited.
• Automation code is not run. It still needs to be copied, and then edited so it matches the framework. This is the best way to deal with auto-generated code, rather than just blindly following it and assuming its correct.
• Accessiblity checking is a really useful additional feature. The accessibilty report is simple and easy to read, with a clear guidance of the conformance levels and provide additional test ideas.

Cons:

• The ideas generated are very limited as you can only select individual elements or forms. If elements are designed to work with other elements, you can’t generate ideas based on that interaction.
• To generate test ideas, the element needs to be available on the page. You can’t pre-plan for testing before the developers have started working on the feature.
• Automated test code can only be written in javascript or typescript.
• The automated test generator does not include Selenium as an option. It is limited to Cypress and Playwright.
• The automated test code appears to be limited to UI tests, and doesn’t create API tests.
• Test Craft is not very user friendly when doing multiple actions on a single element. For example, if you wanted to generate test ideas and check accessibility, you will need to reselect the element each time. It would be useful if it was possible to retain the selected element while testing.

Final Thoughts

For this days challenge, we were supposed to research 3 or more tools. This is actually not an easy task as most AI-powered tools were not open source and required payment. Tools which are not open-source are usually the best and come with customer support. However, it is not always easy to get approval from our workplace to download them or even pay for them (although most offer a free trial). I was unable (and not willing) to sign up for a free trial for all the companies that claim to offer useful AI-Powered tools. This doesn’t mean I’ll never try them out, but will need to do so at a time when convenient for work (and when I’ve been able to get approval from my work to try them out).

TestCraft was free and available as an extension. A lot of companies still restrict what extensions can be added to web browsers. However, the fact it doesn’t have to be downloaded and is free will make getting approval to use it a lot easier.

That being said, the fact that TestCraft is a free web extension does show. The user experience could be improved by allowing elements to be retained so they can be used for multiple actions. However, the ability to focus on an individual element on the webpage is a really useful feature for both test idea generation and accessibilty testing. The ability to focus on these tools, especially for pages that contain multiple features like dashboards, provides a much more focused test report. If TestCraft was able to include user interactions in its analysis, it would be a really simple yet powerful testing tool.

Useful Resources

TestCraft
A tool that generates test ideas which generates test suggestions, including ones that focus on accessibility testing.

Day 6 of the 30 Days of Automation in Testing Challenge
Check out how other testers completed the challenge.