Reviewing Capgemini’s “World Quality Report 2024-25”

It’s that time of year again, with the publication of the 2024-25 World Quality Report!

I’ve reviewed this annual epic from Capgemini for the last four years and will do so again here, albeit for the last time (see my reviews of the 2018/19, 2020/21, 2022/23 and 2023/24 reports in previous blog posts).

This 16th edition is titled “New futures in focus”, only slightly different from last year’s “The future up close”.

I’ve taken a similar approach to reviewing this year’s effort, comparing and contrasting it with the previous reports where appropriate. This review is again lengthy, but you’ll still save plenty of time by reading my summary compared to devouring the 106 pages of the report itself!

TL;DR

It was a bad omen when the first thing I noticed on opening the PDF report in Chrome was a mistake in the report’s title meta-data – “Brochure Potrait” appeared in the document header and in the name of the browser tab. Moving right along…

The 16th edition of the “World Quality Report” is the fifth consecutive report I’ve reviewed in detail on this blog. It’s a hefty report, yet consistently fails to produce the genuine insights I’d expect following such a mammoth undertaking (including about 900 hours of interviews with senior representatives at large organizations). In some ways, I’d argue it’s a report “by CTOs, for CTOs”.

The previous year’s report made bold claims about AI and these continue in the current report. While the authors conclude that uptake of Gen AI is huge and revolutionary, I’m not convinced that the data supports this view and neither does my experience in the industry.

“Quality Engineering” is not clearly differentiated from testing again and many other terms are used without clear definition (e.g. Green IT) so there’s significant potential for confusion, both in interpreting the results and for respondents answering the questions.

I still think it would be valuable to include some questions around human testing to help us to understand what’s going in these large organizations, an observation I’ve made in all of my previous reviews of these reports.

Once again, the focus areas of the report changed almost completely, from eight last year to six (largely different) areas this year, making cross-report comparisons difficult or impossible. The sample set of organizations appears to be the same as last year (maybe in the interests of year-on-year comparison), so I really don’t understand why the report doesn’t stick to a standard set of focus areas year on year.

There continue to be many problems with this report. The lack of responses from smaller organizations mean that the results remain heavily skewed to very large corporate environments. There are numerous errors that really should have been picked up in proofing and the report is poorly copyedited, resulting in a report that feels like a bunch of disparate sections stuck together with no consistent voice. It’s good to see that the data visualizations this year are much simpler and more consistent than last year, though, making the results much easier to interpret.

While these reports blow with the wind of trends, I’d recommend that you keep an eye on trends in testing but don’t get prematurely attached to them. Focusing on building excellent foundations in the craft of testing will likely be time better spent, enabling you to navigate the winds of change throughout your career. Big name industry reports like this one carry substantial weight – regardless of their content – so stay mindful of the hype and adopt a critical thinking mindset when it comes to the conclusions made in reports like this.

About the survey (p96-101)

This report maintains its trend of becoming longer every year, running to 106 pages – up from 96 pages last year and “just” 80 pages the year before that.

I find it useful to look at the “About the survey” section of the report first to understand where the data came from to build it and support its recommendations and conclusions.

Notably new in the study design this year is that “…advanced AI-driven tools were integrated into the research process to enhance data quality”, so this needs to be kept in mind when looking at the findings of the report, I think.

The survey size was 1775, up slightly from the previous report (with 1750). Given the very similar survey size, it’s surprising the resulting report is longer.

The organizations taking part were again all of over 1000 employees, with the largest number (34% of responses) coming from organizations of over 10,000 employees. The response breakdown by organizational size was essentially the same as that of the previous four reports, strongly suggesting that it’s the same organizations contributing every time. I’d love to see the responses from those in smaller organizations where the technological and organizational context is likely to be very different.

While responses came from 33 countries (up from 32 in the previous report), they were heavily skewed to North America and Western Europe, with the US alone contributing 16% and then France with 8%. Industry sector spread was similar to past reports, with “Hi-Tech” (19%), “Financial Services” (15%) and “Public Sector/Government” (11%) topping the list (and all exactly the same percentages as the previous report, reinforcing my hypothesis that the same organizations are reporting each time).

The types of people who provided survey responses this year was also very similar to previous reports, with CIOs at the top (24% again), followed by QA Testing Managers and IT Directors. These three roles comprised over half (59%) of all responses – again exactly the same breakdown as the previous report.

Introduction (p4-5)

The introduction sets the scene for a big focus on AI (or “Gen AI”) in the report, so no surprises here. This finding troubles me, given the large organization focus of this report’s underlying data:

The other striking result is that there is more acknowledgment of the importance of quality (or rather the risk and impact of insufficient quality) – but organizations still need to work to give true strategic attention to the topic of quality.

Executive summary (p6-7)

The high-level summary of the report also makes a big deal about AI, but still manages to make some noteworthy observations.

On the model for where QE (“Quality Engineering”, a term that’s used throughout, never defined and sometimes used interchangeably with testing) fits into teams:

First, the integration of quality engineers within agile teams has become a standard practice, with 40% of organizations now embedding these experts directly into their agile processes. Second, we are witnessing a rise in organizations that not only integrate quality engineers into agile teams but also maintain dedicated Quality Engineering roles operating independently to ensure comprehensive coverage and oversight.

There’s a refreshing honesty around the actual use cases for AI:

The debate on which Quality Engineering & Testing activities will benefit most from Gen AI remains unresolved. This year’s survey highlights a growing focus on leveraging Gen AI for test reporting and data generation over test case creation.

While I noted in my review of last year’s report that the expectations of “QE experts” were getting very high (specifically including skills in coding, BDD and TDD alongside their idea of the typical QE skillset), this year they ramp it up even more:

Over the past decade, the shift to Agile methodologies, cloud computing, and smarter technologies has transformed quality engineers into SDETs (Software Development Engineers in Test) and, further, into full-stack test engineers. The skill set requirements for quality engineers have now expanded even further, encompassing data proficiency, AI expertise, Gen AI capabilities, and product engineering skills. However, this evolution does not diminish the fundamental need for risk-based test strategies, human collaboration, and deep business expertise—elements that remain crucial for ensuring comprehensive and effective Quality Engineering.

This unicorn “full-stack test engineer” nonsense really needs to stop. In my recent experience, it’s hard enough to find good testers without all these additional expectations on individuals. Many of the skills they’re throwing onto the “full-stack” are genuine specialties in their own right and we should treat them as such.

Key recommendations (p8-9)

This year’s recommendations are across six areas (down from eight last year and it’s not obvious which ones are the same and which are different, though “Intelligent Products Validation” appears similar to the previous “Intelligent product testing” and “Quality Engineering in Sustainability” probably relates to the previous “Quality & Sustainability”), as follows:

• Quality Engineering in Agile
• Quality Engineering Automation
• Quality Engineering and AI
• Intelligent Products Validation
• Quality Engineering in Sustainability
• Data Quality

Changing the areas and categories every year is very unhelpful and makes cross-report comparison too difficult. If I was being cynical, I would argue that this could be a deliberate move to make it hard or impossible to see how well their predictions pan out over time. There are three to five recommendations made in each area, so let’s dig into the details, area by area.

Current trends in Quality Engineering & Testing (p10-61)

Half of the report is focused on current trends, broken down into the six areas detailed in the previous section. As usual, the most interesting content is to be found in this part of the report. I’ve broken down my analysis into the same sections as the report. Sitting comfortably?

Quality Engineering in Agile

The introductory spiel for this section has an unusual tone and I particularly dislike the “Great reset” language (triggering inevitable World Economic Forum flashbacks to the atrocities of the pandemic response for me). This “reset” mantra appears again later in this section of the report.

The first set of remarkable data is around the “top 5 skills for your Quality Engineering associates” with an astonishing 70% saying “Quality Engineering skills”!:

There is a data error in this first chart: note that “AI/ML and Gen AI skills” are repeated, with one showing 66% and the other 57%. Putting two and two together from the commentary on this data, it seems that the 57% stat is for “coding skills”.

The next set of data relates to organizational structure:

I find the options on offer here quite difficult to differentiate and I can imagine respondents having difficulty in choosing the right options for their organization’s context. On this, the authors say:

A notable change is the decrease in the use of traditional Testing Centers of Excellence (TCoE), with only 27% of respondents reporting their continued use. This marks a substantial drop from the 70% who relied on TCoEs last year. While the results from last year’s survey seem high, likely due to conflicting interpretations of “TCoE”, the decrease this year is clear and consistent with other survey responses. Concurrently, 40% of respondents now have quality engineers embedded within Agile teams, highlighting a trend towards embedding Quality Engineering into Agile workflows.

The 70% claiming to use TCoEs in the last report always looked wrong (as I pointed out in my review last time) so it’s no surprise that a very different response was found this time. This also shows some confirmation bias on the part of the authors and they are acknowledging that their poor (or, more accurately, complete lack of) definition of the terms they use is likely impacting on the validity of the results.

The supporting commentary for figures 3 and 6 appear to be the wrong way around so it makes for a confusing read! Moving on to looking at challenges for QE adoption:

As development skills have become less critical and the focus has intensified on Gen AI and core Quality Engineering competencies, it appears that the broader value of Quality Engineering is not being fully recognized. The core problem may not lie in the alignment with development teams, but rather in demonstrating tangible value. Despite an increase in the use of advanced technologies like Gen AI and expanded automation coverage, the perceived value of Quality Engineering remains underwhelming.

This is quite a remarkable observation and is a sad indictment of trend following, if that is indeed what these large organizations are trying to do. The WQR has been championing the move to QE – and away from what I’d call testing – for many years, but this change of focus appears to be failing to achieve good outcomes for anyone.

Turning to their recommendations in this area (remember, it’s “Quality Engineering in Agile”), these two stood out to me:

Integrate quality engineers directly into product teams to ensure their work
is closely connected with product development and outcomes.

Maintain the independence of testing. As systems continue to increase in
complexity with multiple technologies and hosting locations, the benefit of
an independent testing team will pay dividends.

These recommendations seem to contradict each other, unless “quality engineers” are not being thought of as providing testing services. This murky distinction between QE and testing in this report (and commonly in the broader industry) is leading to a lot of confusion, pointless rebranding and problems for those of us focused on testing as a craft in its own right.

Quality Engineering Automation

This section of the report leads with AI, of course, asking about the extent to which organizations are using it to enhance the “maturity of test automation”:

My read of this data is that, at best, 29% of the respondents are really doing anything meaningful with Gen AI today. The authors choose to put a more positive spin on uptake:

In 2023, 69% of organizations were experimenting with innovative automation solutions like Low Code/No Code or AI-based automation frameworks. Fast forward to 2024, and the landscape has shifted by leaps and bounds. New futures are in focus – as 29% of organizations have fully integrated Gen AI into their test automation processes, while 42% are actively exploring its potential.

Further to this, when asked about the benefits of Gen AI in enhancing test automation, there were no surprises – “Faster automation” came out top (72%) and “Reduce testing effort/resources” was close behind at 62%.

The following claim doesn’t appear to be supported by any of the data in this report:

The survey results reveal that the global average level of test automation has now gradually increased to 44%

It’s hard to know what this really means – I can imagine different organizations measuring their “level of test automation” quite differently (especially as this is a difficult thing to measure meaningfully), so this average level is probably not indicative of anything – 44% might be great, might be terrible or might mean nothing at all.

The data around business outcomes achieved through test automation caught my eye:

So “Over half of the respondents highlighted that automation reduces manual effort” while the most popular response suggests that there is “improved testing coverage” which increases “confidence in IT”. I find this interesting as those business stakeholders gaining confidence from seeing increases in reported test coverage likely have no idea what’s being measured or the quality of the automation that’s been built.

Turning inevitably to “cost benefits of automation”:

It’s pleasing to see that a large proportion of respondents don’t use cost benefits as the primary driver for test automation, but the authors still claim “One of the key benefits of automation is reducing operating costs”. There appears to be another error here, I assume the the last bar should be labelled “Decreased operational costs due to additional tooling” (since increasing these costs wouldn’t generally be seen as beneficial).

Looking at the “talent requirements” around automation next:

I think it’s a big ask that “31% of respondents identified the need for full-stack engineers – quality professionals with additional expertise across the technology stack, including infrastructure, cloud, performance, resiliency, and reliability.” It’s good to see such a low percentage of respondents believing “developers can do all forms of testing so separate testing is not required”, but I struggle to believe that “Manual testing is still prevalent due to specific application architectures” in only 10% of organizations (and I don’t see why the latter part of that response was included or relevant). If we are to believe the previous claim that the “global average level of test automation” is only 44%, what testing activities are covering the rest if only 10% is “manual testing”?

The recommendations are not very exciting, though the emphasis on increasing the use of AI stuck out to me:

Harness the potential of Gen AI to enhance and accelerate test automation. Gen AI goes far beyond the generation of automated test scripts and helps with the realization of self-adaptive test automation system, driving efficiency and effectiveness.

I didn’t see any data in this section of the report to support the efficiency or effectiveness claims made in this recommendation.

Quality Engineering and AI

Given the authors’ overwhelming focus on AI, I was surprised to see this being one of the shorter sections of the report. The opening gambit makes a bold claim:

The results from this year’s survey indicate what we believe is the new future – Gen AI-augmented Quality Engineering. We found that 68% of respondents have moved beyond the experimentation phase and have adopted Gen AI platforms to improve their overall IT efficiency and accelerate their speed to market.

The 68% stat is a little misleading, as it’s based on these results:

The authors have combined the first two responses to produce their 68% stat, but half of these respondents are not actively using Gen AI solutions yet. The next set of data is quite interesting, looking at the testing-related use cases for Gen AI:

The authors note that their client experiences don’t match this data, so they continue to recommend focusing on the bottom two use cases as “there are greater gains to be made in those areas”. I’m encouraged by this statement, though:

Gen AI isn’t about replacing the human touch or magically improving testing quality on its own. Instead, it’s a game-changer for boosting the productivity of quality engineers.

This section closes with a big call to action:

Although the sheer volume of data may feel overwhelming, one thing is clear: Gen AI will revolutionize Quality Engineering. Whether you jump in headfirst or just dip your toes in, you need to start your adoption now!

The jury is still out on the use of Gen AI for most of the organizations contributing to this report, so this call to action seems too strong – and makes me wonder whether the authors are hallucinating just like the Gen AIs they’re discussing.

Intelligent Products Validation

The first two data sets in this section of the report are closely related, essentially asking about the importance of different types of testing for “intelligent products” (a term that isn’t clearly defined in the report):

These results seem contradictory to me. For example, looking at “Security”, 60% rate the security “test phase” as being very important (and the highest of all the phases), yet only 23% said that security was the most important aspect of validating an intelligent product (a worrying stat in itself!). These contradictory results don’t seem to worry the authors, though, who conclude:

When it comes to testing intelligent products, the emphasis on different test phases directly mirrors what respondents consider crucial for validation

I didn’t spot much else of interest in this section, with the recommendations unsurprisingly focusing on increasing the use of AI.

Quality Engineering in Sustainability

Turning to sustainability and “Green IT” (another term used in the report that is not clearly defined), the opening data relates to the prioritization of sustainability:

The authors conclude that “… a whopping total of 98% of organizations acknowledge that sustainability is extremely crucial to them!” while, in reality, the first three options are worded in such a way that I’d fully expect every organization to pick one of them (especially given the senior folks being interviewed). There’s another data error in this chart too, with the response percentages adding up to 102%.

The next set of data looks at focus areas to validate to drive Green IT:

I don’t understand how some of these choices are related to sustainability or Green IT, e.g. “the ability of IT systems, devices or software to work together for seamless functionality” doesn’t seem to me to have anything to do with efficient use of resources, sustainability, etc.

I find the data in the next chart impossible to believe:

The claim that almost half (43%) of the surveyed organizations are monitoring the environmental impact of every type of testing beggars belief. I have no idea how you would even go about doing that in any meaningful way. This data seems to feed into the recommendation to “Practice sustainable testing”, whatever that means.

Data Quality

There are few revelations in the section of the report relating the quality of test data. When it comes to provisioning test data, it appears more organizations are turning to AI (unsurprisingly):

There is a data/proofing error in the commentary on this data, saying “More organizations are turning to AI-generated test data (49%)”.

Looking at the issue of bias in test data next:

Again, AI is being seen as silver bullet here but kudos to the authors for acknowledging the potential issues with this (especially given the prevalence of using AI to also generate the data!):

Almost a third of organizations rely on AI to check data quality and remove biases (34%), but this approach often lacks transparency and context, which can unintentionally reinforce existing biases.

In most ways, this section of the report feels like the same old, same old with a sprinkling of AI while not suggesting any genuine improvements in the area of test data management, an observation also made by the authors in conclusion:

For over 15 years, we have been asking questions about data and its importance in the World Quality Report. Each year, organizations talk about focusing more on this, yet the same perceptions persist about the quality and importance of data. Despite the critical role data plays in AI and organizational success, many organizations still do not give it the focus it deserves.

Sector analysis (p62-95)

The sector analysis has generally not been as interesting as the trends section in previous reports and the pattern continues this year. The authors identify the same eight sectors as the previous year’s report (albeit presented in a different order for some reason), viz.

• Automotive
• Manufacturing
• Consumer products, retail and distribution
• Healthcare and life sciences
• Public sector
• Financial services
• Telco, media and technology
• Energy, utilities, natural resources and chemicals

A few things caught my attention in the sector analysis section:

• In the Retail sector, 31% “Believe that an environmental impact is monitored for each testing activity.”
• In the Public sector, 34% “Prefer to use developers to perform automated testing rather than dedicated SDETs” and this is the first time I’ve seen the conflation of AI and “shift-left”, I think: “The AI revolution, particularly the shift-left approach, is driving the need for faster, cheaper, and more predictable solutions.”
• In the Financial Services sector, “Approximately 50% of financial institutions are now looking to reduce their dependency on IT services from India and the US, instead opting to relocate resources to Latin America (LATAM)…. For example, Mexico has become the fourth largest IT market, with a growing pool of IT professionals and STEM graduates.” This is an interesting development and makes sense from a timezone perspective for US organizations, so it’ll be fascinating to see how the established outsourcing players respond to this geographical shift. A remarkable 75% “Responded that Quality Engineering skills are considered the most critical for Quality Engineering associates in the financial sector”!
• In the Telco sector, “… the need for critical testing is still new, and while organizations are slowly understanding the implications of overlooking it, the impact will be increasingly apparent in the future.”

(There are more basic proofing errors in this section of the report with repeated text in both the Retail (in the “Shifts in consumer buying channels” section) and Public (in the “Managing complexity in Cloud-based environments”) sectors.)

Geography-specific reports

The main World Quality Report was again supplemented by a number of short reports for specific locales. I only reviewed the Australia/New Zealand report and spotted a couple of interesting observations.

Firstly:

In ANZ, there’s been a shift from a scaled Agile approach to a more balanced methodology. Organizations are reassessing the value of dedicated testing teams versus integrating testing within engineering squads. While early testing and shift-left practices are gaining momentum, the specialized expertise in functional assurance, integration, and continuous testing is being reaffirmed.

I’m not sure how to interpret this, whether they’re saying there’s a move towards dedicated testing teams or away from them and towards embedded testers in teams. This is another example of poor wording in the report that should have been caught by decent copy editing.

The comment around automation is pretty stunning:

Automation … continues to be both an opportunity and a challenge. While some organizations are early adopters, many still struggle with the complexities of automating business-as-usual (BAU) projects.

Calling anyone undertaking automation today an “early adopter” seems absurd to me when we’ve had automation in various forms available to us in testing for decades. This very report in previous years has indicated the growing adoption and importance of automation.

Continuing the trend of “AI everywhere” in this report, it says this about using AI specifically for “hyper-automating test design”:

Gen AI is rapidly becoming a hot topic in ANZ, with organizations exploring its potential to address efficiency pain points within the testing cycle. While there is growing interest in leveraging Gen AI for hyper-automating test design, there is also a cautious approach towards trusting its outcomes. Some organizations have paused their experimentation to centralize control and ensure reliability.

I read this as saying “we tried to automate test case creation using AI and it didn’t work very well”!