Operations leaders face unprecedented challenges. They have to manage the new complexity of becoming cloud native and anticipate the implications of GenAI. If that’s not enough, they also have to find answers to the Digital Dichotomy, balancing the macroeconomic “slowdown” with the “big hurry” to innovate to keep up with innovation pacesetters. Yet, it is not a question of doing one or the other—they must address all those challenges simultaneously.
Against this background, quality assurance (QA), or simply “testing,” as it was called in the old days, can no longer be a reactive afterthought coupled with an unwillingness to invest in quality. Instead, it must become an integral part of the software development lifecycle (SDLC) and take on a much more holistic responsibility for assuring transformational outcomes.
This is the context for HFS’s seminal study on assuring the Generative Enterprise™ (HFS Horizons: Assuring the Generative Enterprise™, 2024). We sought to understand better where organizations are with their QA efforts and how they are trying to ensure transformational outcomes. We also explored how they assure change agents such as automation, AI, and blockchain. Finally, we looked at the adoption of GenAI through the lens of QA. In the following, we share the insights gleaned from our study.
Lofty aspirations for quality assurance
Finding your bearings on all things QA is difficult because there is a massive gap between the aspirations (and lip service) for QA and QA functions’ maturity and enterprises’ willingness to invest in them. While the mature end of the market is pivoting toward quality engineering (QE) with a focus on achieving continuous testing, data-driven decision-making, and cross-functional collaboration, two-thirds of the market is stuck at a lower maturity level, often still working with a Waterfall methodology, as Exhibit 1 depicts. Simply put, most organizations are on the left-hand side of this infographic.
Exhibit 1: Pivoting to quality engineering is about aligning quality assurance to customer journeys
Source: HFS Research, 2024
Yet, we might finally see organizations embracing the transformation of their quality assurance functions. According to our study’s reference clients, 95% described the primary value delivered by their service provider today as the ability to drive functional optimizations with selective quality assurance capabilities. Simply put, they expect services on the left-hand side of Exhibit 1. However, in two years, most of these organizations expect a transformation of their quality assurance functions, intending to drive experience-led outcomes and stakeholder experiences while creating new sources of value through ecosystem synergy.
“Shift right” is starting to augment “shift left”
Discussing these issues with QA leaders, technology partners, and service providers could crystalize more nuanced market dynamics. Most organizations have embraced “shift left” principles emphasizing the early and proactive involvement of quality assurance activities in software development. We are seeing mature QA functions augment this with “shift right” principles advocating quality activities also at the later stages of the SDLC.
Regarding organizations’ QA priorities, four clusters jump out for us: First, carving out a budget for QE architecture modernization and transformation. This goes back to the discussion of the Digital Dichotomy and the need to self-fund innovation. Second, solving the conflict of production quality versus speed to market. Not least, with innovation cycles dramatically compressed with the ascent to GenAI, this is a hard nut to crack. Third, overcoming a new complexity to provide integrated assurance for apps, infrastructure, and platforms or progressing toward the OneOffice™, as HFS would put it. And fourth, driving change management to foster transformation. Yet, large chunks of the QA community are stuck in a tools and technology mindset.
There is much noise but little assurance on GenAI
Let’s zoom in on the topic that comes up in every discussion we have, regardless of the context. While many providers hype use cases around domain knowledge and code creation, grown-up discussions about how to assure GenAI are sparse. As such, Cognizant’s Artificial Intelligent Lifecycle Assurance (AILA) and Infosys’ AI Assurance Platform are more exceptions than rules. This provides indicators for the enterprise adoption of GenAI. We are still at the very beginning of enterprise adoption. The core value proposition of GenAI use cases in QA is having a higher accuracy with less data. Thus, providers can drive new levels of automation to generate test scenarios and test cases.
While we had many discussions on the infusion of QA with GenAI, deliberations on governance on GenAI are still nascent. Where we had honest discussions, providers reflected that large language models (LLMs) are largely unfamiliar entities. One executive framed this aptly: In the context of GenAI, we must show humility to the unknown. Thus, the predominant way to engage is through experimentation. However, there is no beating around the bush that we were a tad underwhelmed by the discussions of assuring GenAI outcomes; outside of the service providers, we had some stimulating conversations. For instance, MunichRe provides insurance for AI, while German start-up QuantiPi blends quality assurance with governance for GenAI.
North Star (autonomous) persona-based testing
Beyond the broader pivot to QE, what have we learned about the evolution of quality requirements? Like cloud-native operations are shifting toward persona-based solutions, QE is shifting toward persona-based testing, requiring capabilities to support better specific stakeholders such as product teams, site reliability engineering (SRE), DevOps engineers, and beyond. But to be clear, this is the North Star, and only a few organizations have it on their roadmap. Looking at these requirements in the context of GenAI, the key is blending prompt engineering with specific persona scenarios. Using prompts to generate automation scripts such as Selenium can significantly enhance the scale of automation. Lastly, using the input from one prompt to generate another prompt can leverage GenAI to industrialize offerings.
Horizon 3 market leaders blend a compelling vision of transformation QA with nuanced approaches to assure change agents such as GenAI
Last but by no means least, congratulations to the Horizon 3 market leaders. These leaders’ shared characteristics include blending a compelling vision of transformational QA with nuanced approaches to assure change agents such as GenAI. The wheat separates from the chaff when providers ensure transformation outcomes are enabled rather than depicting functional testing and an overreliance on tools and technology. The leaders are pushing the envelope on transformation with new themes such as cross-functional testing, zero-touch testing, black-box testing, and beyond. Exhibit 2 outlines the detailed rankings of our research.
Exhibit 2: The vanguard of the quality assurance services ecosystem
Accenture and Wipro stand out, clearly outlining the evolution to QE. They shift the focus of their narratives by depicting transformational journeys and outcomes rather than getting stuck in tools and technology. Infosys supports clients’ pivots to product-centric delivery and intelligent ecosystems, while TCS infuses emerging technologies into QE and embeds them at the core of transformation. Capgemini surprised us with a nuanced and thoughtful narrative on adopting GenAI. Cognizant was demonstrating test automation chops and strongly emphasized customer experience assurance. Perhaps surprisingly to some, Persistent is going deep on cloud-native transformation by investing ahead of the market in digital engineering and GenAI capabilities.
The Bottom Line: The QA community needs to emancipate itself
The innovation delivered by the QA community continues to be stupendous. Yet, the community does a modest job articulating the goals and outcomes those change agents achieve. Without snapping out of this tool- and solution-centric view of QA, it is difficult to articulate a more value-driven approach, where QA executives get the limelight they crave to discuss and decide the significant sourcing issues. Stepping up to QE transformation and putting QE at the heart of transformation and software development is an opportunity for the community to emancipate itself from being boxed in a technology and tools mindset. Discussions with the market leaders in Horizon 3 gave us some hope that we are getting closer to this.
HFS subscribers can download the report and find many more details here.
Posted in : Artificial Intelligence, GenAI, Generative Enterprise, HFS Horizons, The Generative Enterprise
