AI Tooling In DevOps

Patrick Debois and myself have interesting conversations about DevOps, and AI tooling in DevOps, in Twitter. Sometime last month, I decided to ping him to see if we could do a podcast for the Software Testing and Quality Talks channel about DevOps and GenAI. Patrick kindly agreed, and we had a very lively and interesting recording.

My focus during the discussion was from the quality perspective. I brought up issues related to GenAI and how they can be addressed. Being a non-deterministic system, testing a GenAI system has its great challenges. Patrick suggested techniques like tools for specific usage (for example, calculator), RAG and LLM agents as potential solutions to the challenges.

The challenges to the testing and quality community remains. In addition to the steep learning curve about how GenAI works, and how to effectively test it, the application testing side has its own challenges with respect to domain versatility, data quality, and adapting to changing and evolving scenarios in the industry and business.

Patrick highlighted key areas in DevOps where GenAI could be useful. The major challenge is the genAI algorithm itself which generates content on its own accord, sometimes breaking the rules that it is fed with. Sometimes, it is difficult to find out if these are hallucinations or otherwise. The metrics provided by the industry are not very helpful in qualitative aspects, and to me, they sound like vanity metrics. Although there are tools that provide syntactic and semantic correctness measurements, they can go only to certain extent. Human assessment is always a bullet in the GenAI testing strategy framework.

It sounds to me that the architecture is more than one can swallow in terms of validation and verification, the testing angle. The key might be in automation, but automation is limited in scope and will not cater for all possibilities.

There is hope. Based on the international standards, working on a practical framework is under works, and let us see how it pans out. My primary focus is on the functional and security aspects of GenAI, with performance the next consideration.

What are your thoughts on GenAI and how AI tooling can help DevOps? Please feel free to comment.

Also, reach out to me if your organisation is looking for help on quality aspects of GenAI and its applicability to your software development life cycle.

The post AI Tooling In DevOps appeared first on Venkat Ramakrishnan.