Testing SaaS – Quality in a “Buy not Build” Culture
It may surprise some people as it’s not commonly talked about in the industry but not every company will develop its own IT systems. Not every company has an agile development culture, some have a culture of buying the best of the market solutions and integrate them together.
This brings about a very different challenge from a Quality Engineering/Testing perspective. Suddenly the value isn’t in having testers embedded in the sprints to ensure tests are written and automated before a release is marked as complete. It’s a scenario that has certainly opened my eyes and one that I am now comfortable with and am looking for ways to push the quality agenda forward.
Caveat: While I may have described the process in a step by step format, it doesn’t mean it has to be in a waterfall environment, some vendors will be able to integrate into more agile delivery processes, but unless they are defining a bespoke solution just for you, they will likely have release cycles outside of your control
The very first step of identifying the problem that needs solving is similar, people come together to work out what is needed, maybe some loose requirements are defined, maybe a rough idea of ideal design is drawn up but then the first major difference occurs…
Rather than straight away looking at design in detail, the question will be asked “what options are there on the market for this?”. Depending on the size and governance within the organisation, the next step could be as simple as someone choosing a vendor/solution that solves the problem, or it may require a process of selection which would include two stages:
- RFI – Request for Information where multiple vendors would be engaged to learn about the options they could possible offer and share the outline of the problem that needs solving.
- RFP – once a narrowed list of vendors, a Request for Proposal process can occurs where these vendors would submit proposals of how they could solve the problem.
Advocating for Quality in an RFP Process
Within the procurement process of finding the right solution, quality can (and should) play a part in the decision. There are 2 key stages where QA/QE/Testing teams can provide input to the process:
- Testing Questions to be answered as part of submission – these may include questions around how the vendor performs the testing of their solution, their defect management process, test environments they may provide with their solution and any automation packs/guidance they can provide (if any) to enable you to run automation when changes in your IT estate may affect this solution. These answers can be rated as part of the assessment of each vendor and these scores can then be discussed as part of choosing the final solution.
- Clauses in Contracts – Once a final vendor has been chosen, ensure that there are clauses in the contract that hold them to the levels of quality you expect them to deliver. Ultimately, you need to have confidence in what is being provided to you as you won’t have time to functionally test their solution fully (if atall). These clauses may include SLAs on defect fixes, providing all testing reports for deliveries and agreements on measures around defect leakage (defects which they should have found before integrating).
Once these contracts are signed, you should have a basic understanding of the guardrails you will be working within for this solution. A full solution design can then happen in collaboration with the vendor of choice.
Then you can start looking at the actual testing you will be able to perform
Remember, Your Context is Key
There is a common myth around saas solutions that “you won’t need to do your own testing because the vendor will cover it”. I can categorically say now that this is NOT TRUE! In all likelihood, most vendors will not develop and test their solution in your exact context (i.e, with your exact integrations into your other systems, with you exact data configurations etc), so it is crucial that you build into your testing approach, time to run some basic scenarios in this exact setup. Call this phase “Vendor Acceptance Testing or Vendor Smoke Testing”, ideally you would be able to automate these tests and provide feedback very shortly after receiving a release from the vendor, meaning you can effectively reject the build if it fails the VAT/VST rather than diving into further testing before finding those critical issues.
It’s All About the Integrations
This is where you can get into the meat of any testing that may need doing, if you have verified that the system basically works in your contexts configuration, now it is time to plug it into your IT estate, run considerable about of production like test data through it and do various levels of interactions with upstream and downstream systems. Does it communicate via APIs with these other systems? Is there an overall integration layer that all systems plug into? Are there message queues involved? This is about getting into the detail and proving that the data flow isn’t impacted by slotting this system into the architecture.
Without going into the detail of why integration testing is needed, this is your chance to validate the full stack of systems plugged together which perform a particular User Journey, such as a user booking tickets but being able to see all the detail from the bookings appear in the finance systems and seats being reserved at a venue for a particular event.
This is the section of the testing that could benefit most from automation, but the challenge being, when plugging multiple vendor systems together, the automation may initially take more time, but the ROI will be worth it if you can get it right.
Care about Performance
The challenge with not building the solution in-house is that performance is sometimes difficult to get right. If you are a retail company who has peaks for Black Friday, you need to feel confident it can handle the demand. While this should be discussed at the RFP and subsequent deeper design that is needed, it can cause more problems if this is not discovered until the integration testing phase. Set out your performance markers as early as possible and push back if early tests show any signs of concern.
It’s also key to look at the full User Journey impact on performance, while this new system might be performing OK, has it done something to the data it pushes out which means the dependant system now runs slower?
Don’t Forget the Users
As with most system releases, particularly if they are for internal users, a UAT phase will be necessary. It’s key to remember that these users could possibly work in quite a procedural way so dropping a new system into their workflow could be quite disorientating. So it is really crucial that the UAT is coordinated in such a way to understand what the users day looks like and how this new system will impact that flow. It therefore wouldn’t be about the test team writing all the tests, but instead working with key users to understand the workflows they work through and ensure they have aligned journeys with the new system to what they did in the old setup.
Quality may look Different but Just As Important
For anyone who has worked in pure tech companies, you tend to be able to get close to the code and have the ability to make changes to code in an instant. When working with SaaS solutions, you have far less ability to see the code and have any influence on it directly, so it is key to find other ways to get the information you need, to get confidence from the vendor on their quality practices and build the right testing process to ensure you get the most out of the solution for your company.
