“The Purpose of Analysis is Insight”
Testers want to analyse the product and process to improve quality. Once every engineering team had a statistician, that is not the case today. Testers can help their team by using statistics, such as control charts, to do analysis. There are a lot of resources to help us use statistics, for example, this blog post is based on a chapter from Understanding Variation: The Key to Managing Chaos. Control charts are a form of statistics that provide insight into processes and were part of the philosophy that W. Edwards Deming used to rebuild the Japanese economy and turn Ford into America’s most profitable car manufacturer.
A few years ago I was Test Lead in a company that treated each bug it needed to fix as an incident. Each time we fixed a bug we did root cause analysis with the Five Why’s, and I kept a line chart showing the count of how often we needed to fix a bug. The chart could have looked something like this:
The line chart shows that there is never more than one bug at a time and bugs become more frequent towards the end of the year. The line chart does not provide deep insight.
I can display the count of bugs in a control chart. The control chart has three additional lines, the average (mean) plus upper and lower limits, which are calculated from the time series data. Viewing the count of bugs in a control chart gives me a little more insight. The chart shows that the time series stays within the upper and lower limits, which means that the number of bugs is under statistical control.
I created an XMR chart with this data. An XMR chart consists of an X chart and an MR chart. An X chart is a control chart and is called an X chart because x̄ is the symbol for average. The MR chart contains the moving ranges. The moving ranges are the differences between the data points in the time series in the X chart.
XMR charts are relatively easy to create. They are based on the time series data in the X chart. Moving ranges are the differences between the data points in the X chart time series data and can be calculated using an abs function. The averages in both charts can be calculated using an average function. The upper and lower limits in the X chart and the upper limit in the moving ranges are calculated using a simple equation on Pages 40-41 in Understanding Variation: The Key to Managing Chaos.
The X chart shows that the number of bugs per month rose during the year, then fell and that it rose close to the upper limit towards the end of the year. The upper and lower limits are not targets or goals that have been added to the chart, they have been calculated from the data and so are the voice of the process. If the data points go outside the limits the process is said to be out of statistical control. The Moving Ranges chart shows that the rate of increase rose twice.
the rate of increase rose twice.
The analysis provided by the XMR chart provides insight regarding the rate at which bugs need to be fixed.
The X chart shows that the rate of bugs needing to be fixed is increasing. The increases are occurring within the upper and lower limits. Changes in the time series outside the upper and lower limits are due to special causes, that is fleeting events. Change within the upper and lower limits are due to common causes that is they are the faults of the system. The increase in the rate at which bugs need to be fixed is within the upper and lower limits and so is a fault of the system.
The Moving Ranges (MR) chart shows two dates when the increases occur.
The company can use this analysis to decide if it wants to find and address the causes of the increases. It may decide that because the rate of fixing bugs is under statistical control it has more important issues to deal with. However, if it does want to find and address the causes of the rate of increase the analysis provided by the XMR chart will help.
“The purpose of analysis is insight”[3] and control charts provide useful insights. A better insight into rare data, such as the fixing of bugs for this company, is gained by analysing the rate of fixing bugs rather than by counting the bugs. “In general, counts are weaker than measurements”[2].
Testers can use XMR charts, including control charts, to enable their team to get insights into their product and processes.
Thank you Rob Park for prompting me to explore and write this.
Here are some resources that will help you if you would like to use XR charts and control charts to gain insights:
Introductory guides to control charts:
- A Beginners Guide to Control Charts Deming Institute
- Statistical Process Control Charts | ASQ (includes a spreadsheet template for control charts)
- Getting Started with Process Behavior Charts by Rob Park
A useful guide if you want to create an XMR or control chart:
- Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler
A video introduction to control charts:
An API that can be used to create control charts
- Geckoboard dataset API I am using Geckoboard dataset API to create control chart dashboards.
References
[1] Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler (1993, p102)
[2] Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler (1993, p104)
[3] Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler (1993, p33)