Gain insights by using control charts to analyse your performance test results
On Friday 16 May 1924 Walter Shewhart gave his manager at Bell Telephone Laboratories a memo. The memo “suggested a way of using statistics to improve quality in telephones.[1]” Shewhart’s memo proposed using Statistical Process Control, including Control Charts for visualisation, to improve quality. Shewhart sparked “a revolution in quality control”[2] that can help us analyse the results of our performance tests.
Control charts visualise time series data along with the data’s average and upper and lower natural process limits. “Faults from fleeting events[3]” are described as special causes and appear outside a control chart’s upper and lower natural process limits. Faults of the system are described as common causes and appear on a control chart within the upper and lower natural process limits. The natural process limits are calculated from the variations in the time series data[4].
The above control chart shows UK inflation as it spiked post covid. The control chart shows that UK inflation had a fault outside the natural process limits and so had a special cause. Seeing whether the data from performance tests falls inside our outside natural process limits enables a tester to understand if the faults have a special cause ( a ‘fleeting event’) or a common cause (a fault of the system). This is very useful when analysing the results of performance tests because a fault due to a fleeting event should be treated differently from a fault with the system.
Eight patterns, known as ‘Nelson Rules’, can be used to analyse data variation within the chart’s natural process limits.
A control chart lets managers track variation and compare datasets. After the introduction of control charts managers could find and fix the causes of defects, rather than blame the staff for faults.
Control charts can also be called Process Behaviour Charts.
We now have one hundred years of learning from using control charts, which are now widely used. If you have a Fitbit you will be using control charts because the personal ranges for your health data are based on control charts. Here are links to guides to using them in different processes including DevOps automated governance, analysis of unit testing, load testing, customer satisfaction, health care, nuclear power, car manufacturing, education quality management system, managing stock levels, and marketing Campaigns.
Thank you to John Willis, Dennis Sergent, Rob Park and the Deming Profound Book Club members for helping me to gain a deeper understanding of control charts.
Here are some resources that will help you if you would like to use control charts to gain insights from your performance test results:
Introductory guides to control charts:
- A Beginners Guide to Control Charts Deming Institute
- Statistical Process Control Charts | ASQ (includes a spreadsheet template for control charts)
- Getting Started with Process Behavior Charts by Rob Park
A useful guide if you want to create a control chart:
- Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler
A video introduction to control charts:
An API that can be used to create control charts
- Geckoboard dataset API I am using Geckoboard dataset API to create control chart dashboards.
A One Day course at Coventry University:
References
[1] Quality or Else by Lloyd Dobyns and Clare Crawford-Mason (1991, p52)
[2] Manufacturing the Future by Stephen B. Adams and Orville R. Butler (1999, p216)
[3] Out of the Crisis by W. Edwards Deming (1982, p314)
[4] Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler (1993, p35)