Quality assuring analysis
Analytical quality assurance (AQA) ensures our analysis is accurate, reliable, and free from errors that could lead to incorrect conclusions or misguided decisions.
The Aqua book
The Aqua book provides high-level guidance on producing quality analysis for government. This book sets out how departments should ensure their work is fit-for-purpose through verification and validation.
These checks apply to anything that can be loosely defined as a “model”. If your work takes an input, processes it, and produces an output, this comes under the scope of AQA. This includes but is not limited to visualisations, spreadsheets, machine learning models, and even back-of-napkin-type calculations.
The Aqua book establishes four principles which must be considered when undertaking any work involving data/models:
- Proportionality of response: the extent of the analytical quality assurance effort should be proportionate in response to the risks associated with the intended use of the analysis. These risks include financial, legal, operational and reputational impacts. In addition, analysis that is frequently used to support a decision-making process may require a more comprehensive analytical quality assurance response
- Assurance throughout development: quality assurance considerations should be taken into account throughout the life cycle of the analysis and not only at the end. Effective communication is crucial when understanding the problem, designing the analytical approach, conducting the analysis and relaying the outputs
- Verification and validation: analytical quality assurance is more than checking that the analysis is error-free and satisfies its specification (verification). It must also include checks that the analysis is appropriate, that is, fit for the purpose for which it is being used (validation)
- Analysis with RIGOUR: quality analysis needs to repeatable ®, independent (I), grounded in reality (G), objective (O), have understood and managed uncertainty (U), and the results should address the initial question robustly ®. It is important to establish how much we can rely upon the analysis for a given problem
Note that AQA is not just about software quality assurance. It can also include dealing with ethical considerations, reasons for choosing the method/technique, and validating analytical assumptions and caveats.
Additional Aqua book resources are available, and the Government Analytical Function, Government Data Quality Hub, and other departments have also produced:
- guides to ensure your work is fit for purpose when working to very tight deadlines
- guides to ensure your data is fit for purpose
- a curriculum around quality assurance, validation, and data linkage