By: Chris Peace
Risk assessments – how ineffective can they be?
It is not uncommon for an investigation to be carried out into events resulting in major (and, sometimes, minor) consequences. The findings of such investigations may be used, for example, to develop better controls, determine the level of insurance payments, allocate blame or provide evidence in court cases. However, it seems unusual for such an investigation to include the review of a risk assessment that preceded the event in question, and rare for such a review to be published.
On 22 August 2015 an ex-military Hawker Hunter jet crashed during the Shoreham air show in the UK, resulting in multiple fatalities and injuries. As part of its investigation, the UK Air Accident Investigation Branch (AAIB) asked the Health and Safety Laboratory (HSL, part of the Health and Safety Executive) to review a risk assessment (the Shoreham Air Show Air Display Risk Assessment) apparently completed on 14 August 2015. The HSL report formed part of the AAIB report (AAIB, 2017, pp. 323-343) and provides salutary lessons for anyone who carries out or updates an earlier risk assessment, especially if it is foreseeable that the consequences might include serious harm to people or property.
It might be thought that a risk assessment covering the "airside” operation of a high-powered, ex-military jet aircraft at a public display adjacent to a busy main road and built-up area would be anything but formulaic, that it would apply a range of risk techniques and engage with many stakeholders. The HSL report shows many shortcomings in the 2015 risk assessment, including failure to:
- thoroughly review risk assessments from 2013 and 2014 to identify any changes in the context or the nature of risks
- learn from incidents or near-hits at previous air shows
- include a map of the vicinity to help understand the context of the air show and what lay "over the boundary fence"
- follow official guidance published by the regulatory agency (the UK Civil Aviation Authority) or give an explanation for using a different approach
- engage with all relevant stakeholders
- ensure that all stakeholders had a shared understanding of risk
- record the qualifications and experience of the risk assessors
- systematically develop a comprehensive list of hazards or causal factors
- go beyond a simplistic risk matrix (Peace, 2017) and risk register
- explain why quantitative risk analysis was not possible
- identify “practicable” actions and assess whether they were "reasonable" ( ie, apply the reasonably practicable test) (Peace, forthcoming)
- effectively communicate the risks to decision makers
- fully set out and correctly evaluate all relevant factors, including controversies and uncertainty
However, the Shoreham Air Show Air Display Risk Assessment document is reported by HSL to have included:
- the purpose of, or terms of reference for, the risk assessment
- safety assessment criteria
- a claim that the risk assessors had a “wealth of flying experience in general and display related experience in particular”.
While not stated explicitly in the report, it is possible that the official guidance itself may have been deficient, a matter of concern for any regulatory agency giving guidance on compliance with legislation.
Apart from remedying the failures noted above, what else might have been done better? Experience shows that engaging key stakeholders in the use of bow-tie analysis as part of the risk assessment process would have identified more causal factors and a wider range of possible events, resulting in a better understanding of possible consequences, and discovery of control gaps.
What might this report have to do with risk assessments other than for air shows?
Many organisations carry out activities once or twice a year (eg, complete a maintenance shutdown; accept a major delivery; carry out a major stock-take). There might be some records from similar, earlier activities, perhaps including a risk assessment. The HSL report gives some ideas for how the review of an earlier risk assessment might be carried out and so contribute to successful completion of the objectives.
The HSL report is also worth reading just to see how ineffective some risk assessments are. Participants on short courses are often surprised by what should be included in a risk assessment and my research at Victoria University also shows we can do better.
Improving risk assessments is not difficult. We have developed a risk canvas that places key features on a pre-printed A1 sheet for use in workshops and training courses. Analysis of feedback shows it has helped improve understanding of the inputs to, and the process for, effective risk assessments. The results of this applied research will be submitted for publication later this year.
Chris Peace provides consultancy, training and mentoring services. He is researching the effectiveness of risk assessments in informing decision makers at Victoria University and can be contacted at email@example.com.
AAIB. (2017). Report on the accident to Hawker Hunter T7, G-BXFI near Shoreham Airport on 22 August 2015. Aircraft Accident Report 1/2017Aldershot: Air Accident Investigation Board. Retrieved from www.gov.uk/aaib-reports/aircraft-accident-report-aar-1-2017-g-bxfi-22-august-2015, 30 July 2017.