1.2 A framework for understanding confidence in artificial intelligence (AI)

Chapter 1: Introduction

The framework for understanding what influences confidence in artificial intelligence (AI) within health and care settings was presented in the first report.

The first report1 presented a framework for understanding what influences confidence in artificial intelligence (AI) within health and care settings, which was developed following analysis of the academic literature and the interviews conducted for this research.

Establishing confidence in AI can be conceptualised as the following.

Increasing confidence in AI by establishing its trustworthiness (applies to all AI used in healthcare)
  • Trustworthiness can be established through the governance of AI technologies, which conveys adherence to standards and best practice, and suggests readiness for implementation.
  • Trustworthiness can also be established through the robust evaluation and implementation of AI technologies in health and care settings.
  • Increasing confidence is desirable in this context.
Assessing appropriate confidence in AI at the point of use (applies only to AI used for clinical decision making)
  • During clinical decision making, clinicians should determine appropriate confidence in AI-derived information and balance this with other sources of clinical information.
  • Appropriate confidence in AI-derived information will vary depending on the technology and the clinical context.
  • High confidence is not always desirable in this context. For example, it may be entirely reasonable to consider a specific AI technology as trustworthy, but for the appropriate confidence in a particular prediction from that technology to be low because it contradicts strong clinical evidence or because the AI is being used in an unusual clinical situation. The challenge is to enable users to make context-dependent value judgements and continuously ascertain the appropriate level of confidence in AI-derived information.

Figure 1 illustrates the conceptual framework and lists corresponding factors that influence confidence in AI. These comprise factors that relate to governance and implementation, which can establish a system’s trustworthiness and increase confidence. Clinical use factors affect the assessment of confidence during clinical decision making on a case-by-case basis (see first report1 for detailed analysis of these factors).

The figure displays two chevrons that are grouped sequentially and point up to a semicircle. The chevron at the bottom is titled Governance, and on top of that is a second chevron titled Implementation. The Implementation and Governance chevrons are summarised as 'Increasing confidence for all AI used in healthcare'. The semicircle is titled 'Clinical Use' and is summarised as 'Assessing appropriate confidence in AI for a specific clinical decision'. The right part of the figure lists influencing factors that correspond to Governance, Implementation and Clinical Use. Factors related to Governance include regulation and standards, evaluation and validation, guidelines and liability. Factors related to Implementation include strategy and culture, technical implementation, local validation and systems impact. Factors relating to Clinical Use include clinician attitudes, clinical context, AI model design, workflow integration and cognitive biases.
Figure 1: Framework for understanding confidence in AI among the healthcare workforce

References

1 Nix M, Onisiforou G, Painter A. Understanding healthcare workers’ confidence in AI. Health Education England & NHS AI Lab. 2022. https://digital-transformation.hee.nhs.uk/binaries/content/assets/digital-transformation/dart-ed/understandingconfidenceinai-may22.pdf Accessed 29 June, 2022.

Page last reviewed: 18 April 2023
Next review due: 18 April 2024