We use some essential cookies to make this website work. We’d like to set some additional cookies to understand how you use the website and to improve it. We also use cookies set by other sites to help us deliver content from their services. You can read more about our cookies before you choose.
This is a new website – your feedback will help us to improve it.
A framework for understanding confidence in artificial intelligence (AI)
Executive Summary and Report Overview
This report presents a framework for understanding what influences confidence in artificial intelligence (AI) within health and care settings, which was developed following analysis of the academic literature and the interviews conducted for this research.
Establishing confidence in AI can be conceptualised as the following.
Increasing confidence in AI by establishing its trustworthiness (applies to all AI used in healthcare).
Trustworthiness can be established through the governance of AI technologies, which conveys adherence to standards and best practice, and suggests readiness for implementation.
Trustworthiness can also be established through the robust evaluation and implementationof AI technologies in health and care settings.
Increasing confidence is desirable in this context.
Assessing appropriate confidence in AI (applies only to AI used for clinical decision making).
During clinical decision making, clinicians should determine appropriate confidence in AI-derived information and balance this with other sources of clinical information.
Appropriate confidence in AI-derived information will vary depending on the technology and the clinical context.
High confidence is not always desirable in this context. For example, it may be entirely reasonable to consider a specific AI technology as trustworthy, but for the appropriate confidence in a particular prediction from that technology to be low because it contradicts strong clinical evidence or because the AI is being used in an unusual clinical situation. The challenge is to enable users to make context-dependent value judgements and continuously ascertain the appropriate level of confidence in AI-derived information.
Figure A illustrates the conceptual framework and lists corresponding factors that influence confidence in AI. These comprise factors that relate to governance and implementation, which can establish a system’s trustworthiness and increase confidence. Clinical use factors affect the assessment of confidence during clinical decision making on a case-by-case basis.
Figure A: Framework for understanding confidence in AI among the healthcare workforce
The primary focus of this report is understanding and assessing appropriate levels of confidence in AI-derived information during clinical use. However, as appropriate confidence in AI for clinical decision making is premised on establishing the trustworthiness of these technologies, the key elements of governance and implementation that underpin confidence are addressed first before clinical use is discussed in more detail.
Governance
Increasing confidence through the governance of AI technologies
Robust governance underpins the trustworthiness of AI technologies, and can increase confidence in workers who commission, implement and use AI for any task in health and care settings. Aspects of such governance can include:
Regulatory frameworks and standards
Regulation and standards can provide assurance that AI technologies have been developed responsibly, work as advertised and are safe for users and patients. Interviewees for this research highlighted several areas where developments in regulation could increase confidence in AI. These include the regulation of AI technologies (through AI-specific medical device regulation), the regulation of healthcare settings (through guidance on the safe and effective use of AI technologies), and the regulation of professionals who develop, validate and use AI (through advice from regulators of healthcare workers).
Evaluation and validation
AI technologies classed as medical devices require internal validation for Medicines and Healthcare products Regulatory Agency (MHRA) approval, but external validation, prospective clinical studies and, in some cases, local validation can build confidence in an AI’s performance. Several standards and tools have or are being developed for medical devices and clinical research to guide approaches to the evaluation of AI products, including the National Institute for Health and Care Excellence (NICE) evidence standards framework.
Guidelines
Guidelines on the procurement, development and use of AI can enhance confidence when adopting AI in healthcare settings. Clinical guidelines from entities such as NICE and the Royal Colleges can also support confidence in using AI technologies.
Liability
Clarity on liability across different AI technologies is crucial to securing the workforce’s confidence in using AI technologies. Currently, there is uncertainty as to who will be held to account if AI products are used to make clinical decisions that lead to patient harm. Responsibility could fall to the clinician who uses the technology, the deploying organisation, the industry innovator that developed the technology or those who validated and approved the technology for clinical use. Various legal frameworks may be applicable including negligence, product liability and vicarious liability. The NHS AI Lab’s Regulations programme is exploring these issues in greater depth, including through its ‘Liability and Accountability’ portfolio of work.
Implementation
Increasing confidence through the robust implementation of AI technologies
Interviewees for this research highlighted that the safe, effective, and ethical implementation of AI in health and care settings underpins the trustworthiness of AI technologies, and contributes to the workforce’s confidence in these technologies. Such implementation can include:
Strategy and culture
The leadership, management and governance bodies within health and care settings establishing AI as a strategic asset, and maintaining organisational cultures conducive to innovation, collaboration, and public engagement. This can include co-developing AI technologies ‘from the ground up‘ with industry innovators, by engaging and involving multi-disciplinary teams (including clinicians, information technology and governance specialists, clinical domain experts and data scientists) and internal decision-makers early in discussions about their needs and implementation challenges.
Technical implementation
Addressing any technical implementation challenges involving information technology infrastructures, interoperability, and data governance requirements is also crucial. Interviewees noted that agreed information technology and governance arrangements are instrumental to healthcare workers’ confidence in using AI technologies. A focus on the digitalisation of health and care services is an important prerequisite to the adoption of AI technologies, as supported by the NHS’s What Good Looks Like framework.
Local validation
Procurement or commissioning entities within health and care settings will need to decide whether to validate the performance of AI technologies to ensure its performance translates to local data, patient populations and clinical scenarios. There are many unknowns and potential risks involved in ‘translating’ AI technologies from controlled development and validation settings to complex and highly individual real-world settings. These risks can relate to the ability of settings to understand the suitability and performance of the AI technologies locally, to maintain the ongoing rigour of that performance, and to minimise any unfair impact on, or harm to, patients.
Systems impact
Healthcare workers will be more confident in AI technologies that are safely and efficiently integrated into existing workflow systems that should include pathways for reporting safety events. An ethical approach to AI will also be essential to achieving confidence in AI. Interviewees noted that, at a minimum, this can include the principles of fairness, transparency, and accountability and ensuring equitable benefits across patients.
Clinical use
Assessing appropriate confidence in AI-derived information during clinical decision making
Figure B shows that a clinician’s actual confidence (user confidence) in AI-derived information for decision making may be inappropriately high or low if it does not match (along the ‘ideal’ line) the appropriate level for a clinical case. As discussed in this section, this appropriate level is likely to vary from case to case, depending on clinical factors and the AI-derived information itself.
To avoid inappropriately high or low levels of confidence, clinicians need to determine the appropriate level of confidence in the specific AI-derived information available at the point of making each clinical decision.
Figure B: Ideal and inappropriate levels of confidence
A complex set of considerations can dictate how to determine an appropriate level of confidence in AI-derived information, depending both on the technology and the clinical scenario, and with certain AI technologies and use-cases presenting lower clinical or organisational risks.
Synthesising and evaluating information from many disparate sources are key skills in clinical decision making, whether for diagnosis, prognostication or treatment. If incorporated and considered with appropriate confidence, information from AI technologies has the potential to make clinical decision making safer, more effective, and more efficient.
To achieve this, clinicians will need to understand when AI-derived information should and should not be relied upon, and how to modify their decision making process to accommodate and best utilise this information. This might include considering factors like:
other sources of clinical information and how to balance these with AI-derived information in decision making
the clinical case for which the AI is being used
the intended use of the AI technology
Several factors can influence how clinicians view AI-derived information (their user confidence), potentially leading to inappropriately high or low levels of confidence. These include:
Clinicians’ attitudes
General digital literacy, familiarity with technologies and computer systems in the workplace, and past experiences with AI or other innovations can influence assessments of confidence in AI-derived information.
Clinical context
The clinical context in which AI is used can influence confidence in the technologies and in the derived information, including in relation to the levels of clinical risk and the degree of human oversight in the AI decision making workflow.
AI model design
Various design characteristics can influence confidence in AI technologies. For example, the way AI predictions are presented (such as diagnoses, risk scores, or stratification recommendations) can affect how clinicians process information and potentially influence their ability to establish appropriate confidence in AI-derived information.
Cognitive biases
Cognitive biases, including automation bias, aversion bias, alert fatigue, confirmation bias and rejection bias can affect AI-assisted decision making. The propensity towards these biases may be affected by choices made about the point of integration of AI information into the decision making workflow, or the way such information is presented. Interviewees for this research highlighted that enabling clinicians to recognise their inherent biases, and understand how these affect their use of AI-derived information should be a key focus of related training and education. Failure to do so may lead to unnecessary clinical risk or the diminished patient benefit from AI technologies in healthcare.
Page last reviewed: 11 April 2023
Next review due: 11 April 2024
