5.3 Cognitive biases and appropriate confidence in artificial intelligence (AI)-assisted clinical reasoning and decision making (CRDM)

This section describes some of the most common cognitive biases clinicians are susceptible to when using artificial intelligence (AI)-derived information for clinical reasoning and decision making (CRDM).

Interactions between humans and systems can be complex as they involve aspects of human psychology.

These complexities are key to understanding confidence in artificial intelligence (AI)-assisted clinical reasoning and decision making (CRDM), particularly in the context of cognitive biases that can occur when humans use shortcuts to make rapid value judgements. These mental shortcuts are necessary to assimilate and evaluate complex information quickly, as described in 5.1.1.

Interviewees for this research cautioned that, for a new type of information like an AI prediction, mental shortcuts may be inaccurate, and lead to biased value judgments and inappropriate levels of confidence in the information. They stressed that clinicians will need to identify and mitigate these biases during AI-assisted CRDM, as failure to do so may lead to unnecessary clinical risk. This will depend on a fundamental understanding of cognitive biases and how they apply to CRDM (both with and without AI), which is perceived as a gap in current clinical training and education.

This section describes some of the most common cognitive biases clinicians are susceptible to when using AI-derived information for CRDM. It then discusses how clinical settings, choices in workflow integration and how AI presents information can influence how these biases can adversely affect AI-assisted CRDM.

5.3.1 Cognitive biases affecting AI-assisted CRDM

Several cognitive biases are relevant to AI-assisted CRDM, although this list is not exhaustive.

• Automation bias – the tendency to accept the AI recommendation uncritically, potentially due to time pressure, or under-confidence in the clinical task (for example, in non-specialists). A clinician may tend not to critically appraise AI-derived information in the context of the other available evidence, or without consideration of patient-specific factors affecting the model suitability for this case.
• Aversion bias – the tendency to be sceptical of AI, despite strong evidence that its performance at evaluation was good. An experienced clinician may prefer to rely on tried and tested methods, leading to a tendency to dismiss AI predictions as unnecessary for CRDM, regardless of whether these align or misalign with their clinical judgement.
• Alert fatigue – ignoring alerts provided by an AI system due to history or perception of too many incorrect cases (for example, false positives). This can often be the result of an over-conservatively calibrated AI algorithm, or the users’ aversion bias, and occurs often in high volume decision making settings (for example, Accident and Emergency). Ultimately, the pressure on the downstream resources can overwhelm staff and reduce safety for other patients, making alert-fatigue responses to the AI technology more likely.
• Confirmation bias – accepting AI-derived information uncritically when it agrees with the clinician’s intuition, potentially ignoring the evidence suggesting that relying on AI for the specific case should be low. If the AI is operating outside its locus of confidence, this can provide inappropriate reassurance about a decision.
• Rejection bias – rejecting AI recommendations without due consideration when they contradict clinical intuition, and potentially missing the opportunity to critically evaluate whether the AI could have detected something the clinician has not considered.

As noted, interviewees for this research stressed the importance of enabling clinicians to recognise their inherent biases, and to understand how these affect their use of AI-derived information in CRDM, as a key focus of related training and education.

The optimal outcome would be for clinicians to maintain a balanced attitude between enthusiasm for, and healthy scepticism of, AI technologies. Interviewees concluded that the ‘critical eye’ of an experienced clinician is a vital tool in maintaining patient safety when considering AI-derived information, but it must not become so critical that the value of digital transformation through AI is lost.

5.3.2 Clinical settings and cognitive biases

Feedback from the interviews conducted for this research and analysis of related literature suggest that the impact of the biases described in 5.3.1 will depend on the clinical setting and use case.

The nature and extent of the task that AI is assigned within a decision making process affect the level of clinical risk associated with that decision (as discussed also in 5.2.2). This perceived level of risk can affect a clinician’s predisposition towards cognitive biases.¹¹⁸

As described by interviewees for this research, most AI technologies implemented in their settings are limited to decision-support tasks that have low direct clinical risks and maintain human involvement. For example, tools that 'flag’ likely serious cases for prioritised radiology reporting still maintain that all images will be read by a human expert.

Nevertheless, interviewees cautioned that the risk of AI adversely influencing human decision making should not be ignored, even in inherently low-risk scenarios. It is possible that human operators come to rely on the lack of an ‘AI flag’ to indicate low risk, potentially becoming biased towards missing diagnoses when AI flags are absent, particularly when they are under pressure.

Further, in the potential implementation of AI into higher risk CRDM settings, great care will be needed to ensure that the AI is implemented in a way that minimises the risk of cognitive bias for users. This includes users being aware of how aspects of the AI’s implementation can impact their decision making.

Requirements for validating AI, as discussed in section 3.2, can mitigate some of this risk. For example, the external validation requirements of the NHS Breast Screening Programme³³ can provide a high level of confidence in AI in inherently high risk, but tightly controlled settings (such as an AI second reader for mammography screening). Existing human reader workflows are designed to mitigate cognitive biases through independent readers operating in isolation, with defined processes for arbitration. AI-enabled workflows following these same approaches will inherently be more robust to cognitive biases. In such an approach, human readers will be less affected by cognitive biases if they are operating or reaching their own conclusions independently of the AI.¹¹⁹

In less formal or more high-pressure situations such as A&E triage, there tends to be stronger polarisation in inherent clinician attitudes to AI (see section 5.2.1),¹²⁰ with a higher likelihood that clinicians are strongly in favour or strongly averse to its use. This may be due in part to the rapid nature of the CRDM process in these situations, and the limited opportunity to extensively critique AI-derived information.

Furthermore, as conventional CRDM relies heavily on cognitive shortcuts to assimilate complex information quickly, the introduction of AI into this type of CRDM process is more susceptible to the cognitive biases described earlier.

Interviewees for this research supported this view, perceiving that it would be easy to have inappropriately high confidence in AI technologies that provide an immediate output. However, some workers may have inappropriately low confidence in AI predictions, if they feel they don’t understand the technology or aren’t convinced by the robustness or generalisability of the evaluation data. In general, interviewees felt that changing these inherent attitudes will be challenging, except in cases of underperformance and clinical incidents.

5.3.3 AI workflow integration and cognitive biases

Feedback from the interviews conducted for this research and analysis of related literature suggest that the design of the workflow and timing of introduction of AI-derived information are key to mitigating cognitive biases.

Automation bias is more likely when the AI-derived information is presented early in the CRDM pathway, as users may cease searching for potentially contradictory evidence if the prediction appears superficially reasonable.¹¹⁴

Without careful workflow design, there is a notable risk that once an AI technology is implemented, evaluated and in successful routine use, users become increasingly uncritical of the AI outputs, resulting in a higher potential for automation bias and clinical error. ^68,89 This is a known phenomenon with any technology embedded in practice, particularly if failure cases are rare, or unreported. As discussed in 5.2.1, less experienced clinicians (for the task in question) will generally be more reliant on AI predictions, making them more susceptible to automation bias than experts, who are more likely to retain a degree of scepticism and be willing to critique the algorithm.¹²¹

One approach to mitigate automation bias, suggested by interviewees for this research, involves delaying the availability of AI-derived information until an initial human opinion has been formed (and recorded). The user would then be required to record whether they have amended their decision or not, in the light of the AI prediction. This has been described in the literature as the ‘integrative-AI’ approach to clinical decision making.¹²²

In cases where the AI contradicts the initial decision, the clinician is immediately aware of the conflict, whereas with a conventional AI-first approach their own opinion may never have been fully formed. This approach requires a clinician to justify their response to the AI prediction considering their original decision, mitigating the risks of automation and aversion biases.

While suggesting this approach, interviewees recognised that the recording of the initial and final decisions should be as automated and seamless as possible. It should not be overly burdensome to the clinician, who should be focused on the clinical decision, not the record keeping.¹²³ The degree of detail required would be strongly dependent on the clinical scenario.

Research suggests that alert fatigue¹⁰³ can be mitigated by careful calibration of algorithms,¹²⁴ and should be a consideration in the ongoing monitoring of AI-assisted CRDM. The way in which alerts are presented and acknowledged is a key factor in user-interface design for AI systems and can mitigate alert fatigue if done carefully. Designing CRDM systems with prompts and incentives that aim to overcome alert fatigue may be beneficial in these situations.¹²⁵

Confirmation bias becomes a clinical risk when an erroneous human opinion is confirmed by AI, allowing one error to compound another. This has been shown to be more likely when AI-derived information is presented alongside conventional information, allowing the clinician to confirm aspects of the situation and ignore other contradictory factors, which may be critical.¹²² Hence, confirmation bias can be mitigated by presenting the AI prediction later in the pathway, to ensure a thorough human assessment has been performed beforehand, lending further support to the case for integrative-AI workflows.¹²²

Rejection bias is more difficult to mitigate through workflow design, as presenting AI information late in the pathway can increase the likelihood of rejecting it inappropriately.¹²⁶

In order to make AI-assisted CRDM robust against cognitive biases and especially in inherently higher-risk scenarios, interviewees for this research suggested using a decision record system to record and retain the initial clinician assessment, AI prediction and clinician's final decision. This approach is already used in specialties such as radiology, where an initial report is often made and verified by an on-call radiologist in a time-critical scenario, and then a specialist consultant may review and add an addendum, with the initial report retained for reference. 

Interviewees noted that a similar approach with AI-assisted CRDM would provide transparency and encourage careful critical assessment of AI-derived information. The full, recorded, decision process would be available for further review, arbitration by an external specialist and, ultimately, any internal investigations or legal process that may occur in case of a clinically significant error. This approach also affords the possibility of continuous feedback, learning and improvement, both for internal teams and AI developers. When clinical incidents do occur and cause potential or actual patient harm, it has been suggested that cognitive bias assessment should be integrated into the root-cause analysis and reporting for incidents involving AI-assisted CRDM.⁹

The need to record whether a human decision has remained unchanged in the light of contradictory AI information should encourage clinicians to carefully consider their reasons for acceptance or rejection of the AI output and should help to mitigate the risk of the cognitive biases described above.

Finally, this approach would assist in communicating with patients as to how AI has been involved in their care and how the process leading to a clinical decision has happened.

References

¹¹⁸ Kliegr T, Bahník Š, Fürnkranz J. A review of possible effects of cognitive biases on interpretation of rule-based machine learning models. Artif Intell. 2021;295:103458. doi:10.1016/j.artint.2021.103458

³³ Interim guidance on incorporating artificial intelligence into the NHS Breast Screening Programme. Gov.uk. https://www.gov.uk/government/publications/artificial-intelligence-in-the-nhs-breast-screening-programme/interim-guidance-on-incorporating-artificial-intelligence-into-the-nhs-breast-screening-programme. Published 2021. Accessed March 7, 2022.

¹¹⁹ Hickman SE, Baxter GC, Gilbert FJ. Adoption of artificial intelligence in breast imaging: evaluation, ethical constraints and limitations. Br J Cancer. 2021;125(1):15-22. doi:10.1038/s41416-021-01333-w

¹²⁰ Stewart J, Sprivulis P, Dwivedi G. Artificial intelligence and machine learning in emergency medicine. EMA - Emerg Med Australas. 2018;30(6):870-874. doi:10.1111/1742-6723.13145

¹¹⁴ Challen R, Denny J, Pitt M, Gompels L, Edwards T, Tsaneva-Atanasova K. Artificial intelligence, bias and clinical safety. BMJ Qual Saf. 2019;28(3):231-237. doi:10.1136/bmjqs-2018-008370

⁶⁸ Gaube S, Suresh H, Raue M, et al. Do as AI say: susceptibility in deployment of clinical decision-aids. npj Digit Med. 2021;4(1):1-8. doi:10.1038/s41746-021-00385-9

⁸⁹ Buçinca Z, Malaya MB, Gajos KZ. To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. 2021;5(April). doi:10.1145/3449287

¹²¹ Goddard K, Roudsari A, Wyatt JC. Automation bias: A systematic review of frequency, effect mediators, and mitigators. J Am Med Informatics Assoc. 2012;19(1):121-127. doi:10.1136/amiajnl-2011-000089

¹²² Boddard K, Roudsari A, Wyatt JC. Automation bias: A systematic review of frequency, effect mediators, and mitigators. J Am Med Informatics Assoc. 2012;19(1):121-127. doi:10.1136/amiajnl-2011-000089

¹²³ Dymek C, Kim B, Melton GB, Payne TH, Singh H, Hsiao CJ. Building the evidence-base to reduce electronic health record-related clinician burden. J Am Med Inform Assoc. 2021;28(5):1057-1061. doi:10.1093/jamia/ocaa238

¹⁰³ Yu KH, Kohane IS. Framing the challenges of artificial intelligence in medicine. BMJ Qual Saf. 2019;28(3):238-241. doi:10.1136/bmjqs-2018-008551

¹²⁴ Co Z, Holmgren AJ, Classen DC, et al. The tradeoffs between safety and alert fatigue: Data from a national evaluation of hospital medication-related clinical decision support. J Am Med Informatics Assoc. 2020;27(8):1252-1258. doi:10.1093/jamia/ocaa098

¹²⁵ Medlock S, Wyatt JC, Patel VL, Shortliffe EH, Abu-Hanna A. Modeling information flows in clinical decision support: Key insights for enhancing system effectiveness. J Am Med Informatics Assoc. 2016;23(5):1001-1006. doi:10.1093/jamia/ocv177

¹²⁶ Burton JW, Stein MK, Jensen TB. A systematic review of algorithm aversion in augmented decision making. J Behav Decis Mak. 2020;33(2):220-239. doi:10.1002/bdm.2155

⁹ Sinha S, Al Huraimel K. Transforming Healthcare with AI. In: Reimagining Businesses with AI; 2020:33-54. doi:10.1002/9781119709183.ch3