Governing AI in Healthcare: what´s the value of checklists?
In clinical environments where every decision can affect a patient’s life, governing the use of artificial intelligence (AI) is not just a regulatory matter — it’s a matter of clinical trust, patient safety, and ethical responsibility. With the EU Artificial Intelligence Act in force [1], tools like the checklist-based governance methodology proposed by Bignami et al [2] are due to become commonplace in hospital boards and risk committees. Checklists are attractive governance tools: they translate regulatory principles into discrete questions, support documentation, and help multidisciplinary teams track readiness. Yet in healthcare — where data is messy, patient populations vary, and clinical workflows are complex — we also see situations where traditional checklists struggle to capture what matters most. As we witness the rapid deployment of AI in all realms of society, it’s worth asking: Do checklists help us govern AI well — or do they risk creating a false sense of complacency?
AI errors, bias and liability challenges
Risks of AI are well documented. For example, a high-profile research paper introducing Google’s Med-Gemini model in radiology mistakenly referred to a non-existent brain structure, highlighting how even well-resourced systems can generate hallucinations that lapse past clinical reviewers unnoticed. Real-world error modes like this underscore the need for dynamic model validation, real-time monitoring, and robust human oversight — far beyond ticking a box on a checklist [3].
Also well-established is that AI models trained on non-representative data risk encoding and amplifying healthcare disparities. Survey studies in AI healthcare research show that predictive tools can perform unequally across demographic groups, raising concerns about fairness in diagnostics and treatment recommendations. As we’ve learned from bias research in clinical AI, diverse data and fairness auditing are essential governance components [4].
Finally, experts have warned that as clinicians increasingly rely on algorithmic recommendations, establishing accountability in cases of harm becomes legally complex. Patients and providers alike may struggle to identify whether clinicians, developers, or organisations should bear responsibility — a challenge that checklists alone cannot resolve without clear institutional escalation protocols and ethical frameworks [5]. These examples remind us that clinical AI governance must be both rigorous and context-aware — not only procedural.
A proposed checklist
To address the challenges above, Bignami et al designed a checklist as an operational tool to help healthcare organisations systematically assess and govern the adoption and use of AI systems — particularly in clinical environments such as anaesthesia and intensive care units (ICUs) [2]. Its purpose is to support responsible, safe, ethical and legally compliant AI integration, in line with regulatory obligations under the EU Artificial Intelligence Act. Importantly, it is question-based rather than prescriptive, helping professionals ask the right questions rather than simply tick boxes.
The checklist is composed of two main domains: “Clinical and Technical Validation” and “Governance and Compliance”. Each domain focuses on different aspects of trustworthy AI adoption
The clinical and technical validation domain evaluates the performance, robustness, safety and applicability of AI systems in the specific clinical context where they will be used. Key elements include validating real-world clinical performance, regulatory and GDPR compliance, representative data, robust post-deployment monitoring, and adequate training of clinicians and staff to ensure safe use and oversight.
The governance and compliance domain focuses on organisational structures, processes, and legal responsibilities that ensure AI is used ethically, traceably, and in compliance with law and internal policy. Key elements include risk-based legal compliance, clear accountability for AI oversight and clinical use, robust traceability and auditability, defined human-in-the-loop procedures, role-specific staff training, and regular auditing and updates to keep governance aligned with evidence and regulation.
Do checklists bring value to healthcare and clinical practice?
Checklists may add value in healthcare AI governance by bringing structure to multidisciplinary review, helping clinicians, data scientists, IT, legal, and quality teams align around shared validation steps, documentation standards, and risk assessments. They also support audit trails and compliance by providing a clear, traceable account of testing, validation, monitoring, responsibilities, and corrective actions when reviewed by regulators or internal auditors. In addition, checklists encourage lifecycle thinking, reinforcing that AI governance is a continuous process spanning vendor selection, deployment, and ongoing performance monitoring and drift detection, rather than a one-off approval step [6].
Despite their utility, checklists often struggle with the qualitative dimensions of clinical AI governance:
- Nuance and clinical context: A checklist item such as “Has model performance been validated?” does not capture how performance varies by sub-population, clinical setting, or data quality — issues that directly affect patient outcomes.
- Trust and clinician acceptance: Research shows clinicians don’t uniformly “accept or reject” AI recommendations; instead, they negotiate trust in what the AI suggests, adjusting advice based on clinical judgment. This kind of adaptation is not easily captured in checklist responses [7].
- Sociotechnical factors: Patient perception, empathy, workflow integration, and clinician training are all vital for AI to function safely in practice. These variables are hard to codify in binary governance items but are essential in ethical, human-centred AI.
Toward Richer Governance Practices
AI governance in healthcare is positioned at the intersection of technology, medicine, and human judgement. Checklists help us organise conversation and document evidence. Checklists provide a structural baseline, but meaningful and safe AI adoption depends on governance that is grounded in real clinical practice and day-to-day use. Only when paired with active, context-sensitive governance cultures — where clinicians, data scientists, and ethicists continually reflect on outcomes — can we truly steward AI in ways that enhance safety, equity, and trust [8]. Healthcare organisations thus need to consider:
- Multidisciplinary AI governance boards with clinical, ethical, legal, and patient perspectives.
- Scenario-based validation that asks how AI performs under different patient conditions or edge cases.
- Continuous learning loops where post-market monitoring informs updates to governance tools.
- Ethics and equity audits that examine how AI impacts different patient populations.
This means moving beyond “checklists as compliance” toward checklists as catalysts for organisational learning and ethical alignment.
In healthcare, the stakes are high. A box checked does not always mean a patient safe. But a culture that scrutinises AI at every turn just might.
References
- Artificial Intelligence Act. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act), 13 June 2024, http://data.europa.eu/eli/reg/2024/1689/oj
- Bignami E, Darhour LJ, Franco G, Guarnieri M, Bellini V. AI policy in healthcare: a checklist-based methodology for structured implementation. J Anesth Analg Crit Care. 2025 Sep 25;5(1):56. doi: 10.1186/s44158-025-00278-3.
- Chinta SV, Wang Z, Palikhe A. AI-driven healthcare: a review on ensuring fairness and mitigating bias. arXiv:2407.19655, https://arxiv.org/abs/2407.19655
- The Verge, 4.8.2025. Google´s healthcare AI made up a body part. https://www.theverge.com/health/718049/google-med-gemini-basilar-ganglia-paper-typo-hallucination
- Angus DC, Khera R, Lieu T, et al. AI, Health, and Health Care Today and Tomorrow: The JAMA Summit Report on Artificial Intelligence. JAMA. 2025;334(18):1650–1664. doi:10.1001/jama.2025.18490
- Wells BJ, Nguyen HM, McWilliams A et al. A practical framework for appropriate implementation and review of artificial intelligence (FAIR-AI) in healthcare. npj Digit.Med. 8**, 514 (2025). https://doi.org/10.1038/s41746-025-01900-y
- Sivaraman V, Bokuwski LA, Levin J et al. Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care. arXiv:2302.00096, https://arxiv.org/abs/2302.00096
- Kim JY, Hasan A, Kueper J et al Establishing organizational AI governance in healthcare: a case study in Canada. NPJ Digit Med. 2025 Aug 15;8(1):522. doi: 10.1038/s41746-025-01909-3
