In patient care, the consequences of getting AI wrong are easy to understand. When a diagnostic tool makes an incorrect recommendation, the risk is immediate and identifiable.
In health research, the consequences can be less visible, but potentially more significant.
Imagine AI being used to analyse clinical trial data and establish if a new cancer treatment is safe. An error could influence the evidence submitted for regulatory approval, with consequences felt for years. We have not yet had a robust conversation about how much scrutiny each stage of research requires.
Research needs its own AI governance conversation
AI governance in healthcare has focused mainly on direct care, particularly diagnostic tools and clinical decision support. The recently published National Commission into the Regulation of AI in Healthcare's recommendations make almost no reference to health data research, beyond guidance on when evaluating a product requires formal research approval.
Capabilities needed to assure AI in research are broadly similar to those in care delivery. Researchers need to understand how models have been developed, demonstrate their efficacy, identify bias and maintain traceability.
The difficulty is the breadth of what we mean by research. At one end, an academic researcher might use AI to develop an early hypothesis. At the other, a pharmaceutical company might use AI to draft a clinical study report; key documentation required to bring a new drug to market that often represents years of work and billions of pounds of investment.
Increasingly, clinical trials are evaluating interventions that combine a drug with a digital application. But what happens when an AI-enabled digital intervention becomes part of the treatment being evaluated?
As research programmes progress towards a full clinical roll out, the requirements for assurance should become more demanding. That requires a common methodology for assessing risk, regardless of whether the AI is developed by an academic institution, pharmaceutical company or commercial supplier.
The technology has moved beyond point solutions
AI is moving towards integrated AI capabilities that operate across multiple stages of research.
Traditional governance often assumes a technology can be evaluated against a defined purpose and that it will behave consistently. This doesn't hold for AI because the real world rarely resembles the environment a model was trained in.
Models can perform differently across populations; their performance can deteriorate and changes to underlying data can affect outputs in subtle ways.
A national conversation about assurance
We need to move beyond assurance as a one-off approval exercise towards ongoing monitoring, traceability and reassessment; but it's currently unclear who is responsible for establishing a consistent approach across the health research ecosystem.
There is an argument for something akin to the independent standards and assurance associated with national statistics, providing a common basis for confidence in research involving AI.
That does not require another regulator, but instead a shared expectation of the evidence showing a model is appropriate for its intended use.
There's an opportunity for the Health Data Research Service (HDRS) to convene that conversation, drawing on the expertise of existing research organisations, regulators, secure data environments and technology providers.
A minimum viable national AI assurance environment could be a useful starting point to establish common approaches to model evaluation, provenance, performance monitoring and accountability. Crucially, the same underlying methodology should apply when AI tooling is used within secure NHS research environments or brought into the research process by external organisations.
Common standards could enable researchers to bring specialist AI tools into secure data environments without compromising control over sensitive information.
Finding the middle ground
We cannot stop the advancement of AI, nor should we want to prevent organisations from exploring its huge potential for accelerating research.
We risk polarising the debate between early adopters who move before governance is fully developed and those who wait for someone else to prove the technology is safe.
Health research has room for both experimentation and rigorous assurance, provided the expectations are proportionate to the consequences.
Academic research environments and innovation sandboxes can support earlier experimentation. But once it involves clinical trial participants or generates evidence for clinical decisions, scrutiny should increase.
HDRS could play an important role in championing useful applications already demonstrating value, while bringing the research community together to develop the assurance capabilities needed for more consequential uses.
The opportunity is not simply to make research faster. It is to establish a system in which researchers can innovate with confidence, ensure evidence remains trustworthy and the benefits of new discoveries reach patients sooner.
