Our roundtable of experts comprised: Robin Matthews, chief technology officer (CTO), PocDoc; Dr Jingqing Zhang, head of AI, Pangaea Data; Dr Rishi Das-Gupta, chief executive, Health Innovation Network (HIN) South London; Dr Jihad Malasi GP, ICB clinical lead in mental health at Kent and Medway ICB and NICE medtech committee member; and Andrew Davies, executive director, digital health, ABHI.

Scaling challenge

Opening the discussion, Das-Gupta said predictive modelling AI algorithms had performed differently when being deployed by HIN South London in areas geographically close together.

‘Some of the algorithms performed well at Guy's and St Thomas' and not very well at Lewisham and Greenwich and they're only about three miles apart,' he noted.

‘Trying to deploy something at scale is challenging.'

Dr Zhang agreed a solely data-driven machine learning approach to build predictive models can perform differently in various locations.

‘We are seeing this in different countries and different backgrounds where data-driven models can perform very differently if they are not appropriately calibrated for local populations,' he said.

Dr Zhang stressed the importance that AI models should follow well defined clinical guidelines, rather than relying solely on patterns learned from data, to improve AI reliability, consistency and clinical relevance across different healthcare settings.

Davies highlighted the need to ensure the validity of inputted data in order to optimise algorithm performance consistency.

System tensions

Davies pointed out there was a tension in the system between the rapid pace of delivery and slower critical evaluation.

‘We're going to need the technology to monitor these technologies,' he observed.

Matthews acknowledged the need for effective governance oversight but stressed the benefits of being able to cheaply and rapidly roll out new technology.

‘I'm optimistic that with the right guardrails, AI can help us innovate faster and help more people,' he noted.

The CTO said AI had fundamentally transformed PocDoc's mission to reduce preventable diseases such as cardiovascular disease, Type 2 diabetes and chronic kidney disease.

‘We've been able to design, build and scale a new category of In Vitro Diagnostic device that enables people at home to test themselves, to understand their risks and be guided towards actions that can help reduce those risks,' he commented.

Das-Gupta welcomed the way the MHRA was approaching regulation to create a dynamic environment to help develop new products that can be scaled across the country.

Ethical questions

Dr Malasi tackled the thorny issue of the ethical adoption of AI, stressing it should adhere to the four medical principles of Childress and Beauchamp: respect for autonomy; nonmaleficence; beneficence; and justice.

He said finding how to have a conversation with patients about how and where their data was being used remained an ‘open question'.

Dr Malasi added: ‘The NHS does not do a good job of explaining issues surrounding data storage and ownership. Ministers often suggest that patient data can and should be used to "unlock" research or other benefits. Patients need to be made aware and consent to these uses. A key example is the Foresight/FDP predictive model which has hit a roadblock with the BMA & RCGP Informatics Committee and which is currently under review by the ICO to my knowledge.'

Davies proposed giving patients incentives to share their data with GPs in order to provide a more holistic view of their health.

The digital health director also highlighted concerns about GPs using ChatGPT, commenting: ‘We need to address how we combine regulated professions with unregulated tools and where responsibility and liability lies.'

Health equity

Matthews warned against the blanket perception that AI technologies increased health equity polarisation, highlighting how PocDoc helped reduce health inequities by taking healthcare out into the community.

Dr Malasi acknowledged the support being provided by some AI tech providers in accessing difficult to reach communities as in the case of PocDoc.

Das-Gupta agreed, commenting: ‘AI is a fantastic tool for coming to people where they are in their lives.'

Trade-offs

Dr Malasi warned trade-offs would be needed in terms of sharing budgets between technologies and medicines and spending on other NHS resources.

‘We will eventually have to displace something because we have finite resources,' he noted. 

‘I think we need to think more and have a clearer discussion about what gets decommissioned or what doesn't.'

Personalised pathways

Das Gupta called for the development of personalised AI pathways that acknowledged cultural differences.

‘On a personal level, often the algorithms will be identifying things that because of my Bengali heritage may not be appropriate in my context,' he said.

Concluding, the HIN chief executive stressed the importance of delivering care in traditional ways and not losing sight of other opportunities presented in public health, including where people live, health literacy and the social determinants of health.

Das-Gupta re-asserted that AI model roll out needed to be tailored to the local population, taking account of risk factors and social deprivation.