How Should Pharma Companies Audit AI Vendors?

The use of AI in clinical trials is now progressing from a stage of experimentation to one of execution. However, that doesn’t mean most AI platforms are suitable for regulated research. It's increasingly become imperative for pharma enterprises to audit AI vendors effectively. 

In the recent episode of The Clinical Compass podcast, host Shubhangi Dua, Podcast Producer and B2B Journalist, is joined by Dr Amber Hill, Founder and CEO of Research Grid. They talk about how pharmaceutical companies and their leaders should approach the use of AI in clinical trials. Particularly, Hill points out that pharmaceutical companies should focus on AI governance, auditability and AI-native architecture rather than on flashy demonstrations or on promises of automation.

Some trends pharma companies seem to be falling for are the speed of AI adoption. Especially vendors are “attempting to adopt AI before being abreast of the technology completely and its various types of models they should be using,” states Hill.

She further says that pharmaceutical companies need to know which AI models are viable for a use case, how to train AI on the right data sets and correct information in addition to ensuring those AI models are traceable. 

AI is rapidly being adopted without adequate parameters. For instance, healthcare-grade artificial intelligence (AI) and regulatory-grade artificial intelligence (AI) should be specialised AI models whose architecture is purpose-built for clinical trials and fully traceable, but some vendors with black-box AI models are using these titles without adequate checks of their architecture. Models need to be built in a way that regulatory bodies like the FDA or the MHRA can audit them to ensure a secure trail of how the model reaches its outputs. 

The AI boom isn't the problem. 

The problem is that many vendors are marketing AI before they've built systems that meet the standards required for regulated clinical research.

Given that pharmaceutical companies worldwide are spending billions of dollars on digital transformation, the next source of competitive advantage might not lie in using more AI, but in selecting the right AI.

Also Watch: AI in Pharma: Hype vs. What Actually Works

View interview AI in Pharma: Why the Future of Healthcare Starts With Patients, Not Tech

AI in Pharma: Why the Future of Healthcare Starts With Patients, Not Tech

Kate O’Reilly, President & Chair of the Healthcare Businesswomen’s Association (HBA) Dublin-Ireland Chapter & Healthcare Transformation Partner at Roche, discusses AI in pharma, patient engagement, and the future of healthcare innovation.

Image

What is AI-Native & How is It Relevant to Clinical Trials?

What does it mean to have AI-native AI and why is this relevant to clinical trials? This is the key question, Hill believes, every pharma company needs to answer before laying out the AI adoption plan. 

Unlike AI wrappers, which are built on top of third-party large language AI models or other models, AI-native platforms are specifically designed, engineered and audited for the clinical trial workflow and bottlenecks they are intending to automate or resolve.

“It's not retrofitted, and the AI has learned from a model that has been built in-house for that purpose,” Hill tells Dua, alluding to AI-native models.

She explains that the main thing to consider is the AI architecture. Pharma leaders need to look into how AI models are being built, who is responsible for their creation, and whether or not the system is designed and engineered from the start to be AI-native.

Model ownership and production are another important thing to check off the list. Some questions pharma companies need to ask are: “Who has made that model? Is it a third-party consultancy company? And does that vendor actually own that model? Has it been designed and engineered in-house from the ground up?”

They need to check if the AI model has been engineered on top of open-source infrastructure like OpenAI or AWS or Anthropic or any of the other providers.

For pharmaceutical companies, that distinction has important implications. 

How Should AI Vendors be Assessed in Pharma?

A platform designed from the ground up provides better visibility regarding model ownership, cybersecurity, infrastructure, and audit trails. These features are essential for regulated clinical trials. 

On the other hand, a great many vendors develop their products using existing 'black box' models that they do not own and about which they have only a partial understanding.

Hill says that if anything went wrong, they would have no way of knowing what had happened without getting in touch with OpenAI or Google or another owner of the original ‘black-box’AI model, who will be unlikely to backtrack the issue without Intellectual Property conflicts of interest. 

“When you're thinking about traceability, you need to keep in mind if something were to go wrong, could that engineering team backtrack what's gone wrong and fix it? Or do they not own the infrastructure in its entirety to actually go back far enough to fix the problem?” 

These questions are imperative to aid pharma companies in auditing a vendor that is fundamentally AI.

Overall, Hill recommends that every AI vendor should be assessed against four core areas:

• AI architecture and engineering
• Model ownership and intellectual property
• Data governance and infrastructure
• Traceability and auditability

In sectors that are subject to regulation, such as the pharmaceutical industry, these factors are becoming just as important as functionality or cost.

Is Agentic AI Ready For Use in Clinical Research?

The question on everyone’s mind right now is if agentic AI is ready for use in clinical research. With the recent news on OpenAI’s rogue AI agents hacking into other legitimate companies' systems, such as the AI firm Hugging Face has caused the markets have been on the fence.

Agentic AI has become one of the most talked-about subjects in enterprise technology. Hill believes the industry is going too far ahead of itself. She states that agentic AI is not yet suitable for our industry since it is difficult to audit and because it has not been designed or engineered with safety and traceability in mind.

Since many agentic systems are based on generative AI and third-party foundation models, they are capable of hallucinating, of producing unpredictable outputs, and of lacking full traceability.

That presents an unacceptable risk in clinical trials.

Hill cautions that you are working with human lives and the fact that, in regulatory agencies, you may not get a second chance at that clinical trial.

“This is one of the riskiest industries there is because you're dealing with human lives, you're dealing with experimental protocols, and you're dealing with regulatory bodies where you might not get another shot at that clinical trial,” emphasises the CEO.

Rather, she thinks that AI provides the most value when it comes to solving clearly defined operational bottlenecks rather than trying to take the place of scientific decision-making.

A few examples at Research Grid are the use of specialised AI models to speed up patient recruitment, making site selection easier, or automation of paperwork to cut down on admin work.

"AI is very good at recognising patterns," Hill says. "But it has to be combined with the appropriate pain point that it is intended to address."

AI is a tool, though it does come with risks, and it is necessary to understand those risks at their core. With the speed of AI adoption in the life sciences increasing, companies which focus on AI governance, AI-native infrastructure and clinical AI that is ready for regulatory approval are likely to be in a better position. 

Also Watch: AI in Pharma: Why the Future of Healthcare Starts With Patients, Not Tech

Takeaways

• AI-native platforms are better suited for regulated clinical trials than retrofitted AI models.
• Pharma leaders should prioritise AI governance, auditability and data ownership before investing.
• Agentic AI is not yet ready for high-risk clinical research due to traceability and safety concerns.
• Purpose-built AI can accelerate patient recruitment, documentation and clinical trial operations.
• Understanding AI architecture is becoming essential for successful pharmaceutical AI adoption.

Chapters

• 00:00 Introduction to AI in Clinical Research
• 02:36 Challenges in AI Adoption for Clinical Trials
• 05:20 Auditing AI Models for Clinical Research
• 11:02 Complexity of AI in Clinical Trials
• 14:38 Industry Missteps in AI Implementation
• 17:28 Understanding Agentic AI
• 22:45 Evaluating AI Solutions for Pharma
• 27:46 Key Takeaways for Pharma Leaders

To learn more about Research Grid's AI-native clinical trial automation platform and its solutions for patient recruitment, trial operations and back-office automation, visit Research Grid.