Can Agentic AI Strengthen Clinical Safety Monitoring?

Agentic AI in clinical safety monitoring could help clinical teams identify and interpret treatment-related safety signals that remain buried in unstructured medical records.

During MedSafety Week 2026, we examine whether AI-assisted detection can make the review of immune-related adverse events more efficient, consistent, and clinically useful.

Image

Why Adverse Event Detection Needs to Improve

MedSafety Week 2026 runs from 2 to 8 November, with the message “Don’t ignore side effects. Report them.”

The campaign highlights a persistent challenge for medicines safety: Patients and healthcare professionals need to recognise and report suspected side effects, while regulators and researchers need reliable information to understand risks across a medicine’s lifecycle.

That challenge is particularly important in oncology.

Immune checkpoint inhibitors (ICIs) have become an important part of cancer treatment, but they can trigger immune-related adverse events (irAEs).

These reactions occur when treatment activates the immune system in ways that affect healthy tissues and organs. They can involve the skin, thyroid, liver, bowel, lungs, or heart, among other sites.

There are also practical weaknesses in current safety workflows. Clinically important adverse events are often described in free-text progress notes, nursing records, consultation reports, and discharge summaries.

Additionally, electronic health records may not capture these events completely in structured fields.

As a result, research teams often rely on manual chart review to identify what happened, when it happened, how severe it became, and whether clinicians attributed it to treatment.

This process takes time and can produce inconsistent interpretations.

It is hoped that large language model (LLM)s, working alongside human reviewers, can improve the extraction and curation of this information.

Agentic AI Versus Manual Review of irAEs

Published in Nature Cancer in 2026, a research team assessed an agentic AI system designed to extract clinically relevant information from oncology records.

The team developed the system using clinical notes from patients treated with immune checkpoint inhibitors at an academic medical centre in Boston between 2015 and 2024.

Rather than asking one language model to identify an adverse event in a single step, the researchers used a multi-agent architecture.

Six specialised agents worked through different aspects of the task, supported by self-consistency mechanisms intended to improve the reliability of the extracted information.

The system examined six important irAEs of myocarditis, dermatitis thyroiditis, hepatitis, colitis, and pneumonitis.

For each event, the system aimed to extract more than a simple yes-or-no classification. It assessed whether the event was present, when it occurred, its severity, its attribution to immunotherapy, and the certainty of that attribution.

It also highlighted the supporting text in the clinical record for a human reviewer to examine the evidence behind the suggested labels.

This level of detail could make a big difference for safety review.

An historical episode of pneumonitis, for example, has a different clinical meaning from a new episode emerging during treatment. Similarly, identifying a possible adverse event is not the same as establishing its severity or determining whether clinicians consider it treatment-related.

The researcher team then evaluated the system through three stages, moving from retrospective testing to prospective deployment and then a controlled assessment of human-AI collaboration.

Retrospective development and testing

The initial development process used 289 expert-annotated clinical notes. The researchers optimised the system using a subset of 26 notes and evaluated its performance on a held-out set of 263 notes.

The best-performing configuration achieved a macro-averaged F1 score of 0.92 for detecting current adverse events. Its performance for multi-class severity grading was lower, at 0.66. Attribution, assessed without separating events by temporality, reached an F1 score of 0.94.

These results show why the different extraction tasks should not be treated as interchangeable. The system was strong at identifying whether a current event appeared in a note, but assigning the correct severity category presented a more difficult problem.

Prospective deployment in clinical records

Retrospective performance does not necessarily predict how a system will behave in a live clinical environment. Documentation styles change, patient populations differ, and clinical notes rarely follow a perfectly consistent structure.

The team therefore moved the system into a prospective, silent deployment lasting three months, from May to July 2025.

During this time it processed 884 clinical notes in near-real time without influencing patient care.

The system generated 31,824 label adjudications covering severity, attribution, and certainty across current and historical time points.

Its binary F1 performance ranged from 0.72 to 0.79, depending on the label and timeframe, while attribution extraction achieved an F1 score of 0.77, while certainty classification reached 0.80.

Performance was lower than in the retrospective evaluation, but the system retained useful detection capability under operational conditions. This decline is important because it illustrates the difference between testing against a prepared dataset and handling the changing language and complexity of real clinical documentation.

Image

Testing the human-AI workflow

The final stage of assessment examined whether AI assistance actually improved the work performed by clinical research staff.

The researchers conducted a randomised crossover study involving 17 participants who completed both manual and AI-assisted annotation tasks.

The results were encouraging. AI assistance reduced annotation time by an average of 2.71 minutes per note.

The researchers also reported a 45% increase in the odds of complete-match accuracy, with an odds ratio of 1.45 (95% confidence interval: 1.01–2.08; P-value = 0.045).

Perhaps the most striking finding concerned consistency between reviewers. Without AI assistance, interannotator agreement for current severity grades was relatively low, with Krippendorff’s alpha ranging from 0.22 to 0.51 across the two note sets. With AI assistance, agreement increased to between 0.82 and 0.85.

In the post-study survey, 15 of 17 participants (88%) preferred the AI-assisted workflow.

Participants also rated the assisted process more favourably for efficiency and confidence in their assigned labels.

The findings suggest that the system did more than accelerate the extraction of information. It helped reviewers apply a more consistent interpretation to complex clinical narratives.

However, the distinction between assistance and independent clinical judgement remains essential.

The researchers evaluated a workflow in which people reviewed AI-generated suggestions. They did not demonstrate that an autonomous system could safely make final clinical decisions or replace established safety reporting responsibilities.

View interview How Should Pharma Companies Audit AI Vendors?

How Should Pharma Companies Audit AI Vendors?

“This is one of the riskiest industries there is because you're dealing with human lives, you're dealing with experimental protocols, and you're dealing with regulatory bodies where you might not get another shot at that clinical trial."

Image

What the Results Mean for Clinical Development and Drug Safety

The results show an opportunity for an AI agent to improve how sponsors, clinical research teams, and healthcare organisations convert routine clinical documentation into structured safety information.

For Clinical Development specifically, the value lies in making adverse event information more accessible and consistent.

Clinical trial teams need to understand not only which events occur, but also their timing, severity, clinical context, and potential relationship to treatment.

When those details remain buried in free-text records, retrospective review becomes labour-intensive and the resulting datasets may be incomplete or inconsistent.

An agentic system could help reduce that burden by identifying relevant data and presenting structured information for human assessment.

This could support more efficient safety review, trial data curation, and the development of real-world evidence.

It could also help research teams examine how adverse events evolve over time. For example, a system that distinguishes active from historical events may help reviewers understand whether a toxicity is emerging, recurring, or resolving.

That information could be useful when assessing treatment exposure, documenting clinical outcomes, and examining patterns across patient populations.

However, better extraction is not the same as better pharmacovigilance by itself.

This is important in MedSafety Week. The World Health Organisation campaign encourages patients, caregivers, and healthcare professionals to recognise and report suspected side effects through national reporting systems.

Its purpose is to improve the information available for ongoing medicines safety monitoring.

The Nature Cancer study investigates how AI can help extract and organise adverse event information already recorded in clinical documentation.

Both activities contribute to a stronger safety ecosystem, but they serve different functions.

An AI system that identifies a possible adverse event in a clinical note does not automatically submit a valid individual case safety report, establish causality, or fulfil a sponsor’s regulatory reporting obligations.

For clinical trial sponsors, those responsibilities remain governed by established requirements.

The U.S. Food and Drug Administration (FDA) guidance on sponsor safety reporting explains expectations for reviewing safety information and reporting and qualifying suspected adverse reactions.

Further on the Regulatory side, the ICH E2A framework distinguishes seriousness from severity, a distinction that is important when classifying and escalating safety events.

This is particularly relevant to AI-assisted event grading.

A severe event is not necessarily a serious adverse event under regulatory definitions. Seriousness depends on outcomes and criteria such as death, life-threatening events, hospitalisation, or significant disability.

A model that confuses these concepts could create errors in downstream safety workflows.

Sponsors and those others responsible for pharmacovigilance activities should therefore assess where AI-assisted extraction fits within their existing safety processes.

The system may help identify records for review, organise evidence, and reduce repetitive annotation. It should not become an unexamined decision-maker in processes that require clinical interpretation, regulatory judgement, and documented accountability.

The potential value is also relevant to sponsors working with Contract Research Organisations and external data managers. If AI-assisted review becomes part of a trial workflow, responsibilities for system oversight, access to records, quality checks, and escalation need to be clear.

Pharmatica has explored related governance questions in our Insights into LLMs in clinical trial screening and CRO and vendor oversight.

View interview AI in Pharma: Why the Future of Healthcare Starts With Patients, Not Tech

AI in Pharma: Why the Future of Healthcare Starts With Patients, Not Tech

Kate O’Reilly, President & Chair of the Healthcare Businesswomen’s Association (HBA) Dublin-Ireland Chapter & Healthcare Transformation Partner at Roche, discusses AI in pharma, patient engagement, and the future of healthcare innovation.

Image

The Limitations in AI-Assisted Safety Monitoring

The Nature Cancer study provides a good example of what agentic AI might contribute to clinical safety workflows. However, large challenges and limitations exist.

First, the research study took place within a single academic health system.

In practice, clinical language, documentation practices, electronic health record configurations, and oncology workflows vary between institutions. A model that performs well in one environment may not achieve the same results elsewhere.

Therefore, external validation across different hospitals, health systems, and oncology settings is therefore essential before organisations try a similar approach at scale.

Second, the research focused on only six irAEs, and not on injection site reaction, cytokine release storm, and ICANS.

While the reviewed six AEs represent important toxicities associated with immune checkpoint inhibitors, they do not cover the full range of adverse events encountered in oncology or clinical development.

Expanding the system to other toxicities would require further evaluation, particularly where symptoms are ambiguous, multiple conditions overlap, or the clinical record contains incomplete information.

Thirdly, the primary system used a proprietary model, GPT-4.1-mini. Performance may differ with other commercial or open-source models.

Organisations considering deployment would need to assess model selection, reproducibility, data protection, system access, and the implications of future model updates.

The prospective findings also deserve attention. The binary F1 scores of 0.72–0.79 were lower than the retrospective result for current-event detection. That gap shows why a strong retrospective benchmark cannot serve as the only evidence of readiness.

Clinical language changes, patient populations evolve, and new documentation patterns can affect performance. Therefore, continuous monitoring is necessary to identify deterioration and determine when a model requires reassessment.

There is another important limitation in that the study did not establish that AI-assisted extraction improves patient outcomes.

It measured detection performance, annotation efficiency, accuracy, and consistency. It did not demonstrate that the system shortened the time to treatment for an adverse event, reduced hospitalisations, prevented treatment interruptions, or improved patient safety.

An efficient workflow is valuable, but it is not automatically a clinical benefit.

The next stage of research must establish whether more consistent adverse event information leads to better decisions and measurable improvements in patient care.

AI Governance Must Extend Beyond Model Accuracy for Drug Safety Review

Sponsors and teams responsible for drug safety review should also examine the risks created when AI-generated labels influence human review.

Reviewers may accept model suggestions too readily, particularly when systems present outputs confidently or when staff face pressure to process large volumes of records.

There is the real possibility that AI assistance can reduce differences between annotators partly because reviewers anchor on the model’s suggestions.

Greater agreement is useful, but agreement alone does not prove that every label is correct. AI assistance can mean that a shared error can become more consistent without becoming more accurate.

A responsible implementation should therefore include:

• Evidence traceability: Reviewers should be able to inspect the clinical text supporting each suggested event, grade, and attribution.
• Human review and escalation: Uncertain, severe, or potentially serious events should be directed to appropriately qualified staff.
• Performance monitoring: Organisations should assess false positives, missed events, grading errors, and changes in performance over time.
• Data protection: Patient records contain sensitive information, so access, processing, retention, and model-provider arrangements require careful control.
• Clear accountability: Sponsors and healthcare organisations must define who approves safety decisions, manages system changes and responds when the AI produces an incorrect result.

The study’s reported computational cost of approximately US$0.02 per note for the best-performing configuration is encouraging, but it should not be mistaken for the full cost of deployment.

Integration, security, quality assurance, staff training, governance, and ongoing monitoring all contribute to the operational investment.

The study authors also made their custom code and prompts available through the AEGIS GitHub repository. This provides an opportunity for further technical examination and independent research, although access to the underlying patient-level clinical notes remains restricted to protect privacy.

For sponsors, the commercial question is not simply whether an AI system can review notes more cheaply, but whether the system can be integrated into an accountable process that improves the quality and usability of safety information without introducing new risks.

Image

From Better Safety Detection to Better Drug-Safety Decisions

The wider opportunity of AI integration in drug safety is to make safety information more useful throughout clinical development.

Agentic AI in pharmacovigilance could help teams identify clinically relevant information earlier, structure complex records, and direct human attention towards events that need further assessment.

However, no form of AI can replace pharmacovigilance professionals, clinical investigators, or regulatory reporting processes.

An agentic system, combined with human verification, may be able to improve the efficiency and consistency of adverse event curation.

The next priority should be to test whether these workflow improvements translate into better safety decisions. That will require independent validation, clear accountability, robust data governance, and studies that measure clinical outcomes as well as operational performance.

This is the central message of MedSafety Week. Better medicines safety depends on information being recognised, recorded, assessed, and shared. AI may strengthen parts of that process, but it cannot replace the responsibility of patients, healthcare professionals, investigators, and sponsors to report and act on suspected adverse events.

At Pharmatica, we provide Insights into the evidence and Clinical Development systems contributing to pharmaceutical development. Our Insights connect advances in clinical research with the practical decisions that affect patient safety, trial quality, and the future of medicines.

Pharmatica: Insight. Connection. Impact.