bims-arines Biomed News
on AI in evidence synthesis
Issue of 2026–10–11
fourteen papers selected by
Farhad Shokraneh, Systematic Review Consultants LTD



  1. Res Synth Methods. 2026 Oct 08. 1-25
      Evidence synthesis (ES) is costly and time-intensive. Artificial intelligence (AI) tools are increasingly used to automate or semi-automate ES tasks. We conducted a living rapid review of AI tools that automate or semi-automate ES tasks. We searched for published articles and preprints through MEDLINE, Embase, and the Association for Computing Machinery Digital Library from January 1, 2021 through October 15, 2025, supplemented by hand-searching. We included existing reviews to capture evaluations before 2021. Screening was conducted in duplicate at the title/abstract level, with single full-text screening and data extraction verified by a second reviewer. We included 134 studies. Performance varied by task and evaluation approach. Search automation tools showed poor recall (median 17%), although human-in-the-loop approaches improved performance. Tools for identifying randomized controlled trials demonstrated high median recall (98%) and precision (92%). Semi-automated abstract screening tools using active learning reduced screening burden by approximately 50% while maintaining 95% recall. Large language models demonstrated variable accuracy for abstract screening (median 88% recall), full-text screening (median 96% recall), and data extraction (median 63% accuracy). Risk of bias (RoB) assessment tools showed moderate agreement with human reviewers for the Cochrane RoB tool version 1 (median 71%) but performed less well with other RoB tools. AI tools can meaningfully reduce burden in study design identification and abstract screening but perform inconsistently across other evidence synthesis tasks. Tools for search automation, data extraction, and RoB assessment require further development. No existing tool can currently replace human expertise.
    Keywords:  Systematic review methods; artificial intelligence; automation; evidence synthesis
    DOI:  https://doi.org/10.1017/rsm.2026.10118
  2. medRxiv. 2026 Aug 14. pii: 2026.08.12.26360335. [Epub ahead of print]
       IMPORTANCE: Systematic reviews and meta-analyses inform suicide-prevention policy and practice, but broad database searches are difficult to screen manually. This limits capture of upstream interventions, such as economic policies, with indirect effects on suicide. Reliable automated screening could make broader and more comprehensive evidence syntheses feasible.
    OBJECTIVE: To develop and validate ScreenAgent, a large language model (LLM) agent for title and abstract screening, and a review-specific method for prospectively estimating screening performance.
    DESIGN SETTING AND PARTICIPANTS: ScreenAgent was validated internally on a prospective meta-analysis, and externally on two published systematic reviews. The correct include and exclude decisions followed standard systematic-review screening methodology.
    EXPOSURES: ScreenAgent, an LLM agent returning structured include-or-exclude decisions. Records it marked for inclusion were re-checked by a second, cascade pass using a higher-effort LLM. For the external reviews, the agent's prompt was tuned automatically on a small set of labeled examples.
    MAIN OUTCOMES AND MEASURES: We calculated sensitivity, specificity, workload reduction (the percentage of records removed from human review), and agent-versus-human reliability via Cohen kappa. Sensitivity was estimated by direct comparison (internal) and 5-fold cross-validation (external).
    RESULTS: In the internal validation, ScreenAgent identified 43 of 44 eligible studies (sensitivity 97.7%; 95% CI, 88.2%-99.6%) with a generic prompt applied without any review-specific optimization, specificity 98.0%, and a measured full-corpus workload reduction of 99.4%. The cost was $855.91 for the full 201,064-record corpus (0.43 US cents per record). Agent-versus-human-consensus agreement exceeded human-versus-human agreement (Cohen kappa 0.75 vs 0.64; percent agreement 97.3% vs 95.4%). For two external validation studies, automatic tuning resulted in a cross-validated sensitivity of 95.9% (95% CI, 90.0%-98.4%) and 97.4% (90.9%-99.3%), with workload reductions of 97.4% and 98.4%.
    CONCLUSIONS AND RELEVANCE: Suicide prevention efforts often require rapid consolidation of evidence because of the inherent challenges of single studies trying to prevent rare outcomes. On both internal and external validation sets, ScreenAgent identified nearly all eligible studies with human-level reliability for a fraction of a US cent per record while keeping human reviewers as the final arbiters. By making broad searches feasible and screening performance measurable beforehand, this approach can serve as a transparent methodology to strengthen the speed at which we can inform and advance suicide prevention efforts.
    Key point: Question: Can an LLM agent screen records accurately to make broad-scale meta-analyses in suicide prevention more feasible?Findings: Across a prospective review and two published reviews, ScreenAgent identified nearly all eligible studies (internal sensitivity 97.7%; external 95.9/97.4%) with human-level reliability (Cohen kappa 0.75 agent-vs-consensus, vs 0.64 between humans) and 98.0% specificity.Meaning: Systematic reviews and meta-analyses are critical for evidence synthesis but are time and labor intensive. An LLM screening agent identified nearly all eligible studies with human-level reliability while reducing human workload by 99% at a fraction of a cent per record, which makes large-scale syntheses tractable.
    DOI:  https://doi.org/10.64898/2026.08.12.26360335
  3. J Evid Based Med. 2026 Oct 06. e70193
    ADVANCED Working Group
       AIM: The proliferation of clinical practice guidelines presents significant challenges for ensuring evidence-based care, since evaluating their trustworthiness with the Reporting Items for practice Guidelines in HealThcare (RIGHT) criteria is both time-consuming and resource-intensive. To address this, the Artificial intelligence-empowered DeVelopment and AssessmeNt of healthCare guidElines and stanDards (ADVANCED) working group developed ad^Writer, an artificial intelligence system, designed to automate guideline reporting quality assessment. This study aimed to determine the agreement and reliability of ad^Writer compared with human evaluations.
    METHODS: Guidelines were identified from four published systematic reviews, and through direct communication with the original authors, 60 were ultimately included. These guidelines were evaluated using ad^Writer with two different large language models, each performing three independent assessments. A majority rule was applied to synthesize results. Results from systematic reviews, which served as the reference standard. Agreement, consistency, and stability of the artificial intelligence-based assessments were examined using Bland-Altman plots, accuracy measures, and intraclass correlation coefficient (ICC), with a focus on identifying patterns of performance across individual RIGHT items.
    RESULTS: Both models showed good agreement with human but had wide limits of agreement. DeepSeek performed better (mean difference = -1.40, standard deviation = 5.00, 95% limits of agreement ([-11.22, 8.42]) and had higher accuracy scores. Both achieved ≥ 75% accuracy on 23 items (65.71%), however, the results varied. ChatGPT showed higher consistency (ICC = 0.93) compared to DeepSeek (ICC = 0.88).
    CONCLUSIONS: The ad^Writer system leverages large language models to demonstrate the potential for rapid, accurate, and reproducible guideline quality assessments based on RIGHT. By serving as a supportive tool for manual evaluation, this approach enhances the efficiency of the guideline appraisal process, saving time and human resources for stakeholders.
    Keywords:  RIGHT; artificial intelligence; clinical practice guidelines; reporting quality
    DOI:  https://doi.org/10.1111/jebm.70193
  4. J Med Internet Res. 2026 Oct 09. 28 e94809
       BACKGROUND: Limited recent literature has evaluated the use of large language models (LLMs) in the qualitative analysis of health data; more research is needed to expand the generalizability of LLM use and to evaluate potential ethical considerations.
    OBJECTIVE: Our research sought to (1) describe the process of using AI to analyze qualitative in-depth interview data, (2) identify similarities and differences between the human and AI-generated analyses to compare the quality and rigor of the two techniques and describe the strengths and weaknesses of each approach, and (3) make recommendations regarding the bounds of ethics and the role of researcher bias in AI-assisted qualitative research.
    METHODS: Nested within a larger mixed methods study, 17 Ugandan women recovering from female genital fistula surgery participated in hour-long semistructured interviews exploring their mental, physical, and overall health trajectories. Each interview lasted about an hour and was audio-recorded. Following translation and transcription, the data underwent thematic analysis by humans and by Versa, a University of California, San Francisco (UCSF) developed LLM powered by ChatGPT-4o. The AI analysis was conducted using 2 strategies: inductively and deductively. Finally, the analytic outputs were compared by MP and reviewed by AME and MG to evaluate the code frequency and alignment (most frequently used codes and equivalent concepts vs unique concepts between human and AI codes), thematic robustness (differentiation of distinct concepts and labels that convey substantive findings), analysis quality (narrative depth and excerpt accuracy), and efficiency of each method (total person-hours spent on comparable analytic tasks).
    RESULTS: A comparative analysis revealed significant thematic overlap between the human-synthesized and AI-generated outputs, though notable differences in granularity and efficiency emerged. When inductively coding, Versa identified 39 codes, whereas human researchers used a more expansive set of 54 codes. While Versa's thematic analyses were generally accurate regarding how the interview data were represented in the analyses, the human-synthesized analysis was more robust because it incorporated compelling excerpts, and the summary content provided greater depth and range that the Versa analyses lacked. The disparity in efficiency was stark: the human analysis took approximately 15 hours to complete, while Versa produced the analysis in about 2.5 hours.
    CONCLUSIONS: While Versa significantly expedited the initial coding phase, it still relied heavily on human researchers to create appropriate prompts and input all the data into the chat. Versa's inability to replicate the narrative depth and description of human synthesis suggests that LLMs currently lack the interpretive sensitivity required to capture the lived experiences. Using Versa alone does not currently yield a high-quality analysis; significant human engagement is needed to maintain ethical and interpretive rigor.
    TRIAL REGISTRATION: ClinicalTrials.gov NCT05437939; https://clinicaltrials.gov/study/NCT05437939.
    Keywords:  ChatGPT; Uganda; artificial intelligence; data analysis; fistula; generative AI; interviews as topic; large language models; obstetric surgery; qualitative research
    DOI:  https://doi.org/10.2196/94809
  5. medRxiv. 2026 Aug 12. pii: 2026.08.10.26360108. [Epub ahead of print]
       Background: Large language models (LLMs) that use retrieval-augmented generation (RAG) are increasingly used to answer clinical questions, although the evaluation of these systems remains limited. Building on previous studies conducted by our team, this case report aimed to improve upon this knowledge gap by applying a reusable methodology to compare the performance of eight LLMs that utilize RAG techniques for evidence synthesis.
    Case Presentation: Eight commercially available RAG LLM tools (OpenEvidence, Undermind, Consensus, SciSpace, Elicit, MediSearch, EvidenceHunt, and Scite) were evaluated using twelve ChatGPT-generated clinical questions on the topics of treatment, etiology, and prognosis. To enable comparison, we prompted ChatGPT to identify all key unique medical concepts from the full set of LLM responses to each question. Concepts were categorized as critical ("must-have") or non-critical ("nice-to-have") for answering the clinical question. Experienced information scientists were consulted at each step for their expertise. Descriptive statistics and Kruskal-Wallis tests were used to compare performance across tools and question categories. No significant differences were found among the eight RAG LLMs in their coverage of "must-have" (p=0.95) or "nice-to-have" (p=0.16) key unique medical concepts, and no single tool consistently captured all identified concepts.
    Conclusions: These findings suggest that RAG LLMs may be supplementary tools for evidence retrieval and synthesis but cannot, at this time, fully replace comprehensive expert review of the medical literature. The evaluation framework presented here may be a useful model for future comparative assessments of rapidly evolving AI evidence synthesis tools.
    DOI:  https://doi.org/10.64898/2026.08.10.26360108
  6. Quant Biol. 2026 Dec;14(4): e70050
      Manual curation of gene-disease-phenotype relationships from the human genetics literature is a persistent bottleneck for maintaining its bioinformatics databases. Whereas large language models (LLMs) offer a promising alternative, there is currently no systematic benchmark that evaluates whether state-of-the-art commercial LLMs can perform this task reliably on the full-text articles. To address this gap, we introduce a standardized benchmark comprising 406 full-text articles covering 180 congenital heart disease-associated genes, and a multi-dimensional evaluation framework that incorporates fuzzy matching to account for synonyms and partial matches. We benchmarked seven state-of-the-art LLMs, GPT-4o, Claude-Opus-4, DeepSeek-R1, Grok-4, Qwen-3.5, Gemini-2.5 (Pro), and GPT-5 on the extraction of structured gene, disease, and phenotype fields. The top-performing model, Grok-4, achieved 97.6% overall accuracy, whereas the lowest-performing model reached approximately 88%, still surpassing many prior benchmarks employing zero-shot or n-shot prompting in biomedical relation extraction (RE) tasks. Our results provide a rigorous characterization of current LLMs capabilities and limitations. This paper contains two components. First, we conducted a human genetics field benchmark study on LLMs against a curated database. Second we developed the evaluation framework for this task. The benchmark dataset, evaluation framework, and model benchmarking outputs are made available online to support future studies in a reproducible manner.
    Keywords:  CHDgene; Claude‐Opus 4; DeepSeek‐R1; GPT‐4o; LLM; biomedical curation; congenital heart disease; evidence synthesis; gene annotation; generative artificial intelligence
    DOI:  https://doi.org/10.1002/qub2.70050
  7. PLOS Digit Health. 2026 Oct;5(10): e0001768
      Systematic reviews of randomized controlled trials (RCTs) support evidence-based clinical advancement. Clinical trial reporting guidelines, including CONSORT, establish common data elements that support systematic synthesis of published trial outcomes. However, the rapid rise of digital health technologies presents new challenges for reporting guidelines, and substantial real-world reporting heterogeneity has been observed. Therefore, machine-assisted evaluation of reporting adherence offers a pathway for more agile information management to accommodate the diverse and disruptive nature of digital health technology-enabled RCTs. Here, we propose an accelerated retrieval-to-abstraction literature synthesis framework to conduct article screening, text and table data preprocessing, and full-text review using an unmodified pretrained large language model. We apply this methodology in two digital health technology-enabled RCT case studies evaluating adherence to technology and digital health equity guideline reporting items. Large language model performance was on par with inter-annotator agreement F1 scores (≥0.9) in both screening and information-extraction tasks. These findings support the scalability and rigor of accelerated evidence synthesis and demonstrate the importance of common reporting elements across digital health technology-enabled RCTs.
    DOI:  https://doi.org/10.1371/journal.pdig.0001768
  8. Nature. 2026 Oct 01.
      
    Keywords:  Computer science; Language; Machine learning
    DOI:  https://doi.org/10.1038/d41586-026-03044-9
  9. Curr Biol. 2026 Oct 05. pii: S0960-9822(26)01029-8. [Epub ahead of print]36(19): R1019-R1020
      Much of our shared knowledge about biodiversity, including data on species' traits, relationships, and occurrences, is stored in the published literature. This body of knowledge can span multiple centuries, languages, and publication formats, making it challenging to collect, compare, and synthesize data across entire clades. New artificial intelligence (AI) tools, including large language models (LLMs) in particular, are well suited to integrate and summarize large bodies of text and figures. However, the biodiversity literature is too large and multi-layered to be reliably interrogated by simply placing publications into an LLM context window. For the typical tasks of biodiversity scientists (e.g., assembling descriptive characters, reconstructing nomenclatural history), we need structured, provenance-aware, and deterministic data retrieval. Here we present corpus: a tool for converting biodiversity literature PDFs for a clade of organisms into a searchable knowledge base (a 'corpuscle') of text, figures, and references. Users connect an LLM client via corpus' Model Context Protocol (MCP) server, to interact directly with the processed literature through predefined search tools. corpus addresses critical gaps in LLM applications for biology by transforming the literature into an explicit retrieval layer, rather than relying on hidden training data or probabilistic recollection.
    DOI:  https://doi.org/10.1016/j.cub.2026.08.019
  10. Comput Struct Biotechnol J. 2026 ;35(3): 0161
      Manual curation of nanotoxicity data remains a major bottleneck in the scalable construction of large datasets because key physicochemical and toxicological variables are often distributed across text, figures, tables, captions, legends, and Materials and Methods sections. Here, we developed a multimodal large language model (LLM) workflow for automated extraction and structured curation of nanotoxicity data from research articles. Four multimodal LLMs were benchmarked on curated set of 17 oxide nanotoxicity articles. GPT-5.4 provided the best balance between extraction accuracy and efficiency, achieving a mean extraction F1 score of 96.1% and an assay results F1 score of 86.5% on the main benchmark. The optimized workflow was then applied to 821 articles, generating 16,786 structured records and substantially reducing the manual recovery of figure-derived quantitative data. Automated machine learning (AutoML) modeling showed that the physicochemical (PChem) score-filtered subset achieved higher overall predictive performance, whereas record-count balancing among the 4 most represented materials reduced performance. Models trained only on records for ZnO, TiO2, Fe3O4, or SiO2 improved prediction for the corresponding material compared with multi-material models, but applicability domain analysis showed that these gains occurred within narrower descriptor-space coverage. These findings demonstrate that multimodal extraction can move nanotoxicity curation beyond text-only workflows while supporting data-aware model selection across broad-coverage, quality-filtered, focused multi-material, and single-material prediction strategies.
    DOI:  https://doi.org/10.34133/csbj.0161
  11. Science. 2026 Oct 08. 394(6820): 166-167
      Artificial intelligence can now solve research problems in mathematics and verify the answers.
    DOI:  https://doi.org/10.1126/science.ael6628
  12. Res Social Adm Pharm. 2026 Sep 30. pii: S1551-7411(26)00191-9. [Epub ahead of print]
       BACKGROUND: Recent advances in machine learning (ML) have sparked the rise of artificial intelligence (AI) in pharmacovigilance (PV). Several studies suggest these technologies could address key PV challenges, including underreporting, manual case processing, and delayed signal detection.
    PURPOSE: This systematic review aimed to map AI/ML applications across post-marketing PV functions, characterize methodological and geographic trends, and identify evidence gaps in external validation and implementation readiness.
    METHODS: Following PRISMA 2020 and a Population, Concept, Context (PCC) framework, eight bibliographic databases were searched from January 2021 through October 2024, supplemented by grey literature and reference-list screening. Sixty-seven studies were included. Two reviewers independently screened, extracted, and appraised studies, with a third reviewer verifying extraction. Methodological quality was assessed using APPRAISE-AI, and certainty of evidence was considered using GRADE principles.
    RESULTS: Safety-signal detection was the most common application (50.5%), followed by adverse effect prediction (21%), case-processing efficiency (15%), drug-drug interaction detection (7.5%), and high-risk patient identification (6%). XGBoost and ensemble methods dominated structured-data applications, while BERT-family transformers prevailed for unstructured text. However, external validation was the weakest APPRAISE-AI domain (mean score 52%), and prospective implementation studies were rare.
    CONCLUSION: Artificial intelligence demonstrates substantial potential to transform pharmacovigilance from reactive to proactive systems. However, current evidence is largely retrospective, with limited external validation and real-world implementation. Although BERT and XGBoost demonstrate promising performance in signal detection, heterogeneous methodologies limit conclusions regarding generalizability and clinical impact. Responsible integration into safety-monitoring workflows requires rigorous independent validation, prospective human-in-the-loop evaluation, standardized performance metrics, and geographically representative datasets. Addressing data privacy, algorithmic fairness, human oversight, and regulatory requirements will also be crucial. Future research should therefore focus on robust validation and real-world implementation to ensure that AI translates into reliable, clinically meaningful improvements in patient safety.
    Keywords:  Artificial intelligence; Drug safety; External validation; Machine learning; Pharmacovigilance
    DOI:  https://doi.org/10.1016/j.sapharm.2026.09.001
  13. Technol Health Care. 2026 Oct 10. 9287329261495502
      BackgroundEfficient organization and retrieval of rapidly growing biomedical literature require representations that capture contextual semantics and explicit biomedical relationships. However, existing methods typically model text and ontology structure separately, producing incomplete representations with limited cluster separability.ObjectiveThis study addresses the incomplete feature representation and low cluster separability that emerge as textual and ontological information are modeled independently, with the goal of enhancing clustering performance on biomedical literature.MethodsUCTE-OGC applies PubMedBERT with unsupervised contrastive learning to encode article titles and abstracts. In parallel, it learns relation-aware GO representations through DIFF-CR edge weighting, Node2Vec, and graph convolutional networks. An attention-based fusion module projects both representations into a shared latent space and optimizes their integration using a symmetric NT-Xent objective. The resulting embeddings are evaluated with K-Means, Gaussian mixture models, and spectral clustering.ResultsExperiments on a PubMed-derived dataset showed that UCTE-OGC produced higher observed clustering performance than the comparison methods under the reported settings and achieved its best results with spectral clustering. Ablation experiments indicated that contrastive learning and GO structural information made complementary contributions.ConclusionIntegrating contrastive text representations with relation-aware GO graph modeling provides effective embeddings for organizing and retrieving large biomedical literature collections.
    Keywords:  Biomedical literature; PubMedBERT; attention-based fusion; contrastive learning; gene ontology (GO); graph convolutional network; semantic similarity
    DOI:  https://doi.org/10.1177/09287329261495502