bims-arines Biomed News
on AI in evidence synthesis
Issue of 2026–07–19
eleven papers selected by
Farhad Shokraneh, Systematic Review Consultants LTD



  1. Value Health. 2026 Jul 14. pii: S1098-3015(26)02532-5. [Epub ahead of print]
       OBJECTIVES: Systematic literature reviews (SLRs) underpin life sciences research but are resource intensive. Generative artificial intelligence, particularly large language models (LLMs), may accelerate key SLR tasks, yet performance and reliability for evidence synthesis remain unclear. This manuscript aims to review current evidence on GenAI performance across core SLR tasks.
    METHODS: We conducted a PRISMA-adapted rapid evidence assessment of English-language biomedical studies published from November 2022 to July 2025 evaluating GenAI or LLMs for systematic literature review tasks, including search strategy development, title/abstract screening, full-text screening, data extraction, risk-of-bias assessment, qualitative synthesis, report writing, and end-to-end review generation. Findings were summarized qualitatively by task.
    RESULTS: Among 115 included studies, evidence supporting the use of GenAI was strongest for title/abstract screening (n=51) and data extraction (n=33). Selected high-quality evaluations reported sensitivities ≥90%, workload reductions of 27-71%, and human-comparable or superior performance in calibrated human-in-the-loop workflows. Evidence for full-text screening (n=15) and risk-of-bias assessment (n=17) was more variable, showing gains in structured or fine-tuned implementations but persistent limitations in specificity and nuanced judgment. For search strategy development, qualitative synthesis, and report writing, GenAI was most effective as a supportive tool; fully autonomous end-to-end SLR generation was unreliable.
    CONCLUSIONS: GenAI can improve efficiency across multiple SLR tasks when used in hybrid human-AI workflows. Current evidence supports targeted, task-specific adoption with transparent reporting and human oversight, rather than full automation.
    Keywords:  Evidence Synthesis; Generative Artificial Intelligence; HEOR; Health Technology Assessment; Large Language Models; Systematic Literature Review
    DOI:  https://doi.org/10.1016/j.jval.2026.06.012
  2. Orthop Traumatol Surg Res. 2026 Jul 16. pii: S1877-0568(26)00214-8. [Epub ahead of print] 104793
       BACKGROUND: The volume of published research has expanded rapidly, intensifying the need for reliable, regularly updated evidence syntheses. Systematic reviews (SRs) and meta-analyses (MAs) remain the highest level of evidence, but their completion is time-consuming and resource-intensive, often extending beyond one year. Artificial intelligence (AI) is increasingly applied to automate stages of this workflow, yet validation of performance and methodological rigor remains limited. This narrative review addresses three questions: 1) at which stages of the SR/MA workflow can AI be used reliably; 2) how should AI performance be benchmarked for each task; and 3) how can the residual error remaining after human checking be quantified.
    METHODS: We outline the principles and key steps of a traditional meta-analysis and examine the contribution of AI at each stage. Literature was drawn from major biomedical databases; no systematic search was performed. AI performance is summarized by task, with benchmarking and residual-error quantification considered separately.
    RESULTS: Across the workflow, AI maturity is heterogeneous. In a 2025 systematic review of 19 comparative studies, generative AI missed 68.0%-96.0% of relevant studies in searching, made incorrect inclusion decisions in 0.0%-29.0% and incorrect exclusions in 1.0%-83.0%, incorrect data extractions in 4.0%-31.0%, and incorrect risk-of-bias assessments in 10.0%-56.0%. A 2025 scoping review identified 37 articles: 15/37 (41%) searching, 14/37 (38%) study selection, 11/37 (30%) extraction, 33/37 (89%) GPT-based, 21/37 (57%) validation studies; 20/37 (54%) drew a promising conclusion, none as a validated implementation. Title/abstract screening was most mature (sensitivity 99.2%, specificity 83.6%) versus full-text screening (97.6% and 47.4%). For structured extraction across 30 articles, one tool reached 92.0% precision, recall, and F1, but sensitivity fell to 77.0%-80.0% for review-specific variables, with confabulations in 4/90 (4%) of data points; across 107 trials, extraction accuracy was 96.2% (97.9% with refinement) while mean time fell from 86.9 to 14.7 min per trial. Risk-of-bias assessment was most fragile (kappa 0.51, 95% CI: 0.36-0.66). For residual error, 0 errors among 60 checked items corresponds to an upper one-sided 95% bound of 4.9%.
    DISCUSSION: AI is transforming SR/MA toward hybrid human-machine workflows rather than replacing them. Performance is task-specific, and metrics alone are insufficient: residual uncertainty should be quantified. Current evidence supports AI as an assistive technology under human oversight, not autonomous use.
    LEVEL OF EVIDENCE: lV; narrative review.
    Keywords:  AI-assisted evidence synthesis (systematic review/meta-analysis/human oversight); Data extraction (structured variables/field-level accuracy/confabulation); Hybrid AI workflow (LLM benchmarking/prompt validation/inter-model entropy); Residual error quantification (Clopper-Pearson/rule of three/manual checking); Risk-of-bias assessment (RoB 2/ROBINS-I/NOS/kappa); Study screening (title-abstract/full-text/recall-specificity/WSS@95%)
    DOI:  https://doi.org/10.1016/j.otsr.2026.104793
  3. Interact J Med Res. 2026 Jul 14. 15 e94317
       Unlabelled: Deduplication across search results is one of the earliest and most critical steps in the systematic review methodological process; yet, existing solutions often lack the transparency, auditability, and reproducibility required by rigorous systematic review standards. Many automated deduplication tools introduce bias through opaque, nonconfigurable algorithmic decisions, while also potentially removing relevant references through false positive identification. We provide a tutorial on the Rayyan Method, including the Systematic Auto Resolver feature for the deduplication process. This method is defined by an enhanced deduplication approach that combines high-sensitivity duplicate detection with user-controlled resolution criteria. Systematic Auto Resolver enables research teams to define, apply, and document their own deduplication standards rather than relying on predetermined automated decisions. Users apply deduplication criteria, iteratively reviewing results after each pass, maintaining complete control over methodological decisions. The Rayyan Method addresses critical limitations in current deduplication approaches by supporting methodological rigor through user-controlled resolution while enhancing efficiency, transparency, and reproducibility. By empowering research teams to define their own deduplication criteria, this approach supports and aligns with the prescribed methodologically rigorous systematic review process. The method provides a citable framework for researchers to comprehensively document their deduplication methodology.
    Keywords:  deduplication; evidence synthesis; software demonstration; systematic review methodology; transparency
    DOI:  https://doi.org/10.2196/94317
  4. J Med Libr Assoc. 2026 Jul 01. 114(3): 191-207
       Objectives: To synthesize and map literature on automated indexing of the biomedical literature, with a focus on the Medical Text Indexer (MTI) at the National Library of Medicine (NLM). We review the drivers, benefits, and challenges of automated indexing, evolution of the MTI from 2000-2025, and impacts on information retrieval in MEDLINE.
    Methods: We conducted a scoping review following the JBI Manual for Evidence Synthesis and reported findings using PRISMA-ScR and PRISMA-S. We searched several bibliographic databases, key journals, conference proceedings, and grey literature sources, with no restrictions on language or study design. Eligible publications were published 2000-2025 and focused on MTI development. Screening, data charting, and thematic analysis were conducted by multiple reviewers.
    Results: We included 64 publications, with most originating from the United States (n=53, 83%) and five from Canada (8%). Study methods included evaluation or comparative studies (65%), qualitative descriptions (25%), and mixed methods (11%). MTI evolved from a rules-based recommendation tool in 2002 to the neural network-based MTIX in 2024. Despite numerous enhancements to the MTI, human curation remains necessary for approximately one-third of records to correct inaccuracies, capture missed concepts, and address errors arising from figurative language or algorithmic biases.
    Conclusions: This review synthesizes twenty-five years of MTI research (2000-2025). Despite reduced indexing times and a markedly improved algorithm, the MTIX has not yet achieved full equivalence to human indexing. Our findings suggest searchers should watch for algorithmic ambiguities in their MEDLINE searching and adapt accordingly. Health sciences librarians should work with stakeholders, including authors, to shape future algorithmic indexing methods, outputs, evaluation and research.
    Keywords:  Algorithmic indexing; Automated indexing; Medical Subject Headings (MeSH); Medical Text Indexer (MTI); National Library of Medicine (NLM); Neural Network, Search Filters; PubMed/MEDLINE Searching
    DOI:  https://doi.org/10.5195/jmla.2026.2406
  5. J Intern Med. 2026 Jul 14.
      
    Keywords:  biomedical literature; citation errors; evidence synthesis; fabricated references; medRxiv; research integrity
    DOI:  https://doi.org/10.1111/joim.70139
  6. Otol Neurotol Open. 2024 Sep;4(3): e059
       Background: The incorporation of artificial intelligence (AI), especially large language models like Generative Pretrained Transformer 4 (GPT-4), into medical practice is a burgeoning field of interest. This research evaluates the applicability of GPT-4 in otology by analyzing its responses to queries based on otologic clinical practice guidelines.
    Methods: Key guidelines from otology were selected, and corresponding questions were formulated to examine GPT-4's interpretation and response accuracy. Two independent reviewers assessed the AI-generated answers for accuracy and completeness, using a structured Likert scale. A re-evaluation was conducted to evaluate the reproducibility of the results.
    Results: The analysis showed a high accuracy level (mean score: 4.75 of 5) and completeness (mean score: 2.88 of 3) in GPT-4's responses. The interrater agreement, as indicated by Cohen κ, was substantial. GPT-4 consistently advised on individualized treatment plans and professional consultation, particularly for guidelines with weaker evidence, demonstrating its cautious approach to handling medical information.
    Conclusion: GPT-4 exhibits promising potential as an auxiliary tool in otology, providing accurate and comprehensive information. However, its role should be viewed as supplementary, with emphasis on continual updates and careful monitoring to align with evolving medical knowledge. Future studies are recommended to further explore the depth of AI application in diverse clinical scenarios and its real-time impact on clinical outcomes.
    Keywords:  Artificial intelligence; Hearing loss; Otitis media; Otology; Tinnitus; sudden
    DOI:  https://doi.org/10.1097/ONO.0000000000000059
  7. JMIR Res Protoc. 2026 Jul 16. 15 e101691
       BACKGROUND: Pharmacovigilance aims to protect patient safety by identifying and managing adverse events associated with pharmaceuticals. Determining the causality of these adverse events is central at both the individual case and population levels; however, it is increasingly challenging as the volume and complexity of safety data grow. Although AI and related technologies have been proposed to support causality assessment, limited research has examined how these methods are used, their information and quality requirements, or how associated risks are addressed.
    OBJECTIVE: This scoping review aims to determine the available evidence on AI-based methods for causality assessment in pharmacovigilance. The primary objective is to characterize how these methods are applied or proposed with a focus on their functional roles, reported data inputs and information needs, and associated risks. Secondary objectives include comparing applications at the individual case and population levels; describing the types of AI-based techniques and automation tools used in causality assessment workflows; and summarizing reported data quality considerations and governance mechanisms, including risk management approaches.
    METHODS: Sources describing or proposing AI-based approaches, including data-driven models (machine learning, natural language processing, knowledge graphs, and causal inference) and knowledge- or rule-based systems implementing causal assessment logic, will be eligible. Searches will be conducted in PubMed, Web of Science Core Collection, ProQuest, EBSCOhost, and Ichushi Web and will be restricted to English- and Japanese-language sources. Two reviewers will independently screen records and full-text articles, with disagreements resolved by a third reviewer. Data will be charted on use cases, information inputs, data quality dimensions, model characteristics, governance mechanisms, and identified risks. Synthesis will follow a reflexive thematic analysis approach and be reported in accordance with PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) guidelines informed by applicable PRISMA-S (Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension) elements.
    RESULTS: This protocol was registered in the Open Science Framework platform on December 23, 2025. The registration was subsequently updated on May 19, 2026, to reflect an extension to the data collection period. A preliminary database search was conducted in December 2025, retrieving a total of 760 records, of which the preliminary title and abstract screening identified 196 (25.8%) articles for full-text review. Database searches are scheduled for July 2026. Data charting is scheduled for August 2026, and synthesis is scheduled for September 2026. Findings are expected to be submitted for publication by the end of December 2026.
    CONCLUSIONS: This review is expected to provide a structured map of AI-based applications for causality assessment in pharmacovigilance, clarify reported information inputs and data quality dimensions, and synthesize risk management and governance approaches. The findings are expected to inform methodological development, practical implementation, and the governance of AI-supported causality assessment.
    TRIAL REGISTRATION: Open Science Framework 10.17605/OSF.IO/QVF5C; https://osf.io/qvf5c/overview.
    INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/101691.
    Keywords:  AI; artificial intelligence; causality assessment; machine learning; pharmacovigilance; scoping review
    DOI:  https://doi.org/10.2196/101691
  8. Acad Pediatr. 2026 Jul 14. pii: S1876-2859(26)00180-4. [Epub ahead of print] 103398
       OBJECTIVE: Since the 1994 "Back to Sleep" campaign, pediatricians have promoted evidence-based infant safe sleep practices to reduce sleep-related infant deaths. However, caregivers increasingly seek guidance online. We sought to determine the accuracy of large language model (LLM) responses to caregiver questions about infant safe sleep, compared with the American Academy of Pediatrics' (AAP) 2022 recommendations.
    DESIGN: Nine caregiver questions adapted from Reddit New Parents forum were mapped to core AAP safe sleep topics. Each was entered into three LLMs: ChatGPT 5, Gemini 2.5 Flash, and Claude Sonnet 4.5, three times within the same day to assess stability. Three reviewers scored responses on a 0-2 scale for accuracy (primary outcome), completeness, and empathy. Stability reflected similarity across repeated responses. Readability was calculated using the Flesch-Kincaid grade level. Mean scores were compared using descriptive statistics, analysis of variance, and post hoc testing.
    RESULTS: Mean accuracy varied significantly across models. Gemini had the highest accuracy score (mean 1.85), followed by Claude (1.44), and ChatGPT (1.30). Gemini was significantly more accurate than ChatGPT (p=0.01). All models scored high in empathy (2). There were no significant differences in completeness and stability between models. ChatGPT had the lowest average readability, with all models' reading levels between grades seven to nine (7.64 vs 9.15 vs 8.82, p=0.01). Direct guideline questions yielded higher accuracy than nuanced questions.
    CONCLUSION: LLMs offer inconsistently accurate but empathetic infant safe sleep advice with frequent deviations from AAP recommendations. Pediatric oversight and collaboration with technology developers are essential to ensure safe, evidence-based information for families.
    Keywords:  artificial intelligence; infant safe sleep; injury prevention; patient/family education
    DOI:  https://doi.org/10.1016/j.acap.2026.103398
  9. J Med Internet Res. 2026 Jul 13.
       BACKGROUND: Widespread antiretroviral therapy has greatly extended the life expectancy of people living with HIV (PLWH), making cardiovascular disease (CVD) one of their primary comorbidities. Nevertheless, significant cross-specialty knowledge gaps persist in routine clinical practice. Siloed disciplinary expertise results in low clinical adherence to guideline-recommended risk management interventions, highlighting an urgent demand for integrated, evidence-based tools that break down interdisciplinary barriers. Large language models (LLMs) have demonstrated robust medical knowledge retrieval and reasoning capacity in recent years, yet no systematic evaluation has determined whether these models can bridge such knowledge gaps and facilitate multidisciplinary collaborative CVD management for PLWH.
    OBJECTIVE: This study compared the performance of four mainstream AI models (Deepseek-V3, Deepseek-R1, ChatGPT-4o, ChatGPT-o4-mini) and 12 human clinicians (8 infectious disease specialists and 4 cardiologists) in addressing guideline-based CVD management tasks for PLWH.
    METHODS: Based on four authoritative domestic and international guidelines on HIV and CVD care, a structured 25-question assessment battery was developed via two rounds of Delphi expert consultation, with standard reference answers and an evaluation framework finalized through expert consensus. Responses of the four LLMs were generated with standardized prompts, while 12 clinicians answered identical questions in one-on-one structured interviews, with all verbal replies transcribed verbatim. Six multidisciplinary experts independently rated all responses across four dimensions: accuracy, completeness, readability and reliability, using a 4-point ordinal scale ranging from 1 (poor) to 4 (excellent). Cumulative link mixed models (CLMMs) were applied to analyze intergroup differences.
    RESULTS: All AI models achieved statistically significantly higher scores than clinicians across all evaluation dimensions (p < 0.01). The AI group had mean scores of 3.44-3.68 (median = 4, CV: 0.145-0.178). Restricted by individual factors including specialty background, knowledge reserve, clinical experience, clinicians obtained lower mean scores of 1.78-2.05 (median = 2, CV: 0.428-0.473) with markedly greater score dispersion. Among all AI models, Deepseek-R1 delivered the optimal performance and showed statistically significant advantages over ChatGPT-4o, ChatGPT-o4-mini and Deepseek-V3 (all p < 0.01). Specialty-stratified CLMM analysis revealed no significant overall score difference between cardiologists and infectious disease specialists (OR = 0.92, 95% CI: 0.84-1.01, p = 0.094). Dimension-specific CLMMs combined with Wilcoxon rank-sum tests confirmed that cardiologists only earned significantly higher scores in the accuracy dimension (OR = 0.81, 95% CI: 0.67-0.97, p = 0.0261). Domain-specific performance divergence was observed: cardiologists outperformed infectious disease specialists in CVD risk assessment (2.26 vs 1.83), whereas infectious disease specialists achieved higher scores on drug adverse effect evaluation (2.23 vs 1.65).
    CONCLUSIONS: This structured Q&A study on CVD management for PLWH found that LLMs outperformed human clinicians on all assessment metrics, with Deepseek-R1 attaining a distinctly superior composite score. The findings support the promising potential of Deepseek-R1 as a cross-disciplinary decision-support tool: it integrates multi-domain complex clinical knowledge, which may help address cross-specialty knowledge barriers, could improve the completeness and precision of clinical information output, and may enhance communication and decision-making efficiency for patients with complicated multimorbidity. To maximize clinical benefits, AI systems should be integrated into multidisciplinary care workflows alongside targeted clinical training to optimize the management of complex comorbidities among PLWH.
    CLINICALTRIAL:
    DOI:  https://doi.org/10.2196/89858
  10. Minerva Urol Nephrol. 2026 Jun;78(3): 411-418
       BACKGROUND: The integration of Artificial Intelligence (AI), particularly Large Language Models (LLMs) like ChatGPT, into healthcare is reshaping patient information-seeking behaviors. While prior studies have evaluated ChatGPT's performance across various urological domains, no research has yet addressed its ability to provide guideline-concordant recommendations for Bladder Cancer (BCa) management based on histopathological findings. This study aimed to evaluate the concordance between ChatGPT-4 and urologists' therapeutic recommendations for patients undergoing transurethral resection of bladder tumor (TURBT), using the European Association of Urology (EAU) guidelines as the reference.
    METHODS: We retrospectively analyzed 219 consecutive patients who underwent TURBT for BCa at a tertiary referral center between January and June 2024. Data collected included demographics, oncological history, and histopathology reviewed according to WHO 2022 standards. Patients were risk-stratified using the 2021 EAU Non Muscle-Invasive Bladder Cancer (NMIBC) risk model. Two urologists independently defined treatment plans per 2024 EAU guidelines. ChatGPT-4 (accessed April 2025) was queried using histopathological reports and previous BCa history. Recommendations from ChatGPT and urologists were compared for agreement, evaluated using Cohen's Kappa coefficient.
    RESULTS: Among the 219 patients (mean age 72.36 years, 78.5% male), 42.9% had de novo BCa, 6.8% underwent II look TURBT, and 50.2% had recurrence. Histopathology revealed 68.5% pure urothelial carcinoma and 53.8% high-grade tumors. NMIBC risk classification identified 5.5% low, 19.6% intermediate, 32.9% high, and 9.1% very-high risk cases. Overall agreement between urologist and ChatGPT recommendations was poor (Cohen's K=-0.06). Stratified analyses showed moderate agreement for first-time BCa (K=0.23) and very high agreement in muscle-invasive BCa (MIBC) cases (K=0.88). In contrast, recurrent cases and II look TURBT showed low to negative concordance (K=-0.24 and K=-0.6, respectively). ChatGPT displayed a notably conservative attitude towards recommending early radical cystectomy, aligning with guidelines in only 3 of 21 cases.
    CONCLUSIONS: Despite good readability and accessibility, ChatGPT-4's therapeutic recommendations based solely on histopathological reports lacked consistency with EAU guidelines, especially in complex scenarios such as recurrent disease or post-intravesical therapy management. Concordance was higher in straightforward cases like initial diagnosis or MIBC. These findings underscore the limitations of AI when clinical context is incomplete and highlight the irreplaceable role of clinician judgment. ChatGPT may serve as a supplementary tool for patient education but should not be used independently for clinical decision-making in BCa.
    DOI:  https://doi.org/10.23736/S2724-6051.26.06641-3
  11. Nat Med. 2026 Jul 17.
      
    Keywords:  Machine Learning; Medical research
    DOI:  https://doi.org/10.1038/d41591-026-00037-z