bims-arines Biomed News
on AI in evidence synthesis
Issue of 2026–07–12
ten papers selected by
Farhad Shokraneh, Systematic Review Consultants LTD
  1. Leveraging artificial intelligence for evidence synthesis.
  2. Artificial Intelligence in Evidence Synthesis: Benefits, Limitations, and the Importance of Transparent Reporting.
  3. Qualitative Analysis of Discrepancy Patterns Between Large Language Models and Human Reviewers in Abstract Screening for Systematic Reviews.
  4. Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study.
  5. Custom GPT models for complex rheumatology systematic reviews: A two-part evaluation of data extraction and prognosis appraisal.
  6. Large language models vs. conventional search in identifying delirium guidelines for a scoping review.
  7. Agreement between ChatGPT and human-derived multilevel meta-analyses: a reproducibility study across clinical evidence syntheses.
  8. Artificial Intelligence for Evidence Synthesis of Emerging Biologics to Improve Skeletal Health in Osteogenesis Imperfecta: Systematic Review and Meta-Analysis.
  9. Leveraging generative artificial intelligence for the development of non-interventional research study protocols: a proof-of-concept feasibility study.
  10. Tackling challenges in large language model-based data extraction via context engineering: A commentary on Jansen et al. (2025).
  1. JBI Evid Synth. 2026 Jul 01. 24(7): 1280-1282
    Leveraging artificial intelligence for evidence synthesis.
    Barbara Nussbaumer-Streit, Marc Streit, Kylie Porritt.
      
    DOI:  https://doi.org/10.11124/JBIES-26-00371
  2. J Am Acad Dermatol. 2026 Jul 08. pii: S0190-9622(26)03082-3. [Epub ahead of print]
    Artificial Intelligence in Evidence Synthesis: Benefits, Limitations, and the Importance of Transparent Reporting.
    Michelle A Kahn, Katharine Tepper, Shoshana Marmon.
      
    Keywords:  Artificial Intelligence; LLMs; Machine Learning; PRISMA; Retrieval-augmented AI; Systematic Reviews
    DOI:  https://doi.org/10.1016/j.jaad.2026.05.139
  3. J Epidemiol. 2026 Jul 04.
    Qualitative Analysis of Discrepancy Patterns Between Large Language Models and Human Reviewers in Abstract Screening for Systematic Reviews.
    Kyung Hwa Lee, Hakyoung Kim, Dae Sik Yang, Won Sup Yoon, Chai Hong Rim.
       BACKGROUND: We evaluated the reliability of large language models (LLMs) for abstract screening under real-world review practices and qualitatively characterized model-human discordance to inform safe workflow integration.
    METHODS: We evaluated GPT-4.0, GPT-5.0, and GPT-5.0-mini on two curated systematic review datasets representing contrasting topic densities, defined by the target-to-background ratio (TBR): a core-subject dataset (TBR 69%) in which the target intervention was central, and a peripheral-subject dataset (TBR 4.4%) in which the target intervention was incidental. Final inclusion after full-text review served as the reference standard. We developed a qualitative taxonomy of disagreements, classifying false negatives as intended human leniency, gray-zone ambiguity, or true LLM misses, and false positives as implicit or additional human exclusion rules, gray-zone ambiguity, or nominal inclusions that increase workload only.
    RESULTS: GPT-5.0-mini achieved the best sensitivity-efficiency trade-off (core-subject: 91% sensitivity with 96.7% workload reduction; peripheral-subject: 83% sensitivity with 92.7% workload reduction) and negative predictive value >99% in both datasets. Disagreement was lower when relevance was central (core-subject: 1.6%, 7/430) with no true LLM misses (0/430). In the peripheral-subject dataset, disagreement was higher (10.6%, 74/696), driven mainly by intended human leniency among false negatives (52/56) and gray-zone ambiguity among false positives (12/18), while true LLM misses remained rare (0.4%, 3/696).
    CONCLUSION: Many model-human disagreements reflect topic- and workflow-dependent screening conventions rather than intrinsic model failure. LLM-assisted screening may improve efficiency without compromising reliability when accompanied by appropriate safeguards for ambiguous records.
    Keywords:  Abstract Screening Automation; GPT Models; Large Language Models (LLMs); Qualitative Error Analysis; Systematic Review
    DOI:  https://doi.org/10.2188/jea.JE20260233
  4. PLoS One. 2026 ;21(7): e0353155
    Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study.
    Siddharth Gandhi, Arveen Shokravi, Yashan Chelliahpillai, Michael Balas.
       OBJECTIVE: To evaluate the reliability and diagnostic performance of ChatGPT-o3 in conducting Risk of Bias (RoB) assessments of randomized clinical trials (RCTs) using the Cochrane RoB 2.0 tool.
    MATERIALS AND METHODS: This methodological validation study analyzed 50 RCTs sampled from 50 published meta-analyses. Each trial was independently assessed by the original systematic review authors (OSRAs), our masked human panel, and ChatGPT-o3. Structured prompts based on RoB 2.0 guidelines were used to elicit ChatGPT-o3 assessments. Agreement was evaluated using weighted Cohen's kappa and Gwet's AC2. Diagnostic performance was measured by sensitivity, specificity, and balanced accuracy, with human ratings as the reference.
    RESULTS: ChatGPT-o3 classified 34% of trials as high risk, compared with 22% by our panel, and 12% by the OSRAs. Agreement was modest (median κ: 0.33 with our panel; 0.14 with OSRAs). Overall Gwet's AC2 was 0.30. For detecting high-risk trials, ChatGPT-o3 achieved a sensitivity of 0.46, specificity of 0.69, and balanced accuracy of 0.57. For low-risk trials, its sensitivity was 0.47, specificity was 0.86, and balanced accuracy was 0.66.
    DISCUSSION: The results indicate that ChatGPT-o3 produced more conservative RoB ratings than human reviewers, identifying a greater percentage of trials as having a high RoB.
    CONCLUSION: While unsuitable to be used as a sole assessor, ChatGPT-o3 may serve as an adjunct tool to enhance the efficiency and consistency of RoB assessments in systematic reviews.
    DOI:  https://doi.org/10.1371/journal.pone.0353155
  5. Digit Health. 2026 Jan-Dec;12:12 20552076261467480
    Custom GPT models for complex rheumatology systematic reviews: A two-part evaluation of data extraction and prognosis appraisal.
    Pamela Munguía-Realpozo, Edith Ramírez-Lara, Claudia Mendoza-Pinto, Ivet Etchegaray-Morales, Juan Carlos Solis-Poblano, Marco Alejandro Trinidad-González, Jorge Ayón-Aguilar, Máximo Alejandro García-Flores, Álvaro José Montiel-Jarquín.
       Background: Systematic reviews are essential for evidence-based practice but remain resource-intensive, particularly during full-text data extraction and structured risk-of-bias appraisal in prognostic research. These challenges are amplified in complex autoimmune diseases such as systemic lupus erythematosus (SLE). Recent advances in large language models (LLMs) have raised interest in their potential; however, rigorous benchmarking against expert reviewers in real-world rheumatology settings is limited.
    Objective: To evaluate the feasibility, agreement, and efficiency of customized GPT-based LLMs across two systematic-review tasks: 1) study-level data extraction in metabolomics studies of SLE, and 2) prognostic risk-of-bias appraisal in rheumatology studies using QUIPS.
    Methods: This two-part methodological study was nested within two PROSPERO-registered reviews. For data extraction, fifteen full-text SLE metabolomics studies were processed by human reviewers and by a customized GPT model using a shared, structured template; concordance across predefined fields and extraction time per study were compared. For prognostic appraisal, nineteen rheumatology prognostic studies with adjudicated human QUIPS domain ratings (Low/Moderate/High) were reappraised in 2025 using a customized ChatGPT model (GPT-Reviewer). Agreement with the human reference was quantified using weighted kappa (quadratic weights) with 95% confidence intervals.
    Results: GPT-Reviewer generated complete domain-level QUIPS judgments for all 19 studies, with heterogeneous concordance versus adjudicated human ratings. Domain-specific κw was 0.001 for study participation (95% CI 0.000-0.002) and outcome measurement, 0.129 for study attrition (95% CI 0.028-0.241), 0.137 for prognostic factor measurement (95% CI 0.000-0.478), 0.286 for statistical analysis/reporting (95% CI 0.161-0.322), and 0.681 for study confounding (95% CI 0.488-0.847). The mean extraction time was shorter for the GPT model than for human reviewers (5.7 vs. 30.4 minutes per study).
    Conclusions: Customized GPT-based LLMs are best deployed as complementary tools within human-in-the-loop workflows; improved handling of tables/supplements and domain-specific calibration are needed before routine use in complex rheumatology evidence synthesis.
    Keywords:  artificial intelligence; metabolomics; prognosis; rheumatology; systemic lupus erythematosus
    DOI:  https://doi.org/10.1177/20552076261467480
  6. BMC Med Res Methodol. 2026 Jul 10.
    Large language models vs. conventional search in identifying delirium guidelines for a scoping review.
    Mary Faherty, Ronan Madden, Breeda Herlihy, Siobhan Bowman, Alasdair MacLullich, Suzanne Timmons.
       BACKGROUND: A literature search of information sources is a key methodological step for evidence synthesis, including scoping reviews. This study compares the performance of large language models with conventional systematic literature search methods in identifying published delirium clinical practice guidelines.
    METHODS: Comparative study using parallel searches. Four large language models (ChatGPT, Google Gemini, Claude and Grok) were queried using structured prompts, with results assessed for precision, sensitivity, and efficiency, versus a conventional systematic literature search using established bibliographic databases and grey literature sources.
    RESULTS: Large language models sourced 17 of the 45 guidelines identified through a comparable conventional search, missing 28, but also identified two guidelines not found through the conventional search. The sensitivity of individual large language models ranged from 9% to 29%, using the conventional search as a reference standard. Of the 19 guidelines retrieved through large language models, 63% were deemed high quality, compared with 56% identified through the conventional search. Precision of individual large language models varied from 22% to 60%, far exceeding the precision of the conventional search (1.7%). Large language model-based searches required substantially less execution time compared to conventional methods.
    CONCLUSIONS: While large language models have a significantly lower yield in retrieving clinical guidelines compared to the conventional search, their efficiency, along with their capacity to identify some unique articles, suggests a possible role in complementing conventional search methods in scoping reviews.
    Keywords:  Artificial intelligence; Clinical practice guidelines; Large language models; Literature search; Precision; Scoping review; Search strategy; Sensitivity
    DOI:  https://doi.org/10.1186/s12874-026-02941-x
  7. BMC Med Res Methodol. 2026 Jun 05.
    Agreement between ChatGPT and human-derived multilevel meta-analyses: a reproducibility study across clinical evidence syntheses.
    Nikolai Ramadanov, Hannah Jaeger, Mikhail Salzmann, Robert Hable.
       BACKGROUND: Large language models are increasingly integrated into biomedical research workflows, yet their ability to reliably reproduce complex statistical analyses remains insufficiently evaluated. While prior studies suggest that ChatGPT can replicate conventional meta-analyses, its performance in recalculating statistically demanding multilevel meta-analyses is unknown.
    METHODS: We conducted a systematic review to identify published multilevel meta-analyses (i.e., models accounting for dependency among multiple effect sizes within studies) of hip-related clinical studies that provided extraction tables enabling independent recalculation. For each outcome, pooled estimates and heterogeneity parameters were recomputed using ChatGPT under predefined analytical assumptions and compared with statistician-derived results. Agreement was assessed using absolute and relative differences and Bland-Altman analyses for pooled effects, and qualitative comparison of heterogeneity measures (I², τ²) based on absolute differences.
    RESULTS: Seven studies, each reporting a multilevel meta-analysis, comprising more than 40 pooled effect estimates were included. ChatGPT-derived pooled estimates showed close numerical agreement with human-derived results, with small median absolute differences for continuous (2.25) and binary outcomes (0.005) and minimal overall bias. Heterogeneity estimates demonstrated similar concordance, with predominantly small differences that did not alter qualitative interpretation. Larger deviations were limited to a small number of outcomes with extreme heterogeneity or inconsistent input structures.
    CONCLUSION: ChatGPT can reproducibly approximate the results of multilevel meta-analyses under predefined analytical conditions. These findings support the potential role of large language models as adjunct tools for reproducibility assessment and methodological validation in evidence synthesis, while emphasizing the continued need for expert oversight.
    Keywords:  ChatGPT; digital health; evidence synthesis; large language models; multilevel meta-analysis; reproducibility; research methodology
    DOI:  https://doi.org/10.1186/s12874-026-02879-0
  8. J Med Internet Res. 2026 Jul 10. 28 e85840
    Artificial Intelligence for Evidence Synthesis of Emerging Biologics to Improve Skeletal Health in Osteogenesis Imperfecta: Systematic Review and Meta-Analysis.
    Chengfei Li, Zonglin Dai, Wing Chung Tang, Zesen Gao, Vivien Kin Yi Chan, Mariana Ramirez-Posada, Jiyeong Kim, Eleni Linos, C L Cheung, Ian Chi Kei Wong, Dong Dong, Michael To, Dawn Craig, Xue Li.
       Background: Osteogenesis imperfecta (OI) is a rare genetic disorder characterized by bone fragility and recurrent fractures. Emerging biologics demonstrate promise by targeting bone-remodeling pathways, yet evidence for their efficacy and safety remains fragmented and heterogeneous, and no prior systematic review in OI has incorporated artificial intelligence (AI) to synthesize it.
    Objective: This study aims to systematically evaluate the efficacy and safety of novel biologics in patients with OI using an AI-assisted workflow for evidence synthesis.
    Methods: We conducted a systematic review and meta-analysis of interventional trials of denosumab, setrusumab, teriparatide, romosozumab, and fresolimumab. Data were retrieved from PubMed, Web of Science, Embase, ScienceDirect, the Cochrane Library, and ClinicalTrials.gov up to December 1, 2025. Eligible studies enrolled individuals with OI, reported areal bone mineral density (aBMD) and/or fractures, and were randomized, nonrandomized, or single-arm studies; case series were excluded. As a methodological feature, GPT-4o was integrated into the workflow to perform a parallel 2-stage screening (title/abstract and full text) and to assist with risk of bias assessment using an adapted Cochrane RoB 2 tool. The primary outcome, percentage change in aBMD, was synthesized using a random-effects meta-analysis. GPT-4o was benchmarked against human reviewers using sensitivity, specificity, and weighted Cohen κ.
    Results: Thirteen trials (n=684) were systematically reviewed, of which 10 (n=333) contributed to meta-analyses. In children, denosumab produced the greatest 12-month increase in lumbar spine aBMD (25.49%, 95% CI 17.14%-33.84%). In adults, setrusumab at 12 months yielded the highest improvement (9.38%, 95% CI 6.5%-12.26%). Across trials, no biologic significantly reduced fracture incidence compared to bisphosphonates. Safety profiles varied: denosumab was associated with a high risk of hypercalcemia in children (30.95%), whereas setrusumab had no treatment-related serious adverse events. AI achieved high sensitivity in abstract (97.4%) and full-text (88.9%) screening, and reduced total screening time by over 95%. Although there was substantial agreement with humans in the quality assessment (Cohen κ=0.778, 95% CI 0.710-0.846), the model exhibited optimism and positional biases due to reliance on probabilistic language patterns rather than structured clinical reasoning.
    Conclusions: This review is the first to synthesize and quantitatively compare skeletal outcomes across multiple biologics in OI with an AI-assisted review workflow. Denosumab and setrusumab demonstrate promising efficacy in improving lumbar spine aBMD across ages, although current evidence does not support superior fracture reduction over bisphosphonates. GPT-4o can substantially accelerate evidence synthesis but should be deployed with explicit human oversight in tasks requiring contextual understanding and clinical reasoning. These findings should be interpreted cautiously given the small and heterogeneous trial base. Taken together, our workflow presented how evidence synthesis may be scaled and operationalized in real-world rare disease research.
    Keywords:  ChatGPT; artificial intelligence; biologics; evidence synthesis; osteogenesis imperfecta
    DOI:  https://doi.org/10.2196/85840
  9. BMC Med Res Methodol. 2026 Jul 08.
    Leveraging generative artificial intelligence for the development of non-interventional research study protocols: a proof-of-concept feasibility study.
    Julia Langham, Emma Benbow, Tim Reason, Bill Malcolm, Andy Gimblett, Cheryl Jones, Nathan R Hill.
       BACKGROUND: The writing of study protocols is a labor and time-intensive process. We hypothesized that the writing of some methodological aspects of study protocols for non-interventional studies (NIS) could be amenable to automation using generative artificial intelligence (GenAI), particularly when provided with standardized templates, and a set of common elements, such as study design and objectives.
    METHODS: In this proof-of-concept feasibility study, we explored whether a large language model (LLM), specifically GPT-4, could support the drafting of selected methodological sections of a study protocol for retrospective observational NIS. The sections to be populated were the objectives, study design, study population, data source, data collection methods, statistical analysis methods and study strengths and limitations. A Python application programming interface (API) was used to send instructions, "prompts", to GPT-4 including guideline instructions, examples of text and specific study inputs related to the research question, including study population, objectives, and data source/data collection methods. The LLM-based program was first tested using four case studies. The program was then revised and refined using an additional four previously unseen case studies, to explore whether the prompting framework could be applied across different objectives, populations and datasets within the defined scope of retrospective observational NIS (quality assessment set, wave 1). Finally, the revised program was applied to a further five previously unseen protocols as a final feasibility assessment (quality assessment set, wave 2). A critical appraisal of the nine protocols populated by GPT-4 in the quality assessment stages was conducted to explore the alignment of the GPT-4 text in relation to the original human-written protocols and against standard guidelines. In addition, two GPT-4 written protocols were blind-reviewed (i.e. "author" unknown) through the Company's routine internal scientific review process.
    RESULTS: The critical appraisal of the nine GPT-4-produced protocols from the quality assessment sets suggested that the GPT-4 text aligned well with both the original human produced content and with content required by guidelines. This was further demonstrated by the two blind-reviewed protocols which were approved with only minor comments.
    CONCLUSIONS: Considerable effort and time (~ 3-6 months) were required to develop the structured prompt engineering workflow with prompts that were transferable to other NIS protocols. When provided with integral details to specify the elements of the study question (via human input), GPT-4 showed promising alignment with human-authored reference protocols under a constrained use case. Critical human input remains essential to define the study question, provide structured inputs, and review and revise the generated text as is current practice for human-written protocols.
    Keywords:  Artificial intelligence; Automation; Generative artificial intelligence; Large language model; Non-interventional studies; Real-world evidence; Study protocol
    DOI:  https://doi.org/10.1186/s12874-026-02937-7
  10. Psychol Bull. 2026 Apr;152(4): 404-419
    Tackling challenges in large language model-based data extraction via context engineering: A commentary on Jansen et al. (2025).
    Junsong Lu, X T XiaoTian Wang.
      Systematic reviews, particularly meta-analyses, involve crucial yet labor-intensive and error-prone stages of data extraction. Recent advances in large language models (LLMs) have unlocked new avenues for automating this process, potentially enhancing both efficiency and reliability. Recently, Jansen et al. (2025) systematically evaluated the accuracy and error patterns of LLM-assisted data extraction across 22 reviews published in Psychological Bulletin. Their findings indicated that while achieving acceptable-to-good accuracy for some variables describing study characteristics, LLMs struggled with numerical variables, especially those related to effect sizes. In this commentary, we discuss the current challenges of automated data extraction and potential pathways to improve the work reported in Jansen et al.'s study. We situate our discussion within the framework of context engineering, aiming to refine the information provided to LLMs through dynamic optimization strategies tailored to specific tasks. We identify five key challenges that reflect either LLMs' unique patterns or standard practices in research synthesis: parsing semistructured data, understanding long contexts, performing arithmetic induction, engaging in complex reasoning, and ensuring the reproducibility of coding protocols. We then outline potential solutions inspired by context engineering implementations such as retrieval-augmented generation and tool-integrated reasoning. For illustration, we present four examples: extracting semistructured data via optical character recognition, reliably computing effect sizes through function calls, performing adaptive retrieval with LLM-based agents, and iteratively improving outputs through self-refinement. We conclude by calling for future research in automated data extraction to advance beyond simple instruction-following paradigms toward more reliable forms of context engineering. (PsycInfo Database Record (c) 2026 APA, all rights reserved).
    DOI:  https://doi.org/10.1037/bul0000520
This page is being maintained by Thomas Krichel. It was last updated on 2026‒07‒12 at 0:13.