bims-arines Biomed News
on AI in evidence synthesis
Issue of 2026–10–18
fifteen papers selected by
Farhad Shokraneh, Systematic Review Consultants LTD



  1. Res Synth Methods. 2026 Oct 15. 1-10
      Systematic reviews are the gold standard for evidence synthesis, but the manual title and abstract screening of thousands of references creates a severe bottleneck. Existing automated tools have historically struggled to achieve the near-perfect recall (sensitivity) required for reliable reviews. We developed MechaScreener as a 'zero-shot' automated screening tool that utilises a large language model to rank article relevance. The tool requires no initial training data or manual pre-screening, as MechaScreener directly applies user-provided question elements (PICO) or inclusion/exclusion criteria to assign an ordinal relevance score (1-5) to each reference. We evaluated the tool in two phases: a development phase using five reference libraries to optimise prompts, and an independent evaluation phase using 10 diverse Cochrane review libraries (comprising both randomised controlled trials [RCTs] and non-RCTs) containing over 58,000 references. In the evaluation dataset, MechaScreener achieved a perfect recall of 1.00 (100%) in all 10 reviews, ensuring that no relevant articles were missed (pooled record-level 95% CI: 0.98-1.00). Concurrently, it achieved an overall macro-average specificity of 0.61 (61%, cluster bootstrap 95% CI: 0.46-0.75; pooled record-level specificity 0.59, 95% CI: 0.59-0.60). Specificity varied from 0.21 in broad public health topics to 0.91 in precise pharmacological interventions, reflecting the tool's built-in conservatism when evaluating ambiguous abstracts. By safely eliminating over 60% of irrelevant literature during the initial screening phase without compromising recall, MechaScreener functions as a highly reliable but low-effort 'first-pass' filter, allowing researchers to substantially reduce manual workloads and reallocate resources towards full-text review and data extraction.
    Keywords:  artificial intelligence; automation; evidence synthesis; large language models; literature screening; systematic reviews
    DOI:  https://doi.org/10.1017/rsm.2026.10124
  2. Work. 2026 Oct 16. 10519815261478123
      BackgroundThe emergence of Academic Artificial Intelligence (AAI) agents is reshaping Systematic Literature Reviews (SLR) by expanding evidence screening and synthesis beyond Boolean, metadata-based searches, but its use requires a clear, PRISMA-aligned human oversight model to avoid compromising research integrity.ObjectiveThis opinion article proposes the "AI-augmented researcher" as a multidimensional framework that synergizes human analytical rigor with Large Language Models (LLMs). Using a physical ergonomics case study, the paper establishes a normative integration strategy that demonstrates how AAI can accelerate evidence synthesis while maintaining strict PRISMA compliance under explicit human oversight.MethodologyA Workflow Comparative Analysis (WCA) applying a parallel-group design to physical ergonomics SLR. The procedure synchronized two approaches: 1) A "classical" SLR workflow, using Boolean syntax and paratextual screening (Metadata) across indexed databases; and 2) An AAI-driven workflow, utilizing LLM-based immersion via natural language prompting. Comparative metrics analyzed included execution time, record processing volume, and the identification of emerging trends-such as Ergonomics 4.0-benchmarked against the PRISMA 2020 protocol.ResultsThe AAI-augmented workflow reduced execution time by 96% (39 h to 1.5 h) while increasing record processing by 370%. Qualitatively, iterative prompting fostered a "hermeneutic dialogue," enabling the detection of complex "Ergonomics 4.0" trends obscured in syntactic searches.ConclusionsAAI serves as a cognitive enhancer, reconfiguring the researcher from a "data miner" to an "evidence curator". While efficiency is optimized, strict human oversight remains indispensable to mitigate algorithmic hallucinations and ensure ethical scientific integrity.
    Keywords:  artificial intelligence; efficiency; ergonomics; human factors; information storage and retrieval; systematic review; technology; work
    DOI:  https://doi.org/10.1177/10519815261478123
  3. J Med Syst. 2026 Oct 15. pii: 150. [Epub ahead of print]50(1):
      Large language models (LLMs) have demonstrated human-level performance in literature review screening phases and have been evaluated in systematic-review workflows. However, LLM performance in scoping reviews, which are broader and more heterogeneous, remains understudied. We developed a modular LLM-supported framework for scoping-review workflows and compared the performance of two general-purpose LLMs against human reviewers on a reference scoping review. The reference review's keyword search results (n = 1336) and prompt sets mirroring workflow stages were provided to GPT-5 and gpt-oss-20b (OpenAI, August 2025). Outputs were benchmarked against human decisions for agreement; disagreements were adjudicated. For title-abstract screening and final-inclusion concordance, unanimous and adjudicated decisions were combined into a composite reference standard. In title-abstract screening, GPT-5 and gpt-oss-20b workflows showed substantial interrater agreement with human consensus (AC1: 0.769; 0.722); reference concordance was 90.9% (p = 0.08) and 92.4% (p = 0.004) versus humans' 89.0%. For final inclusion status, interrater agreement was almost-perfect (AC1: 0.948; 0.853); reference concordance was 95.4% (p = 0.23) and 92.4% (p < 0.001) versus humans' 96.1%. Across data categorization/synthesis, interrater agreement was fair to almost-perfect (AC1: 0.493-0.949; 0.342-0.929); total variation distances (representing distribution divergence) were 1.55%-28.0%. All sources found comparably low definition-adherent use rates of the term "human digital twin" (humans 12.1%; GPT-5 13.0%; gpt-oss-20b 18.0%). Both LLM-supported workflows achieved fair to almost-perfect agreement and high screening/final-inclusion reference concordance. The main synthesis finding was directionally consistent across workflows, with all sources identifying infrequent correct use of "human digital twin"; total variation distance may be useful in assessing distribution-level concordance. Limitations include a single domain/vendor, fixed prompts, and predefined taxonomies; broader evaluations are warranted. In this single-domain benchmark, LLM-supported scoping-review workflows showed feasibility for supporting and potentially streamlining selected review stages, with broadly comparable performance on evaluated endpoints and conclusion-level concordance. Generalizability, reproducibility, and operational efficiency beyond this benchmark remain to be established.
    Keywords:  Abstract screening; Artificial intelligence; Large language models; Research questions; Scoping reviews
    DOI:  https://doi.org/10.1007/s10916-026-02477-z
  4. Front Res Metr Anal. 2026 ;11 1919972
      The rapid expansion of biomedical literature requires automated methods for accurate and efficient information extraction. This study addresses relation classification: given a pair of annotated biomedical entities in a research article title and abstract, assigning the relation that holds between them from a pre-defined set. We evaluate a retrieval-augmented generation (RAG) framework in which semantically similar annotated examples are retrieved from a training pool and supplied to a large language model as in-context demonstrations. Four open-source LLMs are compared against a supervised DeBERTa+CNN baseline on GutBrainIE, and four further corpora (ChemProt, BioRED, its BioCreative VIII release BC8, and DDI-2013) are used to test whether the findings generalize. We report macro- and micro-averaged precision, recall and F1-score, and the rate of predictions that fall outside the defined relation set. On GutBrainIE, Qwen3:14B with 25 retrieved examples reaches 0.877 macro and 0.866 micro F1, against 0.440 and 0.562 for the supervised baseline, while the smaller Llama-3.1-8B and Gemma3:12B models do not exceed the baseline on macro F1. Macro F1 rises from 0.105 without demonstrations to 0.877 with 25, with the steepest gains over the first ten retrieved examples and subsequent variation smaller than the run-to-run standard deviation. Predictions falling outside the defined relation set drop from 39 at zero shots to at most one between 20 and 35, so retrieval improves output validity as well as accuracy. Across the four further corpora these gains do not generalize uniformly: the best configuration exceeds the supervised baseline on ChemProt, improves macro but not micro F1 on BioRED and BC8, and falls below it on both measures on DDI-2013. Since DDI-2013 is the most balanced corpus in the set and, like ChemProt, is annotated within single sentences, that reversal is not explained by class imbalance or annotation scope, but is consistent with what the relation labels encode: whether they name relationships recoverable from biomedical knowledge, or boundaries set by an annotation guideline.
    Keywords:  GutBrainIE; Qwen; biomedical relation classification; few-shot learning; large language models; out-of-schema prediction; relation classification; retrieval-augmented generation
    DOI:  https://doi.org/10.3389/frma.2026.1919972
  5. Front Toxicol. 2026 ;8 1877128
      Artificial intelligence (AI), particularly in the form of large language models (LLMs), offers promising support for toxicological risk assessment (TRA), which integrates multiple lines of evidence to support health-based conclusions, including those used in regulatory decision-making. Effective use of LLMs in TRA requires a clear understanding of their strengths and limitations, especially in the context of regulatory expectations for transparency, reproducibility, and accountability. This work explores the emerging role of LLMs in TRA through three integrated components: an overview of LLMs in the context of TRA, an exemplar case study examining LLM-assisted derivation (mainly using ChatGPT) of the Permitted Daily Exposure (PDE) for acetyl tributyl citrate under the draft ICH Q3E guideline, and a workflow hypothesis for the integration of LLMs into TRA. Observations from the case study suggest that LLMs may support pattern recognition, targeted information retrieval, and rapid summarization of large bodies of text, facilitating literature triage, extraction of study details, and evidence summaries. At the same time, the case study highlights important limitations for LLMs. Observed variability in LLM outputs reflects inherent model stochastic behavior, model limitations, and the intrinsic complexity of TRA, where different experts may reasonably reach distinct, yet scientifically defensible conclusions based on the available evidence. The observations from the case study, interpreted in the context of the available literature, are used to inform a hypothesis that LLMs may be most effectively integrated into TRA through structured, modular, human-supervised workflows. This hypothesis reflects the view that the suitability of LLM-assisted TRA depends not only on LLM capabilities, but also on how LLMs are integrated into the assessment process to address both LLM variability and the interpretive nature of TRA. The proposed pragmatic workflow separates data search, information extraction, and integrated summarization while maintaining human-in-the-loop control over interpretation, expert judgment, and final conclusions.
    Keywords:  AI; LLM; PDE; human-in-the-loop; pragmatic workflow; toxicological risk assessment
    DOI:  https://doi.org/10.3389/ftox.2026.1877128
  6. EULAR Rheumatol Open. 2026 Sep;2(3): 100193
       Objectives: Evaluating compliance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement can be time-consuming and subjective. This study compares STROBE assessments from large language models (LLMs), a human reviewer panel, and the original manuscript authors in observational rheumatology research.
    Methods: Guided by the Guidelines for Reporting Reliability and Agreement Studies and Development, Evaluation, and Assessment of Large Language Model Pathway B frameworks, 17 rheumatology articles (11 cohort, 4 cross-sectional, and 2 case-control) were independently assessed. Evaluations used the 22-item STROBE checklist, completed by the authors, a 5-person human panel (ranging from junior to senior professionals), and 2 LLMs (ChatGPT-5.2 and Gemini 3 Pro). Interrater reliability was calculated using Gwet's Agreement Coefficient (AC1) with 95% CIs.
    Results: Overall agreement across all reviewers was 85.0% (AC1 = 0.826 [95% CI: 0.801-0.851]). Domain stratification showed almost perfect agreement for 'Presentation & Context' (AC1 = 0.841 [95% CI: 0.810-0.872]) and substantial agreement for 'Methodological Rigor' (AC1 = 0.803 [95% CI: 0.761-0.845]). Although LLMs achieved complete agreement with all human reviewers on standard formatting elements, their agreement declined on complex methodological items, with some pairwise comparisons yielding negative AC1 values. Intra-LLM and cross-version agreement across repeated independent runs was high, and estimates were stable across publication periods, providing no clear evidence of data leakage.
    Conclusions: While LLMs show potential for basic STROBE screening, their lower agreement with human experts on complex methodological items likely reflects a reliance on surface-level information. These models appear more reliable for standardising straightforward checks than for replacing expert human judgement in evaluating observational research.
    DOI:  https://doi.org/10.1016/j.ero.2026.100193
  7. Clin Trials. 2026 Oct 16. 17407745261491903
       BACKGROUND: Detecting discrepancies between registered and published trial outcomes currently requires manual expert review, limiting surveillance to small samples. No scalable automated system exists for continuous monitoring of outcome reporting integrity across the clinical trial literature. We aimed to develop and validate an automated screening pipeline for primary outcome discrepancies and to characterise its performance at scale.
    METHODS: We developed an automated pipeline that retrieves trial registrations from ClinicalTrials.gov, matches them to publications via PubMed and screens for primary outcome discrepancies using a term-matching algorithm. We applied the pipeline to all 46,365 completed interventional trials with posted results (2015-2025) and validated performance against manual expert review of 123 trial-publication pairs, with independent large language model re-review as a second rater.
    RESULTS: The pipeline processed 46,365 trials in under 48 h, matching 27,555 (59.4%) to publications and confirming 13,595 pairs via trial identifier verification. Automated screening classified 10,231 (75.3%) as concordant and flagged 3353 (24.7%) for review. Against manual review of 123 pairs, the screen achieved 100% specificity and 16.7% sensitivity (1 of 6 discrepancies); the flag's positive predictive value could not be estimated because the validation sample contained only one flagged pair. Independent large language model re-review showed 99.2% agreement (Cohen's kappa = 0.91). Manual review estimated a true discrepancy rate of 4.9% (95% confidence interval = 1.6%-8.9%), predominantly minor modifications (timeframe extensions, operationalisation changes) rather than substantive outcome switching. Automated flag rates were not associated with primary outcome statistical significance (21.0% vs 20.6%; p = 0.745).
    DISCUSSION: Automated screening can efficiently triage tens of thousands of trials for outcome reporting surveillance, deprioritising 75% of pairs as concordant and concentrating manual review on the flagged subset. The pipeline provides a foundation for continuous, near-real-time monitoring of outcome reporting integrity that could be integrated into journal peer review or regulatory workflows.
    Keywords:  ClinicalTrials.gov; automated surveillance; clinical trial registration; large language model; outcome reporting; research integrity; screening pipeline
    DOI:  https://doi.org/10.1177/17407745261491903
  8. Front Nutr. 2026 ;13 1924184
      Cardiometabolic diseases remain a primary global health challenge, with nutritional interventions serving as a critical pillar of management. However, the rapidly expanding volume of nutritional research, spanning high-dimensional multiomics, real-world continuous monitoring, and highly heterogeneous clinical trials, has created fragmented knowledge silos, leading to a profound translation lag between scientific discovery and clinical practice. To overcome this methodological bottleneck, we propose a conceptual framework for an "AI-Driven Evidence Ecosystem." This perspective outlines a four-phase closed-loop architecture: (1) AI-assisted mechanistic decoding of multiomics and the development of digital gut twins; (2) data-optimized clinical research integrating traditional trials with real-world digital biomarkers; (3) a methodologically agnostic, multi-agent living evidence synthesis engine capable of automating complex data extraction and mitigating the translation gap; and (4) the deployment of conversational AI for public knowledge translation and personalized patient empowerment. By addressing concurrent ethical challenges, algorithmic biases, and the risk of researcher de-skilling, this ecosystem paradigm leverages artificial intelligence not merely as an automation tool, but as an indispensable bridge. Ultimately, it transforms isolated, stagnant data into a dynamic, translatable evidence network, reshaping the future of precision cardiometabolic care.
    Keywords:  artificial intelligence; cardiometabolic nutrition; evidence synthesis; patient empowerment; precision medicine; real-world data
    DOI:  https://doi.org/10.3389/fnut.2026.1924184
  9. bioRxiv. 2026 Sep 29. pii: 2026.09.23.753935. [Epub ahead of print]
      Curated protein records and their supporting publications organize biological evidence at different levels of detail. Here, we used AI to independently answer a set of biological questions using either UniProt records or the full texts of the literature cited by those records, and then used AI-based evaluation to compare the resulting answers in terms of information coverage and semantic agreement. Fifty human proteins were selected from ten groups in a kinase sequence collection. Target-identity filtering retained 1,749 protein-publication records representing 1,655 distinct publications. For each protein, an AI language model generated one answer set from UniProt and another from the associated literature. A separate AI-based semantic evaluation compared all 2,500 pairwise combinations of the 50 UniProt-derived and 50 literature-derived answer sets, producing 87,500 question-level comparisons. Literature provided answers to 1,211 of 1,750 protein-question combinations, compared with 1,151 for UniProt. After correction for multiple testing across the 35 questions, literature coverage was significantly higher for autoinhibition and inhibitor type. Mean AI-assigned semantic agreement scores were higher for matched protein pairs (0.692) than for unmatched proteins within the same kinase group (0.384) and for proteins from different kinase groups (0.279). The matching literature-derived answer set ranked first for 49 of the 50 UniProt-derived queries, even after questions about gene symbol and kinase family were excluded. The sole exception was GUCY2C, whose UniProt-derived answer set showed slightly greater agreement with the NPR2 literature-derived answer set than with the GUCY2C literature-derived set. TYR proteins had the greatest overall answer coverage, whereas CK1 and RGC proteins had comparatively limited coverage in UniProt and the literature, respectively. Overall, AI analysis showed that UniProt records and their cited publications contain complementary yet strongly protein-specific biological information. Reporting gaps, question scope, and score aggregation influenced apparent agreement, underscoring that the framework evaluates consistency between AI-derived answers, not their biological accuracy.
    DOI:  https://doi.org/10.64898/2026.09.23.753935
  10. Integr Environ Assess Manag. 2026 Oct 10. pii: vjag163. [Epub ahead of print]
      Extracting and formatting data is among the most tedious elements of assessing environmental chemical occurrence and exposure. We present a toolkit (eData) to streamline this process, generate FAIR data, and maximise traceability and transparency: a FAIR Data Reporting Format (DRF) that provides data structure, vocabulary, and validation, and a Large Language Model (LLM)-based data extraction application that quickly formats literature data. Data and metadata for a study of interest can be rapidly extracted to the eData format by a range of commercially available LLMs, and visualised for checking in the application's graphical interface. This blends advantages of LLM extraction with human interpretation, speeding extraction without impairing reproducibility. Data can be exported from the application at any point in this process and saved locally. Furthermore, the application supports a streamlined Criteria for Reporting and Evaluating Exposure Datasets (CREED) data quality assessments that automatically compiles data relevant to the assessment for reporting. All resulting data can also be uploaded to the repository service Zenodo via a dedicated module. Our evaluation showed a substantial time saving per paper (from 3+ to around 1 hour) and extracted consistent data, passing 85% of format validation checks. Our toolkit is a valuable addition to the FAIRification of environmental chemical occurrence and exposure data.
    Keywords:  data extraction; data format; exposure assessment; large language model; pollution
    DOI:  https://doi.org/10.1093/inteam/vjag163
  11. Pharmacoecon Open. 2026 Oct 12.
       OBJECTIVE: Health economic models (HEMs) are crucial for assessing the cost-effectiveness of healthcare interventions, informing policy decisions and guiding reimbursement strategies. This study examines the feasibility of employing generative artificial intelligence (Gen-AI) for the conceptualisation of HEMs. Two proof-of-concept studies were conducted, implementing a first-of-its-kind agentic Gen-AI-augmented approach for model conceptualisation.
    METHODS: An agentic Gen-AI framework was developed, incorporating the chain-of-thought (CoT) reasoning algorithm through advanced prompt engineering, compounded with self-consistency, executed in an external Python programming environment. Within this framework, an agentic execution layer (the 'agent') autonomously orchestrated a multi-step retrieval-reasoning-generation task. Additionally, a graph-based retrieval-augmented generation (RAG) technique was implemented by linking the large language model (LLM) to an external database curated with user-selected data specific to HEM. Two case studies were explored: breast cancer, representing a data-rich disease area, and transthyretin amyloid polyneuropathy (ATTR-PN), exemplifying a data-sparse rare disease.
    RESULTS: The agentic Gen-AI framework successfully generated two key outputs that were subsequently validated against published literature and expert review from health economic (HE) modelers and clinicians: (1) a disease concept diagram illustrating the progression of the disease and its associated health states, and (2) a model structure diagram depicting patient transitions between various health states.
    CONCLUSIONS: The findings suggest that Gen-AI-augmented methodologies can enhance multiple aspects of HEM conceptualisation, including problem definition, disease and treatment effect pathway development, model structure recommendations, and drafting of model specification plans. To the best of the authors' knowledge, this is the first study to integrate cutting-edge Gen-AI techniques such as human-in-the-loop (HITL), chain of thought-self-consistency (CoT-SC), and graph-based RAG for HEM conceptualisation or broader health economics and outcomes research (HEOR) applications.
    DOI:  https://doi.org/10.1007/s41669-026-00687-2
  12. Nature. 2026 Oct;658(8136): 830-831
      
    Keywords:  Information technology; Machine learning; Research management; Technology
    DOI:  https://doi.org/10.1038/d41586-026-03227-4
  13. Elife. 2026 Oct 14. pii: e113137. [Epub ahead of print]15
      Large language models are very good at providing concise answers. But if we only see the answer without knowing the original question or the context behind it, that answer may not be very useful. The important issue is not simply how much information an AI summary removes but whether we can still recover the missing context. Thus, AI literature tools should also report how much information was removed and how recoverable their omissions are.
    Keywords:  AI; AI systems; citations; computational biology; large language models; none; systems biology; warrant
    DOI:  https://doi.org/10.7554/eLife.113137
  14. Acad Radiol. 2026 Oct 10. pii: S1076-6332(26)00782-8. [Epub ahead of print]
       RATIONALE AND OBJECTIVES: Large Language models (LLMs) are novel tools with promising applications in imaging selection for clinical scenarios, but information is limited on the accuracy of an LLM trained on a medically specific dataset compared to a generalized dataset. In this analysis, we assessed the accuracy of the imaging recommendations by LLMs ChatGPT-3.5, ChatGPT-5.0, and OpenEvidence for neuroradiology, using the American College of Radiology (ACR) Appropriateness Criteria as the reference standard.
    MATERIALS AND METHODS: Prompts from the ACR Appropriateness Criteria priority clinical areas within neurology were used to build open-ended (OE) or select-all-that-apply (SATA) questions. Each question was posed three times, and the responses were scored against ACR guidelines. The performance of each LLM on OE and SATA was compared, and then was further analyzed by acuity. Mean accuracy scores were calculated by averaging performance across OE and SATA questions for each model. Analysis of variance was then conducted to assess differences in performance among the models.
    RESULTS: A total of 78 prompts were assessed in 11 priority clinical areas. These areas included cerebrovascular conditions, neurological pain, and spine pathologies. Amongst SATA, Open Evidence performed the best (79.5%) compared to GPT5 (71.2%) and GPT3.5 (44.4%). For the OE prompts, Open Evidence was more accurate (86.3%) than GPT3.5 (69.2%) but not GPT5 (80.6%). Amongst the models, no differences were found based on acuity.
    CONCLUSION: Overall, OpenEvidence performed the best for both OE and SATA prompts. ChatGPT-5 performed better than ChatGPT3.5 for SATA prompts. LLM performance was unaffected by the acuity of the case.
    Keywords:  Artificial intelligence; Neuroradiology; Patient safety
    DOI:  https://doi.org/10.1016/j.acra.2026.09.049