bims-arines Biomed News
on AI in evidence synthesis
Issue of 2026–11–15
ten papers selected by
Farhad Shokraneh, Systematic Review Consultants LTD



  1. Cancer Causes Control. 2026 Nov 09. pii: 192. [Epub ahead of print]37(12):
       PURPOSE: The 2018 World Cancer Research Fund (WCRF)/American Institute for Cancer Research Third Expert Report (TER) on diet, adiposity, physical activity, and risk of 19 cancers could be enhanced with new data. A framework is needed to prioritize future systematic reviews.
    METHODS: We searched PubMed (January 2019-February 2024) for meta-analyses, pooled analyses, randomized controlled trials (RCTs), Mendelian randomization (MR) studies, and large (> 100,000 participants) cohort studies. We assessed TER findings using conditional power (CP) and fail-safe number (FSN) statistics. We developed an exposure-based prioritization score (PS) by awarding or subtracting points considering the quantity, statistical significance, direction, and novelty of associations.
    RESULTS: We compared 366 meta-analyses, 121 pooled analyses, 19 RCTs, 174 MR studies, and 391 cohort studies covering 151 exposures and 28 cancers with 1,371 TER meta-analyses. Based on CP, nonsignificant TER associations likely to become significant with additional evidence included folate and colorectal, waist circumference and lung, total fat and ovarian, tea and ovarian, and red meat and kidney cancers. The FSN indicated that most significant TER associations are unlikely to change with additional evidence. The median PS was 6 (range: - 15 to 163), with top scores observed for anthropometric measurements (PSheight=40 to PSBMI=163), physical activity (PS=100), sedentary behavior (PS=64), alcohol (PS=52), tea (PS=36), dietary fiber (PS=31), milk/dairy (PS=29), micronutrients (PSretinol=27 to PSiron=38), vitamins (PSB12=22 to PSvitD=91), soy (PS=24), isoflavones (PS=23), and sugar-sweetened beverages (PS=22).
    CONCLUSION: The prioritization framework can help identify impactful systematic reviews to complement TER conclusions and enhance our understanding of emerging research.
    Keywords:  Cancer; Modifiable lifestyle factors; Prioritization framework; Systematic literature review; WCRF International CUP Global
    DOI:  https://doi.org/10.1007/s10552-026-02268-4
  2. JMIR Bioinform Biotechnol. 2026 Nov 09. 7 e99716
       Background: The assessment of the risk of bias (RoB) of each study is a necessary but labor-intensive step in conducting a systematic review (SR). There is a potential for AI to reduce the time and effort needed for RoB assessment and improve consistency between reviewers, but little is known about the accuracy of AI-based RoB assessments.
    Objective: In this study, we aim to evaluate the performance of a large language model (LLM) when given the questions from the RoB tool and the relevant passages from the full-text papers in assessing RoB in observational studies of environmental exposures and health outcomes.
    Methods: We evaluated the performance of GPT-5-mini on 128 observational studies from a SR of per- and polyfluoroalkyl substances and health outcomes conducted by the National Academies of Sciences. The navigation guide (NavGuide), a domain-based RoB tool designed for environmental studies, was used. For each paper-domain pair, the LLM was provided with the questions and guidance from the NavGuide and the human-identified text passages addressing that domain. The LLM returned a structured RoB rating, which was compared to the human-adjudicated ratings to quantify agreement and identify patterns of discrepancy.
    Results: GPT-5-mini RoB assessments for exact matches (64.5%; κ=0.199, Gwet AC2=0.882, F1-score=0.311) remained lower than the concordance between humans and the reference standard (87.5%-91.4%). Performance improved substantially for partial matches, reflecting agreement in direction but not necessarily in magnitude (eg, "low" and "probably low"), with percent concordance of 91.9% (κ=0.241, Gwet AC2=0.910, F1-score=0.614; for humans, 97.9%-98.6% concordance). Performance varied across the NavGuide domains, with the worst performance in domain 1 (selection bias) and the best in domain 4 (outcome assessment). The models tended to be more stringent than humans, often assigning higher RoB ratings (eg, "probably low" when humans assigned "low").
    Conclusions: Our results suggest that although concordance was moderate and inconsistent across domains, LLMs may be sufficiently accurate for use with human oversight. LLMs can flag cases for human adjudication or consistency checking but are not ready for use as an independent second reviewer without human oversight.
    Keywords:  AI; environmental health; large language models; risk of bias assessment; systematic review methods
    DOI:  https://doi.org/10.2196/99716
  3. J Med Internet Res. 2026 Nov 13. 28 e108001
       Background: Given that randomized controlled trials (RCTs) are the gold standard for effectiveness research, assessing their risk of bias (RoB) becomes an indispensable step in systematic reviews. Yet this task remains time-consuming and prone to reviewer subjectivity. While large language models may accelerate the process, they lack transparent, traceable judgments linked to trial reports.
    Objective: To bridge methodological gaps and improve both efficiency and consistency, this study developed and externally validated TRoB-1, a fine-tuned open-weight large language model workflow for RoB assessment of RCTs.
    Methods: TRoB-1 was developed specifically for the Cochrane risk of bias tool version 1 (RoB 1), which is the most widely adopted instrument for critical appraisal of RCTs. The workflow generates preliminary judgments and supporting quotations from reviewer-selected passages for the target RoB domains. The workflow was built on 2-stage supervised fine-tuning of Gemma 2-9B-Instruct using 4-bit quantized low-rank adaptation. The training set comprised 4321 RoB records extracted from systematic reviews of chronic obstructive pulmonary disease, diabetes, and hypertension. The validation set consisted of 161 records from recently published RCTs covering a broader disease area. The reference standard was established by 2 independent reviewers. Two clinicians blinded to the reference standard verified TRoB-1-assisted assessments through a locally hosted web interface. Using the same reference standard, the performance of TRoB-1 raw outputs was compared with that of the Gemma 2 base workflow, and clinician-verified outputs were compared with raw TRoB-1 outputs.
    Results: TRoB-1 successfully generated structured outputs for all 161 validation records. In independent clinician-assisted assessments using the same workflow, raw TRoB-1 accuracy ranged from 73.3% (n=118) to 73.9% (n=119), compared with 52.2% (n=84) to 60.2% (n=97) for the Gemma 2 base model. After verification, accuracy ranged from 82.0% (n=132) to 84.5% (n=136) and macro-F1 from 0.709 to 0.776. Accuracy remained higher for raw TRoB-1 after excluding invalid base-model outputs, and verification did not change any correct judgment to an incorrect one. Clinician 1 corrected 13 initially incorrect judgments and clinician 2 corrected 18. The F1-score for judgments of high-RoB increased from 0.600 to 0.784 for clinician 1 and from 0.571 to 0.821 for clinician 2.
    Conclusions: TRoB-1 combined task-specific fine-tuning, evidence-linked outputs, and clinician verification in a transparent workflow for reviewer-supported RoB assessment. The TRoB-1 web interface provides a practical platform for integrating RoB into evidence synthesis, particularly for living systematic reviews where timely updates are critical.
    Keywords:  large language model; randomized controlled trial; risk of bias; validation; workflow
    DOI:  https://doi.org/10.2196/108001
  4. J Biomed Semantics. 2026 Aug 12. pii: 24. [Epub ahead of print]17(1):
       BACKGROUND: The rapid expansion of biomedical literature has made comprehensive manual synthesis increasingly difficult, creating a pressing need for AI systems capable of reasoning over verified evidence rather than merely retrieving it. Existing retrieval-augmented generation (RAG) methods often struggle with complex biomedical queries that require accurate citation, iterative reasoning, and multi-step evidence synthesis. To address this challenge, we developed Queryome, a deep research system composed of specialized large language model (LLM) agents that dynamically adapt their orchestration to the complexity of each query. Queryome operates on a hybrid semantic-lexical retrieval engine covering 28.3 million PubMed abstracts and performs iterative, evidence-grounded synthesis through a hierarchical multi-agent architecture.
    RESULTS: On the MIRAGE benchmark, Queryome achieved 89.93% accuracy, outperforming the best prior system by up to 9.96% points. On the biomedical subset of Humanity's Last Exam (HLE), it improved reasoning accuracy from 14.6% to 24.9% relative to the base reasoning model. In a review article construction task, Queryome achieved the highest composite score among deep research systems from OpenAI, Google, Perplexity, and Scite.ai, demonstrating strong capabilities in literature retrieval and synthesis.
    CONCLUSIONS: Queryome demonstrates that hierarchical, dynamically orchestrated multi-agent systems can substantially enhance evidence-based reasoning and synthesis over biomedical literature. Its consistent performance gains across diverse benchmarks suggest that agentic RAG architectures offer a promising path toward reliable, citation-grounded biomedical knowledge synthesis at scale.
    Keywords:  Agentic AI; Agentic RAG; Deep research; Large language models; Literature mining
    DOI:  https://doi.org/10.1186/s13326-026-00366-5
  5. Brief Bioinform. 2026 Nov 02. pii: bbag624. [Epub ahead of print]27(6):
      Identifying interactions between biological entities is a cornerstone of molecular research, but assembling such lists from the literature is slow and tedious. For many research questions, no curated database exists, leaving researchers to survey the relevant literature themselves. We present interaction-finder, a tool that automates this process: given a topic string and user-defined entity types, it discovers relevant literature through LLM-guided iterative search, extracts candidate associations from full-text articles, and produces a ranked list where every association is backed by quoted passages verified against the source text. A self-contained interactive HTML report enables rapid triage of the results. Evaluated across 60 topics in three domains (celltype-cellmarker, disease-gene, and ligand-receptor), interaction-finder recovers up to 4.3× as many known associations as the best of single-shot prompting and an off-the-shelf deep-research framework, with all extracted quotes verified against source text. To assess candidates unrecognised from the gold-standard databases, we scored each candidate using an independent LLM judge blind to the tool's reasoning. Across the three domains, unverified candidates score similarly to gold-standard associations. We find the gold-standard associations are enriched at the top of our ranked candidates, with an overall recall@20 of 0.61. Interaction-finder is freely available at https://github.com/tecosaur/interaction_finder under an MIT licence.
    Keywords:  biomedical relation extraction; large language models; literature mining
    DOI:  https://doi.org/10.1093/bib/bbag624
  6. Implement Sci. 2026 Aug 14. pii: 66. [Epub ahead of print]21(1):
       BACKGROUND: Artificial intelligence (AI) encompasses computational systems that perform tasks typically requiring human intelligence, including machine learning, generative AI, and agentic applications. The use of AI to support implementation activities is growing, but its applications and evaluation remain poorly characterised. We conducted the first cycle of a living scoping review to identify how AI is being used and evaluated across implementation science and practice.
    METHODS: We followed JBI and Cochrane guidance and reported findings per PRISMA-ScR and PRISMA-LSR. We searched six databases through April 6, 2026, and included sources describing or evaluating AI to support implementation research or practice activities. We excluded adjacent uses such as knowledge synthesis automation. We extracted study characteristics, AI approaches, implementation tasks, evaluation methods, outcomes, and risks, and synthesised findings descriptively.
    RESULTS: We identified 7,203 records and included 40 sources, 34 (85%) of which were published since 2021. Thirty-one sources described, developed, or evaluated a specific AI system, comprising 22 primary research articles, three protocols, three conference abstracts, and three other sources. The remaining nine conceptual, methodological, framework or review articles discussed potential applications of AI in implementation science. Across all 40 sources, AI was most often used or proposed to support evaluation (25/40), implementation strategy selection and tailoring (23/40), barrier and facilitator assessment (21/40), and implementation monitoring (20/40). Among the 31 sources involving a specific AI system, the use of conventional machine learning was most common (10/31), followed by generative AI (7/31), multicomponent AI systems or studies comparing AI approaches (5/31), and non-generative deep learning (3/31). Implementation-related outcomes were reported or prospectively specified in 25/31 sources, technical performance in 19/31, human-centred outcomes in 12/31, time or efficiency in 10/31, clinical or health system outcomes in 9/31, equity in 3/31, and cost or resource outcomes in 2/31. Risks were discussed in about half of sources, although assessment of harms and environmental or societal consequences was rare.
    CONCLUSIONS: AI is in a nascent stage of supporting implementation science and practice. Reported use is narrow and methodologically underdeveloped, and much routine use is likely unpublished. The field needs rigorous, comparative, prospective, and equity-attentive research to establish whether and how AI improves implementation methods, processes, and outcomes.
    REGISTRATION: Open Science Framework, May 2025: https://doi.org/10.17605/OSF.IO/2Q5DV.
    Keywords:  AI; Equity; Evaluation; Implementation science; Large language models; Machine learning; Natural language processing; Risks; Scoping review
    DOI:  https://doi.org/10.1186/s13012-026-01535-x
  7. Transl Androl Urol. 2026 Oct 30. 15(10): 378
       Background: Interest in regenerative therapies for erectile dysfunction (ED), including stem cell therapy (SCT), low-intensity shockwave therapy (LiSWT), and platelet-rich plasma (PRP), has increased despite limited evidence and investigational designations from professional societies. Concurrently, patients increasingly use artificial intelligence (AI) large language model (LLM) chatbots to obtain health information. We evaluated the completeness and guideline concordance of AI-generated responses to patient-style questions about regenerative ED therapies, and their consistency across repeated queries.
    Methods: We performed a cross-sectional evaluation of four general-purpose LLM platforms: ChatGPT-4o, Gemini 2.5, Claude 3.7, and Copilot. Six patient-style prompts addressing general inquiry, evidence appraisal, safety, and treatment-seeking behavior were submitted for each therapy (SCT, LiSWT, PRP) in two independent sessions on separate days, yielding 144 responses. The primary outcome was the presence of nine prespecified counseling elements using a binary checklist. The secondary outcome was concordance with the Sexual Medicine Society of North America (SMSNA) position statement, assessed across efficacy framing, recommendation against routine use, and regulatory-status accuracy. Within-platform consistency across two sessions was assessed using Cohen's kappa.
    Results: Most responses identified investigational or non-FDA-approved status (80.6%), recommended physician consultation (76.4%), and acknowledged limitations in current research (75%). References to society guidelines (14.6%), safety profiles (38.9%), alternative therapies (23.6%), and cost (31.9%) were uncommon. Peer-reviewed citations were present in fewer than half of responses (47.9%), despite frequent references to "research". Among the 130 responses that characterized efficacy, only 36% conveyed the guideline-concordant position that clinical efficacy has not been established for regenerative therapies. The majority of responses instead framed efficacy optimistically despite the lack of robust efficacy evidence. An explicit recommendation against routine clinical use appeared in only 18.8% of responses. Within-platform consistency was substantial for Copilot, Claude, and Gemini (κ=0.61-0.67) and moderate for ChatGPT (κ=0.54).
    Conclusions: General-purpose AI chatbots provide incomplete counseling regarding regenerative ED therapies and inconsistently reflect the core elements of the SMSNA position statement. Chatbot responses specifically contain gaps in disclosing safety profiles, approved alternatives, and cost. Deliberate integration of evidence-based guidance into consumer-facing LLMs is needed to prevent AI-driven misconceptions. Clinicians should assess AI use and address AI-derived expectations during clinical encounters.
    Keywords:  Erectile dysfunction (ED); artificial intelligence chatbots; consumer health information; extracorporeal shockwave therapy; platelet-rich plasma
    DOI:  https://doi.org/10.21037/tau-2026-0399
  8. JAMIA Open. 2026 Dec;9(6): ooag218
       Objectives: Conducting qualitative health research with marginalized populations presents persistent challenges related to trust, disclosure, and resource constraints. Traditional in-person interviews can be costly and difficult to scale, while participants may hesitate to share sensitive experiences openly. This study assessed the feasibility and acceptability of a large language model (LLM)-powered conversational interview system as an alternative to semi-structured qualitative interviews with marginalized populations in a community-engaged health research context.
    Materials and Methods: We developed a web-based, LLM-powered interview system informed by a community-based participatory research framework. The system was tested with 28 caregivers of children with asthma living in subsidized housing. Postinterview evaluation sessions and researcher observation notes were analyzed to assess the system's feasibility and acceptability and identify usability, accessibility, technical, and implementation considerations.
    Results: Participants reported that the system was easy to use and appreciated its empathetic and nonjudgmental tone. They also rated the system positively for clarity, comfort, and accessibility, indicating that conversational artificial intelligence (AI) systems can be adapted to support qualitative data collection in underserved communities. Identified challenges included intermittent technical errors, limited contextual adaptability, and variations in participants' comfort with digital technology.
    Discussion: Large language model-powered conversational interview systems, when guided by community engagement, ethical design principles, and human support, offer a feasible and acceptable approach to qualitative health research with underserved populations. Hybrid human-AI models may offer a scalable yet inclusive pathway for expanding participation and advancing health equity in qualitative research.
    Conclusion: These findings support further development and evaluation of LLM-powered interview systems for community-engaged qualitative research. Ongoing technical improvement, ethical safeguards, and participatory design will be critical to ensuring these systems are used responsibly and effectively to advance health equity.
    Keywords:  artificial intelligence; community-based participatory research; health equity; large language model; semi-structured interview
    DOI:  https://doi.org/10.1093/jamiaopen/ooag218
  9. BMC Health Serv Res. 2026 Aug 12. pii: 1571. [Epub ahead of print]26(1):
       BACKGROUND: Qualitative research is essential for shaping person-centered public health policy and practices, particularly within complex care systems like transplant care, which involve multidisciplinary teams managing clinically complex patients through coordinated, multi‑step care amid variation in practices, policies, and partnerships across centers. However, its time and resource demands limit scalability and timeliness. Generative artificial intelligence (AI) technologies offer unique opportunities to transform and accelerate qualitative research; yet strengths and limitations of its use in transplant research remain unclear.
    METHODS: Two case studies compared AI-Media's LEXI Recorded (January-July 2024 version) and Microsoft Copilot (July 2025 version) against human researchers for transcription and thematic analysis, respectively. Fourteen semi-structured interviews with dialysis and transplant center leaders (January-July 2024) were transcribed using LEXI Recorded, with quality assurance conducted by researchers. Comparisons between initial AI-generated transcripts and final transcripts focused on time and labor costs, document formatting, speaker identification, flow and coherence, content accuracy, and redaction of identifiers. Transcription accuracy was evaluated using character error rate (CER) relative to final transcripts. For thematic analysis, two qualitative researchers manually coded 616 open-ended survey responses from dialysis staff; subsequently, an analyst applied the same methods using Microsoft Copilot. Comparisons focused on thematic overlap, efficiency, and dimensions of Lincoln and Guba's trustworthiness framework, including credibility, transferability, dependability, and confirmability.
    RESULTS: AI transcription produced complete transcript drafts with an average CER of 1.77% (range: 0.82%-2.87%), demonstrating strong but variable accuracy and an estimated savings of four hours and $362 (~ 17 min and $26 per transcript). However, AI-generated transcripts performed worse in document formatting, flow and coherence, speaker identification, accuracy, and redaction of identifiers. While thematic analysis comparisons revealed overlapping themes, notable differences emerged across trustworthiness dimensions. Manual analysis offered context-sensitive insights, clear attribution of perspectives, and synthesis across social and systemic factors, incorporating emotional resonance and theoretical frameworks to inform policy and practice. AI-generated themes were compartmentalized, with limited connection to broader implications, often flattening complex human experiences.
    CONCLUSIONS: Contemporary AI tools offer significant efficiency gains in transcription and analysis but fall short in contextual richness. Findings support using structured, researcher-led hybrid human-generative AI workflows to ensure ethical, rigorous qualitative research using AI.
    Keywords:  Generative artificial intelligence; Kidney transplantation; Qualitative research; Thematic analysis; Transcription
    DOI:  https://doi.org/10.1186/s12913-026-15350-z
  10. medRxiv. 2026 Oct 21. pii: 2026.10.20.26365969. [Epub ahead of print]
      Biomedical artificial intelligence (AI) increasingly relies on heterogeneous data sources, yet differences in data structure, terminology, evidence quality, and provenance complicate their integration into reliable knowledge resources. Here, we develop an evidence-aware framework for processing and integrating heterogeneous pharmacovigilance data from FDA Adverse Event Reporting System (FAERS) reports, biomedical literature, clinical trials, and electronic health records in Mass General Brigham. The framework combines large language model (LLM)-based information extraction, ontology-driven semantic normalization, cross-source evidence assessment, statistical and machine learning-based prioritization, and ingredient-based adverse drug event (ADE) relationship inference to transform heterogeneous evidence into a structured and evidence-based knowledge resource. The constructed knowledge system contains 11,588 drug nodes, 44,843 ingredient nodes, 3,275 ADE nodes, 993 indication nodes, and more than 2.47 million relationships, with each drug-ADE relationship assigned a confidence score. We further developed a drug-level web interface for accessing the knowledge resource and validated its utility in downstream AI tasks. Specifically, incorporating the developed knowledge resource improved the comprehensiveness, relevance, and overall quality of LLM-based drug-specific ADE synthesis and improved the precision and F1 score of clinical ADE relation extraction. These results demonstrate that AI-assisted processing combined with evidence-aware curation can transform heterogeneous biomedical evidence into reusable, AI-ready knowledge with explicit provenance and evidence strength information. The framework provides a general strategy for integrating heterogeneous biomedical data to support reproducible knowledge construction and downstream biomedical AI applications.
    DOI:  https://doi.org/10.64898/2026.10.20.26365969