JMIR Form Res. 2026 Nov 16. 10
e97813
Background: Automation can be a useful efficiency tool for saving time in systematic reviews, but a trust gap and a steep learning curve persist concerning its accuracy and ease of use. It remains unknown whether these adoption barriers apply to the more complex process of systematic gray literature reviews (SGLRs), especially given the rapid evolution of large language models.
Objective: This study seeks to identify the preferred features of SGLRs generated using an automation tool (automated SGLRs).
Methods: Participants aged 18 years or older with experience using evidence from systematic reviews were recruited. A cross-sectional online Best-Worst Scaling Questionnaire was distributed through the authors' contacts, professional networks, and social media platforms. Descriptive statistics and multinomial logistic regression were used to estimate the baseline model for preferred features of automated SGLRs, while mixed logit analyses were used to explore subgroup heterogeneity.
Results: A total of 168 respondents, comprising researchers, academicians, health care professionals, health policymakers, and postgraduates, completed the study. Multinomial logistic regression found that the top 3 preferred features for automated SGLRs were (1) high precision and sensitivity comparable to manual methods (best-worst [BW] score 269, and M-logit estimate B=2.26, odds ratio [OR] 9.60, 95% CI 7.72-11.93; P<.001), (2) acknowledgment and detailed explanation of the AI algorithms (BW score: 218, B=2.06, OR 7.85, 95% CI 6.35-9.7; P<.001), and (3) involvement of more than 2 independent human reviewers in the process (BW score: 118, B=1.54, OR 4.68, 95% CI 3.84-5.72; P<.001), when compared to the reference feature, which was the acceptance of automated SGLR output by peer colleagues. Subgroup analysis using a mixed logit model suggested that respondents aged 18 to 29 years exhibited the highest relative utility weights for detailed explanations (B=0.60, OR 1.81, 95% CI 1.29-2.55) and acceptance by everyone (B=0.35, OR 1.41, 95% CI 1.05-1.90) compared to other age groups. In contrast, academicians (B=1.36, OR 3.91, 95% CI 1.51-10.17) might prefer the detailed explanation feature of automated SGLRs more than those in other professions.
Conclusions: The study concludes that automation tool developers should prioritize the accuracy of automated SGLRs, ensure algorithmic transparency, and integrate human-in-the-loop processes, as respondents most strongly preferred these features, which may support user trust.
Keywords: best-worst scaling; desirable features; gray literature; machine learning; systematic review