800-Participant Randomized Experiment: Source Transparency Plus Accuracy Warning Cut Acceptance of Wrong Answers From 45% to 21%
A randomized online experiment published in Scientific Reports on November 14 tested whether visible source information and an accuracy warning support more selective reliance on generative-AI answers. The study, by Xing, Zhang, Zhang and colleagues, was received on August 14 and accepted on November 11.
The final analytic sample was 800 adult participants assigned to one of four cue conditions. Each participant evaluated 24 AI-style answers, yielding 19,200 trial-level observations, in a design crossing source transparency (absent vs. present), accuracy warning (absent vs. present), and answer correctness (incorrect vs. correct, within participant).
Participants accepted correct answers more often than incorrect answers, and the correct-incorrect acceptance gap was largest in the source-plus-warning condition. In the model restricted to incorrect-answer trials, acceptance of incorrect answers was highest in the control condition at 45% (1080/2400), and lower in the source-only condition at 36% (864/2400), the warning-only condition at 28% (672/2400), and the source-plus-warning condition at 21% (504/2400). Verification behaviour was highest in the combined condition, at 52% (2496/4800).
The authors said brief on-screen cues lowered erroneous-answer acceptance and encouraged verification in low-risk information-evaluation tasks, while noting that the distinction between response bias and sensitivity remains an important boundary condition.
Source: Scientific Reports
Published by GSM · USA (original Chinese edition)
Photo: 图：ChatGPT 的对话界面截图，实验中的生成式答案即以此类界面呈现。作者：Lugab89，CC BY 3.0，维基共享资源。 / Photo: A ChatGPT conversation interface screenshot; the generative answers in the experiment were presented in interfaces of this kind. Credit: Lugab89, CC BY 3.0, via Wikimedia Commons.
