Transformer Models Predict Paper Impact Metrics From Titles and Abstracts, Evaluated Across One Million Articles
A study published November 14 in Scientific Reports tested whether Transformer models can predict paper impact metrics from titles and abstracts alone. The authors, Cameron C. Young and Marc D. Succi, targeted three metrics: citations per year, Altmetric Score, and the relative citation ratio (RCR) from the U.S. National Institutes of Health iCite system.
Using BioBERT as the representative baseline model, the final evaluation covered one million general science and medical articles. On the medical dataset, BioBERT reached an F1 of 0.494, accuracy of 0.504, and MCC of 0.382 for citations per year; an F1 of 0.468 and MCC of 0.354 for Altmetric Score; and an F1 of 0.516 and MCC of 0.405 for iCite RCR.
The models performed best at distinguishing papers in the highest and lowest impact quintiles, and were weaker on the middle quintiles.
Source: Scientific
Published by GSM · USA (original Chinese edition)
Photo: 图：Transformer 模型的整体架构示意图，左侧为编码器、右侧为解码器。来源：Daniel Voigt Godoy (dvgodoy)，Wikimedia Commons，CC BY 4.0。 / Photo: Diagram of the overall Transformer model architecture, encoder on the left and decoder on the right. Source: Daniel Voigt Godoy (dvgodoy), Wikimedia Commons, CC BY 4.0.
