Search

When deciding which submissions should be peer reviewed, eLife editors consider whether they will be able to find high-quality reviewers, and whether the reviews will be valuable to the scientific community.

This paper summarizes the 18th Academic Publishing in Europe (APE) Conference: Berlin Re-Visited: Building Technological Support for Scholarship and Scientific Publishing, held as a hybrid event 10 and 11 January 2023, and organized by the Berlin Institute of Scholarly Publishing (BISP), a not-for-profit organization dedicated to bringing publishers, researchers, funders, and policymakers together. This year's conference theme "Out with the old, in with the new!" was discussed and presented in keynote speeches, the APE lecture, and several panel discussions. Current challenges within scholarly publishing, e.g., with research integrity, trust, and research assessment, have much to do with the old ways of doing things. To move science forward, new technologies and innovations, like the decentralized web, FAIR digital objects, and blockchain technology are needed to shape new paradigms. In many sessions it was stressed that not just new technologies are needed to move science forward, but human collaboration and partnerships as well. The changing role of the journal and the importance to recognize more diverse research outputs, beyond the journal article, was a topic of importance. Not only research assessment reforms and improved collaboration amongst different stakeholder groups are needed to address this, new publishing systems, better metadata, and open infrastructures as well. A session presenting different start-ups showcased how Artificial Intelligence, Natural Language Processing and software technology can be used to tackle problems in e.g., finding relevant funders and peer reviewers, and detecting image plagiarism. It was also discussed what publishers can do to help achieve the Sustainable Development Goals. Collaboration, data transparency and sharing best practices amongst researchers, funders, and policy makers are key. Another important topic during this year's APE was how publishers can support Early Career Researchers: establishing new workflows and infrastructures that enable publication of a wider range of research outputs, and broader recognition of these outputs, will incentivize ECRs. This year's APE was concluded with the APE Award Ceremony. The winner - Vsevolod Solovyov from Prophy - has used AI to enhance the current peer review system and made a very important contribution to improving the scholarly communication system. [ABSTRACT FROM AUTHOR]

This paper is based on a presentation from a panel discussion on Innovation, Technology and Infrastructure at the 2023 APE Conference. It provides an overview of the use, misuse and value of bibliometrics in research assessment, examines the responsible use of metrics and the importance of going beyond existing bibliometric indicators of scholarly impact. [ABSTRACT FROM AUTHOR]

Assess whether ChatGPT 4.0 is accurate enough to perform research evaluations on journal articles to automate this time-consuming task. Test the extent to which ChatGPT-4 can assess the quality of journal articles using a case study of the published scoring guidelines of the UK Research Excellence Framework (REF) 2021 to create a research evaluation ChatGPT. This was applied to 51 of my own articles and compared against my own quality judgements. ChatGPT-4 can produce plausible document summaries and quality evaluation rationales that match the REF criteria. Its overall scores have weak correlations with my self-evaluation scores of the same documents (averaging r=0.281 over 15 iterations, with 8 being statistically significantly different from 0). In contrast, the average scores from the 15 iterations produced a statistically significant positive correlation of 0.509. Thus, averaging scores from multiple ChatGPT-4 rounds seems more effective than individual scores. The positive correlation may be due to ChatGPT being able to extract the author's significance, rigour, and originality claims from inside each paper. If my weakest articles are removed, then the correlation with average scores (r=0.200) falls below statistical significance, suggesting that ChatGPT struggles to make fine-grained evaluations. The data is self-evaluations of a convenience sample of articles from one academic in one field. Overall, ChatGPT does not yet seem to be accurate enough to be trusted for any formal or informal research quality evaluation tasks. Research evaluators, including journal editors, should therefore take steps to control its use. This is the first published attempt at post-publication expert review accuracy testing for ChatGPT. [ABSTRACT FROM AUTHOR]

Definition: In the context of open science, universities, research-performing and funding organizations and authorities worldwide are moving towards more responsible research assessment (RRA). In 2022, the European Coalition for Advancing Research Assessment (CoARA) published an agreement with ten commitments, including the recognition of the "diversity of contributions to, and careers in, research", the "focus on qualitative evaluation for which peer review is central, supported by responsible use of quantitative indicators", and the "abandon (of) inappropriate uses in research assessment of journal- and publication-based metrics". Research assessment (RA) is essential for research of the highest quality. The transformation of assessment indicators and procedures directly affects the underlying research information management infrastructures (also called current research information systems) which collect and store metadata on research activities and outputs. This entry investigates the impact of RRA on these systems, on their development and implementation, their data model and governance, including digital ethics. [ABSTRACT FROM AUTHOR]

The Research Excellence Framework (REF) follows a set of submission rules. Here we analyse whether the submission rules of the impact element arguably shaped the submission sizes of the submitting units – a group of academics researching on a specific subject area in higher education providers – to the REF in 2014. The number of impact case studies required was determined by the number of full-time equivalent (FTE) staff members submitted by the unit. We argue that units that did not have an extra impact case study or units' perception of lower quality of additional impact case study made some units to lower their submission sizes. We show that there were proportionately more submissions with a size just below the threshold FTE – a threshold used to identify the numbers of impact case studies – than those just above the threshold, suggesting that some units arguably decreased their size to return fewer impact studies. [ABSTRACT FROM AUTHOR]