Search

Research assessment reform is crucial for the social sustainability of research infrastructures (RIs): RIs can only thrive in the long term if the researchers who contribute to their development and growth receive academic credit for the kind of work they do in and around research infrastructures. To put it bluntly, research infrastructures have a vested interest in supporting the reform of research assessment. But, conversely, ongoing attempts to reform research assessment can also benefit from the work of research infrastructures because RIs have a great deal of experience creating and maintaining public services for producing, curating and harvesting both traditional and non-traditional academic outputs. The goal of this paper is to outline DARIAH’s position on the importance of research assessment reform for thematic RIs and the importance of thematic RIs for research assessment reform at the European level.

Research assessment reform is crucial for the social sustainability of research infrastructures (RIs): RIs can only thrive in the long term if the researchers who contribute to their development and growth receive academic credit for the kind of work they do in and around research infrastructures. To put it bluntly, research infrastructures have a vested interest in supporting the reform of research assessment. But, conversely, ongoing attempts to reform research assessment can also benefit from the work of research infrastructures because RIs have a great deal of experience creating and maintaining public services for producing, curating and harvesting both traditional and non-traditional academic outputs. The goal of this paper is to outline DARIAH’s position on the importance of research assessment reform for thematic RIs and the importance of thematic RIs for research assessment reform at the European level.

The Openness Position Paper published by the Arqus European University Alliance emphasises that Arqus institutions, in line with the policies, roadmaps and strategies of the EU and a wide range of stakeholders, are striving jointly to make further progress towards realising Open Science. The Position Paper identifies and acknowledges aims and values of Open Science and relates them to values, principles, and standards shared by the Arqus Alliance, followed by a vision for a future with Open Science. In the interest of a nuanced picture, the Position Paper discusses not only desired effects, but also possible areas of tension related to Open Science. It presents a wide range of specific aims and recommendations for each of the eleven elements of Open Science defined by the Arqus Openness Task Force: Governance Publications (including Open Access) Data (including research data management, FAIR and Open Data) Infrastructures (including support staff, Open Science software and tools, repositories, Open Labs) Methods (including source code, preregistration, materials, workflows, protocols, lab notes) Awareness and training (including education of early-stage researchers) Evaluation (including Open Metrics, research assessment, Open Peer Review, rewards and incentives) Communication (including multilingualism) Citizen Science Open Education Open Innovation The Position Paper concludes with an annex that highlights the progress already made in the implementation and support of Open Science practices at Arqus institutions., Cofunded by the Erasmus+Programme of the European Union

Although regression models play a central role in the analysis of medical research projects, there still exist many misconceptions on various aspects of modeling leading to faulty analyses. Indeed, the rapidly developing statistical methodology and its recent advances in regression modeling do not seem to be adequately reflected in many medical publications. This problem of knowledge transfer from statistical research to application was identified by some medical journals, which have published series of statistical tutorials and (shorter) papers mainly addressing medical researchers. The aim of this review was to assess the current level of knowledge with regard to regression modeling contained in such statistical papers. We searched for target series by a request to international statistical experts. We identified 23 series including 57 topic-relevant articles. Within each article, two independent raters analyzed the content by investigating 44 predefined aspects on regression modeling. We assessed to what extent the aspects were explained and if examples, software advices, and recommendations for or against specific methods were given. Most series (21/23) included at least one article on multivariable regression. Logistic regression was the most frequently described regression type (19/23), followed by linear regression (18/23), Cox regression and survival models (12/23) and Poisson regression (3/23). Most general aspects on regression modeling, e.g. model assumptions, reporting and interpretation of regression results, were covered. We did not find many misconceptions or misleading recommendations, but we identified relevant gaps, in particular with respect to addressing nonlinear effects of continuous predictors, model specification and variable selection. Specific recommendations on software were rarely given. Statistical guidance should be developed for nonlinear effects, model specification and variable selection to better support medical researchers who perform or interpret regression analyses.

The breakthrough potentials of research papers can be explained by their boundary-spanning qualities. Here, for the first time, we apply the structural variation analysis (SVA) model and its affiliated metrics to investigate the extent to which such qualities characterize a group of Nobel Prize winning papers. We find that these papers share remarkable boundary-spanning traits, marked by exceptional abilities to connect disparate and topically-diverse clusters of research papers. Further, their publications exert structural variations on the scale that significantly alters the betweenness centrality distributions of existing intellectual space. Overall, SVA not only provides a set of leading indicators for describing future Nobel Prize winning papers, but also broadens our understanding of the similar prize-winning properties that may have been overlooked among other regular publications., 27 pages, 8 figures, 9 tables. Submitted to Frontiers in Research Metrics and Analytics

ObjectivesRecently, the results of two economic evaluations were published both of which seemingly demonstrate the cost-effectiveness of sofosbuvir-based regimens for the treatment of chronic hepatitis C genotype 1 infection in Germany. Both analyses were sponsored by the manufacturer of sofosbuvir and use a different methodology: Whereas one evaluation is based on a conventional cost-utility analysis, the other rests upon the efficiency-frontier method used by the German Institute for Quality and Efficiency in Health Care (IQWiG). The purpose of this study is to reanalysis the results of both economic evaluations in combination.DesignReanalysis of published decision modelling results.SettingPrimary care in Germany.ParticipantsPatients with chronic hepatitis C genotype 1 infection (treatment-naïve and -experienced, cirrhotic and non-cirrhotic).InterventionsSofosbuvir, other anti-hepatitis C virus drugs, and no treatment.Primary and secondary outcome measuresCost per unit of health benefit and cost per quality-adjusted life year.ResultsReanalysis of the results of both economic evaluations in combination reveals an unclear rationale for choosing the selected cost-effectiveness methods as well as a potential publication bias, favoring the product of the manufacturer. Based on the reanalysis, sofosbuvir is not cost-effective in treatment-experienced non-cirrhotic patients, potentially lacks cost-effectiveness in treatment-experienced cirrhotic patients, and is only partially cost-effective in treatment-naïve non-cirrhotic patients. Taken together, these results indicate a lack of cost-effectiveness in three quarters of the German patient population.ConclusionsTwo economic evaluations on sofosbuvir suggest, in combination, that sofosbuvir cannot be considered a cost-effective treatment in three quarters of the German patient population.

In our planned study, we shall empirically study the assessment of cited papers within the framework of the anchoring-and-adjustment heuristic. We are interested in the question whether citation decisions are (mainly) driven by the quality of cited references. The design of our study is oriented towards the study by Teplitskiy, Duede [10]. We shall undertake a survey of corresponding authors with an available email address in the Web of Science database. The authors are asked to assess the quality of papers that they cited in previous papers. Some authors will be assigned to three treatment groups that receive further information alongside the cited paper: citation information, information on the publishing journal (journal impact factor), or a numerical access code to enter the survey. The control group will not receive any further numerical information. In the statistical analyses, we estimate how (strongly) the quality assessments of the cited papers are adjusted by the respondents to the anchor value (citation, journal, or access code). Thus, we are interested in whether possible adjustments in the assessments can not only be produced by quality-related information (citation or journal), but also by numbers that are not related to quality, i.e. the access code. The results of the study may have important implications for quality assessments of papers by researchers and the role of numbers, citations, and journal metrics in assessment processes.

“There is no problem that a library card can't solve” according to author Eleanor Brown [1]. This advice is sound, probably for both life and science, but even the best tool (like the library) is most effective when accompanied by instructions and a basic understanding of how and when to use it. For many budding scientists, the first day in a new lab setting often involves a stack of papers, an email full of links to pertinent articles, or some promise of a richer understanding so long as one reads enough of the scientific literature. However, the purpose and approach to reading a scientific article is unlike that of reading a news story, novel, or even a textbook and can initially seem unapproachable. Having good habits for reading scientific literature is key to setting oneself up for success, identifying new research questions, and filling in the gaps in one’s current understanding; developing these good habits is the first crucial step. Advice typically centers around two main tips: read actively and read often. However, active reading, or reading with an intent to understand, is both a learned skill and a level of effort. Although there is no one best way to do this, we present 10 simple rules, relevant to novices and seasoned scientists alike, to teach our strategy for active reading based on our experience as readers and as mentors of undergraduate and graduate researchers, medical students, fellows, and early career faculty. Rules 1–5 are big picture recommendations. Rules 6–8 relate to philosophy of reading. Rules 9–10 guide the “now what?” questions one should ask after reading and how to integrate what was learned into one’s own science.

BackgroundMost research manuscripts are not accepted for publication on first submission. A major part of the resubmission process is reformatting to another journal's specific requirements, a process separate from revising the scientific content. There has been little research to understand the magnitude of the burden imposed by the current resubmission process.MethodsWe analyzed original research article submission requirements from twelve randomly selected journals in each of eight scientific and clinical focus areas from the InCites Journal Citation Reports database. From the 96 journals selected, we randomly identified three recently published manuscripts and sent surveys to those first and/or corresponding authors (288 total) to solicit information on time spent reformatting resubmissions and opinions on the process.FindingsThere was significant variation in manuscript submission requirements for journals within the same scientific focus and only 4% of journals offered a fully format-free initial submission. Of 203 authors responding (71.5% response rate), only 11.8% expressed satisfaction with the resubmission process and 91% desired reforming the current system. Time spent on reformatting delays most publications by at least two weeks and by over three months in about 20% of manuscripts. The effort to comply with submission requirements has significant global economic burden, estimated at over $1.1 billion dollars annually when accounting for a research team's time.InterpretationWe demonstrate that there is significant resource utilization associated with resubmitting manuscripts, heretofore not properly quantified. The vast majority of authors are not satisfied with the current process. Addressing these issues by reconciling reformatting requirements among journals or adopting a universal format-free initial submission policy would help resolve a major subject for the scientific research community and provide more efficient dissemination of findings.

The UK is one of the largest funders of health research in the world, but little is known about how health funding is spent. Our study explores whether major UK public and charitable health research funders support the research of UK-based scientists producing the most highly-cited research. To address this question, we searched for UK-based authors of peer-reviewed papers that were published between January 2006 and February 2018 and received over 1000 citations in Scopus. We explored whether these authors have held a grant from the National Institute for Health Research (NIHR), the Medical Research Council (MRC) and the Wellcome Trust and compared the results with UK-based researchers who serve currently on the boards of these bodies. From the 1,370 papers relevant to medical, biomedical, life and health sciences with more than 1000 citations in the period examined, we identified 223 individuals from a UK institution at the time of publication who were either first/last or single authors. Of those, 164 are still in UK academic institutions, while 59 are not currently in UK academia (have left the country, are retired, or work in other sectors). Of the 164 individuals, only 59 (36%; 95% CI: 29-43%) currently hold an active grant from one of the three funders. Only 79 (48%; 95% CI: 41-56%) have held an active grant from any of the three funders between 2006-2017. Conversely, 457 of the 664 board members of MRC, Wellcome Trust, and NIHR (69%; 95% CI: 65-72%) have held an active grant in the same period by any of these funders. Only 7 out of 655 board members (1.1%) were first, last or single authors of an extremely highly-cited paper.\ud \ud There are many reasons why the majority of the most influential UK authors do not hold a grant from the country’s major public and charitable funding bodies. Nevertheless, the results are worrisome and subscribe to similar patterns shown in the US. We discuss possible implications and suggest ways forward.

We propose to use Twitter data as social-spatial sensors. This study deals with the question whether research papers on certain diseases are perceived by people in regions (worldwide) that are especially concerned by these diseases. Since (some) Twitter data contain location information, it is possible to spatially map the activity of Twitter users referring to certain papers (e.g., dealing with tuberculosis). The resulting maps reveal whether heavy activity on Twitter is correlated with large numbers of people having certain diseases. In this study, we focus on tuberculosis, human immunodeficiency virus (HIV), and malaria, since the World Health Organization ranks these diseases as the top three causes of death worldwide by a single infectious agent. The results of the social-spatial Twitter maps (and additionally performed regression models) reveal the usefulness of the proposed sensor approach. One receives an impression of how research papers on the diseases have been perceived by people in regions that are especially concerned by these diseases. Our study demonstrates a promising approach for using Twitter data for research evaluation purposes beyond simple counting of tweets.

Papers describing software are an important part of computational fields of scientific research. These “software papers” are unique in a number of ways, and they require special consideration to improve their impact on the scientific community and their efficacy at conveying important information. Here, we discuss 10 specific rules for writing software papers, covering some of the different scenarios and publication types that might be encountered, and important questions from which all computational researchers would benefit by asking along the way., Author summary Computational researchers have a responsibility to ensure that the software they write stands up to the same scientific scrutiny as traditional research studies. These 10 simple rules make doing so easier by enhancing usability, reproducibility, transparency, and other crucial characteristics that aren’t taught in most computer science or research methods curricula.

AimsTo evaluate the relative efficiencies of five Internet-based digital and three paper-based scientific surveys and to estimate the costs for different-sized cohorts.MethodsInvitations to participate in a survey were distributed via e-mail to employees of two university hospitals (E1 and E2) and to members of a medical association (E3), as a link placed in a special text on the municipal homepage regularly read by the administrative employees of two cities (H1 and H2), and paper-based to workers at an automobile enterprise (P1) and college (P2) and senior (P3) students. The main parameters analyzed included the numbers of invited and actual participants, and the time and cost to complete the survey. Statistical analysis was descriptive, except for the Kruskal-Wallis-H-test, which was used to compare the three recruitment methods. Cost efficiencies were compared and extrapolated to different-sized cohorts.ResultsThe ratios of completely answered questionnaires to distributed questionnaires were between 81.5% (E1) and 97.4% (P2). Between 6.4% (P1) and 57.0% (P2) of the invited participants completely answered the questionnaires. The costs per completely answered questionnaire were $0.57-$1.41 (E1-3), $1.70 and $0.80 for H1 and H2, respectively, and $3.36-$4.21 (P1-3). Based on our results, electronic surveys with 10, 20, 30, or 42 questions would be estimated to be most cost (and time) efficient if more than 101.6-225.9 (128.2-391.7), 139.8-229.2 (93.8-193.6), 165.8-230.6 (68.7-115.7), or 188.2-231.5 (44.4-72.7) participants were required, respectively.ConclusionsThe study efficiency depended on the technical modalities of the survey methods and engagement of the participants. Depending on our study design, our results suggest that in similar projects that will certainly have more than two to three hundred required participants, the most efficient way of conducting a questionnaire-based survey is likely via the Internet with a digital questionnaire, specifically via a centralized e-mail.

This study analyzes USPTO patents in the period 1998–2017. The number of science-related patents has increased twice as fast as the number of patents and scientific publications, and the number of cited papers per patent has almost doubled. These results vary substantially from one scientific and technological field to another. The proportion of the research papers cited by a patent has doubled. It refers to papers that are mostly published by the countries that have developed both scientific and technological capability and, surprisingly, are mainly used by inventors abroad. However, a weak relationship between the number of citations received from patents and papers reveals that the assessment of research performance needs some changes as the percentage of papers related to the innovations has grown over time.

There is demand from science funders, industry, and the public that science should become more risk-taking, more out-of-the-box, and more interdisciplinary. Is it possible to tell how interdisciplinary and out-of-the-box scientific papers are, or which papers are mainstream? Here we use the bibliographic coupling network, derived from all physics papers that were published in the Physical Review journals in the past century, to try to identify them as mainstream, out-of-the-box, or interdisciplinary. We show that the network clusters into scientific fields. The position of individual papers with respect to these clusters allows us to estimate their degree of mainstreamness or interdisciplinary. We show that over the past decades the fraction of mainstream papers increases, the fraction of out-of-the-box decreases, and the fraction of interdisciplinary papers remains constant. Studying the rewards of papers, we find that in terms of absolute citations, both, mainstream and interdisciplinary papers are rewarded. In the long run, mainstream papers perform less than interdisciplinary ones in terms of citation rates. We conclude that to avoid a trend towards mainstreamness a new incentive scheme is necessary., 12 pages, 9 figures

Poor research reporting is a major contributing factor to low study reproducibility, financial and animal waste. The ARRIVE (Animal Research: Reporting of In Vivo Experiments) guidelines were developed to improve reporting quality and many journals support these guidelines. The influence of this support is unknown. We hypothesized that papers published in journals supporting the ARRIVE guidelines would show improved reporting compared with those in non-supporting journals. In a retrospective, observational cohort study, papers from 5 ARRIVE supporting (SUPP) and 2 non-supporting (nonSUPP) journals, published before (2009) and 5 years after (2015) the ARRIVE guidelines, were selected. Adherence to the ARRIVE checklist of 20 items was independently evaluated by two reviewers and items assessed as fully, partially or not reported. Mean percentages of items reported were compared between journal types and years with an unequal variance t-test. Individual items and sub-items were compared with a chi-square test. From an initial cohort of 956, 236 papers were included: 120 from 2009 (SUPP; n = 52, nonSUPP; n = 68), 116 from 2015 (SUPP; n = 61, nonSUPP; n = 55). The percentage of fully reported items was similar between journal types in 2009 (SUPP: 55.3 ± 11.5% [SD]; nonSUPP: 51.8 ± 9.0%; p = 0.07, 95% CI of mean difference -0.3-7.3%) and 2015 (SUPP: 60.5 ± 11.2%; nonSUPP; 60.2 ± 10.0%; p = 0.89, 95%CI -3.6-4.2%). The small increase in fully reported items between years was similar for both journal types (p = 0.09, 95% CI -0.5-4.3%). No paper fully reported 100% of items on the ARRIVE checklist and measures associated with bias were poorly reported. These results suggest that journal support for the ARRIVE guidelines has not resulted in a meaningful improvement in reporting quality, contributing to ongoing waste in animal research.

Quantifying the impact of a scholarly paper is of great significance, yet the effect of geographical distance of cited papers has not been explored. In this paper, we examine 30,596 papers published in Physical Review C, and identify the relationship between citations and geographical distances between author affiliations. Subsequently, a relative citation weight is applied to assess the impact of a scholarly paper. A higher-order weighted quantum PageRank algorithm is also developed to address the behavior of multiple step citation flow. Capturing the citation dynamics with higher-order dependencies reveals the actual impact of papers, including necessary self-citations that are sometimes excluded in prior studies. Quantum PageRank is utilized in this paper to help differentiating nodes whose PageRank values are identical. Refereed/Peer-reviewed

In this study, we address the question whether (and to what extent, respectively) altmetrics are related to the scientific quality of papers (as measured by peer assessments). Only a few studies have previously investigated the relationship between altmetrics and assessments by peers. In the first step, we analyse the underlying dimensions of measurement for traditional metrics (citation counts) and altmetrics - by using principal component analysis (PCA) and factor analysis (FA). In the second step, we test the relationship between the dimensions and quality of papers (as measured by the post-publication peer-review system of F1000Prime assessments) - using regression analysis. The results of the PCA and FA show that altmetrics operate along different dimensions, whereas Mendeley counts are related to citation counts, and tweets form a separate dimension. The results of the regression analysis indicate that citation-based metrics and readership counts are significantly more related to quality, than tweets. This result on the one hand questions the use of Twitter counts for research evaluation purposes and on the other hand indicates potential use of Mendeley reader counts.

Background There is widespread evidence that statistical methods play an important role in original research articles, especially in medical research. The evaluation of statistical methods and reporting in journals suffers from a lack of standardized methods for assessing the use of statistics. The objective of this study was to develop and evaluate an instrument to assess the statistical intensity in research articles in a standardized way. Methods A checklist-type measure scale was developed by selecting and refining items from previous reports about the statistical contents of medical journal articles and from published guidelines for statistical reporting. A total of 840 original medical research articles that were published between 2007–2015 in 16 journals were evaluated to test the scoring instrument. The total sum of all items was used to assess the intensity between sub-fields and journals. Inter-rater agreement was examined using a random sample of 40 articles. Four raters read and evaluated the selected articles using the developed instrument. Results The scale consisted of 66 items. The total summary score adequately discriminated between research articles according to their study design characteristics. The new instrument could also discriminate between journals according to their statistical intensity. The inter-observer agreement measured by the ICC was 0.88 between all four raters. Individual item analysis showed very high agreement between the rater pairs, the percentage agreement ranged from 91.7% to 95.2%. Conclusions A reliable and applicable instrument for evaluating the statistical intensity in research papers was developed. It is a helpful tool for comparing the statistical intensity between sub-fields and journals. The novel instrument may be applied in manuscript peer review to identify papers in need of additional statistical review.

Scientific progress is driven by important, infrequent discoveries that cannot be readily identified and quantified, which makes research assessment very difficult. Bibliometric indicators of important discoveries have been formulated using an empirical approach, based on the mathematical properties of the high-citation tail of the citation distribution. To investigate the theoretical basis of such formulations this study compares the US/European research performance ratios expressed in terms of highly cited papers and Nobel Prize-winning discoveries. The research performance ratio in terms of papers was studied from the citation distributions in the fields of chemistry, physics, and biochemistry and molecular biology. It varied as a function of the citation level. Selecting an appropriate high citation level, the ratios in terms of highly cited papers were compared with the corresponding ratios for Nobel Prize-winning discoveries in Chemistry, Physics, and Physiology or Medicine. Research performance ratios expressed in terms of highly cited papers and Nobel Prize-winning discoveries are reasonably similar, and suggest that the research success of the United States is almost 3 times that of Europe. A similar conclusion was obtained using articles published in Nature and Science.