Search

In the last decade, a growing number of studies focused on the qualitative/quantitative analysis of bibliometric-database errors. Most of these studies relied on the identification and (manual) examination of relatively limited samples of errors. Using an automated procedure, we collected a large corpus of more than 10,000 errors in the two multidisciplinary databases Scopus and Web of Science (WoS), mainly including articles in the Engineering-Manufacturing field. Based on the manual examination of a portion (of about 10%) of these errors, this paper provides a preliminary analysis and classification, identifying similarities and differences between Scopus and WoS. The analysis reveals interesting results, such as: (i) although Scopus seems more accurate than WoS, it tends to forget to index more papers, causing the loss of the relevant citations given/obtained, (ii) both databases have relatively serious problems in managing the so-called Online-First articles, and (iii) lack of correlation between databases, regarding the distribution of the errors in several error categories. The description is supported by practical examples concerning a variety of errors in the Scopus and WoS databases.

Recent studies have shown that the Scopus bibliometric database is probably less accurate than one thinks. As a further evidence of this fact, this paper presents a structured collection of several weird typologies of database errors, which can therefore be classified as horrors . Some of them concern the incorrect indexing of so-called Online-First paper, duplicate publications, and the missing/incorrect indexing of references. A crucial point is that most of these errors could probably be avoided by adopting some basic data checking systems. Although this paper does not provide a quantitative and systematic analysis (which will be provided in a future publication), it can be easily understood that these errors can have serious consequences such as: (i) making it difficult or even impossible to retrieve some documents, and (ii) distorting bibliometric indicators/metrics relating to journals, individual scientists or research institutions. Our attention is focused on the Scopus database, although preliminary data show that the Web of Science database is far from being free from these errors. The tone of the paper is deliberately provocative, in order to emphasize the seriousness of these errors.

A recent paper (Canavero et al., 2014. Journal of the American Society for Information Science and Technology, doi:10.1109/TPC.2013.2255935) performed a bibliometric analysis of an extensive set of scientific journals within the Engineering field, published by IEEE (Institute of Electrical and Electronics Engineers). The analysis was based on (i) the citation impact of journal articles and (ii) the reputation of journal authors in terms of total scientific production and relevant citation impact. The goal of this paper is to complement the prior analysis, investigating on the different citation cultures of these journals, depending on the sub-field/specialty of interest. To perform this evaluation, it is suggested a novel technique, which takes into account the connections between journals and some highly specialized communities of scientists, known as IEEE Technical Societies and Councils. After showing significant differences in terms of propensity to cite, probably attributable to the large variety of sub-fields and specialties covered by IEEE journals, it is presented a simplified technique for the sub-field normalization of the results of the prior study. The main contribution of this work is (1) providing an empirical confirmation of the complexity of the problem of normalization, even for journals within the same field but different sub-fields/specialties, and (2) showing how the use of highly specialized information on a journal reference sub-field(s) may be helpful for improving the estimation of the journal propensity to cite. Description is supported by a large amount of empirical data.

The aim of this brief communication is to reply to a letter by Kosmulski (Journal of Informetrics 6(3):368–369, 2012), which criticizes a recent indicator called “success-index”. The most interesting features of this indicator, presented in Franceschini et al. (Scientometrics, in press ), are: (i) allowing the selection of an “elite” subset from a set of publications and (ii) implementing the field-normalization at the level of an individual publication. We show that the Kosmulski's criticism is unfair and inappropriate, as it is the result of a misinterpretation of the indicator.

The paper "Critical remarks on the Italian research assessment exercise VQR 2011-2014" (Franceschini & Maisano, 2017) analyzed some vulnerabilities of the recently concluded Italian assessment exercise. Some apical (former and current)members of ANVUR promptly commented on our criticisms through a letter to the editor (Benedetto, Checchi, Graziosi, & Malgarini, 2017). We believe that this letter is not very convincing. In the following, we provide a rejoinder to the comments directed to our paper.

Evaluating the scientific output of researchers, research institutions, academic departments and even universities is a challenging issue. To do this, bibliometric indicators are helpful tools, more and more familiar to research and governmental institutions. This paper proposes a structured method to compare academic research groups within the same discipline, by means of some Hirsch (h) based bibliometric indicators. Precisely, five different typologies of indicators are used so as to depict groups’ bibliometric positioning within the scientific community. A specific analysis concerning the Italian researchers in the scientific sector of Production Technology and Manufacturing Systems is developed. The analysis is supported by empirical data and can be extended to research groups associated to other scientific sectors.