Search

In the evaluation of scientific publications' impact, the interplay between intrinsic quality and non-scientific factors remains a subject of debate. While peer review traditionally assesses quality, bibliometric techniques gauge scholarly impact. This study investigates the role of non-scientific attributes alongside quality scores from peer review in determining scholarly impact. Leveraging data from the first Italian Research Assessment Exercise (VTR 2001-2003) and Web of Science citations, we analyse the relationship between quality scores, non-scientific factors, and publication short- and long-term impact. Our findings shed light on the significance of non-scientific elements overlooked in peer review, offering policymakers and research management insights in choosing evaluation methodologies. Sections delve into the debate, identify non-scientific influences, detail methodologies, present results, and discuss implications.

Similar to how innovations often find success in fields other than their original domains, in this study we explore whether the same holds true for scientific discoveries. We investigate the flow of knowledge across scientific disciplines, focusing on connections between citing and cited publications. Specifically, we analyze the connections among cited publications from 2015 indexed in the Web of Science and their citing counterparts to measure rates of knowledge dissemination within and across different fields. Our study aims to address key research questions concerning the disparities between inter- and intra-domain knowledge dissemination rates, the correlation between knowledge dissemination types and scholarly impact, as well as the evolution of knowledge dissemination patterns over time. These findings deepen our understanding of knowledge flows and offer practical insights with significant implications for evaluative bibliometrics.

The paper examines extent of bias in the performance rankings of research organisations when the assessments are based on unsupervised author-name disambiguation algorithms. It compares the outcomes of a research performance evaluation exercise of Italian universities using the unsupervised approach by Caron and van Eck (2014) for derivation of the universities' research staff, with those of a benchmark using the supervised algorithm of D'Angelo, Giuffrida, and Abramo (2011), which avails of input data. The methodology developed could be replicated for comparative analyses in other frameworks of national or international interest, meaning that practitioners would have a precise measure of the extent of distortions inherent in any evaluation exercises using unsupervised algorithms. This could in turn be useful in informing policy-makers' decisions on whether to invest in building national research staff databases, instead of settling for the unsupervised approaches with their measurement biases.

Geographic proximity is acknowledged to be a key factor in research collaborations. Specifically, it can work as a possible substitute for institutional proximity. The present study investigates the relevance of the "proximity" effect for different types of national research collaborations. We apply a bibliometric approach based on the Italian 2010-2017 scientific production indexed in the Web of Science. On such dataset, we apply statistical tools for analyzing if and to what extent geographical distance between co-authors in the byline of a publication varies across collaboration types, scientific disciplines, and along time. Results can inform policies aimed at effectively stimulating cross-sector collaborations, and also bear direct practical implications for research performance assessments.

University-industry research collaboration is one of the major research policy priorities of advanced economies. In this study, we try to identify the main drivers that could influence the propensity of academics to engage in research collaborations with the private sector, in order to better inform policies and initiatives to foster such collaborations. At this purpose, we apply an inferential model to a dataset of 32,792 Italian professors in order to analyze the relative impact of individual and contextual factors affecting the propensity of academics to engage in collaboration with industry, at overall level and across disciplines. The outcomes reveal that the typical profile of the professor collaborating with industry is a male under age 40, full professor, very high performer, with highly diversified research, and who has a certain tradition in collaborating with industry. This professor is likely to be part of a staff used to collaborating with industry, in a small university, typically a polytechnic, located in the north of the country., Comment: arXiv admin note: text overlap with arXiv:2102.05364

This study investigates the convergence of two bibliometric approaches to the measurement of interdisciplinary research: one based on analyzing disciplinary diversity in the reference list of publications, the other based on the disciplinary diversity of authors of publications. In particular we measure the variety, balance, disparity and integrated diversity index of, respectively, single-author, multi-author single-field, and multi-author multi-field publications. We find that, in general, the diversity of the reference list grows with the number of fields reflected in a paper's authors' list and, to a lesser extent, with the number of authors being equal the number of fields. Further, we find that when fields belonging to different disciplines are reflected in the authors' list, the disparity in the reference list is higher than in the case of fields belonging to the same discipline. However, this general tendency varies across disciplines, and noticeable exceptions are found at individual paper level.

This study on public-private research collaboration measures the variation over time of the propensity of academics to collaborate with colleagues from private companies. It also investigates the change in weights of the main drivers underlying the academics' propensity to collaborate, and whether the type profile of the collaborating academics changes. To do this, the study applies an inferential model on a dataset of professors working in Italian universities in consecutive periods, 2010-2013 and 2014-2017. The results, obtained at overall and field levels, support the formulation of policies aimed at fostering public-private research collaborations, and should be taken into account in post-assessment of their effectiveness., Comment: arXiv admin note: text overlap with arXiv:2102.05364

In this study, the scientific performance of Italian and Norwegian university professors is analysed using bibliometric indicators. The study is based on over 36,000 individuals and their publication output during the period 2011-2015. Applying a multidimensional indicator in which several aspects of the research performance are captured, we find large differences in the performance of men and women. These gender differences are evident across all analysed levels, such as country, field, and academic position. However, most of the gender differences can be explained by the tails of the distributions-in particular, there is a much higher proportion of men among the top 10% performing scientists. For the remaining 90% of the population, the gender differences are practically non-existent. The results of the two countries, which differ in terms of the societal role of women, are contrasting. Further, we discuss possible biases that are intrinsic in quantitative performance indicators, which might disfavour female researchers.

The disambiguation of author names is an important and challenging task in bibliometrics. We propose an approach that relies on an external source of information for selecting and validating clusters of publications identified through an unsupervised author name disambiguation method. The application of the proposed approach to a random sample of Italian scholars shows encouraging results, with an overall precision, recall, and F-Measure of over 96%. The proposed approach can serve as a starting point for large-scale census of publication portfolios for bibliometric analyses at the level of individual researchers.

Policy makers are interested in the influence of geographic distance on knowledge flows, however these can be expected to vary across research fields. The effects of geographic distance on flows are analyzed by means of citations to scientific literature. The field of observation consists of the 2010-2012 Italian publications and relevant citations up to the close of 2017. The geographic proximity effect is analyzed at national, continental, and intercontinental level in 244 fields, and results as evident at national level and in some cases at continental level, but not at intercontinental level. For flows between Italian municipalities, citations decrease with distance in all fields. At continental level, four fields are identified having knowledge flows that grow with distance; at intercontinental level, this occurs in 26 fields. The influence of distance is more limited in the fields of Humanities and Social sciences, much more significant in the Sciences, mainly in the Natural sciences., Comment: arXiv admin note: text overlap with arXiv:2002.00742, arXiv:2103.13819

The formulation of national research policies would benefit greatly from reliable strategic analysis of the scientific infrastructure, aimed at identifying the relevant strengths and weaknesses at field level. Bibliometric methodologies thus far proposed in the literature are not completely satisfactory. This work proposes a novel "output-to-input-oriented" approach, which permits identification of research strengths and weaknesses on the basis of the ratios of top scientists and highly cited articles to research expenditures in each field. The proposed approach is applied to the Italian academic system. 2012-2016 scientific publications are analyzed, in the 218 research fields where bibliometric assessment is appropriate.

Citations acknowledge the impact a scientific publication has on subsequent work. At the same time, deciding how and when to cite a paper, is also heavily influenced by social factors. In this work, we conduct an empirical analysis based on a dataset of 2010-2012 global publications in chemical engineering. We use social network analysis and text mining to measure publication attributes and understand which variables can better help predicting their future success. Controlling for intrinsic quality of a publication and for the number of authors in the byline, we are able to predict scholarly impact of a paper in terms of citations received six years after publication with almost 80 percent accuracy. Results suggest that, all other things being equal, it is better to co-publish with rotating co-authors and write the papers' abstract using more positive words, and a more complex, thus more informative, language. Publications that result from the collaboration of different social groups also attract more citations.

This work analyzes the variation over time of the effect of geographic distance on knowledge flows. The flows are measured through the citations exchanged between scientific publications, including and excluding self-citations. To calculate geographic distances between citing and cited publication, each publication is associated with a "prevailing" territory, according to the authors' affiliations. We then apply a gravity model to account for the research size of the territories, in terms of cognitive proximity of citing-cited publications. The field of observation is the 2010-2017 world publications citing the 2010-2012 Italian publications, as indexed in the Web of Science. The results show that in domestic knowledge flows, geographic proximity remains an influential factor through time, although with differences among disciplines and trends of attenuating effects. Finally, we replicate the analyses of knowledge flows but with the exclusion of self-citations: in this manner the effect of geographic proximity seems reduced, particularly at the national scale, but the differences (with vs without self-citations) lessen through time. As shown in previous works, the effect of distance on continental flows is modest (imperceptible for intercontinental flows), yet here too time has some influence, including concerning exclusion of self-citations.

In this work we ask whether and to what extent applying a predictor of publications' impact better than early citations, has an effect on the assessment of research performance of individual scientists. Specifically, we measure the total impact of Italian professors in the sciences and economics in a period of time, valuing their publications first by early citations and then by a weighted combination of early citations and impact factor of the hosting journal. As expected, scores and ranks by the two indicators show a very strong correlation, but there occur also significant shifts in many fields, mainly in Economics and statistics, and Mathematics and computer science. The higher the share of uncited professors in a field and the shorter the citation time window, the more recommendable the recourse to the above combination.

The growing complexity of scientific challenges demands increasingly intense research collaboration, both domestic and international. The resulting trend affects not only the modes of producing new knowledge, but also the way it is disseminated within scientific communities. This paper analyses the relationship between the "degree of internationalization" of a country's scientific production and that of the relevant citing publications. The empirical analysis is based on 2010-2012 Italian publications. Findings show: i) the probability of being cited increases with the degree of internationalization of the research team; ii) totally domestic research teams tend to cite to a greater extent totally domestic publications; iii) vice versa, publications resulting from international collaborations tend to be more cited by totally foreign publications rather than by publications including domestic authors. These results emerge both at overall and at discipline level. Findings might inform research policies geared towards internationalization.

This work applies a new approach to measure knowledge flows. Assuming that citation linkages between articles imply a flow of knowledge from the cited to the citing authors, we investigate the geographic flows of scientific knowledge produced in Italy across its regions, at both overall and field level. Furthermore, we measure the the specialization indexes for outflows and inflows of knowledge by a given region. Findings show that larger regions in terms of research output are more likely net exporters of new knowledge. At the same time, we register a positive correlation between the share of intraregional flows and the size of overall scientific output of a region., Comment: arXiv admin note: text overlap with arXiv:1909.08911

The massive shock of the COVID-19 pandemic is already showing its negative effects on economies around the world, unprecedented in recent history. COVID-19 infections and containment measures have caused a general slowdown in research and new knowledge production. Because of the link between R&D spending and economic growth, it is to be expected then that a slowdown in research activities will slow in turn the global recovery from the pandemic. Many recent studies also claim an uneven impact on scientific production across gender. In this paper, we investigate the phenomenon across countries, analysing preprint depositions. Differently from other works, that compare the number of preprint depositions before and after the pandemic outbreak, we analyse the depositions trends across geographical areas, and contrast after-pandemic depositions with expected ones. Differently from common belief and initial evidence, in few countries female scientists increased their scientific output while males plunged.

This study is about public-private research collaboration. In particular, we want to measure how the propensity of academics to collaborate with their colleagues from private firms varies over time and whether the typical profile of such academics change. Furthermore, we investigate the change of the weights of main drivers underlying the academics' propensity to collaborate with industry. In order to achieve such goals, we apply an inferential model on a dataset of professors working in Italian universities in two subsequent periods, 2010-2013 and 2014-2017. Results can be useful for supporting the definition of policies aimed at fostering public-private research collaborations, and should be taken into account when assessing their effectiveness afterwards.

We investigate the changes in the self-citation behavior of Italian professors following the introduction of a citation-based incentive scheme, for national accreditation to academic appointments. Previous contributions on self-citation behavior have either focused on small samples or relied on simple models, not controlling for all confounding factors. The present work adopts a complex statistics model implemented on bibliometric individual data for over 15,000 Italian professors. Controlling for a number of covariates (number of citable papers published by the author; presence of international authors; number of co-authors; degree of the professor's specialization), the average increase in self-citation rates following introduction of the ASN is of 9.5%. The increase is common to all disciplines and academic ranks, albeit with diverse magnitude. Moreover, the increase is sensitive to the relative incentive, depending on the status of the scholar with respect to the scientific accreditation. A further analysis shows that there is much heterogeneity in the individual patterns of self-citing behavior, albeit with very few outliers.

Governments and organizations design performance-based research funding systems (PBFRS) for strategic aims, such as to selectively allocate scarce resources and stimulate research efficiency. In this work we analyze the relative change in research productivity of Italian universities after the introduction of such a system, featuring financial and reputational incentives. Using a bibliometric approach, we compare the relative research performance of universities before and after introduction of PBFRS, at the overall, discipline and field levels. The findings show convergence in the universities' performance, due above all to the remarkable improvement of the lowest performers. Geographically, the universities of the south (versus central and northern Italy) achieved the greatest improvement in relative performance. The methodology, and results, should be of use to university management and policy-makers.

Literature about the scholarly impact of scientific research offers very few contributions on private sector research, and the comparison with public sector. In this work, we try to fill this gap examining the citation-based impact of Italian 2010-2017 publications distinguishing authorship by the private sector from the public sector. In particular, we investigate the relation between different forms of collaboration and impact: how intra-sector private publications compare to public, and how private-public joint publications compare to intra-sector extramural collaborations. Finally, we assess the different effect of international collaboration on private and public research impact, and whether there occur differences across research fields., Comment: 28 pages, 12 tables

In this paper we develop a methodology to assess the scientific wealth of territories at field level. Our methodology uses a bibliometric approach based on the observation of academic research performance and overall scientific production in each territory. We apply it to assess disparities in the Italian territories in the medical specialties at the front line of the COVID-19 emergency. Italy has been the first among western countries to be severely affected by the onset of the COVID-19 pandemic. The study reveals remarkable inequities across territories, with scientific weaknesses concentrated in the south. Policies for rebalancing the north-south divide should also consider, in addition to tangible assets, the gap in production and availability of quality medical knowledge., Comment: 25 Pages, 5 Figures, 4 Tables

In this study we are analysing the research performance of Italian and Norwegian professors using constituent components of the Fractional Scientific Strength (FSS) indicator. The main focus is on differences across fields in publication output and citation impact. The overall performance (FSS) of the two countries, which differ considerably in research size and profile, is remarkedly similar. However, an in-depth analysis shows that there are large underlying performance differences. An average Italian professor publishes more papers than a Norwegian, while the citation impact of the research output is higher for the Norwegians. In addition, at field level the pattern varies along both dimensions, and we analyse in which fields each country have their relative strengths. Overall, this study contributes to further insights on how the research performance of different countries may be analysed and compared, to inform research policy., Comment: arXiv admin note: text overlap with arXiv:1907.02043

This paper analyses the influence of geographic distance on knowledge flows, measured through citations to scientific publications. Previous works using the same approach are limited to single disciplines. In this study, we analyse the Italian scientific production in all disciplines matured in the period 2010-2012. To calculate the geographic distances between citing and cited publications, each one is associated with a "prevalent" territory on the basis of the authors' affiliations. The results of the application of a gravity model, estimated using ordinary least squares regression, show that despite the spread of IT, geographic distance continues to be an influential factor in the process of knowledge flows between territories. In particular, the analysis reveals that the effect of geographic distance on knowledge flows is significant at the national level, not negligible at the continental level, but completely irrelevant at the intercontinental level.

In analogy to the technology balance of payments, in this paper we propose a possible way to set up a "balance of knowledge flows" (BKF), recording world flows of knowledge within the scientific community. Adopting a pure bibliometric approach, the "knowledge" traced in the BKF is that produced and exchanged by the scientific community by means of publications and relevant citations. A description of the theoretical foundation of such a tool is presented together with its empirical testing over the scientific production of four different countries. The BKF can be part of yearly reports of science and technology indicators, aimed at informing research policy.

The ability to predict the long-term impact of a scientific article soon after its publication is of great value towards accurate assessment of research performance. In this work we test the hypothesis that good predictions of long-term citation counts can be obtained through a combination of a publication's early citations and the impact factor of the hosting journal. The test is performed on a corpus of 123,128 WoS publications authored by Italian scientists, using linear regression models. The average accuracy of the prediction is good for citation time windows above two years, decreases for lowly-cited publications, and varies across disciplines. As expected, the role of the impact factor in the combination becomes negligible after only two years from publication.

The intention of this work is to analyze top scientists' collaboration behavior at the "international", "domestic extramural" and "intramural" levels, and compare it to that of their lesser performing colleagues. The field of observation consists of the entire faculty of the Italian academic system, and so the coauthorship of scientific publications by over 12,000 professors. The broader aim is to improve understanding of the causal nexus between research collaboration and performance. The analysis is thus longitudinal, over two successive five-year periods. Results show a strong increase in the propensity to collaborate at domestic level (both extramural and intramural), however this is less for scientists who remain or become top, than it is for their lower-performing colleagues. In contrast, the increase in international collaboration behavior is greater for scientists who become or remain top than it is for their peers. The increase in productivity by those who acquire top scientist status is due precisely to the greater average impact of the publications achieved in collaboration with foreign colleagues., Comment: arXiv admin note: text overlap with arXiv:1906.00754

Altmetrics promise useful support for assessing the impact of scientific works, including beyond the scholarly community and with very limited citation windows. Unfortunately, altmetrics scores are currently available only for recent articles and cannot be used as covariates in predicting long term impact of publications. However, the study of their statistical properties is a subject of evident interest to scientometricians. Applying the same approaches used in the literature to assess the universality of citation distributions, the intention here is to test whether the universal distribution also holds for Mendeley readerships. Results of the analysis carried out on a sample of publications randomly extracted from the Web of Science confirm that readerships seem to share similar shapes across fields and can be rescaled to a common and universal form. Such rescaling results as not particularly effective on the right tails. In other regions, rescaling causes a good collapse of field specific distributions, even for very recent publications.

This work investigates the research performance of foreign faculty in the Italian academic system. Incoming professors compose l'1% of total faculty across the sciences, although with variations by discipline. Their scientific performance measured over 2010-2014 is on average better than that of their Italian colleagues: the greatest difference is for associate professors. Psychology is the discipline with the greatest concentration of top foreign scientists. However there are notable shares of unproductive foreign professors or of those with mediocre performance. The findings stimulate reflection on issues of national policy concerning attractiveness of the higher education system to skilled people from abroad, given the ongoing heavy Italian brain drain., Comment: arXiv admin note: text overlap with arXiv:1907.02043, arXiv:1810.12207

In this work, we try to answer the question of which method, peer review vs bibliometrics, better predicts the future overall scholarly impact of scientific publications. We measure the agreement between peer review evaluations of Web of Science indexed publications submitted to the first Italian research assessment exercise and long-term citations of the same publications. We do the same for an early citation-based indicator. We find that the latter shows stronger predictive power, i.e., it more reliably predicts late citations in all the disciplinary areas examined, and for any citation time window starting one year after publication.

The current work addresses a theme previously unexplored in the literature: that of whether the results arising from research activity in fields other than the scientist's pri-mary field have greater value than the others. Operationally, the authors proceed by identifying: the scientific production of each researcher under observation; field classifi-cation of the publications; the field containing the greatest number of the researcher's publications; attribution of value of each publication. The results show that diversifica-tion at the aggregate level does not pay off, although there are some exceptions at the level of individual disciplines. The implications at policy level are notable. Since the in-centive systems of research organizations are based on the impact of scientific output, the scientists concerned could resist engaging in multidisciplinary projects.

This is the first ever attempt of application in a country other than Italy of a research efficiency indicator (FSS), to assess and compare the performance of professors and universities, within and between countries. A special attention has been devoted to the presentation of the methodology developed to set up a common field classification scheme of professors, and to overcome the limited availability of comparable input data. Results of the comparison between countries, carried out in the 2011-2015 period, show similar average performances of professors, but noticeable differences in the distributions, whereby Norwegian professors are more concentrated in the tails. Norway shows notable higher performance in Mathematics and Earth and Space Sciences, while Italy in Biomedical Research and Engineering.

This work analyzes the differences in collaboration behavior between males and females among a particular type of scholars: top scientists, and as compared to non top scientists. The field of observation consists of the Italian academic system and the co-authorships of scientific publications by 11,145 professors. The results obtained from a cross-sectional analysis covering the five-year period 2006-2010 show that there are no significant differences in the overall propensity to collaborate in the top scientists of the two genders. At the level of single disciplines there are no differences in collaboration behavior, except in the case of: i) international collaborations, for Mathematics and Chemistry - where the propensity for collaboration is greater for males; and ii) extramural domestic collaborations in Physics, in which it is the females that show greater propensity for collaboration. Because international collaboration is positively correlated to research performance, findings can inform science policy aimed at increasing the representation of female top performers.

The present work investigates the relations between amplitude and type of collaboration (intramural, extramural domestic or international) and output of specialized versus diversified research. By specialized or diversified research, we mean within or beyond the author's dominant research topic. The field of observation is the scientific production over five years from about 23,500 academics. The analyses are conducted at the aggregate and disciplinary level. The results lead to the conclusion that in general, the output of diversified research is no more frequently the fruit of collaboration than is specialized research. At the level of the particular collaboration types, international collaborations weakly underlie the specialized kind of research output; on the contrary, extramural domestic and intramural collaborations are weakly associated with diversified research. While the weakness of association remains, exceptions are observed at the level of the individual disciplines.

It is well known that women are underrepresented in the academic systems of many countries. Gender discrimination is one of the factors that could contribute to this phenomenon. This study considers a recent national academic recruitment campaign in Italy, examining whether women are subject to more or less bias than men. The findings show that no gender-related differences occur among the candidates who benefit from positive bias, while among those candidates affected by negative bias, the incidence of women is lower than that of men. Among the factors that determine success in a competition for an academic position, the number of the applicant's career years in the same university as the committee members assumes greater weight for male candidates than for females. Being of the same gender as the committee president is also a factor that assumes greater weight for male applicants. On the other hand, for female applicants, the presence of a full professor in the same university with the same family name as the candidate assumes greater weight than for male candidates., Comment: arXiv admin note: text overlap with arXiv:1810.13236, arXiv:1810.12207

We investigate whether and in what measure scientists tend to diversify their research activity, and if this tendency varies according to their belonging to different disciplinary areas. We analyze the nature of research diversification along three dimensions: extent of diversification, intensity of diversification, and degree of relatedness of topics in which researchers diversifies. For this purpose we propose three bibliometric indicators, based on the disciplinary placement of scientific output of individual scientists. The empirical investigation shows that the extent of diversification is lowest for scientists in Mathematics and highest in Chemistry; intensity of diversification is lowest in Earth sciences and highest in Industrial and information engineering; and degree of relatedness is lowest in Earth sciences and highest in Chemistry., Comment: arXiv admin note: text overlap with arXiv:1810.12633

The growing complexity of challenges involved in scientific progress demands ever more frequent application of competencies and knowledge from different scientific fields. The present work analyzes the degree of collaboration among scientists from different disciplines in order to identify the most frequent "combinations of knowledge" in research activity. The methodology adopts an innovative bibliometric approach based on the disciplinary affiliation of publication co-authors. The field of observation includes all publications (173,134) indexed in the Science Citation Index Expanded (SCI-E) for the five years 2004-2008, authored by all scientists in the hard sciences (43,223) at Italian universities (68). The analysis examines 205 research fields grouped in nine disciplines. Identifying the fields with the highest potential of interdisciplinary collaboration is useful to inform research polices at national and regional levels, as well as management strategies at the institutional level.

Italy lags quite behind vis-a'-vis other industrialized countries, in public to private technology transfer. One of the possible causes might be the mismatch between new knowledge supplied by public research and industry demand. We test this hypothesis through a survey of leading public research scientists in four high-tech sectors. The findings show that most research project results seem to be of immediate industrial interest, which contrasts with the low patent and licensing performances of Italian public research institutions. For one third of all the results of the research, there are no Italian companies able to exploit them. The same, however, is not true for the remaining results, which shows that the misalignment between public supply and industry demand alone cannot account for poor technology transfer. What emerges from our investigation is that a closer coordination of research policy and industrial policy is required, as well as closer attention to initiatives which may support the transfer of public research results to domestic industry.

In benchmarking international research, although publication and citation analyses should not be used to compare different disciplines, scientometrists frequently fail to resist the temptation to present rankings based on total publications and citations. Such measures are affected by significant distortions, due to the uneven fertility across scientific disciplines and the dishomogeneity of scientific specialisation among nations and universities. In this paper, we provide an indication of the extent of the distortions when comparative bibliometric analyses fail to recognise the range of levels of scientific fertility, not only within a given major disciplinary area but also within the different scientific disciplines encompassed by the same area.

Bibliometric indicators are increasingly used in support of decisions for recruitment, career advancement, rewarding and selective funding for scientists. Given the importance of the applications, bibliometricians are obligated to carry out empirical testing of the robustness of the indicators, in simulations of real contexts. In this work we compare the results of national-scale research assessments at the individual level, based on three different indexes: the h-index, g-index and "fractional scientific strength", or FSS, an indicator previously proposed by the authors. For each index, we construct and compare rankings lists of all Italian academic researchers working in the hard sciences over the period 2001-2005. The analysis quantifies the shifts in ranks that occur when researchers' productivity rankings by simple indicators such as h- or g-index are compared with that by more accurate FSS., Comment: arXiv admin note: text overlap with arXiv:1810.12844