Search

There is a growing interest in determining the factors that influence a journal’s flipping to Open Access (OA). Using semi-structured interviews combined with bibliometric indicators, this paper uncovers the perception of Spanish managers related to OA and the decision to flip. The key research questions are twofold: How well do bibliometric measures reflect the changes in the status of the journal? How do journal managers perceive the flipping process? In order to answer these, twelve semi-structured interviews were conducted with journal managers of Spanish Journals. The findings suggest the great majority of managers are aware of the indicators, but only two considered they reflect their reality. The results indicate as the main motivations to flip to OA: being imposed by the host institution, economic reasons, and increase visibility and internationalization. An increase in the number of submissions, visibility, or internationalization since the transition is perceived as a benefit while the loss of interchanges with other institutions is seen as the major drawback. Although flipping to OA is perceived by the managers to have many advantages, it raises some challenges too, especially the need for funding, lack of resources capacity for technical support, and the creation of alliances.

Purpose Building upon pioneering work by Francis Narin and others, a new methodological approach to assessing the technological impact of scientific research is presented. Design/methodology/approach It is based on the analysis of citations made in patent families included in the PATSTAT database that is to scientific papers indexed in Scopus. Findings An advanced citation matching procedure is applied to the data in order to construct two indicators of technological impact: on the citing (patent) side, the country/region in which protection is sought and a patent family's propensity to cite scientific papers are taken into account, and on the cited (paper) side, a relative citation rate is defined for patent citations to papers that is similar to the scientific paper-to-paper citation rate in classical bibliometrics. Research limitations The results are limited by the available data, in our case Scopus and PATSTAT, and especially by the lack of standardization of references in patents. This required a matching procedure that is neither trivial nor exact. Practical implications Results at the country/region, document type, and publication age levels are presented. The country/region-level results in particular reveal features that have remained hidden in analyses of straight counts. Especially notable is that the rankings of some Asian countries/regions move upwards when the proposed normalized indicator of technological impact is applied as against the case with straight counts of patent citations to those countries/regions’ published papers. Originality/value In our opinion, the level of sophistication of the indicators proposed in the current paper is unparalleled in the scientific literature, and provides a solid basis for the assessment of the technological impact of scientific research in countries/regions and institutions.

Nationally oriented scientific-scholarly journals are considered from a methodological-informetric viewpoint, analysing data extracted from Scimago Journal Rank based on Scopus. An operational definition is proposed of a journal’s degree of national orientation based on the geographical distribution of its publishing or citing authors, and the role of international collaboration and a country’s total publication output is discussed. A comprehensive analysis is presented of trends up until 2019 in national orientation and citation impact of national journals entering Scopus, extending outcomes in earlier studies. A method to analyse national journals of given countries is applied to the set of former USSR republics and Eastern and Central European states which were under socialism, distinguishing between domestic and foreign national journals. The possible influence is highlighted of factors related to a journal’s access status, publication language and subject field, international scientific migration and collaboration, database coverage policies, the size of a national research community, historical-political factors and national research assessment and funding policies.

A critical discussion is presented for the Author Metrics Database (AMD) created by Ioannides et al. (2016, 2020) containing citation-based indicators for 165,000 authors publishing in journals indexed in Scopus. It is concluded that the AMD is a rich intermediary dataset open for further analysis to all interested users. However, its indicators suggest a false precision and lack transparency. The theoretical and statistical basis of the database’s key composite impact indicator is weak, and information on whether or not underlying author publication lists were validated is lacking. The paper aims to broaden the perspective on the further development of an AMD, highlighting its bottom-up, interactive use, aptness for self-assessment and educational function for a wide user community. POLICY HIGHLIGHTS Scopus diverges from Eugene Garfield’s original concept of the Science Citation Index, as citation impact plays a weaker role as journal selection criterion. The transparency of the Article Metrics Database (AMD) is seriously hampered by the lack of information on whether the data were verified by scientists themselves. A complex composite indicator in the AMD decides whether or not a particular author is included. Its components are strongly statistically dependent and are largely based on the position an author has in a paper’s author sequence but lack a sound theoretical foundation. An assessment of an individual researcher cannot be merely based on whether or not he or she is included in the AMD. The issue as to how to deal with multi-authored papers in research assessment of individuals can to some extent be enlightened by bibliometric indicators but cannot be solved bibliometrically. This is why the Composite Indicator suggests a false precision. The AMD focuses almost exclusively on senior scientists. Early career scientists and emerging research groups who will shape science and scholarship in the near future hardly appear in the AMD. Desktop bibliometrics using the AMD as a sole source of information must be rejected. Using the AMD as a starting point in a more extensive bibliometric data collection makes it de facto a promotion tool for other Elsevier products. An alternative approach is an interactive, bottom-up bibliometric tool designed for self-assessment and educational purposes, showing how bibliometric indicators depend upon the way in which initial publication lists, author benchmark sets, subject delimitations, thresholds and evaluative assumptions are chosen. Research assessment is much more than just bibliometrics. It requires an overarching evaluative framework based on normative views on what constitutes research performance and which policy objectives should be achieved.

This paper examines the citation impact of papers published in scientific-scholarly journals upon patentable technology, as reflected in examiner- or inventor-given references in granted patents. It analyses data created by SCImago Research Group, linking PATSTAT's scientific non-patent references (SNPRs) to source documents indexed in Scopus. The frequency of patent citations to journal papers is calculated per discipline, year, institutional sector, journal subject category, and for "top" journals. PATSTAT/Scopus-based statistics are compared to those derived from Web of Science/USPTO linkage. A detailed assessment is presented of the technological impact of research publications in social sciences and humanities (SSH). Several subject fields perform well in terms of the number of citations from patents, especially Library & Information Science, Language & Linguistics, Education, and Law, but many of the most cited journals find themselves in the interface between SSH and biomedical or natural sciences. Analyses of the titles of citing patents and cited papers are presented that shed light upon the cognitive content of patent citations. It is proposed to develop more advanced indicators of citation impact of papers upon patents, and ways to combine citation counts with citation content and context analysis., Author copy (peer-reviewed pre-print) of a paper accepted for publication in Scientometrics in April 2020

This paper presents a statistical analysis of the relationship between three science indicators applied in earlier bibliometric studies, namely research leadership based on corresponding authorship, international collaboration using international co-authorship data, and field-normalized citation impact. Indicators at the level of countries are extracted from the SIR database created by SCImago Research Group from publication records indexed for Elsevier's Scopus. The relationship between authorship and citation-based indicators is found to be complex, as it reflects a country's phase of scientific development and the coverage policy of the database. Moreover, one should distinguish a genuine leadership effect from a purely statistical effect due to fractional counting. Further analyses at the level of institutions and qualitative validation studies are recommended., Peer-reviewed author copy of a paper published in Journal of Informetrics

Trends are analysed in the annual number of documents published by Russian institutions and indexed in Scopus and Web of Science, giving special attention to the time period starting in the year 2013 in which the Project 5-100 was launched by the Russian Government. Numbers are broken down by document type, publication language, type of source, research discipline, country and source. It is concluded that Russian publication counts strongly depend upon the database used, and upon changes in database coverage, and that one should be cautious when using indicators derived from WoS, and especially from Scopus, as tools in the measurement of research performance and international orientation of the Russian science system., Author copy of a manuscript accepted for publication in the journal Scientometrics, May 2018

Policy highlights: If evaluation is defined as “a systematic determination of a subject’s merit, worth and significance, using criteria governed by a set of standards”, there is no evaluation without an evaluative framework specifying these criteria and standards. On the other hand, evaluative informetrics itself, defined as the study of evaluative aspects of science and scholarship using citation analysis, altmetrics and other indicators, does not evaluate. However, informetric indicators are often used in research assessment processes. To obtain a better understanding of their role, the links between evaluative informetrics and ‘values’ are investigated, and a series of practical guidelines are proposed. Informetricians should maintain in their evaluative informetric studies a neutral position toward the policy issues addressed and the criteria specified in an evaluative framework. As professional experts, informetricians’ competence lies primarily in the development and application of analytical models within the context of a given evaluative framework. Informetric researchers could propose that evaluators and policy makers incorporate fundamental scientific values such as openness and adopting a critical attitude in assessment processes. Informetricians could also promote and participate in an overall discussion within the academic community and the research policy domain about the objectives and criteria in research assessment processes and the role of informetric tools therein. Evaluative informetrics is defined as the study of evaluative aspects of science and scholarship using informetric data and methodologies, such as citation analysis and altmetrics. Following the main lines of an article by the Dutch philosopher O.D. Duintjer, nine interfaces are distinguished between quantitative science studies, especially evaluative informetrics, and the domain of values, including scientific, socio-historical, political, ethical and personal norms and objectives. Special attention is given to the “principle of value neutrality” at the meta-level of methodological rules guiding scientific inquiry and to the crucial, independent role of evaluative frameworks in research evaluation. The implications of the various relationships between science and values for research practices in evaluative informetrics and for its application in research assessment are considered.

The effects of Open Access (OA) upon journal performance are investigated. The key research question holds: How does the citation impact and publication output of journals switching ('flipping') from non-OA to Gold-OA develop after their switch to Gold-OA? A review is given of the literature, with an emphasis on studies dealing with flipping journals. Two study sets with 119 and 100 flipping journals, derived from two different OA data sources (DOAJ and OAD), are compared with two control groups, one based on a standard bibliometric criterion, and a second controlling for a journal's national orientation. Comparing post-switch indicators with pre-switch ones in paired T-tests, evidence was obtained of an OA Citation advantage but not of an OA Publication Advantage. Shifts in the affiliation countries of publishing and citing authors are characterized in terms of countries' income class and geographical world region. Suggestions are made for qualitative follow-up studies to obtain more insight into OA flipping or reverse-flipping, Comment: Author copy of a manuscript accepted for publication in the journal Scientometrics (18 May 2020)

The heterogeneity of the Higher Education (HE) Institutions is one of the main critical issues in the assessment of their performance. This paper adopts a multi-level and multi-dimensional perspective, combining national (macro) and institution (micro) level data, and measuring both research and teaching activity, using performance indicators derived from the European Tertiary Education Register, CWTS Leiden Ranking, and PATSTAT patent database. Clustering and efficiency analysis are combined to characterize the heterogeneity of national HE systems in European countries, and reveal the potential of using micro level data to characterize national level performance. Large differences are observed between the European countries, partially due to the fact that they are in different phases of their scientific (and economic) development and of the re-structuring of their HE systems. Evidence is found that universities specializing either in teaching or in research tend to have a higher efficiency than those institutions balancing research and teaching. Tradeoffs are observed between undergraduate and post-graduate activities, and a “Matthew cumulative effect” seems in place on the European institutions analysed: high quality research is able to attract external funds that stimulate innovative and patenting activities that in turn are self-reinforcing to the scientific activities. The results reveal once more the limits and dangers of one-dimensional approaches to the performance of HEIs.

This paper analyses the infuence of a Brazilian institution delivering the corresponding author on its scientifc citation impact, distinguishing between its collaborative papers with foreign institutions and those resulting from national collaboration. We retrieved from Scopus database a total of 607,454 Brazilian documents for all 443 Brazilian institutions with at least 100 documents published from 2003 to 2015. We evaluated the diference between the normalized citation impact as corresponding author and that as non-corresponding author, applying paired t-tests both for international and for national collaboration. As result, for international collaboration, it was observed that the normalized citation impact achieved by Brazilian institutions depends upon corresponding author status, and that, in case of non-corresponding authorship, the impact shows a signifcant beneft when the paper has a corresponding author from a foreign institution. In national collaboration, the institutions beneft as non-corresponding author, although the diference is too small to infuence the practice of the institutions’ scientifc policies. Thus, the indicator of corresponding author provides additional information relevant to Brazilian institutions in international collaboration, but not in national institutional co-authorship, which is more infuenced by the institution’s recognized scientifc tradition and publishing strategies/practices

This editorial gives an outline of the scope an mission of the journal Scholarly Assessment Reports.

IntroductionThe article “Are nationally oriented journals indexed in Scopus becoming more international? The effect of publicationlanguage and access modality” published recently in Journal of Informetrics (Moed et al., 2020) defines the internationalorientation of a journal (INO-P) as the percentage of papers published by the most productive author country in the journal.“Author country” refers to the affiliation country of the publishing authors.The indicator INO-P is based on data elaborated by Scimago Research Group and obtained from Elsevier’s Scopus. It iscalculated by dividing the number of documents published from the most productive author country by the total numberof documents published in the journal, and by multiplying this ratio with the factor 100 in order to express this ratio as apercentage.The two main conclusions on the study were as follows. (figure Presented).

An exploratory, descriptive analysis is presented of the national orientation of scientific, scholarly journals as reflected in the affiliations of publishing or citing authors. It calculates for journals covered in Scopus an Index of National Orientation (INO), and analyses the distribution of INO values across disciplines and countries, and the correlation between INO values and journal impact factors. The study did not find solid evidence that journal impact factors are good measures of journal internationality in terms of the geographical distribution of publishing or citing authors, as the relationship between a journal's national orientation and its citation impact is found to be inverse U-shaped. In addition, journals publishing in English are not necessarily internationally oriented in terms of the affiliations of publishing or citing authors; in social sciences and humanities also USA has their nationally oriented literatures. The paper examines the extent to which nationally oriented journals entering Scopus in earlier years, have become in recent years more international. It is found that in the study set about 40 per cent of such journals does reveal traces of internationalization, while the use of English as publication language and an Open Access (OA) status are important determinants., Preprint of a paper accepted for publication in Journal of Informetrics dd 14 Jan 2020

Agradecimientos al Plan Propio de Investigación de la Universidad de Granada por la financiación a través del Programa Visiting Scholars, Presentado en la 23rd International Conference on Science and Technology Indicators (STI 2018) - Leiden, This paper analyses altmetric differences by type of access to documents. We distinguish between three types of access to the document: green OA, gold OA and non-OA. Here we report a preliminary analysis based on two journals from Physics. We use this descriptive analysis as a proof-of-concept before expanding our study to other fields and introducing more advanced statistical methods. Our results show that gold OA documents are best covered in Altmetric.com and receive higher mentions than documents with other types of access. This is especially troublesome in the case of green OA, as it reflects that altmetric indicators do promote a very specific type of access closely linked with the publishing industry (Torres-Salinas, Robinson-Garcia, & Moed, 2018). Furthermore, it has negative implications for the effectiveness of institutional policies promoting green OA through the creation of repositories and OA infrastructure. This paper does not intend to delve into the reasons behind these differences, but to be a first step. Further research will require the implementation of a large-scale study in which other types of access and documents (i.e., not published pre-prints) could be included, as well as other scientific fields covered., Visiting Scholars programme from the University of Granada

The starting point of this paper is a desktop research assessment model that does not take properly into account the complexities of research assessment, but rather bases itself on a series of highly simplifying, questionable assumptions related to the availability, validity and evaluative significance of research performance indicators, and to funding policy criteria. The paper presents a critique of this model, and proposes alternative assessment approaches, based on an explicit evaluative framework, focusing on preconditions to performance or communication effectiveness rather than on performance itself, combining metrics and expert knowledge, and using metrics primarily to set minimum standards. Giving special attention to early career scientists in emerging research groups, the paper discusses the limits of classical bibliometric indicators and altmetrics. It proposes alternative funding formula of research institutions aimed to support emerging research groups., Comment: Author copy of a paper published in FEMS-Microbiology Letter, August 2018

This chapter focuses on the analysis of current publication trends in gold Open Access (OA). The purpose of the chapter is to develop a full understanding on country patterns, OA journals characteristics and citation differences between gold OA and non-gold OA publications. For this, we will first review current literature regarding Open Access and its relation with its so-called citation advantage. Starting with a chronological perspective we will describe its development, how different countries are promoting OA publishing, and its effects on the journal publishing industry. We will deepen the analysis by investigating the research output produced by different units of analysis. First, we will focus on the production of countries with a special emphasis on citation and disciplinary differences. A point of interest will be identification of national idiosyncrasies and the relation between OA publication and research of local interest. This will lead to our second unit of analysis, OA journals indexed in Web of Science. Here we will deepen on journals characteristics and publisher types to clearly identify factors which may affect citation differences between OA and traditional journals which may not necessarily be derived from the OA factor. Gold OA publishing is being encouraged in many countries as opposed to Green OA. This chapter aims at fully understanding how it affects researchers’ publication patterns and whether it ensures an alleged citation advantage as opposed to non-gold OA publications., Forthcoming in Glanzel, W., Moed, H.F., Schmoch U., Thelwall, M. (2018). Springer Handbook of Science and Technology Indicators. Springer

The purpose of this study was to examine whether scientific mobility, either between countries or between affiliations has an effect on researchers’ productivity and impact. In order to investigate this issue, we examined the relationships between the number of institutional affiliations and countries of the top 100 authors in seven disciplines. The selected authors’ profiles contained the number of affiliations and countries each author is assigned. We studied the number of affiliations and countries and compared them to three bibliometric indicators: the number of publications in international, peer-reviewed journals, h-index and Field Weighted Citations Impact. Our findings show that although there are differences in the relationship between mobility, productivity and impact between disciplines, mobility between at least two affiliations has an overall positive effect on both output and impact while mobility between countries does not. Therefore, in most disciplines positive impact and productivity effects are tracked in affiliation mobility within a single country.

This paper proposes an Ontology-Based Data Management (OBDM) approach to coordinate, integrate and maintain the data needed for Science, Technology and Innovation (STI) policy development. The OBDM approach is a form of integration of information in which the global schema of data is substituted by the conceptual model of the domain, formally specified through an ontology. Implemented in Sapientia, the ontology of multi-dimensional research assessment, it offers a transparent platform as the base for the assessment process; it enables one to define and specify in an unambiguous way the indicators on which the evaluation is based, and to track their evolution over time; also it allows to the analysis of the effects of the actual use of the indicators on the behavior of scholars, and spot opportunistic behaviors; and it provides a monitoring system to track over time the changes in the established evaluation criteria and their consequences for the research system. It is argued that easier access to and a more transparent view of scientific-scholarly outcomes help to improve the understanding of basic science and the communication of research outcomes to the wider public. An OBDM approach could successfully contribute to solve some of the key issues in the integration of heterogeneous data for STI policies.

This article introduces the Multidimensional Research Assessment Matrix of scientific output. Its base notion holds that the choice of metrics to be applied in a research assessment process depends on the unit of assessment, the research dimension to be assessed, and the purposes and policy context of the assessment. An indicator may by highly useful within one assessment process, but less so in another. For instance, publication counts are useful tools to help discriminate between those staff members who are research active, and those who are not, but are of little value if active scientists are to be compared with one another according to their research performance. This paper gives a systematic account of the potential usefulness and limitations of a set of 10 important metrics, including altmetrics, applied at the level of individual articles, individual researchers, research groups, and institutions. It presents a typology of research impact dimensions and indicates which metrics are the most appropriate to measure each dimension. It introduces the concept of a “meta-analysis” of the units under assessment in which metrics are not used as tools to evaluate individual units, but to reach policy inferences regarding the objectives and general setup of an assessment process.

A statistical analysis of full text downloads of articles in Elsevier's ScienceDirect covering all disciplines reveals large differences in download frequencies, their skewness, and their correlation with Scopus-based citation counts, between disciplines, journals, and document types. Download counts tend to be 2 orders of magnitude higher and less skewedly distributed than citations. A mathematical model based on the sum of two exponentials does not adequately capture monthly download counts. The degree of correlation at the article level within a journal is similar to that at the journal level in the discipline covered by that journal, suggesting that the differences between journals are, to a large extent, discipline specific. Despite the fact that in all studied journals download and citation counts per article positively correlate, little overlap may exist between the set of articles appearing in the top of the citation distribution and that with the most frequently downloaded ones. Usage and citation leaks, bulk downloading, differences between reader and author populations in a subject field, the type of document or its content, differences in obsolescence patterns between downloads and citations, and different functions of reading and citing in the research process all provide possible explanations of differences between download and citation distributions.

This study was designed to examine the manner by which academic personnel including students, faculty, librarians and administrators, not only access and read scientific literature bust also interacts with it. In addition, the study examines whether there are differences in accessing, reading and interaction behavior between different groups in academia. Using an online survey, which was completed by 416 respondents worldwide, we found that PDF is the most common format for reading, downloading and saving. Despite the advances of technology, most participants prefer to print PDF pages and note, highlight and summarize them manually. Mobile devices such as E-readers, tablets and smart phones are not very popular as means of interaction with scientific literature.

Policy highlights • This paper criticizes a “quick-and-dirty” desktop model for the use of metrics in the assessment of academic research performance, and proposes a series of alternatives. • It considers often used indicators: publication and citation counts, university rankings, journal impact factors, and social media-based metrics. • It is argued that research output and impact are multi-dimensional concepts; when used to assess individuals and groups, these indicators suffer from severe limitations: • Metrics for individual researchers suggest a “false precision”; university rankings are semi-objective and semi-multidimensional; informetric evidence of the validity of journal impact measures is thin; and social media-based indicators should at best be used as complementary measures. • The paper proposes alternatives to the desktop application model: Combine metrics and expert knowledge; assess research groups rather than individuals; use indicators to define minimum standards; and use funding formula that reward promising, emerging research groups. • It proposes a two-level model in which institutions develop their own assessment and funding policies, combining metrics with expert and background knowledge, while at a national level a meta-institutional agency 'marginally' tests the institutions’ internal assessment 'processes'. • According to this model, an 'inappropriate' type of metrics use is when a 'meta'-institutional agency is concerned directly with the assessment of individuals or groups 'within' an institution. • The proposed model is 'not' politically neutral. A normative assumption is that of the autonomy of academic institutions. The meta-institutional entity acknowledges that it is the primary responsibility of the institutions themselves to conduct quality control. • Rather than having one meta-national agency defining what is research quality and what is not, and how it should be measured, the proposed model facilitates each institution to define its own quality criteria and internal policy objectives, and to make these public. • But this 'freedom' of institutions is accompanied by a series of 'obligations'. As a necessary condition, institutions should conceptualize and implement their internal quality control and funding procedures. • Although a meta-institutional agency may help to improve an institution’s internal processes, a repeatedly negative outcome of a marginal test may have negative consequences for the institution’s research funding. This paper discusses a subject as complex as the assessment of scientific-scholarly research for evaluative purposes. It focuses on the use of informetric or bibliometric indicators in academic research assessment. It proposes a series of analytical distinctions. Moreover, it draws conclusions regarding the validity and usefulness of indicators frequently used in the assessment of individual scholars, scholarly institutions and journals. The paper criticizes a so called desktop application model based upon a set of simplistic, poorly founded assumptions about the potential of indicators and the essence of research evaluation. It proposes a more reflexive, theoretically founded, two-level model for the use of metrics of academic research assessment.

We propose a method for selecting the research guarantor when papers are co-authored. The method is simply based on identifying the corresponding author. The method is here applied to global scientific output based on the SCOPUS database in order to build a new output distribution by country. This new distribution is then compared with previous output distributions by country but which were based on whole or fractional counting, not only for the total output but also for the excellence output (papers belonging to the 10 % most cited papers). The comparison allows one to examine the effect of the different methodological approaches on the scientific performance indicators assigned to countries. In some cases, there was a very large variation in scientific performance between the total output (whole counting) and output as research guarantor. The research guarantor approach is especially interesting when used with the excellence output where the quantity of excellent papers is also a quality indicator. The impact of excellent papers naturally has less variability as they are all top-cited papers.

This manuscript is a preprint version of Part 1 (General Introduction and Synopsis) of the book Applied Evaluative Informetrics, to be published by Springer in the summer of 2017. This book presents an introduction to the field of applied evaluative informetrics, and is written for interested scholars and students from all domains of science and scholarship. It sketches the field’s history, recent achievements, and its potential and limits. It explains the notion of multi-dimensional research performance, and discusses the pros and cons of 28 citation-, patent-, reputation- and altmetrics-based indicators. In addition, it presents quantitative research assessment as an evaluation science, and focuses on the role of extra-informetric factors in the development of indicators, and on the policy context of their application. It also discusses the way forward, both for users and for developers of informetric tools.

This paper aims at contributing to the on-going discussion about building and applying bibliometric indicators. It sheds light on their properties and requirements concerning six different aspects: deterministic versus probabilistic approach, application-related properties, the time dependence, normalization issues, size dependence and network indicators. © 2012 Akadémiai Kiadó, Budapest, Hungary. ispartof: Scientometrics vol:96 issue:1 pages:381-394 status: published

This article presents an analysis of the funding policies of three research councils at the Netherlands Organization for Scientific Research (NWO). The key issue is the extent to which these three councils recognized scientific excellence, and particularly, whether they succeeded in rewarding the grants of the most successful and influential researchers. Data on grant applications provided by NWO for the time period 2000--4 were combined with bibliometric indicators of past research performance of applicants and non-applicants derived from Thomson Reuters' Web of Science. It is found that the three councils did support scientific excellence, in the following sense. Firstly, they tend to attract research proposals from the better groups in the fields they cover. Secondly, the applicants whose submitted proposals were granted--and the research groups they represent--tend to generate a higher citation impact at their international research fronts than those whose submissions were rejected. Although there are some differences in the outcomes among the three councils, this conclusion is valid for each council. On the other hand, for applicants with more than three granted applications we observed a rather variable pattern: in one council these performed at the same level as researchers whose applications were all rejected; in another council these applicants outperformed the rejected applicants; and in another council the number of applicants with more than three granted applications was very small. Copyright The Author 2012. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oup.com, Oxford University Press.

This paper explores a new indicator of journal citation impact, denoted as source normalized impact per paper (SNIP). It measures a journal's contextual citation impact, taking into account characteristics of its properly defined subject field, especially the frequency at which authors cite other papers in their reference lists, the rapidity of maturing of citation impact, and the extent to which a database used for the assessment covers the field's literature. It further develops Eugene Garfield's notions of a field's 'citation potential' defined as the average length of references lists in a field and determining the probability of being cited, and the need in fair performance assessments to correct for differences between subject fields. A journal's subject field is defined as the set of papers citing that journal. SNIP is defined as the ratio of the journal's citation count per paper and the citation potential in its subject field. It aims to allow direct comparison of sources in different subject fields. Citation potential is shown to vary not only between journal subject categories - groupings of journals sharing a research field - or disciplines (e.g., journals in mathematics, engineering and social sciences tend to have lower values than titles in life sciences), but also between journals within the same subject category. For instance, basic journals tend to show higher citation potentials than applied or clinical journals, and journals covering emerging topics higher than periodicals in classical subjects or more general journals. SNIP corrects for such differences. Its strengths and limitations are critically discussed, and suggestions are made for further research. All empirical results are derived from Elsevier's Scopus., Comment: Version 13 November 2009; 23 pages; 1 appendix; 7 tables; 8 figures

There is a greater need to evaluate the research, and yet it is increasingly more complex to do so. The main criticisms that the inadequate use of informetric indicators receive are exposed. Alternative ways to use them to measure the research performance in an informed and responsible way, exploiting its potential adequately, are suggested.

This paper introduces two journal metrics recently endorsed by Elsevier's Scopus: SCImago Journal Rank (SJR) and Source Normalized Impact per Paper (SNIP). SJR weights citations according to the status of the citing journal and aims to measure journal prestige rather than popularity. SNIP compensates for disparities in citation potential and aims to account for differences in topicality across research fields. The paper underlines important points to keep in mind when using journal metrics in general; it presents the main features of the two indicators, comparing them one with another, and with a journal impact measure similar to Thomson Reuters' journal impact factor; and it discusses their potentialities in regard to specific interests of a user and theoretical beliefs about the nature of the concept of journal performance.

This paper explores the use of Library Catalog Analysis (LCA), defined as the application of bibliometric or informetric techniques to a set of library online catalogs, to describe quantitatively a scientific-scholarly field on the basis of published book titles. It focuses on its value as a tool in studies of Social Sciences and Humanities, especially its cognitive structures, main book publishers and the research performance of its actors. The paper proposes an analogy model between traditional citation analysis of journal articles and Library Catalog Analysis of book titles. It presents the outcomes of an exploratory study of book titles in Economics included in 42 academic library catalogs from 7 countries. It describes the process of data collection and cleaning, and applies a series of indicators and thematic mapping techniques. It illustrates how LCA can be fruitfully used to assess book production and research performance at the level of an individual researcher, a research department, an entire country and a book publisher. It discusses a number of issues that should be addressed in follow-up studies and concludes that LCA of published book titles can be developed into a powerful and useful tool in studies of Social Sciences and Humanities.

This paper presents bibliometric characteristics of the 386 most frequently publishing world universities and of a (partly overlapping) set of 529 European universities. Rather than presenting a ranking itself, it presents a statistical analysis of ranking data, focusing on more general patterns. It compares US universities with European institutions; countries with a strong concentration of academic research activities among universities with nations showing a more even distribution; a ranking of universities based on indicators calculated for all research fields combined with one compiled for a single field (oncology); general with specialised universities; and rankings based on a single indicator with maps combining social network analysis and a series of indicators. It highlights important factors that should be taken into account in the interpretation of rankings of research universities based on bibliometric indicators. Moreover, it illustrates policy-relevant research questions that may be addressed in secondary analyses of ranking data. In this way, this paper aims at contributing to a public information system on research universities. Copyright , Beech Tree Publishing.

A longitudinal bibliometric analysis of publications indexed in Thomson Reuters' Incites and Elsevier's Scopus, and published from Persian Gulf States and neighbouring Middle East countries, shows clear effects of major political events during the past 35 years. Predictions made in 2006 by the US diplomat Richard N. Haass on political changes in the Middle East have come true in the Gulf States' national scientific research systems, to the extent that Iran has become in 2015 by far the leading country in the Persian Gulf, and South-East Asian countries including China, Malaysia and South Korea have become major scientific collaborators, displacing the USA and other large Western countries. But collaborations patterns among Persian Gulf States show no apparent relationship with differences in Islam denominations., Version 31 March 2016 accepted for publication in Scientometrics

To provide users insight into the value and limits of world university rankings, a comparative analysis is conducted of 5 ranking systems: ARWU, Leiden, THE, QS and U-Multirank. It links these systems with one another at the level of individual institutions, and analyses the overlap in institutional coverage, geographical coverage, how indicators are calculated from raw data, the skewness of indicator distributions, and statistical correlations between indicators. Four secondary analyses are presented investigating national academic systems and selected pairs of indicators. It is argued that current systems are still one-dimensional in the sense that they provide finalized, seemingly unrelated indicator values rather than offering a data set and tools to observe patterns in multi-faceted data. By systematically comparing different systems, more insight is provided into how their institutional coverage, rating methods, the selection of indicators and their normalizations influence the ranking positions of given institutions., Comment: Author copy of a paper accepted for publication in Scientometrics, 15 Nov. 2016 v2

The paper discusses the strengths and limitations of ‘metrics’ and peer review in large-scale evaluations of scholarly research performance. A real challenge is to combine the two methodologies in such a way that the strength of the first compensates for the limitations of the second, and vice versa. It underlines the need to systematically take into account the unintended effects of the use of metrics. It proposes a set of general criteria for the proper use of bibliometric indicators within peer-review processes, and applies these to a particular case: the UK Research Assessment Exercise (RAE). Copyright , Beech Tree Publishing.

A new methodology is proposed for comparing Google Scholar (GS) with other citation indexes. It focuses on the coverage and citation impact of sources, indexing speed, and data quality, including the effect of duplicate citation counts. The method compares GS with Elsevier's Scopus, and is applied to a limited set of articles published in 12 journals from six subject fields, so that its findings cannot be generalized to all journals or fields. The study is exploratory, and hypothesis generating rather than hypothesis-testing. It confirms findings on source coverage and citation impact obtained in earlier studies. The ratio of GS over Scopus citation varies across subject fields between 1.0 and 4.0, while Open Access journals in the sample show higher ratios than their non-OA counterparts. The linear correlation between GS and Scopus citation counts at the article level is high: Pearson's R is in the range of 0.8-0.9. A median Scopus indexing delay of two months compared to GS is largely though not exclusively due to missing cited references in articles in press in Scopus. The effect of double citation counts in GS due to multiple citations with identical or substantially similar meta-data occurs in less than 2 per cent of cases. Pros and cons of article-based and what is termed as concept-based citation indexes are discussed., Version 26 April 2016 accepted for publication in Journal of Informetrics

We analyze a set of three databases at different levels of aggregation: (a) a database of approximately 10 6 publications from 247 countries published from 1980–2001, (b) a database of 508 academic institutions from the European Union (EU) and 408 institutes from the United States for the 11-year period of 1991–2001, and (c) a database of 2,330 Flemish authors published in the period from 1980–2000. At all levels of aggregation we find that the mean annual growth rates of publications is independent of the number of publications of the various units involved. We also find that the standard deviation of the distribution of annual growth rates decays with the number of publications as a power law with exponent 0.3. These findings are consistent with those of recent studies of systems such as the size of research and development funding budgets of countries, the research publication volumes of U.S. universities, and the size of business firms.

This article examines whether domestic or international mobility of scientists have positive effect on the productivity or impact of their work. We analyzed 100 top producing authors from 7 disciplines and found that mobility between at least two affiliations increases both output (number of publications) and impact (number of citations). However, mobility between countries does not seem to have a positive impact as domestic affiliation mobility.

A framework is proposed for comparing different types of bibliometric indicators, introducing the notion of an Indicator Comparison Report. It provides a comprehensive overview of the main differences and similarities of indicators. The comparison shows both the strong points and the limitations of each of the indicators at stake, rather than over-promoting one indicator and ignoring the benefits of alternative constructs. It focuses on base notions, assumptions, and application contexts, which makes it more intelligible to non-experts. As an illustration, a comparison report is presented for the original and the modified SNIP (Source Normalized Impact per Paper) indicator of journal citation impact., Comment: Author copy of an article accepted for publication in Scientometrics, posted on 17 Oct 2015

The Impact Factor introduced by Eugene Garfield is a fundamental citation-based measure for significance and performance of scientific journals. It is perhaps the most popular bibliometric product used in bibliometrics itself, as well as outside the scientific community. First, a concise review of the background and history of the ISI impact factor and the basic ideas underlying it are given. A cross-citation matrix is used to visualise the construction of the Impact Factor and several related journal citation measures.Both strengths and flaws of the impact factor are discussed. Several attempts made by different authors to introduce more sophisticated journal citation measures and the reasons why many indicators aiming at a correction of methodological limitations of the Impact Factor were not successful are described. The next section is devoted to the analysis of basic technical and methodological aspects. In this context, the most important sources of possible biases and distortions for calculation and use of journal citation measures are studied. Thereafter, main characteristics of application contexts are summarised. The last section is concerned with questions of statistical reliability of journal citation measures. It is shown that in contrast to a common misbelief statistical methods can be applied to discrete "skewed" distributions, and that the statistical reliability of these statistics can be used as a basis for application of journal impact measures in comparative analyses. Finally, the question of sufficiency or insufficiency of a single, howsoever complex measure for characterising the citation impact of scientific journals is discussed.

The article "Caveats for the journal and field normalizations in the CWTS (`Leiden') evaluations of research performance", published by Tobias Opthof and Loet Leydesdorff (arXiv:1002.2769) deals with a subject as important as the application of so called field normalized indicators of citation impact in the assessment of research performance of individual researchers and research groups. Field normalization aims to account for differences in citation practices across scientific-scholarly subject fields. As the primary author of the papers presenting the "Leiden" indicators and of many reports and articles reporting on the outcomes of assessments actually using these measures, I comment on the 3 main issues addressed in the paper by Opthof and Leydesdorff., Letter to the editor; reply to paper arXiv:1002.2769

Trends in the research output and impact of universities in the Netherlands are examined, as reflected in scientific articles in journals pro-cessed for the Science Citation Index. At the level of sub fields in the natural, technical and life sciences, there has been hardly any concentration of research activities among Dutch universities during the 80s and 90s. In the 80s and to a lesser extent in the 90s, levelling of universities' research output in natural and life sciences is a dominant trend. Changes in distribution of students among the universities and the outcomes of evaluation studies conducted in the past probably have affected this trend positively towards uniformity in output. The academic systems in Sweden, Italy, Germany, Spain, Denmark and particularly Great Britain show a stronger concentration of research articles among universities than the Dutch academic system. Copyright , Beech Tree Publishing.

QUANTITATIVE STUDY OF SCIENCE AND TECHNOLOGY is a rapidly developing field, and its development is closely linked to a number of general tendencies in scholarship throughout the world. National governments and research organisations and institutions need systematic evaluations to optimise their research allocations, re-orient their research support, justify research organisations, restructure research in particular fields, or augment research productivity.

This article presents two bibliometric techniques for the study of international scientific migration and international collaboration. The first is based on tracking of an author's career in terms of the institutions from which he or she has published their research articles, and the second on co-authorship relations between researchers active in different countries. To illustrate this analysis, we present results of 17 selected countries.