Search

In this article, we compare the reception of Spanish sociology by domestic and international audiences using citation counts as an indicator of audience interest. We compare papers highly cited in a national language database with those highly cited in the Web of Science. We find differences that might prove worrying within a national research evaluation system that emphasizes publishing in high impact factor, Web of Science indexed journals. We suggest that in the social sciences, such an emphasis may induce not simply more research excellence, but also narrower research agendas with long-term consequences for the Spanish sociology community. [ABSTRACT FROM AUTHOR]

Using the population of all university professors (N = 13,479) in the province of Quebec, Canada, this article analyses the concentration of funding, papers and citations at the level of individual researchers. It shows that each of these distributions is different, citations being the most concentrated followed by funding, papers published and, finally, number of funded projects. Concentration measures also vary between disciplines; social sciences and humanities generally being the most concentrated. The article also shows that the correspondence between the elites defined by each of these measures is limited. In fact, only 3.2% of the researchers are in the top 10% for all indicators, while about 20% are in the top 10% for at least one of the indicators. The article concludes with a discussion of the causes of these observed differences and formulates a few hypotheses. [ABSTRACT FROM AUTHOR]

Applying 'standard' publication and citation measures to the social sciences is fast becoming an outmoded practice, yet we have still to develop credible quantitative alternatives to inform research evaluation exercises. This paper reports the outcomes of a comparative pilot study of five Australian economics departments which tested data produced using novel bibliometric, esteem, and 'user engagement' measures. The results were presented to a group of expert peers drawn from the economics groups studied. There was support for developing some novel bibliometric indicators and 'user engagement' measures, but esteem indicators were roundly rejected. [ABSTRACT FROM AUTHOR]

In this paper, we report on the application of Google Scholar (GS)-based metrics in the formal assessment of research programs. Involved were programs in the fields of Education, Pedagogical Sciences, and Anthropology in The Netherlands. Also, a comparative analysis has been conducted of the results based on GS and Web of Science (WoS). Studies critical of GS point at its reliability of data. We show how the reliability of the GS data for the bibliometric analysis of the assessment can be improved by excluding non-verifiable citing sources from the full secondorder GS citing data. The study of the background of these second-order sources demonstrates a broadening of the citing sources. The comparison of GS with WoS citations for the publications of the programs shows that it is promising to use GS for fields with lower degrees of coverage in WoS, in particular for fields that produce more diverse types of output than just research articles. Restrictions to the use of GS are the intensive manual data handling and cleaning, necessary for a feasible and proper data collection. We discuss wider implications of the findings for bibliometric analysis and for the practices and policies in research evaluation. [ABSTRACT FROM AUTHOR]

This article uses high- and low-impact citation indicators for the evaluation of the citation performance of research units at different aggregate levels using a dataset of about 3.6 million articles published in 1998-2002 in the natural and the social sciences with a 5-year citation window. The difficulty is that a large proportion of individual articles are assigned to multiple subfields. To control for wide differences in citation practices at the subfield level, we apply a novel normalization procedure in the multiplicative approach in which each paper is wholly counted as many times as necessary in the several categories to which it is assigned at each aggregation level. The methodology is applied to a partition of the world into three geographical areas: the USA, the European Union (EU), and the Rest of the World. The main findings are the following two. (1) Although normalization does not systematically bias the results against any area, it reduces the US/EU high-impact gap in the all-sciences case by a non-negligible 14.4%. (2) The dominance of the USA over the EU in the basic and applied research published in the periodical literature is almost universal at all aggregation levels. From the high-impact perspective, for example, the USA is ahead of the EU in 77 out of 80 disciplines, and all of 20 fields. For all sciences as a whole, the US high-impact indicator is 61% greater than that of the EU. [ABSTRACT FROM AUTHOR]

Recently, the need to contribute to the evaluation of the scientific, social, and political impact of Social Sciences and Humanities (SSH) research has become a demand of policy makers and society. The international scientific community has made significant advances that have transformed the impact of evaluation landscape. This article reviews the existing scientific knowledge on evaluation tools and techniques that are applied to assess the scientific impact of SSH research; the changing structure of social and political impacts of SSH research is investigated based on an overarching research question: to what extent do scholars attempt to apply methods, instruments, and approaches that take into account the distinctive features of SSH? The review also includes examples of European Union (EU) projects that demonstrate these impacts. This article culminates in a discussion of the development of the assessment of different impacts and identifies limitations, and areas and topics to explore in the future. [ABSTRACT FROM AUTHOR]

This paper enquires into eponymy as an ex post approach for research evaluation giving modalities of presentation, a developmental procedure and critique for this complex phenomenon. Spanish cases of eponymy from the educational field are reviewed. They were obtained from a name index from educational treaties, a local data base and informal interviews with experts. [ABSTRACT FROM AUTHOR]

In this study we carried out a bibliometric analysis of the social sciences at three European universities (Vienna, Zurich, Oslo). The data source for this investigation was the Web of Science SSCI and publications between 2000 and 2006 were included for retrieval. Apart from the analysis of the publication output, activity and impact, networks of co-authorships, disciplines and references were explored. In general the results reveal that overall Oslo outperformed the other two universities which show similar publication activities and outputs. Using the University of Oslo as a benchmark should help to outline different strategies (research, publication and cooperation) to enhance the international visibility of the output of the University of Vienna in different fields of the social sciences. [ABSTRACT FROM AUTHOR]

Impact is at one and the same time an object of derision and acclaim, anxiety and confidence. It is a troubled terrain, discussed from quite different directions, and there seems little prospect of developing a common conversation between those who traverse it. This reflective paper seeks to outline a common core of questions that define the impact of impact. While it offers no answers to them, it establishes the grounds on which the debate can at least be taken forward in the future. These questions are: What is impact? Impact for whom? What are the domains in which it is displayed? What are its indicators? How is it measured? [ABSTRACT FROM AUTHOR]

This article explores the current literature on 'research impact' in the social sciences and humanities (SSH). By providing a comprehensive review of available literature, drawing on national and international experiences, we take a systematic look at the impact agenda within SSH. The primary objective of this article is to examine key methodological components used to assess research impact comparing the advantages and disadvantages of each method. The study finds that research impact is a highly complex and contested concept in the SSH literature. Drawing on the strong methodological pluralism emerging in the literature, we conclude that there is considerable room for researchers, universities, and funding agencies to establish impact assessment tools directed towards specific missions while avoiding catch-all indicators and universal metrics. [ABSTRACT FROM AUTHOR]

Increasingly, research funders include societal impact as a criterion in evaluation procedures. The European Commission is no exception to this trend. Societal impact determines one-third of a project's success in receiving funding from the Societal Challenges in Horizon 2020 (H2020). Yet, there are large differences in terms of science and technology performance between countries that participate in the programme. In this article, we (1) compare societal impact practices in the social sciences and humanities in high-performing countries (HPCs) and low-performing countries (LPCs) to the evaluation of societal impact in funding procedures at the European level and (2) reflect upon consequences for the competition for research funding in the European funding arena. To this end, we introduce the concept of 'societal impact capacity' as well as a framework to analyse it. The analysis of 60 case studies from 16 countries across Europe shows that (1) researchers from HPCs have a higher impact capacity than those from LPCs and (2) researchers from HPCs report more details about impact than those from LPCs. This suggests that researchers from HPCs are better equipped to score well on the impact criterion when applying for funding than researchers from LPCs. We conclude with policy recommendations for the organization and evaluation of societal impact. [ABSTRACT FROM AUTHOR]

The capacity to recognize the contribution of individual researchers and their organizational unit is likely to be at odds with the purpose of a national research assessment, due to the broader approach to identifying research strengths and its classification at disciplinary levels. The Australian research assessment exercise, Excellence in Research for Australia (ERA), is devised to determine research quality in this wider context. To explore the impact of research classification on the visibility of research contribution, this study analysed articles published by authors affiliated with two social sciences and humanities (SSH) fields--'education' and 'language, communication and culture'. Article classification was examined for agreement with the authors' affiliated organizational unit's field. The influence of national and international co-authorship on article classification was also examined. Articles were distributed across different classification categories and a substantial proportion did not match with the authors' organizational unit's field. National and international co-authorship by education-affiliated authors was found to be significantly associated with publishing outside their organizational unit's field. As the first investigation to focus on how research is distributed by the ERA's classification scheme when applied to journal articles by SSH authors, the study provides empirical evidence of the challenges involved in recognizing the contribution of organizational units. This work builds on the existing literature relating to classification and research evaluation and has the potential to inform research managers of the complexities in setting strategic research priorities based on ERA outcomes. [ABSTRACT FROM AUTHOR]

Evaluation has relied on survey data sources. Different fields of social sciences (political science, sociology, and economics) were revolutionized using modern survey methods and techniques for analysing these data. However, the research frontier moves to the use of administrative data. Administrative data are highly preferable to survey data because (1) full population files are generally available and offer much larger sample sizes, (2) administrative data have a longitudinal structure that enables evaluators to follow individuals over time and address many critical policy questions, (3) administrative data do not suffer from high rates of non-response, attrition, and under-reporting. This article presents results from an evaluation of publicly funded R&D assistance provided to New Zealand firms. The grants are intended to lead to additional R&D undertaken by the firm, which in turn is expected to increase the economic performance of a firm. We use the Statistics New Zealand prototype Longitudinal Business Database (LBD), which contains high quality and comprehensive administrative firm-level data. The problem of evaluation is that while the programmes’ impact can truly be assessed only by comparing actual and counterfactual outcomes, the counterfactual is not observed. So the challenge of an evaluation is to create a convincing and reasonable comparison group for assisted firms in light of this ‘missing data’. Our methodology involves matching firms that received assistance to comparable unassisted firms based on firm characteristics. We then compare changes in performances of the assisted group with the group of matched unassisted firms. [ABSTRACT FROM PUBLISHER]

Books are a key channel in the scientific communication processes for researchers in Social Sciences and Humanities. Although the best way of assessing the monographs would involve its individualized reading, this method is practically unfeasible both in terms of cost and time. An indicator which could be used is the prestige of the publisher of monographs and it is essential to take into account the academy’s opinion to determine it. This article reports a survey launched to 11,647 lecturers and Spanish researchers, with a response rate of 26%, in which the respondents were asked to point those quality indicators they consider the most relevant for books as well as their opinion on the quality of publishers. As the main results, there are reported the top-rated publishers, as well as the characteristics which, according to the respondents’ opinion, determine their quality; among these characteristics, stands out the fact of counting with an assessment system with external reviewers. Regarding the list of publishers, it is also explained the indicator which allows its ranking. It is also defended the use of the results in the form of categorization. [ABSTRACT FROM AUTHOR]

This article studies publication patterns in the social sciences and humanities (SSH) in Flanders and Norway using two databases that both cover all SSH peer-reviewed journal articles by university scholars for the period 2005-9. The coverage of journal articles by the Web of Science (WoS) and the proportion of articles published in English are studied in detail applying the same methodologies to both databases. The study of WoS coverage and language use is chosen because the performance-based funding systems that are in place in both countries have given different emphasis to publishing in WoS covered journals. The results show very similar, almost identical evolutions in the use of English as a publication language. The proportion of articles covered by the WoS, however, is stable for Norway but has increased rapidly for Flanders. This finding shows that the parameters used in a performance-based funding system may influence the publishing patterns of researchers. [ABSTRACT FROM AUTHOR]

Does past performance influence success in grant applications? We tested whether the decisions of the Netherlands Research Council for the Economic and Social Sciences correlate with the past performances of applicants in publications and citations, and with the results of the Council's peer reviews. The Council proves successful in distinguishing grant applicants with above-average from below-average performance, but within the former group there was no correlation between past performance and receiving a grant. When comparing the best-performing researchers who were denied funding with those who received it, the rejected researchers significantly outperformed the funded ones. The best rejected proposals score on average as high on the outcomes of the peer-review process as the accepted proposals. The Council successfully corrected for gender effects during the selection process. We explain why these findings may apply beyond this case. However, if research councils are not able to select the 'best' researchers, perhaps they should reconsider their mission. We discuss the role of research councils in the science system in terms of variation, innovation and quality control. [ABSTRACT FROM AUTHOR]

An overview of current scientific journals evaluation initiatives in Spain is presented. Most of these models are focused on social sciences and humanities journals because of their special features, and are targeted to research evaluation activities. Indicators developed in these systems are analysed, including the methodological difficulties that they entail and the relative merits of each in terms of efficiency. Moreover, the criteria used by the evaluation agencies in relation to scientific journals are examined. Finally, a model based on a diverse range of indicators on editorial quality, international visibility, peer review and impact is proposed. [ABSTRACT FROM AUTHOR]