Search

In this paper, we provide an overview of the seventh annual edition of the CLEF eHealth evaluation lab. CLEF eHealth 2019 continues our evaluation resource building efforts around the easing and support of patients, their next-of-kins, clinical staff, and health scientists in understanding, accessing, and authoring electronic health information in a multilingual setting. This year’s lab advertised three tasks: Task 1 on indexing non-technical summaries of German animal experiments with International Classification of Diseases, Version 10 codes; Task 2 on technology assisted reviews in empirical medicine building on 2017 and 2018 tasks in English; and Task 3 on consumer health search in mono- and multilingual settings that builds on the 2013–18 Information Retrieval tasks. In total nine teams took part in these tasks (six in Task 1 and three in Task 2). Herein, we describe the resources created for these tasks and evaluation methodology adopted. We also provide a brief summary of participants of this year’s challenges and results obtained. As in previous years, the organizers have made data and tools associated with the lab tasks available for future research and development.

The PROMISE network of excellence organized a two-days brainstorming workshop on 30th and 31st May 2012 in Padua, Italy, to discuss and envisage future directions and perspectives for the evaluation of information access and retrieval systems in multiple languages and multiple media. This document reports on the outcomes of this event and provides details about the six envisaged research lines: search applications; contextual evaluation; challenges in test collection design and exploitation; component-based evaluation; ongoing evaluation; and signal-aware evaluation. The ultimate goal of the PROMISE retreat is to stimulate and involve the research community along these research lines and to provide funding agencies with effective and scientifically sound ideas for coordinating and supporting information access research.

Participative Research labOratory for Multimedia and Multilingual Information Systems Evaluation (PROMISE) is a Network of Excellence, starting in conjunction with this first independent CLEF 2010 conference, and designed to support and develop the evaluation of multilingual and multimedia information access systems, largely through the activities taking place in Cross-Language Evaluation Forum (CLEF) today, and taking it forward in important new ways. PROMISE is coordinated by the University of Padua, and comprises 10 partners: the Swedish Institute for Computer Science, the University of Amsterdam, Sapienza University of Rome, University of Applied Sciences of Western Switzerland, the Information Retrieval Facility, the Zurich University of Applied Sciences, the Humboldt University of Berlin, the Evaluation and Language Resources Distribution Agency, and the Centre for the Evaluation of Language Communication Technologies. The single most important step forward for multilingual and multimedia information access which PROMISE will work towards is to provide an open evaluation infrastructure in order to support automation and collaboration in the evaluation process.

The Cross Language Evaluation Forum has been an activity of DELOS for the last eight years. During this period, it has promoted research in the domain of multilingual information retrieval. This activity has produced considerable results; in particular it has encouraged experimentation with all kinds of multilingual information access - from the development of systems for monolingual text retrieval operating on multiple languages to the implementation of full multilingual multimedia search functionality. However, despite the research advances, there are still very few operational systems in existence, and these are limited to the most widely used languages. The challenge that must now be faced is how to best transfer the research results to the wider market place. The paper provides a critical assessment of the main results achieved by CLEF so far and discusses plans now underway to extend its activities in order to have a more direct impact on the application sector with the introduction of a new activity: TrebleCLEF.

The report describes a feasibility study on how to implement multilingual information access functionality in The European Library (TEL) system. This work has been conducted as part of a collaboration between DELOS, the European Network of Excellence on Digital Libraries, funded by the EC Sixth Framework Programme, and the TEL service, fully funded by the participant national libraries, members of the Conference of European National Librarians (CENL), which aims at providing a co-operative framework for integrated access to the major collections of the European national libraries. This report analyzes the viability of two approaches, potentially to be used in conjunction: 1. using an isolated query translation feature, which, as far as possible, isolates the problem of multilingual searching from the remainder of the system, and 2. using expansion techniques and pseudo translation on bibliographical records, to alleviate problems of translation errors and vocabulary coverage often encountered in systems for multilingual searching

The Cross-Language Evaluation Forum (CLEF) is now in its fourth year of activity. We summarize the main lessons learned during this period, outline the state-of-the-art of the research reported in the CLEF experiments and discuss the contribution that this initiative has made to research and development in the multilingual information access domain. We also make proposals for future directions in system evaluation aimed at meeting emerging needs.

The objective of the Cross-Language Evaluation Forum (CLEF) is to promote research in the multilingual information access domain. In this short paper, we list the achievements of CLEF during its first four years of activity and describe how the range of tasks has been considerably expanded during this period. The aim of the paper is to demonstrate the importance of evaluation initiatives with respect to system research and development and to show how essential it is for such initiatives to keep abreast of and even anticipate the emerging needs of both system developers and application communities if they are to have a future.

This paper describes the official runs of the Twente/TNO group for CLEF 2002. We participated in the Dutch and Finnish monolingual and the Dutch bilingual tasks. In addition this paper reports on an experiment that was carried out during the assessment of the Dutch results for the CLEF 2002. The goal of the experiment was to examine possible influences on the assessments caused by the use of highlighting in the assessment program.

This paper describes the official runs of our team for CLEF 2002. Wetook part in the monolingual tasks for each of the seven non-English languages for which CLEF provides document collections (Dutch, Finnish, French, German, Italian, Spanish, and Swedish). We also conducted our first experiments for the bilingual task (English to Dutch, and English to German), and took part in the GIRT and Amaryllis tasks. Finally, we experimented with the combination ofruns.

The Cross-Language Evaluation Forum (CLEF) is now in its fourth year of activity. We summarize the main lessons learned during this period, outline the state-of-the-art of the research reported in the CLEF experiments and discuss the contribution that this initiative has made to research and development in the multilingual information access domain. We also make proposals for future directions in system evaluation aimed at meeting emerging needs.

The aim of the Cross-Language Evaluation Forum (CLEF) is to develop and maintain an infrastructure for the evaluation of information retrieval system operating on European languages in both monolingual and cross-language contexts, and to create test-suites of reusable data that can be employed by system developers for benchmarking purposes. The paper discusses the criteria that have been followed in the creation of the multilingual test-bed and provides information on the evaluation methodology adopted and the actual contents of the test collection

We describe the organization of the CLEF 2001 evaluation campaign, outline the guidelines given to participants, and explain the techniques and measures used in CLEF campaigns for result calculation and analysis

The aim of the Cross-Language Evaluation Forum (CLEF) is to develop and maintain an infrastructure for the evaluation of information retrieval systems operating on European languages in both monolingual and cross-language contexts, and to create testsuites of reusable data that can be employed by system developers for benchmarking purposes. Two CLEF evaluation campaigns have been held so far (CLEF 2000 and CLEF 2001); CLEF 2002 is now under way. The paper describes the objectives and the organisation of these campaigns, and gives a first assessment of the results. In conclusion, plans for future CLEF campaigns are reported.

Starting in 1997, the National Institute of Standards and Technology conducted 3 years of evaluation of cross-language information retrieval systems in the Text REtrieval Conference (TREC). Twentytwo participating systems used topics (test questions) in one language to retrieve documents written in English, French, German, and Italian. A large-scale multilingual test collection has been built and a new technique for building such a collection in a distributed manner was devised.

The goals of the CLEF (Cross-Language Evaluation Forum) series of evaluation campaigns for information retrieval systems operating on European languages are described. The difficulties of organizing an activity which aims at an objective evaluation of systems running on and over a number of different languages are examined. The discussion includes an analysis of the first results and proposals for possible developments in the future.

The Cross-Language Evaluation Forum (CLEF) provides an infrastructure aimed at supporting the development, testing and evaluation systems for cross-language information retrieval, and for monolingual information retrieval of European Languages other than English. Originally started as a track at the TREC-6 conference, CLEF became an independent initiative in 2000 when the coordination moved to Europe. The diversity of languages commonly spoken in Europe has led to a multilingual distributed setup that was chosen to best accommodate the unique linguistic properties of each language and the implications for topic development and relevance assessments. The task in CLEF 2000 involved a multilingual document collection in four core languages and several additional topic languages.

There are three main ways in which cross-language information retrieval approaches attempt to "cross the language barrier" - through query translation, or document translation, or both. (Oard, 1997). CLIR research started out with experiments using controlled vocabularies and associated dictionaries and thesauri, but nowadays free text approaches are most common. These approaches also dominate experiments in past and present CLIR tracks. Free text methods can be further classified according to the resources used to cross the language boundary: machine translation, machine-readable dictionaries, or corpus-based resources. Machine translation (MT) seems an obvious choice for cross-language information retrieval systems. It also played a large role in the TREC-8 experiments of a number of groups. However, CLIR is a difficult problem to solve on the basis of MT alone: queries that users typically enter into a retrieval system are rarely complete sentences and provide little context for sense disambiguation.Corpus-based approaches are also popular. Groups experimenting with such approaches during this or former CLIR tracks include Eurospider, IBM and the University of Montreal.Lastly, a significant number of cross-language retrieval approaches make use of existing linguistic resources, mainly machine-readable bilingual dictionaries. Various ideas have been proposed to address some of the problems associated with dictionary-based translations, such as ambiguities and vocabulary coverage. One of the groups that have investigated the use of such dictionaries is the Twenty-One consortium.

We describe the creation of an infrastructure for the testing of cross-language text retrieval systems within the context of the Text REtrieval Conferences (TREC) organised by the US National Institute of Standards and Technology (NIST). The approach adopted and the issues that had to be taken into consideration when building a multilingual test suite and developing appropriate evaluation procedures to test cross-language systems are described. From 2000 on, a cross-language evaluation activity for European languages known as CLEF (Cross-Language Evaluation Forum) will be coordinated in Europe, while TREC will focus on Asian languages. The implications of the move to Europe and the intentions for the future are discussed.

An abstract is not available.

Each national site is responsible for maintaining and operating a search service for its own languages, so that the needs of distinct language communities can be catered for by native speakers of that language. The languages covered are currently French, German, Italian and Spanish, plus also English. Differences in the partners' document collections and indexing mechanisms have led to the implementation of different search strategies, depending on the collection to be queried. The cross-language search component of EuroSearch thus consists of an integration of lexicon- and corpus-based search mechanisms and two distinct types of searching will be activated: o query translation using a multilingual lexicon; enhanced by an experimental corpus-based mechanism o similarity thesaurus technology.

We test the utility of European language stemmers created using the Snowball language [1]. This allows us to experiment with PLIERS in languages other than English. We also report on some BM25 tuning constant experiments conducted in order to find the best settings for our searches.

The Personalised Information Retrieval Lab (PIR-CLEF 2019) lab is an initiative aimed at both providing and critically analysing the evaluation of Personalization in Information Retrieval (PIR) applications. PIR-CLEF 2019 is the second edition of the Lab after the successful Pilot lab organised at CLEF 2017 and the first edition of the Lab at CLEF 2018. PIR-CLEF 2019 provided registered participants with two tracks: the Web Search Task and the Medical Search Task. The Web Search Task continues the activities introduced in the previous editions of the PIR-CLEF Lab, while the Medical Search Track focuses on personalisation within an ad hoc search task introduced in previous editions of the CLEF eHealth Lab.

Se describe algunos experimentos realizados para estudiar la relación de los términos, sobre todo de la asociación y semejanza, usados en los tesauros españoles con el objetivo de mejorar la recuperación de la información., This paper presents some experiments carried out this year in the Spanish monolingual task at CLEF2002. The objective is to continue our research on term expansion. Last year we presented results regarding stemming. Now, our effort is centred on term expansion using thesauri. Many words that derive from the same stem have a close semantic content. However other words with very different stems also have semantically close senses. In this case, the analysis of the relationships between words in a document collection can be used to construct a thesaurus of related terms. The thesaurus can then be used to expand a term with the best related terms. This paper describes some experiments carried out to study term expansion using association and similarity thesauri.

Sheffield’s contribution to the interactive cross language information retrieval track took the approach of comparing users’ abilities to judge the relevance of machine translated French documents against ones written in the users’ native language: English. Conducting such an experiment is challenging, and the issues surrounding the experimental design are discussed. Experimental results strongly suggest that users are just as capable of judging relevance of translated documents as they are for documents in their native language