Search

This paper describes the PASCAL Network of Excellence first Recognising Textual Entailment (RTE-1) Challenge benchmarkSee http://www.pascal-network.org/Challenges/RTE/ for the first and second RTE challenges.. The RTE task is defined as recognizing, given two text fragments, whether the meaning of one text can be inferred (entailed) from the other. This application-independent task is suggested as capturing major inferences about the variability of semantic expression which are commonly needed across multiple applications. The Challenge has raised noticeable attention in the research community, attracting 17 submissions from diverse groups, suggesting the generic relevance of the task. [ABSTRACT FROM AUTHOR]

The PASCAL Challenge's textual entailment recognition task, or RTE, presents intriguing opportunities to test various implications of the strong language universal constraint posited by Wu's (1995, 1997) Inversion Transduction Grammar (ITG) hypothesis. The ITG Hypothesis provides a strong inductive bias, and has been repeatedly shown empirically to yield both efficiency and accuracy gains for numerous language acquisition tasks. Since the RTE challenge abstracts over many tasks, it invites meaningful analysis of the ITG Hypothesis across tasks including information retrieval, comparable documents, reading comprehension, question answering, information extraction, machine translation, and paraphrase acquisition. We investigate two new models for the RTE problem that employ simple generic Bracketing ITGs. Experimental results show that, even in the absence of any thesaurus to accommodate lexical variation between the Text and the Hypothesis strings, surprisingly strong results for a number of the task subsets are obtainable from the Bracketing ITG's structure matching bias alone. [ABSTRACT FROM AUTHOR]

We describe the system we used at the PASCAL-2005 Recognizing Textual Entailment Challenge. Our method for recognizing entailment is based on calculating "directed" sentence similarity: checking the directed "semantic" word overlap between the text and the hypothesis. We use frequency-based term weighting in combination with two different word similarity measures. Although one version of the system shows significant improvement over randomly guessing decisions (with an accuracy score of 57.3), we show that this is only due to a subset of the data that can be equally well handled by simple word overlap. Furthermore, we give an in-depth analysis of the system and the data of the challenge. [ABSTRACT FROM AUTHOR]

This paper describes the system that LCC has devised to perform textual entailment recognition for the PASCAL RTE Challenge. Our system transforms each text-hypothesis pair into a two-layered logic form representation that expresses the lexical, syntactic, and semantic attributes of the text and hypothesis. A large set of natural language axioms are constructed for each text-hypothesis pair that help connect concepts in the hypothesis with concepts in the text. Our natural language logic prover is then used to prove entailment through abductive reasoning. The system's performance in the challenge resulted in an accuracy of 55%. [ABSTRACT FROM AUTHOR]

We use logical inference techniques for recognising textual entailment, with theorem proving operating on deep semantic interpretations as the backbone of our system. However, the performance of theorem proving on its own turns out to be highly dependent on a wide range of background knowledge, which is not necessarily included in publically available knowledge sources. Therefore, we achieve robustness via two extensions. Firstly, we incorporate model building, a technique borrowed from automated reasoning, and show that it is a useful robust method to approximate entailment. Secondly, we use machine learning to combine these deep semantic analysis techniques with simple shallow word overlap. The resulting hybrid model achieves high accuracy on the RTE testset, given the state of the art. Our results also show that the various techniques that we employ perform very differently on some of the subsets of the RTE corpus and as a result, it is useful to use the nature of the dataset as a feature. [ABSTRACT FROM AUTHOR]

This paper describes Macquarie University's Centre for Language Technology contribution to the PASCAL 2005 Recognizing Textual Entailment challenge. Our main aim was to test the practicability of a purely logical approach. For this, atomic propositions were extracted from both the text and the entailment hypothesis and they were expressed in a custom logical notation. The text entails the hypothesis if every proposition of the hypothesis is entailed by some proposition in the text. To extract the propositions and encode them into a logical notation the system uses the output of Link Parser. To detect the independent entailment relations the system relies on the use of Otter and WordNet. [ABSTRACT FROM AUTHOR]

In this paper we present a classifier for Recognising Textual Entailment (RTE) and Semantic Equivalence. We evaluate the performance of this classifier using an evaluation framework provided by the PASCAL RTE Challenge Workshop. Sentence-pairs are represented as a set of features, which are used by our decision tree classifier to determine if an entailment relationship exisits between each sentence-pair in the RTE test corpus. [ABSTRACT FROM AUTHOR]

The system for semantic evaluation VENSES (Venice Semantic Evaluation System) is organized as a pipeline of two subsystems: the first is a reduced version of GETARUN, our system for Text Understanding. The output of the system is a flat list of augmented head-dependent structures with Grammatical Relations and Semantic Roles labels. The evaluation system is made up of two main modules: the first is a sequence of linguistic rules; the second is a quantitatively based measurement of input structures and predicates. VENSES measures semantic similarity which may range from identical linguistic items, to synonymous, lexically similar, or just morphologically derivable. Both modules go through General Consistency checks which are targeted to high level semantic attributes like presence of modality, negation, and opacity operators, temporal and spatial location checks. Results in cws, recall and precision are homogeneous for both training and test corpus and fare higher than 60%. [ABSTRACT FROM AUTHOR]

The Computational Linguistics at Concordia laboratory system for textual entailment determination is based on shallow, partial predicate-argument structure matching combined with a WordNet-based lexical similarity measure. In this paper we describe experiments with different system settings conducted to assess the potential and limitations of partial predicate-argument structures in textual entailment determination. [ABSTRACT FROM AUTHOR]

In this paper, we discuss paradigms for evaluating open-domain semantic interpretation as they apply to the PASCAL Recognizing Textual Entailment (RTE) evaluation (Dagan et al. 2005). We focus on three aspects critical to a successful evaluation: creation of large quantities of reasonably good training data, analysis of inter-annotator agreement, and joint analysis of test item difficulty and test-taker proficiency (Rasch analysis). We found that although RTE does not correspond to a "real" or naturally occurring language processing task, it nonetheless provides clear and simple metrics, a tolerable cost of corpus development, good annotator reliability (with the potential to exploit the remaining variability), and the possibility of finding noisy but plentiful training material. [ABSTRACT FROM AUTHOR]

This paper describes the Bar-Ilan system participating in the Recognising Textual Entailment Challenge. The paper proposes first a general probabilistic setting that formalizes the notion of textual entailment. We then describe a concrete alignment-based model for lexical entailment, which utilizes web co-occurrence statistics in a bag of words representation. Finally, we report the results of the model on the Recognising Textual Entailment challenge dataset along with some analysis. [ABSTRACT FROM AUTHOR]

Semantic entailment is the problem of determining if the meaning of a given sentence entails that of another. We present a principled approach to semantic entailment that builds on inducing re-representations of text snippets into a hierarchical knowledge representation along with an optimization-based inferential mechanism that makes use of it to prove semantic entailment. This paper provides details and analysis of the knowledge representation and knowledge resources issues encountered. We analyze our system's behavior on the PASCAL text collectionhttp://www.pascal-network.org/Challenges/RTE/ and the PARC collection of question-answer pairsThe data is available at http://l2r.cs.uiuc.edu/~cogcomp/data.php. This is used to motivate and explain some of the design decisions in our hierarchical knowledge representation, that is centered around a predicate-argument type abstract representation of text. [ABSTRACT FROM AUTHOR]

Textual Entailment recognition is a very difficult task as it is one of the fundamental problems in any semantic theory of natural language. As in many other NLP tasks, Machine Learning may offer important tools to better understand the problem. In this paper, we will investigate the usefulness of Machine Learning algorithms to address an apparently simple and well defined classification problem: the recognition of Textual Entailment. Due to its specificity, we propose an original feature space, the distance feature space, where we model the distance between the elements of the candidate entailment pairs. The method has been tested on the data of the Recognizing Textual Entailment (RTE) Challenge. [ABSTRACT FROM AUTHOR]

The Recognizing Textual Entailment System shown here is based on the use of a broad-coverage parser to extract dependency relationships; in addition, WordNet relations are used to recognize entailment at the lexical level. The work investigates whether the mapping of dependency trees from text and hypothesis give better evidence of entailment than the matching of plain text alone. While the use of WordNet seems to improve system's performance, the notion of mapping between trees here explored (inclusion) shows no improvement, suggesting that other notions of tree mappings should be explored such as tree edit distances or tree alignment distances. [ABSTRACT FROM AUTHOR]

This paper addresses Textual Entailment (i.e. recognizing that the meaning of a text entails the meaning of another text) using a Tree Edit Distance algorithm between the syntactic trees of the two texts. A key aspect of the approach is the estimation of the cost for the editing operations (i.e. Insertion, Deletion, Substitution) among words. The aim of the paper is to compare the contribution of two different lexical resources for recognizing textual entailment: WordNet and a word-similarity database. In both cases we derive entailment rules that are used by the Tree Edit Distance Algorithm. We carried out a number of experiments over the PASCAL-RTE dataset in order to estimate the contribution of different combinations of the available resources. [ABSTRACT FROM AUTHOR]

We describe our submission to the PASCAL Recognizing Textual Entailment Challenge, which attempts to isolate the set of Text-Hypothesis pairs whose categorization can be accurately predicted based solely on syntactic cues. Two human annotators examined each pair, showing that a surprisingly large proportion of the data - 34% of the test items - can be handled with syntax alone, while adding information from a general-purpose thesaurus increases this to 48%. [ABSTRACT FROM AUTHOR]

The Bleu algorithm has been used in many different fields. Another possible application is the automatic recognition of textual entailment. Bleu works at the lexical level, by comparing a candidate text with several reference texts in order to calculate how close the candidate text is to the references. In this case, the candidate is the text part of the entailment and the hypothesis is the unique reference. The algorithm achieves an accuracy of around 50%. Moreover, in this paper we explore the application of Bleu-like algorithms, finding that they can reach an accuracy of around 56%, which proves its possible use as a baseline for the task of recognizing entailment. [ABSTRACT FROM AUTHOR]

The PASCAL Visual Object Classes Challenge ran from February to March 2005. The goal of the challenge was to recognize objects from a number of visual object classes in realistic scenes (i.e. not pre-segmented objects). Four object classes were selected: motorbikes, bicycles, cars and people. Twelve teams entered the challenge. In this chapter we provide details of the datasets, algorithms used by the teams, evaluation criteria, and results achieved. [ABSTRACT FROM AUTHOR]

In this paper we present lessons learned in the Evaluating Predictive Uncertainty Challenge. We describe the methods we used in regression challenges, including our winning method for the Outaouais data set. We then turn our attention to the more general problem of scoring in probabilistic machine learning challenges. It is widely accepted that scoring rules should be proper in the sense that the true generative distribution has the best expected score; we note that while this is useful, it does not guarantee finding the best methods for practical machine learning tasks. We point out some problems in local scoring rules such as the negative logarithm of predictive density (NLPD), and illustrate with examples that many of these problems can be avoided by a distance-sensitive rule such as the continuous ranked probability score (CRPS). [ABSTRACT FROM AUTHOR]

This article describes the competitive associative net called CAN2 and cross-validation which we have used for making prediction and estimating predictive uncertainty on the regression problems at the Evaluating Predictive Uncertainty Challenge. The CAN2 with an efficient batch learning method for reducing empirical (training) error is combined with cross-validation for making prediction (generalization) error small and estimating predictive distribution accurately. From an analogy of Bayesian learning, a stochastic analysis is derived to indicate a validity of our method. [ABSTRACT FROM AUTHOR]

In many regression tasks, in addition to an accurate estimate of the conditional mean of the target distribution, an indication of the predictive uncertainty is also required. There are two principal sources of this uncertainty: the noise process contaminating the data and the uncertainty in estimating the model parameters based on a limited sample of training data. Both of them can be summarised in the predictive variance which can then be used to give confidence intervals. In this paper, we present various schemes for providing predictive variances for kernel ridge regression, especially in the case of a heteroscedastic regression, where the variance of the noise process contaminating the data is a smooth function of the explanatory variables. The use of leave-one-out cross-validation is shown to eliminate the bias inherent in estimates of the predictive variance. Results obtained on all three regression tasks comprising the predictive uncertainty challenge demonstrate the value of this approach. [ABSTRACT FROM AUTHOR]

Decision trees, a popular choice for classification, have their limitation in providing probability estimates, requiring smoothing at the leaves. Typically, smoothing methods such as Laplace or m-estimate are applied at the decision tree leaves to overcome the systematic bias introduced by the frequency-based estimates. In this work, we show that an ensemble of decision trees significantly improves the quality of the probability estimates produced at the decision tree leaves. The ensemble overcomes the myopia of the leaf frequency based estimates. We show the effectiveness of the probabilistic decision trees as a part of the Predictive Uncertainty Challenge. We also include three additional highly imbalanced datasets in our study. We show that the ensemble methods significantly improve not only the quality of the probability estimates but also the AUC for the imbalanced datasets. [ABSTRACT FROM AUTHOR]

I submitted entries for the two classification problems — "Catalysis" and "Gatineau" — in the Evaluating Predictive Uncertainty Challenge. My entry for Catalysis was the best one; my entry for Gatineau was the third best, behind two similar entries by Nitesh Chawla. The Catalysis dataset was later revealed to be about predicting a property of yeast proteins from expression levels of the genes encoding them. The nature of the Gatineau dataset has not been revealed, for proprietary reasons. The two datasets are similar in number of input variables that are available for predicting the binary outcome (617 for Catalysis, 1092 for Gatineau). They differ substantially in the number of cases available for training (1173 for Catalysis, 5176 for Gatineau) and in the fractions of cases that are in the two classes (43%/57% for Catalysis, 9%/91% for Gatineau). [ABSTRACT FROM AUTHOR]

We describe an approach to regression based on building a probabilistic model with the aid of visualization. The "stereopsis" data set in the predictive uncertainty challenge is used as a case study, for which we constructed a mixture of neural network experts model. We describe both the ideal Bayesian approach and computational shortcuts required to obtain timely results. [ABSTRACT FROM AUTHOR]

This Chapter presents the PASCALPattern Analysis, Statistical Modelling and Computational Learning (PASCAL) Network of Excellence, part of the IST Programme of the European Community, IST-2002-506778. Evaluating Predictive Uncertainty Challenge, introduces the contributed Chapters by the participants who obtained outstanding results, and provides a discussion with some lessons to be learnt. The Challenge was set up to evaluate the ability of Machine Learning algorithms to provide good "probabilistic predictions", rather than just the usual "point predictions" with no measure of uncertainty, in regression and classification problems. Parti-cipants had to compete on a number of regression and classification tasks, and were evaluated by both traditional losses that only take into account point predictions and losses we proposed that evaluate the quality of the probabilistic predictions. [ABSTRACT FROM AUTHOR]

This paper introduces DoMDL, a powerful and flexible document model capable to represent multi-edition, structured, multimedia documents that can be disseminated in multiple manifestation formats. This model also allows any document to be associated with multiple metadata descriptions in different formats and to include semantic relationships with other documents and parts of them. The paper discusses also how the OpenDLib Digital Library Management System exploits this model to abstract from the specific organization and structure of the documents that are imported from different heterogeneous information sources in order to provide virtual documents that fulfill the needs of the different DL user communities. [ABSTRACT FROM AUTHOR]

The architecture of a digital library service system strongly influences its capabilities. In this paper we report on our experience with a distributed digital library system, OpenDLib, and we describe why, in the attempt to better satisfy the user requirements, we decided to develop DILIGENT, a service-oriented digital library infrastructure on the Grid. The paper describes and compare the two systems by focusing, in particular, on that part of the architecture that controls and supplies the necessary features for creating and managing digital libraries. [ABSTRACT FROM AUTHOR]

While there is strong motivation for using Gaussian Processes (GPs) due to their excellent performance in regression and classification problems, their computational complexity makes them impractical when the size of the training set exceeds a few thousand cases. This has motivated the recent proliferation of a number of cost-effective approximations to GPs, both for classification and for regression. In this paper we analyze one popular approximation to GPs for regression: the reduced rank approximation. While generally GPs are equivalent to infinite linear models, we show that Reduced Rank Gaussian Processes (RRGPs) are equivalent to finite sparse linear models. We also introduce the concept of degenerate GPs and show that they correspond to inappropriate priors. We show how to modify the RRGP to prevent it from being degenerate at test time. Training RRGPs consists both in learning the covariance function hyperparameters and the support set. We propose a method for learning hyperparameters for a given support set. We also review the Sparse Greedy GP (SGGP) approximation (Smola and Bartlett, 2001), which is a way of learning the support set for given hyperparameters based on approximating the posterior. We propose an alternative method to the SGGP that has better generalization capabilities. Finally we make experiments to compare the different ways of training a RRGP. We provide some Matlab code for learning RRGPs. [ABSTRACT FROM AUTHOR]

This paper presents a new technique for supporting query formulation and processing experimentally integrated in the OpenDLib search service. This technique provides a better support for unified search by enhancing the capability of the digital library to satisfy the user needs. The paper presents the theory underlying the proposed technique and describes how it has been exploited in the OpenDLib system. [ABSTRACT FROM AUTHOR]

This paper shows the interaction among the algebraic-analytical data fields theory, the complete descriptions that may be truncated for practical visual tasks and the different level of processing in the early visual pathway, in order to establish a measure of the representation of similarity between images. Our work is based on the approach of modeling the representational capabilities of systems of receptive fields found in early mammalian vision. It is well known that a representation scheme with a metric in the space of representation induce, in a natural way, a similarity measure between images. We propose in this paper to compare different representation schemes, based in previous theorems about completnes on data fields, according to the induced similarity measure. [ABSTRACT FROM AUTHOR]