Search

We introduce using images for word sense disambiguation, either alone, or in conjunction with traditional text based methods. The approach is based on a recently developed method for automatically annotating images by using a statistical model for the joint probability for image regions and words. The model itself is learned from a data base of images with associated text. To use the model for word sense disambiguation, we constrain the predicted words to be possible senses for the word under consideration. When word prediction is constrained to a narrow set of choices (such as possible senses), it can be quite reliable. We report on experiments using the resulting sense probabilities as is, as well as augmenting a state of the art text based word sense disambiguation algorithm. In order to evaluate our approach, we developed a new corpus, ImCor, which consists of a substantive portion of the Corel image data set associated with disambiguated text drawn from the SemCor corpus. Our experiments using this corpus suggest that visual information can be very useful in disambiguating word senses. It also illustrates that associated non-textual information such as image data can help ground language meaning.

İç mekan tanıma, insan yapımı yapıların yüksek sınıf içi varyasyonlar ve sınıfarası benzerlikler göstermesi sebebiyle klasik mekan tanıma alanının zorlu birproblemidir. Resmin bütüncül temsillerini çıkarmak gibi en ileri mekan tanımateknikleri iç mekanlarda düşük performans göstermektedirler. Nesnelerin belirlenmesive ardından onların mekanlarla ilişkilendirilmesi gibi ara kademeler kullanandiğer yöntemlerin de oldukça karmaşık bir ortamda nesnelerin başarıyla lokalizeedilmesi ve tanınması handikapları vardır.Kodkitabı olarak da bilinen görsel kelimeler kümesi tekniğinden faydalanaraken yakın komşu yöntemine dayalı bir metrik fonksiyonu ile bu zorluklarınüstesinden gelebilen bir sınıflandırma yöntemi öneriyoruz. Kodkitabı oluşumuöznitelik uzayının mozaikleştirilmesi olarak ele alınırsa, verilen bir resim için,öznitelik vektörlerinin Voronoi hücrelerinin ortalarına olan öğrenilmiş ağırlıklıuzaklıklarının resmin kategorisi için güçlü bir gösterge olduğunu gözlemledik.Yöntemimiz tek bir tanımlayıcı ile bir iç mekan testinde en gelişmiş yaklaşımlarıgeçmekte ve genel bir mekan veri kümesinde rekabetçi sonuçlar üretmektedir.Bu çalışmada her ne kadar temel sorunumuz iç mekan kategorizasyonu olsa da,önerilen metrik fonksiyonunu otomatik etiketleme problemine de bir temel uygulamaoluşturmak için kullanıyoruz. Gittikçe artan sayısal medya ile, otomatik olarak resimlereanlamsal etiketler çıkarma problemine son on yılda araştırmacılar büyük ilgigöstermişlerdir. Bu tarz içerikleri manüel olarak etiketlemek gibi geleneksel yaklaşımlarçok bıktırıcı ve zaman harcayıcı olarak değerlendirilmektedir. Bu neden resimlerianlamsal olarak başarıyla açıklayan anahtar kelimelerle otomatik olarak etiketleme,çözülmeyi bekleyen önemli bir sorundur. Indoor scene recognition is a challenging problem in the classical scene recognitiondomain due to the severe intra-class variations and inter-class similarities ofman-made indoor structures. State-of-the-art scene recognition techniques suchas capturing holistic representations of an image demonstrate low performance onindoor scenes. Other methods that introduce intermediate steps such as identifyingobjects and associating them with scenes have the handicap of successfullylocalizing and recognizing the objects in a highly cluttered and sophisticatedenvironment.We propose a classication method that can handle such diculties of theproblem domain by employing a metric function based on the nearest-neighborclassication procedure using the bag-of-visual words scheme, the so-called codebooks.Considering the codebook construction as a Voronoi tessellation of thefeature space, we have observed that, given an image, a learned weighted distanceof the extracted feature vectors to the center of the Voronoi cells gives a strongindication of the image's category. Our method outperforms state-of-the-art approacheson an indoor scene recognition benchmark and achieves competitiveresults on a general scene dataset, using a single type of descriptor.In this study although our primary focus is indoor scene categorization, we alsoemploy the proposed metric function to create a baseline implementation for theauto-annotation problem. With the growing amount of digital media, the problemof auto-annotating images with semantic labels has received signicant interestfrom researches in the last decade. Traditional approaches where such content ismanually tagged has been found to be too tedious and a time-consuming process.Hence, succesfully labeling images with keywords describing the semantics is acrucial task yet to be accomplished. 57

This dissertation delves into the use of textual metadata for image understanding. We seek to exploit this additional textual information as weak supervision to improve the learning of recognition models. There is a recent and growing interest for methods that exploit such data because they can potentially alleviate the need for manual annotation, which is a costly and time-consuming process. We focus on two types of visual data with associated textual information. First, we exploit news images that come with descriptive captions to address several face related tasks, including face verification, which is the task of deciding whether two images depict the same individual, and face naming, the problem of associating faces in a data set to their correct names. Second, we consider data consisting of images with user tags. We explore models for automatically predicting tags for new images, i.e. image auto-annotation, which can also used for keyword-based image search. We also study a multimodal semi-supervised learning scenario for image categorisation. In this setting, the tags are assumed to be present in both labelled and unlabelled training data, while they are absent from the test data. Our work builds on the observation that most of these tasks can be solved if perfectly adequate similarity measures are used. We therefore introduce novel approaches that involve metric learning, nearest neighbour models and graph-based methods to learn, from the visual and textual data, task-specific similarities. For faces, our similarities focus on the identities of the individuals while, for images, they address more general semantic visual concepts. Experimentally, our approaches achieve stateof- the-art results on several standard and challenging data sets. On both types of data, we clearly show that learning using additional textual information improves the performance of visual recognition systems., La présente thèse s'intéresse à l'utilisation de méta-données textuelles pour l'analyse d'image. Nous cherchons à utiliser ces informations additionelles comme supervision faible pour l'apprentissage de modèles de reconnaissance visuelle. Nous avons observé un récent et grandissant intérêt pour les méthodes capables d'exploiter ce type de données car celles-ci peuvent potentiellement supprimer le besoin d'annotations manuelles, qui sont coûteuses en temps et en ressources. Nous concentrons nos efforts sur deux types de données visuelles associées à des informations textuelles. Tout d'abord, nous utilisons des images de dépêches qui sont accompagnées de légendes descriptives pour s'attaquer à plusieurs problèmes liés à la reconnaissance de visages. Parmi ces problèmes, la vérification de visages est la tâche consistant à décider si deux images représentent la même personne, et le nommage de visages cherche à associer les visages d'une base de données à leur noms corrects. Ensuite, nous explorons des modèles pour prédire automatiquement les labels pertinents pour des images, un problème connu sous le nom d'annotation automatique d'image. Ces modèles peuvent aussi être utilisés pour effectuer des recherches d'images à partir de mots-clés. Nous étudions enfin un scénario d'apprentissage multimodal semi-supervisé pour la catégorisation d'image. Dans ce cadre de travail, les labels sont supposés présents pour les données d'apprentissage, qu'elles soient manuellement annotées ou non, et absentes des données de test. Nos travaux se basent sur l'observation que la plupart de ces problèmes peuvent être résolus si des mesures de similarité parfaitement adaptées sont utilisées. Nous proposons donc de nouvelles approches qui combinent apprentissage de distance, modèles par plus proches voisins et méthodes par graphes pour apprendre, à partir de données visuelles et textuelles, des similarités visuelles spécifiques à chaque problème. Dans le cas des visages, nos similarités se concentrent sur l'identité des individus tandis que, pour les images, elles concernent des concepts sémantiques plus généraux. Expérimentalement, nos approches obtiennent des performances à l'état de l'art sur plusieurs bases de données complexes. Pour les deux types de données considérés, nous montrons clairement que l'apprentissage bénéficie de l'information textuelle supplémentaire résultant en l'amélioration de la performance des systèmes de reconnaissance visuelle.