Search

Recommender systems are typically evaluated using either offline methods, online methods, or through user studies. In this paper we take an episode mining approach to analysing recommender system data and we demonstrate how we can use SNIPER, a tool for interactive pattern mining, to analyse and understand the behaviour of recommender systems. We describe the required data format, and present a useful scenario of how a user can interact with the system to answer questions about the quality of recommendations.

User profiling in social media has gained a lot of attention due to its varied set of applications in advertising, marketing, recruiting, and law enforcement. Among the various techniques for user modeling, there is fairly limited work on how to merge multiple sources or modalities of user data - such as text, images, and relations - to arrive at more accurate user profiles. In this paper, we propose a deep learning approach that extracts and fuses information across different modalities. Our hybrid user profiling framework utilizes a shared representation between modalities to integrate three sources of data at the feature level, and combines the decision of separate networks that operate on each combination of data sources at the decision level. Our experimental results on more than 5K Facebook users demonstrate that our approach outperforms competing approaches for inferring age, gender and personality traits of social media users. We get highly accurate results with AUC values of more than 0.9 for the task of age prediction and 0.95 for the task of gender prediction.

In this paper, we introduce a novel multimodal fashion search paradigm where e-commerce data is searched with a multimodal query composed of both an image and text. In this setting, the query image shows a fashion product that the user likes and the query text allows to change certain product attributes to fit the product to the user’s desire. Multimodal search gives users the means to clearly express what they are looking for. This is in contrast to current e-commerce search mechanisms, which are cumbersome and often fail to grasp the customer’s needs. Multimodal search requires intermodal representations of visual and textual fashion attributes which can be mixed and matched to form the user’s desired product, and which have a mechanism to indicate when a visual and textual fashion attribute represent the same concept. With a neural network, we induce a common, multimodal space for visual and textual fashion attributes where their inner product measures their semantic similarity. We build a multimodal retrieval model which operates on the obtained intermodal representations and which ranks images based on their relevance to a multimodal query. We demonstrate that our model is able to retrieve images that both exhibit the necessary query image attributes and satisfy the query texts. Moreover, we show that our model substantially outperforms two state-of-the-art retrieval models adapted to multimodal fashion search. ispartof: pages:342-350 ispartof: Proceedings of the 11th ACM International Conference on Web Search and Data Mining vol:2018-Febuary pages:342-350 ispartof: The 11th ACM International Conference on Web Search and Data Mining location:Los Angeles date:5 Feb - 9 Feb 2018 status: published

In this paper we develop a neural network which learns inter-modal representations for fashion attributes to be utilized in a cross-modal search tool. Our neural network learns from organic e-commerce data, which is characterized by clean image material, but noisy and incomplete product descriptions. First, we experiment with techniques to segment e-commerce images and their product descriptions into respectively image and text fragments denoting fashion attributes. Here, we propose a rule-based image segmentation approach which exploits the cleanness of e-commerce images. Next, we design an objective function which encourages our model to induce a common embedding space where a semantically related image fragment and text fragment have a high inner product. This objective function incorporates similarity information of image fragments to obtain better intermodal representations. A key insight is that similar looking image fragments should be described with the same text fragments. We explicitly require this in our objective function, and as such recover information which was lost due to noise and incompleteness in the product descriptions. We evaluate the inferred intermodal representations in cross-modal search. We demonstrate that the neural network model trained with our objective function on image fragments acquired with our rule-based segmentation approach improves the results of image search with textual queries by 198% for recall@1 and by 181% for recall@5 compared to results obtained by a state-of-the-art image search system on the same benchmark dataset. ispartof: pages:1-10 ispartof: Proceedings of the KDD 2017 Workshop on Machine Learning Meets Fashion vol:2017 pages:1-10 ispartof: KDD 2017 Workshop on Machine Learning Meets Fashion location:Halifax, Canada date:14 Aug - 14 Aug 2017 status: published

Multimodal information fusion both at the signal and the semantics levels is a core part in most multimedia applications, including multimedia indexing, retrieval, summarization and others. Early or late fusion of modality-specific processing results has been addressed in multimedia prototypes since their very early days, through various methodologies including rule-based approaches, information-theoretic models and machine learning. Vision and Language are two of the predominant modalities that are being fused and which have attracted special attention in international challenges with a long history of results, such as TRECVid, ImageClef and others. During the last decade, vision-language semantic integration has attracted attention from traditionally non-interdisciplinary research communities, such as Computer Vision and Natural Language Processing. This is due to the fact that one modality can greatly assist the processing of another providing cues for disambiguation, complementary information and noise/error filtering. The latest boom of deep learning methods has opened up new directions in joint modelling of visual and co-occurring verbal information in multimedia discourse. The workshop on Vision and Language Integration Meets Multimedia Fusion has been held during the workshop weekend of the ACM Multimedia 2016 Conference and the European Conference on Computer Vision (ECCV 2016) on October 16, 2016 in Amsterdam, the Netherlands. The proceedings contain seven selected long papers, which have been orally presented at the workshop, and three abstracts of the invited keynote speeches. The papers and abstracts discuss data collection, representation learning, deep learning approaches, matrix and tensor factorization methods and graph based clustering with regard to the fusion of multimedia data. A variety of applications is presented including image captioning, summarization of news, video hyperlinking, sub-shot segmentation of user generated video, cross-modal classification, cross-modal question-answering, and the detection of misleading metadata of user generated video. The workshop is organized and supported by the EU COST action iV&L Net, the European Network on Integrating Vision and Language: Combining Computer Vision and Language Processing for Advanced Search, Retrieval, Annotation and Description of Visual Data (IC 1307--2014-2018).

We propose a new unified framework for monolingual (MoIR) and cross-lingual information retrieval (CLIR) which relies on the induction of dense real-valued word vectors known as word embeddings (WE) from comparable data. To this end, we make several important contributions: (1) We present a novel word representation learning model called Bilingual Word Embeddings Skip-Gram (BWESG) which is the first model able to learn bilingual word embeddings solely on the basis of document-aligned comparable data; (2) We demonstrate a simple yet effective approach to building document embeddings from single word embeddings by utilizing models from compositional distributional semantics. BWESG induces a shared cross-lingual embedding vector space in which both words, queries, and documents may be presented as dense real-valued vectors; (3) We build novel ad-hoc MoIR and CLIR models which rely on the induced word and document embeddings and the shared cross-lingual embedding space; (4) Experiments for English and Dutch MoIR, as well as for English-to-Dutch and Dutch-to-English CLIR using benchmarking CLEF 2001-2003 collections and queries demonstrate the utility of our WE-based MoIR and CLIR models. The best results on the CLEF collections are obtained by the combination of the WE-based approach and a unigram language model. We also report on significant improvements in ad-hoc IR tasks of our WE-based framework over the state-of-the-art framework for learning text representations from comparable data based on latent Dirichlet allocation (LDA). ispartof: pages:363-372 ispartof: Proceedings of the 38th Annual ACM SIGIR Conference on Research and Development in Information Retrieval - Full Papers (SIGIR 2015) pages:363-372 ispartof: The 38th international ACM SIGIR conference on research and development in information retrieval (SIGIR 2015) location:Santiago, Chile date:9 Aug - 13 Aug 2015 status: published

The resource limitations of mobile devices continue to impose constraints on the development of complex mobile applications. Performance and resource efficiency remains a challenge. We have examined resource utilization and performance trade-offs when extending an Augmented Reality (AR) application with context-awareness and cloud computing. The hypothesis is that the cost of these technologies is worth the benefits they result in. Our measurements show that filtered image datasets obtained through context-awareness result in lower latency and a reduced memory load when performing all AR computations on the mobile device. However, a cloud computing AR application does not benefit from in-depth context-awareness, as no part of the dataset is stored locally and the latency is approximately constant, relative to the Internet connectivity.

In recent years, there has been a rising interest in cloud computing, which is often used to offer Software as a Service (SaaS) over the Internet. SaaS applications can be offered to clients at a lower cost as they are usually multi-tenant: many end users make use of a single application instance, even when they are from different organisations. It is difficult to offer highly customizable SaaS applications that are still multi-tenant, which is why these SaaS applications are often offered in a one size fits all approach.In some application domains applications must be highly customizable, making it more difficult to migrate them to a cloud environment, and losing the benefits of multi-tenancy. In this paper we compare multiple approaches for the development and management of highly customizable multitenant SaaS applications, and present a methodology for developing and managing these applications. We compare two approaches, an application-based approach focusing on deploying multiple multi-tenant applications variants, and a feature-based approach where applications are composed out of multi-tenant services using a service oriented architecture. In addition, we also discuss a hybrid approach combining properties of both. We conclude that the feature-based approach results in the fewest application instances at runtime resulting in more multi-tenancy.

We study the problem of linking information between different idiomatic usages of the same language, for example, colloquial and formal language. We propose a novel probabilistic topic model called multi-idiomatic LDA (MiLDA). Its modeling principles follow the intuition that certain words are shared between two idioms of the same language, while other words are non-shared, that is, idiom-specific. We demonstrate the ability of our model to learn relations between cross-idiomatic topics in a dataset containing product descriptions and reviews. We intrinsically evaluate our model by the perplexity measure. Following that, as an extrinsic evaluation, we present the utility of the new MiLDA topic model in a recently proposed IR task of linking Pinterest pins (given in colloquial English on the users' side) to online webshops (given in formal English on the retailers' side). We show that our multi-idiomatic model outperforms the standard monolingual LDA model and the pure bilingual LDA model both in terms of perplexity and MAP scores in the IR task.

Multilingual topic models are a fairly novel group of unsupervised, language-independent and generative machine learning models. This tutorial covers all key aspects of their probabilistic framework and demonstrates how to easily integrate these models into frameworks for cross-lingual and multilingual Web mining and search. ispartof: pages:681- ispartof: Proceedings of the 7th ACM international conference on Web search and data mining pages:681- ispartof: 7th ACM international conference on Web search and data mining location:NY, New York date:24 Feb - 28 Feb 2014 status: published

This paper gives a short overview of the state-of-the-art and goals of argumentation mining, and it provides ideas for further research. Its content is based on two invited lectures on argumentation mining respectively at the FIRE 2013 conference at the India International Center in New Delhi, India and a lecture given as SICSA distinguished visitor at the University of Dundee, UK in the summer of 2014. ispartof: pages:2:1-2:1 ispartof: Post-proceedings of the forum for information retrieval evaluation (FIRE 2013) vol:04-06-December-2013 pages:2:1-2:1 ispartof: Forum for information retrieval evaluation (FIRE 2013) location:New Delhi, India date:4 Dec - 6 Dec 2013 status: published

The information that users of social network sites post often points towards their interests and hobbies. It can be used to recommend relevant products to users. In this paper we implement and evaluate several information retrieval models for linking the texts of pins of Pinterest to webpages of Amazon, and ranking the pages (which we call webshops) according to the personal interest of the pinner. The results show that models that combine latent concepts composed of related terms with single words yield the best performance.

We propose an unsupervised framework for recognizing animals in videos using subtitles. In this framework, the alignment between animals and their names is performed using an Expectation Maximization algorithm which is adapted to two very dierent circumstances- 1) when the bounding boxes are available and 2) when the frame as a whole is used instead of bounding boxes. With the goal of maximizing precision, recall and F-measure, the experiments compare a multitude of natural language processing approaches and visual features when associating animal names in the subtitles with visual patterns. The proposed unsupervised methods obtain 83.1% F1 using bounding boxes and 65.7% F1 without bounding boxes in a fully automated pipeline. ispartof: pages:9-14 ispartof: Proceedings of the 2nd ACM international workshop on multimedia analysis for ecological data pages:9-14 ispartof: 2nd ACM international workshop on multimedia analysis for ecological data location:Barcelona, Spain date:21 Oct - 21 Oct 2013 status: published

The continued development and maturation of advanced HTML features such as Cascading Style Sheets (CSS), Javascript, and AJAX, as well as their widespread adoption by browsers, has enabled web pages to flourish with sophistication and interactivity. Unfortunately, this presents challenges to the web search community, as a web page's representation in the browser (i.e., what users see) can diverge dramatically from its raw HTML content (i.e., what search engines index and retrieve). For example, interactive pages may contain content in regions that are not visible before a user action, such as focusing a tab, but which are nonetheless still contained within the raw HTML. We study this divergence by comparing raw HTML to its fully rendered form across a number of metrics spanning presentation, geometry, and content, using a large, representative sample of popular web pages. We found that a large divergence currently exists, and we show via a historical analysis that this divergence has grown more pronounced over the last decade. The general finding of our study is that continuing to index the web via simple HTML parsing will diminish the effectiveness of retrieval on the modern web, and that the IR community should work toward more sophisticated web page processing in indexing technology. ispartof: pages:1990-1994 ispartof: Proceedings of the 21st ACM international conference on information and knowledge management (CIKM 2012) pages:1990-1994 ispartof: The 21st ACM international conference on information and knowledge management (CIKM 2012) location:Maui, Hawaii date:29 Oct - 2 Nov 2012 status: published

Does the reason a user visits a Wikipedia page infuence that user's subsequent browsing behavior on Wikipedia? We address this question using aggregate Wikipedia page access data. We conduct: (1) a comparison of browsing behaviors between serendipitous and directed information seekers; and (2) how topic/category influences users' migration from page to page. Our findings indicate that surfer behavior and topic are potentially influential factors, which weakens the random surfer model underlying link-based algorithms, and could affect how web designers should design sites to best meet the diversity of information seeking behaviors. ispartof: pages:155-163 ispartof: Proceedings of the fourth information interaction in context symposium (IIiX 2012) pages:155-163 ispartof: Fourth information interaction in context symposium (IIiX 2012) location:Nijmegen date:21 Aug - 24 Aug 2012 status: published

When undergoing medical treatment in combination with extended stays in hospitals, children have been frequently found to develop an interest in their condition and the course of treatment. PuppyIR A natural means of searching for related information would be to use a web search engine. The medical domain, however, imposes several key challenges on young and inexperienced searchers, such as difficult terminology, potentially frightening topics or non-objective information offered by lobbyists or pharmaceutical companies. To address these problems, we present the design and usability study of EmSe, a search service for children in a hospital environment.

This paper shows how coupling from the past (CFTP) can be used to avoid time and memory bottlenecks in direct local pattern sampling procedures. Such procedures draw controlled amounts of suitably biased samples directly from the pattern space of a given dataset in polynomial time. Previous direct pattern sampling methods can produce patterns in rapid succession after some initial preprocessing phase. This preprocessing phase, however, turns out to be prohibitive in terms of time and memory for many datasets. We show how CFTP can be used to avoid any super-linear preprocessing and memory requirements. This allows to simulate more complex distributions, which previously were intractable. We show for a large number of public real-world datasets that these new algorithms are fast to execute and their pattern collections outperform previous approaches both in unsupervised as well as supervised contexts.

Wikipedia’s rich category structure has helped make it one of the largest semantic taxonomies in existence, a property that has been central to much recent research. However, Wikipedia’s category representation is simplistic: an article contains a single list of categories, with no data about their relative importance. We investigate the ordering of category lists to determine how a category’s position in the list correlates with its relevance to the article and overall significance. We identify a number of interesting connections between a category’s position and its persistence within the article, age, popularity, size, and descriptiveness. ispartof: pages:2309-2312 ispartof: Proceedings of the 20th ACM international conference on information and knowledge management (CIKM 2011) pages:2309-2312 ispartof: 20th ACM international conference on information and knowledge management location:Glasgow, UK date:24 Oct - 28 Oct 2011 status: published

We present a simple and effective approach to complement search results for children’s web queries with child-oriented multimedia results, such as coloring pages and music sheets. Our approach determines appropriate media types for a query by searching Google’s database of frequent queries for co-occurrences of a query’s terms (e.g., “dinosaurs”) with preselected multimedia terms (e.g., “coloring pages”). We show the effectiveness of this approach through an online user evaluation. ispartof: pages:731-732 ispartof: Proceedings of the 33rd annual ACM SIGIR conference on research and development in information retrieval pages:731-732 ispartof: 33rd annual ACM SIGIR conference on research and development in information retrieval location:Geneva date:19 Jul - 23 Jul 2010 status: published

This tutorial focuses on the task of automated information linking in text and multimedia sources. In any task where information is fused from different sources, this linking is a necessary step. To solve the problem we borrow methods from computational linguistics, computer vision and data mining. Although the main focus is on finding equivalence relations in the sources, the tutorial opens views on the recognition of other relation types.

Argumentation is the process by which arguments are constructed and handled. Argumentation constitutes a major component of human intelligence. The ability to engage in argumentation is essential for humans to understand new problems, to perform scientific reasoning, to express, to clarify and to defend their opinions in their daily lives. Argumentation mining aims to detect the arguments presented in a text document, the relations between them and the internal structure of each individual argument. In this paper we analyse the main research questions when dealing with argumentation mining and the different methods we have studied and developed in order to successfully confront the challenges of argumentation mining in legal texts. ispartof: pages:98-109 ispartof: Proceedings of the twelfth international conference on artificial intelligence and law (ICAIL 2009) pages:98-109 ispartof: Twelfth international conference on artificial intelligence and law (ICAIL 2009) location:Barcelona, Spain date:8 Jun - 12 Jun 2009 status: published

This paper provides the results of experiments on the detection of arguments in texts among which are legal texts. The detection is seen as a classification problem. A classifier is trained on a set of annotated arguments. Different feature sets are evaluated involving lexical, syntactic, semantic and discourse properties of the texts. The experiments are a first step in the context of automatically classifying arguments in legal texts according to their rhetorical type and their visualization for convenient access and search.

We present rpref ; our generalization of the bpref evaluation metric for assessing the quality of search engine results, given graded rather than binary user relevance judgments.

Legislative sources are currently accessible via portal web sites. Users demand precise and exhaustive answers to their information queries. When legislation is drafted, it contains text-rich information that is increasingly marked with XML tags. The statute structure as signaled by XML markup can be exploited to more precisely answer free information queries. In this paper we report on several XML retrieval models that we explicitly designed for the retrieval of legislation. We show that the models provide more advanced access to the content of statutes.

Topic segmentation is an important initial step in many text-based tasks. A hierarchical representation of a texts topics is useful in retrieval and allows judging relevancy at different levels of detail. This short paper describes research on generic algorithms for topic detection and segmentation that are applicable on texts of heterogeneous types and domains.