Search

The paper describes the organization, goals, and results of the sEXism Identification in Social neTworks (EXIST)2022 challenge, a shared task proposed for the second year at IberLEF. EXIST 2022 consists of two challenges: sexism identification and sexism categorization of tweets and gabs, both in Spanish and English. We have received a total of 45 runs for the sexism identification task and 29 runs for the sexism categorization task, submitted by 19 different teams. In this paper, we present the dataset, the evaluation methodology, an overview of the proposed systems, and the results obtained. The final dataset consists of more than 12,000 annotated texts from two social networks (Twitter and Gab) labelled following two different procedures: external contributors and trained experts. El artículo describe la organización, objetivos y resultados de EXIST 2022 (sEXism Identification in Social neTworks), una competición que se celebra por segundo año consecutivo en el foro IberLEF. EXIST 2022 consta de dos tareas: detección de sexismo y categorización de sexismo de tweets y gabs, tanto en español como en inglés. Hemos recibido un total de 45 ejecuciones para la tarea de detección de sexismo y 29 ejecuciones para la tarea de categorización de sexismo, enviadas por 19 equipos diferentes. En el presente artículo, presentamos el conjunto de datos, la metodología de evaluación, una descripción general de los sistemas propuestos y los resultados obtenidos. El conjunto final de datos consta de más de 12.000 textos anotados de dos redes sociales (Twitter y Gab) etiquetados siguiendo dos procedimientos diferentes: colaboradores externos y expertos en el dominio. This work was partially supported by the Spanish Ministry of Science and Innovation under the project “FairTransNLP: Midiendo y Cuantificando el sesgo y la justicia en sistemas de PLN” (PID2021-124361OB-C31 and PID2021-124361OB-C32). This work was also partially funded by the Spanish Ministry of Economy and Competitiveness, as part of the research cooperation project “Space for Observation of AI in Spanish” (UNED and RED.ES, M.P., ref. C039/21-OT). The work of Paolo Rosso was in the framework of the research project PROMETEO/2019/121 (DeepPattern) by the Generalitat Valenciana.

The paper describes the organization, goals, and results of the sEXism Identification in Social neTworks (EXIST) challenge, a shared task proposed for the first time at IberLEF 2021. EXIST 2021 proposes two challenges: sexism identification and sexism categorization of tweets and gabs, both in Spanish and English. We have received a total of 70 runs for the sexism identification task and 61 for the sexism categorization challenge, submitted by 31 different teams from 11 countries. We present the dataset, the evaluation methodology, an overview of the proposed systems, and the results obtained. The final dataset consists of more than 11,000 annotated texts from two social networks (Twitter and Gab) and its development has been supervised and monitored by experts in gender issues. El presente artículo describe la organización, objetivos y resultados de la competición sEXism Identification in Social neTworks (EXIST), una tarea propuesta por primera vez en IberLEF 2021. EXIST 2021 propone dos tareas: la identificación y la categorización de sexismo en inglés y español. Se han recibido un total de 70 runs para la tarea de identificación de sexismo y 61 para la categorización de sexismo, enviadas por 31 equipos de 11 países. En este trabajo, se presentan el dataset, la metodología de evaluación, un análisis de los sistemas propuestos por los participantes y los resultados obtenidos. El dataset final está compuesto por más de 11,000 textos anotados procedentes de dos redes sociales (Twitter y Gab) y su elaboración ha sido supervisada por expertas en temas de género. This work was supported by the Spanish Ministry of Science and Innovation under the MISMIS research project on Misinformation and Miscommunication in Social Media (PGC2018-096212-B-C32).

This paper summarizes the main findings of the ADoBo 2021 shared task, proposed in the context of IberLef 2021. In this task, we invited participants to detect lexical borrowings (coming mostly from English) in Spanish newswire texts. This task was framed as a sequence classification problem using BIO encoding. We provided participants with an annotated corpus of lexical borrowings which we split into training, development and test splits. We received submissions from 4 teams with 9 different system runs overall. The results, which range from F1 scores of 37 to 85, suggest that this is a challenging task, especially when out-of-domain or OOV words are considered, and that traditional methods informed with lexicographic information would benefit from taking advantage of current NLP trends. En este artículo presentamos los resultados de ADoBo 2021, la tarea compartida de IberLEF 2021 sobre detección de préstamos léxicos en la prensa española. En esta tarea abordamos la detección de préstamos como un problema de etiquetado de secuencias. A los participantes de la tarea se les proporcionó un corpus de prensa española anotado con préstamos léxicos no asimilados (mayoritariamente anglicismos) siguiendo el esquema BIO. Recibimos nueve sistemas distintos provenientes de cuatro equipos diferentes. Los resultados obtenidos oscilan entre los 37 y los 85 puntos de valor F1, lo que indica que la detección de préstamos léxicos es un problema no resuelto (sobre todo cuando se abordan préstamos no vistos anteriormente) y que el trabajo lexicográfico tradicional podría beneficiarse de incorporar las técnicas actuales del PLN.

En los últimos años, han surgido diversas herramientas de análisis métrico automático de poesía española. Estos sistemas se basan en complejos métodos de silabación y asignación de acentos, los cuales se apoyan en librerías de etiquetado gramatical, cuyo coste computacional es elevado. Este coste incrementa con el cálculo de ambigüedades métricas. El presente artículo parte de la hipótesis de que es posible llevar a cabo un análisis métrico informativo y preciso sin utilizar estos métodos. Se propone un algoritmo que realiza escansiones (número de sílabas, patrón métrico y tipo de verso) sin silabación. El algoritmo resuelve ambigüedades métricas y tiene en cuenta la compensación hemistiquial. Nuestros resultados indican que obtiene una mejora relativa sobre el estado del arte de un 2% en la clasificación de patrones métricos en poesía de medida fija y un 25% en poesía polimétrica. También se ejecuta 21 y 25 veces más rápido, respectivamente. Por último, se ofrece una aplicación de escritorio como herramienta para los investigadores de la poesía española. In recent years, several systems of automated metric analysis of Spanish poetry have emerged. These systems rely on complex methods of syllabification and stress assignment, which use PoS-tagging libraries, whose computational cost is high. This cost increases with the calculation of metric ambiguities. However, it is possible to carry out an informative and accurate metric analysis without using these costly methods. We propose an algorithm that performs accurate scansion (number of syllables, stress pattern and type of verse) without syllabification. It addresses metric ambiguities and takes into account the hemistichs compensation. Our algorithm outperforms the current state of the art by 2% in fixed-metre poetry, and 25% in mixed-metre poetry. It also runs 21 and 25 times faster, respectively. Finally, a desktop application is offered as a tool for researchers of Spanish poetry. Esta investigación se ha desarrollado gracias al proyecto MISMIS-BIAS (PGC2018-096212-B-C32), financiado por el Gobierno de España, Ministerio de Ciencia, Innovación y Universidades.

Given the task of finding influencers of a given domain (i.e. banking) in a social network, in this paper we investigate (i) the importance of characterizing followers for the automatic detection of influencers; (ii) the most effective way to combine signals obtained from followers and from the main profiles for the automatic detection of influencers. In this work, we have modeled the discourse used in two domains, banking and automotive, as well as the language used by the influencers in such domains and by their followers, and used these Language Models to estimate the probability of being influencer. Our most remarkable finding is that influencers not only depend on their expertise on the domain but also on that of their followers, so that the more knowledge and number of experts among their followers, the more probability of being influencer a profile has. Dada la tarea de encontrar influencers en un dominio dado (i.e. banking) en una red social, en este artículo investigamos (i) la importancia de caracterizar a los seguidores para la detección automática de influencers; (ii) la manera más efectiva de combinar señales obtenidas de los seguidores y de los perfiles principales para la detección automática de influencers. En este trabajo, hemos modelado el discurso usado por los usuarios en dos dominios, automotive y banking, así como el lenguaje utilizado por los influencers en dichos dominios y por sus seguidores, y utilizamos estos Modelos de Lenguaje para estimar la probabilidad de ser un influencer. Nuestro mayor descubrimiento es que los influencers no sólo dependen de su conocimiento sobre el dominio sino del de sus seguidores; por lo tanto, cuanto mayor conocimiento y número de expertos haya entre sus seguidores, mayor será la probabilidad que el perfil sea de un influencer. This research was supported by the Spanish Ministry of Science and Innovation (Vemodalen Project, TIN2015-71785-R).

En este artículo introducimos la tarea de caracterización de una entidad a través de las valoraciones que sobre ella se vierten en la red, entendida como la agrupación de valoraciones (positivas y negativas) equivalentes o muy similares. Se trata de un problema práctico que se encuentra en el ámbito de la gestión de la reputación online, para el que todavía no existen tareas ni benchmarks de referencia dentro del Procesamiento del Lenguaje Natural. A través de un estudio de caso se muestran las dificultades intrínsecas de la tarea y se proporciona un baseline basado en el algoritmo de clustering Lingo. We introduce the opinion-based entity profiling task, understood as the automatic grouping of equivalent or very similar online opinions (both positive and negative) about an entity. This is a highly practical problem that arises in the context of online reputation management, but it has not been directly addressed by any Natural Language Processing evaluation campaign or specific benchmarking corpus. We use a case study to illustrate the challenges of the task, and we provide an initial baseline based on the Lingo clustering algorithm. Este trabajo se enmarca dentro del proyecto WebOpinion, financiado por el Ministerio de Industria y Alma Technologies.

The goal of the project is to analyze, experiment, and develop intelligent, interactive and multilingual Text Mining technologies, as a key element of the next generation of search engines, systems with the capacity to find "the need behind the query". This new generation will provide specialized services and interfaces according to the search domain and type of information needed. Moreover, it will integrate textual search (websites) and multimedia search (images, audio, video), it will be able to find and organize information, rather than generating ranked lists of websites.

Este artículo muestra la aplicación del marco de evaluación QARLA sobre los resúmenes evaluados en el foro DUC-2004, para las tareas 2 y 5. El marco QARLA permite evaluar de forma automática los sistemas según diferentes aspectos (métricas de similitud) en relación a un conjunto de resúmenes modelo, identificando así los aspectos más deficitarios de las estrategias de resumen existentes. Por otro lado, el marco QARLA permite combinar y meta-evaluar diferentes métricas de similitud, otorgando más peso a los aspectos que caracterizan a los modelos en relación a los resúmenes automáticos. This article shows an application of the QARLA evaluation framework on DUC-2004 (tasks 2 and 5). The QARLA framework allows to evaluate summaries with regard to different features. Second, it allows to combine and meta-evaluate different similarity metrics, giving more weigh to metrics which characterize models (manual summaries) regarding automatic summaries. Este trabajo ha sido financiado por el Ministerio de Ciencia y Tecnología a través del proyecto HERMES (TIC2000-0335-C03-1).

En este artículo, describimos un modelo interactivo de resumen orientado a síntesis de información (PRISMA). El usuario interactúa con el sistema por medio de listas de conceptos clave extraídos automáticamente. El modelo hace uso de información sintáctica para identificar conceptos clave, organizar y mostrar la información. Asimismo, proponemos y aplicamos una metodología de evaluación de modelos interactivos, que no requiere interacción entre el usuario y el sistema final. In this paper, we describe an information synthesis interactive model (PRISMA). The user interacts with the system by means of automatically extracted key concepts lists. The model uses syntactic knowledge to identify key concepts, to organize and display the information pieces. Also, we propose, and therefore we put into practice, a corpora based methodology of interactive models evaluation. The implementation of this methodology doesn't need interaction between user and system. Este trabajo ha sido financiado por el Ministerio de Ciencia y Tecnología a través del proyecto HERMES (TIC2000-0335-C03-1).

Describimos un algoritmo que combina información léxica (extraída de WordNet 1.6) con información en Internet (directorios de Altavista) para caracterizar automáticamente los sentidos de una palabra con etiquetas de dominio y, al mismo tiempo, detectar y describir nuevos sentidos relevantes en Internet. Esta información puede utilizarse, entre otras cosas, para enriquecer bases de datos léxicas, para facilitar la extracción de corpora anotados semánticamente y derivados de Internet (como material de entrenamiento para sistemas de anotación semántica), o para agrupar sentidos (de dominio coincidente) cuando las distinciones semánticas son demasiado finas para las aplicaciones en que se usan. Este trabajo ha sido financiado parcialmente por la Comisión Interministerial de Ciencia y Tecnología, proyecto Hermes (TIC2000-0335-C03-01).

Tesis doctoral inédita leida en la Universidad Autónoma de Madrid. Facultad de Ciencias, Departamento de Ingenieria Informática. Fecha de lectura: 28-9-1995

With the rise of social media channels such as Twitter --the most popular microblogging service-- the control of what is said about entities --companies, people or products-- online has been shifted from them to users and consumers. This has generated the necessity of monitoring the reputation of those entities online. In this context, it is only natural to witness a significant growth of demand for text mining software for Online Reputation Monitoring: automatic tools that help processing, understanding and aggregating large streams of facts and opinions about a company or individual.Despite the variety of Online Reputation Monitoring tools on the market, there is no standard evaluation framework yet --a widely accepted set of task definitions, evaluation measures and reusable test collections to tackle this problem. In fact, there is even no consensus on what the tasks carried out during the Online Reputation Monitoring process are, on which a system should minimize the effort of the user. In the context of a collective effort to identify and formalize the main challenges in the Online Reputation Monitoring process in Twitter, we have participated in the definition of tasks and subsequent creation of suitable test collections (WePS-3, RepLab 2012 and RepLab 2013 evaluation campaigns) and we have studied in depth two of the identified challenges:'filtering' (Is a tweet related to a given entity of interest?) --modeled as a binary classification task--and 'topic detection' (What is being said about an entity in a given tweet stream?), that consists of clustering tweets according to topics. Compared to previous studies on Twitter, our problem lies in its 'long tail': except for a few exceptions, the volume of information related to a specific entity (organization or company) at a given time is orders of magnitude smaller than Twitter trending topics, making the problem much more challenging than identifyingTwitter trends. We rely on three building blocks to propose different approaches to tackle these two tasks : the use of 'filter keywords', 'external resources' (such as Wikipedia, representative pages of the entity of interest, etc.) and the use of 'entity-specific training data' when available. We have found that the notion of 'filter keywords' --expressions that, if presentin a tweet, indicate a high probability that it is either related or unrelated to the entity of interest-- can be effectively used to tackle the filtering task. Here, (i) specificity of a term to the tweet stream of the entity is a useful feature to identify keywords, and (ii) the association between a term and the entity's Wikipedia page is useful to differentiate positive vs. negative filter keywords, especially when it is averaged by considering its most co-occurrent terms. In addition, exploring the nature of filter keywords also led us to the conclusion that there is a gap between the vocabulary that characterizes a company in Twitter and the vocabulary associated to the company in its homepage, in Wikipedia, and even in the Web at large. We have also found that, when entity-specific training data is available --as in the known-entity scenario-- it is more cost effective to use a simple Bag-of-Words classifier. When enough training data is available (around 700 tweets per entity), Bag-of-Words classifiers can be effectively used for the filtering task. Moreover, they can be used effectively in an active learning scenario, where the system updates its classification model with the stream of annotations and interactions with the system made by the reputation expert along the monitoring process. In this context, we found that by selecting the tweets to be labeled as those on which the classifier is less confident (margin sampling), the cost of creating a bulk training set can be reduced by 90% after inspecting 10% of test data. Unlike many other applications of active learning on Natural Language Processing tasks, margin sampling works better than random sampling. As for the topic detection problem, we considered two main strategies: the first is inspired on the notion of filter keywords and works by clustering terms as an intermediate step towards document clustering. The second --and most successful-- learns a pairwise tweet similarity function from previously annotated data, using all kinds of content-based and Twitter-based features; and then applies a clustering algorithm on the previously learned similarity function. Our experiments indicate that (i) Twitter signals can be used to improve the topic detection process with respect to using content signals only; (ii) learning a similarity function is a flexible and efficient way of introducing supervision in the topic detection clustering process. The performance of our best system is substantially better than state-of-the-art approaches and gets close to the inter-annotator agreement rate of topic detection annotations in the RepLab 2013 dataset --to our knowledge, the largest dataset available for OnlineReputation Monitoring. A detailedqualitative inspection of the data further reveals two types of topics detected by reputation experts: reputation alerts / issues (which usually spike in time) and organizational topics (which are usually stable across time). Along with our contribution to building a standard evaluation framework to study the Online Reputation Monitoring problem from a scientific perspective, we believe that the outcome of our research has practical implications and may help the development of semi-automatic tools to assist reputation experts in their daily work.