Search

Cross-environment activity recognition in smart homes is a very challenging problem, specially for data-driven approaches. Currently, systems developed to work for a certain environment degrade substantially when applied to a new environment, where not only sensors, but also the monitored activities may be different. Some systems require manual labeling and mapping of the new sensor names and activities using an ontology. Ideally, given a new smart home, we would like to be able to deploy the system, which has been trained on other sources, with minimal manual effort and with acceptable performance. In this paper, we propose the use of neural word embeddings to represent sensor activations and activities, which comes with several advantages: (i) the representation of the semantic information of sensor and activity names, and (ii) automatically mapping sensors and activities of different environments into the same semantic space. Based on this novel representation approach, we propose two data-driven activity recognition systems: the first one is a completely unsupervised system based on embedding similarities, while the second one adds a supervised learning regressor on top of them. We compare our approaches with some baselines using four public datasets, showing that data-driven cross-environment activity recognition obtains good results even when sensors and activity labels significantly differ. Our results show promise for reducing manual effort, and are complementary to other efforts using ontologies.

[EN] The egocentric action recognition EAR field has recently increased its popularity due to the affordable and lightweight wearable cameras available nowadays such as GoPro and similars. Therefore, the amount of egocentric data generated has increased, triggering the interest in the understanding of egocentric videos. More specifically, the recognition of actions in egocentric videos has gained popularity due to the challenge that it poses: the wild movement of the camera and the lack of context make it hard to recognise actions with a performance similar to that of third-person vision solutions. This has ignited the research interest on the field and, nowadays, many public datasets and competitions can be found in both the machine learning and the computer vision communities. In this survey, we aim to analyse the literature on egocentric vision methods and algorithms. For that, we propose a taxonomy to divide the literature into various categories with subcategories, contributing a more fine-grained classification of the available methods. We also provide a review of the zero-shot approaches used by the EAR community, a methodology that could help to transfer EAR algorithms to real-world applications. Finally, we summarise the datasets used by researchers in the literature. We gratefully acknowledge the support of the Basque Govern-ment's Department of Education for the predoctoral funding of the first author. This work has been supported by the Spanish Government under the FuturAAL-Context project (RTI2018-101045-B-C21) and by the Basque Government under the Deustek project (IT-1078-16-D).

Being the elder population in constant growth, governments have to cope with higher expenses for elder care from year to year. Helping the elderly to extend their independent lifestyle is of pivotal importance to minimise those costs. That is the goal of the Ambient Assisted Living research field. Through the use of Information and Communication Technologies, it is possible to provide solutions to help the elderly live independently for as long as possible or to predict mental health issues that could seriously harm their independence. The key enablers for these solutions are the egocentric cameras and the egocentric action recognition techniques for the analysis of egocentric videos. This chapter proposes various of those techniques focused on the exploitation of intrinsic egocentric cues.

Integrating outside knowledge for reasoning in visio-linguistic tasks such as visual question answering (VQA) is an open problem. Given that pretrained language models have been shown to include world knowledge, we propose to use a unimodal (text-only) train and inference procedure based on automatic off-the-shelf captioning of images and pretrained language models. Our results on a visual question answering task which requires external knowledge (OK-VQA) show that our text-only model outperforms pretrained multimodal (image-text) models of comparable number of parameters. In contrast, our model is less effective in a standard VQA task (VQA 2.0) confirming that our text-only method is specially effective for tasks requiring external knowledge. In addition, we show that increasing the language model's size improves notably its performance, yielding results comparable to the state-of-the-art with our largest model, significantly outperforming current multimodal systems, even though augmented with external knowledge. Our qualitative analysis on OK-VQA reveals that automatic captions often fail to capture relevant information in the images, which seems to be balanced by the better inference ability of the text-only language models. Our work opens up possibilities to further improve inference in visio-linguistic tasks, Comment: Under review. 25 pages with 4 figures

The interaction of conversational systems with users poses an exciting opportunity for improving them after deployment, but little evidence has been provided of its feasibility. In most applications, users are not able to provide the correct answer to the system, but they are able to provide binary (correct, incorrect) feedback. In this paper we propose feedback-weighted learning based on importance sampling to improve upon an initial supervised system using binary user feedback. We perform simulated experiments on document classification (for development) and Conversational Question Answering datasets like QuAC and DoQA, where binary user feedback is derived from gold annotations. The results show that our method is able to improve over the initial supervised system, getting close to a fully-supervised system that has access to the same labeled examples in in-domain experiments (QuAC), and even matching in out-of-domain experiments (DoQA). Our work opens the prospect to exploit interactions with real users and improve conversational systems after deployment., Comment: Accepted at COLING 2020. 11 pages, 5 figures

Zero-shot learning is a very promising research topic. For a vision-based action recognition system, for instance, zero-shot learning allows to recognise actions never seen during the training phase. Previous works in zero-shot action recognition have exploited in several ways the visual appearance of input videos to infer actions. Here, we propose to add external knowledge to improve the performance of purely vision-based systems. Specifically, we have explored three different sources of knowledge in the form of text corpora. Our resulting system follows the literature and disentangles actions into verbs and objects. In particular, we independently train two vision-based detectors: (i) a verb detector and (ii) an active object detector. During inference, we combine the probability distributions generated from those detectors to obtain a probability distribution of actions. Finally, the vision-based estimation is further combined with an action prior extracted from text corpora (external knowledge). We evaluate our approach on the EGTEA Gaze+ dataset, an Egocentric Action Recognition dataset, demonstrating that the use of external knowledge improves the recognition of actions never seen by the detectors.

One field in which electronic materials have an important role is energy generation, especially within the scope of photovoltaic energy. This paper deals with one of the most relevant enabling technologies within that scope, i.e, the algorithms for maximum power point tracking implemented in the direct current to direct current converters and its modeling through artificial neural networks (ANNs). More specifically, as a proof of concept, we have addressed the problem of modeling a fuzzy logic controller that has shown its performance in previous works, and more specifically the dimensionless duty cycle signal that controls a quadratic boost converter. We achieved a very accurate model since the obtained medium squared error is 3.47 x 10(-6), the maximum error is 16.32 x 10(-3) and the regression coefficient R is 0.99992, all for the test dataset. This neural implementation has obvious advantages such as a higher fault tolerance and a simpler implementation, dispensing with all the complex elements needed to run a fuzzy controller (fuzzifier, defuzzifier, inference engine and knowledge base) because, ultimately, ANNs are sums and products.

Human activity recognition is a key technology for ICT-based (infomation and communication technologies) assistive applications. The most successful activity recognition systems for intelligent environments in terms of performance rely on supervised learning techniques. However, those techniques demand large labelled data sets for specific sensor deployments and monitored person. Such requirements make supervised learning techniques not to scale well to real world deployments, where different sensor infrastructures may be used to monitor different users. In this paper, we present a novel activity recognition system, based on a combination of unsupervised learning techniques and knowledge-based activity models. First, we use a domain-specific data mining algorithm previously developed by Cook et al. to extract the most frequent action sequences executed by a person. Second, we insert knowledge-based activity models in a novel matching algorithm with the aim of inferring what activities are being performed in a given action sequence. The approach results on a scalable activity recognition system, which has been tested on three real data sets. The obtained performance is comparable to supervised learning techniques.

Human activity recognition has the potential to become a real enabler for ambient assisted living technologies. Research on this area demands the execution of complex experiments involving humans interacting with intelligent environments in order to generate meaningful datasets, both for development and validation. Running such experiments is generally expensive and troublesome, slowing down the research process. This paper presents an agent-based simulator for emulating human activities within intelligent environments: MASSHA. Specifically, MASSHA models the behaviour of the occupants of a sensorised environment from a single-user and multiple-user point of view. The accuracy of MASSHA is tested through a sound validation methodology, providing examples of application with three real human activity datasets and comparing these to the activity datasets produced by the simulator. Results show that MASSHA can reproduce behaviour patterns that are similar to those registered in the real datasets, achieving an overall accuracy of 93.52% and 88.10% in frequency and 98.27% and 99.09% in duration for the single-user scenario datasets; and a 99.3% and 88.25% in terms of frequency and duration for the multiple-user scenario.

One of the biggest challenges in modern societies is the improvement of healthy aging and the support to older persons in their daily activities. In particular, given its social and economic impact, the automatic detection of falls has attracted considerable attention in the computer vision and pattern recognition communities. Although the approaches based on wearable sensors have provided high detection rates, some of the potential users are reluctant to wear them and thus their use is not yet normalized. As a consequence, alternative approaches such as vision-based methods have emerged. We firmly believe that the irruption of the Smart Environments and the Internet of Things paradigms, together with the increasing number of cameras in our daily environment, forms an optimal context for vision-based systems. Consequently, here we propose a vision-based solution using Convolutional Neural Networks to decide if a sequence of frames contains a person falling. To model the video motion and make the system scenario independent, we use optical flow images as input to the networks followed by a novel three-step training phase. Furthermore, our method is evaluated in three public datasets achieving the state-of-the-art results in all three of them.

Generating an image from its textual description requires both a certain level of language understanding and common sense knowledge about the spatial relations of the physical entities being described. In this work, we focus on inferring the spatial relation between entities, a key step in the process of composing scenes based on text. More specifically, given a caption containing a mention to a subject and the location and size of the bounding box of that subject, our goal is to predict the location and size of an object mentioned in the caption. Previous work did not use the caption text information, but a manually provided relation holding between the subject and the object. In fact, the used evaluation datasets contain manually annotated ontological triplets but no captions, making the exercise unrealistic: a manual step was required; and systems did not leverage the richer information in captions. Here we present a system that uses the full caption, and Relations in Captions (REC-COCO), a dataset derived from MS-COCO which allows to evaluate spatial relation inference from captions directly. Our experiments show that: (1) it is possible to infer the size and location of an object with respect to a given subject directly from the caption; (2) the use of full text allows to place the object better than using a manually annotated relation. Our work paves the way for systems that, given a caption, decide which entities need to be depicted and their respective location and sizes, in order to then generate the final image., Accepted in Pattern Recognition

Understanding human behaviour is a central task in intelligent environments. Understanding what the user does and how she does it allows to build more reactive and smart environments. In this paper we present a new approach to inter-activity behaviour modelling. This approach is based on the use of multi-scale convolutional neural networks to detect n-grams in action sequences and a novel method of applying soft attention mechanisms at embedding level. The proposed architecture improves our previous architecture based on recurrent networks, obtaining better result predicting the users' actions.

As the average age of the urban population increases, cities must adapt to improve the quality of life of their citizens. The City4Age H2020 project is working on the early detection of the risks related to mild cognitive impairment and frailty and on providing meaningful interventions that prevent these risks. As part of the risk detection process, we have developed a multilevel conceptual model that describes the user behaviour using actions, activities, and intra- and inter-activity behaviour. Using this conceptual model, we have created a deep learning architecture based on long short-term memory networks (LSTMs) that models the inter-activity behaviour. The presented architecture offers a probabilistic model that allows us to predict the user’s next actions and to identify anomalous user behaviours.

The ever-growing life expectancy of people requires the adoption of proper solutions for addressing the particular needs of elderly people in a sustainable way, both from service provision and economic point of view. Mild Cognitive Impairments (MCI) and frailty are typical examples of elderly conditions which, if not timely addressed, can turn out into more complex diseases that are harder and costlier to treat. Information and Communication Technologies (ICTs), and in particular Internet of Things (IoT) technologies, can foster the creation of monitoring and intervention systems, both on an Ambient Assisted Living (AAL) and Smart City scope, for early detecting behavioral changes in elderly people. This allows to timely detect any potential risky situation and properly intervene, with benefits in terms of treatment’s costs. In this context, as part of the H2020-funded City4Age project, this paper presents the data capturing and data management layers of the whole City4Age platform. In particular, this work deals with an unobtrusive data gathering system implementation to collect data about daily activities of elderly people, and with the implementation of the related Linked Open Data (LOD)-based data management system. The collected data are then used by other layers of the platform to perform risk detection algorithms and generate the proper customized interventions. Through the validation of some use-cases, it is demonstrated how this scalable approach, also characterized by unobtrusive and low-cost sensing technologies, can produce data with a high level of abstraction useful to define a risk profile of each elderly person.

Activity Recognition is an important ingredient that allows the interpretation of elementary data. Understanding which activity is going on allows framing an elementary action (e.g. “a movement”) in a proper context. This paper presents an activity recognition system designed to work in urban scenarios, which impose several restrictions: the unfeasibility of having enough annotated datasets, the heterogeneous sensor infrastructures and the presence of very different individuals. The main idea of our system is to combine knowledge- and data-driven techniques, to build a hybrid and scalable activity recognition system for smart cities.

Healthcare environments, as many other real world environments, present many changing and unpredictable situations. In order to use a social robot in such an environment, the robot has to be prepared to deal with all the changing situations. This paper presents a robot self-configuration approach to overcome suitably the commented problems. The approach is based on the integration of a semantic framework, where a reasoner can take decisions about the configuration of robot services and resources. An ontology has been designed to model the robot and the relevant context information. Besides rules are used to encode human knowledge and serve as policies for the reasoner. The approach has been successfully implemented in a mobile robot, which showed to be more capable of solving situations not pre-designed.

Human activity recognition is a key enabler for Ambient Assisted Living and provides a paradigmatic example of how reasoning capacities can be used in such scenarios. In order to detect and recognise human activities, first of all, humans have to be monitored using sensors. The information grabbed by those sensors feed the modelling and inference layers, where the reasoning takes place. In this chapter, the most used inference and reasoning techniques have been introduced. Reasoning systems for AAL have been divided into three different categories, namely, the data-driven approach, the knowledge-driven approach, and the hybrid approach. The most representative examples of those approaches have been presented, describing the advantages and disadvantages. Due to the emergence of the Semantic Web and its application to AAL scenarios, has been devoted to describe the features of OWL and associated semantic reasoners, which can be classified as a knowledge-driven approach.

Being able to recognise human activities by means of sensor and computational devices can be a key competence in order to achieve human centred technologies. For that purpose, it is mandatory to build computational models of the activities which have to be recognised. There are two major approaches for activity modelling:the data-driven and the knowledge-driven approaches. Both of them have advantages and drawbacks. The objective of this work is to combine both modelling approaches with the aim of building dynamic and personalised activity models, using generic knowledge-based models. This would allow implementing modelling processes which can adapt themselves to the evolution of specific people.; Gizakietara egokitutako teknologiak garatzeko, ezinbestekoa da makinek giza jarduerak antzemateko gaitasuna izatea, sentsoreak eta konputazio-gailuak erabiliz. Horretarako, antzeman nahi diren jarduera horien eredu konputazionalak sortu behar dira. Gaur egun, jarduera-ereduak sortzeko garaian, bi joera nagusi aurki daitezke: datuetan oinarritutako ereduak eta ezagutzan oinarritutakoak. Biek ere abantaila eta desabantailak dituzte. Lan honen helburua da bi joerak elkartzea eredu dinamiko eta pertsonalizatuak lortzeko, ezagutzan oinarritutako eredu orokor batzuetatik hasita. Modu horretan, pertsona bakoitzaren bilakaerara egokitutako modelatze-prozesuak lor daitezke.

Open Access article Evaluating human activity recognition systems usually implies following expensive and time-consuming methodologies,where experiments with humans are run with the consequent ethical and legal issues. We propose a novel evaluation methodology to overcome the enumerated problems, which is based on surveys for users and a synthetic dataset generator tool. Surveys allow capturing how different users perform activities of daily living, while the synthetic dataset generator is used to create properly labelled activity datasets modelled with the information extracted from surveys. Important aspects, such as sensor noise, varying time lapses and user erratic behaviour, can also be simulated using the tool. The proposed methodology is shown to have very important advantages that allow researchers to carry out their work more efficiently. To evaluate the approach, a syntheticdatasetgeneratedfollowingtheproposedmethodologyiscomparedtoarealdataset computing the similarity between sensor occurrence frequencies. It is concluded that the similarity between both datasets is more than significant.

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link. We combine knowledge- and data-driven approaches for activity modeling.We develop a novel clustering algorithm that uses prior domain expert knowledge.A new learning algorithm to model activities from extracted clusters.We model a pervasive home environment with real users' inputs for experiments.Automatically learn 100% of activity variations performed by users. Knowledge-driven activity recognition is an emerging and promising research area which has already shown very interesting features and advantages. However, there are also some drawbacks, such as the usage of generic and static activity models. This paper presents an approach to using data-driven techniques to evolve knowledge-driven activity models with a user's behavioral data. The approach includes a novel clustering process where initial incomplete models developed through knowledge engineering are used to detect action clusters which represent activities and aggregate new actions. Based on those action clusters, a learning process is then designed to learn and model varying ways of performing activities in order to acquire complete and specialized activity models. The approach has been tested with real users' inputs, noisy sensors and demanding activity sequences. Initial results have shown that complete and specialized activity models are properly learned with success rates of 100% at the expense of learning some false positive models.

Health care environments, as many other real world environments, present many changing and unpredictable situations. In order to use a social robot in such an environment, the robot has to be prepared to deal with all the changing situations. This paper presents a robot self-configuration approach to overcome suitably the commented problems. The approach is based on the integration of a semantic framework, where a reasoner can take decisions about the configuration of robot services and resources. An ontology has been designed to model the robot and the relevant context information. Besides rules are used to encode human knowledge and serve as policies for the reasoner. The approach has been successfully implemented in a mobile robot, which showed to be more capable of solving not pre-designed situations.

This paper describes an approach to automate professional floor cleaning tasks based on a commercial platform. The described navigation system works in indoor environments where no extra infrastructure is needed and with no previous knowledge of it. A teach&reproduce strategy has been adopted for this purpose. During teaching, the robot maps its environment and the cleaning path. During reproduction, the robot uses a new motion planning algorithm to follow the taught path whilst avoiding obstacles suitably. The new motion planning algorithm is needed due to the special platform and operational requirements. The system presented here is focused on achieving human comparable performance and safety.

RESUMEN: El modelado de actividades humanas es uno de los pasos clave para los sistemas de reconocimiento de actividades, que permite obtener modelos computacionales de las diversas formas en las que los humanos ejecutamos ciertas actividades. Los modelos de actividades no son solo indispensables para el reconocimiento, sino que tambien se pueden usar en otras aplicaciones como la comprension del comportamiento humano o el diagnostico y evolucion de enfermedades mentales. A pesar de la importancia del modelado de actividades, no hay en la literatura una revision completa del mismo. En este articulo, sin embargo, el foco se ha puesto en las tecnicas de modelado, distinguiendo entre modelado basado en conocimiento y en datos. El primero usa el conocimiento previo del dominio para generar los modelos aplicando tecnicas de ingenieria y gestion del conocimiento. Por otro lado, el segundo tiene por objetivo aprender los modelos de actividad directamente usando los datos generados por los usuarios aplicando tecnicas de aprendizaje automatico y mineria de datos. Recientemente han aparecido sistemas de modelado que fusionan el modelado basado en conocimiento y en datos. Su objetivo es combinar las mejores caracteristicas de ambos sistemas, abriendo nuevas lineas de investigacion. Los sistemas de modelado mas relevantes han sido analizados en este articulo. Finalmente, los problemas abiertos de investigacion y el potencial trabajo futuro han sido analizados, para ofrecer una guia a aquellos investigadores que quieran contribuir al area del modelado de actividades. Palabras Clave: modelado de actividades, reconocimiento de actividades, monitorizacion de actividades, basado en el conocimiento, basado en datos

RESUMEN: El modelado de actividades humanas es uno de los pasos clave para los sistemas de reconocimiento de actividades, que permite obtener modelos computacionales de las diversas formas en las que los humanos ejecutamos ciertas actividades. Los modelos de actividades no son solo indispensables para el reconocimiento, sino que tambien se pueden usar en otras aplicaciones como la comprension del comportamiento humano o el diagnostico y evolucion de enfermedades mentales. A pesar de la importancia del modelado de actividades, no hay en la literatura una revision completa del mismo. En este articulo, sin embargo, el foco se ha puesto en las tecnicas de modelado, distinguiendo entre modelado basado en conocimiento y en datos. El primero usa el conocimiento previo del dominio para generar los modelos aplicando tecnicas de ingenieria y gestion del conocimiento. Por otro lado, el segundo tiene por objetivo aprender los modelos de actividad directamente usando los datos generados por los usuarios aplicando tecnicas de aprendizaje automatico y mineria de datos. Recientemente han aparecido sistemas de modelado que fusionan el modelado basado en conocimiento y en datos. Su objetivo es combinar las mejores caracteristicas de ambos sistemas, abriendo nuevas lineas de investigacion. Los sistemas de modelado mas relevantes han sido analizados en este articulo. Finalmente, los problemas abiertos de investigacion y el potencial trabajo futuro han sido analizados, para ofrecer una guia a aquellos investigadores que quieran contribuir al area del modelado de actividades. Palabras Clave: modelado de actividades, reconocimiento de actividades, monitorizacion de actividades, basado en el conocimiento, basado en datos