Search

Hierarchical models have shown their effectiveness for action recognition. However, most of the existing hierarchy construction methods fail to model the complex motion patterns in videos, and thus are vulnerable to the interference of the inevitable noise in action videos. Therefore, we propose a Dynamic Hierarchical Tree (DHT) model to characterize such complex motion for better recognition performance. First, a minimum maximum DTW (mmDTW) is developed to produce more stable atomic actions by limiting the minimum and maximum lengths of atomic actions. Then an aggregation method is utilized to construct a DHT for each video by merging atomic actions from bottom to top. Not only the similarity between frames but also the compatibility of dynamic evolution between frames and segments is exploited for the mmDTW and the aggregation process, making the DHTs more suitable for modeling actions in videos. Finally, a k-Nearest Neighbors Edge Pairs (kNNEP) kernel is proposed to compare two DHTs by using the mean similarity of k nearest neighbors edge pairs.

Human activity recognition (HAR) aims to recognize the actions of the human body through a series of observations and environmental conditions. The analysis of human activities has drawn the attention of the research community in the last two decades due to its widespread applications, diverse nature of activities, and recording infrastructure. Lately, one of the most challenging applications in this framework is to recognize the human body actions using unobtrusive wearable motion sensors. Since the human activities of daily life (e.g., cooking, eating) comprises several repetitive and circumstantial short sequences of actions (e.g., moving arm), it is quite difficult to directly use the sensory data for recognition because the multiple sequences of the same activity data may have large diversity. However, a similarity can be observed in the temporal occurrence of the atomic actions. Therefore, this paper presents a two-level hierarchical method to recognize human activities using a set of wearable sensors. In the first step, the atomic activities are detected from the original sensory data, and their recognition scores are obtained. Secondly, the composite activities are recognized using the scores of atomic actions. We propose two different methods of feature extraction from atomic scores to recognize the composite activities, and they include handcrafted features and the features obtained using the subspace pooling technique. The proposed method is evaluated on the large publicly available CogAge dataset, which contains the instances of both atomic and composite activities. The data is recorded using three unobtrusive wearable devices: smartphone, smartwatch, and smart glasses. We also investigated the performance evaluation of different classification algorithms to recognize the composite activities. The proposed method achieved 79% and 62.8% average recognition accuracies using the handcrafted features and the features obtained using subspace pooling technique, respectively. The recognition results of the proposed technique and their comparison with the existing state-of-the-art techniques confirm its effectiveness.

This paper presents a framework to recognize temporal compositions of atomic actions in videos. Specifically, we propose to express temporal compositions of actions as semantic regular expressions and derive an inference framework using probabilistic automata to recognize complex actions as satisfying these expressions on the input video features. Our approach is different from existing works that either predict long-range complex activities as unordered sets of atomic actions, or retrieve videos using natural language sentences. Instead, the proposed approach allows recognizing complex fine-grained activities using only pretrained action classifiers, without requiring any additional data, annotations or neural network training. To evaluate the potential of our approach, we provide experiments on synthetic datasets and challenging real action recognition datasets, such as MultiTHUMOS and Charades. We conclude that the proposed approach can extend state-of-the-art primitive action classifiers to vastly more complex activities without large performance degradation., Accepted in Workshop on Compositionality in Computer Vision at CVPR, 2020

Although First Person Vision systems can sense the environment from the user’s perspective, they are generally unable to predict his intentions and goals. Since human activities can be decomposed in terms of atomic actions and interactions with objects, intelligent wearable systems would benefit from the ability to anticipate user-object interactions. Even if this task is not trivial, the First Person Vision paradigm can provide important cues to address this challenge. We propose to exploit the dynamics of the scene to recognize next-active-objects before an object interaction begins. We train a classifier to discriminate trajectories leading to an object activation from all others and forecast next-active-objects by analyzing fixed-length trajectory segments within a temporal sliding window. The proposed method compares favorably with respect to several baselines on the Activity of Daily Living (ADL) egocentric dataset comprising 10 h of videos acquired by 20 subjects while performing unconstrained interactions with several objects.

This paper presents an approach to recognize human activities using body poses estimated from RGB-D data.We focus on recognizing complex activities composed of sequential or simultaneous atomic actions characterized by body motions of a single actor. We tackle this problem by introducing a hierarchical compositional model that operates at three levels of abstraction. At the lowest level, geometric and motion descriptors are used to learn a dictionary of body poses. At the intermediate level, sparse compositions of these body poses are used to obtain meaningful representations for atomic human actions. Finally, at the highest level, spatial and temporal compositions of these atomic actions are used to represent complex human activities.Our results show the benefits of using a hierarchical model that exploits the sharing and composition of body poses into atomic actions, and atomic actions into activities.A quantitative evaluation using two benchmark datasets illustrates the advantages of our model to perform action and activity recognition. A novel hierarchical model to recognize human activities using RGB-D data is proposed.The method jointly learns suitable representations at different abstraction levels.The model achieves multi-class discrimination providing useful mid-level annotations.The compositional capabilities of our model also bring robustness to body occlusions.

The importance of embodiment for effective robot performance has been postulated for a long time. Despite this, only relatively recently concrete quantitative models were put forward to characterize the advantages provided by a well-chosen embodiment. We here use one of these models, based on the concept of relevant information, to identify in a minimalistic scenario how and when embodiment affects the decision density. Concretely, we study how embodiment affects information costs when, instead of atomic actions, scripts are introduced, that is, predefined action sequences. Their inclusion can be treated as a straightforward extension of the basic action space. We will demonstrate the effect on informational decision cost of utilizing scripts vs. basic actions using a simple navigation task. Importantly, we will also employ a world with “mislabeled” actions, which we will call a “twisted” world. This is a model which had been used in an earlier study of the influence of embodiment on decision costs. It will turn out that twisted scenarios, as opposed to well-labeled (“embodied”) ones, are significantly more costly in terms of relevant information. This cost is further worsened when the agent is forced to lower the decision density by employing scripts (once a script is triggered, no decisions are taken until the script has run to its end). This adds to our understanding why well-embodied (interpreted in our model as well-labeled) agents should be preferable, in a quantifiable, objective sense.

The Iterated Prisoner's Dilemma has guided research on social dilemmas for decades. However, it distinguishes between only two atomic actions: cooperate and defect. In real-world prisoner's dilemmas, these choices are temporally extended and different strategies may correspond to sequences of actions, reflecting grades of cooperation. We introduce a Sequential Prisoner's Dilemma (SPD) game to better capture the aforementioned characteristics. In this work, we propose a deep multiagent reinforcement-learning approach that investigates the evolution of mutual cooperation in SPD games. Our approach consists of two phases. The first phase is offline: it synthesizes policies with different cooperation degrees and then trains a cooperation degree detection network. The second phase is online: an agent adaptively selects its policy based on the detected degree of opponent cooperation. The effectiveness of our approach is demonstrated in two representative SPD 2D games: the Apple-Pear game and the Fruit Gathering game. Experimental results show that our strategy can avoid being exploited by exploitative opponents and achieve cooperation with cooperative opponents.

This paper addresses the challenging problem of complex human activity understanding from long videos. Towards this goal, we propose a hierarchical description of an activity video, referring to the "which" of activities, "what" of atomic actions, and "when" of atomic actions happening in the video. In our work, each complex activity is characterized as a composition of simple motion units (called atomic actions), and different atomic actions are explained by different video segments. We develop a latent discriminative structural model to detect the complex activity and atomic actions, while learning the temporal structure of atomic actions simultaneously. A segment-annotation mapping matrix is introduced for relating video segments to their associational atomic actions, allowing different video segments to explain different atomic actions. The segment-annotation mapping matrix is treated as latent information in the model, since its ground-truth is not available during both training and testing. Moreover, we present a semi-supervised learning method to automatically predict the atomic action labels of unlabeled training videos when the labeled training data is limited, which could greatly alleviate the laborious and time-consuming annotations of atomic actions for training data. Experiments on three activity datasets demonstrate that our method is able to achieve promising activity recognition results and obtain rich and hierarchical descriptions of activity videos.

In this paper, we present a model for learning atomic actions for complex activities classification. A video sequence is first represented by a collection of visual interest points. Then the model automatically clusters visual words into atomic actions (topics) based on their co-occurrence and temporal proximity in the same activity category using an extension of hierarchical Dirichlet process (HDP) mixture model. Our approach is robust to noisy interest points caused by various conditions because HDP is a generative model. Finally, we use both a Naive Bayesian and a linear SVM classifier for the problem of activity classification. We first use the intermediate result of a synthetic example to demonstrate the superiority of our model, then we apply our model on the complex Olympic Sport 16-class dataset and show that it outperforms other state-of-art methods.

In household environments, robots are expected to conduct many tasks. It is difficult, however, to write programs for all tasks beforehand ddue to task diversity and changing environmental conditions. One basic task of developing autonomous multifunctional robots is to define a set of basic robot actions that are executed unambiguously and checked for completion. A task planner then uses these actions to accomplish complex tasks that home-service robots are expected to do. This paper first proposes a set of tasks for first-generation home-service robots, then systematically decomposes them into sequences of smaller but meaningful actions called molecular actions. Molecular actions are then broken down into yet more primitive actions called atomic actions. Because vision, sound, range sensors, and force sensors are the main means of monitoring task progress and ompletion, atomic actions are classified based on the complexities and frequencies of the sensing algorithms used. The resulting taxonomy of atomic actions serves as a set of basic building blocks for a knowledge-based task planner. Its advantages are verified and demonstrated through experiments.

We demonstrate ways to incorporate nondeterminism in a system designed to formalize the reasoning of agents concerning their abilities and the results of the actions that they may perform. We distinguish between two kinds of nondeterministic choice operators: one that expresses an internal choice, in which the agent decides what action to take, and one that expresses an external choice, which cannot be influenced by the agent. The presence of abilities in our system is the reason why the usual approaches towards nondeterminism cannot be used here. The semantics that we define for nondeterministic actions is based on the idea that composite actions are unravelled in the strings of atomic actions and tests that constitute them. The main notions used in defining this semantics are finite computation sequences and finite computation runs of actions. The results that we obtain meet our intuitions regarding events and abilities in the presence of nondeterminism.

An approach to operator interaction for a dextrous manipulator is described, using a tele-assistance method termed blind grasping. Given approximate initial coordinates for the object location, the blind grasping approach sequences a series of dextrous finger contact movements to establish the object geometry and candidate stable grasp locations. The approach is implemented within the framework of the preference planning tele-assistant task planning architecture. Geometric and semantic models of the world are used as triggers into pre-stored lists of action sequences. Actions are triggered when their pre-conditions fail, representing a state in the world that the action can correct. Failure of the action to change the world state hierarchically triggers sub-actions, until atomic actions instruct lower levels of control. For the first prototype AMADEVS dextrous underwater gripper, it is demonstrated that variously shaped objects centred between the open gripper fingers can generally be grasped using simple...

In online planning with a team of cooperative agents, a straightforward model for decision making which actions the agents should execute can be represented as the problem of Combinatorial Multi-Armed Bandit. Similarly to the most prominent approaches for online planning with polynomial number of possible actions, state-of-the-art algorithms for online planning with exponential number of actions are based on Monte-Carlo sampling. However, without a proper selection of the appropriate subset of actions these techniques cannot be used. The most recent algorithms tackling this problem utilize an assumption of linearity with respect to the combinations of the actions. In this paper, we experimentally analyze robustness of two state-of-the-art algorithms NMC and LSI for online planning with combinatorial actions in various setups of Real-Time and Turn-Taking Strategy games.

A combining network can combine concurrently issued operations on the same variable and fan out responses to the processes that issued the operations. Combining networks have been proposed with the goals of reducing message traffic, hotspots within the network, and contention for shared variables. This paper extends the class of operations that can be combined to include operations from flat atomic actions. We describe a class of networks called isotach networks and show that isotach networks can correctly combine operations from different flat atomic actions. Also we show that isotach combining networks can combine pipelined operations without sacrificing sequential consistency.

We introduce an atomic action based features and demonstrate that it consistently improves performance on human activity recognition. The features are built using auxiliary atomic action data collected in our lab. We train a kernelized SVM classifier for each atomic action class. Then given a local spatio-temporal cuboid of a test video, we represent it using the responses of our atomic action classifiers. This new atomic action feature is discriminative, and has semantic meanings. We perform extensive experiments on four benchmark action recognition datasets. The results show that atomic action features either outperform the corresponding low level features or significantly boost the recognition performance by combining the two.

An examination of the structure of fault-tolerant systems incorporating backward error recovery indicates a partitioning into two broad classes. Two canonical models, each representing a particular class of systems, have been constructed. The first model incorporates objects and actions as the entities for program construction whereas the second model employs communicating processes and conversations. Applications in areas such as office information and banking systems are typically described and built in terms of the first model whereas applications in the area of process control are usually described and built in terms of the second model. The paper claims that the two models are duals of each other and presents arguments and examples to substantiate this claim. It will be shown that the techniques that have been developed within the context of one model turn out to have interesting and hitherto unexplored duals in the other model.

Some time ago, Computer Vision has passed the stage where it detected changes in image sequences, estimated Optical Flow, or began to track people and vehicles in videos. Currently, research in Computer Vision has expanded to extract descriptions of single actions or concatenations of actions from videos, sometimes even the description of agent behavior in the recorded scene. This transition from treating mostly quantitative, geometric descriptions to becoming concerned with more qualitative, conceptual descriptions creates contacts between Computer Vision, Computational Linguistics, and Computational Logic. The latter two disciplines have studied the analysis and combination of conceptual constructs already for decades. Based on selected examples, attention will be drawn to the potential which can be tapped if the emerging thematic overlap of research in these three disciplines is investigated collaboratively. This applies in particular to the development of encompassing systems which rely on methods from all three disciplines, for example by providing Natural Language interfaces to more generally applicable combinations of Knowledge Bases with Computer Vision systems.

This paper presents a method of how to simulate various human actions. Actions are formed in the Action Hierarchy, and are reified in terms of expanding the hierarchy. While inheritance, super classes provide overall information of actions, and sub classes are used as the actual functions of actions. Primitive actions as leaves of the action hierarchy are proposed for an atomic action to be shared in several actions, and used as key frame. The Action Generator guarantees to return accurate primitive actions to an agent based on the Action Library and the Action Hierarchy, and also provides individual patterns of actions.

Emotional-BDI agents are BDI agents whose behaviour is guided not only by beliefs, desires and intentions, but also by the role of emotions in reasoning and decision-making. The $\mathcal{E}_{\rm BDI}$ logic is a formal system for expressing the concepts of the Emotional-BDI model of agency. In this paper we present an improved version of the $\mathcal{E}_{\rm BDI}$ logic and show how it can be used to model the role of three emotions in Emotional-BDI agents: fear , anxiety and self-confidence . We also focus in the computational properties of $\mathcal{E}_{\rm BDI}$ which can lead to its use in automated proof systems.

A typical present-day virtual actor is able to store episodes in an ad hoc manner, which does not allow for reconstructing the actor's personal stories. This paper proposes a virtual RPG actor with a full episodic memory, which allows for this reconstruction. The paper presents the memory architecture, overviews the prototype implementation, presents a benchmark for the efficiency of the memory measurement, and details the conducted tests.

We present a formal system to reason about and specify the behavior of multiple intelligent artificial agents. Essentially, each agent can perform certain actions, and it may possess a variety of information in order to reason about its and other agent's actions. Thus, our KARO-framework tries to deal formally with the notion of Knowledge, possessed by the agents, and their possible execution of actions. In particular, each agent may reason about its —or, alternatively, other's— Abilities to perform certain actions, the possible Results of such an execution and the availability of the Opportunities to take a particular action. Formally, we combine dynamic and epistemic logic into one modal system, and add the notion of ability to it. We demonstrate that there are several options to define the ability to perform a sequentially composed action, and we outline several properties under two alternative choices. Also, the agents' views on the correctness and feasibility of their plans are highlighted. Finally, the complications in the completeness proof for both systems indicate that the presence of abilities in the logic makes the use of infinite proof rules useful, if not inevitable.

This paper is devoted to developed description of informative model of the user, who may be under the threat of socioengineering attack, and some other models, which is connected to the first one: users group model, model of control area, informative objects (documents) model. Specified informative model are included into the base for analyzing of the protection of users of informative system from socioengineering attack. Informative model of the user allows to consider name and surname of the user, his post in the organization, belongings of user to user group, and vulnerability of the user on socioengineering attacks. informative model of user group allows to consider name of user group, it’s description, allows for different atomic actions which user can perform with informative objects, type of access to information objects and information objects, which this group of users can use. Informative model of control area allows to consider name of control area and it’s description. Information model of information objects includes damage estimation of losses of confidentially, losses of integrity and losses of sufficiency. The example of socioengineering attack is brought. Development of this attack is described through suggested informative models.

This paper proposes a formal framework for modelling the interaction of concurrent items of equipment in real-time safety-critical systems and reasoning about their behaviour abstractly. The framework is based on the concept of Coordinated Atomic (CA) actions, an approach widely used for structuring complex activities in fault-tolerant computer systems. It advocates a hierarchical approach and begins with the construction of a mathematical model of the behaviour of an individual item of equipment. Later on, the model is extended to incorporate the concept of a CA action. In the final stage, a formal representation of the ideal behaviour of an abstract CA action is provided. The framework uses Timed CSP - a well-established formalism used for representation and reasoning in real-time systems.

In this paper, we propose a new way of considering reasoning about action and change. Rather than placing a preferential structure onto the models of logical theories, we place such a structure directly on the semantics of the actions involved. In this way, we obtain a preferential semantics of actions by means of which we can not only deal with several of the traditional problems in this area such as the frame and ramification problems, but can generalize these solutions to a context which includes both nondeterministic and concurrent actions. In fact, the net result is an integration of semantical and verificational techniques from the paradigm of imperative and concurrent programs in particular, as known from traditional programming, with the AI perspective. In this paper, the main focus is on semantical (i.e. model theoretical) issues rather than providing a logical calculus, which would be the next step in the endeavor.

Event-driven and time-driven models exhibit a complementary nature, in the sense that they reflect the nature of the contrasting requirements that any real-time system must meet - i.e., performing actions “as soon as possible” or “at the right time”. Even if each of the models is sufficient to model a system, they should be intermixed to improve the expressiveness. This can be achieved by assuming one model as the primitive, and implementing the other on top of it. The paper introduces the rationale for choosing the time-driven model as the primitive, and discusses why this approach is possibly the most suitable for a wide range of control applications. In conclusion, a unifying model, based on the distinction between the atomic actions performed by reactive agents, the control performed by a time-driven control machine, and the planning performed by an event-driven planning machine, is introduced.

We study a problem of actions with typical, but not certain effects. We show how this kind of actions can be incorporated in a dynamic/epistemic multi-agents system in which the knowledge, abilities and opportunities of agents are formalized as well as the results of actions they perform. To cope with complexity of a rational agent behaviour, we consider scenarios composed of traditionally viewed basic actions and atomic actions with typical effects. We focus on a specific type of scenarios reflecting a “typical” pattern of an agent's behaviour. Adopting a model-theoretic approach we formalize a nonmonotonic preferential strategy for these scenarios in order to reason about the final results of their realizations.

In this paper we define a framework designed to formalize certain aspects of the behaviour of rational agents. To this end we start by defining a language L in which not only knowledge and actions of a set of agents can be expressed, but also their abilities. A semantics for the language L is defined; a sound and complete axiomatization with regard to the class of models used to define this semantics is given. We define the notion of action transformation, a function that transforms actions into other actions. The semantics that we define is such that two actions that can be transformed into one another have equivalent results and that the abilities of an agent are closed under the action transformations. We also consider the Can-predicate and the Cannot-predicate that formalize the knowledge of agents about the (in)correctness and (un)feasibility of their plans.

In this article we attempt to describe and analyze the formalisms of pervasive games and pervasive gaming (PG). As the title indicates, PG consists of atomic entities that nevertheless merge into a molecular structure that exhibits emergent features during the actual game-play. The article introduces four axes of PG (mobility, distribution, persistence, and transmediality). Further, it describes and analyses three key units of PG (rules, entities, and mechanics) as well as discusses the role of space in PG by differentiating between tangible space, information-embedded space, and accessibility space. The article is generally concerned with classifying the indispensable components of pervasive games and, in addition, it lists the invariant features of pervasive game-play, meaning the epistemology that is tied to this new kind of gaming situated on the borderline between corporeal and immaterial space. Of particular interest are game rules in pervasive gaming, since they seem to touch upon both the underlying, formal structure of the game (i.e., the ontology of PG) and the actual play vis-à-vis physical and/or virtual constraints (i.e., the PG epistemology).

Episodic memory represents personal history of an entity. Humanlike agents with a full episodic memory are able to reconstruct their personal stories to a large extent. Since these agents typically live in dynamic environments that change beyond their capabilities, their memory must cope with determining trustfulness of memory records. In this paper, we propose an associative network addressing this issue with regard to records about objects an agent met during its live. The network is presently being implemented into our case-study human-like agent with a full episodic memory.

Action rules describe possible transitions of objects from one state to another with respect to a distinguished attribute. Early research on action rule discovery usually required the extraction of classification rules before constructing any action rule. Newest algorithms discover action rules directly from a decision system. To our knowledge, all these algorithms assume that all attributes are symbolic or require prior discretization of all numerical attributes. This paper presents a new approach for generating action rules from datasets with numerical attributes by incorporating a tree classifier and a pruning step based on meta-actions. Meta-actions are seen as a higher-level knowledge (provided by experts) about correlations between different attributes.