Search

This paper describes a novel approach of segmenting and classifying eating gestures from wrist motion using a deep learning neural network. It is inspired by the approach of fully-convolutional neural networks in the task of image segmentation. Our idea is to segment 1D gestures the same way 2D image regions are segmented, by treating each inertial measurement unit (IMU) datum like a pixel. The novelty of our approach lies in training a neural network to recognize data points that describe an eating gesture just like it would be trained to recognize pixels describing an image region. The data for this research is known as the Clemson Cafeteria Dataset. It was collected from 276 participants that ate an unscripted meal at the Harcombe Dining Hall at Clemson University. Each meal consisted of 1 - 4 courses, and 488 such recordings were used for the experiments described in this paper. Sensor readings consist of measurements taken by an accelerometer (x, y, z) and a gyroscope (yaw, pitch, roll). A total of 51,614 unique gestures associated with different activities commonly seen during a meal were identified by 18 trained raters. Our neural network classifier recognized an average of 79.7% of ‘bite’ and 84.7% of ‘drink’ gestures correctly per meal. Overall 77.7% of all gestures were recognized correctly on average per meal. This indicates that a deep learning model can successfully be used to segment eating gestures from a time series recording of IMU data using a technique similar to pixel segmentation within an image.

This paper evaluates the use of a sensor mounted on a finger to detect bites during food consumption. Previous works have demonstrated the capability to detect bites by tracking wrist motion. The key motion is the roll of the wrist that occurs between picking up food and delivering it to the mouth. We hypothesize that the roll axes of the wrist, hand, and finger are congruent, and that therefore the roll motion measured at the wrist can be similarly measured at the finger. We collected data for 10 meals of subjects wearing sensors on both the wrist and finger. Results indicate that the bite counting algorithm obtains similar accuracy at both locations. The practical implication is that an eating activity monitor could be embedded in the form of a ring worn on the finger, and that this form factor could work as well as a watch-like device.

This work describes a new dataset to improve pedometer evaluation. Prior evaluation techniques focus on regular gaits using laboratory assessment to simplify the manual counting of actual steps. Our goal is to analyze pedometer algorithms under more natural conditions that occur during daily living where gaits are frequently changing or remain regular for only brief periods of time. We video recorded 30 participants performing 3 activities: walking around a track, walking through a building, and moving around a room. Walking around a track uses a regular, consistent gait, and represents the traditional approach to pedometer evaluation. Walking through a building and around a room are activities that include varying amounts of pauses and gait changes, and represent a wider variety of normal daily activities. The ground truth time of each step was manually marked in the accelerometer signals by observing the videos. Collectively 60,853 steps were recorded and annotated. A subclass of steps called shifts were identified as those occurring at the beginning and end of regular strides, during gait changes, and during pivots changing the direction of motion. While shifts comprised only .03% of steps in the regular stride activity, they comprised 10–25% of steps in the semi-regular and unstructured activities. We believe these motions should be identified separately, as they provide different accelerometer signals, and likely result in different amounts of energy expenditure. The proposed dataset will be the first to specifically allow for pedometer algorithms to be evaluated on unstructured gaits that more closely model natural activities.

More than one third of adults in the United Statesare now classified as obese. Self monitoring of energy intakehas been shown to have a positive impact on weight loss, butexisting tools for logging eating activities are tedious to use andprone to bias which leads to noncompliance and underestimation. In this paper we describe preliminary results from our ongoingdata collection from 500 free living participants, in an effort toimprove our previous algorithm that detects periods of eating. We see a 75% accuracy in our data collected to date.

In this paper we present a method to compute the egomotion of a range camera using the space envelope. The space envelope is a geometric model that provides more information than a simple segmentation for correspondences and motion estimation. We describe a novel variation of the maximal matching algorithm that matches surface normals to find correspondences. These correspondences are used to compute rotation and translation estimates of the egomotion. We demonstrate our methods on two image sequences containing 70 images. We also discuss the cases where our methods fail, and additional possible methods for exploiting the space envelope.

We describe an automated method to locate the optic nerve in images of the ocular fundus. Our method uses a novel algorithm we call fuzzy convergence to determine the origination of the blood vessel network. We evaluate our method using 31 images of healthy retinas and 50 images of diseased retinas, containing such diverse symptoms as tortuous vessels, choroidal neovascularization, and hemorrhages that completely obscure the actual nerve. On this difficult data set, our method achieved 89% correct detection. We also compare our method against three simpler methods, demonstrating the performance improvement. All our images and data are freely available for other researchers to use in evaluating related methods.

This work considers the problem of calibrating a distributed multi-camera system. As opposed to traditional multi-camera systems, such as stereo heads, in a distributed network the fields-of-view do not all overlap. Our novel method uses a grid of domino calibration targets. We also describe a novel grid-finding algorithm, to expedite the location of image-to-world correspondences. Experiments conducted in a hallway and two connecting rooms, using 12 cameras, demonstrate accuracies of 4.3 mm in world coordinates and 0.28 pixels in image coordinates.

The construction of a surface model from range data may be undertaken at any point in a continuum of scales that reflects the level of detail of the resulting model. This continuum relates the construction parameters to the scale of the model. We propose methods to dynamically reprocess range data at different scales. The construction result from a single scale is automatically evaluated, causing reconstruction at a different scale when user-defined criteria are not met. We demonstrate our methods in constructing a planar b-rep space envelope (a scene representation) for over 400 range images. The experiments demonstrate the ability to construct 100 percent valid models, with the scale of detail within specified requirements.

Knowledge about required functionality of an object can be used as an effective representation for a generic object category (e.g. "chair", "cup", or "hammer"). This approach to object representation and recognition has recently become an active area of research. We explore a scenario in which a robot senses the environment to obtain an initial partial shape model of an object. If the information in this initial model is not sufficient to hypothesize a possible function for the object, then additional view(s) may be suggested. Once a possible function is hypothesized, a plan is formulated for interacting with the object to confirm that its material properties are compatible with the hypothesized function. The module for reasoning about partial shape models has been evaluated on over 200 shape models acquired from range images. The module for carrying out a function verification plan has been evaluated in a simulated environment using the ThingWorld (TW) system.

A methodology for evaluating range image segmentation algorithms is proposed. This methodology involves (1) a common set of 40 laser range finder images and 40 structured light scanner images that have manually specified ground truth and (2) a set of defined performance metrics for instances of correctly segmented, missed, and noise regions, over- and under-segmentation, and accuracy of the recovered geometry. A tool is used to objectively compare a machine generated segmentation against the specified ground truth. Four research groups have contributed to evaluate their own algorithm for segmenting a range image into planar patches.

A new approach is presented for extracting an explicit 3D shape model from a single range image. One novel aspect is that the model represents both observed object surfaces, and surfaces which bound the volume of occluded space. Another novel aspect is that the approach does not require that the range image segmentation be perfect. The low-level segmentation may be such that the model-building process encounters topology versus geometry conflicts. The model-building process is designed to be "fail soft" in the face of such problems. The portion of the 3D model where a problem presents itself is "glued" together in a manner meant to minimize the disturbance in the 3D shape. The goal is to produce a valid boundary-representation which can be processed by higher-level routines. A third novel aspect of this work is that the implementation has been evaluated on over 200 real range images of polyhedral objects, with no operator intervention and all parameters held constant, and obtained a 97% success rate in creating valid b-reps. >

Trilateration calculations are affected by errors in distance measurements from the set of fixed points to the object of interest. When these errors are systemic, each distinct set of fixed points can be said to exhibit a unique set noise. For ultrawideband (UWB) indoor position tracking, the set of fixed points is a set of sensors measuring the distance to a tracked tag. In this paper we develop a noise model for this sensor set noise, along with a particle filter that uses our set noise model. We test our methods on a real UWB system. Our methods showed an approximately 15% improvement in the accuracy over the raw measurements.

We introduce methods for detecting in real-time information concerning bites taken during a meal. Our methods use an orientation sensor placed on the wrist of a user, and analyze the rolling motion of the wrist in order to detect a pattern related to biting behavior. We have built a prototype bite detector device based upon our methods. The device can count the total number of bites the user has taken, and provide the bites-taken rate (bites per minute) of the user. Experiments have been conducted to determine its accuracy. Ten subjects ate a meal of their choice, using utensils (or fingers) of their choice. Video was recorded of subjects eating, and synchronized with our device, in order to evaluate its performance. The sensitivity of the device was found to be 91%. Our methods could find use in a number of applications, including helping a user with obesity, eating disorders or eating rate problems.

Most current dexterous robot hands are expensive and heavy, which limits the possibility of putting them on industrial robots because the maximum payload that an industrial robot can carry is small. Additionally, current robot hands are not very adaptable. Most grasping research doesn't include vision. A few works incorporate visual sensing, however, these concentrate on stationary objects. In this work, we present a novel pneumatic three-finger hand and use our earlier work in model based dynamic intercept to allow the hand to grasp semi-randomly moving objects. Experimental results show that this novel three-finger hand is simple, light and effective

Visual sensing for robotics has been around for decades, but our understanding of a timing model remains crude. By timing model, we refer to the delays (processing lag and motion lag) between "reality" (when a part is sensed), through data processing (the processing of image data to determine part position and orientation), through control (the computation and initiation of robot motion), through "arrival" (when the robot reaches "reality"). In this work we introduce a timing model where sensing and control operate asynchronously. We apply this model to a robotic workcell consisting of a Staubli RX-130 industrial robot manipulator, a network of six cameras for sensing, and an off-the-shelf Adept MV-19 controller. We demonstrate some experiments to show how the model is applied.

We describe an algorithm to fuse intensity data from multiple video cameras to create a spatial-temporal occupancy map. The camera layout is assumed to resemble a security video network. The occupancy map is a two-dimensional raster image, uniformly distributed in the floor-plane. Each map pixel contains a binary value, signifying whether the designated floorspace is empty or occupied. Our algorithm requires only one difference and one look-up table operation to determine each pixel's effect upon the map. This brevity of operations allows the spatial occupancy map to be temporally computed at real-time video rates. We demonstrate our algorithm operating in several dynamic scenarios.

A segmented range image contains a set of regions representing the projections of the surfaces imaged. Many methods have been proposed to perform the segmentation and surface fitting processes; that is not what this paper is about. Instead, we consider a problem very little analysis has been given to: finding boundary relationships among the regions/surfaces. Since a segmented range image gives us a set of surface patches, it is natural to search for the topology of the 3D surface patches (that is, how the surface patches are connected together). A solution to this problem gives a true 3D boundary representation of the scene. >

This paper describes a definition of the range image segmentation (of polyhedral scenes) problem, a data set to use in evaluation, a method for specifying ground truth, and a set of metrics to classify segmentation results against ground truths. >

The dominant architecture for mobile robot perception uses sensors on-board the robot, providing only a first-person perspective on the environment. This work describes a novel mobile robot system that uses an environment-based sensor network, providing a powerful third-person perspective on the environment. In our previous work (1999), we described an algorithm that computes a real-time spatial-temporal occupancy map from the multiple video streams of the sensor network. We also described a novel path-planning algorithm based upon this system. In this work we describe a novel motion control loop that is based upon tracking the mobile robot in the occupancy map. Tracking in the fused perceptual space of the sensor network provides several advantages over tracking individually in a set of raw sensor spaces. We demonstrate a prototype system operating in several dynamic scenarios.

Trilateration calculations are affected by errors in distance measurements from the set of fixed points to the object of interest. When these errors are systemic, each distinct set of fixed points can be said to exhibit a unique set noise. For ultra-wideband (UWB) indoor position tracking, the set of fixed points is a set of sensors measuring the distance to a tracked tag. In this paper we develop a noise model for this sensor set noise, along with a particle filter that uses our set noise model. We test our methods on a real UWB system. Our methods showed an approximately 15% improvement in accuracy over the raw measurements.

The paper describes a function-based reasoning system that analyzes incomplete object shape descriptions of the type that can be acquired from a single view or from a sequence of views in which not all of the object is seen. Results are presented for the analysis of the partial shape descriptions extracted from over 200 real laser range finder images taken by a mobile robot. The work represents the first function-based recognition system to reason from incomplete knowledge of 3D object shape and also the first to analyze 3D shape descriptions acquired from real image data. Results demonstrate that function based recognition is a feasible and practical capability to pursue for autonomous robotic systems. >

In order for a mobile robot to acquire a shape model of an unknown object, it must be able to view the entire exterior of the object. However, in an unstructured environment, it is impossible to know the extent to which the robot can circumnavigate the object. If the entire object cannot be seen, then it is impractical to discuss creating object models which contain only viewable object surfaces. In fact, it is easy to conceive if an object which possesses exterior surfaces that are hidden from any reasonable viewpoint. However, it is generally possible to establish limits to the volume of space that the object can occupy. Such a volume represents the combination of space occluded from view with space actually taken up by the object. A model of this volume is valuable, in that it has the advantage of being a complete, enclosed boundary description. Object recognition routines, for example, may require complete boundary descriptions to work with. Even if complete boundary descriptions are not required, knowing the maximum possible extent of the object could prove valuable, perhaps in differentiating between several partial object model matches. Processing a single view, we build an ''OPUS'' (object plus unseen space) by combining ''object surfaces''--defining the fraction of the exterior of the object that can actually be seen--with ''occlusion surfaces''-- indicating the limits to the volume of space which is occluded from view.© (1992) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

This work introduces the space envelope, a shape model based upon a boundary description. Instead of modeling a single object (or solid), a space envelope encloses a volume of empty space. The advantage is that in any given view, there may be any number of objects, this number being difficult to determine from pixel-data alone. However, there is always one, and only one, volume of visible empty space. Once a model has been constructed defining the space envelope, higher-order operations may be applied to reason about the scene's content. For instance, surface geometries and topology could yield insight into the number of visible objects. The enclosed empty volume may also be used for vision-based navigation (known free-space), while the surfaces are used for view correspondences.Algorithms to automatically construct a planar boundary representation (b-rep) space envelope from a range image are presented. Results for testing the algorithms on over 400 images from four different range cameras are presented.