Search

Abstract Understandings of the three-dimensional social behaviors of freely moving large-size mammals are valuable for both agriculture and life science, yet challenging due to occlusions in close interactions. Although existing animal pose estimation methods captured keypoint trajectories, they ignored deformable surfaces which contained geometric information essential for social interaction prediction and for dealing with the occlusions. In this study, we develop a Multi-Animal Mesh Model Alignment (MAMMAL) system based on an articulated surface mesh model. Our self-designed MAMMAL algorithms automatically enable us to align multi-view images into our mesh model and to capture 3D surface motions of multiple animals, which display better performance upon severe occlusions compared to traditional triangulation and allow complex social analysis. By utilizing MAMMAL, we are able to quantitatively analyze the locomotion, postures, animal-scene interactions, social interactions, as well as detailed tail motions of pigs. Furthermore, experiments on mouse and Beagle dogs demonstrate the generalizability of MAMMAL across different environments and mammal species.

We present a neural hand reconstruction method for monocular 3D hand pose and shape estimation in this paper. Instead of directly representing hand with 3D data, a novel UV position map is introduced to represent hand pose and shape with 2D data, which maps 3D hand surface points to 2D image space. Furthermore, an encoder-decoder neural network is proposed to infer such UV position map from only single image. To train such network with the lack of ground truth training pairs, we propose a novel MANOReg module which employs MANO model as shape prior to constrain high-dimensional space of UV position map. Both quantitative and qualitative experiments demonstrate the effectiveness of our UV position map representation and MANOReg module.

Background: Aiming at free-view exploration of complicated scenes, this paper presents a method for interpolating views among multi RGB cameras. Methods: In this study, we combine the idea of cost volume, which represent 3D information, and 2D semantic segmentation of the scene, to accomplish view synthesis of complicated scenes. We use the idea of cost volume to estimate the depth and confidence map of the scene, and use a multi-layer representation and resolution of the data to optimize the view synthesis of the main object. Results: /Conclusions By applying different treatment methods on different layers of the volume, we can handle complicated scenes containing multiple persons and plentiful occlusions. We also propose the view-interpolation→multi-view reconstruction→view interpolation pipeline to iteratively optimize the result. We test our method on varying data of multi-view scenes and generate decent results. Keywords: View interpolation, Cost volume, Multi-layer processing, Multi-view reconstruction, Iterative optimization

This paper proposes a new method for fast human motion capture based on a single RGB-D sensor. By leveraging the human pose detection results for reinitializing the ICP-based sequential human motion tracking algorithm when tracking failure happens, our system achieves a highly robust and accurate human motion tracking performance even for fast motion. Moreover, the calculation and utilization of semantic tracking loss enable body-part-level motion tracking refinement, which is better than whole-body refinement. Finally, a simple yet effective post-processing method, semantic bidirectional motion blending, for single-view human motion capture is proposed to further improve the tracking accuracy, especially under severe occlusions and fast motion. The results and experiments demonstrate that the proposed method achieves highly accurate and robust human motion capture performance in a very efficient way. Applications include AR&VR, human motion analysis, movie, and gaming.

With the development of consumer light field cameras, the light field imaging has become an extensively used method for capturing the three-dimensional appearance of a scene. The depth estimation often requires a dense sampled light field in the angular domain or a high resolution in the spatial domain. However, there is an inherent trade-off between the angular and spatial resolutions of the light field. Recently, some studies for super-resolving the trade-off light field have been introduced. Rather than the conventional approaches that optimize the depth maps, these approaches focus on maximizing the quality of the super-resolved light field. In this article, we investigate how the depth estimation can benefit from these super-resolution methods. Specifically, we compare the qualities of the estimated depth using (a) the original sparse sampled light fields and the reconstructed dense sampled light fields, and (b) the original low-resolution light fields and the high-resolution light fields. Experiment results evaluate the enhanced depth maps using different super-resolution approaches.

We propose a method to automatically detect 3D poses of closely interactive humans from sparse multi-view images at one time instance. It is a challenging problem due to the strong partial occlusion and truncation between humans and no tracking process to provide priori poses information. To solve this problem, we first obtain 2D joints in every image using OpenPose and human semantic segmentation results from Mask R-CNN. With the 3D joints triangulated from multi-view 2D joints, a two-stage assembling method is proposed to select the correct 3D pose from thousands of pose seeds combined by joint semantic meanings. We further present a novel approach to minimize the interpenetration between human shapes with close interactions. Finally, we test our method on multi-view human-human interaction (MHHI) datasets. Experimental results demonstrate that our method achieves high visualized correct rate and outperforms the existing method in accuracy and real-time capability.

This paper presents a computational model to address one prominent psychological behavior of human beings to recognize images. The basic pursuit of our method can be concluded as that differences among multiple images help visual recognition. Generally speaking, we propose a statistical framework to distinguish what kind of image features capture sufficient category information and what kind of image features are common ones shared in multiple classes. Mathematically, the whole formulation is subject to a generative probabilistic model. Meanwhile, a discriminative functionality is incorporated into the model to interpret the differences among all kinds of images. The whole Bayesian formulation is solved in an Expectation-Maximization paradigm. After finding those discriminative patterns among different images, we design an image categorization algorithm to interpret how these differences help visual recognition within the bag-of-feature framework. The proposed method is verified on a variety of image categorization tasks including outdoor scene images, indoor scene images as well as the airborne SAR images from different perspectives.

3D experience and free-viewpoint navigation are expected to be two essential features of next generation television. In this paper, we present a flexible 3DTV system in which multiview video streams are captured, compressed, transmitted, and finally converted to high-quality 3D video in real time. Our system consists of an 8×8 camera array, 16 producer PCs, a streaming server, multiple clients, and several autostereoscopic displays. The whole system is implemented over IP network to provide multiple users with interactive 2D/3D switching, viewpoint control, and synthesis for dynamic scenes. In our approach, multiple video streams are first captured by a synchronized camera array. Then, we adopt a lengthened-B-field and region of interest- (ROI-) based coding scheme to guarantee a seamless view switching for each user as well as saving per-user transmission bandwidth. Finally, a convenient rendering algorithm is used to synthesize a visually pleasing result by introducing a new metric called Clarity Degree (CD). Experiments on both synthetic and real-world data have verified the feasibility, flexibility, and good performance of our system.