Search

Recognizing actions from still images has been popularly studied recently. In this paper, we model an action class as a flexible number of spatial configurations of body parts by proposing a new spatial sum–product network (SPN). First, we discover a set of parts in image collections via unsupervised learning. Then, our new spatial SPN is applied to model the spatial relationship and also the high-order correlations of parts. To learn robust networks, we further develop a hierarchical spatial SPN method, which models pairwise spatial relationship between parts inside subimages and models the correlation of subimages via extra layers of SPN. Our method is shown to be effective on two benchmark data sets. [ABSTRACT FROM PUBLISHER]

This paper presents new approaches for gait and activity analysis based on data streams of a rotating multibeam (RMB) Lidar sensor. The proposed algorithms are embedded into an integrated 4D vision and visualization system, which is able to analyze and interactively display real scenarios in natural outdoor environments with walking pedestrians. The main focus of the investigations is gait-based person reidentification during tracking and recognition of specific activity patterns, such as bending, waving, making phone calls, and checking the time looking at wristwatches. The descriptors for training and recognition are observed and extracted from realistic outdoor surveillance scenarios, where multiple pedestrians are walking in the field of interest following possibly intersecting trajectories; thus, the observations might often be affected by occlusions or background noise. Since there is no public database available for such scenarios, we created and published a new Lidar-based outdoor gait and activity data set on our website that contains point cloud sequences of 28 different persons extracted and aggregated from 35-min-long measurements. The presented results confirm that both efficient gait-based identification and activity recognition are achievable in the sparse point clouds of a single RMB Lidar sensor. After extracting the people trajectories, we synthesized a free-viewpoint video, in which moving avatar models follow the trajectories of the observed pedestrians in real time, ensuring that the leg movements of the animated avatars are synchronized with the real gait cycles observed in the Lidar stream. [ABSTRACT FROM PUBLISHER]

Prospective authors are requested to submit new, unpublished manuscripts for inclusion in the upcoming event described in this call for papers. [ABSTRACT FROM AUTHOR]

The article offers information on call for papers IEEE Trans. on circuits and systems for video technology special section on video and Language.

In this paper, a simple yet effective network pruning framework is proposed to simultaneously address the problems of pruning indicator, pruning ratio, and efficiency constraint. This paper argues that the pruning decision should depend on the convolutional weights, and thus proposes novel weight-dependent gates (W-Gates) to learn the information from filter weights and obtain binary gates to prune or keep the filters automatically. To prune the network under efficiency constraints, a switchable Efficiency Module is constructed to predict the hardware latency or FLOPs of candidate pruned networks. Combined with the proposed Efficiency Module, W-Gates can perform filter pruning in an efficiency-aware manner and achieve a compact network with a better accuracy-efficiency trade-off. We have demonstrated the effectiveness of the proposed method on ResNet34, ResNet50, and MobileNet V2, respectively achieving up to 1.33/1.28/1.1 higher Top-1 accuracy with lower hardware latency on ImageNet. Compared with state-of-the-art methods, W-Gates also achieves superior performance. [ABSTRACT FROM AUTHOR]

This paper presents the overview and rationale behind the Decoder-Side Depth Estimation (DSDE) mode of the MPEG Immersive Video (MIV) standard, using the Geometry Absent profile, for efficient compression of immersive multiview video. A MIV bitstream generated by an encoder operating in the DSDE mode does not include depth maps. It only contains the information required to reconstruct them in the client or in the cloud: decoded views and metadata. The paper explains the technical details and techniques supported by this novel MIV DSDE mode. The description additionally includes the specification on Geometry Assistance Supplemental Enhancement Information which helps to reduce the complexity of depth estimation, when performed in the cloud or at the decoder side. The depth estimation in MIV is a non-normative part of the decoding process, therefore, any method can be used to compute the depth maps. This paper lists a set of requirements for depth estimation, induced by the specific characteristics of the DSDE. The depth estimation reference software, continuously and collaboratively developed with MIV to meet these requirements, is presented in this paper. Several original experimental results are presented. The efficiency of the DSDE is compared to two MIV profiles. The combined non-transmission of depth maps and efficient coding of textures enabled by the DSDE leads to efficient compression and rendering quality improvement compared to the usual encoder-side depth estimation. Moreover, results of the first evaluation of state-of-the-art multiview depth estimators in the DSDE context, including machine learning techniques, are presented. [ABSTRACT FROM AUTHOR]

Recently, research on six-degree-of-freedom virtual reality (VR) systems based on a light field (LF) has been actively conducted. The LF-based approach is photorealistic and less prone to errors than the existing three-dimensional (3D) modeling-based approach. On the other hand, because the amount of data is very large for LF-based immersive virtual reality, it is important to manage the data efficiently. This paper makes two major contributions to data management for rendering acceleration in this context. First, the GPUs commonly used for high-speed rendering require optimization of the data transfer process between the host memory and the device memory. This paper proposes a simple but effective way to maximize data reuse by finding the proper size of a transfer unit depending on the target applications and the system capability constraints. Also, view synthesis and data transfer are performed in a pipelined manner to hide the data movement time. Second, for the LF rendering of a wide space, data management in the storage-host-GPU hierarchy is attempted for the first time. As the size of the virtual space increases, the required amount of LF data grows rapidly and exceeds the allocatable host memory capacity. Because the loading of data stored in storage is a very slow process, a system-level LF rendering structure that carefully considers the characteristics of the storage-host-GPU memory hierarchy should be designed. In this paper, progressive LF updating is proposed, and it prevents rendering stalls caused by slow storage data loading. Experimental results show that a 360-degree view with a $9000 \times 2048$ resolution can be rendered in an average of 2.57 ms from LF data with a $4096 \times 2048$ resolution, achieving performance close to 400 frames per second. [ABSTRACT FROM AUTHOR]

Since gait can be utilized to identify individuals from a far distance without their interaction and coordination, recently many gait recognition methods have been proposed. However, due to a real-world scenario of clothing changes, a degradation occurs for most of these methods. Thus in this paper, a more efficient gait recognition method is proposed to address the problem of clothing variances. First, part-based gait features are formulated from two different perspectives, i.e., the separated body parts that are more robust to clothing changes and the estimated human skeleton key-point regions. It is reasonable to formulate such features for cloth-changing gait recognition, because these two perspectives are both less vulnerable to clothing changes. Given that each feature has its own advantages and disadvantages, a more efficient gait feature is generated in this paper by assembling these two features together. Moreover, since local features are more discriminative than global features, in this paper more attention is focused on the local short-range features. Also, unlike most methods, in our method we treat the estimated key-point features as a set of word embeddings, and a transformer encoder is specifically used to learn the dependence of each correlative key-points. The robustness and effectiveness of our proposed method are certified by experiments on CASIA Gait Dataset B, and it has achieved the state-of-the-art performance on this dataset. [ABSTRACT FROM AUTHOR]

Textures contain a wealth of image information and are widely used in various fields such as computer graphics and computer vision. With the development of machine learning, the texture synthesis and generation have been greatly improved. As a very common element in everyday life, wallpapers contain a wealth of texture information, making it difficult to annotate with a simple single label. Moreover, wallpaper designers spend significant time to create different styles of wallpaper. For this purpose, this paper proposes to describe wallpaper texture images by using multi-label semantics. Based on these labels and generative adversarial networks, we present a framework for perception driven wallpaper texture generation and style transfer. In this framework, a perceptual model is trained to recognize whether the wallpapers produced by the generator network are sufficiently realistic and have the attribute designated by given perceptual description; these multi-label semantic attributes are treated as condition variables to generate wallpaper images. The generated wallpaper images can be converted to those with well-known artist styles using CycleGAN. Finally, using the aesthetic evaluation method, the generated wallpaper images are quantitatively measured. The experimental results demonstrate that the proposed method can generate wallpaper textures conforming to human aesthetics and have artistic characteristics. [ABSTRACT FROM AUTHOR]

Due to the advancement of camera hardware and machine learning techniques, video object tracking for surveillance has received noticeable attention from the computer vision research community. Object tracking and trajectory modeling have important applications in surveillance video analysis. For example, trajectory clustering, summarization or synopsis generation, and detection of anomalous or abnormal events in videos are mainly being exploited by the research community. However, barring one research work (which is almost a decade old), there is no recent review that emphasizes the use of video object trajectories, particularly in the perspective of visual surveillance. This paper presents a survey of trajectory-based surveillance applications with a focus on clustering, anomaly detection, summarization, and synopsis generation. The methods reviewed in this paper broadly summarize the abovementioned applications. The main purpose of this survey is to summarize the state-of-the-art video object trajectory analysis techniques used in the indoor and outdoor surveillance. [ABSTRACT FROM AUTHOR]

Human beings tend to learn unknown knowledge in a gradual process, from the basic to the complex. Based on this point, we propose a progressive learning method for producing service strategies according to requests, with a hierarchical priori knowledge and reinforcement learning. Service strategy aims to guide how to perform home services and takes into consideration the relationship between actions and objects in home environment. In this paper, strategy generation is regarded as a text generation problem in question answering (QA). Firstly, a hierarchical priori knowledge with service-object correlation at the bottom and action-object correlation at the top is constructed to assist the understanding on the relationship of objects and actions in service strategies. Service-object correlation guides how to select proper objects with the correct order, while action-object correlation associates actions in strategies according to selected objects. Based on the hierarchical priori knowledge, a progressive learning method is proposed to make the model produce effective strategies with a sequential cognition, from service-object correlation (objects) to action-object correlation (actions). After that, reinforcement learning is employed to enhance the progressive guidance, by designing rewards in terms of the hierarchical priori knowledge. Finally, the proposed method is tested with both comparative experiments and ablation studies, and the experimental results demonstrate the superiority in producing comprehensive and logical strategies, indicating that the progressive learning method in our paper can further improve the QA performance. [ABSTRACT FROM AUTHOR]

Digital watermarking technology and watermark attack methods are mutually reinforcing and complementary. Currently, traditional watermark attack methods are relatively mature, but these traditional attack methods will inevitably damage the visual quality of original images (OIs). Therefore, this paper proposes a covert attack method called residual dense based imperceptible watermark attack network (RD-IWAN). First, this paper designs a watermark attack residual dense network (WARDN) based on the residual dense network (RDN), which can effectively remove the watermark information in the middle and low frequency features of the watermarked image (WMI). Second, to improve the attack ability of the network, this paper innovatively proposes a progressive preprocessing method based on the information enhancement preprocessing method. Concurrently, to ensure the imperceptibility of this watermark attack method, a comprehensive loss function that combines the perceptual loss and mean square error loss (MSE) of OI and attacked watermarked image (AWMI) is designed in this study. Finally, attack experiments are designed and performed on watermarks with different embedding strengths and sizes. Experimental results show that, compared to traditional attack methods, the watermark attack method proposed in this paper exhibits stronger attack ability and higher imperceptibility. [ABSTRACT FROM AUTHOR]

Due to the lack of distinctiveness to the background, robust monocular 6DOF pose tracking of less-distinct objects remains an important open problem. In this paper, we firstly analyze the object distinctiveness in the tracking process. Then, we propose a novel contour-part model based robust monocular pose tracking method for less-distinct objects. This paper uses the traditional contour feature for pose tracking in a novel strategy called contour part model. First, the contour part model is built by segmenting the projected contour rendered from the 3D model into contour segments of a certain length adaptively according to the Shi-Tomasi cornerness scores. Then, the correspondence is detected in the input image for each contour part by gradient orientation based template matching. Finally, pose tracking is achieved by solving the PnP problem. Experiment results on semi-synthetic and real images show that the proposed method performs better than the existing methods when pose tracking less-distinct objects and shows great robustness toward interference. Additionally, we combine the edge feature and the regional feature in a simple strategy. The overall performance of the fused method is boosted according to the experimental results on the RBOT dataset. The fact shows that the fusion of the edge and regional features has great potential for improving tracking performance. [ABSTRACT FROM AUTHOR]

The new generation of the high-efficiency video coding (HEVC) video coding standard has improved compression performance and brought considerable coding complexity. Therefore, reducing the perception redundancy of video to obtain a better compression effect is the new direction of video development at present. In this paper, the HEVC is improved and enhanced from the aspects of a video saliency algorithm based on an attention mechanism and a video compression algorithm based on perception priority. In terms of video saliency, this paper proposes a spatial saliency algorithm based on a convolutional neural network and a temporal saliency algorithm based on motion vector. The saliency algorithm can combine the motion estimation results of each block during the HEVC compression on the basis of convolutional neural network and carry out adaptive dynamic fusion of the two, to complete the saliency map of the input video. In the aspect of the video compression algorithm with perception priority, this paper proposes a more flexible QP selection method, which selects its corresponding QP according to the saliency value of CU. At the same time, we propose a new rate-distortion optimization algorithm, which integrates the current block’s saliency feature into the traditional rate-distortion calculation method, to guide the allocation of bits and achieve the purpose of perception priority. The experimental results proved the superiority of the proposed method over the state-of-the-art perceptual coding algorithms in terms of saliency detection and perceptual compression quality. [ABSTRACT FROM AUTHOR]

Co-salient object detection (CoSOD) aims to detect common salient objects sharing the same attributes in an image group. The key issue of CoSOD is how to model the inter-saliency relations within an image group. The major limitation of previous methods is that they pre-define the group-to-one relations within an image group. In this paper, we propose a new concept of structural inter-saliency relations and solve the CoSOD with deep reinforcement learning framework. Firstly, we design a semantic relation graph (SRG) to model the structural inter-saliency relations. Then the feature selecting agent (FS-agent) aims to select the informative features, which can help the SRG effectively model structural inter-saliency relations. Finally, relation updating agent (RU-agent) progressively updates the SRG to focus on the co-salient relations like human decision-making process. Extensive experiments on co-saliency datasets show that because of well modeling inter-saliency relations in image group, our proposed method achieves superior performance compared to the state-of-the-art methods. We hope that this paper can motivate future research for visual co-analysis tasks. [ABSTRACT FROM AUTHOR]

With the rapid development of screen content video applications, screen content coding (SCC) is urgently needed to be used in commercial codecs. However, the extra encoding complexity introduced by the new SCC tools has posed a great challenge for its practical deployment. In this paper, motivated by our observations that there should be a fine-grained mapping between image content and candidate modes, we propose a two-step fast mode decision method to reduce the encoding complexity. First, we propose to use a convolution neural network (CNN) to automatically extract useful features for fine-grained content classification. Second, we build a precise and concise mapping from CUs to candidate modes by simultaneously considering CU content type, CU size, and mode complexity. Note that the spatial correlations between neighboring CUs and current CU are also utilized in candidate modes derivation. In addition to the two-step fast mode decision method, a content-aware early termination algorithm is further proposed to reduce the encoding complexity. Extensive experiments demonstrate that our method achieves better performance compared with state-of-the-art ones, with 50.13% total encoding complexity reduction and only 0.92% BD-rate increase. [ABSTRACT FROM AUTHOR]

In this paper, the problem of human action recognition using 3D reconstruction data is deeply investigated. 3D reconstruction techniques are employed for addressing two of the most challenging issues related to human action recognition in the general case, namely view variance (i.e., when the same action is observed from different viewpoints) and the presence of (self-) occlusions (i.e., when for a given point of view a body part of an individual conceals another body part of the same or another subject). The main contributions of this paper are summarized as follows. The first one is a detailed examination of the use of 3D reconstruction data for performing human action recognition. The latter includes the introduction of appropriate local/global flow/shape descriptors, extensive experiments in challenging publicly available datasets and exhaustive comparisons with state-of-art approaches. The second one is a new local-level 3D flow descriptor, which incorporates spatial and surface information in the flow representation and efficiently handles the problem of defining 3D orientation at every local neighborhood. The third one is a new global-level 3D flow descriptor that efficiently encodes the global motion characteristics in a compact way. The fourth one is a novel global temporal-shape descriptor that extends the notion of 3D shape descriptions for action recognition, by incorporating the temporal dimension. The proposed descriptor efficiently addresses the inherent problems of temporal alignment and compact representation, while also being robust in the presence of noise (compared with similar tracking-based methods of the literature). Overall, this paper significantly improves the state-of-art performance and introduces new research directions in the field of 3D action recognition, following the recent development and wide-spread use of portable, affordable, high-quality and accurate motion capturing devices (e.g., Microsoft Kinect). [ABSTRACT FROM AUTHOR]

The article offers information on miscellaneous topics including papers and research published by the editorial board and their submission; writing style for manuscripts, illustrations; and information regarding the copyright policies of the journal.

Over the past several years, we have witnessed impressive progress in the field of learned image compression. Recent learned image codecs are commonly based on autoencoders, that first encode an image into low-dimensional latent representations and then decode them for reconstruction purposes. To capture spatial dependencies in the latent space, prior works exploit hyperprior and spatial context model to build an entropy model, which estimates the bit-rate for end-to-end rate-distortion optimization. However, such an entropy model is suboptimal from two aspects: (1) It fails to capture global-scope spatial correlations among the latents. (2) Cross-channel relationships of the latents remain unexplored. In this paper, we propose the concept of separate entropy coding to leverage a serial decoding process for causal contextual entropy prediction in the latent space. A causal context model is proposed that separates the latents across channels and makes use of channel-wise relationships to generate highly informative adjacent contexts. Furthermore, we propose a causal global prediction model to find global reference points for accurate predictions of undecoded points. Both these two models facilitate entropy estimation without the transmission of overhead. In addition, we further adopt a new group-separated attention module to build more powerful transform networks. Experimental results demonstrate that our full image compression model outperforms standard VVC/H.266 codec on Kodak dataset in terms of both PSNR and MS-SSIM, yielding the state-of-the-art rate-distortion performance. [ABSTRACT FROM AUTHOR]

Gaming video streaming services have grown tremendously in the past few years, with higher resolutions, higher frame rates and HDR gaming videos getting increasingly adopted among the gaming community. Since gaming content as such is different from non-gaming content, it is imperative to evaluate the performance of the existing encoders to help understand the bandwidth requirements of such services, as well as further improve the compression efficiency of such encoders. Towards this end, we present in this paper GamingHDRVideoSET, a dataset consisting of eighteen 10-bit UHD-HDR gaming videos and encoded video sequences using four different codecs, together with their objective evaluation results. Additionally, the paper discusses the codec compression efficiency of most widely used practical encoders, i.e., x264 (H.264/AVC), x265 (H.265/HEVC) and libvpx (VP9), as well the recently proposed encoder libaom (AV1), on 10-bit, UHD-HDR content gaming content. Our results show that the latest compression standard AV1 results in the best compression efficiency, followed by HEVC, H.264, and VP9. [ABSTRACT FROM AUTHOR]

The world of sports intrinsically involves fast and complex events that are difficult for coaches, trainers and players to analyze, and also for audiences to follow. In fast paced team sports such as soccer, keeping track of all the players and analyzing their performance after every match are very challenging. Current scenarios for identifying the best talents in soccer involve word-of-mouth and coaches/recruiters scouring through hours of manually annotated videos. This is a very expensive and laborious process and also biased by the nature of the recruiters. To alleviate these problems, this paper proposes an automated system that can detect, track, classify the teams of multiple players and identify the player controlling the ball in a video. The system generates three very important tactical statistics for a player: 1) duration of ball possession, 2) number of successful passes and 3) number of successful steals. This is done by training Convolutional Neural Networks (CNNs) to (a) localize and track the players on the field, (b) classify the team of a detected player, (c) identify the player controlling the ball and (d) pooling all the information extracted from (a), (b), and (c) to generate the statistics of players. To overcome the problem that the features learned from specific soccer matches do not necessarily generalize across different soccer matches, the paper proposes minimal amount of match-specific annotation and data augmentation, using a variant of Deep Convolutional Generative Adversarial Networks (DCGAN) to improve the accuracy. Experimental results and ablation studies show that the proposed approach outperforms the state-of-the-art approaches in terms of accuracy and processing speed. [ABSTRACT FROM AUTHOR]

Ill-posed problems are very common in many image processing and computer vision tasks. To solve such problems, a regularization must be imposed. In this paper, we argue why mean curvature is a good regularization for these tasks. From a geometry point of view, we show that minimizing mean curvature is to assume that the ground truth is a piece-wise minimal surface. From a statistics point of view, we show that the mean curvature from natural images is sparse. From an optimization point of view, we show that the gradient of mean curvature regularization can be numerically approximated by very simple filters. This fact significantly simplifies the optimization procedure. Thus, the link between these filters and the gradient of mean curvature regularization is established in this paper. In summary, mean curvature is a proper regularization for various ill-posed problems in image processing and computer vision. [ABSTRACT FROM AUTHOR]

This paper proposes a low-cost and portable polycamera system and accompanying methods for capturing and synthesizing stereoscopic 360° panoramas. The polycamera consists of only four cameras with fisheye lenses. Synthesizing panoramas from only four views is challenging because the cameras view very differently and the captured images have significant distortions and color degradation including vignetting, contrast loss, and blurriness. For coping with these challenges, this paper proposes methods for rectifying the polyview images, estimating depth of the scene, and synthesizing stereoscopic panoramas. The proposed camera is compact in size, light in weight, and inexpensive. The proposed methods allow the synthesis of visually pleasing stereoscopic 360° panoramas using the images captured with the proposed polycamera. We have built a prototype of the polycamera and tested it on a set of scenes with different characteristics of depth ranges and depth variations. The experiments show that the proposed camera and methods are effective in generating stereoscopic 360° panoramas that can be viewed on popular virtual reality displays. [ABSTRACT FROM AUTHOR]

This paper studies the composition in portrait paintings and develops an algorithm to improve the composition of portrait photographs based on example portrait paintings. A study of portrait paintings shows that the placement of the face and the figure is pose-related. Based on this observation, this paper develops an algorithm to improve the composition of a portrait photograph by learning the placement of the face and the figure from an example portrait painting. This example portrait painting is selected based on the similarity of its figure pose to that of the input photograph. This similarity measure is modeled as a graph matching problem. Finally, space cropping is performed using an optimization function to assign a similar location for each body part of the figure in the photograph with that of the figure in the example portrait painting. The experimental results demonstrate the effectiveness of the proposed method. A user study shows that the proposed pose-based composition improvement is preferred more than rule-based methods and learning-based methods. [ABSTRACT FROM AUTHOR]

Encoding multiple videos in parallel and transmitting them as one joint stream over a limited bandwidth have become a popular strategy for broadcasting, which brings an opportunity to allocate different bitrate for each sequence to meet different demands. In this paper, considering visual experience for human beings, we propose a joint rate allocation scheme aims to reach an equal visual quality among all sequences by minimizing the distortion variance of all the sequences (denoted as minVAR problems). Existing methods assigned bits directly in proportion to their complexity measures and we named them as complexity based allocation scheme (CAS) methods. CAS methods rely on the accuracy of the complexity measures which can hardly be improved under limited computing resources. Also complexities may not be directly related to the distortions. To address these problems, we present a novel joint rate-distortion (R-D) based allocation scheme (RDAS) in this paper. Our proposed scheme can fit for different R-D models and in our method we model the R-D relationship with a hyperbolic function (RDAS-H). We also derive a closed-form solution of RDAS-H by a proposed joint R-D relationship. We integrated the RDAS-H method in high efficiency video coding reference software HM16.0. Experimental results demonstrate that our RDAS-H saves 75.29% variance on average over the related CAS-based method, where we apply both low delay and random access configurations with four different overall bandwidths for all classes recommended by the Joint Collaborative Team on Video Coding. Besides, RDAS-H also saves 36.62% variance on average over our previous method. The proposed RDAS-H method improves the performance significantly while requiring negligible computational cost. [ABSTRACT FROM AUTHOR]

The active contour model is a widely used technique for automatic object contour extraction. Existing methods based on this model can perform with high accuracy, even in the case of complex contours, but challenging issues remain, like the need for precise contour initialization for high curvature boundary segments or the handling of cluttered backgrounds. To deal with such issues, this paper presents a salient object extraction method, the first step of which is the introduction of an improved edge map that incorporates edge direction as a feature. The direction information in the small neighborhoods of image feature points is extracted, and the images’ prominent orientations are defined for direction-selective edge extraction. Using such improved edge information, we provide a highly accurate shape contour representation, which we also combine with texture features. The principle of the paper is to interpret an object as the fusion of its components: its extracted contour and its inner texture. Our goal in fusing textural and structural information is twofold: it is applied for automatic contour initialization, and it is also used to establish an improved external force field. This fusion then produces highly accurate salient object extractions. We performed extensive evaluations, which confirm that the presented object extraction method outperforms parametric active contour models and achieves higher efficiency than the majority of the evaluated automatic saliency methods. [ABSTRACT FROM AUTHOR]

Video coding standards are primarily designed for efficient lossy compression, but it is also desirable to support efficient lossless compression within video coding standards using small modifications to the lossy coding architecture. A simple approach is to skip transform and quantization, and simply entropy code the prediction residual. However, this approach is inefficient at compression. A more efficient and popular approach is to skip transform and quantization but also process the residual block in some modes with differential pulse code modulation (DPCM), along the horizontal or vertical direction, prior to entropy coding. This paper explores an alternative approach based on processing the residual block with integer-to-integer (i2i) transforms. I2i transforms can map integer pixels to integer transform coefficients without increasing the dynamic range and can be used for lossless compression. We focus on lossless intra coding and develop novel i2i approximations of the odd type-3 discrete sine transform (ODST-3). Experimental results with the high efficiency video coding (HEVC) reference software show that when the developed i2i approximations of the ODST-3 are used along the DPCM method of HEVC, an average 2.7% improvement of lossless intra frame compression efficiency is achieved over HEVC version 2, which uses only the DPCM method, without a significant increase in computational complexity. [ABSTRACT FROM AUTHOR]

In active and assisted living environments, a major service that can be provided is the automated assessment of elderly people’s well-being. Therefore, activity recognition is required to detect what types of help disabled persons need to support them in their daily life activities. Unfortunately, it is still a difficult task to estimate the size of the required window for online sensor data streams to recognize a specific activity, especially when new sensor events are recorded. This paper proposes a windowing algorithm, which presents promising results to recognize complex human activities for multi-resident homes. The approach is based on the analysis of the sensor data to identify the best fitting sensors that should be considered in a specified window. Moreover, the second part of this paper proposes a set of different statistical spatio-temporal features to recognize human activities. In order to check the overall performance, this approach is tested using the CASAS data set and artificially generated laboratory data using our HBMS simulator. The results show high performance based on different evaluation metrics compared to other approaches. We believe that the proposed windowing approach provides a good approximation of the required window size in order to robustly detect human activities in comparison to other windowing approaches. [ABSTRACT FROM AUTHOR]

Pedestrian detection has remained an important research topic in both the computer vision and multimedia communities because of its importance in practical applications, such as driving assistance and video surveillance. Existing methods compare the response score with a fixed threshold to determine whether a candidate region contains pedestrians and produce dissatisfactory results that contain either missed detections or false detections, which are difficult to balance. This situation has a serious impact under the condition of variable scale. This paper investigates the functional relationship between the scores and scales of pedestrians. By designing experiments with multiple scales, we have found a discriminant surface in the score scale space. Pedestrians can be distinguished at various scale levels according to their locations on the discriminant surface. The proposed approach is evaluated using four challenging pedestrian detection datasets, including Caltech, INRIA, ETH, and KITTI, and the superior experimental results are achieved when compared with baseline methods. [ABSTRACT FROM AUTHOR]

This paper addresses the issue of detecting the use of information hiding at anti-copying 2D barcodes. Prior hidden information detection schemes have their roots in either heuristic-based or Machine Learning (ML). However, prior heuristics-based schemes lack a rigorous theoretical analysis. Prior ML-based information schemes lack robustness because a printed 2D barcode is very much environmentally dependent. Thus, an information hiding detection scheme trained in one environment often does not work well in another environment. In this paper, we propose two hidden information detection schemes for existing anti-copying 2D barcodes. The first scheme directly uses the pixel distance to detect the use of an information hiding scheme in a 2D barcode, referred to as the Pixel Distance Based Detection (PDBD) scheme. The second scheme first calculates the variance of raw signal and the covariance between the recovered and raw signals, and then based on the variance results, detects the use of information hiding scheme in a 2D barcode, referred to as the Pixel Variance Based Detection (PVBD) scheme. Moreover, we design advanced Illegitimately-Copying (IC) attacks to evaluate the security of two existing anti-copying 2D barcodes. We conduct extensive performance comparisons among the proposed schemes and prior schemes under different capturing devices, e.g., different scanners or camera phones. Our experimental results show that the PVBD scheme can correctly detect the existence of hidden information at both the 2LQR code and the LCAC 2D barcode. Moreover, the successful attacking probability of the proposed IC attacks achieves 0.6538 for the 2LQR code and 1 for the LCAC 2D barcode. [ABSTRACT FROM AUTHOR]

Most of the existing single object trackers track the target in a unitary local search window, making them particularly vulnerable to challenging factors such as heavy occlusions and out-of-view movements. Despite the attempts to further incorporate global search, prevailing mechanisms that cooperate local and global search are relatively static, thus are still sub-optimal for improving tracking performance. By further studying the local and global search results, we raise a question: can we allow more dynamics for cooperating both results? In this paper, we propose to introduce more dynamics by devising a dynamic attention-guided multi-trajectory tracking strategy. In particular, we construct dynamic appearance model that contains multiple target templates, each of which provides its own attention for locating the target in the new frame. Guided by different attention, we maintain diversified tracking results for the target to build multi-trajectory tracking history, allowing more candidates to represent the true target trajectory. After spanning the whole sequence, we introduce a multi-trajectory selection network to find the best trajectory that deliver improved tracking performance. Extensive experimental results show that our proposed tracking strategy achieves compelling performance on various large-scale tracking benchmarks. The project page of this paper can be found at https://sites.google.com/view/mt-track/. [ABSTRACT FROM AUTHOR]

Despite significant progress in human behavior analysis over the past few years, most of today’s state-of-the-art algorithms focus on analyzing individual behavior in a simple environment monitored by a single camera. Recently, the widespread availability of cameras and a growing need for public safety have shifted the attention of researchers in video surveillance from individual behavior analysis to group and crowd behavior analysis in multicamera networks. Group behavior analysis provides a novel level for describing events, which are semantically more meaningful, highlighting barely visible relational connections among people. Crowd behavior analysis can also be used for anomaly detection such as panic scenarios, dangerous situations, and illegal behaviors in public spaces. [ABSTRACT FROM PUBLISHER]

Automated scene analysis has been a topic of great interest in computer vision and cognitive science. Recently, with the growth of crowd phenomena in the real world, crowded scene analysis has attracted much attention. However, the visual occlusions and ambiguities in crowded scenes, as well as the complex behaviors and scene semantics, make the analysis a challenging task. In the past few years, an increasing number of works on the crowded scene analysis have been reported, which covered different aspects including crowd motion pattern learning, crowd behavior and activity analyses, and anomaly detection in crowds. This paper surveys the state-of-the-art techniques on this topic. We first provide the background knowledge and the available features related to crowded scenes. Then, existing models, popular algorithms, evaluation protocols, and system performance are provided corresponding to different aspects of the crowded scene analysis. We also outline the available datasets for performance evaluation. Finally, some research problems and promising future directions are presented with discussions. [ABSTRACT FROM PUBLISHER]

In this paper, a novel skeleton-based approach to human time-varying mesh (H-TVM) compression is presented. The topic of TVM compression is new and has many challenges, such as handling the lack of obvious mapping of vertices across frames and handling the variable connectivity across frames, while maintaining efficiency, which are the most important ones. Very few works exist in the literature, while not all of the challenges have been addressed yet. In addition, developing an efficient and real-time solution, handling the above, obviously is a difficult task. We attempt to address the H-TVM compression problem inspired from video coding using different types of frames and trying to efficiently remove inter-frame geometric redundancy utilizing the recent advances in human skeleton tracking. The overall approach focuses on compression efficiency, low distortion, and low computation time enabling for real-time transmission of H-TVMs. It efficiently compresses geometry and vertex attributes of TVMs. In addition, this paper is the first to provide an efficient method for connectivity coding of TVMs, by introducing a modification to the state-of-the-art MPEG-4 TFAN algorithm. Experiments are conducted in the MPEG-3DGC TVM database. The method outperforms the state-of-the-art standardized static mesh coder MPEG-4 TFAN at low bit-rates, while remaining competent at high bit-rates. It gives a practical proof of concept that in the combined problem of geometry, connectivity, and vertex attribute coding of TVMs, efficient inter-frame redundancy removal is possible, establishing ground for further improvements. Finally, this paper proposes a method for motion-based coding of H-TVMs that can further enhance the overall experience when H-TVM compression is used in a tele-immersion scenario. [ABSTRACT FROM AUTHOR]

In this paper, a low-complexity color interpolation algorithm is proposed for the very-large-scale integration (VLSI) implementation in real-time applications. The proposed novel algorithm consists of an edge detector, an anisotropic weighting model, and a filter-based compensator. The anisotropic weighting model is designed to catch more information in horizontal than vertical directions. The filter-based compensation methodology includes a Laplacian and spatial sharpening filters, which are developed to improve the edge information and reduce the blurring effect. In addition, the hardware cost was successfully reduced by hardware sharing and reconfigurable design techniques. The VLSI architecture of the proposed design achieves 200 MHz with 5.2-K gate counts, and its core area is 64~236~\mu m ^{\mathrm {\mathbf {2}}} synthesized by a 0.18- \mu $ m CMOS process. Compared with the previous low-complexity techniques, this paper not only reduces gate counts or power consumption by more than 8% or 91.7%, respectively, but also improves the average color peak signal-to-noise ratio quality by more than 1.6 dB. [ABSTRACT FROM PUBLISHER]

The paper proposes a novel block-wise prediction paradigm based on Convolutional Neural Networks (CNNs) for lossless video coding. A deep neural network model which follows a multi-resolution design is employed for block-wise prediction. Several contributions are proposed to improve neural network training. A first contribution proposes a novel loss function formulation for an efficient network training based on a new approach for patch selection. Another contribution consists in replacing all HEVC-based angular intra-prediction modes with a CNN-based intra-prediction method, where each angular prediction mode is complemented by a CNN-based prediction mode using a specifically trained model. Another contribution consists in an efficient adaptation of the CNN-based intra-prediction residual for lossless video coding. Experimental results on standard test sequences show that the proposed coding system outperforms the HEVC standard with an average bitrate improvement of around 5%. To our knowledge, the paper is the first to replace all the traditional HEVC-based angular intra-prediction modes with an intra-prediction method based on modern Machine Learning techniques for lossless video coding applications. [ABSTRACT FROM AUTHOR]

This paper describes a video coding technology proposal submitted by MediaTek in April 2018 in response to a joint call for proposals (CfP) issued by ITU-T VCEG and ISO/IEC MPEG. The proposal uses the conventional block-based hybrid coding approach with a breakthrough design of block structure, substantial improvements of coding tools, and new inventions of loop filtering techniques. First, the block structure is designed using a highly flexible partitioning scheme, where each coding tree unit in a picture is split into multiple coding units (CUs) by a recursive quaternary tree followed by a recursive binary-ternary tree. The prediction unit (PU) and transform unit (TU) concepts are unified with the CU concept, i.e., PU and TU are basically the same as CU. Moreover, inter prediction, intra prediction, transform, quantization, in-loop filtering, and entropy coding are all improved. Finally, new loop filtering techniques are invented, where the proposed convolutional neural network loop filtering is of particular interest. The proposed video codec achieves 43.81% average bit rate savings based on objective measures compared to the High Efficiency Video Coding (HEVC) anchors and is among the best-performing CfP responses both objectively and subjectively. It has been partially adopted into the working draft of the Versatile Video Coding (VVC) standard. [ABSTRACT FROM AUTHOR]

With the increasing popularity of mobile phones and tablets, the explosive growth of query-by-capture applications calls for a compact representation of the query image feature. Compact descriptors for visual search (CDVS) is a recently released standard from the ISO/IEC moving pictures experts group, which achieves state-of-the-art performance in the context of image retrieval applications. However, they did not consider the matching characteristics in local space in a large-scale database, which might deteriorate the performance. In this paper, we propose a more compact representation with scale invariant feature transform (SIFT) descriptors for the visual query based on Grassmann manifold. Due to the drastic variations in image content, it is not sufficient to capture all the information using a single transform. To achieve more efficient representations, a SIFT manifold partition tree (SMPT) is initially constructed to divide the large dataset into small groups at multiple scales, which aims at capturing more discriminative information. Grassmann manifold is then applied to prune the SMPT and search for the most distinctive transforms. The experimental results demonstrate that the proposed framework achieves state-of-the-art performance on the standard benchmark CDVS dataset. [ABSTRACT FROM AUTHOR]

Image classification needs to consider the existence of image degradations in practice. Although degraded images have various levels of degradation, the degradation levels are usually unknown. This paper proposes a convolutional neural network to classify degraded images by using a restoration network and an ensemble learning. The proposed network can automatically infer ensemble weights by using estimated degradation levels of degraded images and features of restored images, where the degradation levels are estimated internally. The proposed network is mainly discussed with JPEG distortion, while degradations of both Gaussian noise and blurring are also examined. We demonstrate that the proposed network can classify degraded images over various levels of degradation. This paper also reveals how the image-quality of training data for a classification network affects the classification performance of degraded images. [ABSTRACT FROM AUTHOR]

In this paper, we study a simplified affine motion model-based coding framework to overcome the limitation of a translational motion model and maintain low-computational complexity. The proposed framework mainly has three key contributions. First, we propose to reduce the number of affine motion parameters from 6 to 4. The proposed four-parameter affine motion model can not only handle most of the complex motions in natural videos, but also save the bits for two parameters. Second, to efficiently encode the affine motion parameters, we propose two motion prediction modes, i.e., an advanced affine motion vector prediction scheme combined with a gradient-based fast affine motion estimation algorithm and an affine model merge scheme, where the latter attempts to reuse the affine motion parameters (instead of the motion vectors) of neighboring blocks. Third, we propose two fast affine motion compensation algorithms. One is the one-step sub-pixel interpolation that reduces the computations of each interpolation. The other is the interpolation-precision-based adaptive block size motion compensation that performs motion compensation at the block level rather than the pixel level to reduce the number of interpolation. Our proposed techniques have been implemented based on the state-of-the-art high-efficiency video coding standard, and the experimental results show that the proposed techniques altogether achieve, on average, 11.1% and 19.3% bits saving for random access and low-delay configurations, respectively, on typical video sequences that have rich rotation or zooming motions. Meanwhile, the computational complexity increases of both the encoder and the decoder are within an acceptable range. [ABSTRACT FROM AUTHOR]

This paper presents a DC coefficient estimation algorithm for intra-predicted residual blocks in the High-Efficiency Video Coding (HEVC) standard. Discarding the DC coefficient directly in each transform block leads to substantial bit-saving, but at the same time produces strong discontinuities between neighboring blocks. To overcome this problem, we propose an estimation algorithm for the DC coefficient, which solves an optimal offset in a closed-form to recover the corresponding block edges. Then, we embed this algorithm into HEVC in its rate-distortion optimized quantization and sign bit hiding steps. Furthermore, a flag is signaled to decide whether the DC estimation strategy is used for each transform block. Test results under the common test condition show that our algorithm achieves 1.5% and 1.6% BD-rate reduction on average for luma and chroma, respectively, under all intra configuration. In the meantime, our simulation results show that both encoding time and decoding time increase only slightly (about 10%, without any special optimization on programming our proposed algorithm). When testing the proposed DC estimation algorithm on inter coding configurations, including low delay with P pictures, low delay with B pictures, and random access, we can also achieve 0.5%–1.1% bit-rate savings on average, while nearly no extra encoding and decoding time is needed. [ABSTRACT FROM AUTHOR]

Screen content coding plays an important role in many applications. Conventional reference picture management (RPM) strategies developed for natural content may not work well for screen content. This is because many regions in screen content remain static for a long time, causing a lot of repetitive contents to stay in the decoded picture buffer. The repetitive contents are not conducive to inter prediction, but still occupy valuable memory. This paper proposes a diversity-based RPM scheme for screen content coding. The concept of diversity is introduced for the reference picture set (RPS) to help formulate the RPM problem. By maximizing the diversity of RPS, more potentially better predictions are provided. Better compression performance can then be achieved. Meanwhile, the proposed scheme is nonnormative and compatible with existing video coding standards, such as High Efficiency Video Coding. The experimental results show that, for low delay screen content coding, the bit saving of the proposed scheme is 4.9% on average and up to 13.7%, without increasing encoding time. [ABSTRACT FROM AUTHOR]

This paper presents a subband-decomposed image-enhancement algorithm that can preserve the luminance and contrast levels in power-controllable thin-film-transistor liquid-crystal display devices. The proposed algorithm consists of three steps. First, an input image is decomposed to multiscale subbands via subband decomposition. Second, luminance compensation and contrast preservation are achieved simultaneously by adjusting the gain of each subband properly. Third, additional power consumption caused by excessive boosting, while coping with the local dimming through specific clipping, is prevented. The experimental results show that the proposed algorithm provides more natural details and contrast, while preserving more similarities with the original, than the existing methods. Even in terms of the peak signal-to-noise-ratio and the structural similarity index, the proposed algorithm has 27.8% and 3.8% higher values compared to a state-of-the-art method. The real-time operation of the proposed algorithm was verified through intensive optimization and the Compute Unified Device Architecture implementation. In addition, the power-reduction effect of the proposed algorithm on commercial monitors was investigated experimentally. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel annotation placement approach for revealing information about foreground objects in surveillance videos. To arrange positions of annotations, spatiotemporal coherence between annotations and foreground objects is applied. The annotation placement problem is formulated as an optimization problem with respect to spatiotemporal coherence of annotations and foreground objects. The optimization problem is effectively solved using Markov random fields. To the best of our knowledge, this paper is the first work that discusses and solves the annotation placement problem for surveillance videos by considering the relationships between annotations and foreground objects with trajectories. As shown in the experiments, the proposed approach can arrange annotations based on the moving trajectories of foreground objects and prevent the occlusions between different annotations and foreground objects. It also achieves better quantitative and qualitative results compared with state-of-the-art approaches. [ABSTRACT FROM PUBLISHER]

Rate control typically involves two steps: bit allocation and bitrate control. The bit allocation step can be implemented in various fashions depending on how many levels of allocation are desired and whether or not an optimal rate–distortion (R-D) performance is pursued. The bitrate control step has a simple aim in achieving the target bitrate as precisely as possible. In our recent research, we have developed a \lambda -domain rate control algorithm that is capable of controlling the bitrate precisely for High Efficiency Video Coding (HEVC). The initial research showed that the bitrate control in the \lambda -domain can be more precise than the conventional schemes. However, the simple bit allocation scheme adopted in this initial research is unable to achieve an optimal R-D performance reflecting the inherent R-D characteristics governed by the video content. In order to achieve an optimal R-D performance, the bit allocation algorithms need to be developed taking into account the video content of a given sequence. The key issue in deriving the video-content-guided optimal bit allocation algorithm is to build a suitable R-D model to characterize the R-D behavior of the video content. In this paper, to complement the R- \lambda model developed in our initial work, a D- \lambda model is properly constructed to complete a comprehensive framework of \lambda -domain R-D analysis. Based on this comprehensive \lambda -domain R-D analysis framework, a suite of optimal bit allocation algorithms are developed. In particular, we design both picture-level and basic-unit-level bit allocation algorithms based on the fundamental R-D optimization theory to take full advantage of the content-guided principles. The proposed algorithms are implemented in HEVC reference software, and the experimental results demonstrate that they can achieve an obvious R-D performance improvement with a smaller bitrate control error. The proposed bit allocation algorithms have already been adopted by the Joint Collaborative Team on Video Coding and integrated into the HEVC reference software. [ABSTRACT FROM PUBLISHER]

This paper presents a method to improve the traditional mixture of deformable part models (MDPM) method from the learning perspective. First, an object part configuration learning algorithm based on group sparsity constraint is introduced to automatically discover the object part number, size, and location. The algorithm imposes two additional regularization terms in addition to the standard hinge loss function. The first term focuses on automatic part selection and the second term focuses on automatic part placement. Second, this paper introduces an improved MDPM training framework. The framework applies a learned transformation to normalize the prediction score from each individual deformable part model (DPM) into a pseudo probability such that the partition of the entire object appearance feature space becomes less sensitive to the prior distributions of different DPMs. Finally, the two proposed improvements are combined and formulated under the expectation-maximization framework. We evaluate our method mainly using the PASCAL VOC2007 and VOC2010 detection benchmarks and show that the proposed learning algorithms could increase the detection mean AP score by 2.4% and 0.9%, respectively, on these two data sets when using the proposed part selection method and the training algorithm. We also present further in-depth analysis of the proposed algorithm in the experiments. [ABSTRACT FROM PUBLISHER]

The paper proposes a method to effectively fuse multi-exposure inputs and generate high-quality high dynamic range (HDR) images with unpaired datasets. Deep learning-based HDR image generation methods rely heavily on paired datasets. The ground truth images play a leading role in generating reasonable HDR images. Datasets without ground truth are hard to be applied to train deep neural networks. Recently, Generative Adversarial Networks (GAN) have demonstrated their potentials of translating images from source domain $X$ to target domain $Y$ in the absence of paired examples. In this paper, we propose a GAN-based network for solving such problems while generating enjoyable HDR results, named UPHDR-GAN. The proposed method relaxes the constraint of the paired dataset and learns the mapping from the LDR domain to the HDR domain. Although the pair data are missing, UPHDR-GAN can properly handle the ghosting artifacts caused by moving objects or misalignments with the help of the modified GAN loss, the improved discriminator network and the useful initialization phase. The proposed method preserves the details of important regions and improves the total image perceptual quality. Qualitative and quantitative comparisons against the representative methods demonstrate the superiority of the proposed UPHDR-GAN. [ABSTRACT FROM AUTHOR]

The Low Complexity Enhancement Video Coding (LCEVC) specification is a recent standard approved by the ISO/IEC JTC 1/SC 29/WG04 (MPEG) Video Coding. The main goal of LCEVC is to provide a standalone toolset for the enhancement of any other existing codec. It works on top of other coding schemes, resulting in a multi-layer video coding technology, but unlike existing scalable video codecs, adds enhancement layers completely independent from the base video. The LCEVC technology takes as input the decoded video at lower resolution and adds up to two enhancement sub-layers of residuals encoded with specialized low-complexity coding tools, such as simple temporal prediction, frequency transform, quantization, and entropy encoding. This paper provides an overview of the main features of the LCEVC standard: high compression efficiency, low complexity, minimized requirements of memory and processing power. [ABSTRACT FROM AUTHOR]

This paper presents a new objective quality metric that was adapted to the complex characteristics of immersive video (IV) which is prone to errors caused by processing and compression of multiple input views and virtual view synthesis. The proposed metric, IV-PSNR, contains two techniques that allow for the evaluation of quality loss for typical immersive video distortions: corresponding pixel shift and global component difference. The performed experiments compared the proposal with 31 state-of-the-art quality metrics, showing their performance in the assessment of quality in immersive video coding and processing, and in other applications, using commonly used image quality assessment databases– TID2013 and CVIQ. As presented, IV-PSNR outperforms other metrics in immersive video applications and still can be efficiently used in the evaluation of different images and videos. Moreover, basing the metric on the calculation of PSNR allowed the computational complexity to remain low. Publicly available, efficient implementation of IV-PSNR software was provided by the authors of this paper and is used by ISO/IEC MPEG for evaluation and research on the upcoming MPEG Immersive video (MIV) coding standard. [ABSTRACT FROM AUTHOR]

Learning the human depth localization in camera coordinate space plays a crucial role in understanding the behavior and activities of multi-person in 3D scenes. However, existing monocular-based methods rarely combine the global image features and the human body-parts features effectively, resulting in a large gap from the actual location in some cases, e.g., the special body-sized persons and mutual occlusion between humans in the image. This paper presents a novel Robust 3D Human Localization (R3HL) network consisting of two stages: global depth awareness and body-parts depth awareness, to significantly improve the robustness and accuracy of the 3D location. In the first stage, the front-back and far-near relationship estimation module based on multi-person are proposed to make the network extract depth features from the global perspective. In the second stage, the network focuses on the target human. We propose a Pose-guided Multi-person Repulsion (PMR) module to enhance the target human’s features and reduce the interference features produced by the background and other people. In addition, an Adaptive Body-parts Attention (ABA) module is designed to assign different feature weights to each joint. Finally, the human’s absolute depth is obtained through global pooling and fully connected layers. The experimental results show that the attention from the whole image to a single person helps find the absolute location of different body-sized and poses people from diverse scenes. Our method can achieve better performance than other state-of-the-art methods on both indoor and outdoor 3D multi-person datasets. [ABSTRACT FROM AUTHOR]