Search

An energy-efficient convolutional neural network (CNN) accelerator is proposed for the video application. Previous works exploited the sparsity of differential (Diff) frame activation, but the improvement is limited as many Diff-frame data is small but non-zero. Processing of irregular sparse data also leads to low hardware utilization. To solve these problems, two key innovations are proposed in this article. First, we implement a hybrid-precision inter-frame-reuse architecture which takes advantage of both low bit-width and high sparsity of Diff-frame data. This technology can accelerate 3.2x inference speed with no accuracy loss. Second, we design a conv-pattern-aware processing array that achieves the 2.48x-14.2x PE utilization rate to process sparse data for different convolution kernels. The accelerator chip was implemented in 65-nm CMOS technology. To the best of our knowledge, it is the first silicon-proven CNN accelerator that supports inter-frame data reuse. Attributed to the inter-frame similarity, this video CNN accelerator reaches the minimum energy consumption of 24.7 μJ/frame in the MobileNet-slim model, which is 76.3% less than the baseline.

The detection of infrared small targets under low signal-to-clutter ratio (SCR) and complex background conditions has been a challenging and popular research topic. In this article, a spatial-temporal feature-based detection framework is proposed. First, several factors, such as the infrared target's small sample, the sensitive size, and the usual sample selection strategy, that affect the detection of small targets are analyzed. In addition, the small intersection over union (IOU) strategy, which helps to solve the false convergence and sample misjudgment problem, is proposed. Second, aiming at the difficulties due to the target's dim information and complex background, the interframe energy accumulation (IFEA) enhancement mechanism-based end-to-end spatial-temporal feature extraction and target detection framework is proposed. This framework helps to enhance the target's energy, suppress the strong spatially nonstationary clutter, and detect dim small targets. Experimental results show that using the small IOU strategy and IFEA mechanism, the proposed multiple frame-based detection framework performs better than some popular deep learning (DL)-based detection algorithms.

Most existing studies performed video denoising by using multiple adjacent noisy frames to recover one clean frame; however, despite achieving relatively good quality for each individual frame, these approaches may result in visual flickering when the denoised frames are considered in sequence. In this paper, instead of separately restoring each clean frame, we propose a multiframe-to-multiframe (MM) denoising scheme that simultaneously recovers multiple clean frames from consecutive noisy frames. The proposed MM denoising scheme uses a training strategy that optimizes the denoised video from both the spatial and temporal dimensions, enabling better temporal consistency in the denoised video. Furthermore, we present an MM network (MMNet), which adopts a spatiotemporal convolutional architecture that considers both the interframe similarity and single-frame characteristics. Benefiting from the underlying parallel mechanism of the MM denoising scheme, MMNet achieves a highly competitive denoising efficiency. Extensive analyses and experiments demonstrate that MMNet outperforms the state-of-the-art video denoising methods, yielding temporal consistency improvements of at least 13.3% and running more than 2 times faster than the other methods.

Patient motion during dynamic PET imaging can induce errors in myocardial blood flow (MBF) estimation. Motion correction for dynamic cardiac PET is challenging because the rapid tracer kinetics of 82Rb leads to substantial tracer distribution change across different dynamic frames over time, which can cause difficulties for image registration-based motion correction, particularly for early dynamic frames. In this paper, we developed an automatic deep learning-based motion correction (DeepMC) method for dynamic cardiac PET. In this study we focused on the detection and correction of inter-frame rigid translational motion caused by voluntary body movement and pattern change of respiratory motion. A bidirectional-3D LSTM network was developed to fully utilize both local and nonlocal temporal information in the 4D dynamic image data for motion detection. The network was trained and evaluated over motion-free patient scans with simulated motion so that the motion ground-truths are available, where one million samples based on 65 patient scans were used in training, and 600 samples based on 20 patient scans were used in evaluation. The proposed method was also evaluated using additional 10 patient datasets with real motion. We demonstrated that the proposed DeepMC obtained superior performance compared to conventional registration-based methods and other convolutional neural networks (CNN), in terms of motion estimation and MBF quantification accuracy. Once trained, DeepMC is much faster than the registration-based methods and can be easily integrated into the clinical workflow. In the future work, additional investigation is needed to evaluate this approach in a clinical context with realistic patient motion.

In order to deal with the problems of low detection accuracy and efficiency existing in traditional detection methods of body behavior characteristics in sports training, this paper proposes a new detection method of body behavior characteristics in sports training based on the grey correlation model. According to this method, the foreground binary image is obtained by the interframe difference method, and the action information and contour information are extracted, firstly. And then, the behavior feature descriptor is obtained. After the description of body behavior characteristics in sports training is completed, the grey correlation coefficient is calculated, and the complete observation equation of time delay estimation of image grey value under complex sports training background is established to complete the detection of body behavior characteristics. The experimental results show that, compared with the traditional methods, the detection accuracy and efficiency of the text method can always maintain a high level, indicating that the practical application performance of this method is strong. In fact, the grey analysis does not attempt to find the best solution but does provide techniques for determining a good solution.

With the advent of the Internet of Things era, the detection and recognition of moving objects is becoming increasingly important1. The current motion detection and recognition (MDR) technology based on the complementary metal oxide semiconductor (CMOS) image sensors (CIS) platform contains redundant sensing, transmission conversion, processing and memory modules, rendering the existing systems bulky and inefficient in comparison to the human retina. Until now, non-memory capable vision sensors have only been used for static targets, rather than MDR. Here, we present a retina-inspired two-dimensional (2D) heterostructure based retinomorphic hardware device with all-in-one perception, memory and computing capabilities for the detection and recognition of moving trolleys. The proposed 2D retinomorphic device senses an optical stimulus to generate progressively tuneable positive/negative photoresponses and memorizes it, combined with interframe differencing computations, to achieve 100% separation detection of moving trichromatic trolleys without ghosting. The detected motion images are fed into a conductance mapped neural network to achieve fast trolley recognition in as few as four training epochs at 10% noise level, outperforming previous results from similar customized datasets. The prototype demonstration of a 2D retinomorphic device with integrated perceptual memory and computation provides the possibility of building compact, efficient MDR hardware. A retina-inspired two-dimensional material based retinomorphic device exhibits all-in-one perception, memory and computing capabilities for motion detection and recognition.

Action recognition is essential for many human-centered applications in the Internet of Things (IoT). Especially, in the Internet of Medical Things (IoMT), action recognition shows great importance in surgical assistance, patient monitoring, etc. Recently, 3-D skeleton sequence-based action recognition draws broad attention. It is a challenging task that needs effective modeling on intraframe skeleton representations and interframe temporal dynamics. Standard long short-term memory (LSTM)-based models are widely used for sequence modeling due to its long-term memory, yet they are unable to fully model the relationship between different body joints or persons to extract crucial co-occurrence features from different levels. To handle this shortcoming, we propose an attention-based multilevel co-occurrence graph convolutional LSTM (AMCGC-LSTM). By integrating graph convolutional networks (GCNs) into LSTM, the proposed model is capable of leveraging body structural information from skeletons and strengthening the multilevel co-occurrence (MC) feature learning. Specifically, we first design the spatial attention module for feature enhancement of key joints from skeleton inputs. Second, we design MC memory units coupled with GCN to automatically model the spatial relationship between joints, and simultaneously capture the co-occurrence features from different joints, persons, and frames. Finally, we construct aggregated features of MCs (AFMCs) from MC memory units to better represent the intraframe action context encoding, and leverage a concurrent LSTM (Co-LSTM) to further model their temporal dynamics for action recognition. Our model significantly outperforms mainstream methods on NTU RGB+D 60/120 data set, mutual action subset of NTU RGB+D 60/120 data set, and Northewestern-UCLA data set.

We consider the problem of visual odometry for a sequence of video frames using a camera directed perpendicularly downward. We propose an adaptive two-stage visual odometry technology based on sequential determination of interframe shifts and regular correction of current coordinate estimates. At the first stage, the shift between two consecutive frames is determined by the correlation method, with the compared video frames being aligned using the found shift parameters up to a pixel. At the second stage, the shifts are refined with subpixel precision using the optical flow method. To improve reliability, the most consistent estimates of the optical flow are selected. We present the results of experimental studies on publicly available survey data, which confirm the high reliability and accuracy of the estimates.

Video deblurring is a challenging low-level vision task due to variant blur artifacts caused by factors such as depth variations, high-speed movements and camera shakes. Although significant efforts have been devoted to addressing this task, two challenges of capturing temporal patterns and spatial topologies still remain. In this paper, an attention-based interframe compensation scheme is proposed to address the first challenge. The proposed scheme replaces frames in blurry sequences with newly restored frames, and estimates temporal patterns among the replaced sequence to restore the whole sequence. After each replacement, an attention block is employed to exploit dependencies among restored and blurry frames to capture stable temporal patterns. To tackle the second challenge, we propose an adaptive residual block that dynamically fuses multi-level features via learning location-specific weights. Comprehensive experimental results demonstrate that the proposed method achieves state-of-the-art performance in terms of accuracy, visual effect and model size.

Inappropriate vehicle speeding remains a central factor that causes road accidents claiming millions of lives every year. This challenge has raised concerns for vehicle speed estimation as an attempt to promote speed enforcement methods. Traditionally, radar and lidar systems have widely been used for this purpose, despite their several shortfalls: cosine error effects, need for direct line-of-sight, and inability to simultaneously and accurately measure speed from multiple vehicles. The current work proposes an algorithm and a multi-vehicle speed estimation system in a multi-lane road environment to address multi-vehicle speed estimation shortfalls. The proposed solution exploits image processing and computer vision techniques to flag vehicles with inappropriate speeding patterns. A series of experiments showed that the developed system generates more accurate results than those given by the lidar system. In essence, the proposed system can estimate the speed of up to six vehicles concurrently. It can produce an average percentage error of 2.7% relative to the actual speed measured by a speedometer. This error is 5.4% lower than that demonstrated by the lidar system, emphasizing that the proposed system may be a more suitable approach to traffic laws enforcement. Keywords: Computer vision; image processing; inter-frame differencing; vehicle tracking; road accident

This letter first investigates the effects of inter-frame interference (IFI) on the recent eigendecomposition-precoded (EVD) non-orthogonal frequency-division multiplexing (NOFDM) transmission scheme, where subcarrier spacing is set smaller than that of orthogonal frequency-division multiplexing (OFDM). More specifically, the system model of EVD-NOFDM is generalized to support multi-frame transmission, and we propose an optimal power allocation for the multi-frame EVD-NOFDM architecture. The NOFDM scheme’s information rate bound is derived and evaluated by our theoretical analysis by assuming the presence of IFI, whereas most previous NOFDM studies ignored IFI effects. Our performance results demonstrate that the proposed multi-frame NOFDM scheme eliminates the detrimental IFI effects while attaining its specific benefits of bandwidth efficiency.

This paper presents a novel camera motion classification framework based on modeling the compressed domain block motion vectors using the HSI color model. The input to the proposed method is the interframe block motion vectors decoded from the compressed bitstream. The block motion vector’s magnitude and orientation are estimated, followed by assigning motion vector orientation to Hue, motion vector magnitude to Saturation, and keeping Intensity at a fixed value. The HSI assignment is then converted into an RGB image followed by supervised learning utilizing a convolutional neural network to recognize eleven camera motion patterns comprising seven pure camera motion patterns and four mixed camera patterns. The proposed method’s premise is based on posing the camera motion classification problem as a color recognition task. Detailed experimental analysis that includes a comparison with state-of-the-art methods, ablation study, and robustness analysis is carried out utilizing block motion vectors obtained from H.264/AVC encoded videos. Results demonstrate accuracies of over 98 % in recognizing eleven camera patterns for the proposed method.

In this paper, we use an ant colony heuristic method to tackle the integration of data association and state estimation in the presence of cell mitosis, morphological change and uncertainty of measurement. Our approach first models the scouting behavior of an unlabeled ant colony as a chaotic process to generate a set of cell candidates in the current frame, then a labeled ant colony foraging process is modeled to construct an interframe matching between previously estimated cell states and current cell candidates through minimizing the optimal sub-pattern assignment metric for track (OSPA-T). The states of cells in the current frame are finally estimated using labeled ant colonies via a multi-Bernoulli parameter set approximated by individual food pheromone fields and heuristic information within the same region of support, the resulting trail pheromone fields over frames constitutes the cell lineage trees of the tracks. A four-stage track recovery strategy is proposed to monitor the history of all established tracks to reconstruct broken tracks in a computationally economic way. The labeling method used in this work is an improvement on previous techniques. The method has been evaluated on publicly available, challenging cell image sequences, and a satisfied performance improvement is achieved in contrast to the state-of-the-art methods.

For the surveillance video images captured by monocular camera, this paper proposes a method combining foreground detection and deep learning to detect moving pedestrians, making full use of the invariable background of video image. Firstly, the motion region is extracted by the method of interframe difference and background difference. Then, the normalized motion region extracts the feature vectors based on the improved YOLOv3 tiny network. Finally, the trained linear support vector machine is used for pedestrian detection, and the performance of the fusion detection algorithm on caviar dataset is given, which proves the effectiveness of the proposed fusion detection algorithm. Experimental results show that the proposed method not only improves the practical application of pedestrian rerecognition but also reduces the detection range, computational complexity, and false detection rate compared with sliding window method.

PURPOSE: Data-driven rigid motion estimation for PET brain imaging is usually performed using data frames sampled at low temporal resolution to reduce the overall computation time and to provide adequate signal-to-noise ratio in the frames. In recent work it has been demonstrated that list-mode reconstructions of ultra-short frames are sufficient for motion estimation and can be performed very quickly. In this work we take the approach of using image-based registration of reconstructions of very short frames for data-driven motion estimation, and optimize a number of reconstruction and registration parameters (frame duration, MLEM iterations, image pixel size, post-smoothing filter, reference image creation, and registration metric) to ensure accurate registrations while maximizing temporal resolution and minimizing total computation time. METHODS: Data from (18)F-fluorodeoxyglucose (FDG) and (18)F-florbetaben (FBB) tracer studies with varying count rates are analysed, for PET/MR and PET/CT scanners. For framed reconstructions using various parameter combinations inter-frame motion is simulated and image-based registrations are performed to estimate that motion. RESULTS: For FDG and FBB tracers using 4 × 10(5) true and scattered coincidence events per frame ensures that 95% of the registrations will be accurate to within 1 mm of the ground truth. This corresponds to a frame duration of 0.5 –. 1 sec for typical clinical PET activity levels. Using 4 MLEM iterations with no subsets, a transaxial pixel size of 4 mm, a post-smoothing filter with 4–6 mm full-width at half-maximum, and averaging two or more frames to create the reference image provides an optimal set of parameters to produce accurate registrations while keeping the reconstruction and processing time low. CONCLUSIONS: It is shown that very short frames (≤ 1 sec) can be used to provide accurate and quick data-driven rigid motion estimates for use in an event-by-event motion corrected reconstruction.

Abnormal behaviour detection algorithm needs to conduct behaviour analysis on the basis of continuous video inclination tracking, and the robustness of the algorithm is reduced for the occlusion of moving targets, the occlusion of the environment, and the movement of targets with the same colour. For this reason, the optical flow information between RGB (red, green, and blue) images and video frames is used as the input of the network in view of group behaviour. Then, the direction, velocity, acceleration, and energy of the crowd were weighted and fused into a global optical flow descriptor. At the same time, the crowd trajectory map is extracted from the original image of a single frame. Following, in order to realize the detection of large displacement moving target and solve the problem that the traditional optical flow algorithm is only suitable for the detection of displacement moving target, a video abnormal behaviour detection algorithm based on the double-flow convolutional neural network is proposed. The network uses two network branches to learn spatial dimension information and temporal dimension information, respectively, and uses short- and long-time neural network to model the dependency relationship between long-time video frames, so as to obtain the final behaviour classification results. Simulation test results show that the proposed method can achieve good recognition effect on multiple datasets, and the performance of abnormal behaviour detection can be significantly improved by using interframe motion information.

Sensitivity alignment, based on preliminary calibration, is the technique used to improve the operational parameters in multielement photodetectors having spatially uneven sensitivity. This calibration should be repeated from time to time during operations to take care of drifting the detectors parameters. The experts are interested specifically in calibrating and correcting techniques that do not require the standard light source. The method proposed to evaluate and correct uneven sensitivity in scanning photodetectors is aimed at reducing effect of interference on interframe processing. It is based on analyzing an image pair obtained through shifting the photodetector and the scene recorded. When the image shift is known for columns situated in the image overlap area, an underdetermined system of equations can be developed and solved using the least square technique. A method for correcting the solution error caused by the system underdefiniteness is presented. A numerical simulation confirms the efficiency of the method to suppress interframe background and detect moving small objects.

While a large body of evidence has demonstrated the effects of attention on spatial processes, we know much less about attentional effects on the complementary temporal aspects of visual perception. To narrow this knowledge gap, we examined the effects of endogenous attention-the voluntary component of spatial attention-on temporal integration using the Ternus display. In a typical Ternus display, horizontally aligned discs shift by one position across alternating frames that are separated by a varying interframe interval. This display can induce two different motion percepts: all three discs moving together back and forth (group motion), or the two central discs seeming to remain static and the outer disk jumping across them (element motion). Several studies suggest that element motion reflects temporal integration. Thus, we used the rate of element motion percept to measure temporal integration. Attention was manipulated via the degree of certainty regarding the discs' location (Experiment 1), or with central informative arrows (Experiment 2). The pattern of results was similar in both experiments: The participants reported perceiving element motion more often when attention was allocated in advance to the discs' location. These results suggest that attention prolongs the period of time over which information is integrated. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

In this paper, we track the motion of multiple targets in sports videos by a machine learning algorithm and study its tracking technique in depth. In terms of moving target detection, the traditional detection algorithms are analysed theoretically as well as implemented algorithmically, based on which a fusion algorithm of four interframe difference method and background averaging method is proposed for the shortcomings of interframe difference method and background difference method. The fusion algorithm uses the learning rate to update the background in real time and combines morphological processing to correct the foreground, which can effectively cope with the slow change of the background. According to the requirements of real time, accuracy, and occupying less video memory space in intelligent video surveillance systems, this paper improves the streamlined version of the algorithm. The experimental results show that the improved multitarget tracking algorithm effectively improves the Kalman filter-based algorithm to meet the real-time and accuracy requirements in intelligent video surveillance scenarios.

In this paper, we simulate the estimation of motion through an interframe difference detection function model and investigate the spatial-temporal context information correlation filtering target tracking algorithm, which is complex and computationally intensive. The basic theory of spatiotemporal context information and correlation filtering is studied to construct a fast target tracking method. The different computational schemes are designed for the flow of multiframe target detection from background removal to noise reduction, to single-frame detection, and finally to multiframe detection, respectively. This enables the ground-based telescope to effectively detect spatial targets in dense stellar backgrounds in both modes. The method is validated by simulations and experiments and can meet the requirements of real projects. The interframe bit attitude estimation is optimized by using the beam-parity method to reduce the interframe estimation noise; a global optimization strategy based on the bit attitude map is used in the back end to reduce the system computation amount and make the global bit attitude estimation more accurate; a loop detection based on the word pocket model is added to the system to reduce the cumulative error.

A major challenge for semantic video segmentation is how to exploit the spatiotemporal information and produce consistent results for a video sequence. Many previous works utilize the precomputed optical flow to warp the feature maps across adjacent frames. However, the imprecise optical flow and the warping operation without any learnable parameters may not achieve accurate feature warping and only bring a slight improvement. In this paper, we propose a novel framework named Dynamic Warping Network (DWNet) to adaptively warp the interframe features for improving the accuracy of warping-based models. Firstly, we design a flow refinement module (FRM) to optimize the precomputed optical flow. Then, we propose a flow-guided convolution (FG-Conv) to achieve the adaptive feature warping based on the refined optical flow. Furthermore, we introduce the temporal consistency loss including the feature consistency loss and prediction consistency loss to explicitly supervise the warped features instead of simple feature propagation and fusion, which guarantees the temporal consistency of video segmentation. Note that our DWNet adopts extra constraints to improve the temporal consistency in the training phase, while no additional calculation and postprocessing are required during inference. Extensive experiments show that our DWNet can achieve consistent improvement over various strong baselines and achieves state-of-the-art accuracy on the Cityscapes and CamVid benchmark datasets.

Recently, a few image compressed sensing (CS) methods based on deep learning have been developed, which achieve remarkable reconstruction quality with low computational complexity. However, these existing deep learning-based image CS methods focus on exploring intraframe correlation while ignoring interframe cues, resulting in inefficiency when directly applied to video CS. In this paper, we propose a novel video CS framework based on a convolutional neural network (dubbed VCSNet) to explore both intraframe and interframe correlations. Specifically, VCSNet divides the video sequence into multiple groups of pictures (GOPs), of which the first frame is a keyframe that is sampled at a higher sampling ratio than the other nonkeyframes. In a GOP, the block-based framewise sampling by a convolution layer is proposed, which leads to the sampling matrix being automatically optimized. In the reconstruction process, the framewise initial reconstruction by using a linear convolutional neural network is first presented, which effectively utilizes the intraframe correlation. Then, the deep reconstruction with multilevel feature compensation is proposed, which compensates the nonkeyframes with the keyframe in a multilevel feature compensation manner. Such multilevel feature compensation allows the network to better explore both intraframe and interframe correlations. Extensive experiments on six benchmark videos show that VCSNet provides better performance over state-of-the-art video CS methods and deep learning-based image CS methods in both objective and subjective reconstruction quality.

LED-to-camera communication for smartphones promises to enable low-cost, small-size, and intuitive data access to Internet-of-things (IoT) devices. However, for fast and reliable communication under the rolling shutter effect, the packet losses caused by inter-frame gaps and frame drops must be dealt with. In this paper, we introduce LED-to-camera communication between a commercial smartphone and an IoT device with LED flashlight, where the user can intuitively acquire the desired data transmitted from the IoT device to the smartphone. Specifically, we present IoTorch, a fast and reliable LED-to-camera communication that efficiently prevents the packet losses. IoTorch consists of two core mechanisms: 1) a minimum-repetition one-way reliable transmission focusing on the periodicity of inter-frame gaps and 2) an acknowledgement mechanism to overcome frame drops by using a smartphone's built-in flash focusing on its delay characteristics. Additionally, we propose an optimization method to increase the throughput up to 2.92 kbps (i.e., 2.43 times faster than that of the state-of-the-art method that overcomes the inter-frame gaps). Furthermore, we realize a remote wake-up function by using the smartphone's flash as the data transmission trigger. To demonstrate the benefit of IoTorch, we present a sensor data viewer using Arduino and a smartphone console system on TelosB.

Digital video technology has been increasingly needed in various fields, such as telecommunications, entertainment, medicine. Therefore, video compression is required. Motion estimation methods help in improving video compression efficiency by effectively removing the temporal redundancy between successive frames. Several block-based motion estimation (BME) algorithms are being suggested to reduce the coding process’s computational complexity. This paper proposes a new rapid hybrid (BME) algorithm established on the primary search point prediction and advance ending search point strategies. It combines rough adaptive search and effective local search. The coarse search introduces a new motion vector (MV) prediction technique that utilizes the macro-blocks (MBs) Spatio-temporal correlations to optimize the traditional adaptive-rood-pattern search algorithm (ARPS) and speeding up the whole process without affecting the accuracy. In the accurate local search, the cross-formed search pattern using a one-step search (OSS) block matching algorithm is employed, to estimate the actual (MV) with less computation time and further speed up the search efficiency. Exhaustive experiments are performed to demonstrate the present algorithm’s performance over the benchmark schemes concerning specific assessment criteria for results, including the peak signal-to-noise ratio (PSNR), computational complexity and computational gain. The results show that the proposed algorithm is efficient and reliable; it can always give better performance over diamond search (DS) and (ARPS). The conducted test shows an increased performance of search speed while preserving the visual quality of the motion-compensated images, and it achieves 59.76–88.03 speed improvement over (DS) and 20.98–72.06 over (ARPS) for different video sequences. Besides, the suggested method (ARP-OSS) provides the best result compared to (DS) and (ARPS) in terms of time complexity for analyzing all the video samples.

Video shreds of evidence are usually admissible in the court of law all over the world. However, individuals manipulate these videos to either defame or incriminate innocent people. Others indulge in video tampering to falsely escape the wrath of the law against misconducts. One way impostors can forge these videos is through inter-frame video forgery. Thus, the integrity of such videos is under threat. This is because these digital forgeries seriously debase the credibility of video contents as being definite records of events. This leads to an increasing concern about the trustworthiness of video contents. Hence, it continues to affect the social and legal system, forensic investigations, intelligence services, and security and surveillance systems as the case may be. The problem of inter-frame video forgery is increasingly spontaneous as more video-editing software continues to emerge. These video editing tools can easily manipulate videos without leaving obvious traces and these tampered videos become viral. Alarmingly, even the beginner users of these editing tools can alter the contents of digital videos in a manner that renders them practically indistinguishable from the original content by mere observations. This paper, however, leveraged on the concept of correlation coefficients to produce a more elaborate and reliable inter-frame video detection to aid forensic investigations, especially in Nigeria. The model employed the use of the idea of a threshold to efficiently distinguish forged videos from authentic videos. A benchmark and locally manipulated video datasets were used to evaluate the proposed model. Experimentally, our approach performed better than the existing methods. The overall accuracy for all the evaluation metrics such as accuracy, recall, precision and F1-score was 100%. The proposed method implemented in the MATLAB programming language has proven to effectively detect inter-frame forgeries.

In the hybrid video coding framework, quantization is the key technique to achieve lossy compression. The information loss caused by the quantization may be reduced to improve compression efficiency, by using either encoder-side rate-distortion optimized quantization or decoder-side filtering. Nonetheless, the existing studies did not extensively use the already encoded information, i.e., context, to reduce the quantization loss. We address this issue and propose a context-adaptive inverse quantization method, namely, CAIQ. Specifically, for inter-frame coding, we analyze the correlation between the prediction signal (generated by motion compensated prediction) and the residual signal, as well as the correlation within the residual signal itself. We then present linear as well as nonlinear yet lightweight models to exploit the observed correlations in the frequency domain. Our models provide an optional inverse quantization mode by referring to the prediction signal, which is available at the decoder side. Next, block-level mode selection regarding the CAIQ method is used at the encoder side. We integrate the proposed CAIQ method into the reference software of Versatile Video Coding. We perform an extensive study of the models and analyze their resulting compression efficiency gain and encoding/decoding complexity. Experimental results show that our CAIQ method improves compression performance especially for high-resolution videos and at high bit rates.

Overcoming illumination variance is a critical factor in vision-based navigation. Existing methods tackled this radical illumination variance issue by proposing camera control or high dynamic range (HDR) image fusion. Despite these efforts, we have found that the vision-based approaches still suffer from overcoming darkness. This letter presents real-time image synthesizing from carefully controlled seed low dynamic range (LDR) image, to enable visual simultaneous localization and mapping (SLAM) in an extremely dark environment (less than 10 lux). Unlike existing methods, we elaborately select the seed LDR image for HDR fusion to secure interframe consistency, which is important in visual navigation. After selecting the seed image by camera control, we exploit camera response function (CRF) to synthesize HDR images in real-time without requiring GPU. We validate the algorithm via two extremely dark environments, an indoor environment without light and an outdoor night. In both test scenarios, the proposed method enabled reliable visual SLAM even when the light was limited.

Video semantic segmentation aims atgenerating temporal consistent segmentation results and is still a very challenging task in the deep learning era. In this work, we improve prior approaches from two aspects. On the network architecture level, we present the cross and self-attention network (CSANet). As opposed to prior methods, CSANet not only propagates temporal features from adjacent frames, but is also designed to aggregate spatial context within the current frame, which is shown to effectively improve the consistency and robustness of the extracted deep features. On the loss function level, we further propose the inter-frame mutual learning strategy which ensures the cross-attention module to focus on semantically correlated context regions, allowing the segmentation results at different frames to be collaboratively improved. By combining the above two novel designs, we show that our proposed method is able to deliver state-of-the-art performance on the Cityscapes and CamVid benchmarks.

Recently we have proposed a coding algorithm of point cloud geometry based on a rather different approach from the popular octree representation. In our algorithm, the point cloud is decomposed in silhouettes, hence the name Silhouette Coder, and context adaptive arithmetic coding is used to exploit redundancies within the point cloud (intra frame coding), and also using a reference point cloud (inter frame coding). In this letter we build on our previous work and propose a context selection algorithm as a pre-processing stage. With this algorithm, the point cloud is first parsed testing a large number of candidate context locations. The algorithm selects a small number of these contexts that better reflect the current point cloud, and then encode it with this choice. The proposed method further improves the results of our previous coder, Silhouette 4D, by $10 \%$ , on average, on a dynamic point cloud dataset of the JPEG Pleno, and achieves bitrates competitive with some high quality lossy coders such as the MPEG G-PCC.

This paper proposes a reverse procedure detection method for space target streaks based on motion parameter estimation. According to the phase shift characteristics of the spectrum, the interframe phase difference is used to obtain the displacement vectors of the target streak and image background, and then the morphological parameters of the target streak are estimated. In addition, a streak window is designed on the basis of the parameters, utilizing the local grayscale correlation in two image frames to carry out target searching and positioning with the interval of the target displacement vector. Images from different scenes are used to evaluate the adaptability of our method, and the experimental results suggest that the expected targets can be detected successfully with subpixel precision in all telescope operating modes. The proposed frequency-domain reverse procedure detection method avoids the problem of fake target confirmation caused by blind detection in traditional methods and overcomes the interference produced when the target streak passes through a bright star. Moreover, first target tracking and then positioning, which cannot be performed with time-domain methods, can be achieved. Our proposed method is attractive for multiple applications, such as space target cataloging, early warning and orbit prediction.

State-of-the-art video coding tools are submitted to severe performance and energy consumption requirements resulting from high complexity of video standards and from limited energy budgets of portable mobile devices. While providing most of the compression gains, inter frame and intra frame prediction techniques are the most demanding steps, since they compare a huge number of blocks. In such a process, the similarity metric employed affects both the quality of compression and the calculation effort. In this paper we propose the use of Hadamardbased Sum of Absolute Transformed Differences (SATD), in replacement of the traditionally used Sum of Absolute Differences (SAD), as a means of improving the efficiency of video coding. To allow that we explore two Hadamard Transform methods to design efficient SATD architectures, one using the Fast Hadamard Transform (FHT) butterfly and another one using the so-called Transform-Exempted (TE) SATD algorithm. Those methods were combined with architectural decisions (full parallelism, full parallelism with pipelining or multi-cycling) to build a total of six Hadamard-based SATD architectures that were synthesized for a commercial 45nm standard cell library for two operating frequencies. The architectures were simulated with pixel block data to obtain realistic dynamic power and energy estimates. The TE-SATD architectures achieved the lowest energy results: down to 13.13 pJ/SATD in the case of parallel architecture with pipeline. However, considering also the area results when evaluating energy, the best results are given by both methods using multi-cycling (transpose buffer): nearly 20.75 pJ/SATD with up to 63.54% smaller area compared with fully parallel architectures.

Facing COVID-19 epidemic, many countries have recently strengthened epidemic prevention and control measures The reliability of safety management is of great significance to personnel management and control during the COVID-19 epidemic period The focus of security management of early warning is to monitor and identify the moving target The current optical flow method is vulnerable to the influence of light changes and background movement, and it is not very accurate for moving target detection in dynamic complex background In this paper, aiming at the traditional Lucas Kanade optical flow method, the inter frame difference method, mean shift clustering algorithm and morphological processing are combined to optimize and improve on the original basis, so that the moving target detection effect in both simple and complex environments is significantly improved At the same time, the improved algorithm also reduces the execution time to a certain extent, and has a certain resistance to noise interference such as light changes This has a certain ability test value for personnel control during the epidemic © 2020 - IOS Press and the authors All rights reserved

Video forensics is one of the hot topics in multimedia forensics. Nowadays, spreading fake videos across the internet is becoming a profession mainly in politics and entertainment. In this study, a novel two-stage inter-frame video forgery detection technique is proposed. The first stage analyses spatio-temporal feature flow consistency to detect suspicious tamper points. Identifying the type of forgery and validating the recovered video are done in the second stage. Earth mover's distance is used as a similarity metric in both stages. The authors concentrate on a robust inter-frame forgery detection approach which can be applied for any challenging video. Compression at a higher rate, noise addition, and filtering are the anti-forensic tricks used by forgers to fool forensic techniques. However, most of the literature in video forgery detection has handled these issues as post-processing attacks and reported lesser accuracies for it. Hence the authors propose a robust and efficient forgery detection technique capable of identifying all kinds of inter-frame forgeries in videos. Experimental evaluation of the public video data set shows that the proposed approach outperforms existing approaches with an improved rate of robustness.

Visual–Inertial Navigation Systems (VINS) plays an important role in many navigation applications. In order to improve the performance of VINS, a new visual/inertial integrated navigation method, named Sliding-Window Factor Graph optimised algorithm with Dynamic prior information (DSWFG), is proposed. To bound computational complexity, the algorithm limits the scale of data operations through sliding windows, and constructs the states to be optimised in the window with factor graph; at the same time, the prior information for sliding windows is set dynamically to maintain interframe constraints and ensure the accuracy of the state estimation after optimisation. First, the dynamic model of vehicle and the observation equation of VINS are introduced. Next, as a contrast, an Invariant Extended Kalman Filter (InEKF) is constructed. Then, the DSWFG algorithm is described in detail. Finally, based on the test data, the comparison experiments of Extended Kalman Filter (EKF), InEKF and DSWFG algorithms in different motion scenes are presented. The results show that the new method can achieve superior accuracy and stability in almost all motion scenes.

Despite all its irrefutable benefits, the development of steganography methods has sparked ever-increasing concerns over steganography abuse in recent decades. To prevent the inimical usage of steganography, steganalysis approaches have been introduced. Since motion vector manipulation leads to random and indirect changes in the statistics of videos, MV-based video steganography has been the center of attention in recent years. In this paper, we propose a 54-dimentional feature set exploiting spatio-temporal features of motion vectors to blindly detect MV-based stego videos. The idea behind the proposed features originates from two facts. First, there are strong dependencies among neighboring MVs due to utilizing rate-distortion optimization techniques and belonging to the same rigid object or static background. Accordingly, MV manipulation can leave important clues on the differences between each MV and the MVs belonging to the neighboring blocks. Second, a majority of MVs in original videos are locally optimal after decoding concerning the Lagrangian multiplier, notwithstanding the information loss during compression. Motion vector alteration during information embedding can affect these statistics that can be utilized for steganalysis. Experimental results have shown that our features’ performance far exceeds that of state-of-the-art steganalysis methods. This outstanding performance lies in the utilization of complementary spatio-temporal statistics affected by MV manipulation as well as feature dimensionality reduction applied to prevent overfitting. Moreover, unlike other existing MV-based steganalysis methods, our proposed features can be adjusted to various settings of the state-of-the-art video codec standards such as sub-pixel motion estimation and variable-block-size motion estimation.

With the wide application of stereoscopic video technology, the quality of stereoscopic video has attracted people’s attention. Objective stereoscopic video quality assessment (SVQA) is highly challenging, but essential, particularly the no-reference (NR) SVQA method, where reference information is not needed and a large number of samples are required for training and testing sets. However, as far as we know, there are only a few samples in the established stereo video database, which is unsuitable for NR quality assessment and seriously hampers the development of NR-SVQA method. For these difficulties that we encountered, we carry out a comprehensive subjective evaluation of stereoscopic video quality in our newly established TJU-SVQA databases that contain various contents, mixed resolution coding and symmetrically/asymmetrically distorted stereoscopic videos. Furthermore, we propose a new inter-frame cross map to predict the objective quality scores. We compare and analyze the performance of several state-of-the-art 2D and 3D quality evaluation methods on our new databases. The experimental results on our established databases and a public database demonstrate that the proposed method can robustly predict the quality of stereoscopic videos.

In view of the limited bandwidth of underwater video image transmission, a low bit rate underwater video compression coding method is proposed. Based on the preprocessing process of wavelet transform and coefficient down-sampling, the visual redundancy of underwater image is removed and the computational coefficients and coding bits are reduced. At the same time, combined with multi-level wavelet decomposition, inter frame motion compensation, entropy coding and other methods, according to the characteristics of different types of frame image data, reduce the number of calculations and improve the coding efficiency. The experimental results show that the reconstructed image quality can meet the visual requirements, and the average compression ratio of underwater video can meet the requirements of underwater acoustic channel transmission rate.

This paper proposes a new steganography method for hiding data into dynamic GIF (Graphics Interchange Format) images. When using the STC framework, we propose a new algorithm of cost assignment according to the characteristics of dynamic GIF images, including the image palette and the correlation of interframes. We also propose a payload allocation algorithm for different frames. First, we reorder the palette of GIF images to reduce the modifications on pixel values when modifying the index values. As the different modifications on index values would result in different impacts on pixel values, we assign the elements with less impact on pixel values with small embedding costs. Meanwhile, small embedding costs are also assigned for the elements in the regions that the interframe changes are large enough. Finally, we calculate an appropriate payload for each frame using the embedding probability obtained from the proposed distortion function. Experimental results show that the proposed method has a better security performance than state-of-the-art works.

The drone has been applied in various areas due to its small size, high mobility and low price. However, illegal uses of drones have posed huge threats to both public safety and personal privacy. There is an urgent demand for technologies that can timely detect and counter the drones. In this paper, we propose an adaptive switching spatial-temporal fusion detection method for remote flying drones in the airspace using electrical-optical cameras, which can enhance the contrast between the target and background as well as suppressing the noises and clutters simultaneously. For each incoming video frame, a dark-attentive interframe difference method and a row-column separate black-hat method are proposed to generate temporal feature maps (TFM) and spatial feature maps (SFM), respectively, in parallel. Inspired by the phenomenon that the features in TFMs and SFMs both go strong at the regions of the intended target while they do not at other regions where noises and clutters locate, we design an adaptive switching spatial-temporal fusion mechanism to fuse the SFMs and TFMs, generating adaptive switching spatial-temporal feature maps (ASSTFM). Finally, an adaptive local threshold mechanism is used in ASSTFMs to segment the targets from backgrounds. In order to validate the effectiveness of our method, we conduct both offline experiments and field tests. The experiment results manifest that our method is superior to the other seven baseline methods and works more stably for different backgrounds and various types of drones.

The particular characteristics of contents generated by computers and the frequent scene change in screen content videos make the exiting R- $\lambda $ model in the rate control be not suitable to encode screen content videos. A novel rate control scheme designed for screen content videos coding (SCC) is proposed in this paper. Text blocks, screen image blocks and nature image blocks in screen content videos can result in different bitrate-distortion (R-D) relationships. A content-based rate control method is put forward at Coding Tree Unit (CTU) level for this reason. Three independent parameter update modes are adopted for three different types of CTUs according to their corresponding R-D relationship. Furthermore, frequent scene change in screen content videos needs a novel bit allocation scheme for precise rate control. In view of this, the frames in the screen content videos are classified into scene changed frames and scene unchanged frames. Different bit allocation methods are adopted for different types of frames on the basis of inter frame and intra frame complexity. Besides, a region level bit allocation algorithm considering the inter frame continuity and the numbers of different types of CTUs is added between frame level and CTU level bit allocation schemes. Experimental results show that our proposed rate control algorithm respectively achieves 0.88 dB BDPSNR and 1.54 dB BDPSNR increase under the Low Delay coding structure on average, compared with the default rate control algorithm with hierarchical and non-hierarchical bit allocation in the HEVC-SCC.

Nanoscale phase change contrast agents (PCCAs) have been demonstrated to represent great potential for non-invasive extravascular imaging and therapeutic delivery. However, the contrast-to-tissue ratio (CTR) of PCCA images is usually limited due to either physiological motion or incomplete cancelation of tissue signal. Therefore, to improve the CTR of PCCA images in the presence of physiological motion, a new imaging technique, ultrafast inter-frame activation ultrasound (UIAU) imaging, is proposed and validated. Results of studies with controlled motion in tissue-mimicking phantoms indicate UIAU could provide significantly higher CTR (maximum: 17.3±0.9 dB) relative to conventional pulse inversion imaging (maximum CTR: 3.4±1.4 dB). UIAU exhibits a CTR of up to 16.1±1.0 dB relative to 3.9±2.3 dB for differential imaging in the presence of physiological motion at 20 mm/s. In vivo imaging of PCCAs in the rat liver also demonstrate the ability of UIAU for enhancing PCCA image contrast in the presence of physiological motion.

In 5G cellular networks, it is required to accommodate a massive number of Internet-of-Things (IoT) devices in a resource-efficient way. In this letter, we propose a random access with an inter-frame successive interference cancellation (RA-ISIC) for stationary IoT networks. In our proposed scheme, the BS does not discard the collided packets and attempts to recover them through inter-frame SIC operations. We evaluate the performance of our proposed scheme in terms of resource efficiency. Results show that a certain level of system load is required to maximize the efficiency of our proposed scheme, which can be achieved through a probabilistic retransmission (PR) policy. Consequently, our proposed scheme can effectively improve the resource efficiency and can be suitable to accommodate more IoT devices with a smaller amount of resources, compared to the conventional one.

Inter-frame forgery is a common type of video forgery to destroy the video evidence. It occurs in the temporal domain such as frame deletion, frame insertion, frame duplication, and frame shuffling. These forms of forgery are more frequently produced in a surveillance video because the camera position and the scene are relatively stable, where the tampering process is easy to operate and imperceptible. In this paper, we propose an efficient method for inter-frame forgery detection based on histogram of oriented gradients (HOG) and motion energy image (MEI). HOG is obtained from each image as a discriminative feature. In order to detect frame deletion and insertion, the correlation coefficients are used and abnormal points are detected via Grabb’s test. In addition, MEI is applied to edge images of each shot to detect frame duplication and shuffling. Experimental results prove that the proposed method can detect all inter-frame forgeries and achieve higher accuracy with lower execution time.

Massive inter predictive modes are adopted in latest High Efficiency Video Coding (HEVC) standard to eliminate temporal redundancies, which results in stronger inter-frame dependency among neighboring frames than previous standards like H.264. The inter-frame dependency makes currently independent rate-distortion optimization (RDO) non-optimal any more. Quantization parameter (QP) selection algorithm taking inter-frame dependency into consideration is supposed to optimize RDO based rate control greatly. According to our research, the inter-frame dependency is reflected by the linear relationship between QP change (ΔQP) and the resulting change of distortion (ΔD). An adaptive QP selection algorithm for global RDO is proposed based on the modeling function between ΔD and ΔQP in this paper. Firstly, based on intensive statistic analysis, three parameters (initial QP ($\overline {QP}$), the length of pictures of group (GOP), and the average of SATD of one frame) are used to formulate the relationship between ΔD and ΔQP. Secondly, the resulting rate change ΔR relative to ΔQP is also formulated similarly. Thirdly, optimized Lagrangian multiplier (λ) is calculated with these two mathematic models. Finally, we refine QP values based on the optimized λ in terms of dependent RDO. The experimental results show that the proposed frame-level QP selection algorithm can decrease Bj⊘tegaard Delta BitRate (BD-BR) by about 1.62% at the random-access (RA) configuration and 1.13% at the low-delay (LD) configuration, respectively. At the same time, it doesn’t increase complexity significantly.

In this paper, we propose a novel Convolutional Neural Network (CNN) based video coding technique using a video prediction network (VPN) to support enhanced motion prediction in High Efficiency Video Coding (HEVC). Specifically, we design a CNN VPN to generate a virtual reference frame (VRF), which is synthesized using previously coded frames, to improve coding efficiency. The proposed VPN uses two sub-VPN architectures in cascade to predict the current frame in the same time instance. The VRF is expected to have higher temporal correlation than a conventional reference frame, and, thus it is substituted for a conventional reference frame. The proposed technique is incorporated into the HEVC inter-coding framework. Particularly, the VRF is managed in a HEVC reference picture list, so that each prediction unit (PU) can choose a better prediction signal through Rate-Distortion optimization without any additional side information. Furthermore, we modify the HEVC inter-prediction mechanisms of Advanced Motion Vector Prediction and Merge modes adaptively when the current PU uses the VRF as a reference frame. In this manner, the proposed technique can exploit the PU-wise multi-hypothesis prediction techniques in HEVC. Since the proposed VPN can perform both the video interpolation and extrapolation, it can be used for Random Access (RA) and Low Delay B (LD) coding configurations. It is shown in experimental results that the proposed technique provides −2.9% and −5.7% coding gains, respectively, in RA and LD coding configurations as compared to the HEVC reference software, HM 16.6 version.

Image registration (or image alignment), the problem of aligning multiple images with relative displacement, is a crucial step in many multiframe image restoration algorithms. To solve the problem that most existing image registration approaches can only align two images in one inference, we propose a split-attention multiframe alignment network (SAMANet). Pixel-level displacements between multiple images are first estimated at low-resolution scales and then refined gradually with the increase in feature resolution. To better integrate the interframe information, we present a split-attention module (SAM) and a dot-product attention module (DPAM), which can adaptively rescale the cost volume features and optical flow features according to the similarity between features from different images. The experimental results demonstrate the superiority of our SAMANet over state-of-the-art image registration methods in terms of both accuracy and robustness. To solve the “ghosting effect” caused by pixelwise registration, we designed two “ghost” removal modules: warping repetition detection module (WRDM) and attention fusion module (AFM). WRDM detects “ghost” regions during the image warping process without increasing the time complexity of the registration algorithm. AFM uses an attention mechanism to rescale the aligned images and enables the registration network and the subsequent image restoration networks to be trained jointly. To validate the strengths of the proposed approaches, we apply SAMANet, WRDM and AFM to three image/video restoration tasks. Extensive evaluations demonstrate that the proposed methods can enhance the performance of image restoration algorithms and outperform the other compared registration algorithms.

Shots are the basic units for analyzing and retrieving video, and also the essential elements in creating video datasets. The traditional methods of shot detection exhibit unsatisfactory performance for being too sensitive to motion or too much time-consuming. This paper proposes an automatic shot detection method, by employing the fast feature descriptor of Oriented FAST and Rotated BRIEF (ORB) fused with Structural Similarity (SSIM). Firstly, ORB descriptor is used to preselect candidate segments with a high tolerance for rapidly extracting the features of twenty-frame intervals in video sequences. Then, the cut transition is detected by comparing ORB features, fused with SSIM, of consecutive frames in the candidate segment. Finally, the gradual transition is detected by determining the maximum amount of the continuous increasing/decreasing interframe differences in the candidate segment without cut transition. Experimental result indicates that the proposed method can achieve an F1-Score of 92.5% and five times of real-time speed with one CPU on 106049 test frames from the Open-video project, YouTube, and YOUKU. In addition, the proposed method can outperform the existing shot detection methods, including the rule-based and learning-based methods, by testing on the video sequences from the Open-video project and RAI dataset.