Search

Emotion causes constitute a pivotal component in the comprehension of emotional conversations. Recently, a new task named Causal Emotion Entailment (CEE) has been proposed to identify the causal utterances for the target emotional utterance in a conversation. Although researchers have achieved some progress in solving this problem, they failed to adequately incorporate speaker characteristics and overlooked the effects of temporal relations in conversation structures. To fill such a research gap to some extent, we propose a novel causal emotion entailment framework, namely MPEG (Multi-Perspective Enhanced Graph attention network). The training of MPEG consists of three stages. First, we utilize a speaker-aware pre-trained model and two attention mechanisms to obtain the utterance representations that incorporate local contexts as well as the speaker and emotional information. Then, these representations are fed into a graph attention network to model the conversation structures and emotional dynamics from both local and global perspectives. Finally, a fully-connected network is implemented to predict the relationships between emotional utterances and causal utterances. Experimental results show that MPEG achieves state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Compression methods for images and videos are essential for the effective archiving, transmission, and distribution of multimedia data and files. This paper reviews the state-of-the-art in image and video compression, including the most recent developments, algorithms, and methods. This study compiles findings from a variety of studies in an effort to give readers a bird's-eye view of the progress and obstacles in this dynamic sector. A survey of the relevant literature demonstrates that modern compression methods build upon the work of older algorithms like JPEG and MPEG. Compression ratios and picture quality can be enhanced, however, thanks to developments in transform coding, predictive coding, and entropy coding. Further, by combining machine learning and deep learning techniques, we now have access to cutting-edge options for improving compression efficiency and paving the way for adaptive, content-aware compression. Sustainable compression approaches are also highlighted, along with energy efficiency aspects. [ABSTRACT FROM AUTHOR]

The hyperspectral image provides rich spectral information content, which facilitates multiple applications. With the rapid advancement of the spatial and spectral resolution of optical instruments, the image data size has increased by many folds. For that, it requires a compression algorithm having low coding complexity, low coding memory demand and high coding efficiency. In recent years, many coding algorithms are proposed. The wavelet transform-based set-partitioned hyperspectral compression algorithms have superior coding performance. These algorithms employ linked lists or state tables to track the significant/insignificant of the partitioned sets/coefficients. The proposed algorithm uses the pyramid hierarchy property of wavelet transform. The markers are used to track the significance/insignificance of the pyramid level. A single pyramid level has many sets. An insignificant pyramid level having multiple sets is represented as a single bit in proposed compression algorithm, while a single insignificant set in 3D Set Partition Embedded bloCK (3D-SPECK) and 3D-Listless SPECK (3D-LSK) is represented as a single bit. Through this, the requirement of the bits in the proposed algorithm is less than other wavelet transform compression algorithms at the high bit planes. The simulation result shows that the proposed compression algorithm has high coding efficiency with very less coding complexity and moderate coding memory requirement. The reduced coding complexity improves the performance of the image sensor and lowers the power consumption. Thus, the proposed compression algorithm has great potential in low-resource onboard hyperspectral imaging systems. [ABSTRACT FROM AUTHOR]

To reduce the hardware implementation resource consumption of the two-dimensional transform component in H.266 VVC, a unified hardware structure is proposed that supports full-size Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), and full-size Low-Frequency Non-Separable Transform (LFNST). This paper presents an area-efficient hardware architecture for two-dimensional transforms based on a general Regular Multiplier (RM) and a high-throughput hardware design for LFNST in the context of H.266/VVC. The first approach utilizes the high-frequency zeroing characteristics of VVC and the symmetric properties of the DCT-II matrix, allowing the RM-based architecture to use only 256 general multipliers in a fully pipelined structure with a parallelism of 16. The second approach optimizes the transpose operation of the input matrix for LFNST in a parallelism of 16 architecture, aiming to save storage and logic resources. [ABSTRACT FROM AUTHOR]

Omnidirectional images have gained significant popularity and drawn great attention nowadays, which poses challenges to omnidirectional image processing in solving the bottleneck of storage and transmission. Projecting onto a two-dimensional image plane is generally used to compress an omnidirectional image. However, the most commonly used projection format, the equirectangular projection (ERP), results in a significant amount of redundant samples in the polar areas, thus incurring extra bitrate and geometric distortion. We derive the optimal latitude-adaptive bit allocation for each image tile. Subsequently, we propose a greedy algorithm for non-negative integer bit allocation (NNIBA) for non-uniform quantization under an omnidirectional image quality metric WMSE. In our experiment, we design quantization tables based on JPEG and compare our approach with other sampling-related methods. Our method achieves an average bit saving of 7.9% compared with JPEG while outperforming other sampling-related methods. Besides, we compare our non-uniform quantization approach with two proposed bit allocation methods, achieving an average improvement of 0.35 dB and 2.66 dB under WS-PSNR, respectively. The visual quality assessment also confirms the superiority of our method. [ABSTRACT FROM AUTHOR]

Versatile video coding standard known as H.266/VVC has been adopted by the Joint Video Experts Team in July 2020 Bross et al. (ITU-T and ISO/IEC JVET-S2001, 2020). It is the latest video coding standard with advanced tools to enhance coding efficiency. Notably, H.266/VVC introduces new techniques, including Multiple Transform Selection (MTS). In this paper, detailed descriptions of the new transform coding development in VVC standard are presented. The study also highlights the impact of the MTS on the VVC encoder performance through experimental research using VVC reference software VTM-14 for random access configuration. The experimental results show an increase of 8.35% in terms of time execution with an improvement of 1% in the quality of the reconstructed video along with a reduction of 1% in bitrate for a quantification parameter QP equal to 32. For a QP equal to 22, there is an increase of 15.8% in terms of time execution with an improvement of 0.056% in the quality of the reconstructed video along with a reduction of 0.445% in bitrate. [ABSTRACT FROM AUTHOR]

Hyperspectral (HS) images contain rich spatial and spectral information. Due to its large size, it is difficult to store, process, analyze, or transmit the critical information contained in it. The compression of hyperspectral images is inevitable. Many transform based Hyper Spectral Image Compression Algorithms (HSICAs) have been proposed in the past that work for both lossy and lossless compression processes. The transform based HSICA uses linked lists or dedicated markers or array structure to keep track of significant and insignificant sets or coefficients of a transformed HS image. However, these algorithms either suffered from low coding efficiency, high memory requirements, or high coding complexity. This work proposes a transform based HSICA using a curvelet transform to improve the directional elements and the ability to represent edges and other singularities along curves. The proposed HSICA aims to provide superior quality compressed HS images by representing HS images at different scales and directions and to achieve a high compression ratio. Experimental results show that the proposed algorithm has a low coding memory requirement with a 2% to 5% increase in coding gain compared to the other state of art compression algorithms. [ABSTRACT FROM AUTHOR]

This paper explores learned image compression based on traditional and learned discrete wavelet transform (DWT) architectures and learned entropy models for coding DWT subband coefficients. A learned DWT is obtained through the lifting scheme with learned nonlinear predict and update filters. Several learned entropy models, with varying computational complexities, are explored to exploit inter- and intra-DWT subband coefficient dependencies, akin to traditional EZW, SPIHT, or EBCOT algorithms. Experimental results show that when the explored learned entropy models are combined with traditional wavelet filters, such as the CDF 9/7 filters, compression performance that far exceeds that of JPEG2000 can be achieved. When the learned entropy models are combined with the learned DWT, compression performance increases further. The computations in the learned DWT and all entropy models, except one, can be simply parallelized, and thus, the systems provide practical encoding and decoding times on GPUs, unlike other DWT-based learned compression systems in the literature. [ABSTRACT FROM AUTHOR]

Smart grid is proposed as a solution to the problems of production, distribution, monitoring, and control of the electricity in traditional power grids. Smart grid networks place IoT sensor nodes at various grid lines and collect large volume of data about power flow, usage etc. The collected data are analyzed for various applications like demand forecasting, fault diagnosis and fault prediction etc. The sensor nodes and the communication links can be compromised affecting the privacy of consumers. False data can be propagated with malicious intentions. This work proposes a secure and privacy preserving framework for smart grid IoT networks to secure the data and decision at sensor nodes and communication links. The work proposes a novel Data and Decision rules Secure Efficient Smart Grid (DDSESG) framework integrating secure compressive sensing technique with blockchain and interplanetary file system (IPFS) for securing both data and decision. Through experimental analysis, the proposed solution is found to provide higher resiliency against data security attacks at comparative 12.4% lower computation cost, 15% lower communication cost, 19.9% lower storage cost. Forecasting on transformed data in proposed solution had only a marginal 1.08 % difference in accuracy compared to forecasting on original data. [ABSTRACT FROM AUTHOR]

The 3D-zero memory set partitioned embedded block (3D-ZM-SPECK) is an embedded and memory-efficient compression algorithm. Through, the 3D-ZM-SPECK does not require any coding memory, but testing of the sets for each bit-plane increases the complexity significantly which led a big constrain for the low resource HS image sensors. The proposed HS image compression algorithm is the low complexity solution of 3D-ZM-SPECK which reduces the recursive significance test of sets. Simulation results demonstrate that the proposed 3D-M-ZM-SPECK significantly reduces the complexity by ~ 25% with the other state of art HS image compression algorithms. Thus, it is a viable option for the HS image sensors. [ABSTRACT FROM AUTHOR]

With the spread of tampered images, locating the tampered regions in digital images has drawn increasing attention. The existing tampering localization methods, however, suffer from severe performance degradation when the images are subjected to some post-processing, as the tampering traces would be distorted by the post-processing operations. The poor robustness against post-processing has become a bottleneck for the practical applications of image tampering localization techniques. In order to address this issue, this paper proposes a novel re storation-assisted framework for image tampering loc alization (ReLoc). The ReLoc framework mainly consists of an image restoration module and a tampering localization module. The key idea of ReLoc is to use the restoration module to recover a high-quality counterpart from the distorted tampered image, such that the distorted tampering traces can be re-enhanced, facilitating the tampering localization module to identify the tampered regions. To achieve this, the restoration module is optimized not only with the conventional constraints on image visual quality, but also with a forensics-oriented objective function. Furthermore, the restoration module and the localization module are trained alternately, which can stabilize the training process and is beneficial for improving the performance. The robustness of ReLoc has been evaluated by using several common post-processing operations, including lossy compressions, online social network transmission, and image resizing. Extensive experimental results show that ReLoc can significantly improve the localization performance compared to using a restoration-free model. In addition, we have shown that the restoration module in a well-trained ReLoc model is transferable for different localization modules and across different datasets. [ABSTRACT FROM AUTHOR]

Recent studies have shown that side-information estimation (SIE) via JPEG image deblocking/restoration is effective in enhancing steganographic security when the side-information of JPEG rounding errors is unavailable. The polarity map of deblocking errors can work well in modulating handcrafted embedding costs. However, it may not be easy to design an optimal deblocking method that is universal to enhance security for all images, and it is unclear how to better utilize the polarity map in modulating statistical model-based embedding costs. To circumvent the difficulty of deblocking method design, we propose an optimal polarity map selection (OPMS) method leveraging existing well-performed deblocking methods. OPMS is designed based on polarity-oriented synthetic stego and minimum feature distance, so that the selected optimal polarity map (OPM) ensures a high security performance to each image. Besides, we propose a statistical model-based modulation method to better exploiting OPM in a quaternary quantized Gaussian embedding (QQGE) model. Through shifting the mean of the distribution, QQGE can derive effective modulated embedding probabilities and reduce the number of modified coefficients without increasing coding complexity. Experimental results demonstrate that the proposed overall steganographic method, called OPMS-QQGE, greatly surpasses existing state-of-the-art SIE-based methods in resisting both CNN-based and feature-based steganalyzers. [ABSTRACT FROM AUTHOR]

Recent landing marches on Mars have enabled the access to Martian surface images, which act as an important vehicle to demystify the evolution and habitability of Mars, in terms of climate, geography, etc. Transmitting Martian images thus calls for efficient compression methods to ensure the high-quality reconstruction from distant communication, in which the research is yet to start. To address this issue, we propose in this letter a learned structure-based hybrid (LSH) framework to compress Martian images. More specifically, we first observe that the structural consistency exists across Martian images, which motivates us to propose a structural compression network (SCN). The aim of SCN is to compactly represent the structural information of Martian images, thus allowing for the compression at extremely low bit-rates. Then, we propose a detail compensation network (DCN) to reconstruct the missing details when we restore from the structural information, which benefits from improved compression efficiency by reduced bit-rates. The experimental results have verified the superior performances of our LSH method on compressing Martian images, against existing state-of-the-art methods. [ABSTRACT FROM AUTHOR]

One of the main limitations to the adoption of deep learning for image compression is the need to train multiple models to compress at multiple rates. In the case of onboard remote sensing data compression, another limitation is the computational cost of the neural networks. Addressing both limitations, this letter presents a new reduced-complexity architecture for multirate compression of remote sensing images. The proposed architecture enables compressing at a precise user-selected rate while keeping a competitive performance in lossy compression on different sets of remote sensing data. The proposed approach is amenable for onboard deployment. [ABSTRACT FROM AUTHOR]

The synthetic aperture radar (SAR) image is widely used in many remote sensing applications. In order to store and transmit the increasing SAR image data, more efficient compression algorithms are needed. The purpose of this letter is to introduce a new framework for compressing SAR images. First, we propose a novel analysis and synthesis transform based on multi-Resblocks for transforming the original SAR image into a compact latent representation. Then, a Gaussian mixture model (GMM) is used to estimate the latent representation’s distribution. In order to explore the redundancy within the latent representation, the entropy model parameter is estimated by combining the local context, global context, and hyperprior information. In order to evaluate the performance of the proposed algorithm, we conduct experiments on a dataset of SAR images. The results show that the proposed algorithm outperforms JPEG2000 and some state-of-the-art learned image compression schemes in terms of compression performance. [ABSTRACT FROM AUTHOR]

Omnidirectional images have gained significant popularity and drawn great attention nowadays, which poses challenges to omnidirectional image processing in solving the bottleneck of storage and transmission. Projecting onto a two-dimensional image plane is generally used to compress an omnidirectional image. However, the most commonly used projection format, the equirectangular projection (ERP), results in a significant amount of redundant samples in the polar areas, thus incurring extra bitrate and geometric distortion. We derive the optimal latitude-adaptive bit allocation for each image tile. Subsequently, we propose a greedy algorithm for non-negative integer bit allocation (NNIBA) for non-uniform quantization under an omnidirectional image quality metric WMSE. In our experiment, we design quantization tables based on JPEG and compare our approach with other sampling-related methods. Our method achieves an average bit saving of 7.9% compared with JPEG while outperforming other sampling-related methods. Besides, we compare our non-uniform quantization approach with two proposed bit allocation methods, achieving an average improvement of 0.35 dB and 2.66 dB under WS-PSNR, respectively. The visual quality assessment also confirms the superiority of our method.

JPEG, which was developed 30 years ago, is the most widely used image coding format, especially favored by the resource-deficient devices, due to its simplicity and efficiency. With the evolution of the Internet and the popularity of mobile devices, a huge amount of user-generated JPEG images are uploaded to social media sites like Facebook and Flickr or stored in personal computers or notebooks, which leads to an increase in storage cost. However, the performance of JPEG is far from the-state-of-the art coding methods. Therefore, the lossless recompression of JPEG images is urgent to be studied, which will further reduce the storage cost while maintaining the image fidelity. In this paper, a hybrid coding framework for the lossless recompression of JPEG images (LLJPEG) using transform domain intra prediction is proposed, including block partition and intraprediction, transform and quantization, and entropy coding. Specifically, in LLJPEG, intra prediction is first used to obtain a predicted block. Then the predicted block is transformed by DCT and then quantized to obtain the predicted coefficients. After that, the predicted coefficients are subtracted from the original coefficients to get the DCT coefficient residuals. Finally, the DCT residuals are entropy coded. In LLJPEG, some new coding tools are proposed for intra prediction and the entropy coding is redesigned. The experiments show that LLJPEG can reduce the storage space by 29.43% and 26.40% on the Kodak and DIV2K datasets respectively without any loss for JPEG images, while maintaining low decoding complexity. [ABSTRACT FROM AUTHOR]

We present a simple and effective approach for non-blind image deblurring, combining classical techniques and deep learning. In contrast to existing methods that deblur the image directly in the standard image space, we propose to perform an explicit deconvolution process in a feature space by integrating a classical Wiener deconvolution framework with learned deep features. A multi-scale cascaded feature refinement module then predicts the deblurred image from the deconvolved deep features, progressively recovering detail and small-scale structures. The proposed model is trained in an end-to-end manner and evaluated on scenarios with simulated Gaussian noise, saturated pixels, or JPEG compression artifacts as well as real-world images. Moreover, we present detailed analyses of the benefit of the feature-based Wiener deconvolution and of the multi-scale cascaded feature refinement as well as the robustness of the proposed approach. Our extensive experimental results show that the proposed deep Wiener deconvolution network facilitates deblurred results with visibly fewer artifacts and quantitatively outperforms state-of-the-art non-blind image deblurring methods by a wide margin. [ABSTRACT FROM AUTHOR]

Invertible grayscale is a special kind of grayscale from which the original color can be recovered. Given an input color image, this seminal work tries to hide the color information into its grayscale counterpart while making it hard to recognize any anomalies. This powerful functionality is enabled by training a hiding sub-network and restoring sub-network in an end-to-end way. Despite its expressive results, two key limitations exist: 1) The restored color image often suffers from some noticeable visual artifacts in the smooth regions. 2) It is very sensitive to JPEG compression, i.e., the original color information cannot be well recovered once the intermediate grayscale image is compressed by JPEG. To overcome these two limitations, this article introduces adversarial training and JPEG simulator respectively. Specifically, two auxiliary adversarial networks are incorporated to make the intermediate grayscale images and final restored color images indistinguishable from normal grayscale and color images. And the JPEG simulator is utilized to simulate real JPEG compression during the online training so that the hiding and restoring sub-networks can automatically learn to be JPEG robust. Extensive experiments demonstrate that the proposed method is superior to the original invertible grayscale work both qualitatively and quantitatively while ensuring the JPEG robustness. We further show that the proposed framework can be applied under different types of grayscale constraints and achieve excellent results. [ABSTRACT FROM AUTHOR]

Invertible color-to-grayscale conversion is a research issue in grayscale image colorization, which is a complex problem resulting from information loss. This paper presents an innovative invertible color-to-grayscale conversion idea by independently compressing the chromaticity plane and hiding it into the corresponding luminance plane as a watermark. Since the chromaticity and luminance planes are orthogonal in the proposed method, they can be processed efficiently without mutual influence. By using an efficient lossy compression operation, we can save more chromatic information. By using two high-capacity data hiding techniques (reversible watermarking (RW) and least significant bit (LSB) substitution), we can embed the compressed chromatic information into the luminance plane perfectly. For the purpose of integrality authentication, the luminance plane is hashed as part of the embedded information before the embedding. Experimental results have shown that higher quality of reconstructed color images can be achieved by using RW, but the quality of the synthesized grayscale drops sharply. By using LSB substitution, we can obtain high-quality synthesized grayscale and reconstructed color images simultaneously. Furthermore, we have compared the proposed LSB-based scheme with several recently reported state-of-the-art methods to validate the superiority of the proposed approach. [ABSTRACT FROM AUTHOR]

The Low Complexity Enhancement Video Coding (LCEVC) specification is a recent standard approved by the ISO/IEC JTC 1/SC 29/WG04 (MPEG) Video Coding. The main goal of LCEVC is to provide a standalone toolset for the enhancement of any other existing codec. It works on top of other coding schemes, resulting in a multi-layer video coding technology, but unlike existing scalable video codecs, adds enhancement layers completely independent from the base video. The LCEVC technology takes as input the decoded video at lower resolution and adds up to two enhancement sub-layers of residuals encoded with specialized low-complexity coding tools, such as simple temporal prediction, frequency transform, quantization, and entropy encoding. This paper provides an overview of the main features of the LCEVC standard: high compression efficiency, low complexity, minimized requirements of memory and processing power. [ABSTRACT FROM AUTHOR]

As one of the most important topics in image forensics, resampling detection has developed rapidly in recent years. However, the robustness to JPEG compression is still challenging for most classical spectrum-based methods, since JPEG compression severely degrades the image contents and introduces block artifacts in the boundary of the compression grid. In this article, we propose a method to estimate the upscaling factors on double JPEG compressed images in the presence of image upscaling between the two compressions. We first analyze the spectrum of scaled images and give an overall formulation of how the scaling factors along with the parameters of JPEG compression and image contents influence the appearance of tampering artifacts. The expected positions of five kinds of characteristic peaks are analytically derived. Then, we analyze the features of double JPEG compressed images in the block discrete cosine transform (BDCT) domain and present an inverse scaling strategy for the upscaling factor estimation with a detailed proof. Finally, a fusion method is proposed that through frequency-domain analysis, a candidate set of upscaling factors is given, and through analysis in the BDCT domain, the optimal estimation from all candidates is determined. The experimental results demonstrate that the proposed method outperforms other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Screen content data, such as computer-generated photographs, desktop sharing, remote education, video game streaming and screenshot, is one of the most popular visual information carriers in Internet of Video Things. Although lossless compression can guarantee high quality of service for these screen content based industrial applications, it also causes considerable storage space and transmission bandwidth issues. To alleviate these challenges, in this article, we present a visually quasi-lossless coding approach to control the compression distortion below visibility threshold in the human visual system. Specifically, to better quantify the visual redundancy for screen content data, a new visibility threshold method is designed by incorporating blur sensitivity and oblique correction effects. Then, an end-to-end mapping between the visibility threshold and quality control factor is learned and represented as a deep convolutional neural network. The experimental results demonstrate that the proposed method saves the average encoding bits up to 23.15% compared with the latest scheme under the same perceptual quality. [ABSTRACT FROM AUTHOR]

The picturewise just noticeable difference (PJND) for a given image, compression scheme, and subject is the smallest distortion level that the subject can perceive when the image is compressed with this compression scheme. The PJND can be used to determine the compression level at which a given proportion of the population does not notice any distortion in the compressed image. To obtain accurate and diverse results, the PJND must be determined for a large number of subjects and images. This is particularly important when experimental PJND data are used to train deep learning models that can predict a probability distribution model of the PJND for a new image. To date, such subjective studies have been carried out in laboratory environments. However, the number of participants and images in all existing PJND studies is very small because of the challenges involved in setting up laboratory experiments. To address this limitation, we develop a framework to conduct PJND assessments via crowdsourcing. We use a new technique based on slider adjustment and a flicker test to determine the PJND. A pilot study demonstrated that our technique could decrease the study duration by 50% and double the perceptual sensitivity compared to the standard binary search approach that successively compares a test image side by side with its reference image. Our framework includes a robust and systematic scheme to ensure the reliability of the crowdsourced results. Using 1,008 source images and distorted versions obtained with JPEG and BPG compression, we apply our crowdsourcing framework to build the largest PJND dataset, KonJND-1k (Konstanz just noticeable difference 1k dataset). A total of 503 workers participated in the study, yielding 61,030 PJND samples that resulted in an average of 42 samples per source image. The KonJND-1k dataset is available at http://database.mmsp-kn.de/konjnd-1k-database.html [ABSTRACT FROM AUTHOR]

This paper presents the overview and rationale behind the Decoder-Side Depth Estimation (DSDE) mode of the MPEG Immersive Video (MIV) standard, using the Geometry Absent profile, for efficient compression of immersive multiview video. A MIV bitstream generated by an encoder operating in the DSDE mode does not include depth maps. It only contains the information required to reconstruct them in the client or in the cloud: decoded views and metadata. The paper explains the technical details and techniques supported by this novel MIV DSDE mode. The description additionally includes the specification on Geometry Assistance Supplemental Enhancement Information which helps to reduce the complexity of depth estimation, when performed in the cloud or at the decoder side. The depth estimation in MIV is a non-normative part of the decoding process, therefore, any method can be used to compute the depth maps. This paper lists a set of requirements for depth estimation, induced by the specific characteristics of the DSDE. The depth estimation reference software, continuously and collaboratively developed with MIV to meet these requirements, is presented in this paper. Several original experimental results are presented. The efficiency of the DSDE is compared to two MIV profiles. The combined non-transmission of depth maps and efficient coding of textures enabled by the DSDE leads to efficient compression and rendering quality improvement compared to the usual encoder-side depth estimation. Moreover, results of the first evaluation of state-of-the-art multiview depth estimators in the DSDE context, including machine learning techniques, are presented. [ABSTRACT FROM AUTHOR]

JPEG steganalysis aims to detect stego JPEG images. For some robust steganography methods, in order to enhance stego images’ robustness of resisting JPEG recompression from lossy channel such SNS or photo sharing websites, steganographer may intentionally recompress cover image several times with quantization matrix of targeted channel, which thereby make it possible to transmit stego data in such channel for better disguise. In addition, there are huge number of cover JPEG images may be recompressed for various reasons, such as processing by some tools. Thus a better steganalysis method for such images is needed. In this paper, we investigate the steganalysis method for images recompressed with the same quantization matrix, namely, discriminate recompressed JPEG cover images and its stego images. We present some observed phenomenon on recompressed JPEG images, and design methods to enhance the sensitivity of feature based and deep model based steganalysis methods for this task. To verify their effectiveness with different acquisition of recompression prior-knowledge, we conduct experiments in various settings including conventional setting and mixing samples of different recompressing times in training. Their results demonstrate that the proposed method can notably improve detection accuracy on recompressed JPEG images. [ABSTRACT FROM AUTHOR]

In this paper, we compress convolutional neural network (CNN) weights post-training via transform quantization. Previous CNN quantization techniques tend to ignore the joint statistics of weights and activations, producing sub-optimal CNN performance at a given quantization bit-rate, or consider their joint statistics during training only and do not facilitate efficient compression of already trained CNN models. We optimally transform (decorrelate) and quantize the weights post-training using a rate–distortion framework to improve compression at any given quantization bit-rate. Transform quantization unifies quantization and dimensionality reduction (decorrelation) techniques in a single framework to facilitate low bit-rate compression of CNNs and efficient inference in the transform domain. We first introduce a theory of rate and distortion for CNN quantization and pose optimum quantization as a rate–distortion optimization problem. We then show that this problem can be solved using optimal bit-depth allocation following decorrelation by the optimal End-to-end Learned Transform (ELT) we derive in this paper. Experiments demonstrate that transform quantization advances the state of the art in CNN compression in both retrained and non-retrained quantization scenarios. In particular, we find that transform quantization with retraining is able to compress CNN models such as AlexNet, ResNet and DenseNet to very low bit-rates (1–2 bits). [ABSTRACT FROM AUTHOR]

Omnidirectional video is becoming increasingly popular among viewers, but its delivery requires considerable amount of network bandwidth. Today’s streaming services are transmitting the full spatial angle of omnidirectional videos, although most of the transmitted content is not utilized. Due to both limited bandwidth availability and its dynamic fluctuations, adaptive delivery solutions play a key role in supporting high user quality streaming of omnidirectional videos. This paper describes research which extends the MPEG-DASH Spatial Relationship Description by adding scalable video encoding to spatial tiling. It proposes a novel tile-layering based gaze adaptation algorithm for omnidirectional video delivery and employs it in conjunction with multiple tiling schemes. The benefits of the proposed algorithm with diverse tiling schemes are evaluated objectively in terms of bandwidth savings and adaptation latency. The results show a reduction of network bandwidth requirements to about 30% of the original bandwidth value with a low processing latency of 70.87ms. [ABSTRACT FROM AUTHOR]

Due to the large amount of data involved in the three-dimensional (3D) LiDAR point clouds, point cloud compression (PCC) becomes indispensable to many real-time applications. In autonomous driving of connected vehicles for example, point clouds are constantly acquired along the time and subjected to be compressed. Among the existing PCC methods, very few of them have effectively removed the temporal redundancy inherited in the point clouds. To address this issue, a novel lossy LiDAR PCC system is proposed in this paper, which consists of the inter-frame coding and the intra-frame coding. For the former, a deep-learning approach is proposed to conduct bi-directional frame prediction using an asymmetric residual module and 3D space-time convolutions; the proposed network is called the bi-directional prediction network (BPNet). For the latter, a novel range-adaptive floating-point coding (RAFC) algorithm is proposed for encoding the reference frames and the B-frame prediction residuals in the 32-bit floating-point precision. Since the pixel-value distribution of these two types of data are quite different, various encoding modes are designed for providing adaptive selection. Extensive simulation experiments have been conducted using multiple point cloud datasets, and the results clearly show that our proposed PCC system consistently outperforms the state-of-the-art MPEG G-PCC in terms of data fidelity and localization, while delivering real-time performance. [ABSTRACT FROM AUTHOR]

Image compression is one of the most fundamental techniques and commonly used applications in the image and video processing field. Earlier methods built a well-designed pipeline, and efforts were made to improve all modules of the pipeline by handcrafted tuning. Later, tremendous contributions were made, especially when data-driven methods revitalized the domain with their excellent modeling capacities and flexibility in incorporating newly designed modules and constraints. Despite great progress, a systematic benchmark and comprehensive analysis of end-to-end learned image compression methods are lacking. In this paper, we first conduct a comprehensive literature survey of learned image compression methods. The literature is organized based on several aspects to jointly optimize the rate-distortion performance with a neural network, i.e., network architecture, entropy model and rate control. We describe milestones in cutting-edge learned image-compression methods, review a broad range of existing works, and provide insights into their historical development routes. With this survey, the main challenges of image compression methods are revealed, along with opportunities to address the related issues with recent advanced learning methods. This analysis provides an opportunity to take a further step towards higher-efficiency image compression. By introducing a coarse-to-fine hyperprior model for entropy estimation and signal reconstruction, we achieve improved rate-distortion performance, especially on high-resolution images. Extensive benchmark experiments demonstrate the superiority of our model in rate-distortion performance and time complexity on multi-core CPUs and GPUs. [ABSTRACT FROM AUTHOR]

Channel state information (CSI) plays a crucial role in the capacity of multiple-input and multiple-output systems, but CSI feedback occupies substantial precious transmission resources in frequency-division duplexing (FDD) systems. In this work, we propose a data hiding-based CSI feedback framework, namely, EliCsiNet, to eliminate the CSI feedback overhead in FDD systems through deep learning. The key idea is to hide/superimpose CSI in transmitted messages (e.g., images) with no transmission resource occupation and few effects on message semantics. Concretely, we introduce a novel neural network framework in which the user extracts and hides CSI features in images, and the base station recovers the CSI from the transmitted images. However, the essential source coding (e.g., JPEG compression) before data transmission causes two problems in the proposed EliCsiNet framework when applied in practical systems. First, the compression inevitably disturbs the information of the hidden CSI in images and affects the CSI reconstruction accuracy. Therefore, a two-stage separable training strategy, which includes coding-free end-to-end and coding-aware decoder-only training, is adopted to reduce these effects. Second, the bit length of the images coded via JPEG is unpredictable and uncontrollable, and CSI superimposition may lead to an increase in the bit length of the coded images. To avoid this issue, we divide a full image into several sub-blocks and select the one with the smallest length increment. Image entropy is also introduced to accelerate block selection. Simulation results demonstrate that the proposed EliCsiNet framework can eliminate the CSI feedback overhead with few effects on the features properties of transmitted images, including image quality and bit length. [ABSTRACT FROM AUTHOR]

Recently, data of multimedia performs an exponentially blowing tendency, saturating daily life of humans. Various modalities of data, includes images, texts and video, plays important role in different aspects and has wide. However, the key problem of utilizing data of large scale is cost of processing and massive storage. Therefore, for efficient communications and for economical storage requires effective techniques of data compression to reduce the volume of data. Speech coding is a main problem in the area of digital speech processing. The process of converting the voice signals into a more compressed form is speech coding. In this work, we demonstrate that a DCT with a chaotic system combined with run-length coding can be utilized to implement speech coding of very low bit-rate with high reconstruction quality. Experimental result show that compression ratio is about 13% when implemented on Librispeech dataset. [ABSTRACT FROM AUTHOR]

Frame memory compression (FMC) saves power by reducing the data bandwidth for accessing reference frames of video coding standards in the external memory. However, since the continuously accumulated data loss degrades the performance of video coding standards, the quality of the reconstructed reference frame must be improved. Previous lossy frame memory compression methods focused on processing speed or simplicity, but their performance remains limited in terms of the quality of the reconstructed frame. In this study, a Huffman coding with a fixed codebook designed to considerably reduce the latency caused by arithmetic coding in lossless compression schemes is proposed. A novel lossy frame memory compression technique based on the Hadamard transform that does not require additional memory access using a lookup table is also proposed. The result of experiments on several reconstructed videos preprocessed with HEVC show that the proposed method improves BDPSNR by 10.7 dB and BDBR by 40.4% compared with the DWT+SPIHT‐based compression technique which is widely used for lossy frame memory compression. [ABSTRACT FROM AUTHOR]

The great success of deep learning has boosted the fast development of video quality enhancement. However, existing methods mainly focus on enhancing the objective quality of compressed video, and ignore their perceptual quality that plays a key role in determining quality of experience (QoE) of videos. In this paper, we aim at enhancing the perceptual quality of compressed video. Our main observation is that perceptual quality enhancement mostly relies on recovering the high-frequency details with fine textures. Accordingly, we propose a novel generative adversarial network (GAN) based on multi-level wavelet packet transform (WPT), which is called multi-level wavelet-based GAN+ (MW-GAN+), to exploit high-frequency details for enhancing the perceptual quality of compressed video. In MW-GAN+, we first propose a multi-level wavelet pixel-adaptive (MWP) module to extract temporal information across video frames, such that frame similarity can be utilized in recovering high-frequency details. Then, a wavelet reconstruction network, consisting of wavelet-dense residual blocks (WDRB), is developed to recover high-frequency details in a multi-level manner for enhanced frame reconstruction. Finally, we develop a 3D discriminator to encourage temporal coherence with a 3D-CNN based architecture. Experimental results demonstrate the superiority of our method over state-of-the-art methods in enhancing the perceptual quality of compressed video. Our code is available at https://github.com/IceClear/MW-GAN. [ABSTRACT FROM AUTHOR]

Although it has been surpassed by many subsequent coding standards, JPEG occupies a large storage share of the current data hosting service. To reduce the storage costs, DropBox proposed a secondary lossless compression algorithm, Lepton, to further improve the compression rate of JPEG images. However, the bloated probability models defined by Lepton severely restrict its throughput and energy efficiency. To solve this problem, we construct an access probability-based hash function for the probability models, and then propose a hardware-friendly memory optimization method by combining the proposed hash function with N-way Set-Associative unit. Besides, we also propose a synchronization mechanism for the serial accessing probability model, so that the syntax elements can be processed in parallel without changing the resulting bitstream. After that, we implement a high throughput, high energy efficiency, and low-cost Lepton hardware encoder. To the best of our knowledge, this is the first hardware implementation of transparent image recompression. The synthesis result shows that the proposed hardware structure reduces the total area of the probability models by 70.97%. Compared with DropBox’s software solution, the throughput and the energy efficiency of the proposed Lepton hardware encoder are increased by 63 and 1398 times on average. In terms of manufacturing cost, the proposed Lepton hardware encoder is also lower than the general-purpose CPU used by DropBox. [ABSTRACT FROM AUTHOR]

In image forensics, quantization step estimation plays a crucial role in revealing the JPEG compression history. This study focuses on the estimation for images that have been JPEG compressed and re-saved in lossless formats. Several effective methods have been developed, whereas the performance on small sized images still needs to be improved. In this paper, a novel JPEG quantization step estimation method with low complexity and high efficiency is proposed. First, for a given decompressed image, as the special shape of its DCT coefficients distribution is highly related to the quantization step, a function of the candidate step taking a similar shape is designed. Then, the quantization step is determined as the candidate leading to the maximum response of the designed function on the probability density function of DCT coefficients. The relation between the maximum response and the quantization step is verified mathematically, which provides a theoretical basis for the proposed method. In addition, to further improve the estimation performance, two fine adjustment procedures are adopted. Experimental results demonstrate that the proposed method outperforms some state-of-the-art works, especially on small sized images. [ABSTRACT FROM AUTHOR]

Image splicing can be easily used for illegal activities such as falsifying propaganda for political purposes and reporting false news, which may result in negative impacts on society. Hence, it is highly required to detect spliced images and localize the spliced regions. In this work, we propose a multi-task squeeze and excitation network (SE-Network) for splicing localization. The proposed network consists of two streams, namely label mask stream and edge-guided stream, both of which adopt convolutional encoder-decoder architecture. The information from the edge-guided stream is transmitted to the label mask stream for enhancing the discrimination of features between the spliced and host regions. This work has three main contributions. First, image edges, along with label masks and mask edges, are exploited to supply more comprehensive supervision for the localization of spliced regions. Second, the low-level feature maps extracted from shallow layers are fused with the high-level feature maps from deep layers to provide more reliable feature for splicing localization. Finally, several squeeze and excitation attention modules are incorporated into the network to recalibrate the fused features to enhance the feature expression. Extensive experiments show that the proposed multi-task SE-Network outperforms existing splicing localization methods evidently on two synthetic splicing datasets and four benchmark splicing datasets. [ABSTRACT FROM AUTHOR]

Constant bit rate (CBR) videos are widely used in streaming playback applications. However, the image quality of the CBR video is often unstable, especially for scenes with large motion. To this end, we design a new model to represent the distortion of High Efficiency Video Coding (HEVC) constant bit rate video, and propose a neural network for a constant bit rate video quality enhancement (CBREN). We propose a dual-domain restoration module (DRM) to jointly learn the prior knowledge in the pixel domain and the frequency domain. To address the degradation resulting from compression, we propose a two-step quantization degradation estimation strategy. The Inverse DCT (IDCT) Translation Unit (ITU) is used to constrain the quantization table of the constant bit rate video to a suitable range, and the Dynamic Alpha Unit (DAU) is used to fine-tune the quantization table according to the content of each frame. In order to effectively reduce the block distortion of different sizes produced in the compression process, we adopt a multi-scale network. Extensive experiments show that our approach can greatly enhance the quality of CBR compressed video. Moreover, our method can also be applied to constant quantization parameter (CQP) video enhancement tasks, and is certainly superior to existing methods. [ABSTRACT FROM AUTHOR]

Light field imaging, which captures both spatial and angular information, improves user immersion by enabling post-capture actions, such as refocusing and changing view perspective. However, light fields represent very large volumes of data with a lot of redundancy that coding methods try to remove. State-of-the-art coding methods indeed usually focus on improving compression efficiency and overlook other important features in light field compression such as scalability. In this paper, we propose a novel light field image compression method that enables (i) viewport scalability, (ii) quality scalability, (iii) spatial scalability, (iv) random access, and (v) uniform quality distribution among viewports, while keeping compression efficiency high. To this end, light fields in each spatial resolution are divided into sequential viewport layers, and viewports in each layer are encoded using the previously encoded viewports. In each viewport layer, the available viewports are used to synthesize intermediate viewports using a video interpolation deep learning network. The synthesized views are used as virtual reference images to enhance the quality of intermediate views. An image super-resolution method is applied to improve the quality of the lower spatial resolution layer. The super-resolved images are also used as virtual reference images to improve the quality of the higher spatial resolution layer. The proposed structure also improves the flexibility of light field streaming, provides random access to the viewports, and increases error resiliency. The experimental results demonstrate that the proposed method achieves a high compression efficiency and it can adapt to the display type, transmission channel, network condition, processing power, and user needs. [ABSTRACT FROM AUTHOR]

Codecs using spectral-spatial transforms efficiently compress raw camera images captured with a color filter array (CFA-sampled raw images) by changing their RGB color space into a decorrelated color space. This study describes two types of spectral-spatial transform, called extended Star-Tetrix transforms (XSTTs), and their edge-aware versions, called edge-aware XSTTs (EXSTTs), with no extra bits (side information) and little extra complexity. They are obtained by (i) extending the Star-Tetrix transform (STT), which is one of the latest spectral-spatial transforms, to a new version of our previously proposed wavelet-based spectral-spatial transform and a simpler version; (ii) considering that each 2D predict step of the wavelet transform is a combination of two 1D diagonal or horizontal-vertical transforms; (iii) weighting the transforms along the edge directions in the images. Compared with XSTTs, the EXSTTs can decorrelate CFA-sampled raw images well: they reduce the difference in energy between the two green components by about 3.38–30.08 % for high-quality camera images and 8.97–14.47 % for mobile phone images. The experiments on JPEG 2000-based lossless and lossy compression of CFA-sampled raw images show better performance than conventional methods. For high-quality camera images, the XSTTs/EXSTTs produce results equal to or better than the conventional methods: especially for images with many edges, the type-I EXSTT improves them by about 0.03–0.19 bpp in average lossless bitrate and the XSTTs improve them by about 0.16–0.96 dB in average Bjøntegaard delta peak signal-to-noise ratio. For mobile phone images, our previous work perform the best, whereas the XSTTs/EXSTTs show similar trends to the case of high-quality camera images. [ABSTRACT FROM AUTHOR]

Adaptive Fourier decomposition (AFD) is a newly developed signal processing tool that can adaptively decompose any single signal using a Szegö kernel dictionary. To process multiple signals, a novel stochastic-AFD (SAFD) theory was recently proposed. The innovation of this study is twofold. First, a SAFD-based general multi-signal sparse representation learning algorithm is designed and implemented for the first time in the literature, which can be used in many signal and image processing areas. Second, a novel SAFD based image compression framework is proposed. The algorithm design and implementation of the SAFD theory and image compression methods are presented in detail. The proposed compression methods are compared with 13 other state-of-the-art compression methods, including JPEG, JPEG2000, BPG, and other popular deep learning-based methods. The experimental results show that our methods achieve the best balanced performance. The proposed methods are based on single image adaptive sparse representation learning, and they require no pre-training. In addition, the decompression quality or compression efficiency can be easily adjusted by a single parameter, that is, the decomposition level. Our method is supported by a solid mathematical foundation, which has the potential to become a new core technology in image compression. [ABSTRACT FROM AUTHOR]

Point cloud is a set of three-dimensional points in arbitrary order, which is a popular representation of 3D scene in autonomous navigation and immersive applications in recent years. Compression becomes an inevitable issue due to the huge data volume of point cloud. In order to effectively compress attributes of those points, proper reordering is important. The existing voxel-based point cloud attributes compression scheme uses a naive scan for points reordering. In this paper, we theoretically analyzed 3C properties of point cloud, i.e., Compactness, Clustering and Correlation, of different scan-orders defined by different space filling curves and disclosed that the Hilbert curve can provide the best spatial correlation preservation compared with Z-order and Gray-coded curves. It is also statistically verified that the Hilbert curve always has the best ability of attributes correlation preservation for point clouds with different sparsity. We also proposed a fast and iterative Hilbert address code generation method to implement points reordering. The Hilbert scan-order could be combined with various point cloud attribute coding methods. Experiments show that the correlation preservation feature of the proposed scan-order can bring us 6.1% and 6.5% coding gain for prediction and transform coding, respectively. [ABSTRACT FROM AUTHOR]

Existing compression methods typically focus on the removal of signal-level redundancies, while the potential and versatility of decomposing visual data into compact conceptual components still lack further study. To this end, we propose a novel conceptual compression framework that encodes visual data into compact structure and texture representations, then decodes in a deep synthesis fashion, aiming to achieve better visual reconstruction quality, flexible content manipulation, and potential support for various vision tasks. In particular, we propose to compress images by a dual-layered model consisting of two complementary visual features: 1) structure layer represented by structural maps and 2) texture layer characterized by low-dimensional deep representations. At the encoder side, the structural maps and texture representations are individually extracted and compressed, generating the compact, interpretable, inter-operable bitstreams. During the decoding stage, a hierarchical fusion GAN (HF-GAN) is proposed to learn the synthesis paradigm where the textures are rendered into the decoded structural maps, leading to high-quality reconstruction with remarkable visual realism. Extensive experiments on diverse images have demonstrated the superiority of our framework with lower bitrates, higher reconstruction quality, and increased versatility towards visual analysis and content manipulation tasks. [ABSTRACT FROM AUTHOR]

Common representations of light fields use four-dimensional data structures, where a given pixel is closely related not only to its spatial neighbours within the same view, but also to its angular neighbours, co-located in adjacent views. Such structure presents increased redundancy between pixels, when compared with regular single-view images. Then, these redundancies are exploited to obtain compressed representations of the light field, using prediction algorithms specifically tailored to estimate pixel values based on both spatial and angular references. This paper proposes new encoding schemes which take advantage of the four-dimensional light field data structures to improve the coding performance of Minimum Rate Predictors. The proposed methods expand previous research on lossless coding beyond the current state-of-the-art. The experimental results, obtained using both traditional datasets and others more challenging, show bit-rate savings no smaller than 10%, when compared with existing methods for lossless light field compression. [ABSTRACT FROM AUTHOR]

Transform coding removes redundancy by de-correlating the residual data, which plays a crucial role in video coding. Since different transform cores can adapt to different video content and residual data, multiple cores transform has been proposed to improve the coding performance. However, the overhead for representing the transform type is inevitable. This paper intensively studies the statistical characteristics of transform coefficients. Theoretical analysis shows that the implicit method in the Implicitly Selected Transform (IST) is superior to the explicit signalling method. As such, we propose the parity adjustment scheme which seamlessly cooperates with the rate-distortion optimization quantization for IST. Furthermore, we propose two combination methods, size-based and number-based, to optimize IST. Moreover, a restriction region of IST is applied to reduce the complexity. Experimental results on HPM-6.0, which is the reference software of the AVS3 video coding standard, show that our proposed method can achieve 1.76% and 0.76% BD-Rate savings on average under AI and RA configurations, respectively, along with negligible decoding time variations. The comparison results between the proposed method and explicit signalling method illustrate that the proposed method can achieve better coding gain than the latter. Our method has been adopted in AVS3. [ABSTRACT FROM AUTHOR]