Search

The Joint Photographic Experts Group (JPEG) AI learning-based image coding system is an ongoing joint standardization effort between International Organization for Standardization (ISO), International Electrotechnical Commission (IEC), and International Telecommunication Union - Telecommunication Sector (ITU-T) for the development of the first image coding standard based on machine learning (a subset of artificial intelligence), offering a single stream, compact compressed domain representation, targeting both human visualization and machine consumption. The main motivation for this upcoming standard is the excellent performance of tools based on deep neural networks, in image coding, computer vision, and image processing tasks. The JPEG AI aims to develop an image coding standard addressing the needs of a wide range of applications such as cloud storage, visual surveillance, autonomous vehicles and devices, image collection storage and management, live monitoring of visual data, and media distribution. This article presents and discusses the rationale behind the JPEG AI vision, notably how this new standardization initiative aims to shape the future of image coding, through relevant application-driven use cases. The JPEG AI requirements, the JPEG AI history, and current status are also presented, offering a glimpse of the development of the first learning-based image coding standard.

As the coronavirus disease 2019 (COVID-19) spreads around the world, industrial automated medical diagnosis systems have been developed, which complete a large amount of medical diagnosis work through computed tomography (CT) images. In these systems, how to quickly store and transmit such a large amount of CT image information has important research significance. In this paper, a more targeted COVID-19 chest CT image codec is proposed to make image data not only occupy less space but also have higher image quality. First, the bilateral lung contours are extracted to calculate the position information of the region of interest (ROI). Then, a CT image is classified into four types of non-uniform image blocks according to the characteristics of COVID-19 chest CT images and ROI position information. Next, a series of new transformations are proposed for more efficient transform coding. Finally, a flexible quantization strategy is proposed for the adaptive quantization part. In the experiments, the proposed method is superior to some existing methods with similar computational complexity. At the same bit rate, it significantly improves the image quality. This means that chest CT images can still be used for disease diagnosis while taking up less space. In addition, because of the low computational complexity of the proposed method, it can be more easily embedded into the CT equipment with low computational power. IEEE

Compactly representing visual information plays a fundamental role in optimizing the ultimate utility of myriad visual data-centered applications. Numerous approaches have been proposed to efficiently compress the texture and visual features for human visual perception and machine intelligence, respectively; however, much less work has been dedicated to studying the interactions between them. Here, we investigate the integration of feature and texture compression and show that a universal and collaborative visual information representation can be achieved in a hierarchical way. In particular, we study feature and texture compression in a scalable coding framework, where the base layer serves as the deep learning feature and the enhancement layer targets to perfectly reconstruct the texture. Based on the strong generative capability of deep neural networks, the gap between the base feature layer and enhancement layer is further filled with feature-level texture reconstruction, with the goal of further constructing texture representations from features. As such, the residuals between the original and reconstructed texture could be further conveyed in the enhancement layer. To improve the efficiency of the proposed framework, the base layer neural network is trained in a multitask manner such that the learned features enjoy both high-quality reconstruction and high-accuracy analysis. The framework and optimization strategies are further applied in face image compression, and promising coding performance has been achieved in terms of both rate-fidelity and rate-accuracy evaluations.

Recently, more interest in the different plenoptic formats, including digital holograms, has emerged. Aside from other challenges that several steps of the holographic pipeline, from digital acquisition to display, have to face, visual quality assessment of compressed holograms is particularly demanding due to the distinct nature of this 3D image modality when compared to regular 2D imaging. There are few studies on holographic data quality assessment, particularly with respect to the perceptual effects of lossy compression. This work aims to study the quality evaluation of digital hologram reconstructions presented on regular 2D displays in the presence of compression distortions. As there is no established or generally agreed on compression methodology for digital hologram compression on the hologram plane with available implementations, a set of state-of-the-art compression codecs, namely, HEVC, AV1, and JPEG2000, were used for compression of the digital holograms on the object plane. Both computer-generated and optically generated holograms were considered. Two subjective tests were conducted to evaluate distortions caused by compression. The first subjective test was conducted on the reconstructed amplitude images of central views, while the second test was conducted on pseudovideos generated from the reconstructed amplitudes of different views. The subjective quality assessment was based on mean opinion scores. A selection of objective quality metrics was evaluated, and their correlations with mean opinion scores were computed. The VIFp metrics appeared to have the highest correlation.

Traditional video compression approaches build upon the hybrid coding framework with motion-compensated prediction and residual transform coding. In this paper, we propose the first end-to-end deep video compression framework to take advantage of both the classical compression architecture and the powerful non-linear representation ability of neural networks. Our framework employs pixel-wise motion information, which is learned from an optical flow network and further compressed by an auto-encoder network to save bits. The other compression components are also implemented by the well-designed networks for high efficiency. All the modules are jointly optimized by using the rate-distortion trade-off and can collaborate with each other. More importantly, the proposed deep video compression framework is very flexible and can be easily extended by using lightweight or advanced networks for higher speed or better efficiency. We also propose to introduce the adaptive quantization layer to reduce the number of parameters for variable bitrate coding. Comprehensive experimental results demonstrate the effectiveness of the proposed framework on the benchmark datasets.

In the past decade, the development of transform coding techniques has achieved significant progress and several advanced transform tools have been adopted in the new generation Versatile Video Coding (VVC) standard. In this paper, a brief history of transform coding development during VVC standardization is presented, and the transform coding tools in the VVC standard are described in detail together with their initial design, incremental improvements and implementation aspects. To improve coding efficiency, four new transform coding techniques are introduced in VVC, which are namely Multiple Transform Selection (MTS), Low-Frequency Non-separable Secondary Transform (LFNST) and Sub-Block Transform (SBT), as well as a large (64-point) type-2 DCT. The experimental results on VVC reference software (VTM-9.0) show that average 4.5% and 3.6% overall coding gain can be achieved by the VVC transform coding tools for All Intra and Random Access configurations, respectively.

This paper presents the intra prediction and mode coding of the Versatile Video Coding (VVC) standard. This standard was collaboratively developed by the Joint Video Experts Team (JVET). It follows the traditional architecture of a hybrid block-based codec that was also the basis of previous standards. Almost all intra prediction features of VVC either contain substantial modifications in comparison with its predecessor H.265/HEVC or were newly added. The key aspects of these tools are the following: 65 angular intra prediction modes with block shape-adaptive directions and 4-tap interpolation filters are supported as well as the DC and Planar mode, Position Dependent Prediction Combination is applied for most of these modes, Multiple Reference Line Prediction can be used, an intra block can be further subdivided by the Intra Subpartition mode, Matrix-based Intra Prediction is supported, and the chroma prediction signal can be generated by the Cross Component Linear Model method. Finally, the intra prediction mode in VVC is coded separately for luma and chroma. Here, a Most Probable Mode list containing six modes is applied for luma. The individual compression performance of tools is reported in this paper. For the full VVC intra codec, a bitrate saving of 25% on average is reported over H.265/HEVC using an objective metric. Significant subjective benefits are illustrated with specific examples.

Reversible Data Hiding (RDH) in JPEG images is valuable for many applications, such as archive management and image authentication. Recently, there emerged a lot of related works for JPEG RDH, however, the current methods utilize the histogram-shifting-based framework in a constant distortion metric, which does not consider the property of DCT coefficients whose modification distortion is miscellaneous with respect to DCT frequency. In this paper, we propose a novel JPEG RDH scheme under multi-distortion metric. At first, the modification distortion is defined as the impact in the spatial domain caused by modifying DCT coefficients. Different from previous works, here we select coefficients with values ±1, ±2 as cover, which will not cause extra bitrate-expansion and decrease the shift distortion. By minimizing both the modification distortion and shift distortion, a heuristic block selecting strategy is proposed. With the selected coefficients and the corresponding distortion, we use recursive histogram modification under the inconsistent-distortion metric for message embedding. The experimental results show that the proposed method can effectively improve visual quality as well as bring small bitstream expansion. Additionally, the undetectability of the proposed method outperforms those of the existing methods.

In this article, a survey of the point cloud compression (PCC) methods by organizing them with respect to the data structure, coding representation space, and prediction strategies is presented. Two paramount families of approaches reported in the literature—the projection- and octree-based methods—are proven to be efficient for encoding dense and sparse point clouds, respectively. These approaches are the pillars on which the Moving Picture Experts Group Committee developed two PCC standards published as final international standards in 2020 and early 2021, respectively, under the names: video-based PCC and geometry-based PCC. After surveying the current approaches for PCC, the technologies underlying the two standards are described in detail from an encoder perspective, providing guidance for potential standard implementors. In addition, experiment evaluations in terms of compression performances for both solutions are provided.

This article introduces the ISO/IEC MPEG Immersive Video (MIV) standard, MPEG-I Part 12, which is undergoing standardization. The draft MIV standard provides support for viewing immersive volumetric content captured by multiple cameras with six degrees of freedom (6DoF) within a viewing space that is determined by the camera arrangement in the capture rig. The bitstream format and decoding processes of the draft specification along with aspects of the Test Model for Immersive Video (TMIV) reference software encoder, decoder, and renderer are described. The use cases, test conditions, quality assessment methods, and experimental results are provided. In the TMIV, multiple texture and geometry views are coded as atlases of patches using a legacy 2-D video codec, while optimizing for bitrate, pixel rate, and quality. The design of the bitstream format and decoder is based on the visual volumetric video-based coding (V3C) and video-based point cloud compression (V-PCC) standard, MPEG-I Part 5.

Proceedings of the IEEE 109(9), 1463-1493 (2021). doi:10.1109/JPROC.2020.3043399, Published by Institute of Electrical and Electronics Engineers, New York, NY [u.a.]

Recent learned image coding has emerged with superior efficiency to conventional methods. It, however, is criticized for its complexity-exhaustive deep neural network (DNN) architectures, especially on resource-constrained mobile platforms. Thus we devise a two-stage approach: first, a range pre-processing is applied to constrain the dynamic range of feature map activation by leveraging its sparsity nature with densely clustered distribution, then a layer-wise range-adaptive quantization for convolutional parameter (e.g., weight, bias), and simple yet efficient linear scaling and range-dependent normalization for activation are executed, leading to a fully fixed-point inference architecture. All arithmetic operations and associated data tensors are processed using low-bit-width fixed-point numbers, yielding significant reductions of the computational complexity, memory space, and the elimination of platform-dependent inconsistency induced by floating-point operations. We first exemplify such fixed-point inference in a DNN-based image decoder, showing the comparable coding efficiency with its native floating-point model, against the same anchor using the High-Efficiency Video Coding (HEVC)-based intra image coder. We also extend proposed approach to the super-resolution network for learned resolution scaling-based video streaming, and VGG network-based classification tasks, both of which present negligible performance loss. These evidence the generalization of our approach for efficient DNN processing of various tasks. All materials are made publicly accessible at http://njuvision.github.io/fixed-point/ .

During recent years, there have been product launches and research for enabling immersive audio–visual media experiences. For example, a variety of head-mounted displays and 360° cameras are available in the market. To facilitate interoperability between devices and media system components by different vendors, the Moving Picture Experts Group (MPEG) developed the Omnidirectional MediA Format (OMAF), which is arguably the first virtual reality (VR) system standard. OMAF is a storage and streaming format for omnidirectional media, including 360° video and images, spatial audio, and associated timed text. This article provides a comprehensive overview of OMAF.

The file attached to this record is the author's final peer reviewed version. The Satisfied User Ratio (SUR) for a given distortion level is the fraction of subjects that cannot perceive a quality difference between the original image and its compressed version. By predicting the SUR, one can determine the highest distortion level which allows to save bit rate while guaranteeing a good visual quality. We propose the first method to predict the SUR for symmetrically and asymmetrically compressed stereoscopic images. Unlike SUR prediction techniques for 2D images and videos, our method exploits the properties of binocular vision. We first extract features that characterize image quality and image content. Then, we use gradient boosting decision trees to reduce the number of features and train a regression model that learns a mapping function from the features to the SUR values. Experimental results on the SIAT-JSSI and SIAT-JASI datasets show high SUR prediction accuracy for H.265 All-Intra and JPEG2000 symmetrically and asymmetrically compressed stereoscopic images.

Most current reversible data hiding (RDH) techniques are designed for uncompressed images. However, JPEG images are more commonly used in our daily lives. Up to now, several RDH methods for JPEG images have been proposed, yet few of them investigated the adaptive data embedding as the lack of accurate measurement for the embedding distortion. To realize adaptive embedding and optimize the embedding performance, in this article, a novel RDH scheme for JPEG images based on multiple histogram modification (MHM) and rate-distortion optimization is proposed. Firstly, with selected coefficients, the RDH for JPEG images is generalized into a MHM embedding framework. Then, by estimating the embedding distortion, the rate-distortion model is formulated, so that the expansion bins can be adaptively determined for different histograms and images. Finally, to optimize the embedding performance in real time, a greedy algorithm with low computation complexity is proposed to derive the nearly optimal embedding efficiently. Experiments show that the proposed method can yield better embedding performance compared with state-of-the-art methods in terms of both visual quality and file size preservation.

Social networks are everywhere and currently transmitting very large messages. As a result, transmitting secret messages in such an environment is worth researching. However, the images used in transmitting messages are usually compressed with a JPEG compression channel, which is lossy and damages the transmitted data. Therefore, to prevent secret messages from being damaged, a robust JPEG steganography is urgently needed. In this paper, a secure robust JPEG steganographic scheme based on an autoencoder with an adaptive BCH encoding (Bose-Chaudhuri-Hocquenghem encoding) is proposed. In particular, the autoencoder is first pretrained to fit the transformation relationship between the JPEG image before and after compression by the compression channel. In addition, the BCH encoding is adaptively utilized according to the content of cover image to decrease the error rate of secret message extraction. The DCT (Discrete Cosine Transformation) coefficient adjustment based on practical JPEG channel characteristics further improves the robustness and statistical security. Comparisons with prior state-of-the-art schemes demonstrate that the proposed robust JPEG steganographic algorithm can provide a more robust performance and statistical security.

Recently, Li et al. proposed an effective non-additive distortion model for JPEG steganography by preserving Block Boundary Continuity (BBC) in the spatial domain. However, the method based on BBC only explored how the modifications of DCT coefficient pairs at the same mode in adjacent blocks will impact on BBC. In this paper, we propose a method to enhance the BBC, called BBC++, by considering the mutual impact on the BBC from all DCT coefficients in adjacent blocks. To do that, we design updating strategies for both covers and costs in multi-round embedding processions. During the embedding, the cover is updated to repair the BBC after embedding and the corresponding costs are updated to keep the BBC from being destroyed in the next embedding. Experimental results show that the BBC++ can better maintain BBC and outperform previous non-additive distortion steganography when resisting modern JPEG steganalyzers.

The emergent ecosystems of intelligent edge devices in diverse Internet-of-Things (IoT) applications, from automatic surveillance to precision agriculture, increasingly rely on recording and processing a variety of image data. Due to resource constraints, e.g., energy and communication bandwidth requirements, these applications require compressing the recorded images before transmission. For these applications, image compression commonly requires: 1) maintaining features for coarse-grain pattern recognition instead of the high-level details for human perception due to machine-to-machine communications; 2) high compression ratio that leads to improved energy and transmission efficiency; and 3) large dynamic range of compression and an easy tradeoff between compression factor and quality of reconstruction to accommodate a wide diversity of IoT applications as well as their time-varying energy/performance needs. To address these requirements, we propose, MAGIC, a novel machine learning (ML)-guided image compression framework that judiciously sacrifices the visual quality to achieve much higher compression when compared to traditional techniques, while maintaining accuracy for coarse-grained vision tasks. The central idea is to capture application-specific domain knowledge and efficiently utilize it in achieving high compression. We demonstrate that the MAGIC framework is configurable across a wide range of compression/quality and is capable of compressing beyond the standard quality factor limits of both JPEG 2000 and WebP. We perform experiments on representative IoT applications using two vision data sets and show $42.65\times $ compression at similar accuracy with respect to the source. We highlight low variance in compression rate across images using our technique as compared to JPEG 2000 and WebP.

Its cousin, lossy compression, is easier to comprehend. Lossy algorithms are used to get music into the popular MP3 format and turn a digital image into a standard JPEG file. They do this by selectively removing bits, taking what scientists know about the way we see and hear to determine which bits we'd least miss. But no one can make the case that the resulting file is a perfect replica of the original.

This paper proposes a new framework of lossless data hiding (LDH) in JPEG images. The proposed framework contains two algorithms, i.e., the optimized basic LDH and the relay transfer based extension. In the basic algorithm, we aim to preserve the filesize after data embedding. The data hiding process is optimized by variable-length-code (VLC) mapping, combination and permutation. In the extended algorithm, we focus on embedding more bits into the bitstream with a condition that the filesize increment is allowed. To decrease the filesize increment, we propose a relay transfer based algorithm to preprocess the JPEG bitstream. Subsequently, we embed data into the processed bitstream using the basic LDH. Both algorithms provide better performances than previous arts. After lossless data hiding, the marked JPEG bitstream is compliant to common JPEG decoders. Since all operations are implemented on VLCs and the Huffman codes, no distortion is generated on the image pixels. Experimental results demonstrate that the proposed approach outperforms previous methods.

Quantization plays a pivotal role in JPEG compression with respect to the tradeoff between image fidelity and storage size, and the blind estimation of quantization parameters has attracted considerable interest in the fields of image steganalysis and forensics. Existing estimation methods have made great progress, but they usually suffer a sharp decline in accuracy when addressing small-size JPEG decompressed bitmaps due to the insufficiency of coefficients. Aiming to alleviate this issue, this paper proposes a generic clustering-based framework to improve the performance of the existing methods. The core idea is to gather as many coefficients as possible by clustering subbands before feeding them into a step estimator. The proposed framework is implemented using hierarchical clustering with two kinds of histogram-like features. Extensive experiments are conducted to validate the effectiveness of the proposed framework on a variety of images of different sizes and quality factors, and the results show that notable improvements can be achieved. In addition to quantization step estimation, we believe the idea behind the proposed framework might provide inspiration for other forensic tasks to alleviate their performance issues induced by sample insufficiency.

In recent years, online social networks (OSNs) have become extremely popular and been one of the most common ways for storing and distributing images. Naturally, such widespread availability of OSN makes it a viable channel for transmitting additional data along with the image sharing. However, various lossy operations, e.g., resizing and compression, conducted by OSN platforms impose great challenges for designing a robust watermarking scheme over OSN shared images. In this paper, we tackle this challenge and propose a robust high-capacity watermarking technique, by using Facebook as a representative OSN. To achieve the satisfactory robustness, we first probe into Facebook and recover the image manipulation mechanism via a deep convolutional neural network (DCNN) approach. Assisted with the precise knowledge on the lossy channel offered by Facebook, we then suggest a DCT-domain image watermarking method that is highly robust against the lossy operations on Facebook, even without any error correcting codes (ECC). The proposed technique is also extended to other popular OSNs, e.g., Wechat and Twitter. Extensive experimental results are provided to show the superior performance of our method in terms of the embedding capacity, data extraction accuracy, and quality of the reconstructed images.

JPEG is the most widely used lossy image compression standard. When using JPEG with high compression ratios, visual artifacts cannot be avoided. These artifacts not only degrade the user experience but also negatively affect many low-level image processing tasks. Recently, convolutional neural network (CNN)-based compression artifact removal approaches have achieved significant success, however, at the cost of high computational complexity due to an enormous number of parameters. To address this issue, we propose a dual-stream recursive residual network (STRRN) which consists of structure and texture streams for separately reducing the specific artifacts related to high-frequency or low-frequency image components. The outputs of these streams are combined and fed into an aggregation network to further enhance the restored images. By using parameter sharing, the proposed network reduces the total number of training parameters significantly. Moreover, experiments conducted on five commonly used datasets confirm that the proposed STRRN can efficiently reduce the compression artifacts, while using up to 4.6 times less training parameters and 5 times less running time compared to the state-of-the-art approaches.

Could we compress images via standard codecs while avoiding visible artifacts? The answer is obvious -- this is doable as long as the bit budget is generous enough. What if the allocated bit-rate for compression is insufficient? Then unfortunately, artifacts are a fact of life. Many attempts were made over the years to fight this phenomenon, with various degrees of success. In this work we aim to break the unholy connection between bit-rate and image quality, and propose a way to circumvent compression artifacts by pre-editing the incoming image and modifying its content to fit the given bits. We design this editing operation as a learned convolutional neural network, and formulate an optimization problem for its training. Our loss takes into account a proximity between the original image and the edited one, a bit-budget penalty over the proposed image, and a no-reference image quality measure for forcing the outcome to be visually pleasing. The proposed approach is demonstrated on the popular JPEG compression, showing savings in bits and/or improvements in visual quality, obtained with intricate editing effects.

Reversible data hiding (RDH) for JPEG images has attracted extensive attentions in recent years. However, since DCT is a de-correlation operation, it is difficult for JPEG images to employ the image redundancy for RDH design like uncompressed images. In this paper, considering better utilizing the special properties of quantized DCT coefficients, a new RDH scheme for JPEG images based on multiple two-dimensional (2D) histograms modification is proposed. Firstly, by combining every two nonzero alternating current (AC) coefficients of adjacent DCT blocks in each band as a pair, a new coefficients pairing strategy is proposed for a sharper 2D histogram. Then, with a classification process, multiple 1D and 2D histograms are generated, in which the 2D histograms are selected for data embedding and the bins in 1D histograms are shifted for reversibility. Finally, to further enhance the embedding performance, the 2D mappings are adaptively determined for different 2D histograms through a formulated rate-distortion model. Experimental results demonstrate the superiority of the proposed scheme both in visual quality and file size preservation.

Batch steganography aims at properly allocating a large payload to multiple covers, so as to keep the whole covert communication at a satisfactory level of security. JPEG is currently one of the most widely used formats for image storage and transmission. This paper presents an efficient JPEG batch steganographic scheme, which allocates the payload in a linear manner w.r.t. a new heuristic measure - the intrinsic energy of JPEG image contents, in which more concerns are with the high frequency components, and the proposed measure could also be easily generalized to cover selection in batch steganographic applications. And a calibration strategy is elaborately designed to balance the security level when JPEG covers of various QFs are involved in JPEG batch steganography. In this way, the proposed scheme can effectively resolve the problem that the statistical undetectability fluctuates dramatically w.r.t. the size and quality factor when the batch set is involved with various image parameters, and consequently maintains the overall security of the practical JPEG batch steganographic system. Experimental results show that the proposed method exhibits security performance superior or comparable to the state-of-the-art batch schemes while maintaining a low computational cost.

JPEG images are usually corrupted by various undesirable compression artifacts resulted from block-wise coarse quantization on discrete cosine transform coefficients. In recent years, deep convolutional neural networks (CNNs) have made spectacular achievements in compression artifacts reduction. However, most deep CNNs are difficult to be implemented on mobile devices due to their large number of parameters and operations. In this letter, we propose a novel deep CNN called ESCNet for lightweight JPEG compression artifacts reduction, in which enhanced separable convolution (ESConv) is carefully designed to make full use of image multi-scale information for better dense pixel value predictions. Specifically, ESConv consists of a grouped multi-scale dual depth-wise convolution (GMDDConv) and a wide-activated dual point-wise convolution (WDPConv). GMDDConv is dedicated to efficiently extracting abundant image multi-scale spatial features, which will be sent to WDPConv for effective non-linear feature fusion. The experimental results on benchmark datasets show that compared with state-of-the-art methods, our ESCNet not only achieves better performance in both objective indices and subjective quality but also greatly reduces network parameters and operations.

Recent advances show that a reasonable non-additive cost function can significantly improve the security level of additive cost based steganography. So far, there is only one principle, called block boundary continuity (BBC), that has been proposed to define the non-additive cost function for JPEG steganography, and it aims at synchronizing the modification direction of inter-block boundaries in the spatial domain. In this article, we found that JPEG steganography usually introduces more and larger modifications on the boundary than on the inside of each intra-block in the spatial domain, which is another important factor affecting security. Therefore, we present a new principle, called block boundary maintenance (BBM), to minimize the modifications on the spatial block boundaries. In theory, we deduce the BBM principle on how to modify a pair of DCT coefficients of the intra-block to reduce the modifications on the spatial block boundary. According to the BBM principle, we design a new strategy to define non-additive cost functions for JPEG steganography by exploiting the coefficient correlation of the intra-block in the DCT domain. The experimental results show that the BBM-based strategy can minimize modifications on the spatial block boundaries and thus achieve a high-security level when resisting modern JPEG steganalysis. Furthermore, the two principles of BBC and BBM can be fused to further improve the empirical security.

Several recent studies have attempted to fully replace the conventional camera image signal processing (ISP) pipeline with convolutional neural networks (CNNs). However, the previous CNN-based ISPs, simply referred to as ISP-Nets, have not explicitly considered that images have to be lossy-compressed in most cases, especially by the off-the-shelf JPEG. To address this issue, in this paper, we propose a novel compression-aware deep camera ISP learning framework. At first, we introduce a new use case of compression artifacts simulation network (CAS-Net), which operates in the opposite way of commonly used compression artifacts reduction networks. Then, the CAS-Net is connected with an ISP-Net such that the ISP network can be trained with consideration of image compression. Throughout experimental studies, we show that our compression-aware camera ISP network can produce images with a better tradeoff between bit-rate and image quality compared to its compression-agnostic version when the performance is evaluated after JPEG compression.

Imaging devices capable of capturing 360° images were recently released into the market. The spherical images acquired by imaging devices, such as 360° cameras and multiple camera smartphones, are projected onto the 2D image plane to allow the use of the existing image coding standard. The 360° image market is currently dominated by the equirectangular projection format coded by the JPEG image coding standard. Spherical images are characterized by their large size, and they are typically viewed by displaying a default viewport of the entire image first, followed by navigation by the user. Therefore, progressive image coding capable of transmitting the default viewport first is in high demand. However, the progressive image coding methods proposed thus far, such as quality-based and resolution-based progressive coding, do not support the spatial region-based progressiveness necessary to first display the default viewport. This paper proposes a center-to-edge progressive mode for the JPEG image coding standard, which enables transmittance of the discrete cosine transform blocks including the default viewport first by reordering the blocks’ scanning path. This progressive mode was designed to maintain all other encoding structures of the JPEG standard, such that use of the already established toolchain of the JPEG is maintained. Results of the test performed with our proposed mode using eleven 360° images in the equirectangular projection format and six objective image quality parameters show that transmission of default viewports is notably faster than existing JPEG modes and negligible effects on JPEG image coding efficiency.

The emergence of powerful image editing software has substantially facilitated digital image tampering, leading to many security issues. Hence, it is urgent to identify tampered images and localize tampered regions. Although much attention has been devoted to image tampering localization in recent years, it is still challenging to perform tampering localization in practical forensic applications. The reasons include the difficulty of learning discriminative representations of tampering traces and the lack of realistic tampered images for training. Since Photoshop is widely used for image tampering in practice, this paper attempts to address the issue of tampering localization by focusing on the detection of commonly used editing tools and operations in Photoshop. In order to well capture tampering traces, a fully convolutional encoder-decoder architecture is designed, where dense connections and dilated convolutions are adopted for achieving better localization performance. In order to effectively train a model in the case of insufficient tampered images, we design a training data generation strategy by resorting to Photoshop scripting, which can imitate human manipulations and generate large-scale training samples. Extensive experimental results show that the proposed approach outperforms state-of-the-art competitors when the model is trained with only generated images or fine-tuned with a small amount of realistic tampered images. The proposed method also has good robustness against some common post-processing operations.

We propose a finely tunable JPEG format-compliant perceptual encryption (FE) with two novel strategies: (i) bitcuboid-based encryption (BE) and (ii) exception-free signed binarization (ESB). BE is an intra- and inter-bitplane encryption technique that provides finely tunable perceptual degradation by encrypting constrained subspaces (‘bitcubes’) of a cuboid-shaped bit set (‘bitcuboid’). ESB is a binarization technique that redecimalizes encrypted binary sequences into signed decimal coefficients without any exception-handling by shifting the negative binary sequences one-by-one. BE with ESB (BEESB) is applied to the quantized discrete cosine transform (QDCT) domain in JPEG compression. The results of our first experiment show that the BE attains fine tunability, which means scalability of the perceptual degradation level with a single encryption method, by encrypting bitcubes of various types and sizes combinatorially. The results of our second experiment show that the BEESB suppresses the bitrate overheads and that BEESB with one of the most secure options suppresses approximately 0.80-187.58 % more of the bitrate overheads in terms of Bjontegaard delta (BD)-rate compared with conventional methods except for some ones. The results of our third experiment show that BEESB has high resilience against attacks.

This short article reports two imperfections or implementation bugs of the default tone mapping operators (TMOs) of the JPEGXT HDR image compression standard. In JPEGXT, profiles A and C use the same default TMO, while profile B uses a different default TMO. The first imperfection is with the default TMO of profile A and profile C. Specifically, the default TMO leads to a reduced dynamic range of the tone-mapped LDR images. Very often, the reduction can be severe. The mechanism of causing the dynamic range reduction stems from an implementation bug. To resolve the issue, the correction of adding the missed final normalization step to linearly stretch the dynamic range needs to be made. The second imperfection is the highlights clipping in the LDR images from the default TMO of profile B. The clipping could lead to unusable LDR images. Our suggestion is to unify the default TMO within JPEGXT, i.e., to use the corrected TMO of profiles A and C for profile B.

The state-of-the-art 2D-based dynamic point cloud (DPC) compression algorithm is the video-based point cloud compression (V-PCC) developed by the Moving Pictures Experts Group (MPEG). It first projects the DPC patch by patch from 3D to 2D and organizes the projected patches into a video. The video is then efficiently compressed by High Efficiency Video Coding. However, there are many unoccupied pixels that may have a significant influence on the coding efficiency. These unoccupied pixels are currently padded using either the average of 4-neighbors for the geometry or the push-pull algorithm for the color attribute. While these algorithms are simple, the unoccupied pixels are not handled in the most efficient way. In this paper, we divide the unoccupied pixels into two groups: those that should be occupied and those that should not be occupied according to the occupancy map. We then design padding algorithms tailored to each group to improve the rate-distortion performance of the V-PCC reference software, for both the geometry and the color attribute. The first group is the unoccupied pixels that should be occupied according to the block-based occupancy map. We attempt to pad those pixels using the real points in the original DPC to improve the quality of the reconstructed DPC. Additionally, we attempt to maintain the smoothness of each block so as not to negatively influence the video compression efficiency. The second group is the unoccupied pixels that were correctly identified as unoccupied according to the block-based occupancy map. These pixels are useless for improving the reconstructed quality of the DPC. Therefore, we attempt to minimize the bit cost of these pixels without considering their reconstruction qualities. The bit cost is determined by the residue of these pixels obtained by subtracting the prediction pixels from the original pixels. Therefore, we propose padding the residue using the average residue of the occupied pixels in order to minimize the bit cost. The proposed algorithms are implemented in the V-PCC and the corresponding HEVC reference software. The experimental results show the proposed algorithms can bring significant bitrate savings compared with the V-PCC.

The Joint Picture Expert Group (JPEG) committee has been standardizing next-generation image compression, called JPEG XL, to meet the specific needs for a responsive web, wide color gamut, and high dynamic range. JPEG XL supports lossy and lossless compression. A variable-sized discrete cosine transform (DCT) block is used for lossy compression. A block partitioning method is regarded as a critical function for the performance of JPEG XL. The current DCT block partitioning method used in JPEG XL is highly dependent on the compression rate and tends to assign small-sized DCT blocks to homogeneously textured regions (HTRs) having similar or regular patterns. We propose a region-adaptive DCT block partitioning method that assigns larger blocks to the HTR. The proposed method identifies the HTRs by using a combined metric employing a sum-modified Laplacian, zero-crossing, and colorfulness metric for measuring the region homogeneity. Objective, subjective, and visual comparison evaluations with the ten images recommended by the JPEG working group were provided to show the improvement in coding performance. The proposed method shows its superiority in terms of the compression efficiency evaluated using six objective metrics, subjective tests with 15 participants, visual comparison improvements in the HTR, and gains in the execution time.

Images and short videos that produced by social networks surge in recent years. Image/Video encoders, such as JPEG and H.264, are indispensably involved to reduce the transmitting bandwidth. However, based on our observation, the encoding parameters and their candidates are often preset to fixed values (or fixed candidate values) in real-world scenarios, which might not be the optimal bandwidth allocation strategy. Considering that, we propose an efficient group quality optimization (GQO) framework for multi-channel image/video encoding systems in which the encoding parameters are configured in a perceptual-quality-driven manner. The GQO framework employs adaptive hyper network to predict the relationships between encoding parameters, transmitting resources, and perceptual qualities, i.e., just taking the pristine image as input, the adaptive hyper network could accurately yield a global overview of perceptual quality and transmitting resource varied along encoding parameters. A step-by-step optimization procedure is then employed to search the optimal encoding parameter for each channel so that overall perceptual quality could be maximized under limited transmitting resource. Experimental results demonstrate the proposed GQO framework could achieve higher perceptual quality whilst maintain the same bandwidth compared to traditional allocation strategies where encoding parameters are preset.

The video-based point cloud compression (V-PCC, ISO/IEC 23090-5) is the state-of-the-art international standard for compressing dynamic point clouds developed by the moving picture experts group (MPEG). It has been achieved good rate-distortion (RD) performance by employing the 2D-based dynamic point cloud compression. As a brief look, V-PCC first converts the 3D input point cloud into a set of 2D patches followed by a packing process. The packing process then maps the patches into a 2D grid. Such a way allows compressing the patches utilizing the existing video coding standards. Besides the RD performance, complexity is another vital factor to consider in performance evaluations. In the V-PCC encoder, the self-time accounts for on average 15.9% and a maximum of 48.2% of the total-time, which can be a hindrance to realizing real-time V-PCC applications. One of the most computationally intensive modules of V-PCC is the grid-based refining segmentation (G-RS). Thus this paper proposes a fast G-RS method that can adaptively select voxels that need the refining segmentation. More concretely, the proposed method classifies the voxels based on the projection plane indices of 3D points and only applies the refining process to the selected voxels. Experimental results demonstrate that the proposed method reduces the complexity of the refining steps in G-RS, on average, by 60.7% and 62.5% without coding efficiency loss compared to the test model for category 2 (TMC2) version 12.0 reference software under the random access (RA) and all-intra (AI) configurations, respectively.

Image steganography aims to embed secret messages into cover images in an imperceptible manner. While steganalysis tries to identify stegos from covers, which is a special binary classification problem. Recently, some literatures show that the adversarial embedding can mislead the advanced steganalyzers based on convolutional neural network (CNN), and thus enhance the steganography security. Since adding perturbations to stegos may lead to messages extraction failure due to properties of syndrome-trellis codes (STC), the existing adversarial examples are derived from covers or their enhanced versions, while those stegos are not fully utilized. In this paper, we propose a new adversarial embedding scheme for image steganography. Unlike those related works, we first combine multiple gradients of cover and generated stegos to determine the directions of cost modifications. Next, instead of adjusting all or a random part of embedding costs in existing works, we carefully select the candidate costs according to the amplitudes of cover gradients and their costs. Extensive experimental results demonstrate that by adjusting a tiny part of embedding costs (less than 5% in most cases), the proposed method can significantly improve the security of five modern steganographic methods evaluated on both re-trained CNN-based and traditional steganalyzers, and achieve much better security performances compared with related methods. In addition, the security performances evaluated on different image database show that the generalization of the proposed method is good.

The Encrypted Image Reversible Data Hiding (EIRDH) is a hot area of research in multimedia outsourcing scenarios. In such a setting, a content owner possesses confidential images that he wants to upload to an untrusted server. The server is supposed to embed a secret message in the images before sending them to potential receivers, and the receivers expect to recover the original images without any distortion. EIRDH schemes are designed to meet reversible data hiding goals while preserving the confidentiality of the host images. Confidentiality is provided by means of encryption algorithms; however, these algorithms, in turn, introduce the challenge of dealing with ciphered images in both embedding and extraction phases. The current paper tries to propose a privacy-preserving reversible data hiding scheme for JPEG images without the need for any levels of decompression. We propose an elliptic curve-based asymmetric, commutative, and separable EIRDH method, in which no entities other than the sender and the eligible receiver of the hidden message (embedded in the encrypted image) are able to recognize its existence. In particular, even the server that performs the embedding remains completely unaware of the content of the images. An important security feature of our scheme involves the security of its multiple encryptions against Chosen-Plaintext Attack (CPA), which is an attack model that assumes the adversary can obtain the encryptions of the images on his choice to achieve possible extra information to attack. Experimental results show that our scheme achieves full reversibility and a satisfactory level of capacity, while it accurately extracts all the hidden bits.

Identification of decompressed JPEG images, especially those compressed with high JPEG quality factors, is a challenging issue in image forensics. Furthermore, the applicability of the existing JPEG forensic detectors in forgery localization is limited by their inability to cope with spatial misalignment in the $8\times 8$ JPEG grid. This paper proposes a novel forensic detector for decompressed JPEG color images stored in the bitmap format. The basic idea is to exploit the chroma upsampling traces introduced during the JPEG decompression of color images. We first show that the chroma upsampling alters the wide-sense stationary nature of the chroma planes and introduces wide-sense cyclostationarity in the upsampled chroma planes of decompressed JPEG images. As a result, unlike in the case of never-compressed images, the distributions of the odd-even and even-odd pixel pair differences are different for the chroma planes of never-compressed and decompressed JPEG images. We use the two-sample Kolmogorov-Smirnov (KS) test to find the similarity between these distributions and thereby to differentiate between never-compressed and decompressed JPEG images. The experimental results on a large set of images, including realistically-forged images, show the efficacy of the proposed detector even in the presence of spatial desynchronization due to image cropping.

The Joint Video Exploration Team (JVET) recently launched the standardization of the next-generation video coding named Versatile Video Coding (VVC) with the inherited technical framework from its predecessor High-Efficiency Video Coding (HEVC). The simplified Enhanced Multiple Transform (EMT) has been adopted as the primary residual coding transform solution, termed Multiple Transform Selection (MTS). In MTS, only the transform set consisting of DST-VII and DCT-VIII remains, excluding the other transform sets and the dependency on intra prediction modes. Significant coding gains are achieved by introducing new DST/DCT transforms, but the full matrix implementation is relatively costly compared to partial butterfly in terms of both software run-time and operation counts. In this work, we exploit the inherent features existing in DST-VII and DCT-VIII. Instead of repeating the element-wise additions and multiplications in full matrix operation, these features can be leveraged to achieve more efficient implementations which only use partial elements to derive the identical results. Existing transform matrices are further tuned to utilize these (anti-)symmetric features. A partial butterfly-type fast algorithm with dual-implementation support is proposed for DST-VII/DCT-VIII transform in VVC. Complexity analysis including operation counts and software run-time are conducted to validate the effectiveness. In addition, we prove the features are perfectly supported by theory. The proposed fast methods achieve noticeable software run-time savings without compromising on coding performance by comparing with the VVC Test Model VTM-3.0. It is shown that under Common Test Condition (CTC) with inter MTS enabled, an average of 9%, 0%, and 3% decoding time savings are achieved for All Intra (AI), Random Access (RA) and Low Delay B (LDB), respectively. Under low QP test condition with inter MTS enabled, the proposed fast methods achieve 1%, 2% and 4% decoding time savings on average for AI, RA, and LDB, respectively.

While traditional image compression algorithms take a full three-component color representation of an image as input, capturing of such images is done in many applications with Bayer CFA pattern sensors that provide only a single color information per sensor element and position. In order to avoid additional complexity at the encoder side, such CFA pattern images can be compressed directly without prior conversion to a full color image. In this paper, we describe a recent activity of the JPEG committee (ISO SC 29 WG 1) to develop such a compression algorithm in the framework of JPEG XS. It turns out that it is important to understand the ""development process"" from CFA patterns to full color images in order to optimize the image quality of such a compression algorithm, which we will also describe shortly. We introduce (1) a novel decorrelation step upfront processing (the so-called Star-Tetrix transform), along with (2) a pre-emphasis function to improve the compression efficiency of the subsequent compression algorithm (here, JPEG XS). Our experiments clearly indicate a gain over a RGB compression workflow in terms of complexity and quality (between 1.5dB and more than 4dB depending on the target bitrate). A comparison is also made with other state-of-the-art CFA compression techniques.

JPEG lossy image compression is a still image compression algorithm model that is currently widely used in major network media. However, it is unsatisfactory in the quality of compressed images at low bit rates. The objective of this paper is to improve the quality of compressed images and suppress blocking artifacts by improving the JPEG image compression model at low bit rates. First, the image texture adaptive non-uniform rectangular partition (ITANRP) algorithm is proposed which partitions the image into $8\times 8$ size image blocks with high texture complexity and $16\times 16$ size image blocks with low texture complexity. Then, a new transform coding based on the complete orthogonal U-system and all-phase digital filter (APDF) is proposed for coding image blocks with different sizes. Next, a flexible adaptive quantization scheme is designed to quantize image blocks with different sizes by considering the sensitivity of the human visual system (HVS) to different texture complexities. Finally, combining the above method with the JPEG model, a novel image compression algorithm model with low algorithm complexity is proposed to solve the problem in JPEG. The experimental results demonstrate that the performance of our algorithm model outperforms the JPEG image compression algorithms, the quality of the compressed image is greatly improved, and the blocking artifacts are also significantly suppressed.

Recently we have proposed a coding algorithm of point cloud geometry based on a rather different approach from the popular octree representation. In our algorithm, the point cloud is decomposed in silhouettes, hence the name Silhouette Coder, and context adaptive arithmetic coding is used to exploit redundancies within the point cloud (intra frame coding), and also using a reference point cloud (inter frame coding). In this letter we build on our previous work and propose a context selection algorithm as a pre-processing stage. With this algorithm, the point cloud is first parsed testing a large number of candidate context locations. The algorithm selects a small number of these contexts that better reflect the current point cloud, and then encode it with this choice. The proposed method further improves the results of our previous coder, Silhouette 4D, by $10 \%$ , on average, on a dynamic point cloud dataset of the JPEG Pleno, and achieves bitrates competitive with some high quality lossy coders such as the MPEG G-PCC.

For multidimensional data, Space-Filling Curves (SFCs) have been used to improve the execution time of spatial data queries. However, their effect on compression, when used to reorder the uncompressed values, is known to a lesser extent. We investigate the impact of three SFCs on Shuttle Radar Topographic Mission (SRTM) elevation data and Square-Kilometre Array telescope (SKA) radio-astronomy data: two types of datasets to which SFCs have not been extensively applied, within a compression context. This work contributes to the understanding of how such reorderings impact compression performance and affect different compression schemes and preprocessing techniques through their use. We show empirical results from combining eight common compression schemes, the Z-Order, Gray-Code, and Hilbert space-filling curves, and the bitwise preprocessing technique BitShuffle. The Hilbert Curve consistently outperforms the other orderings for the SRTM dataset though the mapping implementation incurs a significant speed penalty. However, the Z-Order and Gray-Code Curves are best for the SKA dataset. Through an analysis of the dataset autocorrelations, file-entropies, and block-entropies; we show that the SKA dataset’s dimensional bias is not exploited as much by the Hilbert Curve compared to the Z-Order and Gray-Code Curves. However, the Hilbert Curve is the most appropriate for the SRTM dataset as it can be modelled as isotropic and has a significantly higher level of local autocorrelation. BitShuffle is necessary to practically compress the SKA data, but does contribute to the compression performance of the SRTM dataset. These curves and BitShuffle are advantageous in reducing block-entropy values for such datasets.

To cope with the abuse of steganography using H.264 videos, i.e., the dominant video format, as the carrier, this paper presents a steganalytic method which works well even in the scenario where both the training data and the prior knowledge of the test data are limited. As a key feature of H.264, intra prediction is incorporated to remove redundancies within one single frame by predicting the current block using previously coded blocks. Unlike in JPEG domain, the quantized discrete cosine transform (QDCT) coefficients in H.264 videos come from the prediction error (residual) blocks (PEBs) instead of the original pixel block, hence we suggest shifting the focal point from the spatial domain to the prediction error domain, i.e., the PEB domain. According to the traits of video coding, 3 types of subtractive PEB (SPEB) are defined to capture the inconsistency between correlated PEBs, and the differences between correlated SPEBs are modeled by first-order Markov chain. Then the so-called SUPERB (SUbtractive Prediction ERror Block) features are engineered by subsets of sample transition probability matrices for a steganalyzer. What’s more, the features derived from IPM (Intra Prediction Mode) transition probabilities are also merged into SUPERB to improve detection ability. Extensive experiments are carried out from different aspects. Performance results demonstrate the effectiveness of SUPERB, particularly its essence of general applicability when the training and test data are of quite different attributes, which is more favorable for real-world applications.

Recently, the advancements in the field of wireless technologies and micro-electro-mechanical systems lead to the development of potential applications in wireless sensor networks (WSNs). The visual sensors in WSN create a significant impact on computer vision based applications such as pattern recognition and image restoration. generate a massive quantity of multimedia data. Since transmission of images consumes more computational resources, various image compression techniques have been proposed. But, most of the existing image compression techniques are not applicable for sensor nodes due to its limitations on energy, bandwidth, memory, and processing capabilities. In this article, we introduce a highly reliable and low-complexity image compression scheme using neighborhood correlation sequence (NCS) algorithm. The NCS algorithm performs the bit reduction operation and then encoded by a codec (such as PPM, Deflate, and Lempel Ziv Markov chain algorithm.) to further compress the image. The proposed NCS algorithm increases the compression performance and decreases the energy utilization of the sensor nodes with high fidelity. Moreover, it achieved a minimum end to end delay of 1074.46 ms at the average bit rate of 4.40 bpp and peak signal to noise ratio of 48.06 on the applied test images. On comparing with state-of-art methods, the proposed method maintains a better tradeoff between compression efficiency and reconstructed image quality.

Image registration is the process of overlaying two or more images of the same scene taken at different times, from different viewpoints, and/or by different sensors. Registration is among the most fundamental operations to generate almost all data products from remotely sensed scenes. It is a challenging task due to inevitable geometric and radiometric distortions during the data acquisition process. The presence of clouds makes the problem even more complicated. This article presents the development of operational software for multitemporal and multiband image registration for KALPANA-1 and INSAT-3A at the Indian Space Research Organization (ISRO).

In terms of feature-based steganalysis for JPEG images, JPEG phase-aware features (e.g., DCTR and GFR) currently provide the best detection performance on modern adaptive steganographic schemes. But in DCTR and GFR, the types of residual images are relatively single. They only use the convolution residuals obtained with the DCT or Gabor filters of fixed size ${8\times 8}$ . In this paper, to further improve DCTR and GFR, two strategies are proposed to enrich the features by diversifying residual images, and the corresponding symmetrization rules for the features are also elaborately designed. First, instead of a single filter size of ${8\times 8}$ , convolution filters of multiple sizes are adopted to generate different residual images. Second, we also compute JPEG phase-aware features from the difference images between two convolution residuals. Since the features from convolution residuals and difference images are diverse and complementary, the combination of these two kinds of features can significantly improve the detection accuracy. Last but not least, different symmetrization rules are accordingly designed for these features by considering filter types, filter sizes, and residual subtraction to decrease the feature dimension and enhance the feature robustness. The experimental results demonstrate the effectiveness of our proposed features, and we can further boost the performance by incorporating the knowledge of the selection channel and using the accelerated weighted histogram method.

The most widely used video encoders share a common hybrid coding framework that includes block-based motion estimation/compensation and block-based transform coding. Despite their high coding efficiency, the encoded videos often exhibit visually annoying artifacts, denoted as Perceivable Encoding Artifacts (PEAs), which significantly degrade the visual Quality-of-Experience (QoE) of end users. To monitor and improve visual QoE, it is crucial to develop subjective and objective measures that can identify and quantify various types of PEAs. In this work, we make the first attempt to build a large-scale subject-labeled database composed of H.265/HEVC compressed videos containing various PEAs. The database, namely the PEA265, includes 4 types of spatial PEAs ( i.e. blurring, blocking, ringing and color bleeding) and 2 types of temporal PEAs ( i.e. flickering and floating). Each containing at least 60,000 image or video patches with positive and negative labels. Based on the PEA265 database, we develop and optimize Convolutional Neural Networks (CNNs) to objectively recognize different types of PEAs. Experiments show that our architecture is capable of identifying the 6 types of PEAs with an accuracy over 86%. To further demonstrate its application, we explore the relationship between collected PEA intensities and subjective quality scores of compressed videos. A quality metric is consequently proposed with superior performance in terms of correlation to Mean Opinion Score (MOS) values. We believe that the PEA265 database and our findings will benefit the future development of video quality assessment methods and perceptually motivated video encoders.