Search

Amid the conflict between the current huge data volumes and the requirement for large-compression-ratio compression in space-borne multichannel SAR, this paper proposes a data compression method that uses the adaptive bit allocation BAQ method in the Doppler domain of space-borne multichannel SAR. Specifically, 3MBAQ, which denotes multichannel, multipulse, multiweight block-adaptive quantization, performs signal reconstruction through a Krieger filter to obtain the Doppler spectrum under azimuthal multichannel and multipulse conditions and then achieves nonuniform bit allocation by using the multiple weight contributions between subband spectra. Furthermore, large-compression-ratio compression can be realized in homogeneous scenarios by using the adaptive bit allocation BAQ method together with the improved bit error control (BEC) algorithm. In this paper, the effectiveness of the 3MBAQ method is verified on GF3 measured data. Moreover, compared to the SQNR and NMSE of the BAQ, A-BAQ, A-BAQ-BEC, MCBAQ-ASQ, and MCBAQ-BEC methods under the same conditions, it is shown that there is an optimal 1.3434 dB improvement in SQNR and a 0.0241 improvement in NMSE using the 3MBAQ algorithm, and the complexity of the algorithm for 3MBAQ is O (N 2 l o g (N)) , which is within acceptable limits, clearly proving the superiority of the 3MBAQ method. [ABSTRACT FROM AUTHOR]

In this paper, a fast common edge extraction method based on a two-level grid indexing strategy was proposed for eliminating the common edge 'cracks'. A coarse-grained first-level grid index was firstly built which maps each polygon with its intersecting polygons in the graphics set; secondly, a fine-grained second-level grid index is established linking each grid cell with the vertices falling into the cell, which can be used to quickly find the common points shared by intersecting polygons. The common edge 'cracks' problem can be solved if the deleted common points are consistent on two adjacent graphics during compression. [ABSTRACT FROM AUTHOR]

In order to realize the real-time progressive compression of massive data and ensure the quality of compressed data, a real-time progressive compression method of massive data based on improved clustering algorithm is proposed in this paper. Through the micro clustering stage of birch method based on K-Medoids clustering, Clustering Feature Tree hierarchy is constructed and numerical clustering features are extracted; Taking this feature as the input of macro clustering order, the Clustering Feature Tree leaf nodes are clustered based on the improved K-Medoids clustering method, and the clustering data cluster set is output; The set is used as the original data of real-time progressive compression, and the data is denoised and compressed by lifting format wavelet transform. On this basis, Huffman coding is used to compress the data losslessly. The test results show that this method has good clustering effect under the optimal number of clustering centers, can complete the real-time progressive compression of a large number of data, and the availability of compressed data is more than 92%. [ABSTRACT FROM AUTHOR]

A number of text compression algorithms have been proposed in the past decades, which have been very effective and usually operate on conventional character/word-based approaches. A novel data compression perspective of data mining is explored in this research and the paper focuses on novel frequent sequence/pattern mining approach to text compression. This work attempts to make use of longer-range correlations between words in languages for achieving better text compression. We propose a novel and efficient method by making the compression of any word-level text in a universal manner for corpora across domains referred as Universal Huffman Tree-based encoding. The major contribution of this work is in terms of avoidance of code table communication to the decoder. Simulation results over benchmark datasets indicate that Universal Huffman encoding employing frequent sequence mining (achieves [20%, 89%] improvement in compression in reduced time. The paper also contributes a usable interface for Data compression that employs the proposed frequent sequent mining-based data compression algorithm. The interface supports features such as feedback, consistency, usability, navigation, visual appeal, performance and accessibility on par with existing compression softwares. The work results in an intelligent data compression software employing knowledge engineering perspective. [ABSTRACT FROM AUTHOR]

In recent years, single device sound scene classification has achieved good results, however, progress in multi-device sound scene classification has been slow. To solve the problem of large differences in the number of multi-device samples, this paper proposed a pairwise feature fusion algorithm. By calculating the differences in the spectrogram for each pair of paired samples and averaging these differences after accumulation, obtaining the average spectral characteristics of each device for the conversion of device samples. The algorithm effectively improved the generalization ability of the model while increasing the number of device samples. Meanwhile, in order to obtain global information, it proposed a lightweight attention module, which could make the model focus on the training of the whole sound sequence information on the basis of reduced computation by performing self-attention operations on the input features after compression in the frequency domain. The experimental results show that the proposed algorithm has better advantages in terms of model size and classification accuracy compared with other methods. [ABSTRACT FROM AUTHOR]

In Mars exploration, hyper-spectrometry plays an important role due to its high spectral resolution. However, due to the technical difficulty and the data size, the spatial resolution or the coverage of hyperspectral data is often limited. This limitation can be alleviated by deep learning-based super-resolution (SR) reconstruction. But the spatial size and batch size of the input training data is limited due to the large number of spectral channels. To improve the efficiency of model training and SR reconstruction, a dataset based on CRISM hyperspectral data is created in this paper, and its redundancy is analyzed in both spectral and spatial spital dimensions. Compression algorithms based on data selection and PCA are used to reduce the size of the input training data. A network that can perform spatial SR and spectral enhancement is also proposed to make the network can be trained with the compressed data. With these compression algorithms and network, high-resolution data with 235 bands can be reconstructed from the low-resolution data with only 40 bands. Compared with the network trained on the original low-resolution data with 235 bands, the model training time and the SR reconstruction runtime can be reduced to 30% and 23% with practically no accuracy loss. The effectiveness of compression algorithms based on data selection also indicates that maybe not all the bands need to be transmitted from the Mars probes or be collected. Furthermore, it would, in principle, help improve the efficiency of satellite data transmission and simplify the design of the hyper-spectrometer. Additionally, a method for spatial dimension correlation evaluation is also proposed in this paper. The spatial compression shows that the proposed method can reflect the correlation of spatial texture between patches, and the model can be acceptably trained with only half of the original data. [ABSTRACT FROM AUTHOR]

This paper presents a backpropagation neural network algorithm for data compression and data storage. Data compression or establishing model ten coding is the most basic idea of traditional data compression. The traditionally designed ideas are mainly based on reducing the redundancy in the information and focus on the coding design, and its compression ratio has been hovering around dozens of percent. After the traditional coding compression of information, it is difficult to further compress by similar methods. In order to solve the above problems, the information that takes up less signal space can be used to represent the information that takes up more signal space to realize data compression. This new design idea of data compression breaks through the traditional limitation of relying only on coding to reduce data redundancy and achieves a higher compression ratio. At the same time, the information after such compression can be repeatedly compressed, and it has a very good performance. This is the basic idea of the combination of neural network and data compression introduced in this paper. According to the theory of multiobjective function optimization, this paper puts forward the theoretical model of multiobjective optimization neural network and studies a multiobjective data compression method based on neural network. According to the change of data characteristics, this method automatically adjusts the structural parameters (connection weight and bias value) of neural network to obtain the largest amount of data compression at the cost of small information loss. This method has the characteristics of strong adaptability, parallel processing, knowledge distributed storage, and anti-interference. Experimental results show that, compared with other methods, the proposed method has significant advantages in performance index, compression time and compression effect, high efficiency. and high-quality robustness. [ABSTRACT FROM AUTHOR]

We are exploring the idea of data pruning via hyperreduction modeling. The main novelty of this paper is a lossy data compression/decompression approach for ploycrystalline data, which is based on a hyperreduction scheme that preserves data driven modeling capabilities after compression. We assume to know a mechanical model whose equations are satisfied by the data. It is shown that the proposed reconstruction of the data performs an oblique projection of selected original data. This is achieved by the solution of reduced mechanical equations. High resolution crystal plasticity finite element simulations demand computational and storage resources that are unusual, especially in cases where hundreds of grains are interacting under cyclic loading. The development of image-based modeling via computed tomography highlights the problem of long-term storage of simulation data by using data pruning. The present paper focuses on modeling cyclic strain-ratcheting as an example of numerical modeling that the proposed algorithm preserves. The size of the remaining sampled data can be user-defined, depending on the needs concerning storage space. The relevance of the pruned data is tested afterwards for statistics on the predicted strain, as if full finite element data were available. The proposed method is compared to the Gappy POD method, when no additional modeling step is expected after data pruning. [ABSTRACT FROM AUTHOR]

Data deduplication saves storage space by identifying and removing repeats in the data stream. Compared with traditional compression methods, data deduplication schemes are more computationally efficient and are thus widely used in large scale storage systems. In this paper, we provide an information-theoretic analysis of the performance of deduplication algorithms on data streams in which repeats are not exact. We introduce a source model in which probabilistic substitutions are considered. More precisely, each symbol in a repeated string is substituted with a given edit probability. Deduplication algorithms in both the fixed-length scheme and the variable-length scheme are studied. The fixed-length deduplication algorithm is shown to be unsuitable for the proposed source model as it does not take into account the edit probability. Two modifications are proposed and shown to have performances within a constant factor of optimal for a specific class of source models with the knowledge of model parameters. We also study the conventional variable-length deduplication algorithm and show that as source entropy becomes smaller, the size of the compressed string vanishes relative to the length of the uncompressed string, leading to high compression ratios. [ABSTRACT FROM AUTHOR]

The exponential growth in remote sensing, coupled with advancements in integrated circuits (IC) design and fabrication technology for communication, has prompted the progress of Wireless Sensor Networks (WSN). WSN comprises of sensor nodes and hubs fit for detecting, processing, and communicating remotely. Sensor nodes have limited resources such as memory, energy and computation capabilities restricting their ability to process large volume of data that is generated. Compressing the data before transmission will help alleviate the problem. Many data compression methods have been proposed but mainly for image processing and a vast majority of them are not pertinent on sensor nodes because of memory impediment, energy utilization and handling speed. To overcome this issue, authors in this research have chosen Run Length Encoding (RLE) and Adaptive Huffman Encoding (AHE) data compression techniques as they can be executed on sensor nodes. Both RLE and AHE are capable of balancing compression ratio and energy utilization. In this paper, a hybrid method comprising RLE and AHE, named as H-RLEAHE, is proposed and further investigated for sensor nodes. In order to verify the efficacy of the data compression algorithms, simulations were run, and the results compared with the compression techniques employing RLE, AHE, H-RLEAHE, and without the use of any compression approach for five distinct scenarios. The results demonstrate the RLE's efficiency, as it surpasses alternative data compression methods in terms of energy efficiency, network speed, packet delivery rate, and residual energy throughout all iterations. [ABSTRACT FROM AUTHOR]

In the context of today's supply-side structural reform, the research on the relationship between foreign trade and economic growth has received extensive attention from researchers. Since China's economic reform and opening up in the late 1970s, China's economy has experienced a relatively long period of rapid growth. From 1978 to the end of 2008, China's economy grew at an average annual rate of about 10%, and its GDP grew from US$9.75 billion in 1978 to US$1,430.69 billion in 2008. The progress is amazing, the total import and export of goods increased from 20.64 billion US dollars in 1978 to 25.6326 billion US dollars in 2008, and the world ranking rose from 32nd in 1978 to 3rd in 2008, second only to the United States and Germany. The analysis of the test results presented in this paper shows that, from 1995 to 2014, China's average annual exports accounted for 23.72% of GDP. Since China joined the World Trade Organization in 2001, China's share of export trade has increased significantly. In 2004, the total value of exports reached 4910.33 billion yuan, equivalent to more than 29% of China's GDP, and continued to rise in the following years. China's total exports as a share of GDP declined between 2009 and 2014 but remained above 20% in both cases. [ABSTRACT FROM AUTHOR]

The underwater wireless sensor network is an important component of the underwater three-dimensional monitoring system. Due to the high bit error rate, high delay, low bandwidth, limited energy, and high dynamic of underwater networks, it is very difficult to realize efficient and reliable data transmission. Therefore, this paper posits that it is not enough to design the routing algorithm only from the perspective of the transmission environment; the comprehensive design of the data transmission algorithm should also be combined with the application. An edge prediction-based adaptive data transmission algorithm (EP-ADTA) is proposed that can dynamically adapt to the needs of underwater monitoring applications and the changes in the transmission environment. EP-ADTA uses the end–edge–cloud architecture to define the underwater wireless sensor networks. The algorithm uses communication nodes as the agents, realizes the monitoring data prediction and compression according to the edge prediction, dynamically selects the transmission route, and controls the data transmission accuracy based on reinforcement learning. The simulation results show that EP-ADTA can meet the accuracy requirements of underwater monitoring applications, dynamically adapt to the changes in the transmission environment, and ensure efficient and reliable data transmission in underwater wireless sensor networks. [ABSTRACT FROM AUTHOR]

To augment situational awareness capabilities for high-frequency disturbances, the modern power system necessitates the implementation of high-sampling rates for real-time Synchro-Waveform (SW) measurements. Nevertheless, the nonlinear nature of data collected from SW measurement devices poses significant challenges to the communication and storage capacities of data centers. To effectively tackle this issue, this paper proposes a Multi-stage Hybrid Coding (MHC) method for the lossless compression of SW data. The MHC method utilizes a well-structured two-stage approach to achieve efficient compression. In the first stage, a variable frame-based method is designed to reduce redundant information and achieve initial compression using multiple periodic recursive compression. Subsequently, the dictionary-based method, specifically the Lempel–Ziv–Markov chain algorithm, is seamlessly integrated to further compress the SW data. Experimental and numerical analyses are conducted using both simulated and real-world source data, with real-time performance validation in SW measurement units. The obtained results convincingly demonstrate that the proposed method enables online compression of both event and non-event SW data, achieving an impressive reduction of over 73.4% in data space, while maintaining a data loss ratio lower than 0.4% for high-density SW data. • Multiple periodic recursive method is proposed to eliminate repetitive information. • A multi-stage hybrid coding method is implemented for the lossless compression. • A series of synchro-waveform events are collected to evaluate the performance. [ABSTRACT FROM AUTHOR]

In view of the large amount of data collected by an edge server, when compression technology is used for data compression, data classification accuracy is reduced and data loss is large. This paper proposes a data compression algorithm based on the chaotic mutation adaptive sparrow search algorithm (CMASSA). Constructing a new fitness function, CMASSA optimizes the hyperparameters of the Convolutional Auto-Encoder Network (CAEN) on the cloud service center, aiming to obtain the optimal CAEN model. The model is sent to the edge server to compress the data at the lower level of edge computing. The effectiveness of CMASSA performance is tested on ten high-dimensional benchmark functions, and the results show that CMASSA outperforms other comparison algorithms. Subsequently, experiments are compared with other literature on the Multi-class Weather Dataset (MWD). Experiments show that under the premise of ensuring a certain compression ratio, the proposed algorithm not only has better accuracy in classification tasks than other algorithms but also maintains a high degree of data reconstruction. [ABSTRACT FROM AUTHOR]

Future space missions will require increased on-board computing power to process and compress massive amounts of data. Consequently, embedded multi-core and GPU platforms are considered, which have been shown beneficial for data processing. However, the acceleration of data compression - an inherently sequential task - has not been explored. In this on-going research paper, we parallelize two space compression standards on both CPUs and GPUs using two candidate embedded GPU platforms for space showing that despite the challenging nature of CCSDS algorithms, their parallelization is possible and can provide significant performance benefits. [ABSTRACT FROM AUTHOR]

Lossless data compression algorithms can use statistical redundancy to represent data using a fewer number of bits in comparison to the original uncompressed data. Run-Length Encoding is one of the simplest lossless compression algorithms in terms of understanding its principles and software implementation, as well as in terms of temporal and spatial complexity. If this principle is applied to individual bits of original uncompressed data without respecting the byte boundaries, this approach is referred to as bit-level Run-Length Encoding. Algorithm for lossless data compression, proposed in this paper, optimizes bit-level Run-Length Encoding data compression, uses special encoding of repeating data blocks, and, if necessary, combines it with delta data transformation or representation of data in its original form intending to increase compression efficiency compared to a conventional bit-level Run-Length Encoding approach. The advantage of the algorithm proposed in this paper is in the increase of the compression ratio in comparison to the bit-level Run-Length Encoding, with the higher time and space consumption as the trade-off. Test results, that were obtained by compression of the segmentation metadata of different volume datasets take place in the last part of the paper. [ABSTRACT FROM AUTHOR]

Multiscale geometric analysis (MGA) is not only characterized by multi-resolution, time-frequency localization, multidirectionality and anisotropy, but also outdoes the limitations of wavelet transform in representing high-dimensional singular data such as edges and contours. Therefore, researchers have been exploring new MGA-based image compression standards rather than the JPEG2000 standard. However, due to the difference in terms of the data structure, redundancy and decorrelation between wavelet and MGA, as well as the complexity of the coding scheme, so far, no definitive researches have been reported on the MGA-based image coding schemes. In addressing this problem, this paper proposes an image data compression approach using the hidden Markov model (HMM)/pulse-coupled neural network (PCNN) model in the contourlet domain. First, a sparse decomposition of an image was performed using a contourlet transform to obtain the coefficients that show the multiscale and multidirectional characteristics. An HMM was then adopted to establish links between coefficients in neighboring subbands of different levels and directions. An Expectation-Maximization (EM) algorithm was also adopted in training the HMM in order to estimate the state probability matrix, which maintains the same structure of the contourlet decomposition coefficients. In addition, each state probability can be classified by the PCNN based on the state probability distribution. Experimental results show that the HMM/PCNN -contourlet model proposed in this paper leads to better compression performance and offer a more flexible encoding scheme. [ABSTRACT FROM AUTHOR]

There are various applications of clustering in the fields of machine learning, data mining, data compression along with pattern recognition. The existent techniques like the Llyods algorithm (sometimes called k-means) were affected by the issue of the algorithm which converges to a local optimum along with no approximation guarantee. For overcoming these shortcomings, an efficient k-means clustering approach is offered by this paper for stream data mining. Coreset is a popular and fundamental concept for k-means clustering in stream data. In each step, reduction determines a coreset of inputs, and represents the error, where P represents number of input points according to nested property of coreset. Hence, a bit reduction in error of final coreset gets n times more accurate. Therefore, this motivated the author to propose a new coreset-reduction algorithm. The proposed algorithm executed on the Covertype dataset, Spambase dataset, Census 1990 dataset, Bigcross dataset, and Tower dataset. Our algorithm outperforms with competitive algorithms like Streamkm++, BICO (BIRCH meets Coresets for k-means clustering), and BIRCH (Balance Iterative Reducing and Clustering using Hierarchies. [ABSTRACT FROM AUTHOR]

Traditional data update synchronization tends to update the entire file directly at a time or selectively update local changes. Although the latter method saves part of the communication bandwidth, but every time the file changes, no matter how much change is immediately updated, increasing the communication load, these two methods are not very good to meet the real-time requirements of the industrial Internet. Aiming at this problem, this paper proposed an end-to-end cloud collaborative data synchronization scheme for industrial Internet. In this scheme, it judged the threshold by calculating the difference file. If the value was within the threshold, the changed data blocks were encoded and compressed and uploaded to the corresponding cloud storage node for decompression storage. Otherwise, relevant information would be temporarily stored in the edge layer. Compared with RS encoding and Rsync differential algorithm, it can be seen that the proposed scheme improves the transmission efficiency of updated data, and improves the security through coding compression, and the overall performance is better than other methods. [ABSTRACT FROM AUTHOR]

Self-organizing feature mapping neural network is a typical unsupervised neural network algorithm, which is often used for clustering analysis and data compression. As the amount of data increases, the time consumption required by the algorithm becomes increasingly large, which becomes a new challenge. To address this issue, a quantum self-organizing feature mapping neural network is proposed in this paper. This algorithm provides a method to obtain the similarity between samples and neurons based on quantum phase estimation and demonstrates the scheme to obtain winning neurons by Grover algorithm. By utilizing the superposition of quantum, the algorithm achieves parallel computing. The time complexity analysis indicates that the proposed algorithm is exponentially faster than the classical counterpart. The quantum circuit has been devised, while numerical simulation and experiment on a heart disease dataset have been conducted programming within the Qiskit framework. Both have verified the feasibility of the algorithm. Moreover, an application of classification has been developed based on the trained self-organizing feature mapping neural network, which demonstrates the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) have demonstrated super performance in most learning tasks. However, a DNN typically contains a large number of parameters and operations, requiring a high-end processing platform for high-speed execution. To address this challenge, hardware-and-software co-design strategies, which involve joint DNN optimization and hardware implementation, can be applied. These strategies reduce the parameters and operations of the DNN, and fit it into a low-resource processing platform. In this paper, a DNN model is used for the analysis of the data captured using an electrochemical method to determine the concentration of a neurotransmitter and the recoding electrode. Next, a DNN miniaturization algorithm is introduced, involving combined pruning and compression, to reduce the DNN resource utilization. Here, the DNN is transformed to have sparse parameters by pruning a percentage of its weights. The Lempel–Ziv–Welch algorithm is then applied to compress the sparse DNN. Next, a DNN overlay is developed, combining the decompression of the DNN parameters and DNN inference, to allow the execution of the DNN on a FPGA on the PYNQ-Z2 board. This approach helps avoid the need for inclusion of a complex quantization algorithm. It compresses the DNN by a factor of 6.18, leading to about 50% reduction in the resource utilization on the FPGA. [ABSTRACT FROM AUTHOR]

• An effective compression algorithm was proposed for real-time transmission data. • A mixed prediction model of LSTM and XGBoost was established for time series data. • High performance of the proposed algorithm was veried by comparative experiments. • Almost 50% compression rate can be reached under one in ten thousand distortions. In this paper, a compression and decompression algorithm is proposed for real-time transmission data by combining discrete wavelet decomposition and reconstruction, hybrid model prediction of long short-term memory (LSTM) and extreme gradient boosting (XGBoost), and the development of quantization sequences. The forward compression and reverse decompression are achieved by continuously acquiring the prediction error and data series with quantization error through the iterative shifting approach. To accurately forecast data series with quantization errors, three classes of data with specific distribution characteristics collected from the measurement-while-drilling (MWD) operation field are decomposed into approximate data and detailed data, which are then predicted by autoregressive integrated moving average (ARIMA), support vector regression (SVR), XGBoost, backpropagation neural network (BPNN), radial basis function neural network (RBFNN), and LSTM models. Particularly, the LSTM model and the XGBoost model are proven to be suitable strategies for working with approximate data and detailed data separately by comparing with conventional approaches in terms of the mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R 2) predictive indexes. Besides, experiment studies show that almost 50% compression performance can be reached under one in ten thousand distortions. It is verified that the proposed algorithm is appropriate and efficient for real-time data transmission applications with high performance. [ABSTRACT FROM AUTHOR]

Financial early warning mechanism is of great significance to the long-term healthy development and stable operation of listed enterprises. This paper adopts the logistic regression early warning model and BP neural network early warning model. Based on the BP neural network t early warning model optimized by the simulated annealing algorithm, the prediction effects of the model are compared from the perspectives of model accuracy and variable importance. Through the comparative analysis of the empirical results of the three methods, it can be seen that the simulated annealing algorithm has many advantages. The combination of the simulated annealing algorithm with multithreading, data compression, and segmentation greatly improves the efficiency of the algorithm and shortens the running time. Using the logistic regression early warning model and BP neural network early warning model and based on the BP neural network t early warning model optimized by the simulated annealing algorithm, the prediction effects of the model are compared from the perspective of model accuracy and variable importance. The results show that the three index dimensions of the BP neural network optimized by the simulated annealing algorithm have good discrimination ability to financial status. The BP neural network early warning model optimized based on the simulated annealing algorithm has good prediction accuracy and good practical significance. [ABSTRACT FROM AUTHOR]

The popularity of diverse IoT-based applications and services continuously generating tremendous amount of data has revealed the significance of data compression (DC). Principal component analysis (PCA) is one of the most commonly employed algorithms for DC. However, when dealing with large-scale matrices, the standard PCA takes a very long time and requires a lot of memory. Therefore, this paper presents a novel distributed stochastic PCA algorithm (DSPCA) for hierarchical sensor network based on gradient-based adaptive PCA (GA-PCA), where the standard PCA is reformulated as a single-pass stochastic setting to find the direction of approximate maximal variance. The step-size in each iteration is obtained by incorporating the stabilized Barzilai-Borwein method with the gradient optimization. This enables DSPCA to be processed with low computational complexity while maintaining a high convergence speed. Computer simulation with two types of datasets displays that the proposed scheme consistently outperforms the representative DC schemes in terms of reconstruction accuracy of original data and explained variance. [ABSTRACT FROM AUTHOR]

Prediction-based compression methods, like prediction by partial matching, achieve a remarkable compression ratio, especially for texts written in natural language. However, they are not efficient in terms of speed. Part of the problem concerns the usage of dynamic entropy encoding, which is considerably slower than the static alternatives. In this paper, we propose a prediction-based compression method that decouples the context model from the frequency model. The separation allows static entropy encoding to be used without a significant overhead in the meta-data embedded in the compressed data. The result is a reasonably efficient algorithm that is particularly suited for small textual files, as the experiments show. We also show it is relatively easy to built strategies designed to handle specific cases, like the compression of files whose symbols are only locally frequent. [ABSTRACT FROM AUTHOR]

In the future, the Internet of things (IoT) may provide huge volumes of data. Smart grids are a class of IoT electricity distribution systems that can control bidirectional energy flows between consumers and service providers (Barman et al. "IOT Based Smart Energy Meter for Efficient Energy Utilization in Smart Grid", 978-1-5386-4769-1 1831.00, 2018 IEEE). A typical smart grid features an advanced metering infrastructure (AMI), which automatically collects meter data from widely distributed sensors. A utility company that intends to use AMI must deploy smart meters, data concentrator units, and a meter data management system (MDMS). Concentrators collect ubiquitous messages from smart meters and transmit aggregated data to their MDMS. Although ubiquitous messages may enhance the efficiency of some electricity grids, any excessive volume of messages causes data congestion. This research considers the Reference Energy Disaggregation Data Set. Our proposed algorithm can compress meter data efficiently. Our contributions are as follows: First, thus far, numerous researchers have attempted to address smart grid problems with AMI systems, and here, we provide a relatively complete review of these attempts. Second, this paper presents a strategy to analyze this problem and propose a compression algorithm to compress meter data. This proposed algorithm combines several existing compression algorithms and operates from 2 to 10% more efficiently than previously published algorithms. [ABSTRACT FROM AUTHOR]

In this paper, we propose a side matching vector quantisation (SMVQ) data hiding scheme for image using multiple linear regression prediction. For each pixel block, the proposed scheme combines the multiple linear regression algorithm and the SMVQ algorithm, so that it can more accurately match the codeword or directly obtain the predicted value closer to the real pixel. Our experimental results show that when the VQ codebook sizes are 128, 256, 512, and 1024, and the SCB size is 16, this scheme obtains a better compression rate and information embedding ability. It can be concluded from the experimental results that this scheme is superior to existing algorithms in terms of compression and embedding capacity. [ABSTRACT FROM AUTHOR]

Numerous surrogate-assisted expensive multi-objective optimization algorithms were proposed to deal with expensive multi-objective optimization problems in the past few years. The accuracy of the surrogate models degrades as the number of decision variables increases. In this paper, we propose a surrogate-assisted expensive multi-objective optimization algorithm based on decision space compression. Several surrogate models are built in the lower dimensional compressed space. The promising points are generated and selected in the lower compressed decision space and decoded to the original decision space for evaluation. Experimental studies show that the proposed algorithm achieves a good performance in handling expensive multi-objective optimization problems with high-dimensional decision space. [ABSTRACT FROM AUTHOR]

K-mer based methods have become prevalent in many areas of bioinformatics. In applications such as database search, they often work with large multi-terabyte-sized datasets. Storing such large datasets is a detriment to tool developers, tool users, and reproducibility efforts. General purpose compressors like gzip, or those designed for read data, are sub-optimal because they do not take into account the specific redundancy pattern in k-mer sets. In our earlier work (Rahman and Medvedev, RECOMB 2020), we presented an algorithm UST-Compress that uses a spectrum-preserving string set representation to compress a set of k-mers to disk. In this paper, we present two improved methods for disk compression of k-mer sets, called ESS-Compress and ESS-Tip-Compress. They use a more relaxed notion of string set representation to further remove redundancy from the representation of UST-Compress. We explore their behavior both theoretically and on real data. We show that they improve the compression sizes achieved by UST-Compress by up to 27 percent, across a breadth of datasets. We also derive lower bounds on how well this type of compression strategy can hope to do. [ABSTRACT FROM AUTHOR]

Recent advancements in sequencing methods have led to significant increase in sequencing data. Increase in sequencing data leads to research challenges such as storage, transfer, processing, etc. data compression techniques have been opted to cope with the storage of these data. There have been good achievements in compression ratio and execution time. This fast-paced advancement has raised major concerns about the security of data. Confidentiality, integrity, authenticity of data needs to be ensured. This paper presents a novel lossless reference-free algorithm that focuses on data compression along with encryption to achieve security in addition to other parameters. The proposed algorithm uses preprocessing of data before applying general-purpose compression library. Genetic algorithm is used to encrypt the data. The technique is validated with experimental results on benchmark datasets. Comparative analysis with state-of-the-art techniques is presented. The results show that the proposed method achieves better results in comparison to existing methods. [ABSTRACT FROM AUTHOR]

This paper details the examination of a particular case of data compression, where the compression algorithm removes the redundancy from data, which occurs when edge-based compression algorithms compress (previously compressed) pixelated images. The newly created redundancy can be removed using another round of compression. This work utilized the JPEG-LS as an example of an edge-based compression algorithm for compressing pixelated images. The output of this process was subjected to another round of compression using a more robust but slower compressor (PAQ8f). The compression ratio of the second compression was, on average, 18%, which is high for random data. The results of the second compression were superior to the lossy JPEG. Under the used data set, lossy JPEG needs to sacrifice 10% on average to realize nearly total lossless compression ratios of the two-successive compressions. To generalize the results, fast general-purpose compression algorithms (7z, bz2, and Gzip) were used too. [ABSTRACT FROM AUTHOR]

The trajectory information generated by the moving object plays an important role in studying the object movement. In this paper, a trajectory data compression algorithm based on the motion state changing is proposed to reduce trajectory data storage space and increase compression speed, which can accurately show the motion state and trajectory characteristics. This study has certain significance for the exploration of mass traffic data and the planning of traffic network. Combining the angle threshold with the velocity threshold of a moving object, the key data points are found and the redundant information is removed. Subsequently, the compressed trajectory is obtained. The experimental results show that the new algorithm can help to improve compression efficiency. The compressed trajectory has high similarity with the original trajectory in movement tendency. [ABSTRACT FROM AUTHOR]

Aiming at solving network delay caused by large chunks of data in industrial Internet of Things, a data compression algorithm based on edge computing is creatively put forward in this paper. The data collected by sensors need to be handled in advance and are then processed by different single packet quantity K and error threshold e for multiple groups of comparative experiments, which greatly reduces the amount of data transmission under the premise of ensuring the instantaneity and effectiveness of data. On the basis of compression processing, an outlier detection algorithm based on isolated forest is proposed, which can accurately identify the anomaly caused by gradual change and sudden change and control and adjust the action of equipment, in order to meet the control requirement. As is shown by experimental simulation, the isolated forest algorithm based on partition outperforms box graph and K-means clustering algorithm based on distance in anomaly detection, which verifies the feasibility and advantages of the former in data compression and detection accuracy. [ABSTRACT FROM AUTHOR]

Controller area network (CAN) has become the most widely used protocol in automotive networks due to its efficiency and robustness. The number of electronic control units (ECUs) in modern vehicles continually increases as more control functions are needed. CAN buses with limited bandwidths may be overloaded due to increases in bus load. Therefore, reducing the bus load is important in high-end vehicles. In this paper, we propose a multi-level data arrangement algorithm to effectively reduce the amount of transmitted data. Our experimental results show that the proposed algorithm improves the data compression ratio by approximately 25 % compared to previous algorithms. [ABSTRACT FROM AUTHOR]

Due to the low compression performance of traditional compression models, we have developed a new HOA based Fractional KCA with 2D-DWT for improving the multispectral image quality. In this paper, we present a novel multispectral image compression method for improving the complexity by maintaining quality reconstruction and also reducing the size of the storage of multispectral images. Initially, Karhunen–Loeve transform (KLT) is used to remove the spatial redundancies. In the second stage, 2D DWT is used to eliminate the intraband spatial redundancies. In the third stage, Fractional KCA (FKCA) is applied to improve the post-transformation process. FKCA is connected to the band of all wavelet sub-bands to minimize the spatial redundancy between intra sub-bands. Finally, the Hybrid Optimal algorithm (HOA) based FKCA is used to eliminate the residual and information redundancy among the neighboring bands. The experimental analysis of proposed 2D-DWT based Fractional KCA shows that the model improves the performance of compression data in terms of PSNR, MSSI, and VIF. Also, the multispectral image dataset shows the proposed compression model outperforms the existing compression models such as FKLT + PCA, ADWT + OADL, and DWT + DCT [ABSTRACT FROM AUTHOR]

The pervasiveness of GPS-enabled devices and wireless communication technologies results in massive trajectory data, incurring expensive cost for storage, transmission, and query processing. To relieve this problem, in this paper we propose a novel framework for compressing trajectory data, REST (Reference-based Spatio-temporal trajectory compression), by which a raw trajectory is represented by concatenation of a series of historical (sub-)trajectories (called reference trajectories) that form the compressed trajectory within a given spatio-temporal deviation threshold. In order to construct a reference trajectory set that can most benefit the subsequent compression, we propose three kinds of techniques to select reference trajectories wisely from a large dataset such that the resulting reference set is more compact yet covering most footprints of trajectories in the area of interest. To address the computational issue caused by the large number of combinations of reference trajectories that may exist for resembling a given trajectory, we propose efficient greedy algorithms that run in the blink of an eye and dynamic programming algorithms that can achieve the optimal compression ratio. Compared to existing work on trajectory compression, our framework has few assumptions about data such as moving within a road network or moving with constant direction and speed, and better compression performance with fairly small spatio-temporal loss. In addition, by indexing the reference trajectories directly with an in-memory R-tree and building connections to the raw trajectories with inverted index, we develop an extremely efficient algorithm that can answer spatio-temporal range queries over trajectories in their compressed form. Extensive experiments on a real taxi trajectory dataset demonstrate the superiority of our framework over existing representative approaches in terms of both compression ratio and efficiency. [ABSTRACT FROM AUTHOR]

Graphical data is comprised of a graph with marks on its edges and vertices. The mark indicates the value of some attribute associated to the respective edge or vertex. Examples of such data arise in social networks, molecular and systems biology, and web graphs, as well as in several other application areas. Our goal is to design schemes that can efficiently compress such graphical data without making assumptions about its stochastic properties. Namely, we wish to develop a universal compression algorithm for graphical data sources. To formalize this goal, we employ the framework of local weak convergence, also called the objective method, which provides a technique to think of a marked graph as a kind of stationary stochastic processes, stationary with respect to movement between vertices of the graph. In recent work, we have generalized a notion of entropy for unmarked graphs in this framework, due to Bordenave and Caputo, to the case of marked graphs. We use this notion to evaluate the efficiency of a compression scheme. The lossless compression scheme we propose in this paper is then proved to be universally optimal in a precise technical sense. It is also capable of performing local data queries in the compressed form. [ABSTRACT FROM AUTHOR]

Wireless sensor networks (WSNs) generate a variety of continuous data streams. To reduce data storage and transmission cost, compression is recommended to be applied to the data streams from every single sensor node. Local compression falls into two categories: lossless and lossy. Lossy compression techniques are generally preferable for sensors in commercial nodes than the lossless ones as they provide a better compression ratio at a lower computational cost. However, the traditional approaches for data compression in WSNs are sensitive to sensor accuracy. They are less efficient when there are abnormal and faulty measurements or missing data. This paper proposes a new lossy compression approach using the Bayesian predictive coding (BPC). Instead of the original signals, predictive coding transmits the error terms which are calculated by subtracting the predicted signals from the actual signals to the receiving node. Its compression performance depends on the accuracy of the adopted prediction technique. BPC combines the Bayesian inference with the predictive coding. Prediction is made by the Bayesian inference instead of regression models as in traditional predictive coding. In this way, it can utilize prior information and provide inferences that are conditional on the data without reliance on asymptotic approximation. Experimental tests show that the BPC is the same efficient as the linear predictive coding when handling independent signals which follow a stationary probability distribution. More than that, the BPC is more robust toward occasionally erroneous or missing sensor data. The proposed approach is based on the physical knowledge of the phenomenon in applications. It can be considered as a complementary approach to the existing lossy compression family for WSNs. [ABSTRACT FROM AUTHOR]

Road-network data compression or simplification reduces the size of the network to occupy less storage with the aim to fit small form-factor routing devices, mobile devices, or embedded systems. Simplification (a) reduces the storage cost of memory and disks, and (b) reduces the I/O and communication overhead. There are several road network compression techniques proposed in the literature. These techniques are evaluated by their compression ratios. However, none of these techniques takes into consideration the possibility that the generated compressed data can be used directly in Map-matching operation which is an essential component for all location-aware services. Map-matching matches a measured latitude and longitude of an object to an edge in the road network graph. In this paper, we propose a novel simplification technique, named COMA, that (1) significantly reduces the size of a given road network graph, (2) achieves high map-matching quality on the simplified graph, and (3) enables the generated compressed road network graph to be used directly in map-matching and location-based applications without a need to decompress it beforehand. COMA smartly deletes those nodes and edges that will not affect the graph connectivity nor causing much of ambiguity in the map-matching of objects' location. COMA employs a controllable parameter; termed a conflict factor C , whereby location aware services can trade the compression gain with map-matching accuracy at varying granularity. We show that the time complexity of our COMA algorithm is O(|N|log|N|). Intensive experimental evaluation based on a real implementation and data demonstrates that COMA can achieve about a 75% compression-ratio while preserving high map-matching quality. Road Network, Simplification, Compression, Spatial, Location, Performance, Accuracy, Efficiency, Scalability. [ABSTRACT FROM AUTHOR]

The size of a high-density-camera-array (HDCA)-based light field image (LFI) is usually very large, containing hundreds of high-resolution views. Therefore, there is an urgent need to efficiently compress it. Currently, no compression algorithms, specially, for the HDCA-based LFI have been designed. In this paper, we propose an algorithm based on a quadtree-based 2D hierarchical coding framework for the HDCA-based LFI data compression. The proposed framework has the following contributions. First, we organize the views of the HDCA-based LFI into a quadtree-based coding structure. Under this structure, all of the views are divided into four quadrants at the first level. Each quadrant is further sub-divided into four quadrants at each subsequent level. The process continues until the desired depth is reached. This quadtree-based coding structure can make full use of the strong inter-view correlations to improve the coding efficiency. In addition, the proposed quadtree-based structure can be easily extended to a general 2D hierarchical structure with variable group of pictures (GOP) sizes to adapt to the reference frame buffer constraint. Second, we try to improve the performance of the 2D hierarchical coding framework using the distance-based criteria for both the reference frame selection and motion vector scaling. Third, a one-pass optimal bit allocation scheme is proposed to further optimize the performance by taking the quality dependencies among various views into consideration. The proposed framework is implemented in the newest video coding standard, high efficiency video coding (HEVC). The experimental results show that the proposed quadtree-based 2D hierarchical coding framework can achieve an average of over 25% bitrate saving compared with the 1D hierarchical coding structure. [ABSTRACT FROM AUTHOR]

Information security is one of the significant research challenges in the IT world. Most of the time, information is transmitted through a network that is not in a secured format. In this paper, the proposed work is to improve the critical strength of the DNA algorithm. The key strength is improved with the combination of the genetic algorithm and the Diffie-Hellman key exchange algorithm. The Shannon entropy measures the data compression while the critical entropy measures the key strength. [ABSTRACT FROM AUTHOR]

Lossy compression of scientific data arrays is a powerful tool to save network bandwidth and storage space. Properly applied lossy compression can reduce the size of a dataset by orders of magnitude keeping all essential information, whereas a wrong choice of lossy compression parameters leads to the loss of valuable data. The paper considers statistical properties of several lossy compression methods implemented in "NetCDF operators" (NCO), a popular tool for handling and transformation of numerical data in NetCDF format. We compare the effects of imprecisions and artifacts resulting from use of a lossy compression of floating-point data arrays. In particular, we show that a popular Bit Grooming algorithm (default in NCO) has sub-optimal accuracy and produces substantial artifacts in multipoint statistics. We suggest a simple implementation of two algorithms that are free from these artifacts and have twice higher precision. Besides that, we suggest a way to rectify the data already processed with Bit Grooming. The algorithm has been contributed to NCO mainstream. The supplementary material contains the implementation of the algorithm in Python 3. [ABSTRACT FROM AUTHOR]

Widespread use of Multichannel Electroencephalograph (MCEEG) in diversified fields ranging from clinical studies to Brain Computer Interface (BCI) application, has put in a lot of thrust in data processing concepts, for effective storage and transmission. The paper proposes a computationally simple and novel methodology Normalized Spatial Pseudo Codec (n-SPC) to compress MCEEG signals. The signals are first normalized followed by two operations namely the spatial coding and pseudo coding operating on integer part and fractional part of the normalized data respectively. The proposed method was evaluated on publicly available EEG databases and results indicate that the algorithm exhibits good storage efficiency with average Compression Ratio (CR) of 4.61 with a computational complexity of only O(zN). The algorithm offers significantly a better decompressed signal quality, quantified by average Peak Signal to Noise Ratio (PSNR) of 21.42 dB. The average encoding and decoding time per sample is 0.3 and 0.04 ms, respectively with an average Percentage Root Mean Square Deviation (PRD) of 5.33. The efficacy was further evaluated using the decompressed signal to detect sleep spindle, from an excerpt of EEG recording and was compared with the visual scoring of two experts, available at the DREAMS Sleep Spindles Database. Hence, the proposed compression scheme can be used in practical MCEEG recording, archiving and BCI and neuromorphic systems. [ABSTRACT FROM AUTHOR]

Today, there is a huge demand for data compression due to the need to reduce the transmission time and increase the capacity of data storage. Data compression is a technique which represents an information, images, video files in a compressed or in a compact format. There are various data compression techniques which keep information as accurately as possible with the fewest number of bits and send it through communication channel. Arithmetic algorithm, Lempel–Ziv 77 (LZ77) and run length encoding with a K-precision (K-RLE) algorithms are lossless data compression algorithms which have lower performance rate because of their processing complexity as well as execution time. This paper presents an efficient parallel approach to reduce execution time for compression algorithms. The proposed OpenMP is an efficient tool for programming within parallel shared-memory environments. Finally, it shows that performance parallel model experimented using Open Multi-Processing (OpenMP) Application Programming Interface through Intel Parallel studio on multicore architecture platform with spec of Core 2 duo—2.4 GHz, 1 Gb RAM machine of parallel approach for compression algorithms has been improved remarkably against sequential approach. The improvement in compression ratio through an efficient parallel approach leads to reduction on transmission cost, reduction in storage space and bandwidth without additional hardware infrastructure. An overall performance evaluation shows arithmetic data compression algorithm with 46% which is better than LZ77 of 44% as well as K-RLE of 37% data compression algorithms. [ABSTRACT FROM AUTHOR]

Forgery detection is essential to verify the integrity and authenticity of images. Existing block-based detection techniques detect forgery in the same image, most of which use similar frameworks while differ in the feature extraction schemes. These methods have high accuracy in detecting the forged regions, but the computational load is heavy when facing exhaustive search problems. This paper describes a forgery detection method based on local binary pattern residue classes and color regions. An image is divided into overlapped blocks. Local binary pattern residue classes are computed for each block. The plane formed by a dimensional and b dimensional from Lab color space is divided into 16 regions. Similar blocks are searched in the overlapped blocks with the same local binary pattern residue class and color region, then they are grouped into several suspicious regions. Finally, we analyze the multi-region relation of these suspicious regions and their areas to locate the tampered regions. The small hole is filled through the morphologic operation. The results of experiments demonstrated that our method has good performance in that it improved detection accuracy and reduced execution time under various challenging conditions. As the proposed method reduces the search range for similar blocks, it has a higher speed than exhaustive search and has comparable detection results at the same time. [ABSTRACT FROM AUTHOR]

• The reconstruction model is established for vibration signals. • The index vector is proposed to obtain all the feature coefficients. • The improve FISTA is proposed for a better performance in reconstruction. • The simulated signals and acquired signals are used to verify the effectiveness. We consider the compressive sensing reconstruction for vibration signals, which are complex due to the harsh working environment. The recent fast iterative shrinkage-thresholding algorithm (FISTA) has paved the way for the signal reconstruction with a low complexity and high efficiency. Unfortunately, when extending to the vibration signals, the current algorithm still has some drawbacks such as the bad reconstruction effect. In this paper, we propose the improved fast iterative shrinkage-thresholding algorithm (IFISTA) to improve the reconstruction effect. Under the new scheme, the reconstruction is promoted by extracting information from the unstable signals in the process of iteration. Then the feature coefficients will be protected from shrinkage during iteration. The effectiveness of the IFISTA is verified by simulated signals and acquired signals. It is showed that the proposed scheme has superior performance in reconstruction and feature protection for vibration signals. [ABSTRACT FROM AUTHOR]

In this paper we investigate the possibility of improving the efficiency of data compression, and thus reducing storage requirements, for seven widely used text document formats. We propose an open-source text compression software library, featuring an advanced word-substitution scheme with static and semidynamic word dictionaries. The empirical results show an average storage space reduction as high as 78 percent compared to uncompressed documents, and as high as 30 percent compared to documents compressed with the free compression software gzip. [ABSTRACT FROM AUTHOR]

Federated Learning (FL), which enables multiple clients to cooperatively train global models without revealing private data, has gained significant attention from researchers in recent years. However, the data samples on each participating device in FL are often not independent and identically distributed (IID), leading to significant statistical heterogeneity challenges. In this paper, we propose FL-Enhance, a novel framework to address the non-IID-ness data issue in FL by leveraging established solutions such as data selection, data compression, and data augmentation. FL-Enhance, specifically, utilizes cGANs that are trained locally on the server level, which represents a relatively novel approach within the FL framework. Also, data compression techniques are applied to preserve privacy during data sharing between clients and servers. We compare our framework with the commonly used SMOTE data augmentation technique and test it with different FL algorithms, including FedAvg, FedNova, and FedOpt. We conducted experiments using both image and tabular data to evaluate the effectiveness of our proposed framework. The experimental findings show that FL-Enhance can substantially enhance the performance of the trained models in situations of severe pathological clients while still preserving privacy, which is the fundamental requirement in the FL context. • We introduce FL-Enhance, a new data-selection-based method to handle pathological non-IID-ness in FL. • We experiment on various datasets and compare FedAvg, FedNova, and FedOpt algorithms. • We are among the first to integrate cGAN into FL especially on tabular datasets. • FL-Enhance provides better privacy protection compared to baseline FL methods. [ABSTRACT FROM AUTHOR]

Remote monitoring of bearing conditions, using wireless sensor network (WSN), is a developing trend in the industrial field. In complicated industrial environments, WSN face three main constraints: low energy, less memory, and low operational capability. Conventional data-compression methods, which concentrate on data compression only, cannot overcome these limitations. Aiming at these problems, this paper proposed a compressed data acquisition and reconstruction scheme based on Compressed Sensing (CS) which is a novel signal-processing technique and applied it for bearing conditions monitoring via WSN. The compressed data acquisition is realized by projection transformation and can greatly reduce the data volume, which needs the nodes to process and transmit. The reconstruction of original signals is achieved in the host computer by complicated algorithms. The bearing vibration signals not only exhibit the sparsity property, but also have specific structures. This paper introduced the block sparse Bayesian learning (BSBL) algorithm which works by utilizing the block property and inherent structures of signals to reconstruct CS sparsity coefficients of transform domains and further recover the original signals. By using the BSBL, CS reconstruction can be improved remarkably. Experiments and analyses showed that BSBL method has good performance and is suitable for practical bearing-condition monitoring. [ABSTRACT FROM AUTHOR]

This paper brings an idea of establishing a downstream bit channel based on reusing of bit sequences and bit compression technique in point-to-multipoint passive optical networks for increasing their summary throughput. The downstream communication of these networks is based on broadcasting of multi-frames towards all active end-point nodes, therefore these frames cold be potentially reused multiple times to carry additional data in a form of segmented bit sequences spread within the entire bitstream. In order to achieve maximum effectiveness of a proposed method, bit addressing technique using relative sequence positions is proposed. Moreover, the presented bit compression technique might further increase the resulting summary throughput, while the application of existing bit compression algorithms cannot result in better performance. The illustrate the potential of a proposed algorithm together with the bit compression technique, the simulations and evaluations were performed and are presented within this paper as well. [ABSTRACT FROM AUTHOR]