Search

File similarity is a numerical indicator that how many duplicated data exist in target files. With this information, we can reduce storage capacity with data deduplication scheme, further it can be exploited in the digital forensic field for finding malicious software. However, measuring file similarity between files can cause a high overhead in terms of processing time and the capacity of disk storage. For this reason, in this paper, we propose a novel file similarity evaluation algorithm called PHISA (Partial Hash Information String Algorithm). To evaluate the performance of the proposed system, we compare PHISA to well-known file similarity tools. The evaluation result shows that PHISA reduces the processing time and increases the similarity evaluation accuracy.

Data deduplication is widely used to reduce storage space requirement in cloud storage system. Especially chunking based deduplication scheme is very useful for handling data storage system that contains duplicated blocks of file stream. In this paper, we introduce a novel data deduplication scheme that manages metadata of cloud storage system in a rapid time. The key points of this paper are using tree map searching and classifying data as global and local metadata. The two-level metadata management scheme is the main aspects to influencing fast performance of the data deduplication.

In cloud storage system, when we search similar documentation files, keyword-based similarity evaluation scheme is well performed. However, if we want to find similar binary files then it is very difficult to satisfy user request. Because there is no widely used binary file search system that supports similarity evaluation among files. File similarity evaluation is essential for digital forensic and data deduplication field. In the file similarity processing time, the CPU consumption and resource overhead of memory are increased as the number of files increase. Moreover, as the file size is getting bigger, the overhead of metadata management is critical. In this paper, we suggest the similarity evaluation scheme using a hybrid chunking which reduce overall processing time of similarity evaluation. Experiment result shows that the proposed system can reduce processing time and data storage capacity effectively.

In this paper, we present an enhanced storage system that supports Byte-index chunking algorithm. The storage system aims to provide efficient data deduplication with high performance and to be performed in rapid time. We describe the overall procedure of Byteindex chunking based storage system including read/write procedure and how the system works. The key idea of Byte-index chunking is to adapt fixed-size block chunk scheme which are distributed to “Index-table” by chunk’s both side boundary values. We have found that Byte-index chunking in storage system provides high performance compared with other chunking schemes. Experiments result shows that the storage system with Byte-index chunking compresses overall data with high deduplication capability and reduce the speed of file processing.

Background/Objectives: Recently, storage systems and backup systems are popularly used and the number of duplicated data is increased drastically. To minimize data storage size and efficient use of network bandwidth, we proposed deduplication systems and file similarity measurement schemes with GPGPU scheme. The GPGPUs are applied to file similarity measurement for computation speedup. Methods/Statistical Analysis: To cope with the problem accompanying the parallelization of the measurement, we compare two implementations with shared memory and preprocessing. In addition, we propose an alternative to Rabin fingerprinting algorithm to lessen the computational burden of the algorithm to the GPUs. We compare the performance of the systems in time elapsed for several files. Findings: First, we found through experiments that the preprocessing was slightly faster than the shared memory scheme for the overlapped region of consecutive data segments which were assigned to different cores. This region should be shared by two cores for fingerprinting. By adapting GPGPU parallelization with the preprocessing technique for file similarity measurement, the proposed system outperformed the systems with a multi-core CPU. Also, it gets faster for the bigger file. In addition, we made the system three times faster by adapting an alternative to Rabin fingerprinting algorithm. It eliminates the computational burden of the algorithm and provides comparable results to the system with the latter. Improvements: The procedure will be beneficial to de-duplication system in determining file similarity and finding duplicated regions of two files. We achieved speedup in the measurement of file similarity by parallelization on GP-GPUs with two methods for overlaps of consecutive data segments and an alternative fingerprinting algorithm.

In traditional target-based data deduplication system, all of the files should be chunked and compared for reducing duplicated data blocks. One of the critical problem of this system arises as the number of files are increasing. The system suffers from computational delay for calculating hash value and processing metadata for handling each file. To overcome this problem, in this paper, we propose a novel data deduplication system using logical partition of storage system. The system applies data deduplication scheme to each logical partition not each file. Experiment result shows that the proposed system is more efficient compared with traditional deduplication scheme where the logical partition is full of files by 50% in terms of deduplication capacity and processing time.

To effectively handle duplicate files, data deduplication schemes are widely used in many storage systems. Data deduplication algorithms reduce storage space by eliminating data to ensure that only single instance of data is stored in storage device. In this paper, we propose an energy efficient file synchronization scheme that provides hybrid data chunking using variable-length chunking (VLC) and fixed-length chunking (FLC). The main idea is to analyze similarities between old and new versions of data and decide which chunking method to apply in synchronizing the files. In particular, the proposed algorithm exploits the file similarity pattern for calculating the energy efficiency of chunking algorithms. We have developed an Android mobile application for file synchronization and measured energy consumption. The experiment results show that the proposed scheme helps save energy in synchronizing files, regardless of file types or amount of redundancies the files have.

A file synchronization program has to exchange lots of metadata information between a client and a server. As data have been growing rapidly in file synchronization systems, the metadata exchanging overhead is more critical now than before. In this paper, we propose a practical approach for minimizing metadata exchange overhead on the file synchronization system using variable-length chunking. The key idea is to minimize metadata by adapting file similarity information and generating only one hash key for several metadata. In the experiment, traditional file synchronization program sends 4.3 Mbyte over 500 MByte data file. However, the proposed algorithm only sends 37 Kbyte metadata.

Data deduplication is widely used in storage systems to prevent duplicated data blocks. In this paper, we suggest a dynamic chunking approach using fixed-length chunking and file similarity technique. The fixed-length chunking struggles with boundary shift problem and shows poor performance when handling duplicated data files. The key idea of this work is to utilize duplicated data information in the file similarity information. We can easily find several duplicated point by comparing hash key value and file offset within file similarity information. We consider these duplicated points as a hint for starting position of chunking. With this approach, we can significantly improve the performance of data deduplication system using fixed-length chunking. In experiment result, the proposed dynamic chunking results in significant performance improvement for deduplication processing capability and shows fast processing time comparable to that of fixed length chunking.

TAR archive format does not support efficient sub-file deletion and update. Therefore it has drawbacks for utilizing disk storage when it used as a backup tool. To address this problem, we propose an extended TAR file format called DTAR (Deduplication Tape Archive) which provides file-level deduplication and compression. The key idea of our work is to adapt user-level file system which supports efficient block insertion/deletion for DTAR format using block relocation. Experiment results show that the proposed system can reduce data storage space and computation time extremely.

In this paper, we propose a file deduplication system considering file similarity information. The key idea of this paper is to reduce the energy consumption of a mobile device by minimizing network traffic. The proposed system can detect duplication ratio of a file using file similarity algorithm. In this work, for a file with high data redundancy, data deduplication is performed during file transferring, otherwise, conventional file transfer scheme works without data deduplication. We implemented Fixed-length Chunking and Variable-length Chunking deduplication algorithm using file similarity. Experiment result shows that the proposed scheme significantly reduces energy consumption of data deduplication.

Tar archive format does not support file deduplication scheme, therefore it has drawbacks for utilizing disk storage system efficiently. In this paper, we propose an extended TAR file format called DTAR (Deduplication Tape Archive) which provides file-level deduplication. Key idea of our work is to provide block aligned compressed file that can speed up file insertion and deletion for archiving. Furthermore, DTAR shows performance enhancement using file similarity scheme for distributing files into several storage nodes. Experiment results show that the proposed system can reduce data storage space efficiently and diminish network data traffic compared to general file transfer system.

The implementation approaches of data deduplication system divide into several modes including SBA(source-based approach), ILA(in-line approach) and PPA(post-process approach). Currently, most commercial systems are implemented and operated in an ILA and PPA approach, and some researchers have focused on the SBA approach. As data deduplication systems are widely used, to choose an appropriate mode considering operation environment becomes more and more important than ever. Because the overhead of each mode and resource usage wasn't fully studied, in some operating environments, the deduplication mode can lead to inefficiency and poor performance. In this study, we propose a data deduplication system supporting multi-mode. The proposed system can be operated in a mode that a user specifies during system operation, therefore, this system can be dynamically adjusted under consideration of system characteristics. In this paper, we operate the proposed system with the SBA, ILA and PPA mode, respectively, and we present the measurement results with a comparative analysis of the mode-specific performance and overhead.

In a mobile device, it is important to minimize battery consumption when we operate an application. In this paper, we propose an energy efficient file transfer scheme for mobile device by exploiting deduplication scheme. Generally, a mobile phone needs more power for WiFi work rather than CPU work, therefore we have to minimize WiFi work. We can minimize the number of packets by adapting deduplication which consumes lots of CPU power. Deduplication work consumes additional CPU power during data hashing, however overall energy consumption can be diminished. Experiment result shows that the proposed scheme significantly reduces battery consumption if a file has over 5% SRP(Space Reduction Percentage).

In this paper we proposes a clustering backup system that exploits file fingerprint mechanism for multi-level deduplication of redundant data. Our approach differs from the traditional file server system. First, we avoid the data redundancy by block-level fingerprint. The proposed approach enables the efficient use of the storage capacity and network bandwidth without the transmission of the duplicate data block. Second, we applied clustering technology because data transfer and I/O time is reduced a fraction of a percent for each node. In this paper, we made several experiments to verify performance of our proposed method. Experiments result shows that storage space is used efficiently and the performance is noticeably improved.