Search

Due to their massive popularity, image files, especially JPEG, offer high potential as carriers of other information. Much of the work to date on this has focused on stenographic ways of hiding information using least significant bit techniques but we believe that the findings in this project have exposed other ways of doing this. We demonstrate that a digital certificate relating to an image file can be inserted inside that image file along with accompanying metadata containing references to the issuing company. Notwithstanding variations between devices and across operating systems and applications, a JPEG file holds its structure very well. Where changes do take place, this is generally in the metadata area and does not affect the encoded image data which is the heart of the file and the part that needs to be verifiable. References to the issuing company can be inserted into the metadata for the file. There is an advantage of having the digital certificate as an integral part of the file to which it applies and consequently travelling with the file. We ultimately prove that the metadata within a file offers the potential to include data that can be used to prove integrity, authenticity and provenance of the digital content within the file. Keywords: Secure authentication, Integrity, Digital certificates, Security, Network security

MapReduce is a popular computing model for parallel data processing on large-scale datasets, which can vary from gigabytes to terabytes and petabytes. Though Hadoop MapReduce normally uses Hadoop Distributed File System (HDFS) local file system, it can be configured to use a remote file system. Then, an interesting question is raised: for a given application, which is the best running platform among the different combinations of scale-up and scale-out Hadoop with remote and local file systems. However, there has been no previous research on how different types of applications (e.g., CPU-intensive, data-intensive) with different characteristics (e.g., input data size) can benefit from the different platforms. Thus, in this paper, we conduct a comprehensive performance measurement of different applications on scale-up and scale-out clusters configured with HDFS and a remote file system (i.e., OFS), respectively. We identify and study how different job characteristics (e.g., input data size, the number of file reads/writes, and the amount of computations) affect the performance of different applications on the different platforms. Based on the measurement results, we also propose a performance prediction model to help users select the best platforms that lead to the minimum latency. Our evaluation using a Facebook workload trace demonstrates the effectiveness of our prediction model. This study is expected to provide a guidance for users to choose the best platform to run different applications with different characteristics in the environment that provides both remote and local storage, such as HPC cluster and cloud environment.

A large class of modern distributed file systems treat metadata services as an independent system component, separately from data servers. The availability of the metadata service is key to the availability of the overall system. Given the high rates of failures observed in large-scale data centers, distributed file systems usually incorporate high-availability (HA) features. A typical approach in the development of distributed file systems is to design and develop metadata services from the ground up, at significant cost in terms of complexity and time, often leading to functional shortcomings. Our motivation in this paper was to improve on this state of things by defining a general-purpose architecture for HA metadata services (which we call RMS) that can be easily incorporated and reused in new or existing file systems, reducing development time. Taking two prominent distributed file systems as case studies, PVFS and HDFS, we developed RMS variants that improve on functional shortcomings of the original HA solutions, while being easy to build and test. Our extensive evaluation of the RMS variant of HDFS shows that it does not incur an overall performance or availability penalty compared to the original implementation.

A middleware is proposed to optimize file fetch process in transparent computing (TC) platform. A single TC server will receive file requests of large scale distributed operating systems, applications or user data from multiple clients. In consideration of limited size of servers memory and the dependency among files, this work proposes a middleware to provide a file fetch sequence satisfying: (1) each client, upon receiving any file, is able to directly load it without waiting for pre-required files (i.e. receive and load); and (2) the server is able to achieve optimization in reducing overall file fetch time cost. The paper firstly addresses the features of valid file fetch sequence generating problem in the middleware. The method solves the concurrency control problem when the file fetch is required for the multiple clients. Then it explores the methods to determine time cost for file fetch sequence. Based on the established model, we propose a heuristic and greedy (HG) algorithm. According to the simulation results, we conclude that HG algorithm is able to reduce overall file fetch time roughly by 50% in the best cases compared with the time cost of traditional approaches. A middleware solution is proposed to optimize file fetch process in transparent computing (TC) environment.The method solves the concurrency control problem when the file fetch is required for the multiple clients.We propose a heuristic and greedy (HG) algorithm.HG algorithm can reduce overall file fetch time roughly by 50% in the best cases compared with the time cost of traditional approaches.

With the fast increase of multimedia contents, efficient forensics investigation methods for multimedia files have been required. In multimedia files, the similarity means that the identical media (audio and video) data are existing among multimedia files. This paper proposes an efficient multimedia file forensics system based on file similarity search of video contents. The proposed system needs two key techniques. First is a media-aware information detection technique. The first critical step for the similarity search is to find the meaningful keyframes or key sequences in the shots through a multimedia file, in order to recognize altered files from the same source file. Second is a video fingerprint-based technique (VFB) for file similarity search. The byte for byte comparison is an inefficient similarity searching method for large files such as multimedia. The VFB technique is an efficient method to extract video features from the large multimedia files. It also provides an independent media-aware identification method for detecting alterations to the source video file (e.g., frame rates, resolutions, and formats, etc.). In this paper, we focus on two key challenges: to generate robust video fingerprints by finding meaningful boundaries of a multimedia file, and to measure video similarity by using fingerprint-based matching. Our evaluation shows that the proposed system is possible to apply to realistic multimedia file forensics tools.

Recently, data replication has received considerable attention in the field of grid computing. The main goal of data replication algorithms is to optimize data access performance by replicating the most popular files. When a file does not exist in the node where it was requested, it necessarily has to be transferred from another node, causing delays in the completion the file requests. The general idea behind data replication is to keep track of the most popular files requested in the grid and create copies of them in selected nodes. In this way, more file requests can be completed over a period of time and average job execution time is reduced. In this paper, we introduce an algorithm that estimates the potential of the files located in each node of the grid, using a binary tree structure. Also, the file scope and the file type are taken into account. By potential of a file, we mean its increasing or decreasing demand over a period of time. The file scope generally refers to the extent of the group of users which are interested or potentially interested in a file. The file types are divided into read and write intensive. Our scheme mainly promotes the high-potential files for replication, based on the temporal locality principle. The simulation results indicate that the proposed scheme can offer better data access performance in terms of the hit ratio and the average job execution time, compared to other state-of-the-art strategies.

Identifying file systems metadata changes such as which files have been added, modified or removed from the file system has many usages. In this paper we present a framework we developed for identifying those changes in increasing speeds. Our framework which is composed of crawling, hashing, and scheduling components, allows to scale the crawl to multiple client workstations that operate in parallel on the same file system in a non-overlapping fashion. Experiments carried using real-world data indicate performance improvement (speedup) of up to 36X using our framework compared to legacy crawling utilities such as Linux’s ‘find’.

The large scale and distribution of cloud computing storage have become the major challenges in cloud forensics for file extraction. Current disk forensic methods do not adapt to cloud computing well and the forensic research on distributed file system is inadequate. To address the forensic problems, this paper uses the Hadoop distributed file system (HDFS) as a case study and proposes a forensic method for efficient file extraction based on three-level (3L) mapping. First, HDFS is analyzed from overall architecture to local file system. Second, the 3L mapping of an HDFS file from HDFS namespace to data blocks on local file system is established and a recovery method for deleted files based on 3L mapping is presented. Third, a multi-node Hadoop framework via Xen virtualization platform is set up to test the performance of the method. The results indicate that the proposed method could succeed in efficient location of large files stored across data nodes, make selective image of disk data and get high recovery rate of deleted files.

In forensic computing, especially in the field of postmortem file system forensics, the reconstruction of lost or deleted files plays a major role. The techniques that can be applied to this end strongly depend on the specifics of the file system in question. Various file systems are already well-investigated, such as FAT16/32, NTFS for Microsoft Windows systems and Ext2/3 as the most relevant file system for Linux systems. There also exist tools, such as the famous Sleuthkit (Carrier), that provide file recovery features for those file systems by interpreting the file system internal data structures. In case of an Ext file system, the interpretation of the so-called superblock is essential to interpret the data. The Ext4 file system can mainly be analyzed with the tools and techniques that have been developed for its predecessor Ext3, because most principles and internal structures remained unchanged. However, a few innovations have been implemented that have to be considered for file recovery. In this paper, we investigate those changes with respect to forensic file recovery and develop a novel approach to identify files in an Ext4 file system even in cases where the superblock is corrupted or overwritten, e.g. because of a re-formatting of the volume. Our approach applies heuristic search patterns for utilizing methods of file carving and combines them with metadata analysis. We implemented our approach as a proof of concept and integrated it into the Sleuthkit framework.

The convergence of high-performance computing and big data, which has become known as the field of extreme big data, is problematic in that file creation in storage systems such as distributed file systems is not optimized. That is, the large workload leads to the simultaneous creation of many files by many processes when creating checkpoints. The need to improve the file creation processes prompted us to design a scale-out distributed file system for post-petascale systems named PPFS. PPFS consists of PPMDS, a scale-out distributed metadata server, and PPOSS, a scalable distributed storage server for flash storage. High file creation performance of PPMDS was achieved by using a key-value store for metadata storage and non-blocking distributed transactions to update multiple entries simultaneously. PPOSS depends on PPOST, an object storage system that manages underlying low-level storage, such as Fusion IO ioDrive, a flash device connected through PCI express supporting OpenNVM. High file creation performance was attained by implementing the PPFS prototype using file creation optimization, termed bulk creation, to reduce the amount of communication between PPMDS and PPOSS. Moreover, to enhance the I/O performance of PPOSS when the client process and PPOSS run on the same node, PPOSS accesses the local storage device directly. The prototype implementation of PPFS with file creation optimization achieves 119,000 operations per second for file creation when using five metadata servers and 128 client processes, thereby exceeding the performance of IndexFS 2.17 times. With local access optimization, PPOSS reached its limit at a block size of 16 KiB, an improvement of 1.5 times compared to before optimization. Furthermore, this evaluation indicates that PPFS has scalability on file creation and IO performance, that is required for post-petascale systems.

The emerging technologies of persistent memory, such as PCM, MRAM, provide opportunities for preserving files in memory. Traditional file system structures may need to be re-studied. Even though there are several file systems proposed for memory, most of them have limited performance without fully utilizing the hardware at the processor side. This paper presents a framework based on a new concept, “File Virtual Address Space”. A file system, Sustainable In-Memory File System (SIMFS), is designed and implemented, which fully utilizes the memory mapping hardware at the file access path. First, SIMFS embeds the address space of an open file into the process’ address space. Then, file accesses are handled by the memory mapping hardware. Several optimization approaches are also presented for the proposed SIMFS. Extensive experiments are conducted. The experimental results show that the throughput of SIMFS achieves significant performance improvement over the state-of-the-art in-memory file systems.

Sub-file hashing and hash-based carving are increasingly popular methods in digital forensics to detect files on hard drives that are incomplete or have been partially overwritten/modified. While these techniques have been shown to be usable in practice and can be implemented efficiently, they face the problem that a-priori specific “target files” need to be available and at hand. While it is always feasible and, in fact, trivial to create case-specific sub-file hash collections, we propose the creation of case-independent sub-file hash databases. For facilitating hash databases which can be publicly shared among investigators, we propose the usage of data from peer-to-peer file sharing networks such as BitTorrent. Most of the file sharing networks in use today rely on large quantities of hash values for integrity checking and chunk identification, and can be leveraged for digital forensics. In this paper we show how these hash values can be of use for identifying possibly vast amounts of data and thus present a feasible solution to cope with the ever-increasing case sizes in digital forensics today. While the methodology used is independent of the used file sharing protocol, we harvested information from the BitTorrent network. In total we collected and analyzed more than 3.2 billion hash values from 2.3 million torrent files, and discuss to what extent they can be used to identify otherwise unknown file fragments and data remnants. Using open-source tools like bulk_extractor and hashdb, these hash values can be directly used to enhance the effectiveness of sub-file hashing at scale.

Distributed file searching in delay tolerant networks formed by mobile devices can potentially support various useful applications. In such networks, nodes often present certain social network properties of their holders in terms of contents (i.e., interests) and contacts. However, current methods in DTNs only consider either content or contact for file searching or dissemination, which limits the file sharing efficiency. In this paper, we first analyze real traces to confirm the importance and necessity of considering both content and contact in file search. We then propose Cont $^2$ , a social-aware file search method that exploits both node contents and contact patterns. First, considering people with common interests tend to share files and gather together, Cont $^2$ virtually groups common-interest nodes into a community to direct file search. Second, considering human mobility follows a certain pattern, Cont $^2$ exploits nodes’ contact frequencies with a community to expedite file searching. To further improve the searching efficiency, Cont $^2$ also integrates sub-communities and parallel forwarding as optional components for file searching. Trace-driven experiments on the GENI testbed and NS-2 simulator show that Cont $^2$ can effectively improve the search efficiency compared to current methods.

File similarity is a numerical indicator that how many duplicated data exist in target files. With this information, we can reduce storage capacity with data deduplication scheme, further it can be exploited in the digital forensic field for finding malicious software. However, measuring file similarity between files can cause a high overhead in terms of processing time and the capacity of disk storage. For this reason, in this paper, we propose a novel file similarity evaluation algorithm called PHISA (Partial Hash Information String Algorithm). To evaluate the performance of the proposed system, we compare PHISA to well-known file similarity tools. The evaluation result shows that PHISA reduces the processing time and increases the similarity evaluation accuracy.

As smart devices are revolutionized in recent years, people may generate enormous amount of various sized data and store them in the local or remote file system in their daily lives. With cheaper and easy to use private cloud storage appliances helping to handle the increasing demand of storing and sharing big volume of data, effective file deduplication schemes can greatly increase the space efficiency in private cloud storage systems as well as preserve network bandwidth. In the paper, we aim at designing and implementing several file deduplication schemes built in the private cloud storage appliance, based on different duplication checking rules, including file name, file size, and file partial/full content hash value. Experiment results show using partial content hashing based file deduplication scheme achieves a reasonably balanced performance without overutilized limited local computational resources.

In content based file sharing peer-to-peer (P2P) [1] network model nodes share files directly with each other without a centralized server. In such a file sharing system, nodes meet and exchange requests and files in the format of text, short videos, and voice clips in different interest categories. Content is various and large file sharing such as the multimedia content is required with the rapid development of the wireless communication technology. File sharing can also mean having an allocated amount of personal file storage in a common file system. A P2P content based file sharing system, for efficient file searching, threshold takes advantage of node mobility by designating stable nodes, which have the most frequent contact with community members, as community coordinators for intra community searching, and highly mobile nodes that visit other communities frequently as community ambassadors for intercommunity searching. The large file sharing needs more stable end to end path and long transmission time. Last but not least, more relationship between nodes will be used to promote the file sharing process. Content based file sharing is helpful for taking certain decisions during file transmission. These decisions will benefit in proper utilization of network resources. In this paper content-based file sharing scheme using threshold is proposed. The user's interest is determined by the proposed scheme before searching and sharing the files in the peer-to-peer network. The resources in the network are utilized as per the contents of the files to be shared. The performance evaluation show that proposed system significantly lowers transmission cost and improves file sharing success rate compared to current methods.

Nowadays, most of the information is stored digitally. Digital information is from a high level of view it is just an array of bits. In order to figure out its real meaning special software which interprets it is required. Therefore, if by evolution of technology this software cannot be executed anymore there is potential risk that also the data interpreted by it becomes not useful. The goal of Digital Preservation is to stop occurrences of such phenomenon. Data is commonly stored in files each file has a specific format or structure, by knowing it user can figure out the real meaning of raw data stored in the file as an array of bits. Digital Preservation considers valid file format as a perquisite for file to be in usable form, with valid is meant that a specific file is structured conform its declared file format. In this paper we throw a spotlight on the accuracy and capability of these file validation tests. Therefore, we present some open source software which are able to automatically identify and verify the file format. We focus more on file types they can identify, and how they work in large scale data sets.

Data Grid allows many organizations to share data across large geographical area. The idea behind data replication is to store copies of the same file at different locations. Therefore, if a copy at a location is lost or not available, it can be brought from another location. Additionally, data replication results in a reduced time and bandwidth because of bringing the file from a closer location. However, the files that need to be replicated have to be selected wisely. In this paper, a round-based data replication strategy is proposed to select the most appropriate files for replication at the end of each round based on a number of factors. The proposed strategy is based on Popular File Replicate First (PFRF) strategy, and it overcomes the drawbacks of PFRF. The simulation results show that the proposed strategy yields better performance in terms of average file delay per request, average file bandwidth consumption per request, and percentage of files found.

Outsourcing a file to a remote cloud storage provides several benefits, including scalability, accessibility, data replication and considerable cost saving. Unfortunately, when we send a file to a remote cloud storage server, we do not know if this file is intact. To address this problem, we present a remote file possession checking protocol. In this protocol, we use three technologies: Firstly, we design a kind of file block tag whose security is based on the discrete logarithm problem (DLP); secondly, we use probability checking method to improve its effectiveness; thirdly, we use tag aggregation method to reduce its communication cost. Our theory analysis and experiment results show that our protocol is practical and secure.

The processing of massive numbers of small files is a challenge in the design of distributed file systems. Currently, the combined-block-storage approach is prevalent. However, the approach employs the traditional file systems such as ExtFS and may cause inefficiency when accessing small files randomly located in the disk. This paper focuses on optimizing the performance of data servers in accessing massive numbers of small files. We present a Flat Lightweight File System (iFlatLFS) to manage small files, which is based on a simple metadata scheme and a flat storage architecture. iFlatLFS is designed to substitute the traditional file system on data servers and can be deployed underneath distributed file systems that store massive numbers of small files. iFlatLFS can greatly simplify the original data access procedure. The new metadata proposed in this paper occupies only a fraction of the metadata size based on traditional file systems. We have implemented iFlatLFS in CentOS 5.5 and integrated it into an open source Distributed File System (DFS), called Taobao FileSystem (TFS), which is developed by a top B2C service provider, Alibaba, in China and is managing over 28.6 billion small photos. We have conducted extensive experiments to verify the performance of iFlatLFS. The results show that when the file size ranges from 1 to 64 KB, iFlatLFS is faster than Ext4 by 48 and 54 percent on average for random read and write in the DFS environment, respectively. Moreover, after iFlatLFS is integrated into TFS, iFlatLFS-based TFS is faster than the existing Ext4-based TFS by 45 and 49 percent on average for random read access and hybrid access (the mix of read and write accesses), respectively.

A peer-to-peer (P2P) file sharing system provides a platform that enables a tremendous number of nodes to share their files. Retrieving desired files efficiently and trustworthily is critical in such a large and jumbled system. However, the issues of efficient searching and trustworthy searching have only been studied separately. Simply combining the methods to achieve the two goals doubles system overhead. In this paper, we first study trace data from Facebook and BitTorrent. Guided by the observations, we propose a system that integrates a social network into a P2P network, named Social-P2P, for simultaneous efficient and trustworthy file sharing. It incorporates three mechanisms: (1) interest/trust-based structure, (2) interest/trust-based file searching, and (3) trust relationship adjustment. By exploiting the social interests and relationships in the social network, the interest/trust-based structure groups common-multi-interest nodes into a cluster and further connects socially close nodes within a cluster. The comparably stable nodes in each cluster form a Distributed Hash Table (DHT) for inter-cluster file searching. In the interest/trust-based file searching mechanism, a file query is forwarded to the cluster of the file by the DHT routing first. Then, it is forwarded along constructed connections within a cluster, which achieves high hit rate and reliable routing. Moreover, sharing files among socially close friends discourages nodes from providing faulty files because people are unlikely to risk their reputation in the real-world. In the trust relationship adjustment mechanism, each node in a routing path adaptively decreases its trust on the node that has forwarded a faulty file in order to avoid routing queries towards misbehaving nodes later on. We conducted extensive trace-driven simulations and implemented a prototype on PlanetLab. Experimental results show that Social-P2P achieves highly efficient and trustworthy file sharing compared to current file sharing systems and trust management systems.

Recently, power-aware distributed file systems for efficient big data processing have increasingly moved toward power proportional designs. However, inefficient gear-shifting in such systems is an important issue that can seriously degrade their performance. To address this issue, we propose and evaluate an efficient gear-shifting power proportional distributed file system. The proposed system utilizes flexible data placement that reduces the amount of reflected data and has an architecture that improves the metadata management to achieve high-efficiency gear-shifting. Extensive empirical experiments using actual machines based on the HDFS demonstrated that the proposed system gains up to \(22\,\%\) better throughput-per-watt performance. Moreover, a suitable metadata management setting corresponding to the amount of data updated while in low gear is found from the experimental results.

This paper proposes a new approach, called file concealer (FC), to conceal files in a computer system. FC modifies metadata about a file in NTFS (New Technology File System) to hide the file. Unlike traditional hooking methods which can be easily detected by antivirus software, experimental results show that it is difficult for antivirus software to detect the files hidden by FC. Moreover, to enhance the concealment capability of FC, FC also rearranges the order of some data sectors of a hidden file. As a result, even if another person finds the original sectors used by the hidden file, it is difficult for him to recover the original content of the hidden file. Experimental results show that even data recovery tools cannot restore the content of a hidden file. All information that is required to restore a hidden file is stored in a file, called recovery file hereafter. When a user uses FC to hide a file, the user can specify any file as a host file, such as an image file, to which the recovery file will be appended. As a result, the user can easily restore a hidden file; however, it is difficult for other person to detect or restore the hidden file and the related recovery file.

At present, as the amount of digitized information is increasing geometrically, the importance of digitized information as critical clues from the perspective of criminal investigation is also increasing. The importance of digital forensics has also magnified accordingly. As a fundamental technique of digital forensics, recovery of data deleted to conceal information is extremely important and needs to be studied. Although file system types and versions are diversified, sufficient studies have not been performed. The XFS file system was designed to process huge files, and thus it is expected to be applied in server systems. In this paper, we analyze the XFS file system using its structure and metadata information; then, we propose a technique to recover the deleted files through previous analysis results, and verify it through testing.

Efficient file query is important to the overall performance of peer-to-peer (P2P) file sharing systems. Clustering peers by their common interests can significantly enhance the efficiency of file query. Clustering peers by their physical proximity can also improve file query performance. However, few current works are able to cluster peers based on both peer interest and physical proximity. Although structured P2Ps provide higher file query efficiency than unstructured P2Ps, it is difficult to realize it due to their strictly defined topologies. In this work, we introduce a Proximity-Aware and Interest-clustered P2P file sharing System (PAIS) based on a structured P2P, which forms physically-close nodes into a cluster and further groups physically-close and common-interest nodes into a sub-cluster based on a hierarchical topology. PAIS uses an intelligent file replication algorithm to further enhance file query efficiency. It creates replicas of files that are frequently requested by a group of physically close nodes in their location. Moreover, PAIS enhances the intra-sub-cluster file searching through several approaches. First, it further classifies the interest of a sub-cluster to a number of sub-interests, and clusters common-sub-interest nodes into a group for file sharing. Second, PAIS builds an overlay for each group that connects lower capacity nodes to higher capacity nodes for distributed file querying while avoiding node overload. Third, to reduce file searching delay, PAIS uses proactive file information collection so that a file requester can know if its requested file is in its nearby nodes. Fourth, to reduce the overhead of the file information collection, PAIS uses bloom filter based file information collection and corresponding distributed file searching. Fifth, to improve the file sharing efficiency, PAIS ranks the bloom filter results in order. Sixth, considering that a recently visited file tends to be visited again, the bloom filter based approach is enhanced by only checking the newly added bloom filter information to reduce file searching delay. Trace-driven experimental results from the real-world PlanetLab testbed demonstrate that PAIS dramatically reduces overhead and enhances the efficiency of file sharing with and without churn. Further, the experimental results show the high effectiveness of the intra-sub-cluster file searching approaches in improving file searching efficiency.

Big Data era is characterized by the explosive increase of image files on the Internet, massive image files bring great challenges to storage. It is required not only the storage efficiency of massive image files but also the accuracy and robustness of massive image file management and retrieval. To meet these requirements, distributed image file storage system based on cognitive theory is proposed. According to the human brain function, humans can correlate image files with thousands of distinct object and action categories and sorted store these files. Thus the authors proposed to sorted store image files according to different visual categories based on human cognition to resemble human memory. The experimental results demonstrate that the proposed distributed image file system (DIFS) based on cognition performs better than Hadoop Distributed File System (HDFS) and FastDFS.

Image File Format is one of the standardized ways of storing digital images, with a large number of image file formats available it would be very difficult and challenging for the user to select appropriate image format for storing the images. Choosing the proper image file format according to the demand of the project or according to a specific parameter is of vital importance. Therefore, in this paper we provide a brief review on most common and different image file formats in order to ease the user to choose appropriate file formats while working with files.

Large-scale small file storage for original pages degrades performance of search engines. In this paper, we first analyze the disadvantages of the existing EXT3 file system in accessing small files. Then, the rate and speed of compression algorithms are verified to choose a proper storage compression algorithm. Meanwhile, we design an original page oriented file organization structure and a read---write query tree to store the large-scale small files which need no modification. The accessing response time and disk space waste are remarkably decreased when search engines use these techniques to store original-page small files.

Retrieving files on personal computers is a fundamental component of interaction, yet there is surprisingly little empirical data characterising how it is carried out in realistic settings. We developed software, called FileMonitor, to dynamically record users׳ file retrieval activities, including data describing the files retrieved and the tools used to retrieve them. We then deployed the system in a four week log study of 26 participants׳ actual file retrievals on their personal computers. Follow-up interviews contextualised the findings. Results are presented in two sections focusing on the files (the number of files, patterns of revisitation, file types, etc.) and on the interface mechanisms used to retrieve them (file browsers, search tools, ‘recent files’ lists, etc.). We conclude by discussing implications for the design of next-generation file retrieval interfaces.

Checkpointing is the predominant storage driver in today's petascale supercomputers and is expected to remain as such in tomorrow's exascale supercomputers. Users typically prefer to checkpoint into a shared file yet parallel file systems often perform poorly for shared file writing. A powerful technique to address this problem is to transparently transform shared file writing into many exclusively written as is done in ADIOS and PLFS. Unfortunately, the metadata to reconstruct the fragments into the original file grows with the number of writers. As such, the current approach cannot scale to exaflop supercomputers due to the large overhead of creating and reassembling the metadata. In this paper, we develop and evaluate algorithms by which patterns in the PLFS metadata can be discovered and then used to replace the current metadata. Our evaluation shows that these patterns reduce the size of the metadata by several orders of magnitude, increase the performance of writes by up to 40 percent, and the performance of reads by up to 480 percent. This contribution therefore can allow current checkpointing models to survive the transition from petato exascale.

In this paper, we propose a methodology for identification of MP3 audio pieces downloaded by BitTorrent client program using cross-correlation of 12 chroma features extracted from audio signal. To prevent illegal distribution of copyrighted audio files from spreading over the internet, the file pieces split by torrent file must be detected as accurately as possible. In order to improve the probability of successfully identifying the audio pieces, we use Peak In Piece Position (PIPP) method which means the maximum peak appeared in a range corresponding to piece position and length is selected to reduce the probability of misidentification. The experimental results show probabilities of successfully identification of the pieces with different piece properties, comparisons between the proposed algorithms and conventional algorithm in terms of precisions and robustness of the proposed algorithms.

File storage load can be balanced in the storage nodes avail in the cloud system by using totally distributed load rebalancing algorithm. Large level distributed systems such as cloud applications come with rising challenges on how to transfer and where to store compute data. Cloud computing is a distributed computing over a network. Node concurrently serves as a computing and storage task. In cloud computing environment, files can also be dynamically created, deleted and append. The file chunks are not distributed as equally among the nodes. Lead to load inequity in a distributed file system. The existing distributed file system depends on single node to manage almost all operations such as chunk reallocation of every data block in the file system. As a result it can be bottleneck resource and a single point of failure. A new technique Random Linear Network Coding (RLNC) is employed in the proposed system. RLNC is performed at the opening when the file is stored in the cloud. Using this strategy, a file will be split into different parts and send distinct parts to each chunk server. RSA algorithm applied to calculate the response time file size and deadlock detection. RSA algorithm used to detect the anomalies of the system. Dynamic scheduling algorithm present in the proposed system to overcome to load inequity problem. This algorithm is compared against a centralized approach in a production system. The results indicate that our proposal is considerably outperforms the prior distributed in terms of load inequity factor, movement cost, and algorithmic overhead.

As leadership computing facilities grow their storage capacity into the multi- petabyte range, the number of files and directories leap into the scale of billions. A complete profiling of such a parallel file system in a production environment presents a unique challenge. On one hand, the time, resources, and negative performance impact on production users can make regular profiling difficult. On the other hand, the result of such profiling can yield much needed understanding of the file system's general characteristics, as well as provide insight to how users write and access their data on a grand scale. This paper presents a lightweight and scalable profiling solution that can efficiently walk, analyze, and profile multi-petabyte parallel file systems. This tool has been deployed and is in regular use on very large-scale production parallel file systems at both Oak Ridge National Lab's Oak Ridge Leadership Facility (OLCF) and Lawrence Livermore National Lab's Livermore Computing (LC) facilities. We present the results of our initial analysis on the data collected from these two large-scale production systems, organized into three use cases: (1) file system snapshot and composition, (2) striping pattern analysis for Lustre, and (3) simulated storage capacity utilization in preparation for future file systems. Our analysis shows that on the OLCF file system, over 96% of user files exhibit the default stripe width, potentially limiting performance on large files by underutilizing storage servers and disks. Our simulated block analysis quantitatively shows the space overhead when doing a forklift system migration. It also reveals that due to the difference in system compositions (OLCF vs. LC), we can achieve better performance and space trade-offs by employing different native file system block sizes.

Key-Value stores provide scalable metadata service for distributed file systems. However, the metadata's organization itself, which is organized using a directory tree structure, does not fit the key-value access pattern, thereby limiting the performance. To address this issue, we propose a distributed file system with a loosely-coupled metadata service, LocoFS, to bridge the performance gap between file system metadata and key-value stores. LocoFS is designed to decouple the dependencies between different kinds of metadata with two techniques. First, LocoFS decouples the directory content and structure, which organizes file and directory index nodes in a flat space while reversely indexing the directory entries. Second, it decouples the file metadata to further improve the key-value access performance. Evaluations show that LocoFS with eight nodes boosts the metadata throughput by 5 times, which approaches 93% throughput of a single-node key-value store, compared to 18% in the state-of-the-art IndexFS.

Data services such as search, discovery, and management in scalable distributed environments have traditionally been decoupled from the underlying file systems, and are often deployed using external databases and indexing services. However, modern data production rates, looming data movement costs, and the lack of metadata, entail revisiting the decoupled file system-data services design philosophy.In this paper, we present TagIt, a scalable data management service framework aimed at scientific datasets, which is tightly integrated into a shared-nothing distributed file system. A key feature of TagIt is a scalable, distributed metadata indexing framework, using which we implement a flexible tagging capability to support data discovery. The tags can also be associated with an active operator, for pre-processing, filtering, or automatic metadata extraction, which we seamlessly offload to file servers in a load-aware fashion. Our evaluation shows that TagIt can expedite data search by up to 10X over the extant decoupled approach.

In the big data era, the distributed file system is getting more and more significant due to the characteristics of\ud its scale-out capability, high availability, and high performance. Different distributed file systems may have different design goals. For example, some of them are designed to have good performance for small file operations, such as GlusterFS, while some of them are designed for large file operations, such as Hadoop distributed file system. With the divergence of big data applications, a distributed file system may provide good performance for some applications but fails for some other applications, that is, there has no universal distributed file system that can produce good performance for all applications. In this\ud paper, we propose a hybrid file system framework, HybridFS, which can deliver satisfactory performance for all applications. HybridFS is composed of multiple distributed file systems with the integration of advantages of these distributed file systems. In HybridFS, on top of multiple distributed file systems, we have designed a metadata management server to perform three functions: file placement, partial metadata store, and dynamic file migration. The file placement is performed based on a decision tree. The partial metadata store is performed for files whose size is less than a few hundred Bytes to increase throughput. The dynamic file migration is performed to balance the storage usage of distributed file systems without throttling performance. We have implemented HybridFS in java on eight nodes and choose Ceph, HDFS, and GlusterFS as designated distributed file systems. The experimental results show that, in the best case, HybridFS can have up to 30% performance improvement of read/write operations over a single distributed file system. In addition, if the difference of storage usage among multiple distributed file systems is less than 40%, the performance of HybridFS is guaranteed, that is, no performance degradation.

Hadoop Distributed File System (HDFS) is a famous distributed file system to access and store large files. It suffers from different memory, page fault, and processing bottlenecks while accessing storing a large number of very small sized files. In this research, we have studied architecture, operation, and behavior of HDFS while serving the access and storage requests for a very large number of very small-sized files. Hadoop provides a solution to this problem i.e. archiving these many small files with a single HAR file and treating this HAR file as a large file. In this study, we have analyzed the working and significance of HAR as well as its cost benefit analysis is done in a comprehensive manner.

Since, the evolution of distributed operating system, distributed file system is come out to be important part in operating system. P2P is a reliable way in Distributed Operating System for file sharing. It was introduced in 1999, later it became a high research interest topic. Peer to Peer network is a type of network, where peers share network workload and other load related tasks. A P2P network can be a period of time connection, where a bunch of computers connected by a USB (Universal Serial Bus) port to transfer or enable disk sharing i.e. file sharing. Currently P2P requires special network that should be designed in P2P way. Nowadays, there is a big influence of browsers in our life. In this project we are going to study of file sharing mechanism in distributed operating system in web browsers, where we will try to find performance bottlenecks which our research will going to be an improvement in file sharing by performance and scalability in distributed file systems. Additionally, we will discuss the scope of Web Torrent file sharing and free-riding in peer to peer networks.

Accidental misuse of shared files by authorised users is a predominant problem. This paper proposes a well-known static analysis approach, namely a type system, to prevent such accidental misuse. We develop a type system that intercepts commands issued by users in a file system and enforces policies on each file. Commands issued by users to manipulate files will be subject to type checking by the type system. Type-checked commands are then guaranteed to not violate policies of the files. The focus of this paper is on a particular policy that allows owners of files (users who created files) to specify the number of times a file can be read by limiting the number of times a file can be copied. Therefore, a file can be read as much as it can be copied. If the file cannot be copied, then it can be read only once. This approach can be extended to other properties.

HopsFS is an open-source, next generation distribution of the Apache Hadoop Distributed File System (HDFS) that replaces the main scalability bottleneck in HDFS, single node in-memory metadata service, with a no-shared state distributed system built on a NewSQL database. By removing the metadata bottleneck in Apache HDFS, HopsFS enables significantly larger cluster sizes, more than an order of magnitude higher throughput, and significantly lower client latencies for large clusters.In this paper, we detail the techniques and optimizations that enable HopsFS to surpass 1 million file system operations per second - at least 16 times higher throughput than HDFS. In particular, we discuss how we exploit recent high performance features from NewSQL databases, such as application defined partitioning, partition-pruned index scans, and distribution aware transactions. Together with more traditional techniques, such as batching and write-ahead caches, we show how many incremental optimizations have enabled a revolution in distributed hierarchical file system performance.

As an open source implementation of GFS, Hadoop Distributed File System (HDFS) has high efficiency on handling the large files. However, due to its own master-slave structure and the storage of metadata, the efficiency is low when dealing with massive small files. It occupies large amount of NameNode memory, reduces access efficiency, and delays concurrent user access. In order to improve this performance efficiency, this paper studies the method of processing small files on HDFS. According to the file storage process, this paper proposes a small file processing scheme based on index mechanism. Before the file is uploaded to the HDFS cluster, the file size is measured. The small files are indexed and merged. If it is a small file, then it will be indexed and processed. And it will be created an index file to save the index information of the small file. At the same time, this scheme introduces the distributed caching strategy to further optimize the I/O operation of small files, so as to improve the reading speed. Experimental results show that compared with the original HDFS and HAR scheme, this scheme has a great improvement in obtain memory efficiency and consumption of memory resources.

The ever-growing demand for high performance computation calls for progressively larger parallel distributed file systems to match their requirement. These file systems can achieve high performance for large I/O operations through distributing load across numerous data servers. However, they fail to provide quality service for applications pertaining to small files. In this paper, we propose a delegable metadata service (DMS) for hiding latency of metadata accesses and optimizing small-file performance. In addition, four techniques have been designed to maintain consistency and efficiency in DMS: pre-allocate serial metahandles, directory-based metadata replacement, packing transaction operations and fine-grained lock revocation. These schemes have been employed in Cappella parallel distributed file system, and various experiments complying with industrial standards have been conducted for evaluation of its efficiency. The results show that our design has achieved significant improvement in performance of both metadata operations and small-file access. Moreover, this scheme is widely applicable for integration within many other distributed file systems.

This paper presents a Fast display Algorithm of large Map File based on memory mapping file technique, designing for reducing the number of pointer moves and the data in memory scheduling ,The algorithm uses a memory-mapped file access operation of a large-scale map file to resolve the problem of traditional file processing methods, which can not read more than 2G map image file; combined image pyramid technology to apply different resolutions depending on the display requirements map image, in order to achieve the purpose of roaming quickly. The experiment and analysis results show that the algorithm is able to achieve the best resolution real-time roaming of large-scale digital map.

At present, MD5 algorithm is commonly used in the file transfer system to conduct the tests of file integrity, but when a file is too large, it will take a long time to generate MD5 code for the whole file, with the result of reducing the transmission efficiency of the file. Based on this conduction, a fast algorithm of file integrity detection is provided, which can generate MD5 code with the file size and some random content of the file. Although the algorithm cannot test files roundly, it is integrated into the file transfer system. The test results show that this algorithm can detect the file integrity faster, and when the file is damaged or tampered with, the probability of detecting changes of the file content is higher than the theoretical value. The algorithm has great value in engineering applications.

[Abstract] Hadoop is composed with MapReduce programming model for distributed processing and HDFS distributed file system. Hadoop is suitable framework for big data processing, but processing of mass small files have many problems. The processing of mass small file in hadoop have problems to created one mapper per one file, and it have problems to needed many memory for store of meta information of file. This paper have comparison evaluation processing method of mass small file with various method in hadoop platform. The processing of general compression format is inadequate because of processing by one mapper regardless of data size. The processing of sequence and hadoop archive file is removed memory problem of namenode by compress and combine of small file. Hadoop archive file is faster then sequence file about combine time of small file. The processing using CombineFileInputFormat class is needed not combine of small file, and it have similar speed big data processing method.

Traditional file systems have some shortages in storing small files, such as randomness of data layout, waste of disk space and lack of inode resources. In this thesis, a log-structured file system named LevelFS based on LevelDB is presented. By setting the write buffer, it can make disk randomized writes of small files into disk sequential writes, and reduce the distance of related data, so as to improve the read and write performance of file system. Experiments show that LevelFS can greatly improve read and write performance of small files without affect the large ones.