Search

Due to their massive popularity, image files, especially JPEG, offer high potential as carriers of other information. Much of the work to date on this has focused on stenographic ways of hiding information using least significant bit techniques but we believe that the findings in this project have exposed other ways of doing this. We demonstrate that a digital certificate relating to an image file can be inserted inside that image file along with accompanying metadata containing references to the issuing company. Notwithstanding variations between devices and across operating systems and applications, a JPEG file holds its structure very well. Where changes do take place, this is generally in the metadata area and does not affect the encoded image data which is the heart of the file and the part that needs to be verifiable. References to the issuing company can be inserted into the metadata for the file. There is an advantage of having the digital certificate as an integral part of the file to which it applies and consequently travelling with the file. We ultimately prove that the metadata within a file offers the potential to include data that can be used to prove integrity, authenticity and provenance of the digital content within the file. Keywords: Secure authentication, Integrity, Digital certificates, Security, Network security

Recently, power-aware distributed file systems for efficient big data processing have increasingly moved toward power proportional designs. However, inefficient gear-shifting in such systems is an important issue that can seriously degrade their performance. To address this issue, we propose and evaluate an efficient gear-shifting power proportional distributed file system. The proposed system utilizes flexible data placement that reduces the amount of reflected data and has an architecture that improves the metadata management to achieve high-efficiency gear-shifting. Extensive empirical experiments using actual machines based on the HDFS demonstrated that the proposed system gains up to \(22\,\%\) better throughput-per-watt performance. Moreover, a suitable metadata management setting corresponding to the amount of data updated while in low gear is found from the experimental results.

In the big data era, the distributed file system is getting more and more significant due to the characteristics of\ud its scale-out capability, high availability, and high performance. Different distributed file systems may have different design goals. For example, some of them are designed to have good performance for small file operations, such as GlusterFS, while some of them are designed for large file operations, such as Hadoop distributed file system. With the divergence of big data applications, a distributed file system may provide good performance for some applications but fails for some other applications, that is, there has no universal distributed file system that can produce good performance for all applications. In this\ud paper, we propose a hybrid file system framework, HybridFS, which can deliver satisfactory performance for all applications. HybridFS is composed of multiple distributed file systems with the integration of advantages of these distributed file systems. In HybridFS, on top of multiple distributed file systems, we have designed a metadata management server to perform three functions: file placement, partial metadata store, and dynamic file migration. The file placement is performed based on a decision tree. The partial metadata store is performed for files whose size is less than a few hundred Bytes to increase throughput. The dynamic file migration is performed to balance the storage usage of distributed file systems without throttling performance. We have implemented HybridFS in java on eight nodes and choose Ceph, HDFS, and GlusterFS as designated distributed file systems. The experimental results show that, in the best case, HybridFS can have up to 30% performance improvement of read/write operations over a single distributed file system. In addition, if the difference of storage usage among multiple distributed file systems is less than 40%, the performance of HybridFS is guaranteed, that is, no performance degradation.

Accidental misuse of shared files by authorised users is a predominant problem. This paper proposes a well-known static analysis approach, namely a type system, to prevent such accidental misuse. We develop a type system that intercepts commands issued by users in a file system and enforces policies on each file. Commands issued by users to manipulate files will be subject to type checking by the type system. Type-checked commands are then guaranteed to not violate policies of the files. The focus of this paper is on a particular policy that allows owners of files (users who created files) to specify the number of times a file can be read by limiting the number of times a file can be copied. Therefore, a file can be read as much as it can be copied. If the file cannot be copied, then it can be read only once. This approach can be extended to other properties.

SUMMARY The numbers of files in file systems have increased dramatically in recent years. Office workers spend much time and effort searching for the documents required for their jobs. To reduce these costs, we propose a new method for recommending files and operations on them. Existing technologies for recommendation, such as collaborative filtering, suffer from two problems. First, they can only work with documents that have been accessed in the past, so that they cannot recommend when only newly generated documents are inputted. Second, they cannot easily handle sequences involving similar or differently ordered elements because of the strict matching used in the access sequences. To solve these problems, such minor variations should be ignored. In our proposed method, we introduce the concepts of abstract files as groups of similar files used for a similar purpose, abstract tasks as groups of similar tasks, and frequent abstract workflows grouped from similar workflows, which are sequences of

Organizing documents in the file system is one of the most tedious and thorny tasks for most computer users. Taxonomies based on hand made directory hierarchies still remain the only possible alternative for most small and medium enterprises, public administrations and individual users. However, both the limitations of the hierarchical organization of file systems and the difficulty of maintaining the coherence within the taxonomy have raised the need for more scalable and effective approaches. Desktop searching applications provide proprietary interfaces that enable content-based searching at the cost of having no control on the indexing and ranking of results. Semantic file systems, instead, leave users the freedom to manage the taxonomy according to their specific needs, but lose the standard file system features. In this paper we describe GFS (graph-based file system) a new hybrid file system that extends the standard hierarchical organization of files with semantic features. GFS allows the user to nest semantic spaces inside the directory hierarchy leaving unaltered system folders. Semantic spaces allow customized file tagging and leverage on browsing to guide file searching. Since GFS does not change the low-level interface to interact with file systems, users can continue to use their favorite file managers to interact with it. Moreover, no changes are required to integrate the semantic features in proprietary software.

Hadoop has been widely used in various clusters to build scalable and high performance distributed file systems. However, Hadoop distributed file system (HDFS) is designed for large file management. In case of small files applications, those metadata requests will flood the network and consume most of the memory in Namenode thus sharply hinders its performance. Therefore, many web applications do not benefit from clusters with centered metanode, like Hadoop. In this paper, we compare our Fat-Btree based data access method, which excludes center node in clusters, with Hadoop. We show their different performance in different file I/O applications.

Files in computers are increasing in number, so we require file management tools to find target files and to classify large groups of files. Our research group has been developing a system that provides virtual directories made up of related files. Many methods to extract inter-file relationships are available, such as word frequency, access co-occurrence, and so on. In practice, users need to select and combine multiple types of inter-file relationships to form groups of files they want. For this purpose, we propose FileSearchCube, a tool for file group analysis that introduces the data-cube concept. FileSearchCube provides cube operations for a set of files, and helps users to find relevant file groups.

This paper introduces GlobalFS, a POSIX-compliant geographically distributed file system. GlobalFS builds on two fundamental building blocks, an atomic multicast group communication abstraction and multiple instances of a single-site data store. We define four execution modes and show how all file system operations can be implemented with these modes while ensuring strong consistency and tolerating failures. We describe the GlobalFS prototype in detail and report on an extensive performance assessment. We have deployed GlobalFS across all EC2 regions and show that the system scales geographically, providing performance comparable to other state-of-the-art distributed file systems for local commands and allowing for strongly consistent operations over the whole system. The code of GlobalFS is available as open source.

Knowledge of the file size distribution is needed to optimize file system design. In particular, if all the files are small, the disk block size should be small, too, to avoid wasting too large a fraction of the disk. On the other hand, if files are generally large, choosing a large block size is good since it leads to more efficient transfers. Only by knowing the file size distribution can reasonable choices be made. In 1984, we published the file size distribution for a university computer science department. We have now made the same measurements 20 years later to see how file sizes have changed. In short, the median file size has more than doubled (from 1080 bytes to 2475 bytes), but large files still dominate the storage requirements.

File systems are designed to control how files are stored and retrieved. Without knowing the context and semantics of file contents, file systems often contain duplicate copies and result in redundant consumptions of storage space and network bandwidth. It has been a complex and challenging issue for enterprises to seek deduplication technologies to reduce cost and increase the storage efficiency. To solve such problem, researchers proposed in-line or offline solutions for primary storages or backup systems at the subfile or whole-file level. Some of the technologies are used for file servers and database systems. Fewer studies focus on the cloud file system deduplication technologies at the application level, especially for the Hadoop distributed file system. It is the goal of this paper to design a file deduplication framework on Hadoop distributed file system for cloud application developers. The architecture, interface, and implementation experiences are also shared in this paper.

Several file retention policy methods propose that a file retention policy should be based on file value. Though such a retention policy might increase the value of accessible files, the method to arrive at such a policy is underresearched. This article discusses how one can arrive at a method for developing file retention policies based on the use values of files. The method’s applicability is initially assessed through a case study at Capgemini, Netherlands. In the case study, we hypothesize that one can develop a file retention policy by testing causal relations between file attributes (as used by file retention methods) and the use value of files. Unfortunately, most file attributes used by file retention methods have a weak correlation with file value, resulting in the conclusion that these methods do not well select out high- and low-value files. This would imply the ineffectiveness of the used attributes in our study or errors in our conceptualization of file value. We continue with the last possibility and develop indicators for file utility (with low utility being waste). With this approach we were able to detect waste files, in a sample of files, with an accuracy of 80%. We therefore not only suggest further research in information waste detection as part of a file retention policy, but also to further explore other file attributes that could better predict file value and file utility.

MemFS is a fully-symmetrical, in-memory distributed runtime file system. Its design is based on uniformly distributing file stripes across the storage nodes belonging to an application by means of a distributed hash function, purposefully sacrificing data locality for balancing both network traffic and memory consumption. This way, reading and writing files can benefit from full network bisection bandwidth, while data distribution is balanced across the storage servers.

Part 3: FORENSIC TECHNIQUES; International audience; A major challenge in digital forensics is the handling of very large amounts of data. Since forensic investigators often have to analyze several terabytes of data in a single case, efficient and effective tools for automatic data identification and filtering are required. A common data identification technique is to match the cryptographic hashes of files with hashes stored in blacklists and whitelists in order to identify contraband and harmless content, respectively. However, blacklists and whitelists are never complete and they miss most of the files encountered in investigations. Also, cryptographic hash matching fails when file content is altered even very slightly. This paper analyzes several distributed systems for their ability to support file content identification. A framework is presented for automated file content identification that searches for file hashes and collects, aggregates and presents the search results. Experiments demonstrate that the framework can provide identifying information for 26% of the test files from their hashed content, helping reduce the workload of forensic investigators.

This paper provides evidence of a distinguished feature of file systems, that we call File System Genome. Such a feature is originated by the locations where the file blocks are placed on the mass-storage device by the operating system during the installation procedure. It appears from our study that the File System Genome is a distinctive feature of each operating system installation. In particular, our extensive set of experiments shows that the installation of the same operating system on two identical hardware configurations generates two different File System Genomes. Further, the application of sound information theory tools, such as min entropy, show that the differences between two File System Genome are considerably relevant. The results provided in this paper constitute the scientific basis for a number of applications in various fields of information technology, such as devices' identification and security.

Performance comparisons between File Signatures and Inverted Files for text retrieval have previously shown several significant shortcomings of file signatures relative to inverted files. The inverted file approach underpins most state-of-the-art search engine algorithms, such as Language and Probabilistic models. It has been widely accepted that traditional file signatures are inferior alternatives to inverted files. This paper describes TopSig, a new approach to the construction of file signatures. Many advances in semantic hashing and dimensionality reduction have been made in recent times, but these were not so far linked to general purpose, signature file based, search engines. This paper introduces a different signature file approach that builds upon and extends these recent advances. We are able to demonstrate significant improvements in the performance of signature file based indexing and retrieval, performance that is comparable to that of state of the art inverted file based systems, including Language models and BM25. These findings suggest that file signatures offer a viable alternative to inverted files in suitable settings and from the theoretical perspective it positions the file signatures model in the class of Vector Space retrieval models., 12 pages, 8 figures, CIKM 2011

File carving is the process of recovering files without the help of (file system) storage metadata. A host of techniques exist to perform file carving, often used in several tools in varying combinations and implementations. This makes it difficult to determine what tool to use in specific investigations or when recovering files in a specific file format. We define recoverability as the set of software requirements for a file carver to recover files in a specified file format. This set can then be used to evaluate what tool to use or which technique to implement, based on external factors such as file format to recover, available time, engineering capacity and data set characteristics. File carving techniques are divided into two groups, format validation and file reconstruction. These groups refer to different parts of a file carver's implementation. Additionally, some techniques may be emphasized or omitted not only because of file format support for them, but based on performance effects that may result from applying them. We discuss a simplified variant of the GIF image file format as an example and show how a structured analysis of the format leads to design decisions for a file carver.

Retrieval of files without the support of file system structures is arguably essential for digital forensics. Files are typically stored as sequences of data blocks, which have to be reconstructed in the retrieval process. This is commonly performed, among other approaches, through file carving, in general detecting the original block sequences by means of signatures of known headers and footers of files. Of course, this creates challenges with fragmented files, where blocks belonging to different files may be interleaved. Ways to classify file blocks into file types relying on their content may provide a support to achieve a successful reconstruction. We propose to classify file blocks using Support Vector Machines (SVMs), and we do so by studying in-depth the impact of an appropriate selection of the features used in the classification process. We analyze several potential features and test their performance over a large and representative collection of file blocks and file types. We find out that SVM classifiers can achieve a good accuracy and that a specific type of features (based on byte frequency distribution) performs well across almost all of the examined file types.

International audience; Semi-join is the most used technique to optimize the treatment of complex relational queries on distributed architectures. However, the overhead related to semi-join computation can be very high due to data skew and to the high cost of communication in distributed architectures. Internet search engines needs to process vast amounts of raw data every day. Hence, systems that manage such data should assure scalability, reliability and availability issues with reasonable query processing time. Hadoop and Google's File System are examples of such systems. In this paper, we present a new algorithm based on Map-Reduce-Merge model and distributed histograms for processing semi-join operations on such systems. A cost analysis of this algorithm shows that our approach is insensitive to data skew while reducing communication and disk Input/Output costs to a minimum.

Περιέχει το πλήρες κείμενο So far little attention has been paid to file format robustness, i.e., a file formats capability for keeping its information as safe as possible in spite of data corruption. The paper on hand reports on the first comprehensive research on this topic. The research work is based on a study on the status quo of file format robustness for various file formats from the image domain. A controlled test corpus was built which comprises files with different format characteristics. The files are the basis for data corruption experiments which are reported on and discussed.

We consider a BitTorrent-like file sharing system, where the peers interested in downloading a large file join an overlay network. The seed node possessing the file stays in the system, whereas all other peers are non-altruistic in the sense that they leave the system as soon as they have downloaded the whole file. We consider a flash crowd scenario, where the peers join the overlay simultaneously. We show that the chunk selection algorithm is critical, propose an analytic approach to the process, and find that the encounters can be restricted to neighbours in a Chord overlay without losing much in performance.

File transfer using peer-to-peer file sharing applications is usually divided into two steps: resource search and resource download. Depending on the file size and its popularity, either of the two phases can become the bottleneck. In this paper we describe both the location and download phases of a generic peer-to-peer file sharing application using a fluid model. The proposed model allows the computation of the transfer time distribution, and it is capable of considering some advanced characteristic such as parallel downloads and on-off peer behavior. Model parameters reflect network, application, resource and user characteristics, and can be tuned to analyze a large number of different real peer-to-peer implementations file transfer using peer-to-peer file sharing applications is usually divided into two steps: resource search and resource download. Depending on the file size and its popularity, either of the two phases can become the bottleneck. In this paper we describe both the location and download phases of a generic peer-to-peer file sharing application using a fluid model. The proposed model allows the computation of the transfer time distribution, and it is capable of considering some advanced characteristic such as parallel downloads and on-off peer behavior. Model parameters reflect network, application, resource and user characteristics, and can be tuned to analyze a large number of different real peer-to-peer implementations.

The need for flexible file sharing in distributed systems is increasing in applications such as calendar management, collaborative editing of documents, collaborative software developments etc. The file sharing policies required in these applications are often very different from the traditional read/write policies. Hence, a flexible way of specifying and implementing sharing policies on individual files in file systems is required. We propose a distributed object-based system model of constructing file systems. The object-based system model is based on a pattern called FlexiFrag. We show how a distributed object-based system and in particular distributed file system can be constructed using the pattern in a flexible way.

Library records and their utilization are described and the various types of file organization available are examined. The serial file with a series of inverted indexes is preferred to the simple serial file or a threaded list file. It is shown how various records should be stored, according to their utilization, in the available storage devices in order to achieve optimum cost-performance.

Big Data is a frequent generation and updating of large volume of data around the clock across the globe by the users. Handling large volume of data in a real time environment is a challenging task. Distributed File System is one of the strategy to handle large volume of data in the real time. Distributed file system is a collection of independent computers that appear to the users of the system as a single coherent system. In Distributed file system common files can be shared between the nodes, the drawbacks are scalability, replication, availability and very expensive to buy a hardware server. To overcome this issue Hadoop Distributed File System came into existence. Hadoop distributed file system to run on cluster of commodity hardware like personal computer and laptop. HDFS provides the scalable, fault-tolerance, cost-efficient storage for Bigdata. Hadoop Distributed File System support data duplication to achieve high data reliability. However additional utilization of storage space is required due to duplication strategy. HDFS Storage space can be managed efficiently by implementing De-duplication techniques. The objective of the research is to eliminate file duplication by implementing De-duplication strategy. A novel and efficient De-duplication system using HDFS approach is introduced in this research work. To implement De-duplication strategy, hash values are computed for files using MD5 and SHA1 algorithms . The generated hash value for a file is checked with the existing file to identify the presence of duplication. If duplication exists, the system will not allow the user to upload the duplicate copy to the HDFS. Hence memory utilization is handled efficiently in HDFS.