Search

Current distributed file systems are designed to support PB-scale even EB-scale data storage. Metadata service, which manages file attribute information and the global namespace tree, is crucial to system performance. Distributed metadata management, using multiple metadata servers (MDS's) to store metadata, provides effective approaches to alleviate the workload of a single server. However, maintaining good metadata locality and keeping load balancing among MDS's is a nontrivial problem. In this paper, we present the first machine learning based model called DeepHash, which leverages the neural network to learn a locality preserving hashing (LPH) mapping scheme. DeepHash first converts the metadata nodes to feature vectors by the network embedding technology. Due to the absence of training labels, i.e., the hash values of metadata nodes, we design a pair loss function with distinctive characters to train DeepHash, and introduce the sampling strategy to improve the training efficiency. Besides, we propose an efficient algorithm to dynamically balance the workload and adopt the cache model to improve query efficiency. The experiments on the Amazon EC2 platform demonstrate that the DeepHash can preserve the metadata locality meanwhile maintaining a high load balancing, which denotes the effectiveness and efficiency of DeepHash compared with traditional and state-of-the-art schemes.

Many massive data processing applications nowadays often need long, continuous, and uninterrupted data accesses. Distributed file systems are used as the back-end storage to provide the global namespace management and reliability guarantee. Due to increasing hardware failures and software issues with the growing system scale, metadata service reliability has become a critical issue as it has a direct impact on file and directory operations. Existing metadata management mechanisms can provide fault tolerance capability to some level but are inadequate. They often have limitations in system availability, state consistence, and performance overhead and lack an effective mechanism to offer metadata reliability. This paper introduces a novel highly reliable metadata service to address these issues in large-scale file systems. Different from traditional strategies, this proposed reliable metadata service adopts a new active-standby architecture for fault tolerance and uses a holistic approach to improve file system availability. A new shared storage pool (SSP) is designed for transparent metadata synchronization and replication between active and standby servers. Based on the SSP, a new policy called multiple actives multiple standbys (MAMS) is presented to perform metadata service recovery in case of failures. A new global state recovery strategy and a smart client fault tolerance mechanism are achieved to maintain the continuity of metadata service. We have implemented such highly reliable metadata service in a prototype file system CFS (Clover file system) and conducted extensive tests to evaluate it. Experimental results confirm that it can significantly improve file system reliability with fast failover under different failure scenarios while having negligible influence on performance. Compared with typical reliability designs in Hadoop Avatar, Hadoop HA, and Boom-FS file systems, the mean-time-to-recovery (MTTR) with the highly reliable metadata service was reduced by 80.23, 65.46 and 28.13 percent, respectively.

With evolving techniques to parallelize Deep Learning (DL) and the growing amount of training data and model complexity, High-Performance Computing (HPC) has become increasingly important for machine learning engineers. Although many compute clusters already use learning accelerators or GPUs, HPC storage systems are not suitable for the I/O requirements of DL workflows. Therefore, users typically copy the whole training data to the worker nodes or distribute partitions. Because DL depends on randomized input data, prior work stated that partitioning impacts DL accuracy. Their solutions focused mainly on training I/O performance on a high-speed network but did not cover the data stage-in process, for example. We show in this paper that, in practice, (unbiased) partitioning is not harmful for distributed DL accuracy. Nevertheless, manual partitioning can be error prone and inefficient. Typically, data must be unpacked and shuffled before it is distributed to nodes. We propose a solution that features both: efficient stage-in and fast access to a global namespace to prevent biases. Our architecture is based around an ad hoc storage system relying on a high-speed interconnect allowing an efficient stage-in of DL data sets into a single global namespace. Our proposed solution does not limit access to parts of the data set or relies on data duplication, also relieving the HPC storage system. We obtain high I/O performance during training and ensure minimal interference with communication of the learning workers. The optimizations are transparent to DL applications and their accuracy is not affected by our architecture.

With the growing volumes of data, many organizations are deploying geo-distributed edge servers and building federations atop of edge servers to improve data sharing, effective collaborations, and analytics. Multiple federation file systems are designed to satisfy such needs, but due to application-specific architectures, these federations neglect some important features that can improve the overall federation performance. In this paper, we address the important challenges of federated file systems, in particular, global namespace, optimal data placement and analysis, efficient data migration across edge servers, and metadata optimizations. To further investigate these challenges, we prototyped the federation file system iStore to emulate the federation and showed the significance of the afore-mentioned key challenges in the federation. The iStore provides unified global namespace atop of geo-distributed edge servers with a generic job and resource-aware data storage and placement algorithm (JRAP), which minimizes the job execution time by considering resources at each edge server. Furthermore, to enable effective data migration, we employed direct channel file layout-aware data transfer and designed a batch-based metadata scheme for federations to reduce the metadata contention with increasing clients. We evaluated the efficacy of various big data applications from data generation to storage and analysis using the iStore on real testbed and simulation.

The Internet of Things leverages Internet technology in cyber-physical systems (CPSs), but the protocols and principles of the Internet were designed for interacting with information systems, not cyber-physical systems. For one, timeliness is not a factor in any widespread Internet technology, with quality-of-service features having been routinely omitted for decades. In addition, for things, safety, freedom from physical harm, is even more important than information security, the focus on the Internet. Nevertheless, properties of the Internet are valuable in CPSs, including a global namespace, reliable (eventual) delivery of messages, end-to-end security through asymmetric encryption, certificate-based authentication, and the ability to aggregate data from a multiplicity of sources in the cloud. This paper discusses and surveys architectural approaches, communication protocols, and programming models that promise to bridge the gap, enabling the use of the Internet technologies even in safety-critical, cyber-physical applications such as factory automation and transportation. Specifically, we argue that smart gateways hosted on edge computers complement cloud-based services; they can provide tighter control over timing and security that is robust against network outages, play an active role in managing interactions between things, and isolate safety-critical services from best-effort services. We explain how time sensitive network technology can be leveraged to reliably orchestrate a multiplicity of things, and how augmenting our programming models with a well-defined notion of time can make systems more deterministic and more testable.

JavaScript is widely used for implementing client-side web applications, and it is common to include JavaScript code from many different hosts. However, in a web browser, all the scripts loaded in the same frame share a single global namespace. As a result, a script may read or even overwrite the global objects or functions in other scripts, causing unexpected behaviors. For example, a script can redefine a function in a different script as an object, so that any call of that function would cause an exception at run time. We systematically investigate the client-side JavaScript code integrity problem caused by JavaScript global identifier conflicts in this paper. We developed a browser-based analysis framework, JSObserver, to collect and analyze the write operations to global memory locations by JavaScript code. We identified three categories of conflicts using JSObserver on the Alexa top 100K websites, and detected 145,918 conflicts on 31,615 websites. We reveal that JavaScript global identifier conflicts are prevalent and could cause behavior deviation at run time. In particular, we discovered that 1,611 redefined functions were called after being overwritten, and many scripts modified the value of cookies or redefined cookie-related functions. Our research demonstrated that JavaScript global identifier conflict is an emerging threat to both the web users and the integrity of web applications.

Chapel is a parallel language designed to support productive programming from laptops to supercomputers. Unlike typical general-purpose approaches to large-scale computing, Chapel supports a global namespace and a global view of control. These make programming at scale more productive and natural by avoiding the need to utilize a Single Program, Multiple Data (SPMD) approach. Chapel is being developed as portable, open-source software and has demonstrated its ability to match or beat C/C++/MPI/OpenMP performance using code that's as easy to read as Python or Swift. In this talk, I will introduce some of Chapel's key features for those who are new to the language. However, most of the talk will focus on unique aspects of compiling Chapel. I will illustrate the transformations and analyses we use to implement Chapel's global view of data and control. I will also describe some optimizations that make use of Chapel's feature set to improve performance without sacrificing ease-of-use. In doing so, I will argue that when designed well, a productive language can enhance a compiler's ability to achieve performance and scalability rather than thwarting it. I'll wrap up by touching on some upcoming challenges for the Chapel compiler, including code generation for GPUs, reducing compilation time, and supporting interactive programming.

Traditional distributed file systems (DFS) use centralized service to manage metadata. Many studies based on this centralized architecture enhanced metadata processing capability by scaling the metadata server cluster, which is however still difficult to keep up with the growing number of clients and the increasingly metadata-intensive applications. Some solutions abandoned the centralized metadata service and improved scalability by embedding a private metadata service in an HPC application, but these solutions are suitable for only some specific applications and the absence of global namespace makes data sharing and management difficult. This paper addresses the shortcomings of existing studies by optimizing the consistency model of client- side metadata cache for the HPC scenario using a novel partial consistency model. It provides the application with strong consistency guarantee for only its workspace, thus improving metadata scalability without adding hardware or sacrificing the versatility and manageability of DFSes. In addition, the paper proposes batch permission management to reduce path traversal overhead, thereby improving metadata processing efficiency. The result is a library (Pacon) that allows existing DFSes to achieve partial consistency for scalable and efficient metadata management. The paper also presents a comprehensive evaluation using intensive benchmarks and representative application. For example, in file creation, Pacon improves the performance of BeeGFS by more than 76.4 times, and outperforms the state-of-the-art metadata management solution (IndexFS) by more than 4.6 times.

Including JavaScript code from many different hosts is a popular practice in developing web applications. For example, to include a social plugin like the Facebook Like button, a web developer needs to only include a script from facebook.net in her/his web page. However, in a web browser, all the identifiers (i.e., variable names and function names) in scripts loaded in the same frame share a single global namespace. Therefore, a script can overwrite any of the global variables and/or global functions defined in another script, causing unexpected behavior. In this work, we develop a browser-based dynamic analysis framework, that monitors and records any writes to JavaScript global variables and global functions. Our tool is able to cover all the code executed in the run time. We detected 778 conflicts across the Alexa top 1K websites. Our results show that global name conflicts can indeed expose web applications to security risks.

The large scale and distribution of cloud computing storage have become the major challenges in cloud forensics for file extraction. Current disk forensic methods do not adapt to cloud computing well and the forensic research on distributed file system is inadequate. To address the forensic problems, this paper uses the Hadoop distributed file system (HDFS) as a case study and proposes a forensic method for efficient file extraction based on three-level (3L) mapping. First, HDFS is analyzed from overall architecture to local file system. Second, the 3L mapping of an HDFS file from HDFS namespace to data blocks on local file system is established and a recovery method for deleted files based on 3L mapping is presented. Third, a multi-node Hadoop framework via Xen virtualization platform is set up to test the performance of the method. The results indicate that the proposed method could succeed in efficient location of large files stored across data nodes, make selective image of disk data and get high recovery rate of deleted files.

According to the results of the implementation of the priority «industry of nanosystems» an analysis of the use mechanism of protection of intellectual property rights in the global namespace. Argues that only the intellectual property identificeret prospects for capturing a share of the global market of science-intensive products.Therefore, to capture and consolidate global technological leadership in the postindustrial society, primarily in the form of its institutionalization in the system of world standards intellectual property. Noted that the reports of UNESCO and the world intellectual property organization (WIPO), published in 2015, the Program of nanoindustry development in the Russian Federation recognized as not effective in the criterion of low patent activity. The data of the patent analysis in three technological fields of nanotechnology, highlighted WIPO’s as mainstream, estimated the share of Russian patents in the world portfolio in each of these areas.The conclusion about the fundamental relevance of this indicator the priority areas as «the number patents with a priority of the Russian Federation, obtained in foreign countries» to assess the country’s share in the global market high-tech products.

In this paper, we present an innovative namespace architecture called the Universal Namespace (UNS). UNS enhances both the local and global namespace with portable, mobile, and exchangeable properties and supports fetch-from-anywhere storage systems without requiring remote mounting file systems. UNS exploits the extended file attributes of file systems and enhances file system metadata with unified remote data access information and methods. We design the proposed UNS to support big data computing on multiple geolocation-based shared storage systems. We describe the innovative concepts of the proposed UNS’s architecture and demonstrate UNS’s performance results from various test cases. Finally, we summarize the benefits and advantages of using UNS in supporting big data convergence of HPC and Cloud scientific workflows. Furthermore, we present the future design and developmental direction of UNS.

In content based file sharing peer-to-peer (P2P) [1] network model nodes share files directly with each other without a centralized server. In such a file sharing system, nodes meet and exchange requests and files in the format of text, short videos, and voice clips in different interest categories. Content is various and large file sharing such as the multimedia content is required with the rapid development of the wireless communication technology. File sharing can also mean having an allocated amount of personal file storage in a common file system. A P2P content based file sharing system, for efficient file searching, threshold takes advantage of node mobility by designating stable nodes, which have the most frequent contact with community members, as community coordinators for intra community searching, and highly mobile nodes that visit other communities frequently as community ambassadors for intercommunity searching. The large file sharing needs more stable end to end path and long transmission time. Last but not least, more relationship between nodes will be used to promote the file sharing process. Content based file sharing is helpful for taking certain decisions during file transmission. These decisions will benefit in proper utilization of network resources. In this paper content-based file sharing scheme using threshold is proposed. The user's interest is determined by the proposed scheme before searching and sharing the files in the peer-to-peer network. The resources in the network are utilized as per the contents of the files to be shared. The performance evaluation show that proposed system significantly lowers transmission cost and improves file sharing success rate compared to current methods.

A high-performance, scalable, and resilient storage subsystem is essential for delivering and maintaining consistent performance and high utilization expected from a modern supercomputer. IBM delivered two systems under the CORAL program, both of which used IBM Spectrum Scale and IBM Elastic Storage Server (ESS) as the storage solution. The larger of the two CORAL clusters is composed of 77 building blocks of ESS, each of which consists of a pair of high-performance I/O Server nodes connected to four high-density storage enclosures. These ESS building blocks are interconnected via a redundant InfiniBand EDR network to form a storage cluster that provides a global namespace aggregating performance over 32,000 commodity disks. The IBM Spectrum Scale for ESS runs high-performance erasure coding on each building block and provides a single global name space across all the building blocks. The IBM Spectrum Scale features deliver a highly resilient, high-performance storage subsystem using ESS. These features include recent improvements for efficient buffer management and fast efficient low-latency communication. CORAL I/O performance results include large-block streaming throughput of over 2.4 TB/s, ability to create over 1 M 32-KB files per second, and enabling an aggregate rate of 30 K zero-length file creates per second in a shared directory from multiple nodes. This article describes the design and implementation of the ESS storage cluster; the innovations required to meet the performance, scale, manageability, and reliability goals; and challenges we had to overcome as we deployed a system of such unprecedented I/O capabilities.

Modern global file systems such as parallel file systems or distributed file systems normally use a global namespace (GNS) to support viewing and accessing files that are independent of their physical storage locations. Using GNS support, we can seamlessly modify and reconfigure physical data storage without affecting how users view and access it. However, with the rapid growth of data size, the current design and implementations of the GNS system is experiencing problems of manageability and scalability. In this paper, we present a new namespace architecture called the Universal Namespace (UNS). UNS enhances both the local and global namespace with portable, parallel data mobile, and exchangeable properties and supports fetch-from-anywhere storage systems without requiring remote mounting. UNS exploits the extended file attributes of file systems and enhances file system metadata with unified remote data access information and methods. We design the proposed UNS to support big data computing on multiple geolocation-based shared storage systems. We describe the innovative concepts of the proposed UNS's architecture and demonstrate UNS's early performance results from various test cases. Finally, we summarize the benefits and advantages of using UNS in supporting remote data sharing for extreme scale computing and big data computing. Furthermore, we present the future design and developmental direction of UNS.

HPC and data center scale application developers are abandoning POSIX IO because file system metadata synchronization and serialization overheads of providing strong consistency and durability are too costly – and often unnecessary – for their applications. Unfortunately, designing file systems with weaker consistency or durability semantics excludes applications that rely on stronger guarantees, forcing developers to re-write their applications or deploy them on a different system. We present a framework and API that lets administrators specify their consistency/durability requirements and dynamically assign them to subtrees in the same namespace, allowing administrators to optimize subtrees over time and space for different workloads. We show similar speedups to related work but more importantly, we show performance improvements when we custom fit subtree semantics to applications such as checkpoint-restart (91.7x speedup), user home directories (0.03 standard deviation from optimal), and users checking for partial results (2% overhead).

The Oak Ridge Leadership Computing Facility (OLCF) runs the No. 4 supercomputer in the world, supported by a petascale file system, to facilitate scientific discovery. In this paper, using the daily file system metadata snapshots collected over 500 days, we have studied the behavioral trends of 1, 362 active users and 380 projects across 35 science domains. In particular, we have analyzed both individual and collective behavior of users and projects, highlighting needs from individual communities and the overall requirements to operate the file system. We have analyzed the metadata across three dimensions, namely (i) the projects' file generation and usage trends, using quantitative file system-centric metrics, (ii) scientific user behavior on the file system, and (iii) the data sharing trends of users and projects. To the best of our knowledge, our work is the first of its kind to provide comprehensive insights on user behavior from multiple science domains through metadata analysis of a large-scale shared file system. We envision that this OLCF case study will provide valuable insights for the design, operation, and management of storage systems at scale, and also encourage other HPC centers to undertake similar such efforts.

Key-Value stores provide scalable metadata service for distributed file systems. However, the metadata's organization itself, which is organized using a directory tree structure, does not fit the key-value access pattern, thereby limiting the performance. To address this issue, we propose a distributed file system with a loosely-coupled metadata service, LocoFS, to bridge the performance gap between file system metadata and key-value stores. LocoFS is designed to decouple the dependencies between different kinds of metadata with two techniques. First, LocoFS decouples the directory content and structure, which organizes file and directory index nodes in a flat space while reversely indexing the directory entries. Second, it decouples the file metadata to further improve the key-value access performance. Evaluations show that LocoFS with eight nodes boosts the metadata throughput by 5 times, which approaches 93% throughput of a single-node key-value store, compared to 18% in the state-of-the-art IndexFS.

A peer-to-peer (P2P) file sharing system provides a platform that enables a tremendous number of nodes to share their files. Retrieving desired files efficiently and trustworthily is critical in such a large and jumbled system. However, the issues of efficient searching and trustworthy searching have only been studied separately. Simply combining the methods to achieve the two goals doubles system overhead. In this paper, we first study trace data from Facebook and BitTorrent. Guided by the observations, we propose a system that integrates a social network into a P2P network, named Social-P2P, for simultaneous efficient and trustworthy file sharing. It incorporates three mechanisms: (1) interest/trust-based structure, (2) interest/trust-based file searching, and (3) trust relationship adjustment. By exploiting the social interests and relationships in the social network, the interest/trust-based structure groups common-multi-interest nodes into a cluster and further connects socially close nodes within a cluster. The comparably stable nodes in each cluster form a Distributed Hash Table (DHT) for inter-cluster file searching. In the interest/trust-based file searching mechanism, a file query is forwarded to the cluster of the file by the DHT routing first. Then, it is forwarded along constructed connections within a cluster, which achieves high hit rate and reliable routing. Moreover, sharing files among socially close friends discourages nodes from providing faulty files because people are unlikely to risk their reputation in the real-world. In the trust relationship adjustment mechanism, each node in a routing path adaptively decreases its trust on the node that has forwarded a faulty file in order to avoid routing queries towards misbehaving nodes later on. We conducted extensive trace-driven simulations and implemented a prototype on PlanetLab. Experimental results show that Social-P2P achieves highly efficient and trustworthy file sharing compared to current file sharing systems and trust management systems.

Big data is an emerging term in the storage industry, and it is data analytics on big storage, i.e., Cloud-scale storage. In Cloud-scale (or EB-scale) file systems, load balancing in request workloads across a metadata server cluster is critical for avoiding performance bottlenecks and improving quality of services.Many good approaches have been proposed for load balancing in distributed file systems. Some of them pay attention to global namespace balancing, making metadata distribution across metadata servers as uniform as possible. However, they do not work well in skew request distributions, which impair load balancing but simultaneously increase the effectiveness of caching and replication. In this paper, we propose Cloud Cache (C2), an adaptive and scalable load balancing scheme for metadata server cluster in EB-scale file systems. It combines adaptive cache diffusion and replication scheme to cope with the request load balancing problem, and it can be integrated into existing distributed metadata management approaches to efficiently improve their load balancing performance. C2 runs as follows: 1) to run adaptive cache diffusion first, if a node is overloaded, loadshedding will be used; otherwise, load-stealing will be used; and 2) to run adaptive replication scheme second, if there is a very popular metadata item (or at least two items) causing a node be overloaded, adaptive replication scheme will be used, in which the very popular item is not split into several nodes using adaptive cache diffusion because of its knapsack property. By conducting performance evaluation in trace-driven simulations, experimental results demonstrate the efficiency and scalability of C2.

Efficient file query is important to the overall performance of peer-to-peer (P2P) file sharing systems. Clustering peers by their common interests can significantly enhance the efficiency of file query. Clustering peers by their physical proximity can also improve file query performance. However, few current works are able to cluster peers based on both peer interest and physical proximity. Although structured P2Ps provide higher file query efficiency than unstructured P2Ps, it is difficult to realize it due to their strictly defined topologies. In this work, we introduce a Proximity-Aware and Interest-clustered P2P file sharing System (PAIS) based on a structured P2P, which forms physically-close nodes into a cluster and further groups physically-close and common-interest nodes into a sub-cluster based on a hierarchical topology. PAIS uses an intelligent file replication algorithm to further enhance file query efficiency. It creates replicas of files that are frequently requested by a group of physically close nodes in their location. Moreover, PAIS enhances the intra-sub-cluster file searching through several approaches. First, it further classifies the interest of a sub-cluster to a number of sub-interests, and clusters common-sub-interest nodes into a group for file sharing. Second, PAIS builds an overlay for each group that connects lower capacity nodes to higher capacity nodes for distributed file querying while avoiding node overload. Third, to reduce file searching delay, PAIS uses proactive file information collection so that a file requester can know if its requested file is in its nearby nodes. Fourth, to reduce the overhead of the file information collection, PAIS uses bloom filter based file information collection and corresponding distributed file searching. Fifth, to improve the file sharing efficiency, PAIS ranks the bloom filter results in order. Sixth, considering that a recently visited file tends to be visited again, the bloom filter based approach is enhanced by only checking the newly added bloom filter information to reduce file searching delay. Trace-driven experimental results from the real-world PlanetLab testbed demonstrate that PAIS dramatically reduces overhead and enhances the efficiency of file sharing with and without churn. Further, the experimental results show the high effectiveness of the intra-sub-cluster file searching approaches in improving file searching efficiency.

With data volumes rapidly growing on a global scale, PB-scale storage systems will not meet the data-storage requirement capacity of applications in the future. It is time for the research and development of EB-scale storage systems that are general purpose, highly scalable, and easily deployed. Metadata server clustering is the core technology for constructing EB-scale storage systems. Based on pNFS and the Blue Whale block device file system EXFS, Blue Whale metadata server clustering technology for EB-scale storage is presented. To begin, the differences between EB-scale metadata server systems and PB-scale are analyzed. Then, the prototype architecture of a metadata server cluster and the distribution strategy of a global namespace are introduced. Following this, the atomic operation protocol for consistency based on a distributed log is presented. Finally, fine-grained metadata migration technology with low latency is proposed. Measurement results confirm that each metadata server can offer more than 10,000 OPS to EB-scale storage systems simultaneously.

File sharing applications in mobile ad hoc networks (MANETs) have attracted more and more attention in recent years. The efficiency of file querying suffers from the distinctive properties of such networks including node mobility and limited communication range and resource. An intuitive method to alleviate this problem is to create file replicas in the network. However, despite the efforts on file replication, no research has focused on the global optimal replica creation with minimum average querying delay. Specifically, current file replication protocols in mobile ad hoc networks have two shortcomings. First, they lack a rule to allocate limited resources to different files in order to minimize the average querying delay. Second, they simply consider storage as available resources for replicas, but neglect the fact that the file holders’ frequency of meeting other nodes also plays an important role in determining file availability. Actually, a node that has a higher meeting frequency with others provides higher availability to its files. This becomes even more evident in sparsely distributed MANETs, in which nodes meet disruptively. In this paper, we introduce a new concept of resource for file replication, which considers both node storage and meeting frequency. We theoretically study the influence of resource allocation on the average querying delay and derive a resource allocation rule to minimize the average querying delay. We further propose a distributed file replication protocol to realize the proposed rule. Extensive trace-driven experiments with synthesized traces and real traces show that our protocol can achieve shorter average querying delay at a lower cost than current replication protocols.

With the rapid development of Internet, cloud environment work becomes more and more popular, the requirements of user experience are increasingly high, research on collaborative work in cloud environment has gradually become the mainstream of collaborative work. Based on this, collaborative work under traditional P2P has been unable to meet the needs of the development of the network changes. According to the new architecture, the higher real-time requirements and others in the cloud environment, this paper improves COT algorithm which can adapt to the C/S architecture, reduce unnecessary data transmission and query conversion time, and achieve higher real-time requirements; In this paper, the file model under cloud storage is improved into cooperative cloud office file management system, and the improved COT algorithm is applied to file management in the cooperative office system in view of the three operations(create, delete, rename) of the file nodes in the file model. In addition, For the sake of verify the feasibility and effectiveness of the algorithm, this paper develops a Cross-Platform real-time cloud office file management system based on WebSocket protocol: CloudFileSystem.

More and more network services integrate the cloud platform to deliver their services. In order to accelerate the data sharing among users, data are usually divided and encapsulated into small pieces before delivery. However, the cloud file system enlarges the basic block size for maximizing the access throughput. This conflict between network transmission and cloud file system may induce high fragmentation in the cloud file system. In this paper, we propose one minimum block defragmentation mechanism (MBDM) to avoid any fragmentation in the cloud file system even if data are exchanged piece by piece. Based on our experiments, the proposed MBDM save 36% time and 46% energy consumption than the traditional BitTorrent (BT) service.

All kinds of intelligent terminal equipment emerge in endlessly and grow explosively. Typical devices, such as smartphones and tablets, are becoming more and more popular, furthermore, intelligent equipment market share has more than that on PCs. And on mobile devices area, the demand for virtualization technology is gradually increasing. As an important branch of computer technology development, operating system level virtualization is becoming a hot spot of current research due to its low overhead and the advantages of lightweight. In this paper, based on the Android system on mobile devices, implement a prototype which can run multiple virtual system instances on only an Android OS by modifying the namespace mechanism in the Linux kernel to and expand into driver namespace mechanism. In addition, put forward a model called the active- inactive model guaranteeing the process if and only if in the active namespace to operate the device drivers, which was designed and implemented a new scheme of operating system level virtualization.

The search efficiency improvement especially concerning search for rare files is a fundamental challenge in mobile peer-to-peer (MP2P) systems. In this paper, we propose a simple and efficient replication technique to increase file availability in MP2P systems which enhances the search efficiency. The aim of our proposal is to maintain a threshold level of availability at all times in order to find a file within a small number of overlay hops even for rare file. We base on a global availability estimation to choose suitable file to replicate and suitable replica degree. Simulations show that our proposal significantly improves search efficiency in terms of search success, search delay and overhead.

Many computing sites, LANL being one of them, have a requirement for long-term retention of mostly cold data. Although the main function of this storage tier is capacity, it does also have a bandwidth requirement. For many years, tape was the best economic solution for this requirement. However, over time, data sets have grown larger more quickly than tape bandwidth has improved. We have now entered a regime in which disk is the more economically efficient medium for this storage tier. Also more and more, data dominates the computing world. There is a "sea" of data out there in many different formats such as file, object, and Key-value that needs to be efficiently managed and effectively used. In this paper, we introduce a new hybrid storage system named MarFS. MarFS is a Near-POSIX File System using scale-out commercial/cloud for data and many POSIX file systems for metadata services. MarFS is an approach to support a data lake for HPC that sits on industry based commodity storage hardware and is a software layer that provides a global namespace and near POSIX semantics. MarFS provides the capability to serve as an umbrella over a variety of underling storage layers. In this paper, we present the system architecture of the proposed MarFS near-POISX file system, we conduct early functional performance testing cases on MarFS's software components, and finally we address current deployment status and future development works of the MarFS.

Organizing documents in the file system is one of the most tedious and thorny tasks for most computer users. Taxonomies based on hand made directory hierarchies still remain the only possible alternative for most small and medium enterprises, public administrations and individual users. However, both the limitations of the hierarchical organization of file systems and the difficulty of maintaining the coherence within the taxonomy have raised the need for more scalable and effective approaches. Desktop searching applications provide proprietary interfaces that enable content-based searching at the cost of having no control on the indexing and ranking of results. Semantic file systems, instead, leave users the freedom to manage the taxonomy according to their specific needs, but lose the standard file system features. In this paper we describe GFS (graph-based file system) a new hybrid file system that extends the standard hierarchical organization of files with semantic features. GFS allows the user to nest semantic spaces inside the directory hierarchy leaving unaltered system folders. Semantic spaces allow customized file tagging and leverage on browsing to guide file searching. Since GFS does not change the low-level interface to interact with file systems, users can continue to use their favorite file managers to interact with it. Moreover, no changes are required to integrate the semantic features in proprietary software.

IoT(Internet of Things) is a concept of connected internet pursuing direct access to devices or sensors in fused environment of personal, industrial and public area. In IoT environment, it is possible to access realtime data, and the data format and topology of devices are diverse. Also, there are bidirectional communications between users and devices to control actuators in IoT. In this point, IoT is different from the conventional internet in which data are produced by human desktops and gathered in server systems by way of one-sided simple internet communications. For the cloud or portal service of IoT, there should be a file management framework supporting systematic naming service and unified data access interface encompassing the variety of IoT things. This paper implements a DB-based virtual file system maintaining attributes of IoT things in a UNIX-styled file system view. Users who logged in the virtual shell are able to explore IoT things by navigating the virtual file system, and able to access IoT things directly via UNIX-styled file I⋅O APIs. The implemented virtual file system is lightweight and flexible because it maintains only directory structure and descriptors for the distributed IoT things. The result of a test for the virtual shell primitives such as mkdir() or chdir() shows the smooth functionality of the virtual file system, Also, the exploring performance of the file system is better than that of Window file system in case of adopting a simple directory cache mechanism.Keywords:IoT(Internet of Things), Cloud Service, VFS(Virtual File System), DB-Based File System

The explosive growth in data volume and complexity imposes great challenges for file systems. To address these challenges, an innovative namespace management scheme is in desperate need to provide both the ease and efficiency of data access. In almost all today's file systems, the namespace management is based on hierarchical directory trees. This tree-based namespace scheme is prone to severe performance bottlenecks and often fails to provide real-time response to complex data lookups. This paper proposes a Semantic-Aware Namespace scheme, called SANE, which provides dynamic and adaptive namespace management for ultra-large storage systems with billions of files. SANE introduces a new naming methodology based on the notion of semantic-aware per-file namespace, which exploits semantic correlations among files, to dynamically aggregate correlated files into small, flat but readily manageable groups to achieve fast and accurate lookups. SANE is implemented as a middleware in conventional file systems and works orthogonally with hierarchical directory trees. The semantic correlations and file groups identified in SANE can also be used to facilitate file prefetching and data de-duplication, among other system-level optimizations. Extensive trace-driven experiments on our prototype implementation validate the efficacy and efficiency of SANE.

With the increase in personal computer clusters in popularity and quantity, message passing between nodes has been an important issue for high failure rate in the network. File access in a cluster file system often contains several sub-operations; each includes one or more network transmissions. Any network failures cause the file system service unavailable. In this paper, we describe a highly reliable message-passing mechanism HR-NET, which tolerates both software and hardware network failures. HR-NET provides fine-grained, connection-level failover across redundant communication paths. With it, the file system can keep passing messages because HR-NET handles failures automatically by either recovery from network failures or failed over to a backup; therefore, it screens network failures from requests or data transmission of cluster file system. Load balance for messages is also achieved to relieve network traffic. For transmission timeout, HR-NET proposes a priority-based message scheduling which dynamically manages messages in an appropriate order to tolerate request–response failures between clients and servers. HR-NET is implemented upon standard network protocol stack. Performance results show that HR-NET can provide almost full underlying network bandwidth with average 6.17% throughput loss and provide a fast recovery. Experiments with cluster file system show that the overall performance degradation is below 8% due to failover of HR-NET while the reliability is highly enhanced.

his paper, provide survey of the proposed namespace management schemes for file system. Namespace management can be used to reduce exhaustive search over all directories. Namespace using semantic correlation can also increase search ability. File system namespace as an information organizing infrastructure is a help to improve system's quality of service such as performance, scalability, and ease of use. This paper discusses the improvement to be made for future proposed namespace schemes. This paper provides reader with the basis for research in namespace schemes for file system.

Peer-to-Peer (P2P) file sharing attracts much attention due to its scalability and robustness. One important metric in measuring the performance of a P2P file sharing system is the amount of time required for all peers to get the files. We refer this time as the file distribution time. Researchers have proposed protocols to minimize the file distribution time under different situation. However, most works are based on the single-file scenario. On the other hand, there are studies showing that in a file sharing application, users may download multiple files at the same time. In this paper, we analyze the minimum time needed to distribute multiple files. We develop an explicit expression for the minimum amount of time needed to distribute multiple files in a heterogeneous P2P fluid model. Unlike the single-file scenario, we demonstrate that the theoretical lower bound in multi-file is not always achievable. With a comprehensive consideration of all the configurations, we elaborate how to partition the bandwidth capacities of both seeds and leechers for a particular file such that the finish time is optimal.

Organisations of all types and sizes have come to rely on file transfer technology to help their businesses run smoothly. But basic file transfer technology is inherently limiting and inadequate. These days, regulatory demands and changing corporate needs are putting pressure on traditional methods and solutions. Organisations of all types and sizes have come to rely on file transfer technology to help their businesses run smoothly. But basic file transfer technology is inherently limiting and inadequate. Many file transfer applications were designed as simple utilities, not as enterprise solutions, and lack the management, control and integration capabilities needed to support today's challenging business environment. The answer is the next generation of managed file transfer (MFT), explains Dan Dunford of Attachmate.

Distributed cryptographic file systems enable file sharing among their users and need the adoption of a key management scheme for the distribution of the cryptographic keys to authorized users according to their specific degree of trust. In this paper we describe the architecture of a basic secure file sharing facility relying on a multi-party threshold-based key-sharing scheme that can be overlaid on top of the existing stackable networked file systems, and discuss its application to the implementation of distributed cryptographic file systems. It provides flexible access control policies supporting multiple combination of roles and trust profiles. A proof of concept prototype implementation within the Linux operating system framework demonstrated its effectiveness in terms of performance and security robustness.

A hierarchical namespace is a common abstraction used for data organization within modern file systems. Fast translation of namespace objects to physical locations is necessary to carry out efficient file system operations. For reasons attributed to modularity, security, and to some extent legacy, namespace translations involves iterative translation of intervening directory objects from the root of the namespace. Namespace resolution is typically a multi-step process, potentially involving serialized I/O operations at each step. In this paper, we propose a rethink of the strategy to fetch pathname entries. Our technique, StepAhead, proactively utilizes hints about namespace translation lookup failures to enable parallel and just-in-time fetching of necessary path translation data into memory to increase cache hits significantly. With StepAhead, we measure an increase in cache hit rates for path translation data across a set of six workloads by as much as 51%, which in turn results in application speed-up of as much as 20%.

In this paper, we propose an overlay routing architecture for information centric networks that operates on hash-based groupings of content routers and namespaces. Proposed architecture creates hash-based clusters within each domain using globally shared hash functions, and distributes the global namespace to different clusters. In doing so, we achieve the following notable improvements in control and data plane operations: (i) reduced communication overhead in setting up and maintaining the routing tables (which is resulted by hierarchical communication framework and effective distribution of network resources), (ii) reduced storage overhead (to manage control/data plane operations), and (iii) improved utilization of system resources at the content routers (i.e., storage/computing), which are allocated to in-network caching and packet forwarding operations, due to use of hash-dependent caching and overlay-driven request forwarding.

As the number of cores grows in commodity architectures so does the likelihood of failures. A distributed actor model potentially facilitates the development of reliable and scalable software on these architectures. Key components include lightweight processes which 'share nothing' and hence can fail independently. Erlang is not only increasingly widely used, but the underlying actor model has been a beacon for programming language design, influencing for example Scala, Clojure and Cloud Haskell.While the Erlang distributed actor model is inherently scalable, we demonstrate that it is limited by some pragmatic factors. We address two network scalability issues here: globally registered process names must be updated on every node (virtual machine) in the system, and any Erlang nodes that communicate maintain an active connection. That is, there is a fully connected O ( n 2 ) network of n nodes.We present the design, implementation, and initial evaluation of a conservative extension of Erlang - Scalable Distributed (SD) Erlang. SD Erlang partitions the global namespace and connection network using s_groups. An s_group is a set of nodes with its own process namespace and with a fully connected network within the s_group, but only individual connections outside it. As a node may belong to more than one s_group it is possible to construct arbitrary connection topologies like trees or rings.We present an operational semantics for the s_group functions, and outline the validation of conformance between the implementation and the semantics using the QuickCheck automatic testing tool. Our preliminary evaluation in comparison with distributed Erlang shows that SD Erlang dramatically improves network scalability even if the number of global operations is tiny (0.01%). Moreover, even in the absence of global operations the reduced connection maintenance overheads mean that SD Erlang scales better beyond 80 nodes (1920 cores). We address the network scalability limitations of distributed Erlang.We present the design and implementation of Scalable Distributed Erlang.We give a semantics for scalable groups and validate the implementation against it.We provide a preliminary evaluation of distributed Erlang and SD Erlang performance.The performance evaluation shows that introducing s_groups improves the scalability.

A peer-to-peer (P2P) file sharing system can be reduced into a system of addition-min fuzzy relation inequalities. For evaluating the satisfaction degree of the potential solutions in such system, we define a ranking function and establish the corresponding evaluation model. Numerical examples are provided to illustrate the feasibility and efficiency of the proposed evaluation model.

When building an app using a library for the platform, it is important to understand the features that the library provides. In this chapter, you’ll look at various features that WinJS provides for building Windows Runtime apps using JavaScript, HTML, and CSS. These features include namespaces, modules, promises, and query selectors. These are features used in almost all the applications that you build using WinJS. By taking advantage of the features like namespaces and modules, you can make your app easier to maintain, instead of adding everything under the global namespace.

Named Data Networking (NDN) has been considered as a promising networking technology that makes up for the limitation of current IP networking. However, due to its early stage, there are still a lot of open issues in NDN research. From the issues, the limitation of changing namespace keeps an application from modifying its namespace or adding additional namespace after once the namespace was fixed. In NDN, since the namespace of an application is tightly coupled with the routing paths of the application, this limitation is not easy to avoid. In order to resolve the issue, we suggest an alias naming scheme that enables several namespaces to coexist for the same application on NDN network. The evaluation showed the proposed scheme worked well as intended.

This paper introduces GlobalFS, a POSIX-compliant geographically distributed file system. GlobalFS builds on two fundamental building blocks, an atomic multicast group communication abstraction and multiple instances of a single-site data store. We define four execution modes and show how all file system operations can be implemented with these modes while ensuring strong consistency and tolerating failures. We describe the GlobalFS prototype in detail and report on an extensive performance assessment. We have deployed GlobalFS across all EC2 regions and show that the system scales geographically, providing performance comparable to other state-of-the-art distributed file systems for local commands and allowing for strongly consistent operations over the whole system. The code of GlobalFS is available as open source.

During the last few decades, Data-intensive File Systems (DiFS), such as Google File System (GFS) and Hadoop Distributed File System (HDFS) have become the key storage architectures for big data processing. These storage systems usually divide files into fixed-sized blocks (or chunks). Each block is replicated (usually three-way) and distributed pseudo-randomly across the cluster. The master node (namenode) uses a huge table to record the locations of each block and its replicas. However, with the increasing size of the data, the block location table and its corresponding maintenance could occupy more than half of the memory space and 30% of processing capacity in master node, which severely limit the scalability and performance of master node. We argue that the physical data distribution and maintenance should be separated out from the metadata management and performed by each storage node autonomously. In this paper, we propose Deister, a novel block management scheme that is built on an invertible deterministic declustering distribution method called Intersected Shifted Declustering (ISD). Deister is amendable to current research on scaling the namespace management in master node. In Deister, the huge table for maintaining the block locations in the master node are eliminated and the maintenance of the block-node mapping is performed autonomously on each data node. Results show that as compared with the HDFS default configuration, Deister is able to achieve identical performance with a saving of about half of the RAM space in master node and is expected to scale to double the size of current single namenode HDFS cluster, pushing the scalability bottleneck of master node back to namespace management.

As building block for cloud storage, distributed file system uses underlying local file systems to manage objects. However, the underlying file system, which is limited by metadata and journaling I/O, significantly affects the performance of the distributed file system. This paper presents an NVM-based file system (referred to as NV-Booster) to accelerate object access for storage node. The NV-Booster leverages byte-addressability and persistency of nonvolatile memory (NVM) to speedup metadata accesses and file system journaling. With NV-Booster, metadata is kept in NVM and accessed in byte-addressable manner through memory bus, while object is stored on hard disk and accessed from I/O bus. In addition, proposed NV-Booster enables fast object search and mapping between object ID and on-disk location with an efficient in-memory namespace management. NV-Booster is implemented in kernel space with NVDIMM and has been extensively evaluated under various workloads. Our experiments show that NV-Booster improves Ceph performance up to 10X, compared to the Ceph with existing local file systems.

Although pNFS (Parallel Network File System) improves the performance of NFS, to optimize the performance of pNFS, we need to deploy high-end network equipment and the data servers need high-end hard disks, which is expensive for many enterprises. As a result, in this work, we presented hpNFS (Hierarchically Parallel Network File System), which is a hierarchical approach of pNFS to decrease the implementation costs and fulfill storage requirements for enterprise. This work is conducted under the ILM (Information Lifecycle Management) [1] [2], where hpNFS is one of the modules of ILM solution. We left a few issues to be resolved by later stage.