Search

Shift is a lightweight framework for high performance local and remote file transfers that provides resiliency across a wide variety of failure scenarios. Shift supports multiple file transport protocols with automatic selection of the most appropriate mechanism between each pair of participating hosts allowing it to adapt to heterogeneous clients with differing software and network access restrictions. File system information is gathered from clients and servers to detect file system equivalence and enable path rewriting so that multiple clients can be automatically spawned in parallel to carry out both single and multi-file transfers to multiple servers selected according to load and availability. This improves both reliability and performance by eliminating single points of failure and overcoming single system bottlenecks. End-to-end integrity is provided using cryptographic hashes at the source and destination with support for partial file retransmission of only corrupted portions. This paper presents the design and implementation of Shift and details the mechanisms utilized to enhance the reliability and performance of file transfers.

Container-based virtualization is the most popular solution for isolating resources among users in a shared testbed. Container achieves good performance but makes the code quite complicated and hard to maintain, to debug and to deploy. We explore an alternative philosophy to enable the isolation based on commodity OS, i.e., utilizing existing features in commodity OS as much as possible rather than introducing complicated containers. Merely granting each user-id in the OS a dedicated and isolated network address as well as specific routing table, we enhance the commodity OS with the functionality of network namespace isolation. We posit that an OS’s built-in features plus our feather-weight enhancement meet basic requirements for separating activities among different users of a shared testbed. Applying our prototype which has been implemented, we demonstrate the functionality of our solution can support a VINI-like environment with marginal cost of engineering and tiny overhead.

Redundant data transfer over a network is one of the important reasons of traffic congestion today. In this paper, we proposed an efficient and secure file sharing model using de-duplication technology to resolve it. A file sharing based de-duplication system reduce bandwidth and storage at both client and server machine. It does not download duplicate blocks that already have been downloaded. To achieve the security of client data, three-tier architecture is proposed in this work. For this purpose SHA-1 hash function is being used, in which 8KB block of data is converted into a 20 bytes digest. Thus the design presents a dramatic reduction in storage space requirement for various workloads and hence reduces time to perform backup in bandwidth constraint environment.

A Grid computing environment allows forming Virtual Organizations (VOs) to aggregate and share resources. We present a VO File System (VOFS) which is a VO-aware distributed file system that allows VO members to share files. VOFS supports access and location transparency by maintaining a common file namespace, which is decentralized in order to improve robustness. VOFS includes a P2P system of file servers, a VO membership service and a policy and role based security mechanism. VOFS can be mounted to a local file system in order to access files using POSIX file API. VOFS can operate in a dynamic Grid environment (e.g. desktop Grids) since it tolerates unplanned resource arrival and departure (churn) while maintaining a single uniform namespace. It supports transparent disconnected operations that allow the user to work on files while being disconnected.

Building large-scale distributed systems that are decentralized is challenging. Distributed Hash Tables, or structured overlays, attempted to make the process of building these distributed systems easier, by abstracting away many of the complexities of the physical topology, and provide a virtual namespace. The virtual namespace can then be exploited, for example to build or maintain other data structures like a tree to support application-level multicast or group communication. Because the namespace is independent of the underlying physical topology, it can be used to help mitigate the impact of network topology change, for example through churn. A challenge is to understand how to exploit the difference between a physical topology and a virtual topology (or namespace). In the first half of the talk, using examples from a number of systems I have worked on over the years, I will highlight how a namespace can be used and the differences between algorithms designed to work in virtual versus physical topologies. In the second half of the talk I will then outline how recently we have begun more deeply exploring the relationship between physical and virtual topologies. The first is exploiting a virtual topology to enable disruption-tolerant routing protocols that can work on networks that are frequently and dynamically partitioning. The second is examining how we can build physical topologies that are more aligned to a virtual topology, and how this makes building servers that run in Internet-scale Data Centres easier to implement.

In this paper, we present a visual representation of access control permissions in a standard hierarchical file system. Our visualization of file permissions leverages treemaps, a popular graphical representation of hierarchical data. In particular, we present a visualization of access control for the NTFS file system that can help a non-expert user understand and manipulate file system permissions in a simple and effective way. While our examples are based on NTFS, our approach can be used for many other hierarchical file systems as well.

Peer-to-peer (P2P) file sharing systems often replicate files to multiple nodes. File replication is beneficial in the sense that it can achieve good query latency, load balance, and reliability. However, it introduces another problem of maintaining mutual consistency among replica when a file is updated. The new file has to be propagated to all of its replica. In this paper, we propose an update propagation algorithm for P2P file sharing over wireless mobile networks (MONET). Compared to the previous algorithms proposed in wired P2P file sharing systems, our algorithm has low communication overhead. It also guarantees safe delivery of updates even when the underlying network is unreliable. This means that our algorithm is well matched to the characteristics of MONET such as limited battery power of mobile nodes, lower network reliability, and frequent disconnection of mobile nodes.

The main obstacles to grid file management come from the fact that grid file resources are typically stored in heterogeneous and distributed environment and accessed through various protocols. In this paper, we propose a grid file management system called Vega [1][2] Hotfile2 for data-intensive application in widely distributed systems and grid environments. Widely distributed and heterogeneous storage resource can converge into a single uniform storage resource using virtualization techniques in Hotfile2. It is a grid middleware provides a uniform file resources view and a high level interface to access widely distributed and heterogeneous data resources. Presented here are two services that we believe are fundamental to any grid file management system: high-performance, secure transport and metadata management. In this paper, we discuss many aspects of our design and implementation in Hotfile2, present the file system interface designed to support distributed applications such as batch job. Experiments are done to illustrate its performance, extensibility, and various other features

Traditional network file systems, like NFS, do not extend to wide-area due to low bandwidth and high network latency. We present WireFS, a Wide Area File System, which enables delegation of metadata management to nodes at client sites (homes). The home of a file stores the most recent copy of the file, serializes all updates, and streams updates to the central file server. WireFS uses access history to migrate the home of a file to the client site which accesses the file most frequently. We formulate the home migration problem as an integer programming problem, and present two algorithms: a dynamic programming approach to find the optimal solution, and a non-optimal but more efficient greedy algorithm. We show through extensive simulations that even in the WAN setting, access latency over WireFS is comparable to NFS performance in the LAN setting; the migration overhead is also marginal.

Distributed computer applications usually need processes that allow them to retrieve, store, and share data in a suitable way. As a consequence, file systems become a basic concern for these kinds of applications. Typical file systems have been designed on a computer network infrastructure whose communication and data exchange support is homogeneous. The Internet encourages the construction of file systems, which deal with heterogeneous computer platforms. This work presents SoapFS, a file system that is able to manage information in a heterogeneous environment such as the Internet. This is possible because it is based on technologies like RPC-XML and SOAP. This paper shows how a set of functions available in SoapFS allows the development of robust and heterogeneous distributed applications that involve remote data retrieval and storage. SoapFS shows competitive performance compared with popular file systems and it can connect different file systems working together like a one virtual file system.

With the advent of Storage Area Networks (SAN) and Network Attached Disks (NASD), a new trend in storage systems design is to move disks from behind storage servers and attach them to a SAN for direct client access. Such designs remove storage servers from the path of data transfer leading to highly scalable file systems. While there exists many file systems built using such direct attached disks, they are all optimized for traditional text-based data and are not well-suited for streaming continuous media, i.e., audio and video data. In this paper we present the architectural details of a scalable distributed continuous media file system built using SAN-attached disks. We describe the implementation details of our Linux-based prototype and show by experimention that the performance of our file system scales linearly with the number of disks and the number of clients, and the file system provides strict bandwidth guarantees for continuous media streams.

The emerging issues of personal and network mobility have created a need for the decoupling of identity from location in Internet addressing. This decoupling requires name resolution systems that can provide scalable resolution of globally unique persistent identifiers of communication endpoints, which may be users, devices, content or services. Recent developments in structured peer-to-peer overlay networks have made possible the scalable resolution of flat names, which opens up new possibilities in the area of naming and name resolution systems. In this paper we propose a scheme to provide authentication and verification in a name resolution system based on structured peer to peer networks such as distributed hash tables (DHTs). We specify how namespace security and global uniqueness may be managed with the use of public key cryptography. We also propose a framework within which independent overlay networks may compose a global namespace.

Social networks offering unprecedented content sharing are rapidly developing over the Internet. Unfortunately, it is often difficult to both locate and manage content in these networks, particularly when they are implemented on current peer-to-peer technologies. In this paper, we describe Wayfinder, a peer-to-peer file system that targets the needs of medium-sized content sharing communities. Wayfinder seeks to advance the state-of-the-art by providing three synergistic abstractions: a global namespace that is uniformly accessible across connected and disconnected operation, content-based queries that can be persistently embedded into the global namespace, and automatic availability management. Interestingly, Wayfinder achieves much of its functionality through the use of a peer-to-peer indexed data storage system called PlanetP: essentially, Wayfinder constructs the global namespace, locates specific files, and performs content searches by posing appropriate queries to PlanetP. We describe this query-based design and present preliminary performance measurements of a prototype implementation.

The explosive growth of the Web contents has led to increasing attention on scalability and availability of file system. Hence, the ways to improve the reliability and availability of system, to achieve the expected reduction in operational expenses and to reduce the operations of management of system have become essential issues. A basic technique for improving reliability of a file system is to mask the effects of failures through replication. Consistency control protocols are implemented to ensure the consistency among replicas. In this paper, we leveraged the concept of intermediate file handle to cover the heterogeneity of file system and proposed an efficient data consistency control scheme supporting dependence checking among writes and management of out-of-ordered requests for file server group. Finally, the results of experiments proved the efficiency of the proposed consistency control mechanism. Above all, easy to implement is our main design consideration.

By merging network and channel interfaces, resulting interfaces allow multiple computers to physically share storage devices. A storage area network (SAN) is a high-speed special-purpose network (or subnetwork) that interconnects different kinds of data storage devices with associated data servers on behalf of a larger network of users. In SAN, computers service local file requests directly from shared storage devices. Direct device access eliminates the server machines as bottlenecks to performance and availability. Communication is unnecessary between computers, since each machine views the storage as being locally attached. SAN provides us to very large physical storage up to 64-bit address space, but traditional file systems can't adapt to the file system for SAN because they have the limitation of scalability. In this paper, we present architectures and features of SANtopia that allows multiple machines to access and share disk and tape devices on a Fibre Channel or SCSI storage network in Linux system. It performs well as a local file system, as a traditional network file system running over IP environments, and as a high performance cluster file system running over storage area networks like Fibre Channel. SANtopia provides a key cluster enabling technology for Linux, helping to bring the availability, scalability, and load balancing benefits of clustering to Linux.

Peer-to-peer networks have been commonly used for tasks such as file sharing or file distribution. We study a class of cooperative file distribution systems where a file is broken up into many chunks that can be downloaded independently. The different peers cooperate by mutually exchanging the different chunks of the file, each peer being client and server at the same time. While such systems are already in widespread use, little is known about their performance and scaling behavior. We develop analytic models that provide insights into how long it takes to deliver a file to N clients. Our results indicate that the service capacity of these systems grows exponentially with the number of chunks a file consists of.

This paper studies network file system’s security requirements, and focuses on security performance constraints. We review security implementations in both, distributed file systems and pure network-based file systems, focusing on security services included in NFSv4’s proposed standard. Finally, we evaluate the tools being used to preserve data confidentiality, integrity and availability in computer communications, in order to evaluate whether they would be suitable for their use in network file systems.

In existing Internet file transfer mechanism, proxy servers play a major role in load balancing and reducing duplicate file access requests for services like FTP and WWW. However, proxy servers are usually unaware of the availability of cached contents on other peer proxy servers. This is a waste of time since duplicate requests are needed. Unnecessary traffic can be reduced if cooperation and coordination among peer proxies can be utilized. In this paper, peer-to-peer support was incorporated in ordinary file transfer and caching mechanism to reduce unnecessary processing time and storage. Through the location service, hosts requesting file services can dynamically determine if a copy is available and its current location. Work load for file servers will be greatly reduced, and personalization of file transfer configuration can be fully supported.

In mobile computing environment, mobile support station provides a wireless communication link to mobile clients and help them to access remote file servers that reside on a fixed network. Mobile hosts send requests to the mobile support station to access files on remote file servers. No longer this environment requires a user to stay at a fixed position in the network and enables almost unrestricted user mobility. The user mobility affects the file access latency. Mobile hosts have severe resource constraints in terms of limited size of non-volatile storage, and the burden of computation and communication load raises file access latency. In this paper, we propose a scheme for reducing the file access latency by using file migration scheme. We develope an on-demand file migration scheme which determines when the file server should migrate files to another server, and when it should transfer files to mobile hosts. Using simulation, we examine the effects of the parameters such as file access frequency, file size, and mobility rate on the file system performance. We present simulation results, providing useful insight into file system latency.

The exponential growth in hardware performance has reduced to a minimum the distinction on the software side between basic functionality, provided by the operating system, and extended functionality, provided by applications. Nowadays, a lot of advanced features such as multimedia handling are already integrated into the system itself. Given this trend, it is surprising that the current vision of the file management system still remains so close to the level of the physical storage structure of a file, following the basic equivalence of file = pathname. In this paper, we present OMX-FS, a multi-user distributed object-oriented file system, based on the generic object model OM. OMX-FS considers files as objects and introduces concepts typical of object-oriented database systems, enabling both application developers and users to interact with the file system at a higher-level of logical abstraction. We describe the implementation of OMX-FS in the Oberon environment.

Changing relative performance of processors, networks, and disks makes it necessary to reconsider algorithms using these three resources. As networks get faster and less congested topologies emerge, it becomes important to use network resources more aggressively to obtain good performance. Substitution of local disk accesses by accesses to remote memory can lead to better balanced resource usage and thus to faster systems. In this work we address the issue of file caching in a networked file system configuration. Distributed block-level in-memory caches are considered. We show that carefully constructed distributed concepts can lead to lower server load and better overall system performance than centralized concepts. Oversimplification, although aimed at gaining performance for single components, may deteriorate overall performance as a result of unbalanced resource usage.

This chapter discusses file servers that are accessible via some form of local network or internetwork. It characterizes file services from the client point of view along a number of dimensions; distributivity of the service itself, the network environment of the service, the file model supported by the service, the kind(s) of atomic update provided by the service, the techniques used for controlling the allocation of files. It then describes the client interfaces to three existing file services. Finally, the impact of these features on design and implementation is discussed. An appendix contains characterizations of a variety of existing network-based file services using the criteria developed in the chapter.