Search

Distributed file systems are often used to replicate a Web site's content among its many servers. However, for content that needs to be dynamically updated and distributed to many servers, file system locking protocols exhibit high latency and heavy network usage. Poor performance arises because the Web-serving workload differs from the assumed workload. To address the shortcomings of file systems, we introduce the publish consistency model well suited to the Web-serving workload and implement it in the producer-consumer locking protocol. A comparison of this protocol against other file system protocols by simulation shows that producer-consumer locking removes almost all latency due to protocol overhead and significantly reduces network load.

We collect and analyze a snapshot of data from 10,568 file systems of 4801 Windows personal computers in a commercial environment. The file systems contain 140 million files totaling 10.5 TB of data. We develop analytical approximations for distributions of file size, file age, file functional lifetime, directory size, and directory depth, and we compare them to previously derived distributions. We find that file and directory sizes are fairly consistent across file systems, but file lifetimes vary widely and are significantly affected by the job function of the user. Larger files tend to be composed of blocks sized in powers of two, which noticeably affects their size distribution. File-name extensions are strongly correlated with file sizes, and extension popularity varies with user job function. On average, file systems are only half full.

The increasing need for data sharing in large-scale distributed systems may place a heavy burden on critical resources such as file servers and networks. Our examination of the workload in one large commercial engineering environment shows that wide-spread sharing of unstable files among tens to hundreds of users is common. Traditional client-based file cacheing techniques are not scalable in such environments. We propose Frolic, a scheme for cluster-based file replication in large-scale distributed file systems. A cluster is a group of workstations and one or more file servers on a local area network. Large distributed systems may have tens or hundreds of clusters connected by a backbone network. By dynamically creating and maintaining replicas of shared files on the file servers in the clusters using those files, we effectively reduce reliance on central servers supporting such files, as well as reduce the distances between the accessing sites and data. We propose and study algorithms for the two main issues in Frolic, 1) locating a valid file replica, and 2) maintaining consistency among replicas. Our simulation experiments using a statistical workload model based upon measurement data and real workload characteristics show that cluster-based file replication can significantly reduce file access delays and server and backbone network utilizations in large-scale distributed systems over a wide range of workload conditions. The workload characteristics most critical to replication performance are: the size of shared files, the number of clusters that modify a file, and the number of consecutive accesses to files from a particular cluster.

This paper describes different methods and techniques used to model, analyze, evaluate and implement distributed file systems. Distributed file systems are characterized by the distributed system hardware and software architecture, in which they are implemented as well as by the file systems' functions. In addition, distributed file system performance depends on the load executed in the system. Modeling and analysis of distributed file systems requires new methods to approximate complexity of the system and to provide a useful solution. The complexity of the distributed file system is reflected in the possible placement of the files, file replication, and migration of files and processes. The synchronization mechanisms are needed to control file access. File sharing involves load sharing in distributed environment.

Recent measurement studies [8] have shown that peer-to-peer (P2P) file sharing applications are the major traffic source in the Internet. P2P applications, such as eDonkey, Kazaa, or BitTorrent, form overlay networks on the application layer and offer its peers to download and share their files with other peers in a highly distributed way. As a consequence, peers act simultaneously as both clients and servers. For a comprehensive survey of P2P technology, we refer to [7].

In distributed systems the efficiency of the network file system is a key performance issue. Replication of files and directories can enhance file system efficiency, but the choice of replication techniques is crucial. This paper studies a number of replication techniques, including remote access, prereplication, weighted voting, and two demand replication schemes: polling and staling. It develops a Markov chain model, which is capable of characterizing properties of file access sequences, including access locality and access bias. The paper compares the replication techniques under three different network file system architectures. The results show that, under reasonable assumptions, demand replication requires fewer file transfers than remote access, especially for files that have a high degree of access locality. Among the demand replication schemes, staling requires fewer auxiliary messages than polling.