Search

Hadoop is a framework for storing and processing huge amounts of data. With HDFS, large data sets can be managed on commodity hardware. MapReduce is a programming model for processing vast amounts of data in parallel. Mapping and reducing can be performed by using the MapReduce programming framework. A very large amount of data is transferred from Mapper to Reducer without any filtering or recursion, resulting in overdrawn bandwidth. In this paper, we introduce an algorithm called Inner MAPping Combiner (IMapC) for the map phase. This algorithm in the Mapper combines the values of recurring keys. In order to test the efficiency of the algorithm, different approaches were tested. According to the test, MapReduce programs that are implemented with the Default Combiner (DC) of IMapC will be 70% more efficient than those that are implemented without one. To make computations significantly faster, this work can be combined with MapReduce. [ABSTRACT FROM AUTHOR]

In the distributed storage system, when data need to be recovered after node failure, the erasure code redundancy method occupies less storage space than the multi-copy method. At present, the repair mechanism using erasure code to reconstruct the failed node only considers the improvement of link bandwidth on the repair rate and does not consider the impact of the selection of data providing node-set on the repair performance. A single node fault data reconstruction method based on the Software Defined Network (SDN) using the erasure code method is designed to solve the above problems. This method collects the network link-state through SDN, establishes a multi-attribute decision-making model of the data providing node-set based on the node performance, and determines the data providing nodes participating in providing data through the ideal point method. Then, the data recovery problem of a single fault node is modeled as the optimization problem of an optimal repair tree, and a hybrid genetic algorithm is designed to solve it. The experimental results show that under the same erasure code scale, after selecting the nodes of the data providing node-set, compared with the traditional tree topology and star topology, the repair delay distribution of the designed single fault node repair method for a distributed storage system is reduced by 15% and 45% respectively, and the repair flow is close to the star topology, which is reduced by 40% compared with the traditional tree repair. [ABSTRACT FROM AUTHOR]

Erasure codes are being extensively deployed in distributed storage systems instead of replication to achieve fault tolerance in a storage efficient manner. While traditional erasure codes are storage efficient, they can result in a significant increase in the amount of data access and downloaded during rebuilding of failed or otherwise unavailable nodes. In this paper, we present a new framework, which we call piggybacking, for constructing distributed storage codes that are efficient in the amount of data read and downloaded during rebuilding, while meeting requirements arising out of system considerations in data centers—maximum-distance-separability (MDS), high-rate, and a small number of so-called substripes. Under this setting, to the best of our knowledge, piggyback codes achieve the minimum average amount of data access and downloaded during rebuilding among all existing explicit solutions. The piggybacking framework also offers a rich design space for constructing codes for a variety of other settings. In particular, we construct codes that require minimum amount of data access and downloaded for rebuilding among all existing solutions for: 1) binary MDS array codes with more than two parities and 2) MDS codes with the smallest locality during rebuilding. In addition, we show how piggybacking can be employed to enable efficient repair of parity nodes in codes that address the rebuilding of only systematic nodes. The basic idea behind the piggybacking framework is to take multiple instances of existing codes and add carefully designed functions of the data from one instance to the others. This framework provides 25% to 50% savings in the average amount of data access and downloaded during rebuilding depending on the choice of the code parameters. [ABSTRACT FROM AUTHOR]

Erasure codes are widely used in distributed storage systems since they provide space-optimal data redundancy to protect against data loss. Despite recent progress on quantifying average service latency when erasure codes are employed, there is very little work on providing differentiated latency among multiple tenants that may have different latency requirements. This paper proposes a novel framework for providing and optimizing differentiated latency in erasure-coded storage by investigating two policies, weighted queue and priority queue, for scheduling tenant requests. For both policies, we quantify service latency for different tenant classes for homogeneous files with arbitrary placement and service time distributions. We develop an optimization framework that jointly minimizes differentiated latency over three decision spaces: 1) data placement; 2) request scheduling; and 3) resource management. Efficient algorithms harnessing bipartite matching and convex optimization techniques are developed to solve the proposed optimization. Our solution enables elastic service-level agreements to meet heterogeneous application requirements. We further prototype our solution with both queuing models applied in an open-source, cloud storage deployment that simulates three geographically distributed data centers through bandwidth reservations. Experimental results validate our theoretical delay analysis and show significant joint latency reduction for different classes of files, providing valuable insights into service differentiation and elastic quality of service in erasure-coded storage systems. [ABSTRACT FROM PUBLISHER]

Maximum distance separable (MDS) array codes are widely used in storage systems due to their computationally efficient encoding and decoding procedures. An MDS code with $r$ redundancy nodes can correct any $r$ node erasures by accessing (reading) all the remaining information in the surviving nodes. However, in practice, $e$ erasures are a more likely failure event, for some $1 \le e < r$ . Hence, a natural question is how much information do we need to access in order to rebuild $e$ storage nodes. We define the rebuilding ratio as the fraction of remaining information accessed during the rebuilding of $e$ erasures. In our previous work, we constructed MDS codes, called zigzag codes, that achieve the optimal rebuilding ratio of $1/r$ for the rebuilding of any systematic node when $e=1$ ; however, all the information needs to be accessed for the rebuilding of the parity node erasure. The (normalized) repair bandwidth is defined as the fraction of information transmitted from the remaining nodes during the rebuilding process. For codes that are not necessarily MDS, Dimakis et al. proposed the regenerating codes framework where any $r$ erasures can be corrected by accessing some of the remaining information, and any $e=1$ erasure can be rebuilt from some subsets of surviving nodes with optimal repair bandwidth. In this paper, we present three results on rebuilding of codes: 1) we show a fundamental outer bound on the storage size of the node and the repair bandwidth similar to the regenerating codes framework, and show that zigzag codes achieve the optimal rebuilding ratio of $e/r$ for systematic nodes of MDS codes, for any $1 \le e \le r$ ; 2) we construct systematic codes that achieve optimal rebuilding ratio of $1/r$ , for any systematic or parity node erasure; and 3) we present error correction algorithms for zigzag codes, and in particular demonstrate how these codes can be corrected beyond their minimum Hamming distances. [ABSTRACT FROM AUTHOR]

High capacity storage systems distribute files across several storage devices (nodes) and apply an erasure code to meet availability and reliability requirements. Since devices can lose network connectivity or fail permanently, a dynamic repair mechanism must be put in place. In such cases a new recovery node gets connected to a given subset of the operating nodes and receives a part of the stored data. The objective of this paper is to investigate data survival for Random Linear Network Coding (RLNC) as a function of topology and communication overhead, defined by the number of connections and the number of transmitted packets to the recovery node, respectively. The paper includes two main contributions. First, a sufficient set of conditions for quasi-infinite longevity of the stored data is derived. Second, a comparison using experimental results shows that RLNC can be up to 50% more effective than traditional erasure codes like Reed-Solomon. [ABSTRACT FROM AUTHOR]

The problem of Multi-user Blind $X$ -secure $T$ -colluding Symmetric Private Information Retrieval from Maximum Distance Separable (MDS) coded storage system with $B$ Byzantine and $U$ unresponsive servers (U-B-MDS-MB-XTSPIR) is studied in this paper. Specifically, a database consisting of multiple files, each labeled by $M$ indices, is stored at the distributed system with $N$ servers according to $(N,K+X)$ MDS codes over $\mathbb {F}_{q}$ such that any group of up to $X$ colluding servers learn nothing about the data files. There are $M$ users, in which each user $m,m=1,\ldots,M$ privately selects an index $\theta _{m}$ and wishes to jointly retrieve the file specified by the $M$ users’ indices $(\theta _{1},\ldots,\theta _{M})$ from the storage system, while keeping its index $\theta _{m}$ private from any $T_{m}$ colluding servers, where there exists $B$ Byzantine servers that can send arbitrary responses maliciously to confuse the users retrieving the desired file and $U$ unresponsive servers that will not respond any message at all. In addition, each user must not learn information about the other users’ indices and the database more than the desired file. An U-B-MDS-MB-XTSPIR scheme is constructed based on Lagrange encoding. The scheme achieves a retrieval rate of $1-\frac {K+X+T_{1}+\ldots +T_{M}+2B-1}{N-U}$ with secrecy rate $\frac {K+X+T_{1}+\ldots +T_{M}-1}{ N-(K+X+T_{1}+\ldots +T_{M}+2B+U-1)}$ on the finite field of size $q\geq N+\max \{K, N-(K+X+T_{1}+\ldots +T_{M}+2B+U-1)\}$ for any number of files. [ABSTRACT FROM AUTHOR]

Fractional repetition (FR) codes are a class of repair efficient erasure codes that can recover a failed storage node with both optimal repair bandwidth and complexity. In this paper, we study the minimum distance of FR codes, which is the smallest number of nodes whose failure leads to the unrecoverable loss of the stored file. We derive a new upper bound on the minimum distance of FR codes, which is tighter than the Singleton bound and a Singleton-like bound that takes locality into account. Based on regular graphs and combinatorial designs, several families of FR codes with optimal minimum distance are obtained. [ABSTRACT FROM AUTHOR]

Regenerating codes are a class of recently developed codes for distributed storage that, like Reed-Solomon codes, permit data recovery from any subset of k nodes within the n-node network. However, regenerating codes possess in addition, the ability to repair a failed node by connecting to an arbitrary subset of d nodes. It has been shown that for the case of functional repair, there is a tradeoff between the amount of data stored per node and the bandwidth required to repair a failed node. A special case of functional repair is exact repair where the replacement node is required to store data identical to that in the failed node. Exact repair is of interest as it greatly simplifies system implementation. The first result of this paper is an explicit, exact-repair code for the point on the storage-bandwidth tradeoff corresponding to the minimum possible repair bandwidth, for the case when d=n-1. This code has a particularly simple graphical description, and most interestingly has the ability to carry out exact repair without any need to perform arithmetic operations. We term this ability of the code to perform repair through mere transfer of data as repair by transfer. The second result of this paper shows that the interior points on the storage-bandwidth tradeoff cannot be achieved under exact repair, thus pointing to the existence of a separate tradeoff under exact repair. Specifically, we identify a set of scenarios which we term as “helper node pooling,” and show that it is the necessity to satisfy such scenarios that overconstrains the system. [ABSTRACT FROM AUTHOR]

A user wants to retrieve a file from a database without revealing the identity of the file retrieved to the operator of the database (server), which is known as the problem of private information retrieval (PIR). If it is further required that the user obtains no information about the other files in the database, the concept of symmetric PIR (SPIR) is introduced to guarantee privacy for both parties. For SPIR, the server(s) need to access some randomness independent of the database, to protect the content of undesired files from the user. The information-theoretic capacity of SPIR is defined as the maximum number of information bits of the desired file retrieved per downloaded bit. In this paper, the problem of SPIR is studied for a distributed storage system with $N$ servers (nodes), where all data (including the files and the randomness) are stored in a distributed way. Specifically, the files are stored by an $(N,K_{C})$ -MDS storage code. The randomness is distributedly stored such that any $K_{C}$ servers store independent randomness information. We consider two scenarios regarding to the ability of the storage nodes to cooperate. In the first scenario considered, the storage nodes do not communicate or collude. It is shown that the SPIR capacity for MDS-coded storage (hence called MDS-SPIR) is $1-\frac {K_{C}}{N}$ , when the amount of the total randomness of distributed nodes (unavailable at the user) is at least $\frac {K_{C}}{N - K_{\vphantom {R_{l}}C}}$ times the file size. Otherwise, the MDS-SPIR capacity equals zero. The second scenario considered is the $T$ -colluding SPIR problem (hence called TSPIR). Specifically, any $T$ out of $N$ servers may collude, that is, they may communicate their interactions with the user to guess the identity of the requested file. In the special case with $K_{C}=1$ , i.e., the database is replicated at each node, the capacity of TSPIR is shown to be $1-\frac {T}{N}$ , with the ratio of the total randomness size relative to the file size be at least $\frac {T}{\vphantom {R_{l}}N - T}$. For TSPIR with MDS-coded storage (called MDS-TSPIR for short), when restricted to schemes with additive randomness where the servers add the randomness to the answers regardless of the queries received, the capacity is proved to equal $1-\frac {K_{C} + T - 1}{N}$ , with total randomness at least $\frac {K_{C} + T - 1}{N - K_{\vphantom {R_{l}}C} - T + 1}$ times the file size. The MDS-TSPIR capacity for general schemes remains an open problem. [ABSTRACT FROM AUTHOR]

In a distributed storage system, recovering from multiple failures is a critical and frequent task that is crucial for maintaining the system’s reliability and fault-tolerance. In this paper, we focus on the problem of repairing multiple failures in a centralized way, which can be desirable in many data storage configurations; furthermore, we show that a significant repair traffic reduction is possible. First, the fundamental trade-off between the repair bandwidth and the storage size for functional repair is established. Using a graph-theoretic formulation, the optimal tradeoff is identified as the solution to an integer optimization problem, for which a closed-form expression is derived. Expressions of the extreme points, namely the minimum storage multi-node repair (MSMR) and minimum bandwidth multi-node repair (MBMR) points, are obtained. Second, we describe a general framework for converting single erasure minimum storage regenerating codes to MSMR codes. The repair strategy for $e$ failures is similar to that for a single failure; however, certain extra requirements need to be satisfied by the repairing functions for a single failure. For illustration, the framework is applied to product-matrix codes and interference alignment codes. Furthermore, we prove that the functional MBMR point is not achievable for linear exact-repair codes. We also show that the exact-repair minimum bandwidth cooperative repair codes achieve an interior point, that lies near the MBMR point, when $k \equiv 1 \mod e$ , $k$ being the minimum number of nodes needed to reconstruct the entire data. Finally, for $k> 2e, e\mid k$ , and $e \mid d$ , where $d$ is the number of helper nodes during repair, we show that the functional repair trade-off is not achievable under exact repair, except for maybe a small portion near the MSMR point, which parallels the results for single-erasure repair by Shah et al. [ABSTRACT FROM AUTHOR]

For locally repairable codes (LRCs), Cadambe and Mazumdar derived the first field-dependent parameter bound, known as the C-M bound. However, the C-M bound depends on an undetermined parameter $ {k}^{({q})}_{{\textbf {opt}}}({n}, {d})$. In this paper, a sphere-packing approach is developed for upper bounding the parameter $ {k}$ for $[ {n}, {k}, {d}]$ linear LRCs with locality $ {r}$. When restricted to the binary field, three upper bounds (i.e., Bound A, Bound B, and Bound C) are derived in an explicit form. More specifically, Bound A holds under the hypothesis that the local repair groups are disjoint and of equal size. Comparing with previous bounds obtained under the same hypothesis, Bound A either covers them as special cases or has an advantage due to its explicit form. Then, the hypothesis is removed in Bound B and Bound C. As the price for explicit form, Bound B specially holds for $ {d}\geq \text {5}$ and Bound C for $ {r}=\textbf {2}$. Through specific comparisons, we show that Bound B and Bound C both tend to outperform the C-M bound, as $ {n}$ goes large. Moreover, a family of binary linear LRCs with $ {d}\geq \textbf {6}$ attaining Bound B are constructed and later extended to a wider range of parameters by a shortening technique. Lastly, most of the bounds and constructions are extended to $ {q}$ -ary LRCs. [ABSTRACT FROM AUTHOR]

Regenerating codes are efficient methods for distributed storage in storage networks, where node failures are common. They guarantee low cost data reconstruction and repair through accessing only a predefined number of arbitrarily chosen storage nodes in the network. In this paper, we consider two simultaneous extensions to the original regenerating codes framework introduced by Dimakis et al.; 1) both data reconstruction and repair are resilient to the presence of a certain number of erroneous nodes in the network and 2) the number of helper nodes in every repair is not fixed, but is a flexible parameter that can be selected during the run-time. We study the fundamental limits of required total repair bandwidth and provide an upper bound for the storage capacity of these codes under these assumptions. We then focus on the minimum repair bandwidth (MBR) case and derive the exact storage capacity by presenting explicit coding schemes with exact repair, which achieve the upper bound of the storage capacity in the considered setup. To this end, we first provide a more natural extension of the well-known product matrix (PM) MBR codes, modified to provide flexibility in choosing the number of helpers in each repair, and simultaneously be robust to erroneous nodes in the network. This is achieved by proving the non-singularity for a family of matrices in large enough finite fields. We next provide another extension of the PM codes, based on a novel repair scheme which enables flexibility in the number of helpers and robustness against erroneous nodes without any extra cost in field size compared with the original PM codes. [ABSTRACT FROM AUTHOR]

In recent years, wireless communication demands from high-speed vehicles have rapidly increased, due to the enrichment of various emerging vehicular applications including road safety, intelligent transportation, in-vehicle entertainment, self-driving, etc. Such high communication demands may result in significant burden on the supporting infrastructure in vehicular networks, which, in turn, deteriorate the quality of experience of vehicular users. Caching popular contents at the edge of the network attracting increasing research interests lately, however, is significantly hindered by the vehicle mobility in vehicular scenarios. To cope with such an issue, this paper explores and proposes a judiciously designed framework termed as IV-Cache to facilitate in-vehicle caching in the vehicular scenario that specifically accounts for the vehicle mobility and vehicle-to-vehicle (V2V) communications vulnerability. At the foundation of this framework, we innovatively propose a storage scheme enhanced with dynamic relays among vehicles to account for the vehicle mobility. The key of the proposed data storage scheme is to maintain the storage data survival in a designated region of interest. Data redundancy by erasure codes is also introduced in order to combat the volatile V2V links. Based on dynamic distributed storage relay (D $^2$ SR), our designed IV-Cache can also flexibly accommodate multi-file popularity. Theoretical analysis and numerical results validate the effectiveness of our proposed D $^2$ SR and IV-Cache schemes. Specifically, the IV-Cache system could serve up to $\text{95}\%$ of the requests for the cached popular contents, and the data stored at the D $^2$ SR system could survive for up to tens of hours under typical road conditions. [ABSTRACT FROM AUTHOR]

Locally repairable codes are desirable for distributed storage systems to improve the repair efficiency. In this paper, a connection between locally repairable codes with multiple disjoint repair sets and packings is derived under the condition that each repair set contains exactly one check symbol. Particularly, conditions under which an optimal locally repairable code corresponds to a packing are also characterized. As an application of this connection, some optimal locally repairable codes can be obtained by packings. Specifically, two constructions of locally repairable codes are proposed which not only generalize some known explicit constructions but also give optimal locally repairable codes with flexible new parameters. [ABSTRACT FROM AUTHOR]

Regenerating codes are a class of distributed storage codes that allow for efficient repair of failed nodes, as compared to traditional erasure codes. An [n, k, d] regenerating code permits the data to be recovered by connecting to any k of the n nodes in the network, while requiring that a failed node be repaired by connecting to any d nodes. The amount of data downloaded for repair is typically much smaller than the size of the source data. Previous constructions of exact-regenerating codes have been confined to the case n=d+1. In this paper, we present optimal, explicit constructions of (a) Minimum Bandwidth Regenerating (MBR) codes for all values of [n, k, d] and (b) Minimum Storage Regenerating (MSR) codes for all [n, k, d\geq 2k-2], using a new product-matrix framework. The product-matrix framework is also shown to significantly simplify system operation. To the best of our knowledge, these are the first constructions of exact-regenerating codes that allow the number n of nodes in the network, to be chosen independent of the other parameters. The paper also contains a simpler description, in the product-matrix framework, of a previously constructed MSR code with [n=d+1, k, d\geq 2k-1]. [ABSTRACT FROM PUBLISHER]

For the successful development of the astrophysics and, accordingly, for obtaining more complete knowledge of the Universe, it is extremely important to combine and comprehensively analyze information of various types (e.g., about charged cosmic particles, gamma rays, neutrinos, etc.) obtained by using divers large-scale experimental setups located throughout the world. It is obvious that all kinds of activities must be performed continually across all stages of the data life cycle to help support effective data management, in particular, the collection and storage of data, its processing and analysis, refining the physical model, making preparations for publication, and data reprocessing taking refinement into account. In this paper we present a general approach to construction and the architecture of a system to be able to collect, store, and provide users' access to astrophysical data. We also suggest a new approach to the construction of a metadata registry based on the blockchain technology. [ABSTRACT FROM AUTHOR]

Modern distributed storage systems offer large capacity to satisfy the exponentially increasing need of storage space. They often use erasure codes to protect against disk and node failures to increase reliability, while trying to meet the latency requirements of the applications and clients. This paper provides an insightful upper bound on the average service delay of such erasure-coded storage with arbitrary service time distribution and consisting of multiple heterogeneous files. Not only does the result supersede known delay bounds that only work for a single file or homogeneous files, it also enables a novel problem of joint latency and storage cost minimization over three dimensions: selecting the erasure code, placement of encoded chunks, and optimizing scheduling policy. The problem is efficiently solved via the computation of a sequence of convex approximations with provable convergence. We further prototype our solution in an open-source cloud storage deployment over three geographically distributed data centers. Experimental results validate our theoretical delay analysis and show significant latency reduction, providing valuable insights into the proposed latency–cost tradeoff in erasure-coded storage. [ABSTRACT FROM PUBLISHER]

The newly presented $(k+m,k)$ Hadamard minimum storage regenerating (MSR) codes are a class of high rate storage codes with optimal repair property for single node failure. In this paper, we propose a new simple optimal repair strategy for $(k+m,k)$ Hadamard MSR codes, which can considerably reduce the computation compared with the original one during the node repair. [ABSTRACT FROM PUBLISHER]

Regenerating codes represent a class of block codes applicable for distributed storage systems. The $[n,k,d]$ regenerating code has data recovery capability while possessing arbitrary $k$ out of $n$ code fragments, and supports the capability for code fragment regeneration through the use of other arbitrary $d$ fragments, for $k\leq d\leq n-1$ . Minimum storage regenerating (MSR) codes are a subset of regenerating codes containing the minimal size of each code fragment. The first explicit construction of MSR codes that can perform exact regeneration (named exact-MSR codes) for $d\geq 2k-2$ has been presented via a product-matrix framework. This paper addresses some of the practical issues on the construction of exact-MSR codes. The major contributions of this paper include as follows. A new product-matrix framework is proposed to directly include all feasible exact-MSR codes for $d\geq 2k-2$ . The mechanism for a systematic version of exact-MSR code is proposed to minimize the computational complexities for the process of message-symbol remapping. Two practical forms of encoding matrices are presented to reduce the size of the finite field. [ABSTRACT FROM AUTHOR]

A significant amount of research on using erasure coding for distributed storage has focused on reducing the amount of data that needs to be transferred to replace failed nodes. This continues to be an active topic as the introduction of faster storage devices looks to put an even greater strain on the network. However, with a few notable exceptions, most published work assumes a flat, static network topology between the nodes of the system. We propose a general framework to find the lowest cost feasible repairs in a more realistic, heterogeneous and dynamic network, and examine how the number of repair strategies to consider can be reduced for three distinct erasure codes. We devote a significant part of the paper to determining the set of feasible repairs for random linear network coding (RLNC) and describe a system of efficient checks using techniques from the arsenal of dynamic programming. Our solution involves decomposing the problem into smaller steps, memorizing, and then reusing intermediate results. All computationally intensive operations are performed prior to the failure of a node to ensure that the repair can start with minimal delay, based on up-to-date network information. We show that all three codes benefit from being network aware and find that the extra computations required for RLNC can be reduced to a viable level for a wide range of parameter values. [ABSTRACT FROM AUTHOR]

Microgrids are key components of future smart grids, which integrate distributed renewable energy generators to efficiently serve the load locally. However, the intermittent nature of renewable energy generations hinders the reliable operation of microgrids. Besides the commonly adopted methods such as deploying energy storage system (ESS) and supplementary fuel generator to address the intermittency issue, energy cooperation among microgrids by enabling their energy exchange for sharing is an appealing new solution. In this paper, we consider the energy management problem for two cooperative microgrids each with individual renewable energy generator and ESS. First, by assuming that the microgrids’ renewable energy generation/load amounts are perfectly known ahead of time, we solve the off-line energy management problem optimally. Based on the obtained solution, we study the impacts of microgrids’ energy cooperation and their ESSs on the total energy cost. Next, inspired by the off-line optimization solution, we propose online algorithms for the real-time energy management of the two cooperative microgrids. It is shown via simulations that the proposed online algorithms perform well in practice, have low complexity, and are also valid under arbitrary realizations of renewable energy generations/loads. Finally, we present one method to extend our proposed online algorithms to the general case of more than two microgrids based on a clustering approach. [ABSTRACT FROM PUBLISHER]

Erasure code-based distributed storage systems are increasingly being used by storage providers for big data storage since they offer the same reliability as replication with a significant decrease in the amount of storage required. But, when it comes to a storage system with data nodes spread across a very large geographical area, the node’s recovery performance is affected by various factors that are both network and computation related. In this paper, we present a XOR-based code supplemented with the ideas of parity duplication and rack awareness that could be adopted in such storage clusters to improve the recovery performance during node failures and compare it with popular implementations of erasure codes, namely Facebook’s Reed-Solomon codes and XORBAS local recovery codes. The code performance along with the proposed ideas are evaluated on a geo-diverse cluster deployed on the NeCTAR research cloud. We also present a scheme for intelligently placing blocks of coded storage depending on the design of the code, inspired by local reconstruction codes. The sum of all these propositions could offer a better solution for applications that are deployed on coded storage systems that are geographically distributed, in which storage constraints make triple replication not affordable, at the same time ensuring minimal recovery time is a strict requirement. [ABSTRACT FROM AUTHOR]

A distributed storage system must support efficient access to stored data while ensuring recovery of temporally unavailable nodes. Another important aspect of a distributed storage system is security. In this paper, we bring these features together and investigate the problem of efficient access to stored data in presence of a passive eavesdropper with access to limited number of nodes. The access efficiency is measured in two different terms, namely, the number of accessed nodes and the volume of generated network traffic. These quantities possess a natural connection to locality and repair bandwidth in distributed storage system. For each of them we derive bounds on parameters and provide explicit constructions based on maximum distance separable codes. Motivated by practical perspectives we propose the techniques to ensure the same workload on each node as well as constructions over small fields based on subfield subcodes, Euclidean geometry codes and Reed-Muller codes. Finally, we derive an asymptotic random coding bound on parameters of a secure distributed storage system and propose further research directions. [ABSTRACT FROM AUTHOR]

Abstract Integration of electric vehicles (EVs) into the smart grid has attracted considerable interest from researchers, governments, and private companies alike. Such integration may bring problems if not conducted well, but EVs can be also used by utilities and other industry stakeholders to enable the smart grid. This paper presents a systematic literature review of the topic and offer a research framework to guide future research and enrich the body of knowledge. The systematic literature review presented in this paper does not contain all the material available on this subject. It does, however, include most of the key publications readily available in a power-utility or technical-reference library together with some of the earlier papers in the field (the anchor papers). For this review, we selected appropriate digital sources (digital libraries and indexing systems; IEEE Xplore and Web of Science), determined the search terms, and conducted a broad automated search. This article also details the components of the research theme—EV integration into the smart grid—as well as its accompanying use cases. The analysis of the relevant papers indicated four types of key research concerns: power-grid, power-system, and smart-grid reliability and the impacts of changes on them. These results can help guide future research to further smart-grid development. Future research can also expand the reach of this research to address its limitations in scope and depth.

In the literature, all the known high-rate MDS codes with the optimal repair bandwidth possess a significantly large sub-packetization level, which may prevent the codes to be implemented in practical systems. To build MDS codes with small sub-packetization level, existing constructions and theoretical bounds imply that one may sacrifice the optimality of the repair bandwidth. Partly motivated by the work of Tamo et al. (IEEE Trans. Inform. Theory, 59(3), 1597–1616, 2013), in this paper, we present a transformation that can greatly reduce the sub-packetization level of MDS codes with the optimal repair bandwidth with respect to the same code length n. As applications of the transformation, four high-rate MDS codes having both small sub-packetization level and near-optimal repair bandwidth can be obtained, where three of them are explicit and the required field sizes are around or even smaller than the code length ${n}$. Additionally, we propose another explicit MDS code which has a similar structure as that of the first resultant code obtained by the generic transformation, but can be built on a smaller finite field. [ABSTRACT FROM AUTHOR]

With the rapid development of Internet of Things (IoT), more and more devices are connected to IoT, leading to an exponential increase in IoT data. However, two problems stand in the way of the further development of IoT. First, IoT data is uploaded directly to the cloud, and the service provider obtains huge profits by analyzing IoT data. However, users can't get benefits, and they have to bear the risk of privacy leakage. Second, centralized storage limits data sharing, which can slow the development of IoT. To solve these problems, this paper proposes an architecture of Blockchain-based Distributed IoT Data Transaction (BDDT). In our proposed BDDT system, users can get benefits by selling their own data, and the service providers buy users' data to gain access. Furthermore, all operations of IoT data transactions are recorded in blockchain. It ensures the traceability and irreversible modification of data transactions. Final, the experimental results prove that the BDDT system has a good performance and excellent safety. [ABSTRACT FROM AUTHOR]

In recent years, many researchers have focused on developing a feasible solution for storing and exchanging medical images in the field of health care. Current practices are deployed on cloud-based centralized data centers, which increase maintenance costs, require massive storage space, and raise privacy concerns about sharing information over a network. Therefore, it is important to design a framework to enable sharing and storing of big medical data efficiently within a trustless environment. In the present paper, we propose a novel proof-of-concept design for a distributed patient-centric image management (PCIM) system that is aimed to ensure safety and control of patient private data without using a centralized infrastructure. In this system, we employed an emerging Ethereum blockchain and a distributed file system technology called Inter-Planetary File System (IPFS). Then, we implemented an Ethereum smart contract called the patient-centric access control protocol to enable a distributed and trustworthy access control policy. IPFS provides the means for decentralized storage of medical images with global accessibility. We describe how the PCIM system architecture facilitates the distributed and secured patient-centric data access across multiple entities such as hospitals, patients, and image requestors. Finally, we deployed a smart contract prototype on an Ethereum testnet blockchain and evaluated the proposed framework within the Windows environment. The evaluation results demonstrated that the proposed scheme is efficient and feasible. [ABSTRACT FROM AUTHOR]

Recently Reed-Solomon (RS) codes were shown to possess a repair scheme that supports repair of failed nodes with optimal repair bandwidth. In this paper, we extend this result in two directions. First, we propose a new repair scheme for the RS codes constructed in [Tamo-Ye-Barg, IEEE Transactions on Information Theory, vol. 65, May 2019] and show that repair is robust to erroneous information provided by the helper nodes while maintaining the optimal repair bandwidth. Second, we construct a new family of RS codes with optimal access for the repair of any single failed node. We also show that the constructed codes can accommodate both features, supporting optimal-access repair with optimal error-correction capability. Going beyond RS codes, we also prove that any scalar MDS code with repair bandwidth attaining the cutset bound affords a repair scheme with optimal access property. [ABSTRACT FROM AUTHOR]

The problem of $X$ -secure $T$ -private information retrieval from MDS coded storage is studied in this paper, where the user wishes to privately retrieve one out of $K$ independent messages that are distributed over $N$ servers according to an MDS code. It is guaranteed that any group of up to $X$ colluding servers learn nothing about the messages and that any group of up to $T$ colluding servers learn nothing about the identity of desired message. A lower bound of achievable rates is proved by presenting a novel scheme based on cross-subspace alignment and a successive decoding with interference cancellation strategy. For large number of messages $(K\rightarrow \infty)$ the achieved rate, which we conjecture to be optimal, improves upon the best known rates previously reported in the literature by Raviv and Karpuk, and generalizes an achievable rate for MDS-TPIR previously found by Freij-Hollanti et al. that is also conjectured to be asymptotically optimal. The setting is then expanded to allow unresponsive and Byzantine servers. Finally, the scheme is applied to find a new lower convex hull of (download, upload) pairs of secure and private distributed matrix multiplication that generalizes, and in certain asymptotic settings strictly improves upon the best known previous results. [ABSTRACT FROM AUTHOR]

Locally repairable codes (LRCs) and codes with hierarchical locality (H-LRCs) have a great significance due to their applications in distributed storage systems. Constructing LRCs or H-LRCs achieving the Singleton-type bound is a challenging task and has received much attention. In this paper, we make use of optimal LRCs to construct a family of optimal H-LRCs. For this purpose we first construct a new class of optimal LRCs via generalized Reed-Solomon codes (GRS codes). Next, based on these optimal LRCs we present some new optimal H-LRCs, of which the parameters are more flexible than those given in and. [ABSTRACT FROM AUTHOR]

An (n,k,l) maximum distance separable (MDS) array code of length n , dimension k=n-r , and sub-packetization l is formed of l\times n matrices over a finite field F , with every column of the matrix stored on a separate node in the distributed storage system and viewed as a coordinate of the codeword. Repair of a failed node (recovery of one erased column) can be performed by accessing a set of d\le n-1 surviving (helper) nodes. The code is said to have the optimal access property if the amount of data accessed at each of the helper nodes meets a lower bound on this quantity. For optimal-access MDS codes with . In our previous work (IEEE Trans. Inf. Theory, vol. 63, no. 4, 2017), for any n and r , we presented an explicit construction of optimal-access MDS codes with sub-packetization . In this paper, we take up the question of reducing the sub-packetization value l to make it to approach the lower bound. We construct an explicit family of optimal-access codes with l=r^\lceil n/r\rceil , which differs from the optimal value by at most a factor of r^2 . These codes can be constructed over any finite field $F$ as long as $|F|\ge r\lceil n/r\rceil $ , and afford low-complexity encoding and decoding procedures. We also define a version of the repair problem that bridges the context of regenerating codes and codes with locality constraints (LRC codes), which we call group repair with optimal access. In this variation, we assume that the set of $n=sm$ nodes is partitioned into $m$ repair groups of size $s$ , and require that the amount of accessed data for repair is the smallest possible whenever the $d=s+k-1$ helper nodes include all the other $s-1$ nodes from the same group as the failed node. For this problem, we construct a family of codes with the group optimal access property. These codes can be constructed over any field $F$ of size $|F|\ge n$ , and also afford low-complexity encoding and decoding procedures. [ABSTRACT FROM PUBLISHER]

Storage quality-of-service (QoS) is a key issue for a storage cloud infrastructure. This paper presents QoSC, a QoS-aware indiscriminate volume Storage Cloud over the dynamic network, based on the Hadoop distributed file system. QoSC employs a data redundancy policy based on indiscriminate recovery volumes and a QoS-aware data placement strategy. We consider the QoS of a storage node as a combination of the transfer bandwidth, the availability of service, the workload (CPU utilization), the free storage space, and the failure rate of DataNodes. We have deployed QoSC on the campus network of Zhejiang University and have conducted a group of experiments and simulations on file storage and retrieval. The results show that QoSC improves the performance of file storage and retrieval and balances the workload among DataNodes, by being aware of QoS of DataNodes. Copyright © 2016 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

A new real-time optimal scheduling model is proposed and analyzed in this paper to aggregate storage benefits for a large-scale electricity consumer. The complete model for optimal operation of storage-based electrical loads considering both the capital and operating expenditures of storage is developed. A real-time load forecaster is incorporated into the optimal scheduling algorithm using soft constraints, slack variables, and penalizing mechanisms. The application of the proposed model to a real-world large-scale electricity consumer is examined and compared with previous models. It is demonstrated that the proposed model outperforms prior models by generating higher profitability of investment in storage, lower storage operating expenditure, and an extended life of the storage plant. [ABSTRACT FROM PUBLISHER]

Maximum distance separable (MDS) codes are optimal error-correcting codes in the sense that they provide the maximum failure tolerance for a given number of parity nodes. Suppose that an MDS code with $k$ information nodes and r=n-k$ parity nodes is used to encode data in a distributed storage system. It is known that if h$ out of the n$ nodes are inaccessible and d$ surviving (helper) nodes are used to recover the lost data, then we need to download at least h/(d+h-k)$ fraction of the data stored in each of the helper nodes (Dimakis et al., 2010 and Cadambe et al., 2013). If this lower bound is achieved for the repair of any h$ erased nodes from any d$ helper nodes, we say that the MDS code has the (h,d)$ -optimal repair property. We study high-rate MDS array codes with the optimal repair property (also known as minimum storage regenerating codes, or MSR codes). Explicit constructions of such codes in the literature are only available for the cases where there are at most three parity nodes, and these existing constructions can only optimally repair a single node failure by accessing all the surviving nodes. In this paper, given any r$ and n$ , we present two explicit constructions of MDS array codes with the (h,d) . The encoding, decoding, repair of failed nodes, and update procedures of these codes all have low complexity. Codes in the second family have the optimal access property and can be constructed over any base field $F$ as long as $|F|\ge n+1$ . Moreover, both code families have the optimal error resilience capability when repairing failed nodes. We also construct several other related families of MDS codes with the optimal repair property. [ABSTRACT FROM PUBLISHER]

We consider non-binary product codes with MDS components and their iterative row-column algebraic decoding on the erasure channel. Both independent and block erasures are considered in this paper. A compact graph representation is introduced on which we define double-diversity edge colorings via the rootcheck concept. An upper bound of the number of decoding iterations is given as a function of the graph size and the color palette size M . Then, we propose a differential evolution edge coloring algorithm that produces colorings with a large population of minimal rootcheck order symbols. The complexity of this algorithm per iteration is o(M^{\aleph }) , for a given differential evolution parameter \aleph , where M^\aleph itself is small with respect to the huge cardinality of the coloring ensemble. Stopping sets of a product code are defined in the context of MDS components and a relationship is established with the graph representation. A full characterization of these stopping sets is given up to a size (d+1)^2 , where $d$ is the minimum Hamming distance of the MDS component code. The performance of MDS-based product codes with and without double-diversity coloring is analyzed in presence of both the block and the independent erasures. In the latter case, ML and iterative decoding are proven to coincide at small channel erasure probability. Furthermore, numerical results show excellent performance in presence of unequal erasure probability due to double-diversity colorings. [ABSTRACT FROM PUBLISHER]

The arrival of the era of big data for Earth observation (EO) indicates that traditional data management models have been unable to meet the needs of remote sensing data in big data environments. With the launch of the first remote sensing satellite, the volume of remote sensing data has also been increasing, and traditional data storage methods have been unable to ensure the efficient management of large amounts of remote sensing data. Therefore, a professional remote sensing big data integration method is sorely needed. In recent years, the emergence of some new technical methods has provided effective solutions for multi-source remote sensing data integration. This paper proposes a multi-source remote sensing data integration framework based on a distributed management model. In this framework, the multi-source remote sensing data are partitioned by the proposed spatial segmentation indexing (SSI) model through spatial grid segmentation. The designed complete information description system, based on International Organization for Standardization (ISO) 19115, can explain multi-source remote sensing data in detail. Then, the distributed storage method of data based on MongoDB is used to store multi-source remote sensing data. The distributed storage method is physically based on the sharding mechanism of the MongoDB database, and it can provide advantages for the security and performance of the preservation of remote sensing data. Finally, several experiments have been designed to test the performance of this framework in integrating multi-source remote sensing data. The results show that the storage and retrieval performance of the distributed remote sensing data integration framework proposed in this paper is superior. At the same time, the grid level of the SSI model proposed in this paper also has an important impact on the storage efficiency of remote sensing data. Therefore, the remote storage data integration framework, based on distributed storage, can provide new technical support and development prospects for big EO data. [ABSTRACT FROM AUTHOR]

To achieve high data availability or reliability in an efficient manner, distributed storage systems must detect whether an observed node failure is permanent or transient, and if necessary, generate replicas to restore the desired level of replication. Given the unpredictability of network dynamics, however, distinguishing permanent and transient failures is extremely difficult. Though timeout-based detectors can be used to avoid mistaking transient failures as permanent failures, it is unknown how the timeout values should be selected to achieve a better tradeoff between detection latency and accuracy. In this paper, we address this fundamental tradeoff from several perspectives. First, we explore the impact of different timeout values on maintenance cost by examining the probability of their false positives and false negatives. Second, we propose a self-configurable failure detector called the Neutralizer based on the idea of counteracting false positives with false negatives. The Neutralizer could enable the system to maintain a desired replication level on average with the least amount of bandwidth. We conduct extensive simulations using real trace data from a widely deployed peer-to-peer system and synthetic traces based on PlanetLab and Microsoft PCs, showing a significant reduction in aggregate bandwidth usage after applying the Neutralizer (especially in an environment with a low average node availability). Overall, we demonstrate that the Neutralizer closely approximates the performance of a perfect ‘oracle’ detector in many cases. Copyright © 2008 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Deferrable power-driven demands such as water and thermal ones possess a capability of energy storage which can be exploited to further optimize electric systems. This paper proposes a new optimized scheduling framework for electric grids by simultaneously exploiting the high-inertia thermal dynamics and lossless water storage. Unlike previous ideas in the literature, both water and thermal demands are managed to realize the long-term and short-term load-shifting strategies, respectively. First, distributed water/thermal loads are aggregated where the virtual storage can be represented by battery models. Then, a further optimized scheduling framework is proposed with a mixed-integer nonlinear programing problem. Comparative studies show that the water storage offers outstanding flexibility for electric system via pumps scheduling, especially for the electric load-shifting strategy in a long time-horizon thanks to the lossless water-storage process. Meanwhile, the thermal storage can directly support short-term electric load-shifting to avoid price spikes of electricity. Numerical results show that the proposed method can reduce the total cost of micro-grids by maximizing the usage of renewable energy sources, avoiding price spikes, and reducing dependence on high-cost centralized energy-storage facilities provided that the critical water-energy demands are preserved. [ABSTRACT FROM AUTHOR]

In this paper, locally repairable codes which have optimal minimum Hamming distance with respect to the bound presented by Prakash et al. are considered. New upper bounds on the length of such optimal codes are derived. The new bounds apply to more general cases, and have weaker requirements compared with the known ones. In this sense, they both improve and generalize previously known bounds. Further, optimal codes are constructed, whose length is order-optimal with respect to the new upper bounds. Notably, the length of the codes is super-linear in the alphabet size. [ABSTRACT FROM AUTHOR]

To ensure data availability and save storage space, storage systems usually save data across multiple storage nodes (or servers) using erasure codes. Storage systems need to reconstruct the complete data to respond to reading requests in the case of the loss of some data blocks when node failure occurs. However, a degraded read in erasure code-based storage systems does not fully utilize node resources and ignores the node's topology. In this paper, we propose a real-time performance evaluation model for storage nodes to evaluate the performance of each node combining a metrics choice and an analytic hierarchy process. We also design a cost evaluation method to calculate the transmission cost by considering the node's topology. By combining the node evaluation method and a distance calculation, we propose an adaptive degraded read optimization strategy, NADE. We further implement the node selection method NADE in Ceph. The evaluation results show the efficiency of the proposed method. [ABSTRACT FROM AUTHOR]

The multi-version coding problem is a recently formulated information-theoretic framework to study the storage cost of consistent key-value data stores. Previous work on multi-version coding considered a completely decentralized asynchronous system where the nodes (servers) are not aware of which updates (versions) of the data are received by the other nodes. In this paper, we relax this assumption and study a system where a node acquires side information of the versions propagated to some other nodes based on the network topology. Specifically, we study a storage system with $n$ nodes over a graph that stores $\nu $ totally ordered versions of an object (message). Each node receives a subset of these $\nu $ versions. A node is aware of which versions that were received by its neighbors in the network graph. Our code constructions show that the side information can result in a better storage cost as compared with the case where the nodes do not exchange side information for some regimes at the expense of the additional latency and the negligible communication overhead of exchanging the side information. Through an information-theoretic converse, we identify surprising scenarios where exchanging tremendous amount of side information does not reduce the storage cost. Finally, we present a case study over Amazon web services (AWS) that demonstrates the potential storage cost reductions of our code constructions. [ABSTRACT FROM AUTHOR]

By introducing several levels of recoverability for locally recoverable codes (LRCs), LRCs with hierarchical locality (H-LRCs) are defined for correcting different numbers of erasures. There are two major ingredients in this paper. The first is to investigate some properties and existence conditions of optimal H-LRCs with respect to a generalized Singleton-like bound for H-LRCs. The second ingredient is to propose several constructions of optimal H-LRCs by employing cyclic codes. Notably, the parameters of these optimal H-LRCs are flexible. [ABSTRACT FROM AUTHOR]

Locally repairable codes are desirable for distributed storage systems to improve the repair efficiency. In this paper, a new combination of codes with locality and codes with multiple disjoint repair sets (also called availability) is introduced. Accordingly, a Singleton-type bound is derived for the new code, which contains those bounds in , , as special cases. Optimal constructions are proposed with respect to the new bound. In addition, these constructions can also generate optimal codes with multiple disjoint repair sets with respect to the bound in , which to the best of our knowledge, are the first explicit constructions that can achieve the bound in. [ABSTRACT FROM AUTHOR]

Content-based image retrieval poses many problems to computer systems. The content of images has to be described by some feature extraction methods. As image databases are often very large, they are sometimes to complex to be processed by traditional computing methods. We have to use big data solutions to fast retrieve images. The paper presents a system for retrieving images in relational databases in a distributed environment. Content of the query image and images in the database is compared using global colour information and local image keypoints. Image keypoint descriptors are indexed by fuzzy sets directly in a relational database by our algorithm. The process is distributed to several machines thanks to the Apache Hadoop software framework with HDFS. [ABSTRACT FROM AUTHOR]

In distributed storage systems, codes with lower repair locality for each coordinate are much more desirable since they can reduce the disk I/O complexity for repairing a failed node. The ith coordinate of a linear code 풞 is said to have locality if there exist δi non-overlapping local repair sets of size no more than ri, where a local repair set of one coordinate is defined as the set of some other coordinates by which one can recover the value at this coordinate. In this paper, we consider linear codes with information locality, where there exists an information set I such that for each , the ith coordinate has locality and and . We derive a lower bound on the codeword length n for any linear [ n, k, d] code with information locality. Particularly, we indicate that some existing bounds can be deduced from our result by restrictions on parameters. [ABSTRACT FROM AUTHOR]

A variety of wireless networks, including applications of Wireless Sensor Networks, Internet of Things, and Cyber-physical Systems, increasingly pervade our homes, retail, transportation systems, and manufacturing processes. Traditional approaches communicate data from all sensors to a central system, and users (humans or machines) query this central point for results, typically via the web. As the number of deployed sensors, and thus generated data streams, is increasing exponentially, this traditional approach may no longer be sustainable or desirable in some application contexts. Therefore, new approaches are required to allow users to directly interact with the network, for example, requesting data directly from sensor nodes. This is difficult, as it requires every node to be capable of point-to-point routing, in addition to identifying a subset of nodes that can fulfil a user's query. This paper presents Dragon, a platform that allows any node in the network to identify all nodes that satisfy user queries, i.e., request data from nodes, and relay the result to the user. The Dragon platform achieves this in a fully distributed way. No central orchestration is required, network overheads are low, and latency is improved over existing comparable methods. Dragon is evaluated on networks of various topologies and different network densities. It is compared with the state-of-the-art algorithms based on summary trees, like Innet and SENS-Join. Dragon is shown to outperform these approaches up to 88% in terms of network traffic required, also a proxy for energy efficiency, and 84% in terms of processing delay. [ABSTRACT FROM PUBLISHER]

Meteorological technology has evolved rapidly in recent years to provide enormous, accurate and personalized advantages in the public service. Large volumes of observational data are generated gradually by technologies such as geographical remote sensing, meteorological radar satellite, etc. that makes data analysis in weather forecasting more precise but also poses a threat to the traditional method of data storage. In this paper, we present MHBase, (Meteorological data based on HBase (Hadoop Database), a distributed real-time query scheme for meteorological data based on HBase. The calibrated data obtained from terminal devices will be partitioned into HBase and persisted to HDFS (the Hadoop Distributed File System). We propose two algorithms (the Indexed Store and the Indexed Retrieve Algorithms) to implement a secondary index using HBase Coprocessors, which allow MHbase to provide high performance data querying on columns other than rowkey. Experimental results show that the performance of MHBase can satisfy the basic demands of meteorological business services. [ABSTRACT FROM AUTHOR]