Search

With the development of synchrophasor measurement techniques, ambient signal-based load modeling is recently proposed to track the time-varying characteristics of power loads. On this basis, the hardware realization of this approach, which includes the local load modeling devices and the data center, is proposed and tested with the field measurement data from China Southern Power Grid. In this way, practical load model parameters can be identified from field measurement data both online and offline by the hardware system. To begin with, the hardware system, which is the first realization of an ambient signal-based load modeling algorithm, offers sufficient field measurements for load modeling. Then, the identification results of the field measurements verify the effectiveness of the ambient signal-based load modeling algorithm. Besides, the identified load model parameters for different substations during different times demonstrate the time-varying and the distributed characteristics of power loads. [ABSTRACT FROM AUTHOR]

Cyberspace is not a “vacuum space”, and it is normal that there are inevitable viruses and worms in cyberspace. Cyberspace security threats stem from the problem of endogenous security, which is caused by the incompleteness of theoretical system and technology of the information field itself. Thus it is impossible and unnecessary for us to build an “aseptic” cyberspace. On the contrast, we must focus on improving the “self-immunity” of network. Literally, endogenous security is an endogenous effect from its own structural factors rather than external ones. The $t/k$ t / k -diagnosis strategy plays a very important role in measuring endogenous network security without prior knowledge, which can significantly enhance the self-diagnosing capability of network. As far as we know, few research involves $t/k$ t / k -diagnosis algorithm and $t/k$ t / k -diagnosability of interconnection networks under MM* model. In this article, we propose a fast $f(r,k+1)/k$ f (r , k + 1) / k -diagnosis algorithm of complexity $O(Nr^2)$ O (N r 2) , say $G$ G MIS $k$ k DIAGMM*, for a general $r$ r -regular network $G$ G under MM* model by designing a 0-comparison subgraph $M_0(G)$ M 0 (G) , where $N$ N is the size of $G$ G . We determine that the $t/k$ t / k -diagnosability $(t(G)/k)^M$ (t (G) / k) M of $G$ G under MM* model is $f(r,k+1)$ f (r , k + 1) by $G$ G MIS $k$ k DIAGMM* algorithm. Moreover, we establish the $(t(G)/k)^M$ (t (G) / k) M of some interconnection networks under MM* model, including BC networks, $(n,l)$ (n , l) -star graph networks, and data center network DCells. Finally, we compare $(t(G)/k)^M$ (t (G) / k) M with diagnosability, conditional diagnosability, pessimistic diagnosability, extra diagnosability, and good-neighbor diagnosability under MM* model. It can be seen that $(t(G)/k)^M$ (t (G) / k) M is greater than other fault diagnosabilities in most cases. [ABSTRACT FROM AUTHOR]

Multi-field packet classification is a crucial component in modern software-defined data center networks. To achieve high throughput and low latency, state-of-the-art algorithms strive to fit the rule lookup data structures into on-die caches; however, they do not scale well with the number of rules. We present a novel approach, NuevoMatch, which improves the memory scaling of existing methods. A new data structure, Range Query Recursive Model Index (RQ-RMI), is the key component that enables NuevoMatch to replace most of the accesses to main memory with model inference computations. We describe an efficient training algorithm that guarantees the correctness of the RQ-RMI-based classification. The use of RQ-RMI allows the rules to be compressed into neural networks that fit into the hardware cache. Further, it takes advantage of the growing support for fast neural network processing in modern CPUs, such as wide vector instructions, achieving a latency of tens of nanoseconds per lookup. Our evaluation using 500K multi-field rules from the standard ClassBench benchmark shows a geometric mean compression factor of $4.9\times $ , $8\times $ , and $82\times $ , and average performance improvement of $2.4\times $ , $2.6\times $ , and $1.6\times $ in throughput compared to CutSplit, NeuroCuts, and TupleMerge, all state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Motivated by applications in data center networks, in this paper, we study the problem of scheduling in an input queued switch. While throughput maximizing algorithms in a switch are well-understood, delay analysis was developed only recently. It was recently shown that the well-known MaxWeight algorithm achieves optimal scaling of mean queue lengths in steady state in the heavy-traffic regime, and is within a factor less than 2 of a universal lower bound. However, MaxWeight is not used in practice because of its high time complexity. In this paper, we study several low complexity algorithms and show that their heavy-traffic performance is identical to that of MaxWeight. We first present a negative result that picking a random schedule does not have optimal heavy-traffic scaling of queue lengths even under uniform traffic. We then show that if one picks the best among two matchings or modifies a random matching even a little, using the so-called flip operation, it leads to MaxWeight like heavy-traffic performance under uniform traffic. We then focus on the case of non-uniform traffic and show that a large class of low time complexity algorithms have the same heavy-traffic performance as MaxWeight, as long as it is ensured that a MaxWeight matching is picked often enough. We also briefly discuss the performance of these algorithms in the large scale heavy-traffic regime when the size of the switch increases simultaneously with the load. Finally, we perform empirical study on a new algorithm to compare its performance with some existing algorithms. [ABSTRACT FROM AUTHOR]

This paper presents the first network configuration verifier that provides fast all-pair reachability analysis of incremental configuration changes for network overlay data center networks (DCNs). Network overlay DCNs leverage distributed routing protocol on edge leaf switches to disseminate overlay routes and establish overlay tunnels. In addition, network overlay DCNs use access control lists, microsegmentation policy, policy-based routing and firewall policy to control east-west and north-south traffic. Although some incremental verification approaches have been proposed, they either do not support certain forwarding features of the network, or are not efficient. Our configuration verifier addresses these issues through the following components: 1) a port predicate based forwarding model that is general to support all features; 2) fine-grained association technique to index possibly affected reachable pairs by changed interfaces in the original network; and 3) required waypoint path computation that finds all reachable pairs related to changed interfaces in the new network. Based on these components, our verifier presents two incremental verification algorithms that are specially designed for different service update cases. Experiment results show that our incremental verification algorithms are accurate and fast. For all-pair reachability, our verifier performs change-impact analysis within 15s for networks with 200 leafs (4000 subnets and 16 million pairs), outperforming existing approaches by up to 10x. [ABSTRACT FROM AUTHOR]

The proliferation of Internet-of-Things (IoT) devices and cloud-computing applications over siloed data centers is motivating renewed interest in the collaborative training of a shared model by multiple individual clients via federated learning (FL). To improve the communication efficiency of FL implementations in wireless systems, recent works have proposed compression and dimension reduction mechanisms, along with digital and analog transmission schemes that account for channel noise, fading, and interference. The prior art has mainly focused on star topologies consisting of distributed clients and a central server. In contrast, this paper studies FL over wireless device-to-device (D2D) networks by providing theoretical insights into the performance of digital and analog implementations of decentralized stochastic gradient descent (DSGD). First, we introduce generic digital and analog wireless implementations of communication-efficient DSGD algorithms, leveraging random linear coding (RLC) for compression and over-the-air computation (AirComp) for simultaneous analog transmissions. Next, under the assumptions of convexity and connectivity, we provide convergence bounds for both implementations. The results demonstrate the dependence of the optimality gap on the connectivity and on the signal-to-noise ratio (SNR) levels in the network. The analysis is corroborated by experiments on an image-classification task. [ABSTRACT FROM AUTHOR]

With the explosive growth in the scale of cloud computing infrastructures, reliability and energy efficiency have become important concerns considering the great complexity of cloud data centers. There is an urgent need for efficient task assignment that can dispatch tasks to appropriate cloud data center servers, which is critical to achieve reliability and energy efficiency in current cloud data centers. Most of the research on task assignment focuses on only one of the objectives of reliability and energy efficiency, while the two objectives are intrinsically conflicting with each other. In this paper, we deal with the problem of task assignment in data centers, with the objective of minimizing the energy consumption while providing failure tolerance to task execution failure. We propose a reliability-aware and energy-efficient task replica assignment algorithm based on running task replicas at a low speed and enabling multiple task replicas to share the same server resources. Each task in a job processed by the cloud computing platform has two instances: main task and task replica (shadow). Each main task runs on an individual server, and the task replica associated with the main task is assigned on a different server. The main tasks run at the full server speed, while the task replicas run at a lower rate than the main tasks. The task replicas can be mapped onto dedicated backup servers or be assigned to the servers on which the main tasks are running. Multiple task replicas can share the same server resources to reduce the number of servers required. We conduct experiments through simulations. Experimental results demonstrate that the proposed algorithm can effectively reduce the energy consumption, while achieving a good balance between the number of servers used and job completion time. [ABSTRACT FROM AUTHOR]

Aerosol particles affect the Earth’s radiative balance and represent one of the largest uncertainties in climate research. The removal of clouds is the first step and critical in the aerosol retrievals. However, the cloud detection is still challenging. Here, a novel simplified cloud detection algorithm (SCDA) is proposed to identify the cloud and clear-sky over land and based on a simplified radiative transfer model (RTM). The fewer input bands, dynamic thresholds, and only one parameter to be modified are the main advantages of the algorithm, which can be applied to different satellite sensors. In this article, we apply SCDA to the Himawari-8 data in 2016 for preliminary analysis. The detection results are validated using Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation (CALIPSO) vertical feature mask (VFM) data and the National Centers for Environmental Information (NCEI) ground-based observation data. We also compare the results with the Himawari-8 cloud products from the Japan Aerospace Exploration Agency (JAXA). Compared with CALIPSO VFM data and NCEI ground-based observation data, the correct rate of SCDA cloud detection result is 86.08% and 79.86%, which are higher than that of Himawari-8 cloud products (85.71% and 78.89%). The correct rate of SCDA clear-sky detection result is 88.33% and 87.85%, which are close to the correct rate of Himawari-8 clear-sky products (90.54% and 88.63%). The overall performance of the SCDA is comparable to that of the threshold method for JAXA Himawari-8 cloud products. Therefore, the SCDA can provide accurate cloud mask with only one threshold to be modified and few input parameters. [ABSTRACT FROM AUTHOR]

In hybrid datacenter networks, wired connections are augmented with wireless links to facilitate data transfers between racks. The usage of mmWave/FSO wireless links enables dynamic bandwidth/capacity allocation with extremely small reconfiguration delay. Also, on-demand workload distribution, where the workload of a job is divided into multiple tasks that can be distributed/routed to different racks to be processed in parallel, allows better utilization of computational resources in data centers. In prior work, the dynamic wireless capacity augmentation and workload distribution decisions were mostly made independently and in a heuristic manner for serving distributed and parallel computing jobs. In this paper, we propose a novel analytical framework and algorithms to jointly optimize both the wireless capacity augmentation and the workload distribution, to minimize the job completion time. We consider workload that is not amenable to pipelining, fully amenable to pipelining, and partially amenable to pipelining. With extensive simulation studies, we show that the gain (in terms of the reduction in the job completion time) can be very substantial when allowing such joint optimization. [ABSTRACT FROM AUTHOR]

Cloud computing has attracted both end-users and cloud service providers (CSPs) in recent years. Improving resource utilization rate (RUtR), such as CPU and memory usages on servers, while maintaining quality of service (QoS) is one key challenge faced by CSPs with warehouse-scale datacenters. Prior works proposed various algorithms to reduce energy cost or to improve RUtR, which either lack the fine-grained task scheduling capabilities, or fail to take a comprehensive system model into consideration. This article presents H2O-Cloud, a Hierarchical and Hybrid Online task scheduling framework for warehouse-scale Cloud service providers, to improve resource usage effectiveness while maintaining QoS. H2O-Cloud is highly scalable and considers comprehensive information, such as various workload scenarios, cloud platform configurations, user request information, and dynamic pricing model. The hierarchy and hybridity of the framework, combined with its deep reinforcement learning (DRL) engines, enable H2O-Cloud to efficiently start on-the-go scheduling and learning in an unpredictable environment without pretraining. Our experiments confirm the high efficiency of the proposed H2O-Cloud when compared to baseline approaches, in terms of energy and cost while maintaining QoS. Compared with a state-of-the-art DRL-based algorithm, H2O-Cloud achieves up to 201.17% energy cost efficiency improvement, 47.88% energy efficiency improvement, and 551.76% reward rate improvement. [ABSTRACT FROM AUTHOR]

Cloud computing is developing rapidly and playing an increasingly important role in many fields. Resource management of data center networks (DCNs) is critical to the performance of cloud computing. Due to the large-scale, distributed, and cross-regional characteristics of DCNs, there is insufficient resource utilization. In this article, we discuss the latest developments in the resource management and optimization of DCNs. We also designed an intelligent resource search algorithm to efficiently search excellent resources in massive computing resources. We propose a novel intelligent resource scheduling based on the locality principle algorithm (RSLP). The proposed strategy can search computing resources dynamically and efficiently, and allocate large resources to large tasks efficiently. Furthermore, we outline critical demands and guidance for researchers and designers to achieve the convergence of artificial intelligence and data centers. [ABSTRACT FROM AUTHOR]

The growth of cloud-based services is mainly supported by the core networking infrastructures of large-scale data centers, while the scalability of these services is influenced by the performance and dependability characteristics of data centers. Hence, the data center network must be agile and reconfigurable in order to respond quickly to the ever-changing application demands and service requirements. The network must also be able to interconnect the big number of nodes, and provide an efficient and fault-tolerant routing service to upper-layer applications. In response to these challenges, the research community began exploring novel interconnect topologies, namely: Flecube, DCell, Ficonn, HyperFlaNet and BCube. However, these topologies either scale too fast (grows exponentially in size), or too slow, and therefore suffer from performance bottlenecks. In this paper, we propose a novel data center topology called LaScaDa (Layered Scalable Data Center) as a new solution for building scalable and cost-effective data center networking infrastructures. The proposed topology organizes nodes in clusters of similar structure, then interconnect these clusters in a well-crafted pattern and system of coordinates for nodes to reduce the number of redundant connections between clusters, while maximizing connectivity. LaScaDa forwards packets between nodes using a new hierarchical row-based routing algorithm. The algorithm constructs the route to the source based on the modular difference between the source and destination coordinates. Furthermore, the proposed topology interconnects a large number of nodes using a small node degree. This strategy increases the number of directly connected clusters and avoids redundant connections. As a result, we get a good quality of nodes in terms of average path length (APL), bisection bandwidth, and aggregated bottleneck throughput. Experimental results show that LaScaDa has better performance than DCell, BCube, and HyperBcube in terms of scalability, while providing a good quality of service. [ABSTRACT FROM AUTHOR]

This article studies edge caching in fog computing networks, where a capacity-aware edge caching framework is proposed by considering both the limited fog cache capacity and the connectivity capacity of base stations (BSs). By allowing cooperation between fog nodes and cloud data center, the average-download-time (ADT) minimization problem is formulated as a multi-class processor queuing process. We prove the convexity of the formulated problem and propose an Alternating Direction Method of Multipliers (ADMM)-based algorithm that can achieve the minimum ADT and converge much faster than existing algorithms. Simulation results demonstrate that the allocation of fog cache capacity and BS connectivity capacity needs to be balanced to take the full advantage of edge caching. While the maximization of the edge-cache-hit-ratio (ECHR) by utilizing all available fog cache capacity is helpful when the BS connectivity capacity is sufficient, it is preferable to keep a lower ECHR and allocate more traffic to the cloud when the BS connectivity capacity is deficient. [ABSTRACT FROM AUTHOR]

MapReduce jobs need to shuffle a large amount of data over the network between mapper and reducer nodes. The shuffle time accounts for a big part of the total running time of the MapReduce jobs. Therefore, optimizing the makespan of shuffle phase can greatly improve the performance of MapReduce jobs. A large fraction of production jobs in data centers are recurring with predictable characteristics, and the recurring jobs split the network into periodic busy and idle time slots, which allows us to better schedule the shuffle data in order to reduce the makespan of shuffle phase with the future predictable network status available. In this paper, we formulate the shuffle scheduling problem with the aim to minimize the makespan of MapReduce shuffle phase by leveraging the predictable periodic network status. We then propose a simple yet effective network-aware shuffle scheduling algorithm (NAS) to reduce the number of idle time slots required to transfer the shuffle data so as to reduce the shuffle makespan. We also prove that the proposed algorithm NAS is a $\frac {3}{2}$ -approximation algorithm to the shuffle scheduling problem when all the future idle time slots have the same duration. We finally conduct experiments through simulations. Experimental results demonstrate the proposed algorithm can effectively reduce the makespan of MapReduce shuffle phase and increase network utilization. [ABSTRACT FROM AUTHOR]

A data-driven smart solid-state transformer (SST) is an attractive building block for reliable and automated distribution power grids. An SST can work as an energy buffer unit in a completely embedded framework, similar to the information internet router, which facilitates a seamless information flow between the local grids and the data centers. An SST connects two or more electric subsystems and then distributes the energy to other subsystems based on a data-driven energy dispatch and intelligent management algorithm. The modern SST-based distribution grid has been envisioned to integrate several renewable energy sources, energy storage devices, and dc or ac loads in a standardized protocol-based plug-n-play technique, which may form a dc or an ac zonal microgrid system. Energy-efficient monitoring, coordination, control management, and economic dispatch are crucial issues for the optimal operation of an SST. This article proposes a data-driven coordinated control strategy for a three-stage multiple active bridge converter-based SST topology, which can ensure improved power quality of the distribution grid. A step-by-step controller design methodology is developed, which can coordinate several control data according to the various SST functionalities. The article presents the simulation and experimental results that validate the effectiveness of the proposed coordinated control approach. [ABSTRACT FROM AUTHOR]

Geo-distributed machine learning (ML) often uses large geo-dispersed data collections produced over time to train global models, without consolidating the data to a central site. In the parameter server architecture, “workers” and “parameter servers” for a geo-distributed ML job should be strategically deployed and adjusted on the fly, to allow easy access to the datasets and fast exchange of the model parameters at any time. Despite many cloud platforms now provide volume discounts to encourage the usage of their ML resources, different geo-distributed ML jobs that run in the clouds often rent cloud resources separately and respectively, thus rarely enjoying the benefit of discounts. We study an ML broker service that aggregates geo-distributed ML jobs into cloud data centers for volume discounts via dynamic online placement and scaling of workers and parameter servers in individual jobs for long-term cost minimization. To decide the number and the placement of workers and parameter servers, we propose an efficient online algorithm which first decomposes the online problem into a series of one-shot optimization problems solvable at each individual time slot by the technique of regularization, and afterwards round the fractional decisions to the integer ones via a carefully-designed dependent rounding method. We prove a parameterized-constant competitive ratio for our online algorithm as the theoretical performance analysis, and also conduct extensive simulation studies to exhibit its close-to-offline-optimum practical performance in realistic settings. [ABSTRACT FROM AUTHOR]

Previous resource provisioning strategies in cloud datacenters allocate physical resources to virtual machines (VMs) based on the predicted resource utilization pattern of VMs. The pattern for VMs of a job is usually derived from historical utilizations of multiple VMs of the job. We observed that these utilization curves are usually misaligned in time, which would lead to resource over-prediction and hence over-provisioning. Since this resource utilization misalignment problem has not been revealed and studied before, in this paper, we study the VM resource utilization from public datacenter traces and Hadoop benchmark jobs to verify the commonness of the utilization misalignments. Then, to reduce resource over-provisioning, we propose three VM resource utilization pattern refinement algorithms to improve the original generated pattern by lowering the cap of the pattern, reducing cap provision duration and varying the minimum value of the pattern. We then extend these algorithms to further improve the resource efficiency by considering periodical resource demand patterns that have multiple pulses in a pattern. These algorithms can be used in any resource provisioning strategy that considers predicted resource utilizations of VMs of a job. We then adopt these refinement algorithms in an initial VM allocation mechanism and test them in trace-driven experiments and real-world testbed experiments. The experimental results show that each improved mechanism can increase resource utilization, and reduce the number of PMs needed to satisfy tenant requests. Also, our extended refinement algorithms are effective in improving resource efficiency of the refinement algorithms. [ABSTRACT FROM AUTHOR]

The emerging deployment of geographically distributed data centers (DCs) incurs a significant amount of data transfers over the Internet. Such transfers are typically charged by Internet service providers with the widely adopted $q$ th percentile charging model. In such a charging model, the time slots with top ( $100-q$ ) percent of data transmission do not affect the total transmission cost and can be viewed as “free.” This brings the opportunity to optimize the scheduling of inter-DC transfers to minimize the entire transmission cost. However, a very little work has been done to exploit those “free” time slots for scheduling inter-DC transfers. The crux is that existing work either lacks a mechanism to accumulate traffic to “free” time slots, or inevitably relies on prior knowledge of future traffic arrival patterns. In this paper, we present TrafficShaper, a new scheduler that shapes the inter-DC traffic to exploit the “free” time slots involved in the $q$ th percentile charging model, so as to reduce or even minimize the transmission cost. When shaping traffic, TrafficShaper advocates a simple principle: more traffic peaks should be scheduled in “free” time slots, while less traffic differentiation should be maintained among the remaining time slots. To this end, TrafficShaper designs a pricing-aware control framework, which makes online decisions for inter-DC transfers without requiring a prior knowledge of traffic arrivals. To verify the performance of TrafficShaper, we conduct rigorous theoretical analysis based on Lyapunov optimization techniques, large-scale trace-driven simulations, and small-scale testbed implementation. Results from rigorous mathematical analyses demonstrate that TrafficShaper can make the transmission cost arbitrarily close to the optimum value. Extensive trace-driven simulation results show that TrafficShaper can reduce the transmission cost by up to 40.23%, compared with the state-of-the-art solutions. The testbed experiments further verify that TrafficShaper can realistically reduce the transmission cost by up to 19.38%. [ABSTRACT FROM AUTHOR]

In cloud environments, the absence of strict network performance guarantees leads to unpredictable job execution times. To address this issue, recently, there have been several proposals on how to provide guaranteed network performance. These proposals, however, rely on computing resource reservation schedules a priori. Unfortunately, this is not practical in today’s cloud environments, where application demands are inherently unpredictable, e.g., due to differences in the input data sets or phenomena, such as failures and stragglers. To overcome these limitations, we designed Kraken, a system that allows to dynamically update minimum guarantees for both network bandwidth and compute resources at runtime. Unlike previous work, Kraken does not require prior knowledge about the resource needs of the applications but allows to modify reservations at runtime. Kraken achieves this through an online resource reservation scheme, which comes with provable optimality guarantees. In this paper, we motivate the need for dynamic resource reservation schemes, present how this is provided by Kraken, and evaluate Kraken via extensive simulations and a preliminary Hadoop prototype. [ABSTRACT FROM PUBLISHER]

The development of cloud computing and data science result in rapid increases of number and scale of data centers. Because of cost and sustainability concerns, energy efficiency has been a major goal for data center architects. Focusing on reducing the cooling power and making full use of available computing power, power budgeting is an increasingly important requirement for data center operations. In this paper, we present a framework of power budgeting, considering both computing power and cooling power, in data centers to maximize the system normalized performance (SNP) of the entire center under a total power budget. Maximizing the SNP for a given power budget is equivalent to maximizing the energy efficiency. We propose a method to partition the total power budget among the cooling and computing infrastructure in a self-consistent way, where the cooling power is sufficient to extract the heat of the computing power. Intertwinedly, we devise an optimal computing power budgeting technique based on dynamic programming algorithm to determine the optimal power caps for the individual servers such that the available power could be efficiently translated to performance improvements. The optimal computing budgeting technique leverages a proposed online throughput predictor based on performance counter measurements to estimate the change in throughput of heterogeneous workloads as a function of allocated server power caps. We demonstrate that our proposed power budgeting method outperforms previous methods by 3-4 percent in terms of SNP using our data center simulation environment. While maintaining the improvement of SNP, our method improve fairness at best by 57 percent. We also evaluate the performance of our method in power saving scenario and dynamic power budgeting case. [ABSTRACT FROM AUTHOR]

Federation of geo-distributed cloud services is a trend in cloud computing that, by spanning multiple data centers at different geographical locations, can provide a cloud platform with much larger capacities. Such a geo-distributed cloud is ideal for supporting large-scale social media applications with dynamic contents and demands. Although promising, its realization presents challenges on how to efficiently store and migrate contents among different cloud sites and how to distribute user requests to the appropriate sites for timely responses at modest costs. These challenges escalate when we consider the persistently increasing contents and volatile user behaviors in a social media application. By exploiting social influences among users, this paper proposes efficient proactive algorithms for dynamic, optimal scaling of a social media application in a geo-distributed cloud. Our key contribution is an online content migration and request distribution algorithm with the following features: 1) future demand prediction by novelly characterizing social influences among the users in a simple but effective epidemic model; 2) one-shot optimal content migration and request distribution based on efficient optimization algorithms to address the predicted demand; and 3) a \Delta(t)-step look-ahead mechanism to adjust the one-shot optimization results toward the offline optimum. We verify the effectiveness of our online algorithm by solid theoretical analysis, as well as thorough comparisons to ready algorithms including the ideal offline optimum, using large-scale experiments with dynamic realistic settings on Amazon Elastic Compute Cloud (EC2). [ABSTRACT FROM PUBLISHER]

In this paper, we investigate the problem of minimizing energy cost for distributed Internet data centers (IDCs) in smart microgrids while taking system dynamics into consideration. Specifically, IDC operators expect to minimize the long-term energy cost with the uncertainties in electricity price, workload, renewable energy generation, and power outage state. At first, we formulate the problem as a stochastic program that captures service request distribution, server provisioning, energy storage management, generator scheduling, power transactions between smart microgrids, and main grids. Second, we use the Lyapunov optimization technique to design an operation algorithm, which enables an explicit tradeoff between energy cost saving and battery investment cost. Finally, the effectiveness of the proposed algorithm is evaluated with practical data. [ABSTRACT FROM PUBLISHER]

Modern stock trading and cluster applications require microsecond latencies and almost no losses in data centers. This paper introduces an algorithm called FineComb that can obtain fine-grain end-to-end loss and latency measurements between edge routers in these networks. Such a mechanism can allow managers to distinguish between latencies and loss singularities caused by servers and those caused by the network. Compared to prior work, such as Lossy Difference Aggregator (LDA), which focused on switch-level latency measurements, the requirement of end-to-end latency measurements introduces the challenge of reordering that occurs commonly in IP networks due to churn. The problem is even more acute in switches across data center networks that employ multipath routing algorithms to exploit the inherent path diversity. Without proper care, a loss estimation algorithm can confound loss and reordering; furthermore, any attempt to aggregate delay estimates in the presence of reordering results in severe errors. FineComb deals with these problems using order-agnostic packet digests and a simple new idea we call stash recovery. Our evaluation demonstrates that FineComb is orders of magnitude more accurate than LDA in loss and delay estimates in the presence of reordering. [ABSTRACT FROM PUBLISHER]

We propose a novel architecture of data center networks (DCN), which adds wireless network card to both servers and routers. Existing traffic redundancy elimination (TRE) mechanisms reduce link loads and increase network capacity in several environments by removing strings that have appeared in earlier packets through encoding and decoding them several hops downstream. This article is the first to explore TRE mechanisms in large-scale DCNs and the first to exploit cooperative TRE among servers. Moreover, it also achieves the `logically centralized' control over the physically distributed states in emerging software defined networks (SDN) paradigm, by sharing information among servers and routers in data centers with wireless cards. We first formulate the TREDaCeN (TRE in Data Center Networks) problem and reduce the cycle cover problem to prove that finding an optimal caching task assignment for TREDaCeN problem is NP-hard. We further describe an offline TREDaCeN algorithm which is proved to have good approximation ratio. We then discuss efficient online zero-delay and semi-distributed implementations of TREDaCeN supported by physical proximity of servers and routers, enabling status updates in a single wireless transmission, using an efficient prioritized schedule. We also address online cache replacement and consistency of information in servers and routers with and without delay. Our framework is tested on different parameters and shows superior performance in comparison to other mechanisms (imported directly to this setting). Our results show the robustness and the trade-off between the `logically centralized' implementation and the overhead on handling inconsistency of distributed information in DCN. [ABSTRACT FROM PUBLISHER]

In this article, we consider a scalability problem associated with software-defined datacenter, of which the unicast/multicast routing states is proven to be NP-complete. We introduce an efficient multiple membership query algorithm, called Scalar-pair Vectors Routing and Forwarding (SVRF), based on the prime theory such as Chinese Remainder Theorem (CRT). Our proposed algorithm simply calculates corresponding output ports of each multicast group by dividing a common scalar-pair with a group-specific key, within pseudo-polynomial time. The result is then used to make a forwarding decision within few cycles through a hardware accelerator. Compared to Bloom filter, our algorithm can achieve remarkable performance in terms of memory consumption, processing time, hardware cost, and 100% delivery accuracy, while applying for a large number of large-scale distinct flows (including unicast and multicast) in a large-scale datacenter networks. Our work may be applied to various research areas of computer science and networking. [ABSTRACT FROM PUBLISHER]