Search

Convolution is a fundamental and computationally expensive primitive and finds ubiquitous in deep neural networks (DNNs). The evolving DNNs have spurred the emergence of numerous accelerators and they successfully achieve high throughput. However, for DNN inference with small batch sizes, the computational resources of the accelerators are often under-utilized, and the overhead of offloading is significant. Compared to accelerators, the CPU can better meet fast response requirements of inference, flexibly handle various models, and is suitable for various scenarios (from edge to data center). Therefore, CPU remains an attractive platform for DNN inference, despite the sub-optimal performance, and resource efficiency. In this paper, we propose CUTE, a scalable CPU-centric and ultra-utilized tensor engine for convolutions. It co-designs data flow and hardware architecture to leverage the data reuse and parallelism of convolutions. CUTE is composed of several small tensor elements (TEs) and two-level buffers. It employs a decoupled access-execution architecture and greedy strategy to feed data to TEs, enabling it to achieve ultra utilization and great scalability. CUTE is tightly coupled with the CPU to minimize offloading latency, thereby providing efficient convolution computing capabilities for the system. Experimental results show that under the same bandwidth, CUTE achieves an average performance improvement of 3.8x compared with the CPU AVX512 unit and 1.6x compared with the CPU AMX unit. Besides, CUTE achieves a speedup of 7.0x and 3.9x over Nvidia V100 GPU and Eyeriss accelerator respectively, due to higher utilization of computing units. [ABSTRACT FROM AUTHOR]

Valuable data is often distributed across multiple data centers (DCs). Deep learning (DL) tasks, constrained by privacy regulations, utilize local training and model averaging to facilitate collaborative training across multiple DCs. However, the hierarchical bandwidth within and between DCs diminishes the training efficiency for decentralized data. Therefore, it is imperative to prioritize research efforts aimed at reducing communication overhead while preserving convergence performance for geographically distributed DL tasks. To address this challenge, we propose a H igh- C onvergence and E fficient- C ommunication (HCEC) training strategy for geographically distributed data. In this paper, we adopt two approaches: (1) to ensure high convergence, we utilize dynamic learning rates and local epochs to avoid local optima; (2) to ensure efficient communication, we introduce the A daptive L ayerwise C ommunication (ALC) method to minimize inter-DC communication costs. The ALC method decides whether to communicate all L -layer model parameters at once or perform L -times communication based on the available bandwidth and computational training overhead. Experimental results show that compared to the model averaging method, HCEC ensures convergence and improves training efficiency by at most 37.9%. [ABSTRACT FROM AUTHOR]

GPUs used in data centers to perform deep learning inference tasks are underutilized. Previous systems tended to deploy a single model on a GPU to ensure that inference tasks met throughput and latency requirements. The rapid increase in one GPU's resources, as well as the emergence of scenarios such as small models and small batches, have exacerbated the issue of low GPU utilization. In this case, a mixed model deployment-based solution can significantly improve GPU utilization while also providing greater flexibility to the inference system's upper layer. The selection of model combinations and optimization strategies in mixed model deployment, however, remain unresolved issues. This paper proposes Optimum , the first model-combination planning and runtime optimization framework for mixed model deployment. Facing enormous search spaces, Optimum uses performance prediction for model combination selection with low search overhead. The predictor is based on a multilayer perceptron. Its input features are the profiling results of the model engine, and the output is the performance degradation. The runtime optimization strategies allow Optimum to perform performance optimization and fine-grained tradeoff. The Optimum prototype is based on CUDA multi-stream and TensorRT. The test results show that we have a flat 10.3% performance improvement over mainstream single-model deployments. We have a performance improvement of up to 7.09% over the state-of-the-art with an order of magnitude reduction in search overhead. [ABSTRACT FROM AUTHOR]

Report shows that in many cloud data centers, computing resources are not used efficiently and thereby cause resource waste. In many cloud nodes, the utilization rate of CPU is only 20%. A large-scale multi-layer smart computing system (such as Deep Learning) require computing of a large amount of labeled data. To improve cost-effectiveness of resources in cloud data centers, running big data-based smart computing applications to utilize residual computing resource capacity is a feasible solution. However, performance loss brought by resource competition and interference between host application and smart computing application is the main challenge for us. In this paper, we design, implement and evaluate the InSTechAH: A big data-oriented cost-effective cluster autoscaling and scheduling scheme in private cloud, which improves the resource utilization as well as to maintain required quality of services by autoscaling and scheduling background smart computing analytics tasks. [ABSTRACT FROM AUTHOR]

Cloud computing technology provides the means to share physical resources among multiple users and data center tenants by exposing them as virtual resources. There is a strong industrial drive to use similar technology and concepts to provide timing sensitive services. One such domain is a chain of connected virtual network functions. This allows the capacity of each function to be scaled up and down by adding or removing virtual resources. In this work, we develop a model of such service chain and pose the dynamic allocation of resources as an optimization problem. We design and present a set of strategies to allow virtual network nodes to be controlled in an optimal fashion subject to latency and buffer constraints. Furthermore, we derive a feedback-law for dynamically adjusting the amount of resources given to each functions in order to ensure that the system remains in the desired state even if there are modeling errors or for a stochastic input. [ABSTRACT FROM AUTHOR]

Modern cloud computing applications have stringent low-latency and high-throughput requirements to meet the increasingly diverse demands from customers. In edge datacenters, the popular load balancing scheme based on packet spraying, i.e., random packet spraying (RPS), makes good use of multiple equal-cost paths to ensure high link utilization and efficient transmission. However, RPS performs poorly under the bursty traffic scenario due to packet loss or even timeout. Therefore, to reduce the tail latency caused by retransmission, we design a coding-based random packet spraying named CRPS. Specifically, the source host transmits forward error correction (FEC) encoded packets and dynamically adjusts the data redundancy based on the packet loss rate. Then the switch randomly sprays the encoded packets across all equal-cost multiple paths to implement parallel transmission. In this way, once enough encoded packets from any parallel paths arrive at the destination host, the original packets can be decoded immediately to address the adverse impact of packet loss. The NS-2 simulation results show that CRPS effectively reduces the probability of timeout and significantly improves the tail flow completion time (FCT) by up to 72% compared with the state-of-the-art multipath transmission schemes. [ABSTRACT FROM AUTHOR]

Today, a vast amount of sensitive data worth millions of dollars is processed in untrusted data centers; hence, the confidentiality and integrity of the code and data are of paramount importance. Given the high incentive of mounting a successful attack, the complexity of attack methods has grown rapidly over the years. The attack methods rely on vulnerabilities present in the system to hijack the control flow of a process and use it to either steal sensitive information or degrade the quality of service. To thwart these attacks, the complexity of the defense methods has also increased in tandem. Researchers have explored different methods to ensure the secure execution of an application. The defense methods range from software-only to hardware-only to hybrid defense methods. In this survey, we focus on the relatively new hybrid form of defense methods where software and hardware work in tandem to protect the control flow of applications. We present a novel three-level taxonomy of these defense mechanisms based on first principles and use them to classify existing defense methods. After presenting the taxonomy, we critically analyze the proposed defense methods, study the evolution of the field and outline the challenges for future work. [ABSTRACT FROM AUTHOR]

Metadata service of geo-distributed file systems plays an important role in many web-based applications, such as supply-chain management, process mining from log files, intelligent aggregation and collaboration among business processes deployed over Industrial Internet, etc. The edge computing paradigm provides an ideal infrastructure for some value-added services by using geo-distributed file systems. However, web-based applications among the geo-distributed file systems often need an intelligent metadata service mechanism for low-latency data visiting and processing. Unfortunately, it is often a challenge in practice, as the edge computing paradigm often lacks an edge data center that is indispensable for metadata storage and metadata operation. In view of this challenge, an architecture of a virtual edge data center with intelligent metadata service is investigated, in this paper, for taking full advantage of the geo-distributed file systems. Concretely, a prototype of the virtual edge data center is proposed by dynamically gathering the idle storage capability of edge servers into a virtual resource pool. Then, its intelligent metadata service mechanism is investigated The experimental results demonstrate the feasibility of the architecture and show the superior performances with comparison analysis. [ABSTRACT FROM AUTHOR]

The evolving edge and IoT devices collect and process massive data and need tremendous storage space for those data. It is more efficient and reliable to establish a distributed storage system for their storage need. However, the conventional erasure codes used in distributed storage are designed for reliable data centers. Edge and IoT devices are highly vulnerable and unreliable. The conventional distributed storage systems cannot prevent malicious attackers and require very high redundancy in wild environments. This paper presents DSPR, a new decentralized distributed storage system with proof of replication for edge and IoT devices. By providing the cryptographic proofs of stored data, the system can actively find the data corruption and recover the data before it is unrecoverable in vulnerable environments. Therefore, we can apply high efficient coding schemes with low storage redundancy. Our LDPC coding achieves about 6x encoding and 20x recovering speedup than Intel ISA-L. With an IoT device Xavier AGX, the system can prove over 180GB of data per day to securely store the data, which is about 4x faster than public auditing schemes with an even stronger threat model that every device in the network can be malicious. [ABSTRACT FROM AUTHOR]

Data center is a fundamental infrastructure of computers and networking equipment to collect, store, process, and distribute huge amounts of data for a variety of applications such as Cyber–Physical–Social Systems, business enterprises and social networking. As the demands of remote data services keep increasing, both the workload of the data center and its power consumption are rapidly rising. An indispensable part of a data center is the cooling system which provides a suitable operation environment, and accounts for around 30% of the power consumption of the data center. Therefore, optimized energy management of data center's cooling system is a highly profitable research area. Generally, a cooling system is made up of a mechanical refrigeration sub-system and a terminal cooling sub-system. Heat generated during operation of the data center will be absorbed by the latter one, and transferred into the outdoor environment via the former one. Depending on the cooling principle, current cooling solutions can be classified into air-cooling, liquid-cooling or free cooling technology. Although air-cooling is widely used in most existing data centers, the other two solutions have attracted more interests due to their excellent cooling effectiveness and higher energy efficiencies. Among the different cooling equipment, the chillers and fans are the major power consumers of the entire cooling system. Therefore, modeling of their power consumption is important for energy management of the cooling system, which can be classified into mechanism-based methods and data-driven methods. Based on the aforementioned models, optimization strategies for the operation management of cooling equipment are proposed to reduce the power consumption of the cooling system, which mainly includes the model predictive control-based methods and reinforcement learning-based methods. This paper is an overview of the data center's cooling system, which mainly includes the mainstream cooling solutions, the power consumption modeling methods and the optimization control strategies. In addition, several current challenges and future work in the data center's cooling system are described. [ABSTRACT FROM AUTHOR]

Virtualized data centers usually consist of heterogeneous servers which have different specifications (performance). Though there usually exist unused heterogeneous servers in such data centers, conventional DVFS (Dynamic Voltage and Frequency Scaling)-based DTM (Dynamic Thermal Management) techniques do not exploit the unused servers to cool down hot servers. In this paper, we propose a novel DTM technique which adaptively exploits external computing resources (unused servers with different performance) as well as internal computing resources (unused CPU cores in the server) available in heterogeneous data centers. Additionally, we also propose to consider locations of the servers when migrating VMs (Virtual Machines) among servers in a rack, which has a large impact on the on-chip temperatures and performance due to the heat conduction; when VMs run on the two closest servers in the rack, the ambient temperature of servers is up to 6.2° higher, compared to the case where VMs run on the two farthest servers, so that on-chip temperature more rapidly increases causing up to 13.5% of performance degradation due to more frequent thermal throttling. When the temperature of a CPU core in a server exceeds a pre-defined thermal threshold, our proposed technique estimates the impact of VM migrations on performance (e.g., performance degradation due to the physical machine migrations and/or core migrations of VMs). Depending on the estimated performance impact of VM migrations, our technique adaptively employs the following three methods: (1) a method that migrates a VM to another distant server with different performance, (2) a method that migrates VMs among CPU cores in the server, and (3) a DVFS-based method. In our experiments, our proposed technique improves performance by 15.1% and saves system-wide EDP by 22.9%, on average, compared to a state-of-the-art DVFS-based DTM technique, satisfying thermal constraints. [ABSTRACT FROM AUTHOR]

Traditional GPU resource allocation approaches, widely adopted in today's data centers as the rise of cloud gaming, only focus on server-side functions while ignoring the client-side, which wastes hardware resources. To solve this problem, cloud-edge integrated architectures are put forward, leaving some workloads to the client-side. However, many cloud-edge integrated frameworks suffer from one big issue: shared-resource interference, stemming from two reasons: (a) GPU resource racing caused by resources overuse for single client, and (b) CPU resource racing caused by resources shortage among clients. This paper presents gRemote , an open-source cloud-rendering system that can address these issues on a GPU resource pool. To mitigate the CPU resource shortage, gRemote improves CPU configuration by expanding CPU resources from server-side to both server- and client-side. To maintain the reasonable GPU usage for individual tasks, gRemote innovates a new resource-sharing mechanism called GPU throttle. Furthermore, remote API-Forwarding brings another type of network bandwidth consumption: command streaming. To optimize the network bandwidth for GPU command transmission, gRemote proposes two methods, command characteristics-oriented (CCO) compression algorithm and command-transmitted methodology. gRemote supports 1,228 OpenGL commands and provides cloud rendering for more than 40 clients with negligible shared-resource interference. With CCO algorithm and command-transmitted methodology, the network bandwidth is saved by more than 90%. [ABSTRACT FROM AUTHOR]

In recent years, big data and data analytics based on Cyber–Physical–Social Systems (CPSS) have become increasingly popular in providing valued services to humans. For many applications of CPSS, adequate computing infrastructure, which can be realized using powerful data centers (DCs), is needed. These DCs can then provide CPSS application developers with flexible and efficient High-Performance-Computing-Communications services. In DCs, the energy consumption of the cooling system which dissipates the heat generated by information technology (IT) devices should be optimized. Since the cooling system is one of the main energy consumers of DCs, optimization of its energy consumption can drastically reduce the operating costs while maintaining stable operation of the IT devices by efficient heat dissipation. Therefore, there is continuing development on improving the performance of cooling systems for DCs using different optimization strategies. In particular, model-based optimization algorithms have had impressive advances, but their deployment in real physical systems often becomes difficult due to limited data, poor optimization efficiency, and potential operation risks. In this paper, we propose a two-layer optimal scheduling framework for room-level cooling of DCs. In the global layer, we use limited data to build a set of novel physically-based empirical models to achieve accurate system energy tracking. Then with defined equipment operating constraints, a genetic algorithm efficiently obtains the optimal plan of all equipment control while ensuring safe system operations. In the local layer, through interactions with the global layer, local precision air conditioners are regulated to stabilize the room temperature within a safe range. To test our solution in a real physical system, we deployed the two-layer optimal scheduling technique in the real DC cooling system of Postal Savings Bank of China in Hefei, China. Our solution achieved an impressive average reduction of 6.1% on cooling load factor. [ABSTRACT FROM AUTHOR]

Virtual machine (VM) placement is a key technique for energy optimization in cloud data centers. Previous work generally focus on how to place the VMs efficiently in servers to optimize the physical resources used (e.g., memory, bandwidth, CPU, etc.), network resources used or cooling energy consumption. These work can optimize the energy consumption of cloud data centers according to one or two aspects (e.g. server, network or cooling), however, these methods may cause increased energy consumption in other aspects. To address this problem, we propose a global-energy-aware VMP (virtual machine placement) strategy to reduce, from multiple aspects, the total energy consumption of data centers. A two-step SAG algorithm is designed to lower the energy consumption of cloud data centers where multiple VMs are deployed. We conduct extensive experiments to evaluate the effectiveness of SAG. Two workloads from real-world data centers are utilized to quantitatively measure and compare the performance of our SAG with other typical algorithms. Experimental results indicate that, compared to other algorithms, our global-energy-aware VMP strategy can reduce the total energy consumption of the cloud data center by 8%–24.9%. [ABSTRACT FROM AUTHOR]

The osmotic computing paradigm sets out the principles and algorithms for simplifying the deployment of Internet of Things (IoT) applications in integrated edge-cloud environments. Various existing simulation frameworks can be used to support integration of cloud and edge computing environments. However, none of these can directly support an osmotic computing environment due to the complexity of IoT applications and heterogeneity of integrated edge-cloud environments. Osmotic computing suggests the migration of workload to/from a cloud data center to edge devices, based on performance and security trigger events. We propose 'IoTSim-Osmosis– a simulation framework to support the testing and validation of osmotic computing applications. In particular, our detailed related work analysis demonstrates that IoTSim-Osmosis is the first simulation framework to enable unified modeling and simulation of complex IoT applications over heterogeneous edge-cloud environments. IoTSim-Osmosis is demonstrated using an electricity management and billing application case study, for benchmarking various run-time QoS parameters, such as IoT battery use, execution time, network transmission time and consumed energy. [ABSTRACT FROM AUTHOR]

The fog/edge computing paradigm is increasingly being adopted to support a range of latency-sensitive IoT services due to its ability to assure the latency requirements of these services while supporting the elastic properties of cloud computing. IoT services that cater to user mobility, however, face a number of challenges in this context. First, since user mobility can incur wireless connectivity issues, executing these services entirely on edge resources, such as smartphones, will result in a rapid drain in the battery charge. In contrast, executing these services entirely on fog resources, such as cloudlets or micro data centers, will incur higher communication costs and increased latencies in the face of fluctuating wireless connectivity and signal strength. Second, a high degree of multi-tenancy on fog resources involving different IoT services can lead to performance interference issues due to resource contention. In order to address these challenges, this paper describes URMILA, which makes dynamic resource management decisions to achieve effective trade-offs between using the fog and edge resources yet ensuring that the latency requirements of the IoT services are met. We evaluate URMILA's capabilities in the context of a real-world use case on an emulated but realistic IoT testbed. [ABSTRACT FROM AUTHOR]

Modern datacenters involve a rich mix of workloads, each of which puts forward different service-level objective, including high throughput, low latency, etc. Currently, most datacenters introduce statistical multiplexing technology and oversubscription to the network design to lower the total cost, which can easily lead to the occurrence of network congestion, especially when the network is highly occupied by throughput-intensive workloads. This paper describes ProCAM, a proactive congestion avoidance mechanism for datacenter networks. As throughput-intensive flows are the chief culprit of network congestion, ProCAM adapts the multi-path routing to control transmission bandwidth and utilizes the predictability of throughput-intensive flows to prearrange optimal coordinate scheme (desynchronize the sending time of concurrent long-flows) beforehand as a proactive manner, by solving the low-congestion transmission model which minimizes network-wide host-to-host transmission latency from a global perspective. In this way, queue length in buffers can be kept at a low level and the performance of latency-sensitive flows can be guaranteed. In the evaluation experiments based on simulation with Mininet and SDN controller Ryu, the extensive simulations show that the proposed ProCAM can achieve high throughput with nearly zero packet loss and low latency when the network is highly occupied. [ABSTRACT FROM AUTHOR]