Search

Background: Genotype imputation as a service is developed to enable researchers to estimate genotypes on haplotyped data without performing whole genome sequencing. However, genotype imputation is computation intensive and thus it remains a challenge to satisfy the high performance requirement of genome wide association study (GWAS). Objective: In this paper, we propose a high performance computing solution for genotype imputation on supercomputers to enhance its execution performance. Method: We design and implement a multi-level parallelization that includes job level, process level and thread level parallelization, enabled by job scheduling management, message passing interface (MPI) and OpenMP, respectively. It involves job distribution, chunk partition and execution, parallelized iteration for imputation and data concatenation. Due to the design of multi-level parallelization, we can exploit the multi-machine/multi-core architecture to improve the performance of genotype imputation. Results: Experiment results show that our proposed method can outperform the Hadoop-based implementation of genotype imputation. Moreover, we conduct the experiments on supercomputers to evaluate the performance of the proposed method. The evaluation shows that it can significantly shorten the execution time, thus improving the performance for genotype imputation. Conclusion: The proposed multi-level parallelization, when deployed as an imputation as a service, will facilitate bioinformatics researchers in Singapore to conduct genotype imputation and enhance the association study.

Existing Hadoop MapReduce fault tolerance strategy causes the computing jobs suffering from high performance penalty during failure recovery. In this paper, we propose Fast Recovery MapReduce (FAR-MR) to improve MapReduce performance in failure recovery. FAR-MR includes a novel fault tolerance strategy that combines distributed checkpointing and proactive push mechanism to support fast recovery from task failure and node failure. With distributed checkpointing, computing progress of each task is recorded as checkpoints periodically and kept in distributed data storage. The recovered task can obtain the last progress of the failed task from the distributed storage during failure recovery. In addition, the proactive push mechanism enables the computing results of map tasks to be proactively transmitted to the nodes hosting reduce tasks of the same computing job. When a failure happens, the partial output results being pushed to the reducer nodes can be used by the reduce tasks without the necessity of re-compute. FAR-MR allows a failed task to be recovered efficiently at any node in the cluster. The performance evaluation has shown that the proposed FAR-MR can improve computing job performance by up to 62% and 45% compared to Hadoop MapReduce in the case of task failure recovery and node failure recovery, respectively.

In view of the problems of exposure bias and insufficient annotation data in the existing Chinese short text summary algorithms, a new pre-training short text summary algorithm named ERNIE-GEN-CTS based on ERNIE-GEN is proposed. This paper uses pre-training language model to realize the double attention of word level and word level for short text, and fills the sub block with semantic information to optimize the exposure bias problem. After the pre-training model of large-scale Chinese corpus, the results are improved in LCSTS data set after the task tuning of the downstream summary, and the validity and stability of the model are verified.

Federated cloud enables data sharing across different organizations. This paper proposes a distributed search index (DS-index) solution to provide distributed search capability in federated cloud, with the target to support multi-attribute range queries with secondary indexes. DS-index deploys a three-layer architecture with a multi-attribute index overlay, a tree-based P2P network layer, and a federated cloud layer. We present a distributed search algorithm with the three-layer architecture. In order to facilitate distributed search, we propose a dynamic mapping algorithm for DS-index to map between different layers. In addition, a Markov Chain-based cost model is defined to facilitate node selection and mapping. With the dynamic mapping algorithm and cost model, the proposed DS-index solution is more cost-effective than the traditional P2P networks as it can reduce the network cost in terms of index maintenance cost and node selection cost. The experiments demonstrate that, our DS-index solution along with its cost model can save the computation resource and reduce network bandwidth consumption by around 30% comparing to the one without cost model. It can also reduce the number of node splits/merges by around 20% comparing to the state-of-the-art solution.

Due to consistent improvements in memory and processor technology, object storage devices (OSDs) have greater memory space and more powerful processing power, which allow the OSDs to execute user-defined programs. Shifting part of an application's processing to the disk drives drops the amount of data transferred across the network and explores the parallelism of large-scale distributed storage systems, reducing the execution time for many basic data analytics tasks. In this paper, we propose a large-scale object-based active storage platform, named Gem, for data analytics in the internet of things (IoT). All data from the IoT that resides in disk drives form objects with attributes, methods and policies. For some applications such as data analytics, application-specific operations are executed by the drive processors. In this way, only the results are returned to clients, rather than data files being read by the clients. Therefore, the platform Gem is able to greatly reduce the overhead of data analytics applications in the Internet of Things. By conducting performance evaluation, experimental results demonstrate the effectiveness and scalability of Gem.

As modern data centers continue to grow in size and complexity to host different kinds of applications, it is required to have an efficient proactive failure management for Data Center reliability. Although Software-Defined Networking (SDN) and its implementation OpenFlow facilitate dynamic management and the configuration of Data center networks, network failure recovery in a timely manner remains great challenging. The centralized SDN controller is responsible for monitoring the entire network health status and maintain the end-to-end connectivity between the hosts. In the event of a link failure, the controller either computes a new backup path reactively on demand and creates flow table entries for the new backup path, or pro-actively computes the backup path a-priori and set up flow table rules for the pre-defined backup path. Switching to the predefined backup path locally results in faster recovery time compared to switching to the backup path that establish on demand. In this paper, we propose a proactive mechanism to provide fast recovery upon a link failure. With the proposed proactive approach, we compute the recovery (or backup) paths for the flows prior to failures and install appropriate rules in the forwarding tables at the switches in advance. Such recovery paths are adaptively updated based on the current load state of the network to improve resource efficiency and reduce congestion. By providing the backup forwarding rules in advance, upon a failure, the failed traffic is rerouted without interacting with the controller, thus ensuring fast recovery. We demonstrate the effectiveness of the proposed mechanisms using an experimental testbed with Openstack platform and simulated environment with Mininet.

Without a central authority, blockchains can easily enable the management of transactions. Smart contracts stored on blockchains are self-executing contractual states that are not controlled by anybody, so they can be trusted. In addition, due to increasing improvements in processor and memory technology, IoT (Internet of Things) devices have more powerful processing power and greater memory space, which allow them to execute user-defined programs, e.g., smart contracts. Shifting part of applications’ tasks to IoT devices reduces the transferred data amount over the IoT network. The parallelism of large-scale storage systems is employed to decrease many basic data analytics tasks’ execution time. Blockchain can be used as smart contracts that facilitate and enforce the negotiation of a contract in the IoT. This chapter proposes a blockchain-based storage system, named Sapphire, for data analytics applications in the Internet of Things. All the IoT data from the devices forms objects with IDs, attributes, policies, and methods. We present an OSD-based smart contract (OSC) approach employed in Sapphire as a transaction protocol, where IoT devices interact with such blockchains. For data analytics applications, the IoT device processors execute application-specific operations. By doing so, only the results are returned to clients instead of data files read by them. Therefore, the Sapphire system can greatly decrease the overhead of data analytics in the Internet of Things.

Prompt response to data access requests is the key performance concern of parallel storage systems. Sort partition (SP) is one of the most promising file allocation solutions for the static scenario due to its sorted partition and placing mechanism. The online variant of SP, hybrid partition (HP), however, does not perform well in the dynamic scenario. The recently proposed balanced allocation with sort (BAS) and its online version balanced allocation with sort for batch (BASB) try to compete with SP and HP by providing better load balancing together with sorting. BAS's performance is close to SP, and BASB greatly outperforms HP. In this paper, we proposed new file placement solutions for the parallel storage systems: Optimized Sort Partition (OSP) for the static scenario and Optimized Sort Partition Online (OSPOnline) for the dynamic scenario. By eliminating the drawbacks of SP/HP and BAS/BASB in the file assignment, OSP and OSPOnline can achieve optimized system response time. Conducted simulations show that OSP and OSPOnline steadily outperform their competitors under various conditions.

In this paper, we propose a new reference-based data compression method for efficient compressing of genome sequencing data in FASTQ format. With the advance of the next sequencing technology, the genome data can be generated faster and cheaper, which brings the challenges for efficient storage of these data when used in cloud computing. In order to efficiently store these types of genome data in cloud, content-aware compressing methods have to be developed to make use of the specific file structures. Compared with existing genome-specific compression methods, our proposed content-aware method focused on high compression ratio by taking advantages of repetitive nature of DNA sequence, and using reference genomes in compressing the sequences inside the FASTQ files. The benchmark results of 8 datasets show that our method can achieve highest compression ratio compared with existing FASTQ file compressors.

Federated cloud facilitates data sharing across multiple organizations. Existing distributed storage systems cannot support efficient data search with complicated query requirements. This paper proposes a DS-index solution to provide distributed search in federated cloud, targeting to support multi-attribute range search with secondary index. DS-index includes a three-layer architecture with a multi-attribute index overlay, a tree-based P2P network layer, and a federated cloud layer. In addition, a dynamic mapping algorithm has been presented for DS-index to map between different layers. We have also defined a Markov Chain-based cost model to facilitate node selection and mapping. The proposed DS-index solution is capable to support multi-attribute range queries in the federated cloud. The experiments have shown that, the DS-index solution with cost model can save the computation resource and reduce network bandwidth consumption by around 30% comparing to the one without cost model. It can also reduce the splitting/ merging times by around 20% comparing to the state-of-the-art solution.

Visualizing and analyzing large-scale datasets are both critical and challenging, as they require substantial resources for data processing and storage. While the speed of supercomputers continues to set higher standard, the I/O systems have not kept in pace, resulting in a significant performance bottleneck. To alleviate the I/O bottleneck for scientific visualization applications, we propose a Visualization via a Heterogeneous Distributed Storage Infrastructure (VH-DSI) solution to improve I/O speed and accelerate overall visualization performance. VH-DSI replaces the traditional parallel file system with a distributed file system to support visualization applications. A new scheduling algorithm HeterSche is proposed in VH-DSI to assign computing tasks to data nodes with the consideration of cluster heterogeneity and data locality. VH-DSI also includes a design to support POSIX-IO for distributed file system. The performance evaluation has shown that the proposed VH-DSI solution can achieve significant performance improvement for visualization applications. Compared to the traditional visualization, the VH-DSI solution reduces the response time by at least 5 times. The HeterSche scheduling algorithm is capable to speed up visualization compared to other scheduling algorithms especially for large scale datasets.

In this paper, an effective medium access control protocol is proposed for Ethernet passive optical networks. By combining the bandwidth guarantee polling (BGP) scheme with the strict priority queuing (SPQ) mechanism, the proposed solution can greatly mitigate the light-load penalty while supporting differentiated services. The performance of the proposed solution has been demonstrated by the simulation results.

The Ethernet passive optical network (EPON) has recently attracted more and more research attentions since it can be a perfect candidate for next generation access networks. Supporting differentiated services (DiffServ) is an important issue for service providers to design an EPON system. In this paper, we deploy the hierarchical scheduling for the medium access control (MAC) layer in EPONs, which consists of the inter-ONU scheduling and intra-ONU scheduling, to provide DiffServ as well as share the upstream bandwidth efficiently among multiple users. The inter-ONU scheduling among the ONUs uses the efficient MAC scheme that we have proposed before. It provides services to different ONUs based on the service level agreement (SLA). The intra-ONU scheduling within the ONU adopts the two-stage queuing (TSQ) mechanism to provide DiffServ to different traffics in the same ONU. It can provide better services to the lower-priority traffics to achieve better fairness than the strict priority queuing (SPQ) mechanism. The combined scheme greatly mitigates the light-load penalty that is obvious in other schemes. The simulation results show that our scheme for the hierarchical scheduling can effectively provide differentiated services for different types of traffics, and also achieve the fairness to some extent.

The ever increasing amounts of digital data being stored in public and private clouds are challenging users to access and manage the data. With the corresponding storage system reaches Petabyte-scale, or even Exabyte-scale, metadata access will become a severe performance bottleneck. Hence, this paper proposes an efficient multi-dimensional metadata index and search solution for cloud data. By proposing some new mechanism for K-D-B tree based index/search and implementing index partitioning technique, our system can achieve optimized performance in terms of memory utilization and search speed. Experiments show that our system performs much better as compared with existing solutions. In addition, our system can safely scale out in a distributed manner with guaranteed performance.

In this paper, an intelligent model for analyzing and predicting cloud computing resource utilization is proposed to enhance on-demand services in cloud computing systems. The model is with the capability to discover active users and mine the system storage utilization patterns. This model is also with learning capabilities to adapt the dynamics in the cloud computing platform by capturing changing patterns of system storage utilization, and it employs data mining means for computing the practical model to be used for prediction and providing inputs for intelligent management in the on-demand cloud computing system. We have evaluated the proposed analysis and prediction model in a cloud computing platform. High prediction accuracies of 95% and 86% have been achieved in 1-day ahead and 7-day ahead system utilization prediction, respectively.

The computational science community is approaching petascale level simulations that will produce massive amount of datasets. While the computational power of supercomputers keep increasing, the I/O systems have not kept pace, resulting in a significant performance bottle neck. We propose a solution, VisDSI, to address the problem by 1) using traditional high performance clusters with disks directly attached within each node, 2) deploying a data-intensive distributed file system on the cluster, and 3) developing a POSIX-compatible I/O layer to enable the traditional visualization applications to smoothly port to this new platform as well as to provide a new I/Osemantic of retrieving the data location with respect to the POSIX standards. In particular, VisDSI guarantees the colocated compute and data storage by introducing a scheduling of work assignments to nodes with local copies of needed data. Compared with the original visualization application, our solution runs at least 10 times faster.

Data Center Ethernet is likely to be deployed as the communication infrastructure for future data centers, which carries multiple types of traffic with very different characteristics and handling requirements. Conventional Spanning Tree Protocol STP cannot meet the requirement of a Data Center Ethernet framework because of its poor bandwidth utilization and lack of multipathing capability. In this paper, we propose a layer2 multipathing solution, namely optimized dynamic load-balancing multipathing ODLBMP, to be deployed in Data Center Ethernet. Our proposed method utilizes all available links and ports for frame delivery and can split traffic of a communication pair along multiple paths. In ODLBMP, the traffic loads of all paths are continuously monitored so that traffic assigned to each path can be dynamically adjusted to avoid path/link over-utilization. Per-flow forwarding is observed in ODLBMP to guarantee the in-order delivery, which is important for most storage traffic. In addition, ODLBMP finely differentiates flows from application perspective so it has more flexibility in traffic splitting and route selection, and achieves better multipath load balancing. Computer simulations show that our proposed algorithm performs better than other compared algorithms, including STP, Transparent Interconnection of Lots of Links, and DLBMP, in all simulation scenarios in terms of frame delivery ratio and network throughput. Copyright © 2013 John Wiley & Sons, Ltd.

In parallel I/O systems, fast response to disk access and load balancing are two important performance objectives pursued by end users and applications. These performances are largely determined by the data allocation strategies, or file assignment algorithms. However, most existing algorithms can only obtain one of the performance objectives, including the well-known algorithms like Greedy, Sort Partition (SP) and Hybrid Partition (HP). New algorithms that can achieve both objectives are very necessary for parallel I/O systems. In this paper, we have proposed two new allocation algorithms for file assignment in parallel I/O systems: an offline Balanced Allocation with Sort (BAS) algorithm and an online Balanced Allocation with Sort for Batch (BASB) algorithm. Both algorithms aim to achieve the optimal mean response time and load balancing at the same time. The experiment results have shown that the BAS algorithm can get the optimal performance on response time among all compared algorithms and better performance on load balancing than SP. For online BASB algorithm, it can achieve the optimal performances on both response time and load balancing among all algorithms for comparison.

Ethernet Passive Optical Network (EPON) has been regarded as the best candidature for next-generation access network, because it represents the convergence of the inexpensive Ethernet equipment and high-bandwidth fiber infrastructure. In EPON networks, multiple Optical Network Units (ONUs) share the upstream bandwidth to transmit data packets to the Optical Line Terminal (OLT). An efficient Medium Access Control (MAC) protocol is required in EPONs to arbitrate upstream transmissions. Many efforts have been put on the design of MAC protocols, especially Dynamic Bandwidth Allocation (DBA) schemes, in EPONs. This chapter aims to present a comprehensive survey on the up-to-date DBA schemes for EPON networks. We have provided a categorization method to classify numerousDBA schemes into corresponding groups. Besides the description and comments for each individual scheme, this chapter has also provided common features as well as the merits and shortcomings for each category of DBA schemes.

The traditional method of centralized backup suffers from single point of failure and scalability problem. It is therefore beneficial to carry out the backup service over geographically distributed networks. Hence, redundant copies of data stored in such networks are essential to ensure the availability and reliability of the backup data. The main objective of this work is to balance and evenly distribute large amounts of backup data for efficient use of disks as well as network bandwidth which leads to faster data access and recovery. We develop an efficient method for data placement and replication. Our method has several attractive features: (1) Balanced and efficient data placement in a distributed storage resource environment. (2) Redistributing as few replicas as possible in the event of storage node addition or failure/removal. (3) There is no central mapping table so that the search performance becomes more efficient. We demonstrate the effectiveness of our method through theoretical analysis and simulation results.

Interleaved Polling with Adaptive Cycle Time (IPACT) has great significance in the Ethernet Passive Optical Network (EPON), which is regarded as one of the best choices for next generation access networks. In this paper, we propose an estimation-based scheme, IPACT with Grant Estimation (IPACT-GE), to improve the performance of the IPACT. By estimating the amount of new packets arriving in one polling cycles and granting transmission windows to ONUs based on the estimation, the proposed scheme can provide a shorter waiting delay for the packets and less buffer occupancy than those of the IPACT scheme at the light traffic load so that the light load penalty could be greatly mitigated when combined with the Strict Priority Queue (SPQ) mechanism to provide the Differentiated Services.

The Ethernet passive optical network (EPON) has recently attracted more and more research attentions since it could be a perfect candidate for next generation access networks. Supporting differentiated services in EPONs is an important issue for service providers to design an EPON system. We have proposed a multiple access control scheme to provide a solution to efficiently share the upstream of an EPON among multiple users. In this paper, we further deploy the efficient multiple access scheme and combine it with a two-stage queuing scheme to provide differentiated services in EPONs. The simulation results show that the combined scheme can effectively provide differentiated services for different classes of traffic.

The Ethernet passive optical network (EPON) has recently attracted more attentions from the academic research and industry since it could be a perfect candidate for next generation access networks. Supporting differentiated services is an important issue for service providers to design an EPON system. In this paper, we propose an urgency fair queuing (UFQ) scheme to support DiffServ among multiple users in EPONs. It can achieve fairness by providing better services to best-effort users while guaranteeing the services for QoS users. The simulation results show that the UFQ scheme can effectively provide differentiated services for different classes of traffic

Ethernet passive optical network (EPON) has been considered the best choice for access networks. Since EPON is supposed to deliver different types of traffic, supporting differentiated services in EPON becomes an important issue. This paper discusses how the differentiated services can be provided by medium access control (MAC) protocol in EPON. We combine the bandwidth guarantee polling (BGP) scheme with the strict priority scheduling specified by IEEE 802.1D to support differentiated services. The simulation results show that our proposed technique can effectively provide differentiated services to different classes of traffic in the EPON

Multiple access control is an important issue in Ethernet passive optical networks (EPON). It is needed for traffic from different ONUs to share the upstream channel. The IEEE 802.3ah Ethernet in the first mile (EFM) task force has specified a multipoint control protocol (MPCP) for this purpose. We propose a novel multiple access control scheme consisting of a parameter-based call admission control (CAC) mechanism, the evenly distributed algorithm (EDA), and the bandwidth guarantee polling (BGP) algorithm. Our scheme can guarantee bandwidth for high-demand customers while providing best-effort service to low-demand customers according to the service level agreement (SLA). Simulation results show that the proposed scheme is very efficient.

While the backbone networks have experienced substantial changes in the last decade; the access networks have not changed much. Recently, passive optical networks (PONs) seem to be ready for commercial deployment in access networks due to the maturity of a number of enabling technologies, long distance and reduced maintenance. Among PON technologies, the Ethernet PON (EPON) presently being standardized by the IEEE 802.3ah Ethernet in the first miles (EFM) task force is most attractive because of its high speed, low cost, familiarity, interoperability and low overhead. In this paper, we propose a novel bandwidth guarantee polling (BGP) scheme that will allow the upstream bandwidth to be shared based on the service level agreement (SLA) between each subscriber and the operator. It is able to provide bandwidth guarantee for premium subscribers according to the SLAs while providing best-effort service to other subscribers. The analytical and simulation results prove that the proposed scheme does best in what it is designed to do compared to another well-known scheme that has not considered providing differentiated services. With business customers preferring premium services with guaranteed bandwidth and residential users preferring low-cost best effort services, our scheme could benefit both groups of subscribers as well as the operators.