Search

Recently, research on six-degree-of-freedom virtual reality (VR) systems based on a light field (LF) has been actively conducted. The LF-based approach is photorealistic and less prone to errors than the existing three-dimensional (3D) modeling-based approach. On the other hand, because the amount of data is very large for LF-based immersive virtual reality, it is important to manage the data efficiently. This paper makes two major contributions to data management for rendering acceleration in this context. First, the GPUs commonly used for high-speed rendering require optimization of the data transfer process between the host memory and the device memory. This paper proposes a simple but effective way to maximize data reuse by finding the proper size of a transfer unit depending on the target applications and the system capability constraints. Also, view synthesis and data transfer are performed in a pipelined manner to hide the data movement time. Second, for the LF rendering of a wide space, data management in the storage-host-GPU hierarchy is attempted for the first time. As the size of the virtual space increases, the required amount of LF data grows rapidly and exceeds the allocatable host memory capacity. Because the loading of data stored in storage is a very slow process, a system-level LF rendering structure that carefully considers the characteristics of the storage-host-GPU memory hierarchy should be designed. In this paper, progressive LF updating is proposed, and it prevents rendering stalls caused by slow storage data loading. Experimental results show that a 360-degree view with a $9000 \times 2048$ resolution can be rendered in an average of 2.57 ms from LF data with a $4096 \times 2048$ resolution, achieving performance close to 400 frames per second. [ABSTRACT FROM AUTHOR]

So-called spatial web queries retrieve web content representing points of interest, such that the points of interest have descriptions that are relevant to query keywords and are located close to a query location. Two broad categories of such queries exist. The first encompasses queries that retrieve single spatial web objects that each satisfy the query arguments. Most proposals belong to this category. The second category, to which this paper's proposal belongs, encompasses queries that support exploratory user behavior and retrieve sets of objects that represent regions of space that may be of interest to the user. Specifically, the paper proposes a new type of query, the top- $k$ k spatial textual cluster retrieval ($k$ k -STC) query that returns the top- $k$ k clusters that (i) are located close to a query location, (ii) contain objects that are relevant with regard to given query keywords, and (iii) have an object density that exceeds a given threshold. To compute this query, we propose a DBSCAN-based approach and an OPTICS-based approach that rely on on-line density-based clustering and that exploit early stop conditions. Empirical studies on real data sets offer evidence that the paper's proposals can find good quality clusters and are capable of excellent performance. [ABSTRACT FROM AUTHOR]

Visual exploration has become an integral part of big spatial data management. With the increase in volume and number of spatial datasets, several specialized mechanisms have been proposed to speed up the exploration of these datasets. However, the existing techniques have major limitations which make them incapable of providing visual exploration for hundreds of thousands of big datasets on a single machine. This paper introduces a new index structure, termed AID*, that facilitates the visual exploration of an arbitrarily large number of big spatial datasets on a single machine. The AID* index defines multi-resolution fixed-size tiles on the input and classifies them as image, data, shallow, or empty tiles, based on their processing cost. Then, it uses this classification to build an index with a minimal index size and construction time, while supporting the desired real-time exploration interface. The index is constructed in parallel, using Hadoop or Spark, and is accessible to end users through a standard web interface similar to Google Maps. The small size of the index allows a single-machine server to host arbitrarily many datasets. Our experiments, on up-to 1 TB of data and 27 billion records, show that the construction of the proposed index is up-to an order of magnitude faster than the baselines without compromising the end-user interactivity. [ABSTRACT FROM AUTHOR]

In this paper, we present a systematic approach to design hardware circuits for bit-dimension permutations. The proposed approach is based on decomposing any bit-dimension permutation into elementary bit-exchanges. Such decomposition is proven to achieve the theoretical minimum number of delays required for the permutation. This offers optimum solutions for multiple well-known problems in the literature that make use of bit-dimension permutations. This includes the design of permutation circuits for the fast Fourier transform, bit reversal, matrix transposition, stride permutations, and Viterbi decoders. [ABSTRACT FROM AUTHOR]

In the last two decades, interactive visualization and analysis have become a central tool in data-driven decision making. Concurrently to the contributions in data visualization, research in data management has produced technology that directly benefits interactive analysis. Here, we contribute a systematic review of 30 years of work in this adjacent field, and highlight techniques and principles we believe to be underappreciated in visualization work. We structure our review along two axes. First, we use task taxonomies from the visualization literature to structure the space of interactions in usual systems. Second, we created a categorization of data management work that strikes a balance between specificity and generality. Concretely, we contribute a characterization of 131 research papers along these two axes. We find that five notions in data management venues fit interactive visualization systems well: materialized views, approximate query processing, user modeling and query prediction, muiti-query optimization, lineage techniques, and indexing techniques. In addition, we find a preponderance of work in materialized views and approximate query processing, most targeting a limited subset of the interaction tasks in the taxonomy we used. This suggests natural avenues of future research both in visualization and data management. Our categorization both changes how we visualization researchers design and build our systems, and highlights where future work is necessary. [ABSTRACT FROM AUTHOR]

High voltage circuit breakers (HVCBs) play a critical role on providing the desired reliability, and resiliency in power systems. In order to extend their lifetime and predict the failures, various maintenance policies could be applied on these critical components. Amongst these strategies, condition-based maintenance (CBM) provides a satisfactory agreement with future smart environment. This paper aims to provide an insight into the relevant developments in this subject and to explore the viable visions compatible with future research stream. Accordingly, three directions, i.e., diagnostic signals, intelligent modelling and using monitoring data in asset management have been addressed in this paper. It presents challenges dealing with real-time assessment of the diagnostic signals relating to measurements, and analyses. Subsequently, the issues associated with using artificial intelligent (AI) and Machine learning for providing intelligent algorithms have been discussed. Finally, the connection between the monitoring data and the asset management approach is investigated. The latter is looking for the subjects including remaining lifetime estimation, prioritization, and health index definitions. This paper has attempted to make a bridge from past to future research trends in the failure diagnosis of HVCBs. [ABSTRACT FROM AUTHOR]

Efficient real-time analytics are an integral part of an increasing number of data management applications, such as computational targeted advertising, algorithmic trading, and Internet of Things. In this paper, we focus primarily on accelerating stream joins, which are arguably one of the most commonly used and resource-intensive operators in stream processing. We propose a scalable circular pipeline design ($\sf{ Circular\text{-}MJ}$ Circular - MJ ) in hardware to orchestrate a multiway join while minimizing data flow disruption. In this circular design, each new tuple (given its origin stream) starts its processing from a specific join core and passes through all respective join cores in a pipeline sequence to produce the final results. We also present a novel two-stage pipeline stream join ($\sf{ Stashed\text{-}MJ}$ Stashed - MJ ) that uses a best-effort buffering technique (referred to as stash) to maintain intermediate results. If an overwrite is detected in the stash, our design automatically resorts to recomputing intermediate results. Finally, we present a parallelized version of our multiway stream join by integrating our proposed pipelines into a parallel unidirectional flow-based architecture ($\sf{ Parallel\text{-}MJ}$ Parallel - MJ ). Our experimental results demonstrate a linear throughput scaling with respect to the numbers of streams and processing cores. [ABSTRACT FROM AUTHOR]

Cloud computing is a promising distributed computing platform for big data applications, e.g., scientific applications, since excessive resources can be obtained from cloud services for processing and storing both existing and generated application datasets. However, when tasks process big data stored in distributed data centers, the inevitable data movements will cause huge bandwidth cost and execution delay. In this paper, we construct a tripartite graph based model to formulate the data replica placement problem and propose a genetic algorithm based data replica placement strategy for scientific applications to reduce data transmissions in cloud. Our approach can reduce 1) the size of moved data, 2) the time of data movement and 3) the number of movements. We conduct experiments to compare the proposed strategy with the random placement strategy used in Hadoop Distributed Files System (HDFS), which demonstrates that our strategy has better performance for scientific applications in clouds. [ABSTRACT FROM AUTHOR]

Erasure codes are being extensively deployed in distributed storage systems instead of replication to achieve fault tolerance in a storage efficient manner. While traditional erasure codes are storage efficient, they can result in a significant increase in the amount of data access and downloaded during rebuilding of failed or otherwise unavailable nodes. In this paper, we present a new framework, which we call piggybacking, for constructing distributed storage codes that are efficient in the amount of data read and downloaded during rebuilding, while meeting requirements arising out of system considerations in data centers—maximum-distance-separability (MDS), high-rate, and a small number of so-called substripes. Under this setting, to the best of our knowledge, piggyback codes achieve the minimum average amount of data access and downloaded during rebuilding among all existing explicit solutions. The piggybacking framework also offers a rich design space for constructing codes for a variety of other settings. In particular, we construct codes that require minimum amount of data access and downloaded for rebuilding among all existing solutions for: 1) binary MDS array codes with more than two parities and 2) MDS codes with the smallest locality during rebuilding. In addition, we show how piggybacking can be employed to enable efficient repair of parity nodes in codes that address the rebuilding of only systematic nodes. The basic idea behind the piggybacking framework is to take multiple instances of existing codes and add carefully designed functions of the data from one instance to the others. This framework provides 25% to 50% savings in the average amount of data access and downloaded during rebuilding depending on the choice of the code parameters. [ABSTRACT FROM AUTHOR]

In a power system, unlike some critical and standalone assets that are equipped with condition monitoring devices, the conditions of most regular in-group assets are acquired through periodic inspection work. Due to their large quantities, significant amount of manual inspection effort and sometimes data management issues, it is not uncommon to see the asset condition data in a target study area becomes unavailable or incomplete. Lack of asset condition data undermines the reliability assessment work. This paper tackles this important problem from an unconventional way – it explores how to generate numerical and non-numerical asset condition data based on condition degradation, condition correlation and categorical distribution models. Empirical knowledge from human experts can also be incorporated in the modeling process. Also, a probabilistic diversification step can be taken to make the generated numerical condition data probabilistic. This method can generate close-to-real asset condition data and has been validated systematically based on two public datasets. An area reliability assessment example based on cables is given to demonstrate the usefulness of this method and its generated data. This method can also be used to conveniently generate hypothetical asset condition data for research purposes. [ABSTRACT FROM AUTHOR]

With 5G on the verge of being adopted as the next mobile network, there is a need to analyze its impact on the landscape of computing and data management. In this paper, we analyze the impact of 5G on both traditional and emerging technologies and project our view on future research challenges and opportunities. With a predicted increase of 10-100x in bandwidth and 5-10x decrease in latency, 5G is expected to be the main enabler for smart cities, smart IoT and efficient healthcare, where machine learning is conducted at the edge. In this context, we investigate how 5G can help the development of federated learning. Network slicing, another key feature of 5G, allows running multiple isolated networks on the same physical infrastructure. However, security remains the main concern in the context of virtualization, multi-tenancy and high device density. Formal verification of 5G networks can be applied to detect security issues in massive virtualized environments. In summary, 5G will make the world even more densely and closely connected. What we have experienced in 4G connectivity will pale in comparison to the vast amounts of possibilities engendered by 5G. [ABSTRACT FROM AUTHOR]

Wireless edge networks in smart industrial environments increasingly operate using advanced sensors and autonomous machines interacting with each other and generating huge amounts of data. Those huge amounts of data are bound to make data management (e.g., for processing, storing, computing) a big challenge. Current data management approaches, relying primarily on centralized data storage, might not be able to cope with the scalability and real time requirements of Industry 4.0 environments, while distributed solutions are increasingly being explored. In this paper, we introduce the problem of distributed data access in multi-hop wireless industrial edge deployments, whereby a set of consumer nodes needs to access data stored in a set of data cache nodes, satisfying the industrial data access delay requirements and at the same time maximizing the network lifetime. We prove that the introduced problem is computationally intractable and, after formulating the objective function, we design a two-step algorithm in order to address it. We use an open testbed with real devices for conducting an experimental investigation on the performance of the algorithm. Then, we provide two online improvements, so that the data distribution can dynamically change before the first node in the network runs out of energy. We compare the performance of the methods via simulations for different numbers of network nodes and data consumers, and we show significant lifetime prolongation and increased energy efficiency when employing the method which is using only decentralized low-power wireless communication instead of the method which is using also centralized local area wireless communication. [ABSTRACT FROM AUTHOR]

The emerging distribution-level phasor measurement unit (D-PMU) is expected to play an important role in enhancing distribution system observability and situational-awareness. It is a demanding yet challenging task to develop advanced D-PMU data management and analytics tools to improve D-PMU data usability and further promote D-PMU projects deployment. This paper focuses on D-PMU data processing and storage. It presents a brief review of existing D-PMU data storage systems and points out their limitations on high-performance, flexibility, and scalability. To overcome the limitations, a unified PMU-data storage system (UPS) is proposed. Specifically, a unified I/O interface between storage servers and computing jobs is developed to effectively reduce the overhead of managing various computing jobs and data analytics over multiple storage infrastructures; and PMUCache with PMUCache partitioning and PMUCache replacement algorithms are designed to support in-situ data processing and shared distributed data storage and further serve the computing jobs/queries with different quality of service (QoS) requirements. Through a series of experiments, it is demonstrated that UPS achieves high performance on fast and big data processing and storage, and efficiently increases the flexibility and scalability of PMU data management systems. [ABSTRACT FROM AUTHOR]

This paper presents a distributed synchrophasor data management (DSDM) technique for wide-area monitoring systems. The proposed technique focuses on managing voltage/current phasor data measured from a specific target bus by defining neighboring phasor measurement units (PMUs). We developed an algorithm for searching neighboring PMUs by interpreting the power network topology as a graph structure. By exploiting the data of neighboring PMUs, we constructed a multi-channel structure to recover low-quality entries by using the parallel implementation of long short-term memory networks. A successive output decision-making rule automatically determines the final output, considering the occurrence of defective input, data quality of the measured data, and operational conditions. The proposed DSDM can complement existing centralized management techniques by preserving information on local dynamics. In particular, because of the multi-channel structure, DSDM can provide robust responses to defective data in neighboring PMUs, such as an unintended change in a network topology. We verified the effectiveness of the DSDM with real-world synchrophasor data as well as simulated data from the IEEE 68-bus test system. [ABSTRACT FROM AUTHOR]

Data quality plays a key role in big data management today. With the explosive growth of data from a variety of sources, the quality of data is faced with multiple problems. Motivated by this, we study the multiple data cleaning on incompleteness and inconsistency with currency reasoning and determination in this paper. We introduce a 4-step framework, named ${\sf Imp3C}$ Imp 3 C , for errors detection and quality improvement in incomplete and inconsistent data without timestamps. We achieve an integrated currency determining method to compute the currency orders among tuples, according to currency constraints. Thus, the inconsistent data and missing values are repaired effectively considering the temporal impact. For both effectiveness and efficiency consideration, we carry out inconsistency repair ahead of incompleteness repair. A currency-related consistency distance metric is defined to measure the similarity between dirty tuples and clean ones more accurately. In addition, currency orders are treated as an important feature in the missing imputation training process. The solution algorithms are introduced in detail with case studies. A thorough experiment on three real-life datasets verifies our method ${\sf Imp3C}$ Imp 3 C improves the performance of data repairing with multiple quality problems. ${\sf Imp3C}$ Imp 3 C outperforms the existing advanced methods, especially in the datasets with complex currency orders. [ABSTRACT FROM AUTHOR]

In mobile communication networks, when a user roams to and accesses a foreign network, the foreign operator needs to request the user’s subscription data from a centralized authentication server, which is managed by the user’s home operator. However, centralized authentication introduces single point of failure. Meanwhile, the real-time participation of the home operator and the trust relationship between the foreign operator and the home operator are difficult to guarantee. In this paper, by adopting blockchain and smart contracts, we propose a secure and efficient access control scheme of user subscription data in roaming scenarios. We further design a flexible user authentication scheme, which utilizes derivable tokens based on the proposed access control scheme. By using blockchain to store and manage user subscription data, access control can be decentralized without any trusted third party. Besides, by implementing automatic verification of access privilege through smart contracts, the limitation of the home operators’ real-time participation is eliminated. In addition, to further improve security and reduce the on-chain storage overhead, we optimize the data encryption and storage scheme utilizing threshold secret sharing. Our security and performance analysis show that the proposed user subscription data access control scheme for roaming service provides high-level security while causing acceptable time and storage overhead. [ABSTRACT FROM AUTHOR]

5G networks and beyond introduce a larger number of Network Elements (NEs) and functions than former cellular generations. The increase in NEs will, thus, result in significantly increasing the Management-Plane (M-Plane) data collected from the NEs. Therefore, the conventional centralized Network Management Systems (NMSs) will face fundamental challenges in processing the M-Plane data. In this paper, we present the concept of Quality of Monitoring (QoM) as a solution, which is able to reduce the M-Plane data already at the NEs. First, QoM aggregates the raw M-Plane data into Key Performance Indicators (KPIs). To these KPIs, the QoM applies a data-driven algorithm to define information loss limits for QoM classes specific for each KPI time series. Then, the QoM applies the classes for compressing the KPI data utilizing a lossy-compression method, which is a derivative of the Piece-Wise Constant Approximation (PWCA) algorithm. To evaluate the performance of the QoM solution, we use M-Plane raw data from a live LTE network and calculate four KPIs, while each KPI has different statistical characteristics. We also define three QoM classes named Exact, Optimized, and Sharp. For all KPIs, the class Optimized has a higher compression rate than the class Exact, while the class Sharp has the highest compression rate. Assuming that, for example, NEs of a network produce 280 MB of raw data containing information that needs to be transferred to the network operations center; we use KPIs to represent the information contents of the data, and QoM solution to transfer the data over the network. As a result, the QoM solution achieves an estimated 95% compression gain from the raw data in transfer. [ABSTRACT FROM AUTHOR]

The important topic of global climate change builds on a huge collection of scientific research. It is common for agencies releasing climate change information to be served with requests for all supporting materials resulting in a particular conclusion. Capturing and presenting global change provenance, linking to the research papers, data sets, models, analyses, observations, satellites, etc., that support the key research findings in this domain can increase understanding and aid in reproducibility of results and conclusions. The U.S. Global Change Research Program is now coordinating the production of a national climate assessment (NCA) that presents our best understanding of global change. We are now developing a global change information system that will present the content of that report and its provenance, including the scientific support for the findings of the assessment. We are using an approach that will present this information both through a human accessible Web site as well as a machine-readable interface for automated mining of the provenance graph. We plan to use the developing World Wide Web Consortium (W3C) PROV data model and ontology for this system. This paper will describe an overview of the process of developing the NCA and how the provenance trail of the report and each of the technical inputs can be captured and represented using the W3C PROV ontology. This will improve the visibility into the assessment process, increase understanding and possibility of reproducibility, and ultimately increase the credibility and trust of the resulting report. [ABSTRACT FROM AUTHOR]

Electric power systems are going through a major upgrade with the integration of advanced technologies to enable the smarter electric grid (SEG). The SEG will use information and communications technology to have enhanced controllability and will become more interactive. This ongoing change also necessitates educating professionals and future generation of engineers to manage evolving complexity of the electric grid. This paper presents experiences in the design and teaching of a unique multidisciplinary team-taught course on smart grid cyber infrastructure to provide a new generation of engineers with a solid foundation of smart grid concepts and their associated challenges. This paper identifies the course topics covered, learning objectives, and assessment activities for the class as well as lessons learned based on course evaluations obtained from multiple offerings. All of the course material is available in public domain and can be easily adopted at another educational institution. [ABSTRACT FROM AUTHOR]

In Australia and many other countries, distribution network service providers (DNSPs) have an obligation to their customers to provide electrical power that is reliable and of high quality. Failure to do so may have significant implications ranging from financial penalties theoretically through to the loss of a license to distribute electricity. In order to ensure the reliability and quality of supply are met, DNSPs engage in monitoring and reporting practice. This paper provides an overview of a large long-running power-quality monitoring project that has involved most of Australia's DNSPs at one time or another. This paper describes the challenges associated with conducting the project as well as some of the important outcomes and lessons learned. A number of novel reporting techniques that have been developed as part of the monitoring project are also presented. A discussion about large-volume data management, and issues related to reporting requirements in future distribution networks is included. [ABSTRACT FROM PUBLISHER]

In the process of developing an infrastructure for providing visualization and visual analytics (VIS) tools to epidemiologists and modeling scientists, we encountered a technical challenge for applying a number of visual designs to numerous datasets rapidly and reliably with limited development resources. In this paper, we present a technical solution to address this challenge. Operationally, we separate the tasks of data management, visual designs, and plots and dashboard deployment in order to streamline the development workflow. Technically, we utilize: an ontology to bring datasets, visual designs, and deployable plots and dashboards under the same management framework; multi-criteria search and ranking algorithms for discovering potential datasets that match a visual design; and a purposely-design user interface for propagating each visual design to appropriate datasets (often in tens and hundreds) and quality-assuring the propagation before the deployment. This technical solution has been used in the development of the RAMPVIS infrastructure for supporting a consortium of epidemiologists and modeling scientists through visualization. [ABSTRACT FROM AUTHOR]

The introduction of ADS-B, a satellite-based aircraft tracking technology, and the increasing installation of ADS-B receiver stations around the globe eases the tracking of aircraft, compared with traditional solutions using secondary radar. Given the large scale of ADS-B implementation and the high frequency of data collection, storing and managing ADS-B induced data has become increasingly difficult: The worldwide ADS-B data easily aggregates to several hundreds of terabyte per year, depending on the spatial coverage and temporal resolution. Standard data management solutions do not work well for ADS-B data, since they either require a large uncompressed index structure or cannot be queried efficiently. In this paper, we propose a novel compressed index structure for managing ADS-B data, called ADS-BI. The essential building blocks are spatio-temporal reference partitioning, reordering, and compression. On top of the partitioned, compressed representation, metadata is stored effectively, and exploited during query answering for typical ATM related task such as trajectory adherence evaluation, as well as complexity and safety metrics assessment by only accessing parts of the compressed data as necessary. Our novel index structure is evaluated on worldwide ADS-B data for a week in November 2016. For comparison, we implemented ten standard compression/indexing methods. The experiments reveal that none of these traditional methods can target the sweet spot between a small storage and efficient query answering. Our novel technique provides fast query answering at smallest storage costs. This paper contributes toward efficient handling of the increasing amount of traffic data in air traffic management, and eventually, toward more efficient and safer air transportation. [ABSTRACT FROM AUTHOR]

This paper reports the results and post-challenge analyses of ChaLearn’s AutoDL challenge series, which helped sorting out a profusion of AutoML solutions for Deep Learning (DL) that had been introduced in a variety of settings, but lacked fair comparisons. All input data modalities (time series, images, videos, text, tabular) were formatted as tensors and all tasks were multi-label classification problems. Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly. In this setting, DL methods dominated, though popular Neural Architecture Search (NAS) was impractical. Solutions relied on fine-tuned pre-trained networks, with architectures matching data modality. Post-challenge tests did not reveal improvements beyond the imposed time limit. While no component is particularly original or novel, a high level modular organization emerged featuring a “meta-learner”, “data ingestor”, “model selector”, “model/learner”, and “evaluator”. This modularity enabled ablation studies, which revealed the importance of (off-platform) meta-learning, ensembling, and efficient data management. Experiments on heterogeneous module combinations further confirm the (local) optimality of the winning solutions. Our challenge legacy includes an ever-lasting benchmark (http://autodl.chalearn.org), the open-sourced code of the winners, and a free “AutoDL self-service.” [ABSTRACT FROM AUTHOR]

Due to the growing popularity of indoor location-based services, indoor data management has received significant research attention in the past few years. However, we observe that the existing indexing and query processing techniques for the indoor space do not fully exploit the properties of the indoor space. Consequently, they provide below par performance which makes them unsuitable for large indoor venues with high query workloads. In this paper, we first propose two novel indexes called Indoor Partitioning Tree (IP-Tree) and Vivid IP-Tree (VIP-Tree) that are carefully designed by utilizing the properties of indoor venues. The proposed indexes are lightweight, have small pre-processing cost and provide near-optimal performance for shortest distance and shortest path queries. We are also the first to study spatial keyword queries in indoor venues. We propose a novel data structure called Keyword Partitioning Tree (KP-Tree) that indexes objects in an indoor partition. We propose an efficient algorithm based on VIP-Tree and KP-Trees to efficiently answer spatial keyword queries. Our extensive experimental study on real and synthetic data sets demonstrates that our proposed indexes outperform the existing solutions by several orders of magnitude. [ABSTRACT FROM AUTHOR]

In this paper, we investigate threshold effects associated with the swapping of signal and noise subspaces in estimating signal parameters from compressed noisy data. The term threshold effect refers to a sharp departure of mean-squared error from the Cramér–Rao bound when the signal-to-noise ratio falls below a threshold SNR. In many cases, the threshold effect is caused by a subspace swap event, when the measured data (or its sample covariance) is better approximated by a subset of components of an orthogonal subspace than by the components of a signal subspace. We derive analytical lower bounds on the probability of a subspace swap in compressively measured noisy data in two canonical models: a first-order model and a second-order model. In the first-order model, the parameters to be estimated modulate the mean of a complex multivariate normal set of measurements. In the second-order model, the parameters modulate the covariance of complex multivariate measurements. In both cases, the probability bounds are tail probabilities of F-distributions, and they apply to any linear compression scheme. These lower bounds guide our understanding of threshold effects and performance breakdowns for parameter estimation using compression. In particular, they can be used to quantify the increase in threshold SNR as a function of a compression ratio C. We demonstrate numerically that this increase in threshold SNR is roughly 10\log10 \ C dB, which is consistent with the performance loss that one would expect when measurements in Gaussian noise are compressed by a factor C. [ABSTRACT FROM PUBLISHER]

This paper describes the design and the architecture of symbiote coprocessor unit (SCU)—a programmable streaming coprocessor for a heterogeneous reconfigurable logic-assisted data stream management systems (DSMSs) such as symbiote. The SCU is intended for streaming applications with real-time event and data processing that have stricter deadlines, high-bandwidth, and high-accuracy requirements. To meet these requirements, the SCU exploits unique characteristics of DSMSs, such as single-pass tuple processing, windowed operators, and inherent data level parallelism, using a single-instruction multiple-data very large instruction word (SIMD-VLIW) microarchitecture and a novel inverted distributed register file. In order to better explain the instruction set, design, and functionality of the various units in the SCU, this paper also provides a brief overview of SymQL—a procedural query language that we developed to describe the queries that can be executed on the SCU. Finally, this paper presents the performance of SCU using four queries that represent common data stream processing use-cases, one of them being similar to a query found in the Linear Road Benchmark. Using these queries on SCU simulation, we show that the SCU outperforms a software-only DSMS running on an AMD Opteron 2350 quad-core processor by 1.5–42 times. [ABSTRACT FROM PUBLISHER]

Wind turbine blades have been constantly increasing since wind energy becomes a popular renewable energy source to generate electricity. Therefore, the wind sector requires a more efficient and representative characterization of vertical wind speed profiles to assess the potential for a wind power plant site. This paper proposes an alternative characterization of vertical wind speed profiles based on Ward's agglomerative clustering algorithm, including both wind speed module and direction data. This approach gives a more accurate incoming wind speed variation around the rotor swept area, and subsequently, provides a more realistic and complete wind speed vector characterization for vertical profiles. Real wind database collected for 2018 in the Forschungsplattformen in Nordund Ostsee (FINO) research platform is used to assess the methodology. A preliminary pre-processing stage is proposed to select the appropriated number of heights and remove missing or incomplete data. Finally, two locations and four heights are selected, and 561588 wind data are characterized. Results and discussion are also included in this paper. The methodology can be applied to other wind database and locations to characterize vertical wind speed profiles and identify the most likely wind data vector patterns.

Mining digitalisation have been receiving significant attention due to the utilisation of advanced technologies, such as IoT, automation, and sensing. However, maximising the potential value of collected data in the mining industry remains a challenge. Therefore, this paper aims to review timely concern topics to facilitate the fusion implementation in mining engineering. Specifically, this review covers recent popular topics, such as, data visualisation, data management, data analytics, data fusion, visual analytics, and mining digital twin construction. In this paper, we aim to draw a comprehensive picture about the fusion of data visualisation and analytics in the big data context, by examining the recent academic research related to these topics. Therefore, this paper reviews the visualisation domain by conventional classification, including scientific visualisation, information visualisation, and visual analytics, associated with the analysis of current digital twin development. Next, according to the challenges and issues related to visualisation development, this paper reviews the data management and data analytics domains as well. Incorporating with the fusion concept, machine learning-oriented fusion applications and potential scenarios in the mining industry have been discussed. In addition, based on the observation across various domains, this paper presents challenges and future potentials of data fusion in mining.

Advanced metering infrastructure (AMI) initiatives are a popular tool to incorporate changes for modernizing the electricity grid, reduce peak loads, and meet energy efficiency targets. There is the looming issue of how to communicate and handle consumer data collected by electric utilities and manage limited communication network resources. Several data relay points are required to collect data distributedly and send them through a communication backhaul. This paper studies the smart meter message concatenation (SMMC) problem of how to efficiently concatenate multiple small smart metering messages arriving at data concentrator units in order to reduce protocol overhead and thus network utilization. This problem needs to deal with the added constraint that each originating message from its source may have its own stated deadline that must be taken into account during the concatenation process. This paper provides hardness results for the SMMC problem, and proposes six heuristics and evaluates them to gain a better understanding of the best data volume reduction policies that can be applied at data concentrators of AMI infrastructures. These results are further tested for feasibility under practical settings based on aspects, such as network and processing delays, tightness of application deadlines, and lossy backhaul links. [ABSTRACT FROM PUBLISHER]

Random projection has been widely used in data classification. It maps high-dimensional data into a low-dimensional subspace in order to reduce the computational cost in solving the related optimization problem. While previous studies are focused on analyzing the classification performance in the low-dimensional space, in this paper, we consider the recovery problem, i.e., how to accurately recover the optimal solution to the original high-dimensional optimization problem based on the solution learned after random projection. We present a simple algorithm, termed dual random projection, which uses the dual solution of the low-dimensional optimization problem to recover the optimal solution to the original problem. Our theoretical analysis shows that with a high probability, the proposed algorithm is able to accurately recover the optimal solution to the original problem, provided that the data matrix is (approximately) low-rank and/or optimal solution is (approximately) sparse. We further show that the proposed algorithm can be applied iteratively to reducing the recovery error exponentially. [ABSTRACT FROM PUBLISHER]

For over many decades, relational database model has been considered as the leading model for data storage and management. However, as the Big Data explosion has generated a large volume of data, alternative models like NoSQL and NewSQL have emerged. With the advancement of communication technology, these database systems have given the potential to change the existing architecture from centralized mechanism to distributed in nature, to deploy as cloud-based solutions. Though all of these evolving technologies mostly focus on performance guarantees, it is still being a major concern how these systems can ensure the security and privacy of the information they handle. Different datastores support different types of integrated security mechanisms, however, most of the non-relational database systems have overlooked the security requirements of modern Big Data applications. This paper reviews security implementations in today's leading database models giving more emphasis on security and privacy attributes. A set of standard security mechanisms have been identified and evaluated based on different security classifications. Further, it provides a thorough review and a comprehensive analysis on maturity of security and privacy implementations in these database models along with future directions/enhancements so that data owners can decide on most appropriate datastore for their data-driven Big Data applications. [ABSTRACT FROM AUTHOR]

Emergency Medical Services (EMS) are of great importance to saving people's lives from emergent accidents and diseases by efficiently picking up patients using ambulances. The transporting capability of an EMS system (e.g., defined as the average pickup time of patients) significantly depends on the real-time redeployment strategy of ambulances. That is, which station should an ambulance be redeployed to, after it becomes available (after it transports a patient to a hospital or after it finishes the in-site treatment for a patient)? However, it is a challenging task concerning with the multiple data D1-D5 as detailed in Introduction. To this end, in this paper, we propose a data-driven real-time ambulance redeployment approach that redeploys an ambulance to a proper station after it becomes available, so as to optimize the transporting capability of an EMS system, considering the aforementioned multiple data D1-D5. Specifically, the proposed approach is comprised of two stages to well consider the D1-D5. First, we propose a method (a safety time-based urgency index) to incorporate D1, D2, and D3 into each ambulance station's urgency degree (D*). Second, we propose an optimal matching algorithm to combine D*, D4, and D5 into the redeployment of the current available ambulance. Experimental results using data collected in real world demonstrate the significant advantages of our approach over many baselines. Comparing with baselines, our approach can save $\sim$ ∼ 4 minutes ($\sim$ ∼ 35 percent) of the average pickup time for each patient, improve the ratio of patients picked up within 10 minutes from 0.684 and 0.803 ($\sim$ ∼ 17 percent), and largely enhance the survival rate of patients ($\sim$ ∼ 12 percent for patients in category A1 and $\sim$ ∼ 17 percent for patients in A2). [ABSTRACT FROM AUTHOR]

Although many active scientific codes use modern Fortran, most contemporary scientific software libraries are implemented in C and C++. Providing their numerical, algorithmic, or data management features to Fortran codes requires writing and maintaining substantial amounts of glue code. This paper introduces a tool that automatically generates native Fortran 2003 interfaces to C and C++ libraries. The tool supports C++ features that have no direct Fortran analog, such as templated functions and exceptions. A set of simple examples demonstrates the utility and scope of the tool, and timing measurements with a mock numerical library illustrate the minimal performance impact of the generated wrapper code. [ABSTRACT FROM AUTHOR]

In order to provide benchmark data sets for radiation detector and algorithm development, a particle transport test bed has been created using experimental data as model input and validation. A detailed radiation measurement campaign at the Combined Arms Collective Training Facility in Fort Indiantown Gap, PA (FTIG), USA, provides sample background radiation levels for a variety of materials present at the site (including cinder block, gravel, asphalt, and soil) using long dwell high-purity germanium (HPGe) measurements. In addition, detailed light detection and ranging data and ground-truth measurements inform model geometry. This paper describes the collected data and the application of these data to create background and injected source synthetic data for an arbitrary gamma-ray detection system using particle transport model detector response calculations and statistical sampling. In the methodology presented here, HPGe measurements inform model source terms while detector response calculations are validated via long dwell measurements using 2” $\times 4$ ” $\times 16$ ” NaI(Tl) detectors at a variety of measurement points. A collection of responses, along with sampling methods and interpolation, can be used to create data sets to gauge radiation detector and algorithm (including detection, identification, and localization) performance under a variety of scenarios. Data collected at the FTIG site are available for query, filtering, visualization, and download at muse.lbl.gov. [ABSTRACT FROM PUBLISHER]

Load balancing is a widely accepted technique for performance optimization of scientific applications on parallel architectures. Indeed, balanced applications do not waste processor cycles on waiting at points of synchronization and data exchange, maximizing this way the utilization of processors. In this paper, we challenge the universality of the load-balancing approach to optimization of the performance of parallel applications. First, we formulate conditions that should be satisfied by the performance profile of an application in order for the application to achieve its best performance via load balancing. Then we use a real-life scientific application, EULAG MPDATA kernel, to demonstrate that its performance profile on a modern parallel architecture, Intel Xeon Phi, significantly deviates from these conditions. Based on this observation, we propose a method of performance optimization of scientific applications through load imbalancing. In the case of data parallel application, the method uses functional performance models of the application to find partitioning that minimizes its computation time but not necessarily balances the load of processors. We apply this method to optimization of MPDATA on Intel Xeon Phi. Experimental results demonstrate that the performance of this carefully optimized load-balanced application can be further improved by 15percent using the proposed load-imbalancing technique. [ABSTRACT FROM PUBLISHER]

With the development of the economy, products are significantly enriched, and uncertainty has been their inherent quality. The probabilistic dynamic skyline (PDS) query is a powerful tool for customers to use in selecting products according to their preferences. However, this query suffers several limitations: it requires the specification of a probabilistic threshold, which reports undesirable results and disregards important results; it only focuses on the objects that have large dynamic skyline probabilities; and, additionally, the results are not stable. To address this concern, in this paper, we formulate an uncertain dynamic skyline (UDS) query over a probabilistic product set. Furthermore, we propose effective pruning strategies for the UDS query, and integrate them into effective algorithms. In addition, a novel query type, namely the top κ favorite probabilistic products (TFPP) query, is presented. The TFPP query is utilized to select κ products which can meet the needs of a customer set at the maximum level. To tackle the TFPP query, we propose a TFPP algorithm and its efficient parallelization. Extensive experiments with a variety of experimental settings illustrate the efficiency and effectiveness of our proposed algorithms. [ABSTRACT FROM PUBLISHER]

Software-Defined Datacenters (SDDC) have been widely used for load-aware data management for different applications across the globe. Due to its centralized architecture, the issues of scalability along with resilience (to overcome the failure of single or multiple controllers) are still challenging because of an exponential increase in the data generated from different smart devices. Most of the solutions reported in the literature for this problem use a single controller which may not address the scalability issues. However, the issues as mentioned above of scalability and resilience in SDDC can be solved by deploying multiple distributed controllers at the control plane. However, the primary concern in a network having various controllers is the optimal Controller Placement Problem (CPP) to resolve the issues of fault-tolerance, latency among controllers, availability, and placement. Hence, to resolve the issues described above, in this paper, we propose Placement Availability Resilient Controller (PARC) scheme. The PARC scheme works in the following four phases: (i) stable network partitioning (ii) localization of controllers using the cooperative game theory (iii) computation of an optimal number of multiple controllers and (iv) computation of minimal extra backup controllers to improve the overall network cost. The numerical results of the PARC scheme are evaluated on Internet2 OS3E topology using POCO-toolset simulated in Matlab. The experimental results demonstrated that the cost of deploying the number of controllers using the PARC scheme has reduced to 12.98%, 8.16%, and 6.25% as compared to the POCO-SA, POCO-MOALO, and CNCP schemes respectively. Moreover, the PARC scheme outperforms the existing state-of-the-art schemes (POCO-SA, POCO-MOALO, and CNCP) for inter-controller as well as switch-to-controller latency. [ABSTRACT FROM AUTHOR]

In this paper, we study the problem of data flow management in the presence of heterogeneous flows — elephant and mice flows — in software-defined networks (SDNs). Most of the researchers considered the homogeneous flows in SDN in the existing literature. The optimal data flow management in the presence of heterogeneous flows is NP-hard. Hence, we propose a game theory-based heterogeneous data flow management scheme, named FlowMan. In FlowMan, initially, we use a generalized Nash bargaining game to obtain a sub-optimal problem, which is NP-complete in nature. By solving it, we get the Pareto optimal solution for data-rate associated with each switch. Thereafter, we use a heuristic method to decide the flow-association with the switches, distributedly, which, in turn, helps to get a Pareto optimal solution. Extensive simulation results depict that FlowMan is capable of ensuring quality-of-service (QoS) for data flow management in the presence of heterogeneous flows. In particular, FlowMan is capable of reducing network delay by 77.8–98.7%, while ensuring 24.6–47.8% increase in network throughput, compared to the existing schemes such as FlowStat and CURE. Additionally, FlowMan ensures that per-flow delay is reduced by 27.7% with balanced load distribution among the SDN switches. [ABSTRACT FROM AUTHOR]

There has been an increasing trend of moving computing activities closer to the edge of the network, particularly in smart city applications (e.g., vehicle-to-everything – V2X). Such a paradigm allows the end user’s requests to be handled/processed by nodes at the edge of the network; thus, reducing latency, and preserving privacy of user data/activities. However, there are a number of challenges in such an edge computing ecosystem. Examples include (1) potential inappropriate utilization of resources at the edge nodes, (2) operational challenges in cache management and data integrity due to data migration between edge nodes, particularly when dealing with vehicular mobility in a V2X application, and (3) high energy consumption due to continuous link breakage and subsequent reestablishment of link(s). Therefore in this paper, we design a blockchain-based secure data processing framework for an edge envisioned V2X environment (hereafter referred to as BloCkEd). Specifically, a multi-layered edge-enabled V2X system model for BloCkEd is presented, which includes the formulation of a multi-objective optimization problem. In addition, BloCkEd comprises an optimal container-based data processing scheme, and a blockchain-based data integrity management scheme, designed to minimize link breakage and reducing latency. Using Chandigarh City, India, as the scenario, we implement and evaluate the proposed approach in terms of its latency, energy consumption, and service level agreement compliance. [ABSTRACT FROM AUTHOR]

Melt pool monitoring in selective laser melting (SLM) is a key challenge to enhance the understanding of the basic process physics, as well as the in situ identification of drifts. The main issues are related to the molten pool’s fast dynamics and small spatial extent, which require high-performance monitoring systems in terms of temporal and spatial resolutions, often resulting in unmanageable data rates. Furthermore, the broadband thermal emission from the process can be viewed in different spectral ranges depending on the sensor and optical system employed. Accordingly, a wide range of sensing configurations is possible. The choice of monitoring parameters is crucial to achieve a solution capable of describing process dynamics with industrial applicability in terms of data transport and analysis. This paper discusses the design of a coaxial monitoring module to assess melt pool dynamics in SLM using temporal and spatial cues related to the process physics. Restrictions regarding data management were considered to ensure future inline process control. The system was tested with an external illuminator to reveal the actual melt pool geometry employing an acquisition rate of 1200 frames per second, 4.3 mm $\times $ 4.3 mm field of view, and $14~\mu \text{m}$ /pixel spatial resolution. The process emission in visible (640 nm) and near-infrared regions (850–1000 nm) was also acquired and the band choice discussed. The proposed solution captured successfully the melting conditions from both spatial and temporal viewpoints. The monitoring system depicted the melt pool shape variations when processing different geometries using both modulated and continuous wave laser emissions. [ABSTRACT FROM AUTHOR]

The proliferation of radio frequency identification (RFID) systems in application domains such as supply chain management requires an IT infrastructure that provides RFID device and data management and supports application development. In this paper, we discuss these application requirements in detail. We also contend that the characteristics of passive RFID technology introduce constraints that are unique to the development of middleware for the RFID domain. These constraints include the occurrence of false negative reads, tag memory variations, the heterogeneous reader landscape, and the limited communication bandwidth available to RFID readers. To address these constraints and the application requirements for filtered and aggregated RFID data, we developed Accada, an open source RFID platform. This paper shows that the Accada implementation, which is based on a set of specifications developed by the EPCglobal community and a number of extensions, such as the surrogate concept and the virtual tag memory service, addresses the majority of the application requirements and limitations of passive RFID technology. [ABSTRACT FROM PUBLISHER]

Data caching at mobile clients is an important technique for improving the performance of wireless data dissemination systems. However, variable data sizes, data updates, limited client resources, and frequent client disconnections make cache management a challenge. In this paper, we propose a gain-based cache replacement policy, Min-SAUD, for wireless data dissemination when cache consistency must be enforced before a cached item is used. Min-SAUD considers several factors that affect cache performance, namely, access probability, update frequency, data size, retrieval delay, and cache validation cost. This paper employs stretch as the major performance metric since it accounts for the data service time and, thus, is fair when items have different sizes. We prove that Min-SAUD achieves optimal stretch under some standard assumptions. Moreover, a series of simulation experiments have been conducted to thoroughly evaluate the performance of Min-SAUD under various system configurations. The simulation results show that, in most cases, the Min-SAUD replacement policy substantially outperforms two existing policies, namely, LRU and SAIU. [ABSTRACT FROM AUTHOR]

Entity-attribute-value (EAV) tables are widely used to store data in electronic medical records and clinical study data management systems. Before they can be used by various analytical (e.g., data mining and machine learning) programs, EAV-modeled data usually must be transformed into conventional relational table format through pivot operations. This time-consuming and resource-intensive process is often performed repeatedly on a regular basis, e.g., to provide a daily refresh of the content in a clinical data warehouse. Thus, it would be beneficial to make pivot operations as efficient as possible. In this paper, we present three techniques for improving the efficiency of pivot operations: 1) filtering out EAV tuples related to unneeded clinical parameters early on; 2) supporting pivoting across multiple EAV tables; and 3) conducting multi-query optimization. We demonstrate the effectiveness of our techniques through implementation. We show that our optimized execution method of pivoting using these techniques significantly outperforms the current basic execution method of pivoting. Our techniques can be used to build a data extraction tool to simplify the specification of and improve the efficiency of extracting data from the EAV tables in electronic medical records and clinical study data management systems. [ABSTRACT FROM AUTHOR]

In resource-constrained embedded systems, on-chip cache memories play an important role in both performance and energy consumption. In contrast to read operations, scant regard has been paid to optimizing write operations even though the energy consumed by write operations in the data cache constitutes a large portion of the total energy consumption. Consequently, this paper proposes a write buffer-oriented (WO) cache architecture that reduces energy consumption in the L1 data cache. Observing that write operations are very likely to be merged in the write buffer because of their high localities, we construct the proposed WO cache architecture to utilize two schemes. First, the write operations update the write buffer but not the L1 data cache, which is updated later by the write buffer after the write operations are merged. Write merging significantly reduces write accesses to the data cache and, consequently, energy consumption. Second, we further reduce energy consumption in the write buffer by filtering out unnecessary read accesses to the write buffer using a read hit predictor. In this paper, we show that the proposed WO cache architecture is applicable to the conventional embedded processors that support both write-through and write-back policies. Further, the experimental results verify that the proposed cache architecture reduces energy consumption in data caches up to 14%. [ABSTRACT FROM PUBLISHER]

Query processing over uncertain data has gained growing attention, because it is necessary to deal with uncertain data in many real-life applications. In this paper, we investigate skyline queries over uncertain data in distributed environments (DSUD query) whose research is only in an early stage. The state-of-the-art algorithm, called e-DSUD algorithm, is designed for processing this query. It has the desirable characteristics of progressiveness and minimum bandwidth consumption. However, it still needs to be perfected in three aspects. (1) Progressiveness. Each time it only returns one query result at most. (2) Efficiency. There are a significant amount of redundant I/O cost and numerous iterations which causes a long total query time. (3) Universality. It is restricted to the case where local skyline tuples are incomparability. To address these concerns, we first present a detailed analysis of the e-DSUD algorithm and then develop an improved framework for the DSUD query, namely IDSUD. Based on the new framework, we propose an adaptive algorithm, called ADSUD, for the DSUD query. In the algorithm, we redefine the approximate global skyline probability and choose local representative tuples due to minimum probabilistic bounding rectangle adaptively. Furthermore, we design a progressive pruning method and apply the reuse mechanism to improve its efficiency. The results of extensive experiments verify the better overall performance of our algorithm than the e-DSUD algorithm. [ABSTRACT FROM AUTHOR]

Data traffic in cellular networks has dramatically increased in recent years as the emergence of various new wireless applications, which imposes an immediate requirement for large network capacity. Although many efforts have been made to enhance wireless channel capacity, they are far from solving the network capacity enhancement problem. Device-to-Device (D2D) communication is recently proposed as a promising technique to increase network capacity. However, most existing work on D2D communications focuses on optimizing throughput or energy efficiency, without considering economic issues. In this paper, we propose a truthful double auction for D2D communications (TAD) in multi-cell cellular networks for trading resources in frequency-time domain, where cellular users with D2D communication capability act as sellers, and other users waiting to access the network act as buyers. Both intra-cell and inter-cell D2D sellers are accommodated in TAD while the competitive space in each cell is extensively exploited to achieve a high auction efficiency. With a sophisticated seller-buyer matching, winner determination and pricing, TAD guarantees individual rationality, budget balance, and truthfulness. Furthermore, we extend our TAD design to handle a more general case that each seller and buyer ask/bid multiple resource units. Extensive simulation results show that TAD can achieve truthfulness as well as high performance in terms of seller/buyer sanctification ratio, auctioneer profit and network throughput. [ABSTRACT FROM AUTHOR]

Group skyline query is a powerful tool for optimal group analysis. Most of the existing group skyline queries select optimal groups by comparing the dominance relationship between aggregate-based points; such feature creates difficulties for users to specify an appropriate aggregate function. Besides, many significant groups that have great attractions to users in practice may be overlooked. To address these issues, the group skyline (GSky) query is formulated on the basis of a general definition of group dominance operator. While the existing GSky query algorithms are effective, there is still room for improvement in terms of progressiveness and efficiency. In this paper, we propose some new lemmas which facilitate direct generation of the GSky query results. Consecutively, we design a layered unit-based (LU) algorithm that applies a layered optimum strategy. Additionally, for the GSky query over the data that are dynamically produced and cannot be indexed, we propose a novel index-independent algorithm, called sorted-based progressive (SP) algorithm. The experimental results demonstrate the effectiveness, efficiency, and progressiveness of the proposed algorithms. By comparing with the state-of-the-art algorithm for the GSky query, our LU algorithm is more scalable and two orders of magnitude faster. [ABSTRACT FROM AUTHOR]

The inclusion of various intelligent electronic devices such as smart meters for AMI is expected to result in intermittent or frequent communications network congestion if additional network infrastructure investments are not made. One approach to deal with such a data volume challenge in smart grids without additional investments to increase network capacity is to aggregate data streams within the network whenever network congestion happens, but this needs to be done carefully so as not to significantly impact the applications that need the data. This paper proposes a novel approach to manage AMI data traffic volume in multi-level data collection trees through data aggregation that estimates the expected network delays messages would suffer and dynamically determines an aggregation policy to be applied at forwarding nodes of the tree to reduce the network delays with which the electric utility can get the necessary information. The proposed algorithm is evaluated for different network congestion scenarios using the NS-3 simulator. The results illustrate that the algorithm is immensely effective in controlling the increase in network latencies as congestion levels increase in AMI networks. In addition, it does well in satisfying quality-of-service (QoS) requirements in terms of the data granularity required by smart grid applications. [ABSTRACT FROM AUTHOR]

Simultaneous modification and navigation of massive 3D models are difficult because repeated data edits affect the data layout and coherency on a secondary storage, which in turn affect the interactive out-of-core rendering performance. In this paper, we propose a novel approach for distributed data management for simultaneous interactive navigation and modification of massive 3D data using the readily available infrastructure of a tiled display. Tiled multi-displays, projection or LCD panel based, driven by a PC cluster, can be viewed as a cluster of storage-compute-display (SCD) nodes. Given a cluster of SCD node infrastructure, we first propose a distributed memory hierarchy for interactive rendering applications. Second, in order to further reduce the latency in such applications, we propose a new data partitioning approach for distributed storage among the SCD nodes that reduces the variance in the data load across the SCD nodes. Our data distribution method takes in a data set of any size, and reorganizes it into smaller partitions, and stores it across the multiple SCD nodes. These nodes store, manage, and coordinate data with other SCD nodes to simultaneously achieve interactive navigation and modification. Specifically, the data is not duplicated across these distributed secondary storage devices. In addition, coherency in data access, due to screen-space adjacency of adjacent displays in the tile, as well as object space adjacency of the data sets, is well leveraged in the design of the data management technique. Empirical evaluation on two large data sets, with different data density distribution, demonstrates that the proposed data management approach achieves superior performance over alternative state-of-the-art methods. [ABSTRACT FROM AUTHOR]