Search

During the past few years, the traffic volume of legitimate traffic and attack traffic has increased manifolds up to Terabytes per second (Tbps). Because of the processing of such a huge traffic volume, it has become implausible to detect high rate attacks in time using conventional DDoS defense architectures. At present, the majority of the DDoS defense systems are deployed predominantly at the victim-end domain But these victim-end defense systems themselves are vulnerable to HR-DDoS attacks as the mammoth volume of attack traffic is generated by such type of attacks. The insufficient computational resources further make the problem more crucial at the victim-end. This paper proposed a distributed and collaborative architecture called E-Had that is capable of efficiently processing a large amount of data by distributing it among a number of mappers and reducers in a Hadoop based cluster. The proposed E-Had system has been comprehensively validated using various publicly available benchmarked datasets and real datasets generated in HA-DDoS testbed in terms of various detection system evaluation metrics. The experimental results clearly show that the proposed detection system is capable of early detection of different scenarios of DDoS attacks along with differentiating them from flash crowds.

Although many researchers talk about a “patient database,” they typically are not referring to a database at all, but instead to a spreadsheet of curated facts about a cohort of patients. This article describes relational database systems and how they differ from spreadsheets. At their core, spreadsheets are only capable of describing one-to-one (1:1) relationships. However, this article demonstrates that clinical medical data encapsulate numerous one-to-many relationships. Consequently, spreadsheets are very inefficient relative to relational database systems, which gracefully manage such data. Databases provide other advantages, in that the data fields are “typed” (that is, they contain specific kinds of data). This prevents users from entering spurious data during data import. Because each record contains a “key,” it becomes impossible to add duplicate information (ie, add the same patient twice). Databases store data in very efficient ways, minimizing space and memory requirements on the host system. Likewise, databases can be queried or manipulated using a highly complex language called SQL. Consequently, it becomes trivial to cull large amounts of data from a vast number of data fields on very precise subsets of patients. Databases can be quite large (terabytes or more in size), yet still are highly efficient to query. Consequently, with the explosion of data available in electronic health records and other data sources, databases become increasingly important to contain or order these data. Ultimately, this will enable the clinical researcher to perform artificial intelligence analyses across vast amounts of clinical data in a way heretofore impossible. This article provides initial guidance in terms of creating a relational database system.

With the emerging of the Internet of Things (IoT) technology, we can analyze and real-time monitoring the power consumption data of buildings or equipment. However, over time, many power data will accumulate to even the terabyte level. Traditional data processing techniques will not handle. It is a challenge to monitor and process the data in a reasonable time. In this paper, we construct an efficient and real-time power monitoring platform, based on open source techniques. The power-data of buildings or equipment is provided through smart meters equipped inside campus buildings. The technologies of data collected and stored on the server side were handled by the Hadoop ecosystem and the data process to Hive data warehouse is executed by the Spark. On the particulars of system evaluation, we make some experiments to compare with the performance of record query, use three different technologies separately: Apache Hive, Apache Spark, and Impala had been evaluated regarding query-response performance. Moreover, for the performance estimation on the data ETL processing and SQL functions. The relevant experiments have been conducted on the same three software modules as well.

Clustering is a popular technique that can help make large datasets more manageable and usable by grouping together similar objects. Most clustering approaches are too computationally expensive for datasets that are very large or complex. This work presents Parallel K-Tree, a hierarchical data structure and clustering algorithm that takes advantage of modern computing environments to cluster extremely large datasets. Parallel K-Tree produces high-quality clusters and scales more efficiently than traditional, parallelized and state-of-the-art approaches. Parallel K-Tree was applied to a large (8 terabyte) collection of Landsat 5 satellite images. This required clustering of 540 billion objects into eight billion clusters — a two orders of magnitude size increase over any reported alternative approach. Furthermore, Parallel K-Tree was executed on just two commodity servers — rather than a high-performance supercomputer.

Genomics next generation sequencing (NGS) and third generation sequencing (TGS) have a broad area of applications in life science, such as Non-Invasive Prenatal Testing (NIPT), ctDNA Testing for Non-Invasive Tumor Personalized Therapy, Whole Genome Sequencing (WGS), Whole Exome Sequencing (WES), RNA Sequencing, etc. The high throughput genome sequencing instrument is capable of sequencing thousands of samples in parallel, generating dozens of terabytes of genomic data in one day. The storage and analysis of the big petabytes of genomic data are approaching a very challenge for much of the biomedical research communities. In this paper, we give the design and implementation of a genomics cloud, which can scale storage and computing abilities flexibly and provides many easy to use genomics analysis software for customers. This paper gives the technical solution for building a genomics cloud based on CWL/WDL, Docker, DAG, NAS and Object Storage System. The implemented cloud platform also frees scientists from the burden of: building high performance cluster, managing millions of genomic data files, and scripting genomics analysis pipelines.

The recent growth of high-resolution spatial data, especially in developing urban environments, is enabling new approaches to civic activism, urban planning and the provision of services necessary for sustainable development. A special area of great potential and urgent need deals with urban expansion through informal settlements (slums). These neighborhoods are too often characterized by a lack of connections, both physical and socioeconomic, with detrimental effects to residents and their cities. Here, we show how a scalable computational approach based on the topological properties of digital maps can identify local infrastructural deficits and propose context-appropriate minimal solutions. We analyze 1 terabyte of OpenStreetMap (OSM) crowdsourced data to create worldwide indices of street block accessibility and local cadastral maps and propose infrastructure extensions with a focus on 120 Low and Middle Income Countries (LMICs) in the Global South. We illustrate how the lack of physical accessibility can be identified in detail, how the complexity and costs of solutions can be assessed and how detailed spatial proposals are generated. We discuss how these diagnostics and solutions provide a multiscalar set of new capabilities&mdash, from individual neighborhoods to global regions&mdash, that can coordinate local community knowledge with political agency, technical capability, and further research.

Does watching more pirated streaming video mean spending less time watching non-pirated streaming video? This study measures whether, and how much, time spent watching pirated video crowds out time spent on streaming video apps. While prior studies have estimated the impact of piracy on sales revenues, our study measures the impact of piracy on time spent on free and paid streaming apps. We combine big data tools with standard econometric techniques, including a two-stage least squares model, to analyze 5.25 terabytes of online activity data from 19,764 American households and their 468,612 devices from 2016 to 2017. The analysis suggests that every minute spent engaged with pirated video sites crowded out about 3.5 minutes of time spent streaming video. Because pirated video files are generally more compressed than non-pirated video files and because they are frequently downloaded as entire files rather than streamed, as with non-pirate sites like Netflix and Amazon, we conclude that our results exhibit closer to a 1-to-1 crowding out effect of piracy on over-the-top streaming video services.

The steadily improving resolution power of sensors results in larger and larger data objects, which cannot be analysed in a reasonable amount of time on single workstations. To speed up the analysis the Divide and Conquer method can be used by splitting (large) data objects into smaller pieces where each piece is analysed on a single node and, finally, the partial results are collected and combined. We apply this method to the validated bio–medical framework Ki67–Analysis that determines the amount of cancer cells in high–resolution images from breast examinations. In previous work, we observed an anomalous behaviour when the framework is applied to subtiles of an image. To this end, we determined for each subtile a so–called Ki67–Analysis score parameter, which is given by the ratio of the number of identified cancer cells and the total number of cells. This parameter turns out to be underestimated the more the smaller the subtiles. The anomaly prevents a direct application of the Divide and Conquer method. In this work, we suggest a novel grey–box testing method for understanding the origin of the anomaly. It allows to identify a class of subtiles for which the Ki67–Analysis score parameter can be determined reasonably well, i.e. for which the Divide and Conquer method can be applied. By demanding the stability of the framework with regard to small additive noise in brightness, “ghost cells” are identified that turn out to be an artefact of the framework. Finally, the challenge of analysing huge single data objects is considered. The upcoming observatory Square Kilometre Array (SKA) will consist of thousands of antennas and telescopes. Due to the exceptional resolution power of SKA, single images from the Universe may be as large as one Petabyte. “Data monster” of that huge size cannot be analysed reasonably fast on traditional computing architectures. The relatively small throughput rates when reading data from disks is a serious bottleneck (memory–wall problem). Memory–based computing offers a change in paradigm: the current processor–centric architecture is replaced by a memory–based architecture. Hewlett Packard Enterprise (HPE) developed a prototype with 48 Terabyte of memory, called Sandbox. Counting words in large files can be considered as a first step towards simulating image processing of “Data Monster” at SKA. We run the big data framework Thrill on the Sandbox and determine the speedup of different setups for distributed word counting.

The progress of next-generation sequencing has a major impact on medical and genomic research. This high-throughput technology can now produce billions of short DNA or RNA fragments in excess of a few terabytes of data in a single run. This leads to massive datasets used by a wide range of applications including personalized cancer treatment and precision medicine. In addition to the hugely increased throughput, the cost of using high-throughput technologies has been dramatically decreasing. A low sequencing cost of around US$1000 per genome has now rendered large population-scale projects feasible. However, to make effective use of the produced data, the design of big data algorithms and their efficient implementation on modern high performance computing systems is required.

In the atmospheric sciences, the size of simulation output continues to grow as computational resources able to handle simulations with fine-scale spatial and temporal resolutions become more accessible. As output size increases, serial data analysis methods become overwhelmed, resulting in either long delays during processing or total failures due to memory constraints. Parallel data analysis methods can alleviate these issues, however atmospheric scientists are often unfamiliar with how to achieve this. Therefore, example methods are needed to help guide the use of parallel processing in the analysis of Big Data from atmospheric simulations. In this work, practical methods are presented by which an analysis may be executed in parallel using the Message Passing Interface (MPI) and Python. These methods first consider the inherent spatial dependencies of a particular data analysis process. By identifying these dependencies, horizontal or vertical distribution of the dataset across processes can be carried out with minimal process intercommunication. In addition, an analysis method is classified as either data-transfer-limited or computationally-limited. In data-transfer-limited problems, data transfer time outweighs processing time. In computationally-limited problems, processing time outweighs data transfer time. The results show that by increasing processor count, the execution time of computationally-limited problems shows improvement. For data-transfer-limited problems, increasing node count offers the greatest improvement. To further improve the performance of computationally-limited problems, a Graphics Processing Unit (GPU) and the Compute Unified Device Architecture (CUDA) framework are used. It is shown that this GPU implementation offers further improvement over the MPI version of the analysis methods tested.

The concept of software agent and big data plays a crucial responsibility in several fields. With new multiple dynamic parameters of software agent with the concept of big data analytic is growing rapidly in few years. The emergent figure of applications helping millions of users and dealing with terabytes of data require to a faster processing prototype. The concept of software agent in Object-Oriented techniques describes an advance and stimulating way of analyzing, developing, and modelling advanced and useful data analysis. This paper intends to identify and describing how such a concept of software agent might be useful to big data in the appearance of an adaptive software agent System based analytics and upcoming rising challenges. Proposed critique focuses at inquire about new attributes and recitation that how such a technology be able to useful to big data in the appearance of an adaptive software-agent system as long as vibrant analytics capabilities. This ongoing review of research has capable outcomes but will be require for discussing and validating.

Performance analysis of scientific parallel applications is essential to use High Performance Computing (HPC) infrastructures efficiently. Nevertheless, collecting detailed data of large-scale parallel programs and long-running applications is infeasible due to the huge amount of performance information generated. Even though there are no technological constraints in storing Terabytes of performance data, the constant flushing of such data to disk introduces a massive overhead into the application that makes the performance measurements worthless. This paper explores the use of Event flow graphs together with wavelet analysis and EZW-encoding to provide MPI event traces that are orders of magnitude smaller while preserving accurate information on timestamped events. Our mechanism compresses the performance data online while the application runs, thus, reducing the pressure put on the I/O system due to buffer flushing. As a result, we achieve lower application perturbation, reduced performance data output, and the possibility to monitor longer application runs.

The New Vacuum Solar Telescope (NVST) is a 1-m solar telescope that aims to observe the fine structures in both the photosphere and the chromosphere of the Sun. The observational data acquired simultaneously from one channel for the chromosphere and two channels for the photosphere bring great challenges to the data storage of NVST. The multi-channel instruments of NVST, including scientific cameras and multi-band spectrometers, generate at least 3 terabytes data per day and require high access performance while storing massive short-exposure images. It is worth studying and implementing a storage system for NVST which would balance the data availability, access performance and the cost of development. In this paper, we build a distributed data storage system (DDSS) for NVST and then deeply evaluate the availability of real-time data storage on a distributed computing environment. The experimental results show that two factors, i.e., the number of concurrent read/write and the file size, are critically important for improving the performance of data access on a distributed environment. Referring to these two factors, three strategies for storing FITS files are presented and implemented to ensure the access performance of the DDSS under conditions of multi-host write and read simultaneously. The real applications of the DDSS proves that the system is capable of meeting the requirements of NVST real-time high performance observational data storage. Our study on the DDSS is the first attempt for modern astronomical telescope systems to store real-time observational data on a low-cost distributed system. The research results and corresponding techniques of the DDSS provide a new option for designing real-time massive astronomical data storage system and will be a reference for future astronomical data storage. (C) 2015 Elsevier B.V. All rights reserved.

Visualizing and navigating through large astronomy images from a remote location with current astronomy display tools can be a frustrating experience in terms of speed and ergonomics, especially on mobile devices. In this paper, we present a high performance, versatile and robust client-server system for remote visualization and analysis of extremely large scientific images. Applications of this work include survey image quality control, interactive data query and exploration, citizen science, as well as public outreach. The proposed software is entirely open source and is designed to be generic and applicable to a variety of datasets. It provides access to floating point data at terabyte scales, with the ability to precisely adjust image settings in real-time. The proposed clients are light-weight, platform-independent web applications built on standard HTML5 web technologies and compatible with both touch and mouse-based devices. We put the system to the test and assess the performance of the system and show that a single server can comfortably handle more than a hundred simultaneous users accessing full precision 32 bit astronomy data., Published in Astronomy & Computing. IIPImage server available from http://iipimage.sourceforge.net . Visiomatic code and demos available from http://www.visiomatic.org/

Next generation sequencing (NGS) technologies produce massive amounts of data requiring a powerful computational infrastructure, high quality bioinformatics software, and skilled personnel to operate the tools. We present a case study of a practical solution to this data management and analysis challenge that simplifies terabyte scale data handling and provides advanced tools for NGS data analysis. These capabilities are implemented using the “Globus Genomics” system, which is an enhanced Galaxy workflow system made available as a service that offers users the capability to process and transfer data easily, reliably and quickly to address end-to-endNGS analysis requirements. The Globus Genomics system is built on Amazon's cloud computing infrastructure. The system takes advantage of elastic scaling of compute resources to run multiple workflows in parallel and it also helps meet the scale-out analysis needs of modern translational genomics research.

Big data is a collection of massive and complex data sets and data volume that include the huge quantities of data, data management capabilities, social media analytics and real-time data. Big data analytics is the process of examining large amounts of data. There exist large amounts of heterogeneous digital data. Big data is about data volume and large data set's measured in terms of terabytes or petabytes. This phenomenon is called Bigdata. After examining of Bigdata, the data has been launched as Big Data analytics. In this paper, presenting the 5Vs characteristics of big data and the technique and technology used to handle big data. The challenges include capturing, analysis, storage, searching, sharing, visualization, transferring and privacy violations. It can neither be worked upon by using traditional SQL queries nor can the relational database management system (RDBMS) be used for storage. Though, a wide variety of scalable database tools and techniques has evolved. Hadoop is an open source distributed data processing is one of the prominent and well known solutions. The NoSQL has a non-relational database with the likes of MongoDB from Apache.

The SEC's EDGAR log file data set is a collection of web server log files that allow researchers to study the demand for SEC filings. This multiple terabyte data set provides researchers with a direct measure of demand for financial reports, but the log files must be filtered to remove downloads by computer programs (or robots), and the sheer size of the files presents big data challenges. This paper compares three methods for counting human views in the EDGAR log files and aggregates the data on a filing-day basis so that it is accessible to desktop hardware and statistical analysis software. Overall, the three methods agree on the robot-human classification for 96 percent of users, but for sample 10-K filings, they can disagree by up to 27 percent. Download counts may be biased by up to 36 percent if multiple views by the same user are counted. Ryans's 2017 method eliminates multiple download counting and appears to effectively classify robots in cases of disagreement among the measures. The choice of measure may be particularly important when studying demand for Forms 10-K, 10-Q, 4, 13F-HR, as well as SEC comment letters. The aggregated data and sample code are available from the author.

Generalized linear mixed-effects models in the context of genome-wide association studies (GWAS) represent a formidable computational challenge: the solution of millions of correlated generalized least-squares problems, and the processing of terabytes of data. We present high performance in-core and out-of-core shared-memory algorithms for GWAS: By taking advantage of domain-specific knowledge, exploiting multi-core parallelism, and handling data efficiently, our algorithms attain unequalled performance. When compared to GenABEL, one of the most widely used libraries for GWAS, on a 12-core processor we obtain 50-fold speedups. As a consequence, our routines enable genome studies of unprecedented size.

When digital forensics started in the mid-1980s most of the software used for analysis came from writing and debugging software. Amongst these tools was the UNIX utility ‘dd’ which was used to create an image of an entire storage device. In the next decade the practice of creating and using ‘an image’ became established as a fundamental base of what we call ‘sound forensic practice’. By virtue of its structure, every file within the media was an integrated part of the image and so we were assured that it was wholesome representation of the digital crime scene. In an age of terabyte media ‘the image’ is becoming increasingly cumbersome to process, simply because of its size. One solution to this lies in the use of distributed systems. However, the data assurance inherent in a single media image file is lost when data is stored in separate files distributed across a system. In this paper we assess current assurance practices and provide some solutions to the need to have assurance within a distributed system.

For ITER, acquiring, managing and archiving its data is an essential task. ITER is foreseen to produce up to one terabyte of data per pulse and several petabytes of data per year. All the produced data needs to be stored and managed. The stored data is expected to serve the data access needs of ITER researchers located both on the ITER premises as well as worldwide during ITER's lifetime and beyond. ITERDB is a data management system being designed for centralized ITER data archival and data access. It is designed to manage and serve both unprocessed and processed data from the ITER plant systems and data analysis workflows. In this paper, we report the ITER Data Archiving System software requirements and priorities that have been identified by working with ITER staff and a large number of stakeholders. We will describe the design challenges and the proposed solutions. We will also present the current state of the ITERDB software architecture design.

Scientific workflows integrate data and computing interfacesas configurable, semi-automatic graphs to solve a scientific problem. Kepler is such a software system for designing, executing, reusing, evolving, archiving and sharing scientific workflows. Electron tomography (ET) enables high-resolution views of complex cellular structures, such as cytoskeletons, organelles, viruses and chromosomes. Imaging investigations produce large datasets. For instance, in Electron Tomography, the size of a 16 fold image tilt series is about 65 Gigabytes with each projection image including 4096 by 4096 pixels. When we use serial sections or montage technique for large field ET, the dataset will be even larger. For higher resolution images with multiple tilt series, the data size may be in terabyte range. Demands of mass data processing and complex algorithms require the integration of diverse codes into flexible software structures. This paper describes a workflow for Electron Tomography Programs in Kepler (EPiK). This EPiKworkflow embeds the tracking process of IMOD, and realizes the main algorithms including filtered backprojection (FBP) from TxBR and iterative reconstruction methods. We have tested the three dimensional (3D) reconstruction process using EPiK on ET data. EPiK can be a potential toolkit for biology researchers with the advantage of logical viewing, easy handling, convenient sharing andfuture extensibility.

Analyzing big data is a challenging problem in cluster computing systems especially when the data volume goes beyond the available computing resources. Sampling is the favored solution for such problems. It summarizes or reduces the amount of data, taking into consideration the statistical characteristics of data distribution. However, the traditional method to sample the massive data by drawing record-by-record is a computationally expensive process because a full scan of the whole data is needed to be performed. While if the massive data is partitioned into a set of data blocks with each block is a random sample data block, the processing time for selecting some blocks as a sample (or different samples) is computationally cheaper. The main purpose of the software described in this paper is to represent the HDFS blocks as a set of random sample data blocks which also stored in HDFS. Our empirical results show that the performance of the partitioning operation is acceptable in the real application especially this operation is only performed once, thereby analysis on terabyte data becomes more natural.

At small scales, storage is straightforward to afford and to use, but at large scales – from several Terabytes (TB) to many Petabytes (PB) and soon Exabytes (EB) – tradeoffs must be made between cost and convenience, and training for use of such resources needs to take such inconveniences into account. A large scale, long term (over 10 year) institutional research data storage archive is described, focusing on both hardware and software. The technology choices give rise to inconveniences, which in turn not only lead to a crucial requirement for training on the proper use of the archive, but also inform the specifics of that training, as does each individual use case.

We present a fast randomized least-squares solver for distributed-memory platforms. Our solver is based on the Blendenpik algorithm, but employs multiple random projection schemes to construct a sketch of the input matrix. These random projection sketching schemes, and in particular the use of the randomized Discrete Cosine Transform, enable our algorithm to scale the distributed memory vanilla implementation of Blendenpik to terabyte-sized matrices and provide up to ×7.5 speedup over a state-of-the-art scalable least-squares solver based on the classic QR algorithm. Experimental evaluations on terabyte scale matrices demonstrate excellent speedups on up to 16,384 cores on a Blue Gene/Q supercomputer.

Connectomics is a strategy for mapping complex neural networks based on high-speed automated electron optical imaging, computational assembly of neural data volumes, web-based navigational tools to explore 10 12 –10 15 byte (terabyte to petabyte) image volumes, and annotation and markup tools to convert images into rich networks with cellular metadata. These collections of network data and associated metadata, analyzed using tools from graph theory and classification theory, can be merged with classical systems theory, giving a more completely parameterized view of how biologic information processing systems are implemented in retina and brain. Networks have two separable features: topology and connection attributes. The first findings from connectomics strongly validate the idea that the topologies of complete retinal networks are far more complex than the simple schematics that emerged from classical anatomy. In particular, connectomics has permitted an aggressive refactoring of the retinal inner plexiform layer, demonstrating that network function cannot be simply inferred from stratification; exposing the complex geometric rules for inserting different cells into a shared network; revealing unexpected bidirectional signaling pathways between mammalian rod and cone systems; documenting selective feedforward systems, novel candidate signaling architectures, new coupling motifs, and the highly complex architecture of the mammalian A II amacrine cell. This is but the beginning, as the underlying principles of connectomics are readily transferrable to non-neural cell complexes and provide new contexts for assessing intercellular communication.

Every day hundreds of millions of people log into social network sites and deposit terabytes of data as they share status updates, photographs, and more. This article explores how background factors, motivations, and social network site experiences relate to people's use of social network site technology to protect their privacy. The findings indicate that during technology-mediated communication on social network sites, not only do traditional privacy factors relate to the technological boundaries people enact, but people's experiences with the mediating technology itself do, too. The results also identify privacy inequalities, in which certain groups are more likely to take advantage of the technology to protect their privacy-suggesting that some individuals' information and reputations may be more at risk than others'.

Scalability, fault tolerance and collaborative processing across possibly dispersed sites are key enablers of modern computer forensics applications, that must be able to elastically accommodate all kinds of digital investigations, without wasting resources or fail to deliver timely outcomes. Traditional tools running in a standalone or client- server setups may fall short when handling the multi terabyte scale of a case just above average or, conversely, lie mainly underutilized when dealing with few digital evidences. A new category of applications that leverage the opportunities offered by modern Cloud Computing (CC) platforms, where scalable computational power and storage capacity can be engaged and decommissioned on demand, allow one to conveniently master huge amounts of information that otherwise could be impossible to wield. This paper discusses the design goals, technical requirements and architecture of AlmaNebula, a conceptual framework for the analysis of digital evidences built on top of a Cloud infrastructure, which aims to embody the concept of “Forensics as a service”.

Post-genomics allegedly allow us to become the ‘managers’ of our own health. And yet, human individuality seems to dissolve into massive data streams. What is the fate of the human subject in the terabyte age? The Human Genome Project already resulted in personalising and depersonalising trends, exemplified by two types of genomes: the anonymised Human Reference Genome versus the personal genomes of genomics celebrities. This ambiguity is radicalised by post-genomics. Life becomes ‘obliterated’: dissolved into letters and symbols (e.g. the nucleotide alphabet), but this is complemented by re-personalising trends. As a case study, I will analyse the Snyderome, involving a prominent geneticist who closely monitored ‘everything’ with the help of precision diagnostics, resulting in a comprehensive (“high coverage”) omics portrait, highly personal and highly impersonal at the same time, captured in massive data sets, setting the stage for a digital panopticon: a molecularised ‘conscience’, the superego of the terabyte age.

Conventional approaches to information retrieval search through all applicable entries in an inverted file for a particular collection in order to find those documents with the highest scores. For particularly large collections this may be extremely time consuming. A solution to this problem is to only search a limited amount of the collection at query-time, in order to speed up the retrieval process. In doing this we can also limit the loss in retrieval efficacy (in terms of accuracy of results). The way we achieve this is to firstly identify the most ''important'' documents within the collection, and sort documents within inverted file lists in order of this ''importance''. In this way we limit the amount of information to be searched at query time by eliminating documents of lesser importance, which not only makes the search more efficient, but also limits loss in retrieval accuracy. Our experiments, carried out on the TREC Terabyte collection, report significant savings, in terms of number of postings examined, without significant loss of effectiveness when based on several measures of importance used in isolation, and in combination. Our results point to several ways in which the computation cost of searching large collections of documents can be significantly reduced.

Purpose Having access to a library of radiological images in the university setting is important for teaching and learning in diagnostic radiography. Modern modalities such as PETCT create data volumes rather than single static 2D images. A PACS repository of images alone does not constitute a teaching library without some text based searchable index. A review of several options for acquiring a digital teaching library (DTL) of such indexed DICOM data is presented. Discussion The data protection principles, current guidance and potential methods for migrating and cleansing large quantities of DICOM data from a clinical PACS prior to transfer to a university setting is discussed. The chosen method is described and the important enabling technology identified. Various methods of index construction are outlined and a method of migrating and cleansing HL7 data from a clinical RIS described. Results Three terabytes of de-normalised DICOM image files were cleansed of patient, staff and geographic identifiers, within the header tags and pixel data. These files were then migrated to an educational PACS hosted at a university. A searchable index database was created based on 90,200 reports and associated data, and 886,263 DICOM headers to enable meaningful results to be found from the 51,304 unique patient specific cases. Conclusion A large DTL in the university setting using PACS technology is becoming a valuable resource for teaching, learning and assessment.

Demands for a broad range of integrated geospatial data-analysis tools and methods for planetary data organization have been growing considerably since the late 1990s when a plethora of missions equipped with new instruments entered planetary orbits or landed on the surface. They sent back terabytes of new data which soon became accessible for the scientific community and public and which needed to be organized. On the terrestrial side, issues of data access, organization and utilization for scientific and economic analyses are handled by using a range of well-established geographic information systems (GIS) that also found their way into the field of planetary sciences in the late 1990s. We here address key issues concerning the field of planetary mapping by making use of established GIS environments and discuss methods of addressing data organization and mapping requirements by using an easily integrable datamodel that is—for the time being—designed as file-geodatabase (FileGDB) environment in ESRI's ArcGIS. A major design-driving requirement for this datamodel is its extensibility and scalability for growing scientific as well as technical needs, e.g., the utilization of such a datamodel for surface mapping of different planetary objects as defined by their respective reference system and by using different instrument data. Furthermore, it is a major goal to construct a generic model which allows to perform combined geologic as well as geomorphologic mapping tasks making use of international standards without loss of information and by maintaining topologic integrity. An integration of such a datamodel within a geospatial DBMS context can practically be performed by individuals as well as groups without having to deal with the details of administrative tasks and data ingestion issues. Besides the actual mapping, key components of such a mapping datamodel deal with the organization and search for image-sensor data and previous mapping efforts, as well as the proper organization of cartographic representations and assignments of geologic/geomorphologic units within their stratigraphic context.

Recent advances in computer visualization have allowed us to develop new tools for analyzing the data gathered during planetary missions, which is important, since these data sets have grown exponentially in recent years to tens of terabytes in size. As part of the Advanced Visualization in Solar System Exploration and Research (ADVISER) project, we utilize several advanced visualization techniques created specifically with planetary image data in mind. The Geoviewer application allows real-time active stereo display of images, which in aggregate have billions of pixels. The ADVISER desktop application platform allows fast three-dimensional visualization of planetary images overlain on digital terrain models. Both applications include tools for easy data ingest and real-time analysis in a programmatic manner. Incorporation of these tools into our everyday scientific workflow has proved important for scientific analysis, discussion, and publication, and enabled effective and exciting educational activities for students from high school through graduate school.

In this paper we present an XML-based framework, called XLIVE, which provides an efficient way to collect data in live forensic cases, according to well-known crime categories. XLIVE is a forensic automated framework that can be used in live forensic investigations for gathering live data on a Windows-based system. In addition, we have also implemented a proof-of-concept, called LRDS (Live Resource Detection System). This approach of examination will be used extensively to deal with terabyte/petabyte digital systems, where other approaches, such as a post-mortem analysis, cannot be adopted.

Sharing scientific and data capture files of gigabyte and terabyte size in conventional data grid systems is inefficient because conventional approaches copy the entire shared file to a user's local storage even when only a tiny file fragment is required. Such transfer schemes consume unnecessary data transmission time and local storage space, with the additional problem of maintaining replica synchronization. Traditionally, replica consistency treats shared files as read-only, consequently sacrificing guaranteed replica consistency. This paper presents a DSM-based fragmented data sharing framework called ''Spigot'' which transfers only the necessary fragments of large files on user demand, thereby reducing data transmission time, wasted network bandwidth and required storage space. Data waiting time is further reduced by overlapping data transmission and data analysis. The DSM concept maintains replica synchronization. Real experiments show reduced turnaround time in data-intensive applications, particularly when fragment size is low and analysis time and network latency are high.

Modern computers are beginning to surpass the design limitations of the aging DOS/MBR partition tables and BIOS boot system. As disk sizes begin to exceed two terabytes and hardware vendors begin to transition from BIOS to EFI, understanding GPT disks in forensic examinations becomes useful. This practitioner paper provides an overview of the GUID Partition Table (GPT) scheme from the perspective of the digital forensic investigator. Methods of analysis and acquisition are shown, and artifacts of forensic relevance are identified. The target audience for this paper is digital forensic practitioners and forensic tool developers.

Many of the physical properties of materials are critically dependent on their microstructure. In recent years, there has been increasing interest in using computer simulations based on phase-field models for the spatial and temporal evolution of microstructures. Although such simulations are computationally expensive, the generated set of microstructures can be stored in a repository and used for further analysis in materials design. However, such an approach requires a substantial amount of storage, for example, approximately 1 Terabyte for a single binary alloy. In this paper, we develop fast data compression and regeneration schemes for two-dimensional microstructures that can reduce storage requirements without compromising the accuracy of computed values, such as stress fields used in analysis. Our main contribution is the development and evaluation of a sparse skeletal representation scheme which outperforms traditional compression schemes. Our results indicate that our scheme can reduce microstructure data size by more than two orders of magnitude while achieving better accuracies for the computed stress fields and order parameters.

D0 and CDF currently are the two largest experiments operating at Tevatron proton-antiproton collider at a center-of-mass energy of 1.96 GeV. Approximately 2.5 fb −1 of data resulting in 3 billion events per experiment have been collected and processed by each experiment in 2002 – 2007. Such an amount of data represents the unique challenge for experiments and requires the extensive use of computing facilities all over the world. The D0 and CDF experience in building successful computing models and making 2000 Terabytes of data available for the physics analyzes is presented as well as the future prospects to handle the upcoming 3 time increase of the available data set in the next 2 years.

This report gives an overview of the status and outlook for the world's research networks and major international links used by the high energy physics and other scientific communities, network technology advances on which our community depends and in which we have an increasingly important role, and the problem of the Digital Divide, which is a primary focus of ICFA's Standing Committee on Inter-regional Connectivity (SCIC). Wide area networks of sufficient, and rapidly increasing end-to-end capability are vital for every phase of high energy physicists' work. Our bandwidth usage, and the typical capacity of the major national backbones and intercontinental links used by our field have progressed by a factor of more than 1000 over the past decade, and the outlook is for a similar increase over the next decade. This striking exponential growth trend, outstripping the growth rates in other areas of information technology, has continued in the past year, with many of the major national, continental and transoceanic networks supporting research and education progressing from a 10 Gigabits/sec (Gbps) backbone to multiple 10 Gbps links in their core. This is complemented by the use of point-to-point “light paths” to support the most demanding applications, including high energy physics, in a growing list of cases. As we approach the era of LHC physics, the growing need to access and transport Terabyte-scale and later 10 to 100 Terabyte datasets among more than 100 “Tier1” and “Tier2” centers at universities and laboratories spread throughout the world has brought the key role of networks, and the ongoing need for their development, sharply into focus. Bandwidth itself on an increasing scale is not enough. Realizing the scientific wealth of the LHC and our other major scientific programs depends crucially on our ability to use the bandwidth efficiently and reliably, with reliable high rates of data throughput, and effectively, where many parallel large-scale data transfers serving the community complete with high probability, often while coexisting with many other streams of network traffic. Responding to these needs, and to the scientific mission, physicists working with network engineers and computer scientists have made substantial progress in the development of protocols and systems that promise to meet these needs, placing our community among the world leaders in the development as well as use of large-scale networks. A great deal of work remains, and is continuing. As we advance in these areas, often (as in the past year) with great rapidity, there is a growing danger that we will leave behind our collaborators in regions with less-developed networks, or with regulatory frameworks or business models that put the required networks financially out of reach. This threatens to further open the Digital Divide that already exists among the regions of the world. In 2002, the SCIC recognized the threat that this Divide represents to our global scientific collaborations, and since that time has worked assiduously to reduce or eliminate it; both within our community, and more broadly in the world research community of which HEP is a part.

The increasing number of documents to be indexed in many environments (Web, intranets, digital libraries) and the limitations of a single centralised index (lack of scalability, server overloading and failures), lead to the use of distributed information retrieval systems to efficiently search and locate the desired information. This work is a case study of different architectures for a distributed information retrieval system, in order to provide a guide to approximate the optimal architecture with a specific set of resources. We analyse the effectiveness of a distributed, replicated and clustered architecture simulating a variable number of workstations (from 1 up to 4096). A collection of approximately 94 million documents and 1 terabyte (TB) of text is used to test the performance of the different architectures. In a purely distributed information retrieval system, the brokers become the bottleneck due to the high number of local answer sets to be sorted. In a replicated system, the network is the bottleneck due to the high number of query servers and the continuous data interchange with the brokers. Finally, we demonstrate that a clustered system will outperform a replicated system if a high number of query servers is used, essentially due to the reduction of the network load. However a change in the distribution of the users' queries could reduce the performance of a clustered system.

Data warehouses collect masses of operational data, allowing analysts to extract information by issuing decision support queries on the otherwise discarded data. In many application areas (e.g. telecommunications), the warehoused data sets are multiple terabytes in size. Parts of these data sets are stored on very large disk arrays, while the remainder is stored on tape-based tertiary storage (which is one to two orders of magnitude less expensive than on-line storage). However, the inherently sequential nature of access to tape-based tertiary storage makes the efficient access to tape-resident data difficult to accomplish through conventional databases.In this paper, we present a way to make access to a massive tape-resident data warehouse easy and efficient. Ad hoc decision support queries usually involve large scale and complex aggregation over the detail data. These queries are difficult to express in SQL, and frequently require self-joins on the detail data (which are prohibitively expensive on the disk-resident data and infeasible to compute on tape-resident data), or unnecessary multiple passes through the detail data. An extension to SQL, the extended multi feature SQL (EMF SQL) expresses complex aggregation computations in a clear manner without using self-joins. The detail data in a data warehouse usually represents a record of past activities, and therefore is temporal. We show that complex queries involving sequences can be easily expressed in EMF SQL. An EMF SQL query can be optimized to minimize the number of passes through the detail data required to evaluate the query, in many cases to only one pass. We describe an efficient query evaluation algorithm along with a query optimization algorithm that minimizes the number of passes through the detail data, and which minimizes the amount of main memory required to evaluate the query. These algorithms are useful not only in the context of tape-resident data warehouses but also in data stream systems which require similar processing techniques.

In numerous scientific disciplines, terabyte and petabyte-scale data collections are emerging as critical community resources. A new class of "data grid" infrastructure is required to support management, transport, distributed access to, and analysis of these datasets by potentially thousands of users. Researchers who face this challenge include the climate modeling community, which performs long-duration computations accompanied by frequent output of very large files that must be further analyzed. We describe the Earth System Grid-I prototype, which brings together advanced analysis, replica management, data transfer, request management, and other technologies to support high-performance, interactive analysis of replicated data. We present performance results that demonstrate our ability to manage the location and movement of large datasets from the user's desktop. We report on experiments conducted over SciNET at SC'2000, where we achieved peak performance of 1.55 Gb/s and sustained performance of 512.9 Mb/s for data transfers between Texas and California. Finally, we describe the development of the next-generation Earth System Grid-II (ESG-II) project. Important issues for ESG-II include security requirements for production environments, efficient data filtering and transport, metadata services for discovery of relevant climate datasets, and sophisticated request or workflow management for complex tasks.

The European DataGrid (EDG) project has reached, after 1.5 years, the middle of its lifetime. In this article we give an overview of the status, components, procedures and plans of the EDG project as of June 2002. The objective of the EDG project is to assist the next generation of scientific exploration, computation and analysis of large-scale data sets—from hundreds of terabytes to petabytes, across widely distributed scientific communities. The primary goal of the first phase of the EDG project was to assemble a distributed testbed to demonstrate that the EDG middleware components could be integrated into a production-quality computational Grid, as well as to gain experience with such a system. The very first version of the EDG testbed was deployed toward the end of 2001. At the first official European Union review of the project on March 1, 2002, it has been found that the project is on the right track to achieve its goals. Since then the EDG middleware, testbed components and procedures have been continuously refined according to the requirements of our user communities and our experience.

This article provides a snapshot of immersive virtual reality (IVR) use for scientific visualization, in the context of the evolution of computing in general and of user interfaces in particular. The main thesis of this article is that IVR has great potential for dealing with the serious problem of exponentially growing scientific datasets. Our ability to produce large datasets both through numerical simulation and through data acquisition via sensors is outrunning our ability to make sense of those datasets. While our idea of “large” datasets used to be measured in hundreds of gigabytes, based at least in part on what we could easily store, manipulate, and display in real time, today's science and engineering are producing terabytes and soon even petabytes, both from observation via sensors and as output from numerical simulation. Clearly, visualization by itself will not solve the problem of understanding truly large datasets that would overwhelm both display capacity and the human visual system. We advocate a human–computer partnership that draws on the strengths of each partner, with algorithmic culling and feature-detection used to identify the small fraction of the data that should be visually examined in detail by the human. Our hope is that IVR will be a potent tool to let humans “see” patterns, trends, and anomalies in their data well beyond what they can do with conventional 3D desktop displays.

Just when it was looking like the underdog, classical computing is striking back. IBM has found a way to simulate quantum computers that have 56 quantum bits, or qubits, on a non-quantum supercomputer--a task previously thought to be impossible. The achievement moves the goalposts in the race for quantum supremacy, the triumph of quantum computers over classical ones. It used to be widely accepted that a classical computer cannot simulate more than 49 qubits due to memory limitations. The closest anyone had come to putting that limit to the test was a 45-qubit simulation at the Swiss Federal Institute of Technology in Zurich, which needed 500 terabytes of memory.

New microscopy techniques (e.g. SPIM, 3D STORM, OPT, etc.) and many other 3D bioimaging modalities (tomographic EM, MRI, CT, etc.) generate terabytes of volumetric data. Whilst highly specialized software packages are available for the analysis and visualization of such data, their use requires specialized skills and they are often costly. Furthermore the capacity to share data and allow interactive exploration by third parties is highly limited.

Group 47 in Woodland Hills CA is working on ways to get around the fact that the drives and discs have a limited lifespan. Instead of writing 1s and 0s as magnetic signals, they write them as microscopic dots onto metal tape, using a laser in a system called DOTS. The tape is then stored in cartridges. A high resolution digital camera can read the data back, but all a future human would need to retrieve the image is knowledge of binary code and a microscope. The firm says the tape should last hundreds of years without degrading and, crucially, doesn't need any special climate-controlled storage. The company received funding from a US intelligence agency last year to develop a prototype, and is now raising funds to build a commercial version. The medium is designed to guide someone with no knowledge of hard drives or computers to read its data--all 1.2 terabytes of it.

Carnegie Mellon University (CMU) is developing robotic technologies and systems for lunar missions. The lunar mission currently being studied is a long duration, time and distance, excursion to revisit the landing sites of previous missions as well as to examine areas and features that were precluded in the previous missions. This mission will be far more than an exploration of a small territory; it is envisioned to be a grand tour of historic sites, interesting features, and unknown areas. A major focus of this mission will be to provide an interactive, live means to enroll public awareness, participation and support into lunar and space missions. To carry out the planned mission, the rover will traverse more than 1000 km, operate for at least two years, perform an array of scientific experiments, transmit more than two terabytes of data and provide entertainment for large numbers of people. To satisfy these goals, concurrent system engineering principals must be must be applied to the rover configuration, otherwise meeting the mission requirements and satisfying the engineering constraints may prove to be impossible.

Measurement Lab (M-Lab) is a collaborative effort founded by Vint Cerf and a large body of network researchers, dedicated to promoting network science at scale. M-Lab provides researchers with geographically distributed server platform to run open source broadband measurement tools. All the data collected by these tools are made publicly available and constitute the largest repository of open broadband measurement data (more than 550 terabytes). These data are being used by researchers as well as by governments and national regulators. Canada recently joined Greece, the US, and the European Union in choosing M-Lab as the backbone of their nation-wide broadband studies. M-Lab's model creates an ecosystem in which researchers, policy-makers, and consumers have access to information drawn from the same pool of open, globally-comparable broadband performance data.

Architectural enhancements are described to increase the performance of the arithmetic accelerator and memory of the nodes in the CM-5 for QCD and a broad range of general problems while maintaining compatibility with existing software, compilers, communications network and I/O subsystems. A factor of 10 increase in performance is obtained by increasing the number of floating point processors by a factor of 4, extending the vector instruction set for dual execution of single-precision arithmetic, and increasing the block rate from 32 to 40 MHz. The required memory bandwidth is obtained by using synchronous DRAMs and 4 floating point processors are packaged into a multichip module which occupies the same area as a present processor package. The proposed 2048 node machine will provide 2.6 Teraflops peak, 0.5 – 1.5 Teraflops sustained on lattices of 322 × 64 − 1283 × 256, will have 256 Gigabytes of memory, 1 Terabyte of disk, an estimated cost of approximately $40 million, and can be built in 2.5 years.

In the mid nineties, the Earth Observing System (EOS) will generate an estimated 10 terabytes of data per day. This enormous amount of data will require the use of sophisticated technologies from real time distributed Artificial Intelligence (AI) and data management. Without regard to the overall problems in distributed AI, efficient models were developed for doing query planning and/or scheduling in intelligent user interfaces that reside in a network environment. Before intelligent query/planning can be done, a model for real time AI planning and/or scheduling must be developed. As Connectionist Models (CM) have shown promise in increasing run times, a connectionist approach to AI planning and/or scheduling is proposed. The solution involves merging a CM rule based system to a general spreading activation model for the generation and selection of plans. The system was implemented in the Rochester Connectionist Simulator and runs on a Sun 3/260.