Search

During the past few years, the traffic volume of legitimate traffic and attack traffic has increased manifolds up to Terabytes per second (Tbps). Because of the processing of such a huge traffic volume, it has become implausible to detect high rate attacks in time using conventional DDoS defense architectures. At present, the majority of the DDoS defense systems are deployed predominantly at the victim-end domain But these victim-end defense systems themselves are vulnerable to HR-DDoS attacks as the mammoth volume of attack traffic is generated by such type of attacks. The insufficient computational resources further make the problem more crucial at the victim-end. This paper proposed a distributed and collaborative architecture called E-Had that is capable of efficiently processing a large amount of data by distributing it among a number of mappers and reducers in a Hadoop based cluster. The proposed E-Had system has been comprehensively validated using various publicly available benchmarked datasets and real datasets generated in HA-DDoS testbed in terms of various detection system evaluation metrics. The experimental results clearly show that the proposed detection system is capable of early detection of different scenarios of DDoS attacks along with differentiating them from flash crowds.

For a marine seismic survey, the recorded and processed data size can reach several terabytes. Storing seismic data sets is costly, and transferring them between storage devices can be challenging. Dictionary learning (DL) has been shown to provide representations with a high level of sparsity. This method stores the shape of the redundant events once and represents each occurrence of these events with a single sparse coefficient. Therefore, an efficient DL-based compression workflow, which is specifically designed for seismic data, is developed here. This compression method differs from conventional compression methods in three respects: (1) The transform domain is not predefined but data-driven, (2) the redundancy in seismic data is fully exploited by learning small-sized dictionaries from local windows of the seismic shot gathers, and (3) two modes (i.e., two different parameterizations) are proposed depending on the geophysical application. Based on a test seismic data set, we have determined superior performance of our workflow in terms of compression ratio (CR) for a wide range of signal-to-residual ratios, such as the zfp software or algorithms from the Seismic Unix package. Using a more realistic data set of marine seismic acquisition, we evaluate the capability of our workflow to preserve the seismic signal for different applications. For applications such as near-real-time transmission and long-term data storage, we observe insignificant signal leakage on a 2D line stack when the DL method reaches a CR higher than 20. For other applications, such as visual quality control of shot gathers, our method preserves the visual aspect of the data even when a CR of 95 is reached.

The advancement of technology has brought up so many elements of ease for human beings that now humans cannot afford to think of a life without all these. But all is not about ease and comfort. Along with all these elements, there are many modern complexities as well as challenges too. Big data analytics and cloud computing are among the elements in which we face many challenges. This document discusses some of the challenges in big data analytics and cloud computing. Challenges for Big Data in Cloud Computing: Just as Big Data provided organizations with terabytes of data, it also posed an issue of managing that data in a traditional framework. How to analyze a large amount of data to remove only the most useful bits? Analyzing these large volumes of data often becomes a very difficult task. In the era of high-speed connectivity, moving large data sets, and providing the details needed to access them is also an issue Conclusion: “All in all, the value of systems based on Big Data technologies lies in the quality and speed of the provided analytics. Noting this, Trovati [19] noted: “Since such analytical tasks that go beyond the scope of data warehouses were previously solved only for individual unique projects, for launching projects on a new set of technologies, it is necessary to develop new industrial models, indicators for specific types of customers. In addition to the industrial focus, a creative approach to data mining is also required - the possibilities for Big Data research are huge”.

The analysis of large amount of sensor data can enhance aircraft operations and maintenance. In contrast to other use cases, aircraft sensor data are accumulated on-board during the flight. The data need to be transferred in a short time while the aircraft is on the ground due to limited and expensive connectivity in the air. This leads to a massive transfer of data in a short time. Therefore, we analyze the data rate demand per aircraft in terms of generated data per flight hour. We determine the influence of different parameters on the needed data rate per aircraft. We conclude that the needed data rates are feasible if the full day is used for transmitting. We show that multiple terabyte of generated data per flight hour can be supported by 5G data rates. Furthermore, we highlight challenges for the deployment of an aircraft data transfer system.

Small and medium sized businesses have lacked of good and automate backup system. Data storages are unreliable and tend to fail without any warning. Whereas, personal computers are already equipped with terabytes of hard drives. This is enormous space for personal or business use. Under normal working conditions, more than half of total storage space is not being used. It is scattered in all connected devices. This paper proposes pooling unused storage resources to create a personal private cloud. The process will be transparent to the users by having a reliable drive with an auto backup built in. The proposed system solution uses a replica-based model in which three sets of the same information would distribute and keep in three different nodes. All the participated nodes are peer-to-peer (P2P) and are decentralized. Therefore, private cloud system can support small or medium businesses as backup system or business sharing drives without adding any cost.

To address the problem of ever-growing scientific data sizes making data movement a major hindrance to analysis, we introduce a novel encoding for scalar fields: a unified tree of resolution and precision, specifically constructed so that valid cuts correspond to sensible approximations of the original field in the precision-resolution space. Furthermore, we introduce a highly flexible encoding of such trees that forms a parameterized family of data hierarchies. We discuss how different parameter choices lead to different trade-offs in practice, and show how specific choices result in known data representation schemes such as zfp [52] , idx [58] , and jpeg 2000 [76] . Finally, we provide system-level details and empirical evidence on how such hierarchies facilitate common approximate queries with minimal data movement and time, using real-world data sets ranging from a few gigabytes to nearly a terabyte in size. Experiments suggest that our new strategy of combining reductions in resolution and precision is competitive with state-of-the-art compression techniques with respect to data quality, while being significantly more flexible and orders of magnitude faster, and requiring significantly reduced resources.

Advances within the toy industry and interconnectedness have resulted in the rapid and pervasive development of smart connected toys (SCTs), built with the capacity to collect terabytes of personal identifiable information, device context data, and play data. Any compromise of data stored, process, or transit can introduce privacy concerns, financial fraud concerns, and safety concerns, such as location identification, which can put child physical safety at risk. This work provides an overview of the SCT privacy problem and the landscape based on previous work to address privacy and safety concerns. We further investigate technical and legislative-related privacy issues in SCT and present a confirmatory study on developers’ privacy views based on the privacy calculus theory. Finally, we present an abstract continuous improvement process-based privacy preservation framework using Plan-Do-Check-Act, which can be adopted and used during the SCT design to minimize privacy breaches and serve as a foundation for the deployment of continuous privacy control improvement in a complex SCT development context.

Streams are temporally ordered, rapid changing, ample in volume, and infinite in nature. It is nearly impossible to store the entire data stream due to its large volume and high velocity. In this work, the principle of parallelism is employed to accelerate stream data computing. GPU-based high-performance computing (HPC) framework is proposed for accelerated processing of big-data streams using the in-memory data structure. We have implemented three parallel algorithms to prove the viability of the framework. The contributions of Mille Cheval are: (1) the viability of streaming on accelerators to increase throughput, (2) carefully chosen hash algorithms to achieve low collision rate and high randomness, and (3) memory sketches for approximation. The objective is to leverage the power of a single node using in-memory computing and hybrid computing. HPC does not always require high-end hardware but well-designed algorithms. Achievements of Mille Cheval are: (1) relative error is 1.32 when error rate and overestimate rate are chosen as 0.001 and (2) the host memory space requirement is just 63 MB for 1 terabyte of data. The proposed algorithms are pragmatic. It is evident from experimental results that the framework demonstrates 10X speed-up as compared with CPU implementations and 3X speed-up as compared with GPU implementations.

Documentation in one form or another is at the heart of any organisation large or small. When it comes to global intergovernmental organisations such as the International Atomic Energy Agency (IAEA...

The next decade will usher in a new era of technological innovation where compute, communications, and intelligence will converge. The number of connected devices is estimated to reach 500 billion by 2030, which is 59× larger than the expected world population [1] . Instead of humans dominating the use of wireless networks, objects or things will become the dominant users. Networks will incorporate intelligent compute to enable new types of applications such as cyberphysical worlds that replicate reality. These applications will require distributed intelligent compute platforms, as processing terabytes of data at a single edge device will be near impossible. Future 6G systems are currently being defined to meet these needs, as shown in Figure 1 , and the essential components of all 6G systems will be intelligence and security [2] . These new wireless systems will need to leverage a distributed compute network that is part of the wireless network so that the edge device can employ a high-performance compute platform that is in close proximity to itself. Such a network device must incorporate flexibility and programmability at low latency while also supporting network and connectivity functions. Heterogeneous integration in 2.5D and 3D can efficiently enhance existing compute platforms such as processors, application-specific integrated circuits, and field programmable gate arrays (FPGAs) to address emerging applications such as intelligent compute-communicate systems at lower cost with faster time to market. This article presents an overview of 2.5D and 3D heterogeneous technologies and how they can be leveraged to create new intelligent-compute-communicate platforms to enable emerging applications

Although many researchers talk about a “patient database,” they typically are not referring to a database at all, but instead to a spreadsheet of curated facts about a cohort of patients. This article describes relational database systems and how they differ from spreadsheets. At their core, spreadsheets are only capable of describing one-to-one (1:1) relationships. However, this article demonstrates that clinical medical data encapsulate numerous one-to-many relationships. Consequently, spreadsheets are very inefficient relative to relational database systems, which gracefully manage such data. Databases provide other advantages, in that the data fields are “typed” (that is, they contain specific kinds of data). This prevents users from entering spurious data during data import. Because each record contains a “key,” it becomes impossible to add duplicate information (ie, add the same patient twice). Databases store data in very efficient ways, minimizing space and memory requirements on the host system. Likewise, databases can be queried or manipulated using a highly complex language called SQL. Consequently, it becomes trivial to cull large amounts of data from a vast number of data fields on very precise subsets of patients. Databases can be quite large (terabytes or more in size), yet still are highly efficient to query. Consequently, with the explosion of data available in electronic health records and other data sources, databases become increasingly important to contain or order these data. Ultimately, this will enable the clinical researcher to perform artificial intelligence analyses across vast amounts of clinical data in a way heretofore impossible. This article provides initial guidance in terms of creating a relational database system.

Motivation Reproducibility is of central importance to the scientific process. The difficulty of consistently replicating and verifying experimental results is magnified in the era of big data, in which bioinformatics analysis often involves complex multi-application pipelines operating on terabytes of data. These processes result in thousands of possible permutations of data preparation steps, software versions and command-line arguments. Existing reproducibility frameworks are cumbersome and involve redesigning computational methods. To address these issues, we developed RepeatFS, a file system that records, replicates and verifies informatics workflows with no alteration to the original methods. RepeatFS also provides several other features to help promote analytical transparency and reproducibility, including provenance visualization and task automation. Results We used RepeatFS to successfully visualize and replicate a variety of bioinformatics tasks consisting of over a million operations with no alteration to the original methods. RepeatFS correctly identified all software inconsistencies that resulted in replication differences. Availabilityand implementation RepeatFS is implemented in Python 3. Its source code and documentation are available at https://github.com/ToniWestbrook/repeatfs. Supplementary information Supplementary data are available at Bioinformatics online.

Optical imaging has rapidly evolved in the last decades. Sophisticated microscopes allowing optical sectioning for three-dimensional imaging or sub-diffraction resolution are available. Due to price and maintenance issues, these microscopes are often shared between users in facilities. Consequently, long-term access is often prohibited and does not allow to monitor slowly evolving biological systems or to validate new models like organoids. Preliminary coarse long-term data that do not require acquisition of terabytes of high-resolution images are important as a first step. By contrast with expensive all-in-one commercialized stations, standard microscopes equipped with incubator stages offer a more cost-effective solution despite imperfect long-run atmosphere and temperature control. Here, we present the Incubascope , a custom-made compact microscope that fits into a table-top incubator. It is cheap and simple to implement, user-friendly and yet provides high imaging performances. The system has a field of view of 5.5 × 8 mm 2 , a 3 μm resolution, a 10 frames per second acquisition rate, and is controlled with a Python-based graphical interface. We exemplify the capabilities of the Incubascope on biological applications such as the hatching of Artemia salina eggs, the growth of the slime mould Physarum polycephalum and of encapsulated spheroids of mammalian cells.

In open-source software development environments; textual, numerical and relationship-based data generated are of interest to researchers. Various data sets are available for this data, which is frequently used in areas such as software engineering and natural language processing. However, since these data sets contain all the data in the environment, the problem arises in the terabytes of data processing. For this reason, almost all of the studies using GitHub data use filtered data according to certain criteria. In this context, using a different data set in each study makes a comparison of the accuracy of the studies quite difficult. In order to solve this problem, a common dataset was created and shared with the researchers, which would allow us to work on many software engineering problems., 7 pages, The article was submitted to Cumhuriyet Science Journal

Biological imaging tools continue to increase in speed, scale, and resolution, often resulting in the collection of gigabytes or even terabytes of data in a single experiment. In comparison, the ability of research laboratories to store and manage this data is lagging greatly. This leads to limits on the collection of valuable data and slows data analysis and research progress. Here we review common ways researchers store data and outline the drawbacks and benefits of each method. We also offer a blueprint and budget estimation for a currently deployed data server used to store large datasets from zebrafish brain activity experiments using light-sheet microscopy. Data storage strategy should be carefully considered and different options compared when designing imaging experiments.

The many terabytes of data that are generated from space missions must be analyzed, and accurate decisions must be made. This article highlights two important areas where automated machine learning (ML) systems are helping to science the data and allowing engineers and scientists to make more informed decisions without drowning in ones and zeros. One such area is exoplanet detection. This application is particularly unique because data analysis is crowdsourced as part of a citizen science effort in which members of the public who are not part of the science team take part, using powerful open source methods and software tools running on clusters of servers available in the cloud.

One challenge impeding the analysis of terabyte scale X-ray scattering data from the Linac Coherent Light Source (LCLS) is determining the number of clusters required for the execution of traditional clustering algorithms. Here, we demonstrate that the previous work using bi-cross validation to determine the number of singular vectors directly maps to the spectral clustering problem of estimating both the number of clusters and hyperparameter values. Applying this method to LCLS X-ray scattering data enables the identification of dropped shots without manually setting boundaries on detector fluence and provides a path toward identifying rare and anomalous events.

The launch of the big data era puts forward challenges for information preservation technology, both in storage capacity and security. Herein, a brand new optical storage medium, transparent glass ceramic (TGC) embedded with photostimulated LiGa5O8: Mn2+ nanocrystals, capable of achieving bit-by-bit optical data write-in and read-out in a photon trapping/detrapping mode, is developed. The highly ordered nanostructure enables light–matter interaction with high encoding/decoding resolution and low bit error rate. Importantly, going beyond traditional 2D optical storage, the high transparency of the studied bulk medium makes 3D volumetric optical data storage (ODS) possible, which brings about the merits of expanded storage capacity and improved information security. Demonstration application confirmed the erasable–rewritable 3D storage of binary data and display items in TGC with intensity/wavelength multiplexing. The present work highlights a great leap in photostimulated material for ODS application and hopefully stimulates the development of new multi-dimensional ODS media. An optical data storage (ODS) system that is cost-effective to make and easily scalable could lay the foundations for data storage technologies with higher capacity and improved security. The era of big data is driving the need for data storage systems with larger capacity, better security and improved performance. However, even the latest two-dimensional optical disk technologies cannot store more than one terabyte of data, leading scientists to explore alternative technologies. Now, a team of Chinese researchers, led by Yuansheng Wang from the Chinese Academy of Sciences, has developed a three-dimensional ODS system that uses light-emitting nanocrystals (NCs) to store and retrieve information. Made from a transparent glass ceramic embedded with photostimulated luminescent rare-earth-doped metallic NCs, the new ODS system can store 3D optical data with increased capacity and improved security compared with current technologies.

Key-value (KV) stores play an increasingly critical role in supporting diverse large-scale applications in modern data centers hosting terabytes of KV items which even might reside on a single server due to virtualization purposes. The combination of the ever-growing volume of KV items and storage/application consolidation is driving a trend of high storage density for KV stores. Shingled Magnetic Recording (SMR) represents a promising technology for increasing disk capacity, which however comes with the increased complexity of handling random writes. To take the best advantages of SMR drives, applications are expected to work in an SMR-friendly way. In this work, we present SEALDB, a Log-Structured Merge tree (LSM-tree) based key-value store that is specifically optimized for SMR drives via avoiding random writes and the corresponding write amplification on SMR drives. First, for LSM-trees, SEALDB collects and groups participating data of each compaction into sets. Using a set as the basic unit for compactions, SEALDB improves compaction efficiency by reducing random I/Os. Second, SEALDB creates variable sized bands on original HM-SMR drives, named dynamic bands. Dynamic bands store sets in an SMR-friendly way to eliminate the auxiliary write amplification from SMR drives. Third, SEALDB employs two light-weight garbage collection (GC) policies to further improve the space efficiency. We demonstrate the advantages of SEALDB via extensive experiments with various workloads. Overall, SEALDB delivers impressive performance compared with LevelDB, e.g., $3.42\times$3.42×/$2.65\times$2.65× faster for random writes (without or with GCs), and $3.96\times$3.96× faster for sequential reads.

With the emerging of the Internet of Things (IoT) technology, we can analyze and real-time monitoring the power consumption data of buildings or equipment. However, over time, many power data will accumulate to even the terabyte level. Traditional data processing techniques will not handle. It is a challenge to monitor and process the data in a reasonable time. In this paper, we construct an efficient and real-time power monitoring platform, based on open source techniques. The power-data of buildings or equipment is provided through smart meters equipped inside campus buildings. The technologies of data collected and stored on the server side were handled by the Hadoop ecosystem and the data process to Hive data warehouse is executed by the Spark. On the particulars of system evaluation, we make some experiments to compare with the performance of record query, use three different technologies separately: Apache Hive, Apache Spark, and Impala had been evaluated regarding query-response performance. Moreover, for the performance estimation on the data ETL processing and SQL functions. The relevant experiments have been conducted on the same three software modules as well.

Clustering is a popular technique that can help make large datasets more manageable and usable by grouping together similar objects. Most clustering approaches are too computationally expensive for datasets that are very large or complex. This work presents Parallel K-Tree, a hierarchical data structure and clustering algorithm that takes advantage of modern computing environments to cluster extremely large datasets. Parallel K-Tree produces high-quality clusters and scales more efficiently than traditional, parallelized and state-of-the-art approaches. Parallel K-Tree was applied to a large (8 terabyte) collection of Landsat 5 satellite images. This required clustering of 540 billion objects into eight billion clusters — a two orders of magnitude size increase over any reported alternative approach. Furthermore, Parallel K-Tree was executed on just two commodity servers — rather than a high-performance supercomputer.

Aligning genomic sequences is a simple method for handling and analyzing the information in Bioinformatics. Here, the author expresses the problems related to BLAST algorithm which is used for the alignment of the sequences of biological data as the data grows huge. The current BLAST algorithm provided by National Centre of Biotechnology Information (stand-alone) cannot address biological data dynamically that are in terabytes. Many schedulers have been proposed to address this issue. The existing sequencing methodology is the one which is based on Hadoop MapReduce framework that executes in a sequential manner by consuming more time and is cost expensive. Hence, the author in this paper has achieved the BLAST-BSPMAPREDUCE (Bulk Synchronization Parallel MapReduce) algorithm by further improving the BLAST (Basic Local Alignment Search Tool) algorithm that is on the basis of Azure Cloud principles to tackle the case of serial execution in Hadoop MapReduce framework. The exploratory study which has been presented in this work demonstrates that the proposed BSPMAPREDUCE has a better coordinating speed of bioinformatics genomic sequences and achieves much higher speed over existing Hadoop-BLAST algorithm and the experiment result shows that the proposed customized scheduler is highly robust and scalable.

The Covid Symptom Study, a smartphone-based surveillance study on COVID-19 symptoms in the population, is an exemplar of big data citizen science. As of May 23rd, 2021, over 5 million participants have collectively logged over 360 million self-assessment reports since its introduction in March 2020. The success of the Covid Symptom Study creates significant technical challenges around effective data curation. The primary issue is scale. The size of the dataset means that it can no longer be readily processed using standard Python-based data analytics software such as Pandas on commodity hardware. Alternative technologies exist but carry a higher technical complexity and are less accessible to many researchers. We present ExeTera, a Python-based open source software package designed to provide Pandas-like data analytics on datasets that approach terabyte scales. We present its design and capabilities, and show how it is a critical component of a data curation pipeline that enables reproducible research across an international research group for the Covid Symptom Study.

As simulations get larger and produce more data, domain scientists are increasingly using in situ methods to analyze and visualize data as they are being produced. Most tools for in situ, which allow domain scientists to monitor their simulations, such as ParaView Catalyst and VisIt Libsim, tend to be bulky and require the scientist to request and wait for compute nodes or download full datasets (of the order of terabytes or petabytes) on desktops/laptops for analysis. In this paper, we present a very lightweight framework that allows users to visualize snapshots of a simulation without having to request additional compute nodes from a supercomputer when analysis or visualization is needed or download full datasets. We extend the Seer framework, an open-source in situ framework, to sub-sample and store simulation data, which can then be accessed and manipulated from the user’s local machine using a dashboard created using Plotly’s Dash interface.

Data of the order of terabytes, petabytes, or beyond is known as Big Data. This data cannot be processed using the traditional database software, and hence there comes the need for Big Data Platforms. By combining the capabilities and features of various big data applications and utilities, Big Data Platforms form a single solution. It is a platform that helps to develop, deploy and manage the big data environment. Hadoop and Spark are the two open-source Big Data Platforms provided by Apache. Both these platforms have many configurational parameters, which can have unforeseen effects on the execution time, accuracy, etc. Manual tuning of these parameters can be tiresome, and hence automatic ways should be needed to tune them. After studying and analyzing various previous works in automating the tuning of these parameters, this paper proposes two algorithms - Grid Search with Finer Tuning and Controlled Random Search. The performance indicator studied in this paper is Execution Time. These algorithms help to tune the parameters automatically. Experimental results have shown a reduction in execution time of about 70% and 50% for Hadoop and 81.19% and 77.77% for Spark by Grid Search with Finer Tuning and Controlled Random Search, respectively., 12 Pages, 9 Figures, 12 Tables, Published with International Journal of Engineering Trends and Technology (IJETT)

The fast growing technology has left a great impact on the human life. Many traditional systems are either replaced or running in parallel with their electronic counterpart. As for example: - the traditional postal system is now nearly replaced by mobile phones and emails. The electronic system is providing more functionalities than their traditional counterparts. Due to social media, peoples may communicate with each other, share their thoughts and moments of life in form of texts, images or videos. On the other hand, to enhance technologies and knowledge many research activities are propelled and data from different sources are gathered in large volume for further analysis. In short today’s world is surrounded with large volume of data in different form. This put a requirement for effective management of these billions of terabytes of electronic data generally called BIG DATA. The effective management must be based on proven mathematical concepts so that chance of casualties may be reduced. The study objective of this is to provide a mathematical appraisal for evolution of distribution of file data and explains some basic solution of primitive problems based on probability theory. The probability theory helps us to formalize the basic tree like architecture of distributed file system if we try to solve the problem like Big Data.

The Gut Cell Atlas (GCA), an initiative funded by the Helmsley Charitable Trust, seeks to create a reference platform to understand the human gut, with a specific focus on Crohn’s disease. Although a primary focus of the GCA is on focusing on single-cell profiling, we seek to provide a framework to integrate other analyses on multi-modality data such as electronic health record data, radiological images, and histology tissues/images. Herein, we use the research electronic data capture (REDCap) system as the central tool for a secure web application that supports protected health information (PHI) restricted access. Our innovations focus on addressing the challenges with tracking all specimens and biopsies, validating manual data entry at scale, and sharing organizational data across the group. We present a scalable, cross-platform barcode printing/record system that integrates with REDCap. The central informatics infrastructure to support our design is a tuple table to track longitudinal data entry and sample tracking. The current data collection (by December 2020) is illustrated with types and formats of the data that the system collects. We estimate that one terabyte is needed for data storage per patient study. Our proposed data sharing informatics system addresses the challenges with integrating physical sample tracking, large files, and manual data entry with REDCap.

Clinical metagenomics is a technique that allows the search for an infectious agent in a biological tissue/fluid sample. Over the past few years, this technique has been refined, while the volume of data increases in an exponential way. DNA sequencers generate gigabytes of data and these data must be paired with bases that exceed one terabyte. The molecular nature of DNA does not allow the use of traditional string search algorithms, since biologically analogous sequences are not always syntactically identical, which makes it difficult to analyze matching sequences. In the case of clinical diagnosis, in specific cases, the sooner the diagnosis is available to the doctor, the higher are the chances of the patient's recovery. Therefore, the data processing must be as short as possible, maintaining the necessary detail and coverage. This paper describes some current techniques of computational processing and acceleration of the search for genomic data and explores possible alternative computational paths to streamline the process of metagenomic diagnosis.

In this era of data security crisis where the world around us works primarily with data, from the giant MNCs, Hospitals, Courts, to small businesses generate billions of terabytes of data every day. With this much data getting produced every day, it raises a massive problem of storing and keeping it safe. Handling this amount of data is the biggest challenge, and if a single piece of information is lost, that can cause a massive loss to the organization. For storing data, we can do it either electronically or physically. The probability of a file getting lost is more with hard copies. Still, most companies, government organizations, and others use old-traditional file management methods, making tracking the file between different sections difficult. Tracking and managing files are essential for every business, which also permits you to assess the overall business growth, which is amongst the most significant resources for future growth. The primary goal of this paper is to develop a more effective and practical approach to tracking files utilizing BLE (Bluetooth Low Energy) beacons and locator nodes for physical copies of information. It provides us with the file's real-time location and decreases the risk of essential files misplaced. And for the protection of electronic files, we propose a more reliable solution with cryptography and decentralization using blockchain.

Spurred by the increasing market needs for big data and cloud services, global server suppliers and hyper- scalers are looking to adopt high-speed and large-capacity memory modules. To fulfill this trend, the brand- new low-voltage operable DDR5 (double data rate 5th generation) memory can be an appropriate solution, with the highest speed of 7.2 Gbps and the largest capacity of 512 GB. However, some critical obstacles, such as increased capacity and high-speed I/O requirements, unstable power noise occurrences, high power consumption, and increase in operating temperature, must be overcome. This poster will cover various technical pathfinding solutions for world's first DDR5 512 GB module with an advanced DRAM process and I/O schemes, package technology, and module architecture regarding improvements in the following four aspects: performance, speed, capacity, and power. This will unveil the industry's first high-performance and large-capacity memory product with 8-stacked DDR5 DRAMs. Samsung believes that this product will pave the way for achieving both higher bandwidth and lower power consumption to inaugurate the era of terabyte DRAM modules for next-gen servers.

Earth observation analysis workflows commonly require mass processing of time series data, with data volumes easily exceeding terabyte magnitude, even for relatively small areas of interest. Cloud processing platforms such as Google Earth Engine (GEE) leverage accessibility to satellite image archives and thus facilitate time series analysis workflows. Instant visualization of time series data and integration with local data sources is, however, currently not implemented or requires coding customized scripts or applications. Here, we present the GEE Timeseries Explorer plugin which grants instant access to GEE from within QGIS. It seamlessly integrates the QGIS user interface with a compact widget for visualizing time series from any predefined or customized GEE image collection. Users can visualize time series profiles for a given coordinate as an interactive plot or visualize images with customized band rendering within the QGIS map canvas. The plugin is available through the QGIS plugin repository and detailed documentation is available online (https://geetimeseriesexplorer.readthedocs.io/).

Big data is a large collection of dataset from heterogeneous sources of data which may be terabytes or petabytes of data. The big data is useful for existing business growth and also supports to create the new business. Handling this much of data is very difficult in database management system. The problems of big data are storing, processing, analyzing, extracting, and privacy. This survey paper, mainly focused on challenges of big data, how to extract the required data from large volume of data, and also various clustering algorithm. For the extraction of data, mapreduce function is used which is mainly used in Google search engine.

The synapse is a central player in the nervous system serving as the key structure that permits the relay of electrical and chemical signals from one neuron to another. The anatomy of the synapse contains important information about the signals and the strength of signal it transmits. Because of their small size, however, electron microscopy (EM) is the only method capable of directly visualizing synapse morphology and remains the gold standard for studying synapse morphology. Historically, EM has been limited to small fields of view and often only in 2D, but recent advances in automated serial EM (“connectomics”) have enabled collecting large EM volumes that capture significant fractions of neurons and the different classes of synapses they receive (i.e. shaft, spine, soma, axon). However, even with recent advances in automatic segmentation methods, extracting neuronal and synaptic profiles from these connectomics datasets are difficult to scale over large EM volumes. Without methods that speed up automatic segmentation over large volumes, the full potential of utilizing these new EM methods to advance studies related to synapse morphologies will never be fully realized. To solve this problem, we describe our work to leverage Argonne leadership-scale supercomputers for segmentation of a 0.6 terabyte dataset using state of the art machine learning-based segmentation methods on a significant fraction of the 11.69 petaFLOPs supercomputer Theta at Argonne National Laboratory. We describe an iterative pipeline that couples human and machine feedback to produce accurate segmentation results in time frames that will make connectomics a more routine method for exploring how synapse biology changes across a number of biological conditions. Finally, we demonstrate how dendritic spines can be algorithmically extracted from the segmentation dataset for analysis of spine morphologies. Advancing this effort at large compute scale is expected to yield benefits in turnaround time for segmentation of individual datasets, accelerating the path to biology results and providing population-level insight into how thousands of synapses originate from different neurons; we expect to also reap benefits in terms of greater accuracy from the more compute-intensive algorithms these systems enable.

Data from emerging applications, such as cybersecurity and social networking, can be abstracted as graphs whose edges are updated sequentially in the form of a stream. The challenging problem of interactive graph stream analytics is the quick response of the queries on terabyte and beyond graph stream data from end users. In this paper, a succinct and efficient double index data structure is designed to build the sketch of a graph stream to meet general queries. A single pass stream model, which includes general sketch building, distributed sketch based analysis algorithms and regression based approximation solution generation, is developed, and a typical graph algorithm—triangle counting—is implemented to evaluate the proposed method. Experimental results on power law and normal distribution graph streams show that our method can generate accurate results (mean relative error less than 4%) with a high performance. All our methods and code have been implemented in an open source framework, Arkouda, and are available from our GitHub repository, Bader-Research. This work provides the large and rapidly growing Python community with a powerful way to handle terabyte and beyond graph stream data using their laptops.

Deployed in 2015, Discovery is New Mexico State University’s commonly-available High-Performance Computing (HPC) cluster. The deployment of Discovery was initiated by Information and Communication Technologies (ICT) employees from the Systems Administration group who wanted to help researchers run their computations on a more powerful system than one they had sitting in their offices. Over the years, the cluster has grown 6 times, and as of March 2021 has 52 compute nodes, 1480 CPU cores, 17 Terabytes of RAM, 30 GPUs, and 1.8 Petabytes of usable storage. Discovery’s hardware is acquired using a combination of university funds, condo-model based funds, and grant funds, causing Discovery to be a heterogeneous system containing several CPU generations. This paper discusses our growth and administration experiences on this heterogeneous system, as well as our outreach and contribution to the HPC community.

We argue that immature data pipelines are preventing a large portion of industry practitioners from leveraging the latest research on recommender systems. We propose our template data stack for machine learning at "reasonable scale", and show how many challenges are solved by embracing a serverless paradigm. Leveraging our experience, we detail how modern open source can provide a pipeline processing terabytes of data with limited infrastructure work., Manuscript version of a work accepted at RecSys 2021 (camera-ready forthcoming)

In today’s world, Modern organizations deal with terabytes of text, such as email, that often plays a significant role in their day to day operations. The user has to face a task of identifying useful information from these data which is difficult and it requires some amount of time. One possible means is to use text summarization. Text summarization is the process of identifying the most valued/meaningful information in a document and compressing that information into a shorter version preserving its overall meaning. Sentiment analysis is about determining the text given by the user whether it is Positive, Negative or Neutral. We used Gensim Algorithm for generating text summary. This algorithm automatically summarizes the given text, by extracting one or more important sentences from the text. This project is about text summarization which includes sentiment analysis. In UI, a text box will be displayed, which is used to take the input text from the user which need to be summarize. Then it will pre-process the text and show the summarized content. It will be taking input as URL of an article and it is going to provide Title, Author, Publication Date, Sentimental analysis, Keywords, URL of an Article.

Distributed denial of service (DDoS) attacks pose a huge threat to the Internet. Follow the rapid usage of the Internet of things (IoT), the DDoS attack is no longer a mere traffic attack, the original attack on the application layer surpasses the attack on the network layer. Furthermore, DDoS attacks using Bonnets result more destructive effects. This research aims to propose a new collaborated active defense framework between Honeypot and cloud platform to detect and defend future DDoS attacks in the context of the IoT with the instantaneous malicious traffic measured in Terabytes.

Interactive machine learning (iML) tools help to make ML accessible to users with limited ML expertise. However, gathering necessary training data and expertise for model-building remains challenging. Transfer learning, a process where learned representations from a model trained on potentially terabytes of data can be transferred to a new, related task, offers the possibility of providing ”building blocks” for non-expert users to quickly and effectively apply ML in their work. However, transfer learning largely remains an expert tool due to its high complexity. In this paper, we design a prototype to understand non-expert user behavior in an interactive environment that supports transfer learning. Our findings reveal a series of data- and perception-driven decision-making strategies non-expert users employ, to (in)effectively transfer elements using their domain expertise. Finally, we synthesize design implications which might inform future interactive transfer learning environments.

Currently, one of the most effective techniques to study the mechanisms of DNA replication is using the electron microscopy. Typical imaging experiments result in terabytes of image data, where finding and classifying the replication forks is done manually by experts and is extremely time consuming.Here, we present a fully automated deep learning based approach for detection and classification of DNA replication forks. It is based on the Faster R-CNN architecture, following with additional classification layers. The experimental results using real data indicate that the proposed method achieves detection accuracy which approaches the performance of expert annotators, even with limited amounts of training data.

Studying solar filaments can help in understanding the distribution of large-scale field in the solar atmosphere and provide useful insights into the nature of the solar magnetic field. Recently, the National Astronomical Observatory of the Chinese Academy of Sciences has collected huge volumes of data about solar filaments from several international observing stations for nearly 100 years. Designing a database, as well as establishing a networked tool for making the best of it, are necessary to provide efficient and convenient access to the data for researchers. In this study, we introduce a solar filament database, involving digital image data of nearly 45 terabytes, based on MySQL and a data management platform named database management system of hundred years of solar filament archives ( http://faste.ctgu.edu.cn ) that adopts Browser/Server mode and is written in the Java programming language. From this data system, users can realise multi-property query on the basis of demands, download data under multiple conditions, perform visual query for information on one single filament, view long-term data statistics of filament properties, and implement extensions and management of the database.

Citrix, a company that works with the likes of the FBI and US military, recently hit the headlines when it fell victim to hackers 1 . The cyber-criminals allegedly stole terabytes of sensitive documents including reports, blueprints and business papers, using a technique called password spraying. This approach, which exploits weak passwords, could be used to infiltrate many more companies like Citrix – and it's one more example of why the traditional username and password combination is no longer fit for purpose, in both our personal and business lives.

Genomics next generation sequencing (NGS) and third generation sequencing (TGS) have a broad area of applications in life science, such as Non-Invasive Prenatal Testing (NIPT), ctDNA Testing for Non-Invasive Tumor Personalized Therapy, Whole Genome Sequencing (WGS), Whole Exome Sequencing (WES), RNA Sequencing, etc. The high throughput genome sequencing instrument is capable of sequencing thousands of samples in parallel, generating dozens of terabytes of genomic data in one day. The storage and analysis of the big petabytes of genomic data are approaching a very challenge for much of the biomedical research communities. In this paper, we give the design and implementation of a genomics cloud, which can scale storage and computing abilities flexibly and provides many easy to use genomics analysis software for customers. This paper gives the technical solution for building a genomics cloud based on CWL/WDL, Docker, DAG, NAS and Object Storage System. The implemented cloud platform also frees scientists from the burden of: building high performance cluster, managing millions of genomic data files, and scripting genomics analysis pipelines.

Large amounts of data at various temporal and spatial scales require terabyte (TB) level storage and computation, both of which are not easy for researchers to access. Cloud data and computing services provide another solution to store, process, share and explore environmental data with low costs, stronger computation capacity and easy access. The purpose of this paper is to examine the benefits and challenges of using freely available satellite data products from Australian Geoscience DataCube and Google Earth Engine (GEE) online data with time series for integrative environmental analysis of the Macquarie-Castlereagh Basin in the last 15 years as a case study. Results revealed that the cloud platform simplifies the procedure of traditional catalog data processing and analysis. The integrated analysis based on the cloud computing and traditional methods represents a great potential as a low-cost, efficient and user-friendly method for global and regional environmental study. The user can save considerable time and cost on data integration. The research shows that there is an excellent promise in performing regional environmental analysis by using a cloud platform. The incoming challenge of the cloud platform is that not all kinds of data are available on the cloud platform. How data are integrated into a single platform while protecting or recognizing the data property, or how one portal can be used to explore data archived on different platforms represent considerable challenges.

When evaluating IR run effectiveness using a test collection, a key question is: What search topics should be used? We explore what happens to measurement accuracy when the number of topics in a test collection is reduced, using the Million Query 2007, TeraByte 2006, and Robust 2004 TREC collections, which all feature more than 50 topics, something that has not been examined in past work. Our analysis finds that a subset of topics can be found that is as accurate as the full topic set at ranking runs. Further, we show that the size of the subset, relative to the full topic set, can be substantially smaller than was shown in past work. We also study the topic subsets in the context of the power of statistical significance tests. We find that there is a trade off with using such sets in that significant results may be missed, but the loss of statistical significance is much smaller than when selecting random subsets. We also find topic subsets that can result in a low accuracy test collection, even when the number of queries in the subset is quite large. These negatively correlated subsets suggest we still lack good methodologies which provide stability guarantees on topic selection in new collections. Finally, we examine whether clustering of topics is an appropriate strategy to find and characterize good topic subsets. Our results contribute to the understanding of information retrieval effectiveness evaluation, and offer insights for the construction of test collections.