Search

In big data analysis at an enterprise, a useful option is the analysis of spatial data by machine learning, using a hybrid smart system. Machine learning permits complex nonlinear prediction with maximum precision and efficiency. A real-time big data prediction network for effective traffic flow is of great practical value. In such a network, the key problem is to construct an adaptive model on the basic of historical data. Traffic flow prediction is critical to traffic management in information technology. Real-time data proliferation led to the development of big data analysis, for which nonlinearity of traffic data is the main source of inaccurate prediction.

In today’s world of digitalization the most important object is data. There are numerous techniques available for retrieval of the data. To store this data over a large period of time various data hardware (DHW) are used. As the data is increasing day by day the more cost is incurred in storing the data. Data proliferation is the term used to refer the large amount of data that is created by government and nongovernment institutions either in structured or unstructured format. To minimize the maintenance cost generated by data proliferation IaaS as cloud computing service can be implemented. IaaS can be considered as virtualized service, which can be used to provide ‘as a service’ to the users via cloud vendors. It provides the access of database resources on demand. One can ‘pay-as-per-use’ of resource from the pool of resources and these resources can be virtualized in cloud based environment. To access the virtualized resources there are various approaches used. Private cloud approach is commonly used to maintain the integrity of the data.

In this paper, a new representation for the chi-squared distribution has been derived over wireless sensor networks (WSN). The underlying correlated data proliferation protocols have strong influence on performance of the deployed system. A proposed model has been deployed to investigate the WSN system over the data proliferation aspect. Initially, the degree of freedom (DOF) factor has been evaluated with respect to scalability issue in the proposal. Further, the factors affecting the outcome of the WSN system assessing data proliferation have been investigated. Moreover, sensor node operations namely: sense count, transmit count and receive redundant count have also been evaluated. Finally, extensive simulation analysis has been carried out to prove the validity of the proposed innovative scheme. However, it has also been investigated that chi-distribution for wireless sensor networks seems intractable with the degree of freedom, when varied with a specific number of nodes.

Cloud computing represents a kind of computing that is based on the sharing of computing resources instead of possessing personal devices or local servers for handling several applications and tasks. This kind of computing includes three distinguished kinds of services provided remotely for clients that can be accessed by using the Internet. Typically, clients work on paying annual or monthly service fees for suppliers, in order to gain access to systems that work on delivering infrastructure as a service, platforms as a service, and software as a service for any subscriber. In this paper, the usefulness and the abuse of the cloud computing are briefly discussed and presented by highlighting the influences of cloud computing in different areas. Moreover, this paper also presents the kinds and services of cloud. In addition, the security issues that cover the cloud security solution requirements, and the cloud security issues, which is one of the biggest issues in recent years in cloud computing were presented in this paper. The security requirement that needs by the cloud computing covers privacy, lack of user control, unauthorized secondary usage, and finally data proliferation and data flow. Meanwhile, the security issues cover including ownership of device, the trust issue and legel aspects. To overcome the security issues, this paper also presents the solution at the end of this paper.

As geospatial missions age, one of the challenges for the usability of data is the availability of relevant and updated metadata with sufficient documentation that can be used by future generations of users to gain knowledge from the original data. Given that remote sensing data undergo many intermediate processing steps, for example, an understanding of the exact algorithms employed and the quality of that data produced could be key considerations for these users. As interest in global climate data is increasing, documentation about older data, their origins, and their provenance are valuable to first-time users attempting to perform historical climate research or comparative analysis of global change. Incomplete or missing documentation could be what stands in the way of a new researcher attempting to use the data. Therefore, preservation of documentation and related metadata is sometimes just as critical as the preservation of the original observational data. The Goddard Earth Sciences–Data and Informa...

Today Big data is one of the most widely spoken about technology that is being explored throughout the world by technology enthusiasts and academic researchers. The reason for this is the enormous data generated every second of each day. Every webpage visited, every text message sent, every post on social networking websites, check-in information, mouse clicks etc. is logged. This data needs to be stored and retrieved efficiently, moreover the data is unstructured therefore the traditional methods of strong data fail. This data needs to be stored and retrieved efficiently There is a need of an efficient, scalable and robust architecture that needs stores enormous amounts of unstructured data, which can be queried as and when required. In this paper, we come up with a novel methodology to build a data warehouse over big data technologies while specifically addressing the issues of scalability and user performance. Our emphasis is on building a data pipeline which can be used as a reference for future research on the methodologies to build a data warehouse over big data technologies for either structured or unstructured data sources. We have demonstrated the processing of data for retrieving the facts from data warehouse using two techniques, namely HiveQL and MapReduce.

The extremely fast grow of Internet Services, Web and Mobile Applications and advance of the related Pervasive, Ubiquity and Cloud Computing concepts have stimulated production of tremendous amounts of data partially available online (call metadata, texts, emails, social media updates, photos, videos, location, etc.). Even with the power of today's modern computers it still big challenge for business and government organizations to manage, search, analyze, and visualize this vast amount of data as information. Data-Intensive computing which is intended to address this problems become quite intense during the last few years yielding strong results. Data intensive computing framework is a complex system which includes hardware, software, communications, and Distributed File System (DFS) architecture. Just small part of this huge amount is structured (Databases, XML, logs) or semi-structured (web pages, email), over 90% of this information is unstructured, what means data does not have predefined structure and model. Generally, unstructured data is useless unless applying data mining and analysis techniques. At the same time, just in case if you can process and understand your data, this data worth anything, otherwise it becomes useless. Two key components of any Data-intensive system are: Data Storage and Data Processing. So, which technologies, techniques, platforms, tools are best for Big Data storing and processing? How Big Data Era effect technological landscape? These and many other questions will be answered during a speech.

Digital data both structured and unstructured data is everywhere and continues to grow approximately every two years. Due to increasing uses of data, creating tremendous data storage challenges in all industry. Object based Storage can be applied in both public and private clouds service models. Object based storage is used to store the data from various users through REST API Using HTTP(S) protocol and it also stores the virtual machines images as objects in cloud environment. Cloud storage services are used by all enterprises, home users and researchers to store their files that vary in size and formats. With the huge amount of stored data which is mostly unstructured, comes the related issue of finding the exact data relevant to the requirements of a user. This paper will cover the need of object storage system in cloud computing for massive unstructured digital static data and why traditional approaches are insufficient to meet this challenge. The basic architecture of object storage systems, how the object storage system supports cloud storage environment to store unstructured data effectively. Additionally we will compare and contrast the major object based storage cloud providers in use today including Google, Microsoft and Amazon.

The amount of data in world is exploding. Data is being collected and stored at unprecedented rates. The challenge is not only to store and manage the vast volume of data, but also to analyze and extract meaningful value from it. In the last decade Data Warehousing technology has been evolved for efficiently storing the data from different sources for business intelligence purpose. In the Age of the Big Data, it is important to remodel the existing warehouse system that will help you and your organization make the most of unstructured data with your existing Data Warehouse. As Big Data continues to revolutionize how we use data, this paper addresses how to leverage big data by effectively integrating it to your data warehouse.

With the rapid development of big data technologies, more and more companies focus on the integration and analysis of the large scale data which are produced when the companies run, in order to discovering and solving their inner problems. Therefore, a data analysis system is in urgent need to process the big data generated by heterogeneous data sources (e.g. online and offline structured data, semi-structured data and non-structured data). Unfortunately, there is no integrated tool which can universally collect heterogeneous data sources, accomplish data ETL, storage, analysis, and knowledge output. In this paper, an integrated tool: CertDB is proposed, which can provide one-stop services of data collection, analysis, and knowledge output. The CertDB provides services to both expert and inexperienced data analysts, by proving both graphical BI interface and programming interface.

Research data is being generated at an ever-increasing rate. This brings challenges in how to store, analyse, and care for the data. A component of this problem is the stewardship of data and associated files that need a safe and secure home for the medium to long-term. As part of typical suites of Research Data Management services, researchers are provided with large allocations of ‘active data storage’. This is often stored on expensive and fast disks to enable efficient transfer and working with large amounts of data. However, over time this active data store fills up, and researchers need a facility to move older but still valuable data to cheaper storage for long-term care. In addition, research funders are increasingly requiring data to be stored in forms that allow it to be described and retrieved in the future. For data that can’t be shared publicly in an open repository, a closed solution is required that can make use of offline or near-line storage for cost efficiency. This paper describes a solution to these requirements, called the Data Vault.

Big Data has nowadays become the most talked latest IT trends. The fact that it can handle all the forms of data which includes unstructured data, big data has now become the preferred choice for analysis of huge amount of data over the Relational Database Management System(RDBMS). Big Data is helpful for the analysis of petabytes of data which is not possible in case of normal database system. But as everything comes with pros and cons so does big data. There are certain challenges which big data analytics is facing. The various challenges include validating the end point input, handling enormous amount of data on a very large scale, ensuring the security of transactional data, ensuring the safe and secure storage of data, sharing of variants of data with third party and analyzing those data without skipping even a single piece of information in order to generate reports and draw a conclusion. In this paper, we will be considering the major challenge to the big data and that is data security. Even with enormous advantages, the industry is taking a backseat for the shift from normal database to big data because of the data privacy concern. Even many big organizations don't consider big data to be a safe option as the data can be accessed by anyone. Certain different methods like the encryption-decryption technique, anonymization based, etc have been suggested by the researchers who are working to overcome the major threat of Big Data i.e. Data security. But unfortunately because of the 3 V's of the Big Data as stated by Gartner i.e. Velocity, Volume and Variety these methods didn't prove to be advantageous. Basically in this paper we will be discussing the various concern issues of Big data, out of which our main focus will be on data security.

The digital economy has been termed a data-driven economy. Many digital business models are based on the collection and the processing of various types of data. Whereas the protection of personal data falls under the EU legal framework, and first and foremost the General Data Protection Regulation, the protection of data that goes beyond personal data may seem to be not protected at all. The latter circumstance is important in light of the EU Commission’s considerations on the introduction of rights related to the protection of non-personal data. Indeed, the question of whether there is a need for the protection of such data – called “industrial data” - and the issue of access to such data have also been at the center of a recent academic debate. Yet, whereas the protection of different types of data is highly relevant when it comes to the transfer and the processing of data, for the question of “access to data” data has to be considered more broadly, i.e. as a digital resource. Since in the digital economy it is no longer individual pieces of data (personal and/or non-personal) that play a role, but rather collections of data, digital data sets (which may consist of both personal and non-personal data) have become a digital asset of companies. The value of such digital data sets derives from the analysis of data – importantly though, the analysis of real-time data. Due to this feature of data, i.e. the fact that data is constantly changing, the question that arises as regards access to data in the digital economy relates not so much to access to data as such, but rather to access to the sources of data. After all, it is only in this case that access can be provided to real-time data. In light of this, the current debate on so-called “industrial data” may appear too static and, besides, it considers only part of the data that is relevant for the companies operating in the digital economy. The question is what the legal framework for an optimal allocation of such a digital resource as data should be and whether the question of access can be solved with the currently available legal tools.

The amount of data in the world seems increasing and computers make it easy to save the data. Companies offer data storage by providing cloud services and the amount of data being stored in these servers is increasing rapidly. In data mining, the data is stored electronically and the search is automated or at least augmented by computer. As the volume of data increases, inexorably, the proportion of it that people understand decreases alarmingly. This paper presents the data leakage problem arises because the services like Facebook and Google store all your data unencrypted on their servers, making it easy for them, or governments and hackers, to monitor the data.

In the Information Technology era information plays vital role in every sphere of the human life. It is very important to gather data from different data sources, store and maintain the data, generate information, generate knowledge and disseminate data, information and knowledge to every stakeholder. Due to vast use of computers and electronics devices and tremendous growth in computing power and storage capacity, there is explosive growth in data collection. The storing of the data in data warehouse enables entire enterprise to access a reliable current database. To analyze this vast amount of data and drawing fruitful conclusions and inferences it needs the special tools called data mining tools. This paper gives overview of the data mining systems and some of its applications.

In today's HPC world, numerous applications executed with high speed. The multi core architecture has provided a wide scope of exploration towards any kind of high-end applications. The paper gives an analogy on ability to handle the high-end input triggers by managing the core utilities at the lower end of the computer. The optimized way of memory block utilization, debugging proliferation and data management at configured input level discussed in the paper. The Data Proliferation framework model named as COMCO (Component for Compiler Optimization) model and it elaborates on how to handle configured inputs at the OS level. The paper gives an exploration of the configured inputs safely at the multi thread level using the COMCO (Component for Compiler Optimization) strategy.

The field of data analysis seeks to extract value from data for either business or scientific benefit. This field has seen a renewed interest with the advent of big data technologies and a new organizational role called data scientist. Even with the new found focus, the task of analyzing large amounts of data is still challenging and time-consuming. The essence of data analysis involves setting up data pipe-lines which consists of several operations that are chained together - starting from data collection, data quality checks, data integration, data analysis and data visualization (including the setting up of interaction paths in that visualization). In our opinion, the challenges stem from from the technology diversity at each stage of the data pipeline as well as the lack of process around the analysis. In this paper we present a platform that aims to significantly reduce the time it takes to build data pipelines. The platform attempts to achieve this in following ways. Allow the user to describe the entire data pipeline with a single language and idioms - all the way from data ingestion to insight expression (via visualization and end-user interaction). Provide a rich library of parts that allow users to quickly assemble a data analysis pipeline in the language. Allow for a collaboration model that allows multiple users to work together on a data analysis pipeline as well as leverage and extend prior work with minimal effort. We studied the efficacy of the platform for a data hackathon competition conducted in our organization. The hackathon provided us with a way to study the impact of the approach. Rich data pipelines which traditionally took weeks to build were constructed and deployed in hours. Consequently, we believe that the complexity of designing and running the data analysis pipeline can be significantly reduced; leading to a marked improvement in the productivity of data analysts/data scientists.

Big Data refer to the analysis of significantly large collection of data that may contain user data, sensor data or machine data. An analyzed Big Data can deliver new business insights, open new markets and create competitive advantages. It consists of data sets that are of large magnitude (Volume), large collection of data which diverse representation include structured, semi structured, or unstructured data (Variety), and should arrive fast (velocity). The real value of data is observed after analyzing i.e. after finding patterns, deriving meanings, making decisions, the ultimate data is available whichever required. This paper discusses the need of Big Data and its analysis with the help of four V's. Further it gives the availability of data in day to day life because of which the requirement of Big Data was generated. The challenges, its privacy, security and analytic are taken into account. The security of Big Data is of major concerned thus different types are cyber attacks and threat detections are need to be discussed.

From traditional to today's data network text documents, pictures, audio and video mainstream, the Internet is gradually changing the structure of the data, the data from the non-structured data, which are growing and a wide variety of unstructured data, the Internet data Storage management has brought new challenges. In this paper, for all kinds of massive unstructured data storage problems of the proposed solutions, summed up the key issues to achieve unified storage of unstructured data, design and implement a batch framework of a non-structured data using a unified data storage features, to solve the problem of unified treatment of various types of unstructured data.

While a lot of work has been published on clustering of data on storage medium, little has been done about automating this process. This is an important area because with data proliferation, human attention has become a precious and expensive resource. Our goal is to develop an automatic and dynamic database clustering technique that will dynamically re-cluster a database with little intervention of a database administrator (DBA) and maintain an acceptable query response time at all times. In this paper we describe the issues that need to be solved when developing such a technique.

This paper summarizes opportunities and challenges of big data. It identifies important research directions and includes a number of questions that have been debated by the panel.

Data integration is the management of data that travels between application systems and organizations. A data interface is the application that passes data between application systems. Most information technology planning organizations focus more on data that persists in data stores and very little on the data that passes between systems, although having the right data, at the right time, in the right format is critical to the core goal of trusted data. Integrating data may involve moving and transforming data between application systems or around the organization; it also involves both data in databases (structured data) and unstructured data. In the age of huge volumes of data, it is sometimes more efficient to distribute the processes out to the data and then consolidate the results.

Data De-duplication has being used comprehensively within Disk-based Backups, Archives and Disaster Recovery. This technology's high data reduction ratio and satisfied performance in secondary data storage system has aroused huge interests in other fields. The primary storage system is indeed a field being long for data reduction. Data de-duplication maybe a chance to alleviate data redundancy problem within that field. Actually, there are still lots of problems and technology challenges in resolving the contradiction between strict performance requirements in the primary storage system and the resources consuming character of the data de-duplication. This article will give an introduction about those technology challenges and propose a new data storage framework trying to improve those problems.

With the proliferation of organizations that independently collect various types of data, with the growing awareness of corporations and public to keep their sensitive data private, and with the ever-increasing need of government and corporate policy makers to learn the behavior of their customers, there is a definite demand for data mining services even when the data owners refuse to provide their data directly. In the past, techniques such as random perturbation were used by data owners prior to sharing the data with a third-party data miner. But, as already proven, even these techniques are prone to privacy-violation.In this paper, we take a completely different approach---each data owner derives association rules locally, sanitizes them if necessary, and sends them to a third-party data miner. The data miner collects local rules from all data owners, regenerates an estimate of global data, and performs global data mining. We suggest schemes to reduce the generation of spurious rules, a possible outcome of data generation from rules. The proposed method is illustrated using an example of association rule data mining. We are currently in the process of formalizing some of the underlying techniques and to make them more efficient.

the management of unstructured data is recognized as one of the major unsolved problems in the information technology (IT) industry, the main reason being that the tools and techniques that have proved so successful transforming structured data into business intelligence and actionable information simply don't work when it comes to unstructured data. This unstructured data is often stored outside the database, separate from its structured data. This separation can cause data management complexities. Or, if the data is associated with structured storage, the file streaming capabilities and performance can be limited. User data is produced and generated on big scale where measurement is hundreds of MB or even couple of GB per user per day. Efficient method for storing and accessing those kinds of data is imperative in modern information society. In this paper we are presenting results of research on using different data types for storing unstructured data within database and classic file system. Research is inspired with current situation in modern information society.

Nowadays, there is a vast amount of information available in the internet. The useful data must be captured and stored for future purposes. One of the major unsolved problems in the information technology (IT) industry is the management of unstructured data. The unstructured data such as multimedia files, documents, spreadsheets, news, emails, memorandums, reports and web pages are difficult to capture and store in the common database storage. The underlying reason is due to the tools and techniques that proved to be so successful in transforming structured data into business intelligence and actionable information, simply do not work when it comes to unstructured data. As a result, new approaches are necessary. Attempts have been undertaken by several researchers to deal with unstructured data, but, so far it is hard to find a tool that can store and retrieve the extracted and classified unstructured data into a structured database system. This paper is to present our research on unstructured data identification, extraction and classification of web pages, which is then transformed into structured format in Extensible Markup Language (XML) document, and later stored into a multimedia database. The contribution of this research is in the approach of capturing the unstructured data and the efficiency of a multimedia database to handle this kind of data. The stored data could give benefits to various communities such as students, lecturers, researchers and IT managers because it can be used for any planning, decision-making, day-today operations, and other future purposes.