Search

The ubiquity of data lakes has created fascinating new challenges for data management research. In this tutorial, we review the state-of-the-art in data management for data lakes. We consider how data lakes are introducing new problems including dataset discovery and how they are changing the requirements for classic problems including data extraction, data cleaning, data integration, data versioning, and metadata management.

We present VISE, or Vehicle Image Search Engine, to support the fast search of similar vehicles from low-resolution traffic camera images. VISE can be used to trace and locate vehicles for applications such as police investigations when high-resolution footage is not available. Our system consists of three components: an interactive user-interface for querying and browsing identified vehicles; a scalable search engine for fast similarity search on millions of visual objects; and an image processing pipeline that extracts feature vectors of objects from video frames. We use transfer learning technique to integrate state-of-the-art Convolutional Neural Networks with two different refinement methods to achieve high retrieval accuracy. We also use an efficient high-dimensional nearest neighbor search index to enable fast retrieval speed. In the demo, our system will offer users an interactive experience exploring a large database of traffic camera images that is growing in real time at 200K frames per day.

We developed T oronto O pen D ata S earch to support the ad hoc , interactive discovery of connections or linkages between datasets. It can be used to efficiently navigate through the open data cloud. Our system consists of three parts: a user-interface provided by a Web application; a scalable backend infrastructure that supports navigational queries; and a dynamic repository of open data tables. Our system uses LSH Ensemble, an efficient index structure, to compute linkages (attributes in two datasets with high containment score) in real time at Internet scale. Our application allows users to navigate along these linkages by joining datasets. LSH Ensemble is scalable, providing millisecond response times for linkage discovery queries even over millions of datasets. Our system offers users a highly interactive experience making unrelated (and unlinked) dynamic collections of datasets appear as a richly connected cloud of data that can be navigated and combined easily in real time.

International audience

Myocardial infarction therapies involving biomaterial injections have shown benefits in inhibiting progression towards heart failure. However, the underlying mechanisms remain unclear. A finite element model of the human left ventricle was developed from magnetic resonance images. An anteroapical infarct was represented at acute (AI) and fibrotic (FI) stage. Hydrogel injections in the infarct region were modelled with layered (L) and bulk (B) distribution. In the FI, injectates reduced end-systolic myofibre stresses from 291.6% to 117.6% (FI-L) and 115.3% (FI-B) of the healthy value, whereas all AI models exhibited sub-healthy stress levels (AI: 90.9%, AI-L: 20.9%, AI-B: 30.5%). Reduction in end-diastolic infarct stress were less pronounced for both FI (FI: 294.1%, FI-L: 176.5%, FI-B: 188.2%) and AI (AI: 94.1%, AI-L: 35.3%, AI-B: 41.2%). In the border zone, injectates reduced end-systolic fibre stress by 8-10% and strain from positive (AI) and zero (FI) to negative. Layered and bulk injectates increased ejection fraction by 7.4% and 8.4% in AI and 14.1% and 13.7% in FI. The layered injectate had a greater impact on infarct stress and strain at acute stage, whereas the bulk injectate exhibited greater benefits at FI stage. These findings were confirmed by our previous in vivo results.

We present BibBase, a system for publishing and managing bibliographic data available in BiBTeX files. BibBase uses a powerful yet light-weight approach to transform BiBTeX files into rich Linked Data as well as custom HTML code and RSS feed that can readily be integrated within a user's website while the data can instantly be queried online on the system's SPARQL endpoint. In this paper, we present an overview of several features of our system. We outline several challenges involved in on-the-fly transformation of highly heterogeneous BiBTeX files into high-quality Linked Data, and present our solution to these challenges.

Repairing erroneous or conflicting data that violate a set of constraints is an important problem in data management. Many automatic or semi-automatic data-repairing algorithms have been proposed in the last few years, each with its own strengths and weaknesses. Bart is an open-source error-generation system conceived to support thorough experimental evaluations of these data-repairing systems. The demo is centered around three main lessons. To start, we discuss how generating errors in data is a complex problem, with several facets. We introduce the important notions of detectability and repairability of an error, that stand at the core of Bart. Then, we show how, by changing the features of errors, it is possible to influence quite significantly the performance of the tools. Finally, we concretely put to work five data-repairing algorithms on dirty data of various kinds generated using Bart, and discuss their performance.

As Web applications mature and evolve, the nature of the semistructured data that drives these applications also changes. An important trend is the need for increased flexibility in the structure of Web documents. Hence, applications cannot rely solely on schemas to provide the complex knowledge needed to visualize, use, query and manage documents. Even when XML Web documents are valid with regard to a schema, the actual structure of such documents may exhibit significant variations across collections for several reasons: the schema may be very lax (e.g., RSS feeds), the schema may be large and different subsets of it may be used in different documents (e.g., industry standards like UBL), or open content models may allow arbitrary schemas to be mixed (e.g., RSS extensions like those used for podcasting). For these reasons, many applications that incorporate XPath queries to process a large Web document collection require an understanding of the actual structure present in the collection, and not just the schema. To support modern Web applications, we introduce DescribeX, a powerful framework that is capable of describing complex XML summaries of Web collections. DescribeX supports the construction of heterogenous summaries that can be declaratively defined and refined by means of axis path regular expression (AxPREs). AxPREs provide the flexibility necessary for declaratively defining complex mappings between instance nodes (in the documents) and summary nodes. These mappings are capable of expressing order and cardinality, among other properties, which can significantly help in the understanding of the structure of large collections of XML documents and enhance the performance of Web applications over these collections. DescribeX captures most summary proposals in the literature by providing (for the first time) a common declarative definition for them. Experimental results demonstrate the scalability of DescribeX summary operations (summary creation, as well as refinement and stabilization, two key enablers for tailoring summaries) on multi-gigabyte Web collections.

International audience; Integration systems are typically evaluated using a few real-world scenarios (e.g., bibliographical or biological datasets) or using synthetic scenarios (e.g., based on star-schemas or other patterns for schemas and constraints). Reusing such evaluations is a cumbersome task because their focus is usually limited to showcasing a specific feature of an approach. This makes it difficult to compare integration solutions, understand their generality, and understand their performance for different application scenarios. Based on this observation, we demonstrate some of the requirements for developing integration benchmarks. We argue that the major abstractions used for integration problems have converged in the last decade which enables the application of robust empirical methods to integration problems (from schema evolution, to data exchange, to answering queries using views and many more). Specifically, we demonstrate that schema mappings are the main abstraction that now drives most integration solutions and show how a metadata generator can be used to create more credible evaluations of the performance and scalability of data integration systems. We will use the demonstration to evangelize for more robust, shared empirical evaluations of data integration systems.

If Lisa visits Dr. Brown, and there is no record of the drug he prescribed her, can we find it? Data sources, much to analysts' dismay, are too often plagued with incompleteness, making business analytics over the data difficult. Data entries with incomplete values are ignored, making some analytic queries fail to accurately describe how an organization is performing. We introduce a principled way of performing value imputation on missing values, allowing a user to choose a correct value after viewing possible values and why they were inferred. We achieve this by turning our data into a graph network and performing link prediction on nodes of interest using the belief propagation algorithm.

Clio is a system for managing and facilitating the complex tasks of heterogeneous data transformation and integration. In Clio, we have collected together a powerful set of data management techniques that have proven invaluable in tackling these difficult problems. In this paper, we present the underlying themes of our approach and present a brief case study.

The DataWeb project seeks to facilitate the querying and browsing of multimedia databases over a wide spectrum of networks and media. We developed an intelligent query facility that builds on the access paradigms supported by current hypertext-style Web applications and on decision support systems. This facility generalizes the basic navigation and abstraction mechanisms of these applications to make them extensible and scalable.

In the new era of Business Intelligence (BI) technology, transforming massive amounts of data into high-quality business information is essential. To achieve this, two non-overlapping worlds need to be aligned: the Information Technology (IT) world, represented by an organization’s operational data sources and the technologies that manage them (data warehouses, schemas, queries, ...), and the business world, portrayed by business plans, strategies and goals that an organization aspires to fulfill. Alignment in this context means mapping business queries into BI queries, and interpreting the data retrieved from the BI query in business terms. We call the creation of this interpretation the vivification problem. The main thesis of this position paper is that solutions to the vivification problem should be based on a formal framework that explicates assumptions and the other ingredients (schemas, queries, etc.) that affect it. Also, that there should be a correctness condition that explicates when a response to a business schema query is correct. The paper defines the vivification problem in detail and sketches approaches towards a solution.

An emerging therapy to limit adverse heart remodelling following myocardial infarction (MI) is the injection of polymers into the infarcted left ventricle (LV). In the few numerical studies carried out in this field, the definition and distribution of the hydrogel in the infarcted myocardium were simplified. In this computational study, a more realistic biomaterial distribution was simulated after which the effect on cardiac function and mechanics was studied. A validated finite element heart model was used in which an antero-apical infarct was defined. Four infarct models were created representing different temporal phases in the progression of a MI. Hydrogel layers were simulated in the infarcted myocardium in each model. Biomechanical and functional improvement of the LV was found after hydrogel inclusion in the ischaemic models representing the early phases of MI. In contrast, only functional but no mechanical restitution was shown in the scar model due to hydrogel presence.

To make semantic search a reality, we need to be able to efficiently publish large data sets containing rich semantic structure. We have tools for translating relational and semi-structured data into RDF, but such translation tools do not have the goal of adding or providing the kind of semantics necessary to achieve the goals of the Semantic Web and semantic search over the Web. In this chapter, we present LinQuer, a tool for creating semantic links within a data source and between data sources. We focus on link discovery over structured (relational) data since many Semantic Web sources are the result of publishing relational data as RDF and since relational engines provide the scalability and flexibility we need for large scale link discovery. The LinQuer framework is based on the declarative specification of linkage requirements by a user. We present algorithms for translating these requirements to queries that can run over relational data sources, potentially using semantic information (such as a class hierarchy or a more general ontology) to enhance the recall of the link discovery. We show that this framework is flexible enough to permit linking real data, including dirty data (which is commonly found on the Web) and data with a variety of semantic connections.

Current theoretical work offers measures of schema equivalence based on the information capacity of schemas. This work is based on the existence of abstract functions satisfying various restrictions between the sets of all instances of two schemas. In considering schemas that arise in practice, however, it is not clear how to reason about the existence of such abstract functions. Further, these notions of equivalence tend to be too liberal in that schemas are often considered equivalent when a practitioner would consider them to be different. As a result, practical integration methodologies have not utilized this theoretical foundation and most of them have relied on ad-hoc approaches. We present results that seek to bridge this gap. First, we consider the problem of deciding information capacity equivalence and dominance of schemas that occur in practice, i.e., those that can express inheritance and simple integrity constraints. We show that this problem is undecidable. This undecidability suggests that in addition to the overly liberal nature of information capacity equivalence, we should look for alternative, more restrictive notions of equivalence that can be effectively tested. To this end, we develop several tests that each serve as sufficient conditions for information capacity equivalence or dominance. Each test is characterized by a set of schema transformations in the following sense: a test declares that Schema S1 is dominated by schema S2 if and only if there is a sequence of transformations that converts S1 to S2. Thus, each test can be understood essentially by understanding the individual transformations used to characterize it. Each of the transformations we consider is a local, structural schema change with a clear underlying intuition. We demonstrate the power of these tests by showing that one can reason about the equivalence and dominance of quite complex schemas. Because our work is based on structural transformations, the same characterizations that underly our tests can be used to guide designers in modifying a schema to meet their equivalence or dominance goals.

Schemas, and more generally metadata specifying structural and semantic constraints, are invaluable in data management. They facilitate conceptual design and enable checking of data consistency. They also play an important role in permitting semantic query optimization, that is, optimization and processing strategies that are often highly effective, but only correct for data conforming to a given schema. While the use of metadata is well-established in relational and XML databases, the same is not true for data streams. The existing work mostly focuses on the specification of dynamic information. In this paper, we consider the specification of static metadata for streams in a model called Stream Schema. We show how Stream Schema can be used to validate the consistency of streams. By explicitly modeling stream constraints, we show that stream queries can be simplified by removing predicates or subqueries that check for consistency. This can greatly enhance pro-grammability of stream processing systems. We also present a set of semantic query optimization strategies that both permit compile-time checking of queries (for example, to detect empty queries) and new runtime processing options, options that would not have been possible without a Stream Schema specification. Case studies on two stream processing platforms (covering different applications and underlying stream models), along with an experimental evaluation, show the benefits of Stream Schema.

Two major forms of information integration, federation and materialization, continue to dominate the market, embedded in separate products, each with their strengths and weaknesses. Application developers must make difficult choices among techniques and products, choices that are hard to change later. We propose a new design principle, Integration Independence, for integration engines. Integration independence frees the application designer from deciding how to integrate data. We then describe a new, adaptive information integration engine that provides the ability to index base data or to materialize transformed data, giving us a flexible platform for experimentation.

Discovering correspondences between schema elements is a crucial task for data integration. Most schema matching tools are semi-automatic, e.g. an expert must tune some parameters (thresholds, weights, etc.). They mainly use several methods to combine and aggregate similarity measures. However, their quality results often decrease when one requires to integrate a new similarity measure or when matching particular domain schemas. This paper describes YAM (Yet Another Matcher), which is a schema matcher factory. Indeed, it enables the generation of a dedicated matcher for a given schema matching scenario, according to user inputs. Our approach is based on machine learning since schema matchers can be seen as classifiers. Several bunches of experiments run against matchers generated by YAM and traditional matching tools show how our approach is able to generate the best matcher for a given scenario.

Discovering links between different data items in a single data source or across different data sources is a challenging problem faced by many information systems today. In particular, the recent Linking Open Data (LOD) community project has highlighted the paramount importance of establishing semantic links among web data sources. Currently, LOD sources provide billions of RDF triples, but only millions of links between data sources. Many of these data sources are published using tools that operate over relational data stored in a standard RDBMS. In this paper, we present a framework for discovery of semantic links from relational data. Our framework is based on declarative specification of linkage requirements by a user. We illustrate the use of our framework using several link discovery algorithms on a real world scenario. Our framework allows data publishers to easily find and publish high-quality links to other data sources, and therefore could significantly enhance the value of the data in the next generation of web.

In this paper, we present YAM, a schema matcher factory. YAM (Yet Another Matcher) is not (yet) another schema matching system as it enables the generation of a la carte schema matchers according to user requirements. These re- quirements include a preference for recall or precision, a training data set (schemas already matched) and provided expert correspondences. YAM uses a knowledge base that includes a (possibly large) set of similarity measures and classifiers. Based on the user requirements, YAM learns how to best apply these tools (similarity measures and clas- sifiers) in concert to achieve the best matching quality. In our demonstration, we will let users apply YAM to build the best schema matcher for different user requirements.

To integrate information, data in different formats, from dif- ferent, potentially overlapping sources, must be related and transformed to meet the users' needs. Ten years ago, Clio introduced nonprocedural schema mappings to describe the relationship between data in heteroge- neous schemas. This enabled powerful tools for mapping discovery and integration code generation, greatly simplifying the integration process. However, further progress is needed. We see an opportunity to raise the level of abstraction further, to encompass both data- and schema-centric integration tasks and to isolate applications from the details of how the integration is accomplished. Holistic information integration supports it- eration across the various integration tasks, leveraging information about both schema and data to improve the integrated result. Integration inde- pendence allows applications to be independent of how, when, and where information integration takes place, making materialization and the tim- ing of transformations an optimization decision that is transparent to applications. In this paper, we define these two important goals, and propose leveraging data mappings to create a framework that supports both data- and schema-level integration tasks.

In this special issue on metadata management, we present a new work on creating, gathering, managing, and understanding metadata. The work in this issue highlights the reality that the lack of metadata and effective techniques for managing them is currently one of the biggest challenges to meaningful use and sharing of the wealth (or should we say glut) of data available today. Our first paper, “Model-Independent SchemaTranslation” by Paolo Atzeni, Paolo Cappellari, Riccardo Torlone, Philip A. Bernstein, and Giorgio Gianforme, considers the schema translation problem—a problem as old as the data management field itself.Model IndependentData andSchemaTranslation (MIDST) is a framework for translating schemas from one data model to another. The explicit creation of metadata representing the relationship between the schema and its translation is central to the approach. The schema mapping is represented in a variant of Datalog with Skolem functions. Such mappings are in the spirit of source-to-target tuplegenerating-dependencies (commonly used in data integration and data exchange) but with special constructs for reasoning about data model specific concepts. These mappings play a crucial role in permitting the customization of the schema translation rules for different applications or domains. They also provide the mechanism through which MIDST can be extended to new data models. Our second paper, “PicShark: Mitigating Metadata Scarcity through Large-Scale P2P Collaboration” by Philippe

In authorization-transparent access control, user queries are formulated against the database schema rather than against authorization views that transform and hide data. The Truman and the Non-Truman are two approaches to authorization transparency where in a Truman model, queries that violate the access restrictions are modified transparently by the system to only reveal accessible data, while in a Non-Truman model, such queries are rejected. The advantage of a Non-Truman model is that the semantics of user queries is not changed by the access-control mechanism. This work presents an access-control mechanism for XML, under the Non-Truman model. Security policies are specified as parameterized rules formulated using XPath. The rules specify relationships between elements, that should be concealed from users. Hence, not only elements, but also edges and paths within an XML document, can be concealed. The access-control mechanism authorizes only valid queries, i.e., queries that do not disclose the existence of concealed relationships. The additional expressive power, provided by these rules, over element-based authorization techniques is illustrated. The proposed access-control mechanism can either serve as a substitute for views or as a layer for verifying that specific relationships are concealed by a view.

Considers the use of wavelet transformations as a dimensionality reduction technique to permit efficient similarity searching over high-dimensional time-series data. While numerous transformations have been proposed and studied, the only wavelet that has been shown to be effective for this application is the Haar wavelet. In this work, we observe that a large class of wavelet transformations (not only orthonormal wavelets but also bi-orthonormal wavelets) can be used to support similarity searching. This class includes the most popular and most effective wavelets being used in image compression. We present a detailed performance study of the effects of using different wavelets on the performance of similarity searching for time-series data. We include several wavelets that outperform both the Haar wavelet and the best-known non-wavelet transformations for this application. To ensure our results are usable by an application engineer, we also show how to configure an indexing strategy for the best-performing transformations. Finally, we identify classes of data that can be indexed efficiently using these wavelet transformations.

We demonstrate how the data management techniques known as On-Line Analytical Processing, or OLAP, can be used to enhance the sophistication and range of software reverse engineering tools. This is the first comprehensive examination of the similarities and differences in these tasks both in how OLAP techniques meet (or fail to meet) the needs of reverse engineering and in how reverse engineering can be recast using data analysis. To permit the seamless integration of these technologies, we extend a multidimensional data model to manage dynamically changing dimensions (over which data can be aggregated). We use a case study of the Apache Web server to show how our solutions permit an integrated view of data, ranging from low level program analysis information to abstract, aggregate information. These high-level abstractions may be provided either by humans (perhaps using a visualization tool) or directly from reverse engineering tools or data mining techniques.

In Toronto's Kanata project, we are investigating the integration and exchange of data and metadata in dynamic, autonomous environments. Our focus is on the development and maintenance of semantic mappings that permit runtime sharing of information.