A Hybrid Data Cleaning Framework Using Markov Logic Networks.

With the increase of dirty data, data cleaning turns into a crux of data analysis. The accuracy limitation of the existing integrity constraints-based cleaning approaches results from insufficient rules. In this paper, we present a novel hybrid data cleaning framework on top of Markov logic networks (MLNs), termed as ${\sf MLNClean}$ MLNClean , which is capable of learning instantiated rules to supplement the insufficient integrity constraints. ${\sf MLNClean}$ MLNClean consists of two steps, i.e., pre-processing and two-stage data cleaning. In the pre-processing step, ${\sf MLNClean}$ MLNClean first infers a set of probable instantiated rules according to MLNs and then builds a two-layer MLN index structure to generate multiple data versions and facilitate the cleaning process. In the two-stage data cleaning step, ${\sf MLNClean}$ MLNClean first presents a concept of reliability score to clean errors within each data version separately, and afterward eliminates the conflict values among different data version using a novel concept of fusion score. Considerable experimental results on both real and synthetic scenarios demonstrate the effectiveness of ${\sf MLNClean}$ MLNClean in practice. [ABSTRACT FROM AUTHOR]