Search

Clustering ensemble, also referred to as consensus clustering, has emerged as a method of combining an ensemble of different clusterings to derive a final clustering that is of better quality and r...

In this paper a novel approach to document clustering has been introduced by defining a representative-based document similarity model that performs probabilistic segmentation of documents into chunks. The frequently occuring chunks that are considered as representatives of the document set, may represent phrases or stem of true words. The representative based document similarity model, containing a term-document matrix with respect to the representatives, is a compact representation of the vector space model that improves quality of document clustering over traditional methods.

In this paper we propose a novel approach to document clustering by introducing a representative-based document similarity model that treats a document as an ordered sequence of words and partitions it into chunks for gaining valuable proximity information between words. Chunks are subsequences in a document that have low internal entropy and high boundary entropy. A chunk can be a phrase, a word or a part of word. We implement a linear time unsupervised algorithm that segments sequence of words into chunks. Chunks that occur frequently are considered as representatives of the document set. The representative based document similarity model, containing a term-document matrix with respect to the representatives, is a compact representation of the vector space model that improves quality of document clustering over traditional methods.