Search

With the high development of Internet, e-commerce websites now routinely have to work with log datasets which are up to a few terabytes in size. How to remove messy data timely with low cost and find out useful information is a problem we have to face. The mining process involves several steps from pre-processing the raw data to establishing the final models. In this paper we describe our method to solve the problem with Map-Reduce. Hadoop[7] is a Map-Reduce implementation develops open-source software for reliable, scalable, distributed computing. Several applications which we have proposed: data extracting, sum operation, join operation and clustering algorithm are applied on hadoop. We can apply them on data pre-processing and detect users with the same interests. In particular, we focus on clustering algorithms. A clustering algorithms which integrate SOM(Self-Organized Map) and fuzzy[13] logic is combined with Map-Reduce and we call it MRSF here. With the help of hadoop cluster, large calculation of jobs with MRSF can be accommodated easily by just adding more nodes or computers to the cluster. From the experiment, we show that MRSF can scale well and efficiently process and analyze extremely large datasets.

In the traditional K-means algorithm, the selection of cluster number and the initial cluster center brings huge affection on the quality of clustering. To reduce the dependence on the initial center and to locate the types of new data rapidly, an algorithm applicable for text data is proposed. In this algorithm, document density is considered as parameter. Documents are divided into blocks first. After that every divided block is clustered separately. Experiment shows that this algorithm not only makes higher quality for clustering, but also does well in the new increasing data.