1. An efficient attribute reduction algorithm using MapReduce
- Author
-
Ding Jiafeng, Zhaohui Jiang, Xuemei Xu, Li Jing, and Linzi Yin
- Subjects
Reduction (complexity) ,Computer science ,05 social sciences ,0202 electrical engineering, electronic engineering, information engineering ,020201 artificial intelligence & image processing ,02 engineering and technology ,Rough set ,0509 other social sciences ,Library and Information Sciences ,050904 information & library sciences ,Algorithm ,Information Systems - Abstract
Classical attribute reduction algorithms based on attribute significance initiate too many jobs ( O(| C|2)) when they run in MapReduce. To improve the efficiencies of these algorithms, we proposed a novel reduction algorithm. Instead of focusing on attribute significance, the notion of a core attribute was applied to construct a new heuristic reduction algorithm, and only | C| jobs were considered to obtain a reduct. The algorithm only included two basic operations: compare and sort. The latter was optimised using the shuffle mechanism in MapReduce, which provided an efficient sorting ability for big data. In particular, we connected jobs in an iterative form to transfer the processing result of the former job to the latter job. Finally, experimental results demonstrated that the proposed attribute reduction algorithm was efficient and significantly improved upon the classical algorithms in runtime and number of jobs.
- Published
- 2019
- Full Text
- View/download PDF