1. Statistical Methodology for Comparison of SAT Solvers
- Author
-
Mladen Nikolić
- Subjects
Theoretical computer science ,Computer science ,Random seed ,02 engineering and technology ,01 natural sciences ,Running time ,Task (project management) ,Set (abstract data type) ,010104 statistics & probability ,Point-biserial correlation coefficient ,0202 electrical engineering, electronic engineering, information engineering ,020201 artificial intelligence & image processing ,Point (geometry) ,0101 mathematics ,Algorithm - Abstract
Evaluating improvements to modern SAT solvers and comparison of two arbitrary solvers is a challenging and important task. Relative performance of two solvers is usually assessed by running them on a set of SAT instances and comparing the number of solved instances and their running time in a straightforward manner. In this paper we point to shortcomings of this approach and advocate more reliable, statistically founded methodologies that could discriminate better between good and bad ideas. We present one such methodology and illustrate its application.
- Published
- 2018
- Full Text
- View/download PDF