Can Likert scales predict choices? Testing the congruence between using Likert scale and comparative judgment on measuring attribution

How people make choices among alternatives are of interest in different areas of psychological research. One paradigm to answer this question is by applying Likert Scale (LS) to compare the agreements to different alternatives, and the respective LS scores are then transferred into rank order of preference. However, using LS to infer choices is somewhat debatable because the measurement format of LS was not designed for revealing psychological preference. In this article, we examined to what extent it is appropriate to use quantitative indicators derived from LS to infer choices, with which we used the Comparative Judgment (CJ) procedure to represent a direct measurement of choice decision to compare. A total of 929 adolescents reported their effort and ability attributions for academic failure and success using both LS and CJ. We found that while using LS is generally accurate in predicting results obtained via CJ, the percentage of people revealing different choices inferred from LS versus CJ was 14.7% and 12.1% for the success and failure scenarios, respectively, suggesting that inferring psychological preference from LS is not without risk, at least for this sample of adolescents from a culturally-Chinese society. Furthermore, the majority of participants displaying incongruent decisions of achievement attribution via LS and CJ showed equivalent LS scores between effort and ability attributions. A goodness-of-fit test was conducted (on a model motivated by the beta-binomial distribution) and successfully eliminated the possibility that the tied LS scores in effort and ability attributions actually represent participants’ true psychological state.