Reliability Analysis of PISA 2018 Reading Literacy Student Questionnaire Based on Item Response Theory (IRT): Turkey Sample

PISA (Program for International Student Assessment) tests have enabled the OECD countries to see not only the success of their students in gaining the ability to solve some daily problems they may encounter in their lives but also the place in the world rankings as a result of an objective evaluation comparing the achievement results of participant countries. Therefore, it is very important for the reliability and prestige of the program that the PISA exams and related student survey results are as unbiassed and error-free as possible. In this paper, validity and reliability study of the PISA 2018 Reading Literacy Student Survey (RLSS) Turkey sample was conducted including the analysis of survey items within the framework of Item Response Theory (IRT). The item fit indexes, item parameters according to the GPCM (Generalized Partial Credit Model), standard error values and item characteristics' reliability of the survey were determined via IRT, respectively. The data set included the answers of 6111 15-year-old Turkish students participated in PISA 2018. In the data analysis, through using the local independence assumption, Q3 statistics; IRT calibrations were tested with the help of the "Mirt v.1.30" program within the scope of "R v.4.0.5". In the study, each set of questions in the PISA student survey was examined independently from each other and each question set was considered as a separate attitude scale. The results showed that, although some of the items in PISA 2018 (RLSS) gave low level information, all the question sets in the test provided an acceptable model fit according to the GPCM. Upon examining the item characteristic curves, it was understood that the survey items showed valid and reliable results for testing different ability levels.