Search

AbstractWhile a rich methodology for analyzing response patterns for accuracy and time-on-task is at hand via Item Response Theory (IRT), tests with time cutoffs are so far harder to handle. Given that this test mode is widely applied, especially in the context of paper-and-pencil testing, there is a lack of psychometric techniques for a relevant number of tests. In this context, the original work of Rasch and his Rasch Poisson Counts model indeed offers an approach for this scenario that is adequate to solve the problem but which leads to model violations in many cases. Recent developments in statistical modeling – the so-called Conway Maxwell Poisson Counts Model (CMPCM) – can solve the problem of under- and overdispersion. We apply this model to the norm data of the ELFE II reading comprehension test and analyze patterns of over- and underdispersion with regard to speededness and mode effects. CMPCM with subtest-specific dispersion was adequate to model the raw test data, with underdispersion occurring mainly in highly speeded subtests with low difficulty and overdispersion in less speeded subtests with high difficulty. Thus, the CMPCM could contribute to psychometric methodology to appropriately model tests with time cutoffs on the subtest level. [ABSTRACT FROM AUTHOR]

Effective evaluation of data quality between data collected in different modes is complicated by the confounding of selection and measurement effects. This study evaluates the utility of propensity score matching (PSM) as a method that has been proposed as a means of removing selection effects across surveys conducted in different modes. Our results show large differences in estimates for the same variables between parallel face-to-face and online surveys, even after matching on standard demographic variables. Moreover, discrepancies in estimates are still present after matching between surveys conducted in the same (online) mode, where differences in measurement properties can be ruled out a priori. Our findings suggest that PSM has substantial limitations as a method for separating measurement and selection differences across modes and should be used only with caution. [ABSTRACT FROM AUTHOR]

Reinterview designs are a potential tool to estimate and adjust for mode measurement effects, that is, relative differences in mode-specific measurement error bias. In 2011, a reinterview design was successfully applied to the Dutch Crime Victimization Survey, which led to a redesign of the survey. Reinterview designs may, however, be very costly, especially when face to face is included as a survey mode. The crucial question is whether benefits outweigh costs, that is, whether the potential increase in the accuracy of survey statistics is worth the investment. The answer to this question depends heavily on the purpose of the reinterview, that is, assessment versus adjustment, the size of the measurement effects, and the relative cost of the modes. Reinterview designs also make a number of assumptions that will not hold for every setting. In this article, we perform a cost–benefit analysis for two surveys, the Dutch Health Survey and the Dutch Labour Force Survey, and discuss the utility and validity of reinterviews. We conclude that a reinterview may not be useful due to relatively small measurement differences for the Labour Force Survey, whereas it may be useful for the Health Survey. [ABSTRACT FROM AUTHOR]

OBJECTIVES: Telephone-administered cognitive assessments are a cost-effective and sometimes necessary alternative to face-to-face assessments. There is limited information in large studies concerning mode effects, or differences in cognition attributable to the assessment method, as a potential measurement threat. We evaluated mode effects on cognitive scores using a population-based sample of community-living older adults. METHODS: We used data from participants aged 65-79 in the 2014 Health and Retirement Study for whom the interview mode was randomized (n = 6,825). We assessed mode differences in test means, whether mode modifies associations of cognition with criterion variables, and formal measurement invariance testing. RESULTS: Relative to face-to-face assessment, telephone assessment was associated with higher scores for memory and calculation (0.06 to 0.013 standard deviations [SD]) and lower scores for nonmemory items (-0.09 to -0.01 SD). Cognition was significantly differentially related to instrumental activities of daily living difficulty depending on assessment mode. Measurement invariance testing identified evidence of mode differences in certain tests as a function of mode: adjusting for underlying cognition, the largest mode differences in memory and attention: immediate noun recall, delayed word recall, and serial-7s scores were higher given telephone administration. DISCUSSION: Differences by mode of administration are apparent in cognitive measurement in older adults, albeit to a small degree in our study, and most pronounced for tests of memory and attention. The importance of accounting for mode differences ultimately depends on ones research question and study sample: not all associations may be affected by mode differences, and such modification may only be apparent among those with lower cognitive functioning.

Objectives : Current knowledge about the causes of offending behavior is heavily reliant on self-reports of offending (SRO). However, methodological research on the impact of modes of administration on SRO is very scarce. Further, the existing evidence conflicts with the general knowledge about responding to sensitive questions. In this study, we aimed to test whether SRO are affected by modes of administration. Methods: We carried out a methodological experiment, with a 2 (interviewer-administered vs. self-administered surveys) × 2 (paper-and-pencil vs. computer- assisted surveys) factorial design. A total of 181 undergraduate students were randomly assigned to one of these conditions and completed the International Self-Report Delinquency 3 (ISRD3) questionnaire. Results: Findings showed an increased odds of reporting offending behavior in self-administered surveys, compared to face-to-face interviews. Paper-and-pencil and computer-assisted modes resulted in comparable estimates of offending. Conclusions: This experiment provides evidence that SRO provide more accurate estimates of offending behavior using self-administered surveys. [ABSTRACT FROM AUTHOR]

Copyright of Österreichische Zeitschrift für Soziologie is the property of Springer Nature and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Background: Surveys are a prominent tool in gerontological research but are hindered by low response rates. This study examined methods to improve response rate among social workers, an understudied cohort in survey methodology literature. Methods: In this brief report, we analyzed factors that influence response rate using mixed-mode survey design (mail vs. online response) and randomized monetary ($10 gift card) and in-kind (continuing education coupon) noncontingent incentives with three reminder postcards among social workers (N = 307). Results: Response rate was 34.1% with more surveys returned via mail than online. Participants who received a gift card responded at a higher rate (46.33%) than those who received continuing education coupons (28.0%). The results add to the literature of survey methodology to improve response rates among social workers and support noncontingent gift cards and mixed-mode survey return procedures as best practices for reducing bias and improving the quality of gerontological research. [ABSTRACT FROM AUTHOR]

Longitudinal surveys are a key data collection tool used to estimate social change. Recent developments have accelerated the move from traditional single-mode longitudinal designs to mixed-mode designs. Nevertheless, there are concerns that mixing survey modes may affect coefficients of change at the individual level. We investigate the impact of mixing survey modes on estimates of change using a quasi-experimental design implemented in a long-running UK panel study. Two types of comparisons are carried out: single-mode (face-to-face) design versus sequential mixed-mode (Web–face-to-face) design, and Web versus face to face. Across 41 variables, we find no differences in estimates of individual-level change across modes (designs). However, correlations between intercepts and slopes, an estimate of convergence of respondents, were significantly different for most variables, which led to some biases in estimates of change. Applied researchers are encouraged to do sensitivity checks to ensure their results are robust to mode effects. [ABSTRACT FROM AUTHOR]

Abstract Background Mode effects, the variations in item and scale properties attributed to the mode of test administration (paper vs. computer), have stimulated research around test equivalence and trend estimation in PISA. The PISA assessment framework provides the backbone to the interpretation of the results of the PISA test scores. However, an identified gap in the current literature is whether mode effects have affected test score interpretation as defined by the assessment framework, and whether the interpretations of the PBA and CBA test scores are comparable. Methods This study uses the 2015 PISA field trial data from thirteen countries to compare test modes through a construct representation approach. It is investigated whether item facets defined by the assessment framework (e.g., different cognitive demands) affect item difficulty comparably across modes using a unidimensional two-group generalized partial credit model (GPCM). Results Linking the assessment framework to item difficulty using linear regression showed that for both maths and science domains, item categorisation relates to item difficulty, however for the reading domain no such conclusion was possible. In comparing PBA to CBA in representations across the three domains, maths had one facet with a significant difference in representation, reading had all three facets significantly different, and for science, four out of six facets had significant differences. Modelling items labelled “mode invariant” in PISA 2015, the results indicated that in every domain, two facets showed significant differences between the test modes. The graphical inspection of difficulty patterns confirmed that reading shows stronger differences while the patterns of the other domains were quite consistent between modes. Conclusions The present study shows that the mode effects on difficulty vary within the task facets proposed by the PISA assessment framework, in particular for reading. These findings shed light on whether the comparability of score interpretation between modes is compromised. Given the limitations of the link between the reading domain and item difficulty, any conclusions in this domain are limited. Importantly, the present study adds a new approach and empirical findings to the investigation of the cross-mode equivalence in PISA domains.

The outbreak of the COVID-19 pandemic has transformed the educational landscape in a way unseen before. Educational institutions are navigating between offline and online learning worldwide. Computer-based testing is rapidly taking over paper-and-pencil testing as the dominant mode of assessment. In some settings, computer-based and paper-and-pencil assessments can also be offered side-by-side, in which case test developers should ensure the evidence of equivalence between both versions. This study aims to establish the equivalency evidence of different delivery modes of the English Competency Test, an English language assessment for civil service officers developed and used by the Human Resources Development Education and Training Center, a civil service training institution under the Ministry of Finance of the Republic of Indonesia. Psychometric analyses were carried out with the Rasch model to measure the unidimensionality, reliability, separation, and standard error of measurement estimates. The findings demonstrate that the paper-and-pencil and computer-based versions of the language assessment exhibit comparatively equivalent psychometric properties. The computer-based version of the English Competency Test is proven to offer a reliable and comparable alternative to the paper-and-pencil version.

Constant changes to COVID-19 restrictions have required adaptability from social scientists including responding to new challenges such as infiltration by bots. This research note presents unexpected encounters of bot infiltration and recruitment during survey data collection under pandemic conditions. The note draws from a household survey on a social housing estate in London, UK conducted in 2021. The survey investigates residents’ lived experiences of the estate and housing turnover. The note discusses the limitations of online data collection, focusing on infiltration by bots and exclusion of marginalised groups. It adds to the emerging literature on bots in survey methods, making recommendations for an iterative verification and sequential multi-stage data cleaning process. It finds that online-only approaches can exclude marginalised groups. The note argues that even under pandemic conditions, face-to-face data collection can have greater reach than online only approaches. It concludes that mixed-mode household surveys can a) mitigate the challenges of a changing research environment; b) reach a broader sample; and c) provide qualitative insight for future research. [ABSTRACT FROM AUTHOR]

The outbreak of the COVID-19 pandemic has transformed the educational landscape in a way unseen before. Educational institutions are navigating between offline and online learning worldwide. Computer-based testing is rapidly taking over paper-and-pencil testing as the dominant mode of assessment. In some settings, computer-based and paper-and-pencil assessments can also be offered side-by-side, in which case test developers should ensure the evidence of equivalence between both versions. This study aims to establish the equivalency evidence of different delivery modes of the English Competency Test, an English language assessment for civil service officers developed and used by the Human Resources Development Education and Training Center, a civil service training institution under the Ministry of Finance of the Republic of Indonesia. Psychometric analyses were carried out with the Rasch model to measure the unidimensionality, reliability, separation, and standard error of measurement estimates. The findings demonstrate that the paper-and-pencil and computer-based versions of the language assessment exhibit comparatively equivalent psychometric properties. The computer-based version of the English Competency Test is proven to offer a reliable and comparable alternative to the paper-and-pencil version. [ABSTRACT FROM AUTHOR]

Internationally, teachers use oral reading fluency (ORF) measurements to monitor learning progress in reading and adapt instruction to the individual needs of students. In ORF measures, the child reads aloud single syllables, words, or short passages, and the teacher rates in parallel at which items the child makes a mistake. Since administering paper-based ORF requires increased effort on the part of teachers, computer-based test administration is available. However, there are still concerns about the comparability of paper-based and computer-based test modes. In our study, we examine mode effects between paper-based and computer-based test scores for both reading speed and reading accuracy using a German-language ORF assessment for progress monitoring. 2nd- and 3rd-year-students (N = 359) with and without special education needs participated in the study. Results show comparable and high reliability (r > 0.76) and no differential item functioning for both test modes. Furthermore, students showed significantly higher reading speed on the paper-based test, while no differences were found in reading accuracy. In the absence of differential item functioning, we discuss how mean differences can be accounted for, how teachers can be trained to use the different test modes, and how computer-based tests can be safeguarded in practice. [ABSTRACT FROM AUTHOR]

Responses to phone surveys tend to exhibit higher rates of social desirability bias and extreme responses when compared to face-to-face surveys. Yet, studies of mode effects typically compare either representative studies that implausibly assume comparability or experimental studies that rely on convenience samples. Our study compares two national probability samples but uses matching to address comparability. We study Costa Rica, a middle-income democracy, to see whether the conventional wisdom drawn from Western Europe and North America extends to the Global South. We analyze two nationally representative surveys, one fielded by phone and one face-to-face, allowing us to compare identically worded items we placed on both surveys. We find that phone respondents exhibited more socially desirable responding and were more likely to choose negative endpoints on scalar items. This suggests that survey researchers and practitioners should carefully assess the tradeoffs in shifting modes or employing mixed modes.

Abstract Numerous panel surveys around the world use multiple modes of data collection to recruit and interview respondents. Previous studies have shown that mixed-mode data collection can improve response rates, reduce nonresponse bias, and reduce survey costs. However, these advantages come at the expense of potential measurement differences between modes. A major challenge in survey research is disentangling measurement error biases from nonresponse biases in order to study how mixing modes affects the development of both error sources over time. In this article, we use linked administrative data to disentangle both nonresponse and measurement error biases in the long-running mixed-mode economic panel study “Labour Market and Social Security” (PASS). Through this study design we answer the question of whether mixing modes reduces nonresponse and measurement error biases compared to a single-mode design. In short, we find that mixing modes reduces nonresponse bias for most variables, particularly in later waves, with only small effects on measurement error bias. The total bias and mean-squared error are both reduced under the mixed-mode design compared to the counterfactual single-mode design, which is a reassuring finding for mixed-mode economic panel surveys.

Background: Mode effects, the variations in item and scale properties attributed to the mode of test administration (paper vs. computer), have stimulated research around test equivalence and trend estimation in PISA. The PISA assessment framework provides the backbone to the interpretation of the results of the PISA test scores. However, an identified gap in the current literature is whether mode effects have affected test score interpretation as defined by the assessment framework, and whether the interpretations of the PBA and CBA test scores are comparable. Methods: This study uses the 2015 PISA field trial data from thirteen countries to compare test modes through a construct representation approach. It is investigated whether item facets defined by the assessment framework (e.g., different cognitive demands) affect item difficulty comparably across modes using a unidimensional two-group generalized partial credit model (GPCM). Results: Linking the assessment framework to item difficulty using linear regression showed that for both maths and science domains, item categorisation relates to item difficulty, however for the reading domain no such conclusion was possible. In comparing PBA to CBA in representations across the three domains, maths had one facet with a significant difference in representation, reading had all three facets significantly different, and for science, four out of six facets had significant differences. Modelling items labelled "mode invariant" in PISA 2015, the results indicated that in every domain, two facets showed significant differences between the test modes. The graphical inspection of difficulty patterns confirmed that reading shows stronger differences while the patterns of the other domains were quite consistent between modes. Conclusions: The present study shows that the mode effects on difficulty vary within the task facets proposed by the PISA assessment framework, in particular for reading. These findings shed light on whether the comparability of score interpretation between modes is compromised. Given the limitations of the link between the reading domain and item difficulty, any conclusions in this domain are limited. Importantly, the present study adds a new approach and empirical findings to the investigation of the cross-mode equivalence in PISA domains. [ABSTRACT FROM AUTHOR]

Numerous panel surveys around the world use multiple modes of data collection to recruit and interview respondents. Previous studies have shown that mixed-mode data collection can improve response rates, reduce nonresponse bias, and reduce survey costs. However, these advantages come at the expense of potential measurement differences between modes. A major challenge in survey research is disentangling measurement error biases from nonresponse biases in order to study how mixing modes affects the development of both error sources over time. In this article, we use linked administrative data to disentangle both nonresponse and measurement error biases in the long-running mixed-mode economic panel study "Labour Market and Social Security" (PASS). Through this study design we answer the question of whether mixing modes reduces nonresponse and measurement error biases compared to a single-mode design. In short, we find that mixing modes reduces nonresponse bias for most variables, particularly in later waves, with only small effects on measurement error bias. The total bias and mean-squared error are both reduced under the mixed-mode design compared to the counterfactual single-mode design, which is a reassuring finding for mixed-mode economic panel surveys. [ABSTRACT FROM AUTHOR]

Copyright of Journal for Educational Research Online / Journal für Bildungsforschung Online is the property of Waxmann Verlag GmbH and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Official monthly statistics about the Dutch labour force are based on the Dutch Labour Force Survey (LFS). The LFS is a continuously conducted survey that is designed as a rotating panel design. Data collection among selected households is based on a mixed‐mode design that uses web interviewing, telephone interviewing and face‐to‐face interviewing. Monthly estimates about the labour force are obtained with a structural time series model. Due to the COVID‐19 pandemic, face‐to‐face interviewing stopped. It was anticipated that this would have a systematic effect on the outcomes of the LFS and that the lockdown at the same time affected the real monthly labour force figures. The lockdown indeed marked a sharp turning point in the evolution of the series of the monthly labour force figures and strongly increased the volatility of these series. In this paper, it is explained how Statistics Netherlands produced monthly labour force figures during the COVID‐19 pandemic. It is shown how the sudden change in the mode effects, because face‐to‐face interviewing stopped, were separated from real period‐to‐period changes in the labour force figures. It is also explained how the time series model is adapted to the increased volatility in the labour force figures. [ABSTRACT FROM AUTHOR]

Internationally, teachers use oral reading fluency (ORF) measurements to monitor learning progress in reading and adapt instruction to the individual needs of students. In ORF measures, the child reads aloud single syllables, words, or short passages, and the teacher rates in parallel at which items the child makes a mistake. Since administering paper-based ORF requires increased effort on the part of teachers, computer-based test administration is available. However, there are still concerns about the comparability of paper-based and computer-based test modes. In our study, we examine mode effects between paper-based and computer-based test scores for both reading speed and reading accuracy using a German-language ORF assessment for progress monitoring. 2nd- and 3rd-year-students (N = 359) with and without special education needs participated in the study. Results show comparable and high reliability (r > 0.76) and no differential item functioning for both test modes. Furthermore, students showed significantly higher reading speed on the paper-based test, while no differences were found in reading accuracy. In the absence of differential item functioning, we discuss how mean differences can be accounted for, how teachers can be trained to use the different test modes, and how computer-based tests can be safeguarded in practice.

A sequential mixed‐mode data collection, online‐to‐telephone, was introduced into the National Child Development Study for the first time at the study's age 55 sweep in 2013. The study included a small experiment, whereby a randomised subset of study members was allocated to a single mode, telephone‐only interview, in order to test for the presence of mode effects on participation and measurement. Relative to telephone‐only, the offer of the Web increased overall participation rates by 5.0 percentage points (82.8% vs. 77.8%; 95% confidence interval for difference: 2.7% to 7.3%). Differences attributable to mode of interview were detected in levels of item non‐response and response values for a limited number of questions. Most notably, response by Web (relative to telephone) was found to have increased the likelihood of non‐response to questions relating to pay and other financial matters, and increased the likelihood of 'less desirable' responses. For example, response by Web resulted in the reporting of more units of alcohol consumed, and more negative responses to subjective questions such as self‐rated health, self‐rated financial status and well‐being. As there was evidence of mode effects, there is the potential for biases in some analyses, unless appropriate techniques are utilised to correct for these. [ABSTRACT FROM AUTHOR]

Over the last two decades, in‐person interviewing costs continued to increase while the data quality advantages traditionally identified with this data collection mode have faded. Consequently, some longitudinal surveys have begun transitioning from in‐person to web data collection despite risks to data quality and longitudinal comparability. This paper addresses the major issues involved in the transition process and proposes a multi‐sample, multi‐phase responsive design that attempts to minimize the data quality risks while preserving the considerable cost savings promised by the transition. The paper describes the design as it was applied to the National Longitudinal Study of Adolescent to Adult Health (Add Health)—a nationally representative panel survey of around 20,000 adolescents selected from grades 7 to 12 (typically 13 to 18 years of age) in the 1994–95 school year. Also described are key results from several experiments embedded within the design and the analysis of mode effects. Also presented are some lessons learned and recommendations for other in‐person panel surveys that may be contemplating a similar transition to web or mixed‐mode data collection. [ABSTRACT FROM AUTHOR]

The presence of an interviewer (face-to-face or via telephone) is hypothesized to motivate respondents to generate an accurate answer and reduce task difficulty, but also to reduce the privacy of the reporting situation. To study this, we used respondents from an existing face-to-face probability sample of the general population who were randomly assigned to face-to-face, telephone and web modes of data collection. The prevalence of indicators of satisficing (e.g., non-differentiation, acquiescence, middle category choices and primacy and recency effects) and socially desirable responding were studied across modes. Results show differences between interviewer modes and web in levels of satisficing (non-differentiation, acquiescence and middle category choices) and in socially desirable responding. There were also unexpected findings of (1) different ways of satisficing by mode and (2) of a telephone mode primacy/positivity effects.

Many surveys collect data using a mixture of modes administered in sequential order. Although the impacts of mixed-mode designs on measurement quality have been extensively studied, their impacts on the measurement quality of unobservable (or latent) constructs is still an understudied area of research. In particular, it is unclear whether latent constructs derived from multi-item scales are measured equivalently across different sequentially-administered modes – an assumption that is often made by analysts, but rarely tested in practice. In this study, we assess the measurement equivalence of several commonly-used multi-item scales collected in a sequential mixed-mode (Web-telephone-face-to-face) survey: the Age 25 wave of the Next Steps cohort study. After controlling for selection via an extensive data-driven weighting procedure, a multi-group confirmatory factor analysis was performed to assess measurement equivalence across the three modes. We show that cross-mode measurement equivalence is achieved for the majority of scales, with partial equivalence established for the remaining scales. Although measurement equivalence was achieved, some differences in the latent means were observed between the modes. We conclude with a discussion of these findings, their potential causes, and implications for survey practice.

In today's digital age, tests are increasingly being delivered on computers. Many of these computerbased tests (CBTs) have been adapted from paper-based tests (PBTs). However, this change in mode of test administration has the potential to introduce construct-irrelevant variance, affecting the validity of score interpretations. Because of this, when scores from a CBT are to be interpreted in the same way as a PBT, evidence is needed to support the reliability and validity these scores (AERA et al. 2014). Numerous studies have investigated the impact of changing the mode of test delivery from paper to computer, not only in terms of their psychometric properties, but also with regard to possible sources of construct-irrelevant variance. This article summarizes the main lessons learned from mode effects studies in education over the past 30 years and discusses some of the questions remaining. [ABSTRACT FROM AUTHOR]

Measuring change over time is one of the main purposes of longitudinal surveys. With an increase in the use of web as a mode of data collection it is important to assess whether the web mode differs from other modes with respect to the number of changes and events that are captured. We examine whether telephone and web data collection modes are comparable with respect to measuring changes over time or experiencing events. Using experimental data from a two-wave pilot of the Swiss Household Panel, we investigate this question for several variables in the domain of work and family. We find differences for the work-related variables, with web respondents more likely to report changes. These differences do not disappear once the socio-demographic composition of the sample is taken into consideration. This suggests that these differences are not driven by observed different characteristics of the respondents who may have self-selected into one or the other mode. Contrary to work-related variables, a termination of a relationship was more common in the telephone group. This shows that one mode does not necessarily measure more change or events than another; it may depend on the variable in question. In addition, the difference in the protocol mattered: a web respondent in a household that participated fully by web sometimes differed from a web respondent in a household that had a household interview by phone. Nonetheless, the telephone group differed more from the various web protocols that the web protocols among themselves. With more household panel surveys introducing web questionnaires in combination with more traditional face-to-face and telephone interviews, this study adds to our understanding of the potential consequences of mixing modes with respect to longitudinal data analysis. [ABSTRACT FROM AUTHOR]

Previous research is inconclusive regarding the effects of paper and web surveys on response burdens. We conducted an establishment survey with random assignment to paper and web modes to examine this issue. We compare how the actual and perceived response burdens differ when respondents complete a survey in the paper mode, in the web mode and when they are allowed to choose between the two modes. Our results show that in the web mode, respondents have a lower estimated time to complete the questionnaire, while we do not find differences between paper and the web on the perceived response time and perceived burden. Even though the response burden in the web mode is lower, our study finds no evidence of an increased response burden when moving an establishment survey from paper to the web. [ABSTRACT FROM AUTHOR]

Conducting qualitative interviews in-person is usually presented as the gold standard, with other modes being seen as inferior. There have been arguments, however, that remote interviews, such as those conducted using the telephone or videoconference technologies, should be seen as equivalent to or even superior to in-person interviews. Evaluations of these claims have been limited by the small number of interviews used to compare modes. We analyze over 300 interviews conducted using three modes: in-person, telephone, and Skype. Our analyses find that in-person interviews have clear advantages when it comes to producing conversation turns and word-dense transcripts and field notes but do not significantly differ from the other two modes in interview length in minutes, subjective interviewer ratings, and substantive coding. We conclude that, although remote interviews might be necessary or advantageous in some situations, they likely do often come at a cost to the richness of information produced by the interviews. [ABSTRACT FROM AUTHOR]

Objectives: We investigate the impact of data collection mode on responses to variables in NSHAP Round 4 and discuss how potential mode differences should (and should not) be addressed in substantive analyses., Methods: Among the set of respondents who were eligible to be contacted remotely in Round 4, we randomly selected 398 to be contacted instead for an in-person interview. We compare response rates and the distributions of selected key outcomes among those 398 respondents to those among the control group who were initially approached remotely. As a contrast, we compare all R4 respondents according to the mode in which they completed the interview, including those not part of the randomized experiment., Results: Among those included in the experiment, there was no evidence of systematic differences in responses to physical and mental health questions between remote and in-person modes, nor in responses to number recall measures. In-person respondents scored moderately lower on cognitive function measures requiring careful attention to a figure and/or task, though this difference became less with each similar item. Remote respondents named fewer social network members. Comparing all respondents according to their final mode yielded substantially different results in all cases., Discussion: Mode did not appear to affect reports of physical and mental health based on a randomized comparison, though it did moderately affect other items in predictable ways. Naïve estimates of mode effects based on comparing all respondents according to mode yielded misleading results, and should not be used to adjust for mode differences in analyses., (© The Author(s) 2024. Published by Oxford University Press on behalf of The Gerontological Society of America. All rights reserved. For commercial re-use, please contact reprints@oup.com for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site—for further information please contact journals.permissions@oup.com.)

The "transportability" of laboratory findings to other instances than the original implementation entails the robustness of rates of observed behaviors and estimated treatment effects to changes in the specific research setting and in the sample under study. In four studies based on incentivized games of fairness, trust, and reciprocity, we evaluate (1) the sensitivity of laboratory results to locally recruited student-subject pools, (2) the comparability of behavioral data collected online and, under varying anonymity conditions, in the laboratory, (3) the generalizability of student-based results to the broader population, and (4) with a replication at Amazon Mechanical Turk, the stability of laboratory results across research contexts. For the class of laboratory designs using incentivized games as measurement instruments of prosocial behavior, we find that rates of behavior and the exact behavioral differences between decision situations do not transport beyond specific implementations. Most clearly, data obtained from standard participant pools differ significantly from those from the broader population. This undermines the use of empirically motivated laboratory studies to establish descriptive parameters of human behavior. Directions of the behavioral differences between games, in contrast, are remarkably robust to changes in samples and settings. Moreover, we find no evidence for either anonymity effects nor mode effects potentially biasing laboratory measurement. These results underscore the capacity of laboratory experiments to establish generalizable causal effects in theory-driven designs. [ABSTRACT FROM AUTHOR]

Copyright of Boğaziçi Journal: Review of Social, Economic & Administrative Studies is the property of Bogazici University, Faculty of Economics & Administrative Sciences and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Web questionnaires are increasingly used to complement traditional data collection in mixed mode surveys. However, the utilization of web data raises concerns whether web questionnaires lead to mode-specific measurement bias. We argue that the magnitude of measurement bias strongly depends on the content of a variable. Based on the Luxembourgish Labour Force Survey, we investigate differences between web and telephone data in terms of objective (i.e., Employment Status) and subjective (i.e., Wage Adequacy and Job Satisfaction) variables. To assess whether differences in outcome variables are caused by sample composition or mode-specific measurement bias, we apply a coarsened exact matching that approximates randomized experiments by reducing dissimilarities between web and telephone samples. We select matching variables with a combination of automatic variable selection via random forest and a literature-driven selection. The results show that objective variables are not affected by mode-specific measurement bias, but web participants report lower satisfaction-levels on subjective variables than telephone participants. Extensive supplementary analyses confirm our results. The present study supports the view that the impact of survey mode depends on the content of a survey and its variables. [ABSTRACT FROM AUTHOR]

International large-scale assessments, such as the Program for International Student Assessment (PISA), are conducted to provide information on the effectiveness of education systems. In PISA, the target population of 15-year-old students is assessed every 3 years. Trends show whether competencies have changed in the countries between PISA cycles. In order to provide valid trend estimates, it is desirable to retain the same test conditions and statistical methods in all PISA cycles. In PISA 2015, however, the test mode changed from paper-based to computer-based tests, and the scaling method was changed. In this paper, we investigate the effects of these changes on trend estimation in PISA using German data from all PISA cycles (2000–2015). Our findings suggest that the change from paper-based to computer-based tests could have a severe impact on trend estimation but that the change of the scaling model did not substantially change the trend estimates.

Abstract Background TIMSS 2019 is the first assessment in the TIMSS transition to a computer-based assessment system, called eTIMSS. The TIMSS 2019 Item Equivalence Study was conducted in advance of the field test in 2017 to examine the potential for mode effects on the psychometric behavior of the TIMSS mathematics and science trend items induced by the change to computer-based administration. Methods The study employed a counterbalanced, within-subjects design to investigate the potential for eTIMSS mode effects. Sample sizes for analysis included 16,894 fourth grade students from 24 countries and 9,164 eighth grade students from 11 countries. Following a review of the differences of the trend items in paper and digital formats, item statistics were examined item by item and aggregated by subject for paperTIMSS and eTIMSS. Then, the TIMSS scaling methods were applied to produce achievement scale scores for each mode. These were used to estimate the expected magnitude of the mode effects on student achievement. Results The results of the study provide support that the mathematics and science constructs assessed by the trend items were mostly unaffected in the transition to eTIMSS at both grades. However, there was an overall mode effect, where items were more difficult for students in digital formats compared to paper. The effect was larger in mathematics than science. Conclusions Because the trend items cannot be expected to be sufficiently equivalent across paperTIMSS and eTIMSS, it was concluded that modifications must be made to the usual item calibration model for TIMSS 2019 to measure trends. Each eTIMSS 2019 trend country will administer paper trend booklets to a nationally representative sample of students, in addition to the usual student sample, to provide a bridge between paperTIMSS and eTIMSS results.

In light of issues such as increasing unit nonresponse in surveys, several studies argue that social media sources such as Twitter can be used as a viable alternative. However, there are also a number of shortcomings with Twitter data such as questions about its representativeness of the wider population and the inability to validate whose data you are collecting. A useful way forward could be to combine survey and Twitter data to supplement and improve both. To do so, consent within a survey is first needed. This study explores the consent decisions in three large representative surveys of the adult British population to link Twitter data to survey responses and the impact that demographics and survey mode have on these outcomes. Findings suggest that consent rates for data linkage are relatively low, and this is in part mediated by mode, where face-to-face surveys have higher consent rates than web versions. These findings are important to understand the potential for linking Twitter and survey data but also to the consent literature generally. [ABSTRACT FROM AUTHOR]

International large-scale assessments, such as the Program for International Student Assessment (PISA), are conducted to provide information on the effectiveness of education systems. In PISA, the target population of 15-year-old students is assessed every 3 years. Trends show whether competencies have changed in the countries between PISA cycles. In order to provide valid trend estimates, it is desirable to retain the same test conditions and statistical methods in all PISA cycles. In PISA 2015, however, the test mode changed from paper-based to computer-based tests, and the scaling method was changed. In this paper, we investigate the effects of these changes on trend estimation in PISA using German data from all PISA cycles (2000–2015). Our findings suggest that the change from paper-based to computer-based tests could have a severe impact on trend estimation but that the change of the scaling model did not substantially change the trend estimates. [ABSTRACT FROM AUTHOR]

Stated preferences methods are extensively applied in health economics to elicit preferences. Although mailed surveys were commonly used to collect data, internet panel (IP) surveys are being increasingly used. This raises questions about the validity of responses and estimated willingness-to-pay (WTP) values generated from IP surveys. We conduct the first study in health to compare a contingent valuation IP survey with a mailed survey using the electoral roll. Our IP has a higher response rate and lower item missing response rate. The difference is reduced but remains when restricting comparisons with valid WTPs. Sample characteristics differ, with significant differences between modes for gender, age, income, and attitudes and knowledge. Although difference in WTP values exist, with the IP resulting in higher values, we find limited evidence that such differences are statistically significant. The mail survey has lower initial cost per response; however, once restricting samples to valid WTP responses with nonmissing respondent information, the cost per response across modes is similar. Our results, suggesting that IPs generate valid and cost-effective values, are encouraging as researchers move increasingly to IPs to collect preference data. [ABSTRACT FROM AUTHOR]

International large-scale assessments (ILSAs) transitioned from paper-based assessments to computer-based assessments (CBAs) facilitating the use of new item types and more effective data collection tools. This allows implementation of more complex test designs and to collect process and response time (RT) data. These new data types can be used to improve data quality and the accuracy of test scores obtained through latent regression (population) models. However, the move to a CBA also poses challenges for comparability and trend measurement, one of the major goals in ISLAs. We provide an overview of current methods used in ILSAs to examine and assure the comparability of data across different assessment modes and methods that improve the accuracy of test scores by making use of new data types provided by a CBA. [ABSTRACT FROM AUTHOR]

This paper compares two modes of administering an election survey: a traditional, door-to-door survey and an identical online version promoted via a Voting Advice Application. Whereas online political surveys are known to suffer from self-selection bias of politically interested respondents, traditional surveys are plagued with socially desirable responding and are susceptible to the effects of satisficing and other fatigue-related effects. Using a propensity score matching methodology, we examine the extent to which such differences exist between the two modes of administration. While we report mixed findings regarding the structure of respondents’ answer patterns, significant differences emerged in relation to social desirability bias with the offline group being more ‘affected’ than the online group.

For many years, reading comprehension in the Programme for International Student Assessment (PISA) was measured via paper‐based assessment (PBA). In the 2015 cycle, computer‐based assessment (CBA) was introduced, raising the question of whether central equivalence criteria required for a valid interpretation of the results are fulfilled. As an extension of the PISA 2012 main study in Germany, a random subsample of two intact PISA reading clusters, either computerized or paper‐based, was assessed using a random group design with an additional within‐subject variation. The results are in line with the hypothesis of construct equivalence. That is, the latent cross‐mode correlation of PISA reading comprehension was not significantly different from the expected correlation between the two clusters. Significant mode effects on item difficulties were observed for a small number of items only. Interindividual differences found in mode effects were negatively correlated with reading comprehension, but were not predicted by basic computer skills or gender. Further differences between modes were found with respect to the number of missing values. [ABSTRACT FROM AUTHOR]

Being of great importance for transportation policy appraisals, we investigate mode and user-type effects in the value of travel time savings (VTTS) using a pooled RP/SP Mixed Logit modeling approach for mode, route and destination choice data. For a representative sample of Austrian workers, our analysis reveals population-weighted median VTTS estimates for car (12.3 Euro/h), public transportation (PT; 8.1 Euro/h), bike (11.7 Euro/h) and walk (10.2 Euro/h). Considering only those respondents who have used car and PT in the observation period (and thus are familiar with both modes), we find that four user characteristics are able to decompose this substantial difference in median VTTS between car and PT (i.e. the total mode effect) of about 4.9 Euro/h: Posterior means of individual and mode-specific VTTS distributions reveal a reduced mode effect for high income (4.6 Euro/h), female (4.5 Euro/h), low educated (4.3 Euro/h) and urban (3.0 Euro/h) user groups. Our results indicate that in the case of Austrian workers, characteristics of the mode are more important than characteristics of the users, and that the travel time spent in PT is valued less than in a car for all investigated user groups. [ABSTRACT FROM AUTHOR]

More and more surveys are conducted online. While web surveys are generally cheaper and tend to have lower measurement error in comparison to other survey modes, especially for sensitive questions, potential advantages might be offset by larger nonresponse bias. This article compares the data quality in a web survey administration to another common mode of survey administration, the telephone. The unique feature of this study is the availability of administrative records for all sampled individuals in combination with a random assignment of survey mode. This specific design allows us to investigate and compare potential bias in survey statistics due to 1) nonresponse error, 2) measurement error, and 3) combined bias of these two error sources and hence, an overall assessment of data quality for two common modes of survey administration, telephone and web. Our results show that overall mean estimates on the web are more biased compared to the telephone mode. Nonresponse and measurement bias tend to reinforce each other in both modes, with nonresponse bias being somewhat more pronounced in the web mode. While measurement error bias tends to be smaller in the web survey implementation, interestingly, our results also show that the web does not consistently outperform the telephone mode for sensitive questions. [ABSTRACT FROM AUTHOR]