Search

Survey research aims to collect robust and reliable data from respondents. However, despite researchers’ efforts in designing questionnaires, survey instruments may be imperfect, and question structure not as clear as could be, thus creating a burden for respondents. If it were possible to detect such problems, this knowledge could be used to predict problems in a questionnaire during pretesting, inform real-time interventions through responsive questionnaire design, or to indicate and correct measurement error after the fact. Previous research has used paradata, specifically response times, to detect difficulties and help improve user experience and data quality. Today, richer data sources are available, for example, movements respondents make with their mouse, as an additional detailed indicator for the respondent–survey interaction. This article uses machine learning techniques to explore the predictive value of mouse-tracking data regarding a question’s difficulty. We use data from a survey on respondents’ employment history and demographic information, in which we experimentally manipulate the difficulty of several questions. Using measures derived from mouse movements, we predict whether respondents have answered the easy or difficult version of a question, using and comparing several state-of-the-art supervised learning methods. We have also developed a personalization method that adjusts for respondents’ baseline mouse behavior and evaluate its performance. For all three manipulated survey questions, we find that including the full set of mouse movement measures and accounting for individual differences in these measures improve prediction performance over response-time-only models. German Research Foundation (DFG)

The International Program in Survey and Data Science (IPSDS) is an online educational program, which can be attended through the Joint Program in Survey Methodology (JPSM) at the University of Maryland (UMD) and a part-time Master of Applied Data Science & Measurement (MDM) at the University of Mannheim and Mannheim Business School (MBS). It is targeted towards and attended by working professionals involved or interested in data collection and data analysis including those working in official statistics. The program conveys competencies in the areas of data collection, data analysis, data storage, and data visualization. The faculty of the program includes researchers and lecturers from both the University of Maryland and the University of Mannheim as well as other organizations such as destatis and Statistics Netherlands in the field of official statistics. The program was awarded the label of ‘European Master in Official Statistics (EMOS)’ under conditions in May 2021. In the article, we summarize the methodological and statistical competencies needed in official statistics and show how IPSDS covers this set of skills. We will present the flipped classroom design used for the IPSDS program and demonstrate that it is especially suited for students who are working professionals at the same time.

Audio computer-assisted self-interviewing (ACASI) has been widely used to collect sensitive information from respondents in face-to-face interviews. Interviewers ask questions that are not sensitive or only moderately sensitive and then allow respondents to self-administer more sensitive questions, listening to audio recordings of the questions and typically entering their responses directly into the same device that the interviewer has used. According to the conventional thinking, ACASI is taken as independent of the face-to-face interaction that almost always precedes it. Presumably as a result of this presumed independence, the respondents’ prior interaction with the interviewer is rarely considered when assessing the quality of ACASI responses. There is no body of existing research that has experimentally investigated how the preceding interviewer–respondent interaction may create sufficient social presence to affect responses in the subsequent ACASI module. The study reported here, a laboratory experiment with eight professional interviewers and 125 respondents, explores the carryover effects of preceding interactions between interviewer and respondent on responses in the subsequent ACASI. We evaluated the impact of the similarity of the live and recorded interviewer’s voice for each respondent as well as respondents’ rapport with interviewers in the preceding interview. We did not find significant main effects of vocal similarity on disclosure in ACASI. However, we found significant interaction effects between vocal similarity and respondents’ rapport ratings in the preceding interview on disclosure in ACASI. When the ACASI voice was similar to the interviewer’s voice in the preceding interaction, respondent-rated rapport led to more disclosure but, when the ACASI voice is clearly different from the interviewer’s voice, respondent-rated rapport in the prior interaction did not affect disclosure.

Researchers are combining self-reports from mobile surveys with passive data collection using sensors and apps on smartphones increasingly more often. While smartphones are commonly used in some groups of individuals, smartphone penetration is significantly lower in other groups. In addition, different operating systems (OSs) limit how mobile data can be collected passively. These limitations cause concern about coverage error in studies targeting the general population. Based on data from the Panel Study Labour Market and Social Security (PASS), an annual probability-based mixed-mode survey on the labor market and poverty in Germany, we find that smartphone ownership and ownership of smartphones with specific OSs are correlated with a number of sociodemographic and substantive variables. The use of weighting techniques based on sociodemographic information available for both owners and nonowners reduces these differences but does not eliminate them.

Interviewer-respondent rapport is generally considered to be beneficial for the quality of the data collected in survey interviews; however, the relationship between rapport and data quality has rarely been directly investigated. We conducted a laboratory experiment in which eight professional interviewers interviewed 125 respondents to see how the rapport between interviewers and respondents is associated with the quality of data—primarily disclosure of sensitive information—collected in these interviews. It is possible that increased rapport between interviewers and respondents might motivate respondents to be more conscientious, increasing disclosure; alternatively, increased rapport might inhibit disclosure because presenting oneself unfavorably is more aversive if respondents have a positive relationship with the interviewer. More specifically, we examined three issues: (1) what the relationship is between rapport and the disclosure of information of varying levels of sensitivity, (2) how rapport is associated with item nonresponse, and (3) whether rapport can be similarly established in video-mediated and computer-assisted personal interviews (CAPIs). We found that (1) increased respondents’ sense of rapport increased disclosure for questions that are highly sensitive compared with questions about topics of moderate sensitivity; (2) increased respondents’ sense of rapport is not associated with a higher level of item nonresponse; and (3) there was no significant difference in respondents’ rapport ratings between video-mediated and CAPI, suggesting that rapport is just as well established in video-mediated interviews as it is in CAPI.

A major concern arising from ubiquitous tracking of individuals’ online activity is that algorithms may be trained to predict personal sensitive information, even for users who do not wish to reveal such information. Although previous research has shown that digital trace data can accurately predict sociodemographic characteristics, little is known about the potentials of such data to predict sensitive outcomes. Against this background, we investigate in this article whether we can accurately predict voting behavior, which is considered personal sensitive information in Germany and subject to strict privacy regulations. Using records of web browsing and mobile device usage of about 2,000 online users eligible to vote in the 2017 German federal election combined with survey data from the same individuals, we find that online activities do not predict (self-reported) voting well in this population. These findings add to the debate about users’ limited control over (inaccurate) personal information flows.

The long-standing approach of using probability samples in social science research has come under pressure through eroding survey response rates, advanced methodology, and easier access to large amounts of data. These factors, along with an increased awareness of the pitfalls of the nonequivalent comparison group design for the estimation of causal effects, have moved the attention of applied researchers away from issues of sampling and toward issues of identification. This article discusses the usability of samples with unknown selection probabilities for various research questions. In doing so, we review assumptions necessary for descriptive and causal inference and discuss research strategies developed to overcome sampling limitations.

In the wake of the digital revolution and connected technologies, societies store an ever-increasing amount of data on humans, their preferences, and behavior. These modern technologies create a trust challenge, insofar as individuals have to trust data collectors such as private organizations, government institutions, and researchers that their data is not misused. Privacy regulations should increase trust because they provide laws that increase transparency and allow for punishment in cases in which the trustee violates trust. The introduction of the General Data Protection Regulation (GDPR) in May 2018 – a wide-reaching regulation in EU law on data protection and privacy that covers millions of individuals in Europe – provides a unique setting to study the impact of privacy regulation on trust in data collectors. We collected survey panel data in Germany around the implementation date and ran a survey experiment with a GDPR information treatment. Our observational and experimental evidence does not support the hypothesis that the GDPR has positively affected trust. This finding and our discussion of the underlying reasons are relevant for the wider research field of trust, privacy, and big data.

Numerous within-household selection methods have been tested in general population surveys since the advent of telephone interviewing. However, very few selection studies, if any, have been conducted with a dual frame (landline and cell phone) sample. Landline and cell phone frames are known to represent demographically different groups of respondents, and selection methods that may result in more representative demographics in a landline frame may actually skew the results when combined with the cell phone frame. This study tested 11 different within-household selection methods with approximately 11,000 landline respondents. A parallel cell phone sample was also collected with 1,000 respondents, and the frames were combined for analysis. The selection methods tested included one probability-based method, four quasi-probability methods and six nonprobability methods. The methods were evaluated on four criteria: response rates, accuracy, demographic representation and substantive results. The demographic representativeness of each method was examined for the landline frame only and for the dual (landline and cell phone) frame combination. The probability method had the lowest response rate, while the nonprobability at-home methods had the highest. Accuracy rates were lowest for the quasi-probability birthday methods. There were few demographic differences between selection methods, and no substantive differences, when combined with the cell phone sample.

This article examines the influence of interviewers on the estimation of regression coefficients from survey data. First, we present theoretical considerations with a focus on measurement errors and nonresponse errors due to interviewers. Then, we show via simulation which of several nonresponse and measurement error scenarios has the biggest impact on the estimate of a slope parameter from a simple linear regression model. When response propensity depends on the dependent variable in a linear regression model, bias in the estimated slope parameter is introduced. We find no evidence that interviewer effects on the response propensity have a large impact on the estimated regression parameters. We do find, however, that interviewer effects on the predictor variable of interest explain a large portion of the bias in the estimated regression parameter. Simulation studies suggest that standard measurement error adjustments using the reliability ratio (i.e., the ratio of the measurement-error-free variance to the observed variance with measurement error) can correct most of the bias introduced by these interviewer effects in a variety of complex settings, suggesting that more routine adjustment for such effects should be considered in regression analysis using survey data.

Abstract. Because survey response rates are consistently declining worldwide, survey researchers strive to obtain as much auxiliary information on sampled units as possible. Surveys using in-person interviewing often request that interviewers collect observations on key features of all sampled units, given that interviewers are the eyes and ears of the survey organization. Unfortunately, these observations are prone to error, which decreases the effectiveness of nonresponse adjustments based on the observations. No studies have investigated the strategies being used by interviewers tasked with making these observations, or examined whether certain strategies improve observation accuracy. This study is the first to examine the associations of observational strategies used by survey interviewers with the accuracy of observations collected by those interviewers. A qualitative analysis followed by multilevel models of observation accuracy shows that focusing on relevant correlates of the feature being observed and considering a diversity of cues are associated with increased observation accuracy.

Standardized interviewing (SI) and conversational interviewing are two approaches to collect survey data that differ in how interviewers address respondent confusion. This article examines interviewer–respondent interactions that occur during these two techniques, focusing on requests for and provisions of clarification. The data derive from an experimental study in Germany, where the face-to-face interviews were audio-recorded. A sample of 111 interviews was coded in detail. We find that conversational interviewers do make use of the ability to clarify respondent confusion. Although the technique improved response accuracy in the main study compared to SI, conversational interviewers seem to provide clarifications even when there is no evidence of respondent confusion. This may lengthen administration time and potentially increase data collection costs. Conversational interviewers also employ neutral probes, which are generally associated with standardized interviews, at an unexpectedly high rate. We conclude with suggestions for practice and directions for future research.

Recent methodological studies have attempted to decompose the interviewer variance introduced in interviewer-administered surveys into its potential sources, using the Total Survey Error framework. These studies have informed the literature on interviewer effects by acknowledging interviewers’ dual roles as recruiters and data collectors, thus examining the relative contributions of nonresponse error variance and measurement error variance among interviewers to total interviewer variance. However, this breakdown may depend on the interviewing technique: some techniques emphasize behaviors designed to reduce variation in the answers collected by interviewers more so than other techniques. The question of whether the contributions of these error sources to total interviewer variance change for different interviewing techniques remains unanswered. Addressing this gap in knowledge has important implications for interviewing practice because the technique used could alter the relative contributions of variance in these error sources to total interviewer variance. This article presents results from an experimental study mounted in Germany that was designed to answer this question about two specific interviewing techniques. A national sample of employed individuals was first selected from a database of official administrative records, then randomly assigned to interviewers who themselves were randomized to conduct either conversational interviewing (CI) or standardized interviewing (SI), and finally measured face-to-face on a variety of cognitively challenging survey questions with official values also available for verifying the accuracy of responses. We find that although nonresponse error variance does exist among interviewers for selected measures (especially respondent age in the CI group), measurement error variance tends to be the more important source of total interviewer variance, regardless of whether interviewers are using CI or SI.

Respondent incentives are widely used to increase response rates, but their effect on nonresponse bias has not been researched as much. To contribute to the research, we analyze an incentive experiment embedded within the third wave of the German household panel survey “Panel Labor Market and Social Security” conducted by the German Institute for Employment Research. Our question is whether attrition bias differs in two incentive plans. In particular, we want to study whether an unconditional €10 cash incentive yields less attrition bias in self-reported labor income and other sociodemographics than a conditional lottery ticket incentive. We find that unconditional cash incentives are more effective than conditional lottery tickets in reducing attrition bias in income and several sociodemographic variables.

Summary Currently, most surveys ask for occupation with open-ended questions. The verbal responses are coded afterwards, which is error prone and expensive. We present an alternative approach that allows occupation coding during the interview. Our new technique uses a supervised learning algorithm to predict candidate job categories. These suggestions are presented to the respondent, who in turn can choose the most appropriate occupation. 72.4% of the respondents selected an occupation when the new instrument was tested in a telephone survey, entailing potential cost savings. To aid further improvements, we identify some factors for how to increase quality and to reduce interview duration.

Summary Several studies have shown that conversational interviewing (CI) reduces response bias for complex survey questions relative to standardized interviewing. However, no studies have addressed concerns about whether CI increases intra-interviewer correlations (IICs) in the responses collected, which could negatively impact the overall quality of survey estimates. The paper reports the results of an experimental investigation addressing this question in a national face-to-face survey. We find that CI improves response quality, as in previous studies, without substantially or frequently increasing IICs. Furthermore, any slight increases in the IICs do not offset the reduced bias in survey estimates engendered by CI.

Numerous surveys link interview data to administrative records, conditional on respondent consent, in order to explore new and innovative research questions. Optimizing the linkage consent rate is a critical step toward realizing the scientific advantages of record linkage and minimizing the risk of linkage consent bias. Linkage consent rates have been shown to be particularly sensitive to certain design features, such as where the consent question is placed in the questionnaire and how the question is framed. However, the interaction of these design features and their relative contributions to the linkage consent rate have never been jointly studied, raising the practical question of which design feature (or combination of features) should be prioritized from a consent rate perspective. We address this knowledge gap by reporting the results of a placement and framing experiment embedded within separate telephone and Web surveys. We find a significant interaction between placement and framing of the linkage consent question on the consent rate. The effect of placement was larger than the effect of framing in both surveys, and the effect of framing was only evident in the Web survey when the consent question was placed at the end of the questionnaire. Both design features had negligible impact on linkage consent bias for a series of administrative variables available for consenters and non-consenters. We conclude this research note with guidance on the optimal administration of the linkage consent question.

A key goal of survey interviews is to collect the highest quality data possible from respondents. In practice, however, it can be difficult to achieve this goal because respondents do not always understand particular survey questions as designers intended. Researchers have used a variety of indicators to identify and predict respondent confusion and difficulty in answering questions in different modes. In web surveys, it is possible to automatically detect response difficulty in real time. The research to date has focused on response latencies—mostly long response times—as evidence of difficulty. In addition to response latencies, however, web surveys offer rich behavioral data, which may predict respondent confusion and difficulty more directly than response times. This article focuses on one such behavior, mouse movements. We examine a set of mouse movements participants engage in when answering questions about experimental scenarios whose difficulty has been manipulated (as confirmed by respondent ratings). This approach makes it possible to determine which movements are general movements, demonstrating how a person interacts with a computer, and which movements are related to response difficulty. We find not only that certain mouse movements are highly predictive of difficulty but also that such movements add considerable value when used in conjunction with response times. The approach developed in this article may be useful in delivering help to confused respondents in real time and as a diagnostic tool to identify confusing questions.

Looping questions are used to collect data about several similar events, such as employment spells, retirement accounts, or marriages. The loops gather information about the number of events experienced as well as details about each one. The questions require respondents to think hard to recall each event and are often lengthy and repetitive. Looping questions can be asked in two formats, and which format a survey uses may aect the quality of the data collected. We develop hypotheses about the eects of format on measurement error in looping questions and test the hypotheses using experimental data from a web survey with a link to administrative records. Results show that one format collects more accurate event reports, but the other format provides higher quality data to the follow up questions. We conclude with guidance for those who write survey questions as well as those who rely on survey data for substantive analyses., Survey Research Methods, Vol 12, No 1 (2018)

Das International Program in Survey and Data Science (IPSDS) zielt darauf ab, den steigenden Bedarf an Fachkraften im Bereich Datenerhebung und Datenanalyse durch einen berufsbegleitenden Studiengang zu decken. Anders als viele Weiterbildungsangebote greift das Programm stark auf moderne Formen des asynchronen und synchronen Lernens zuruck, um in Beschaftigung befindlichen Personen und Personen mit Familienverpflichtungen entgegen zu kommen. Der Studiengang wurde mit finanzieller Hilfe des Bundesministeriums fur Bildung und Forschung im Rahmen der Ausschreibung „Aufstieg durch Bildung: offene Hochschulen“ unterstutzt.

Obtaining informed consent from individuals to participate in voluntary research studies is widely considered to be an ethical research practice. However, there is considerable debate over how consent should be obtained from subjects. Many researchers argue that active (opt-in) consent is the only type of consent that accurately reflects the true wishes of the subject and is closer to the informed consent ideal than passive (opt-out) consent procedures. Opponents of active consent procedures argue that such procedures harm study participation rates and increase the risk of self-selection bias to a greater extent than passive consent procedures. Empirical evaluations of these claims are rare, given the lack of studies that experimentally assign subjects to different consent procedures and utilize a control group (in which no consent is sought) to facilitate comparison. We report on an experiment that overcomes these issues in an study of consent to transfer contact data from a federal register to a third-party data collector for purposes of carrying out a telephone survey. Specifically, we evaluate the impact of requiring consent on survey participation rates, self-selection bias, and the resulting survey estimates. We find that the passive consent procedure does a better job of minimizing self-selection bias and maximizing the validity of the survey estimates (relative to the control group) compared with the active consent procedure. However, neither procedure is ideal: Both consent procedures increase the total self-selection bias and reduce the sample size. We conclude with a general discussion of the main findings and their practical implications.