Search

Die Studie "FReDA - Das familiendemografische Panel" wird gemeinsam vom Bundesinstitut für Bevölkerungsforschung (BiB), GESIS Leibniz-Institut für Sozialwissenschaften und dem pairfam-Konsortium aufgebaut. Es handelt sich um eine Infrastruktur in Deutschland, die das Gender & Generations Programme (GGP) und pairfam unter einem Dach verbindet. Im Kern von FReDA stehen jährliche Wiederholungsbefragungen von 18- bis 49-jährigen Frauen und Männern sowie deren Partnerinnen und Partnern. Hierdurch soll es Datennutzern ermöglicht werden verschiedene familiendemographische, familiensoziologische und familienpsychologische Fragestellungen zu beantworten. Zur Vorbereitung der FReDA-Welle 3b sollten insgesamt elf neu entwickelte bzw. adaptierte Fragen zum Partnerschaftsstatus, zu Ernährungsstilen, globaler Unsicherheit und zum Vertrauen in Institutionen in einem kognitiven Online-Pretest auf ihre Verständlichkeit hin untersucht werden.

Social media data (SMD) have become an important data source in the social sciences. The purpose of this paper is to investigate the experiences and practices of researchers working with SMD in their research and gain insights into researchers' sharing behavior and influencing factors for their decisions. To achieve these aims, we conducted a survey study among researchers working with SMD. The questionnaire covered different topics related to accessing, (re)using, and sharing SMD. To examine attitudes toward data sharing, perceived subjective norms, and perceived behavioral control, we used questions based on the Theory of Planned Behavior (TPB). We employed a combination of qualitative and quantitative analyses. The results of the qualitative analysis show that the main reasons for not sharing SMD were that sharing was not considered or needed, as well as legal and ethical challenges. The quantitative analyses reveal that there are differences in the relative importance of past sharing and reuse experiences, experienced challenges, attitudes, subjective norms, and perceived behavioral control as predictors of future SMD sharing intentions, depending on the way the data should be shared (publicly, with restricted access, or upon personal request). Importantly, the TPB variables have predictive power for all types of SMD sharing.

This paper aims to analyze the effect on measured inequality and its structure of using administrative data instead of survey data. Different analyses are carried out based on the Spanish Survey on Income and Living Conditions (ECV) that continued to ask households for their income despite assigning their income data as provided by the Tax Agency and the Social Security Administration. Our main finding is that the largest discrepancies between administrative and survey data are in the tails of the distribution. In addition to that, there are clear differences in the level and structure of inequality across data sources. These differences matter, and our results should be a wake-up call to interpret the results based on only one source of income data with caution.

Recent advances in web survey methodology were motivated by the observation that respondents increasingly use mobile devices, such as smartphones and tablets, to participate in web surveys. Even though we do not doubt this general observation, we argue that the claim is lacking a solid empirical basis. Most research on increasing mobile device use in web surveys covers limited periods of time and/or analyzes data from only one study or panel. There is a surprising lack of comprehensive overviews on the magnitude of mobile device use in web surveys. In the present study, we explored this research gap by analyzing data from 128 web surveys collected in four different academic studies in Germany between 2012 and 2020. Overall, we found strong empirical evidence for an increase in smartphone use, a stagnation in tablet use, and a decrease in desktop PC use. There was no evidence that the increase in smartphone use will slow down any time soon. Thus, we recommend that survey researchers prepare for a device change in web surveys that may enable new applications in web surveys.

The goal of this research was to determine the best way to present mixed-device surveys. We investigate the effect of survey method (messenger versus regular survey), answer scale, device used, and personal characteristics such as gender, age and education on break-off rate, substantive answers, completion time and respondents' evaluation of the survey. Our research does not suggest that a messenger survey affects mixed-device surveys positively. Further research is necessary to investigate how to optimally present mixed-device surveys in order to increase participation and data quality.

The ease with which large amounts of data can be collected via the Internet has led to a renewed interest in the use of non-probability samples. To that end, this paper performs a case study, comparing two non-probability datasets - one based on a river-sampling ap­proach, one drawn from an online-access panel - to a reference probability sample. Of particular interest is the single-question river-sampling approach, as the data collected for this study presents an attempt to field a multi-item scale with such a sampling method. Each dataset consists of the same psychometric measures for two of the Big-5 personality traits, which are expected to perform independently of sample composition. To assess the similarity of the three datasets we compare their correlation matrices, apply linear and non-linear dimension reduction techniques, and analyze the distance between the datasets. Our results show that there are important limitations when implementing a multi-item scale via a single-question river sample. We find that, while the correlation between our data sets is similar, the samples are composed of persons with different personality traits.

In Germany, the population registers with addresses of individuals can be used for address-based sampling. However, unlike countries with a centralized register, municipalities in Germany administer their registers themselves. This not only makes sampling for a nation­wide survey more costly and cumbersome but may also result in gaps in the gross sample, as selected municipalities may refuse to allow their registers to be used for sampling pur­poses. If substitute municipalities are not available, other sampling methods are required. The present study tested the feasibility of using the address database from Deutsche Post Direkt (ADB-DPD) as an alternative frame for address-based sampling in Germany. We simultaneously conducted two almost identical surveys in the German city of Mannheim with gross samples of equal size (N = 3,000). One sample was drawn from the city’s popu­lation register, the other from the commercial ADB-DPD. Our findings suggest that the ADB-DPD performs well both in terms of survey response and up-to-dateness. Due to relatively low costs and the fast provision of addresses, the ADB-DPD could be particu­larly attractive for survey projects with limited budgets and tight schedules. However, these benefits come at considerable cost. First, the use of the ADB-DPD is limited to self-admin­istered surveys. More importantly, in the net sample of the DPD survey, women and young persons were considerably underrepresented. This indicates coverage issues about which DPD provided no further information. Based on our analyses, we offer practical insights into the feasibility of using the ADB-DPD for sampling purposes and suggest avenues for future research.

Online probability-based panels often apply two or more data collection modes to cover both the online and offline populations with the aim of obtaining results that are more representative of the population of interest. This study used such a panel to investigate how necessary it is, from the coverage error standpoint, to include the offline population by mixing modes in online panel survey research. This study evaluated the problem from three different perspectives: undercoverage bias, bias related to survey item topics and vari­able characteristics, and accuracy of online-only samples relative to nationally representa­tive benchmarks. The results indicated that attitudinal, behavioral, and factual differences between the online and offline populations in Australia are, on average, minor. This means that, considering that survey research commonly includes a relatively low proportion of the offline population, survey estimates would not be significantly affected if probability-based panels did not mix modes and instead were online only, for the majority of topics. The benchmarking analysis showed that mixing the online mode with the offline mode did not improve the average accuracy of estimates relative to nationally representative bench­marks. Based on these findings, it is argued that other online panels should study this issue from different perspectives using the approaches proposed in this paper. There might also be an argument for (temporarily) excluding the offline population in probability-based on­line panel research in particular country contexts as this might have practical implications.

The innovative software system "myEcoCost" enables to gather and communicate resource and environmental data for products and services in global value chains. The system has been developed in the consortium of the European research project myEcoCost and forms a basis of a new, highly automated environmental accounting system für companies and consumers. The prototype of the system, linked to financial accounting of companies, was developed and tested in close collaboration with large and small companies. This brochure gives a brief introduction to the vision linked to myEcoCost: a network formed by collaborative environmental accounting nodes collecting environmental data at each step in a product's value chains. It shows why better life cycle data are needed and how myEcoCost addresses and solves this problem. Furthermore, it presents options for a future upscaling of highly automated environmental accounting for prodcuts and services.

European inequality research often follows the tradition of using occupational based categorical classification to identify the socioeconomic position of individuals or households (e.g., classes, strata, milieus, occupational groups). In the past, European studies have often used the Erikson-Goldthorpe-Portocarero class (EGP) scheme (cf. Erikson & Goldthorpe 1992). However, the EGP scheme has only been validated for Great Britain (cf. Evans 1992). Variants for other countries are largely based on informed plausibility, following the British model, though operationalisation may vary by country. As an alternative to the EGP, two other currently available categorical concepts are the ESeC (European Socioeconomic Classification) and the ESeG (European Socioeconomic Groups), both based on the International Standard Classification of Occupation (ISCO). Both concepts are intended to improve cross-country comparative analysis of social inequality in Europe. However, the ESeC and ESeG differ in their theoretical basis and should not be confused with each other, and the ESeC and ESeG also cannot readily be transferred into each other. The ESeC is available for ISCO-88 (ESeC88) and ISCO-08 (ESeC08). The ESeG is available for ISCO-08. In this paper, we focus on the operationalisation of ESeC with EU-SILC cross-sectional data (2004-2020). Alongside this report, we have published syntax files (SPSS, Stata, and R) which can be used for the operationalisation of ESeC and ESeG.

While survey research has been at the heart of social science for decades and social scientific research with digital trace data has been growing rapidly in the last few years, until now, there are relatively few studies that combine these two data types. This may be surprising given the potential of linking surveys and digital trace data, but at the same time, it is important to note that the collection and analysis of such linked data are challenging in several regards. The three key issues are: (1) data linking including informed consent for individual-level studies, (2) methodological and ethical issues impeding the scientific (re)analysis of linked survey and digital trace data sets, and (3) developing conceptual and theoretical frameworks tailored toward the multidimensionality of such data. This special issue addresses these challenges by presenting cutting-edge methodological work on how to best collect and analyze linked data as well as studies that have successfully combined survey data and digital trace data to find innovative answers to relevant social scientific questions.

This study correlated self-report and trace data measures of political incivility. Specifically, we asked respondents to provide estimates of the degree to which they engage in uncivil political communication online. These estimates were then compared to computational measures of uncivil social media discussion behavior. The results indicated that those who self-disclose uncivil online behavior also tend to generate content on social media that is uncivil as identified by Google's Perspective application programming interface. Taken as a whole, this work suggests that combining self-report and behavioral trace data may be a fruitful means of developing multimethod measures of complex communication behaviors.

Researchers hoping to make inferences about social phenomena using social media data need to answer two critical questions: What is it that a given social media metric tells us? And who does it tell us about? Drawing from prior work on these questions, we examine whether Twitter sentiment about Barack Obama tells us about Americans' attitudes toward the president, the attitudes of particular subsets of individuals, or something else entirely. Specifically, using large-scale survey data, this study assesses how patterns of approval among population subgroups compare to tweets about the president. The findings paint a complex picture of the utility of digital traces. Although attention to subgroups improves the extent to which survey and Twitter data can yield similar conclusions, the results also indicate that sentiment surrounding tweets about the president is no proxy for presidential approval. Instead, after adjusting for demographics, these two metrics tell similar macroscale, long-term stories about presidential approval but very different stories at a more granular level and over shorter time periods.

This perspective article suggests considering the everyday research data management work required to accomplish social media research along different phases in a data lifecycle to inform the ongoing discussion of social media research data’s quality and validity. Our perspective is informed by practical experience of archiving social media data, by results from a series of qualitative interviews with social media researchers, as well as by recent literature in the field. We emphasize how social media researchers are entangled in complexities between social media platform providers, social media users, other actors, as well as legal and ethical frameworks, that all affect their everyday research practices. Research design decisions are made iteratively at different stages, involving many decisions that may potentially impact the quality of research. We show that these decisions are often hidden, but that making them visible allows us to better understand what drives social media research into specific directions. Consequently, we argue that untangling and documenting choices during the research lifecycle, especially when researchers pursue specific approaches and may have actively decided against others (often due to external factors) is necessary and will help to spot and address structural challenges in the social media research ecosystem that go beyond critiques of individual opportunistic approaches to easily accessible data.

Facebook has been credited with expanding political activity by simultaneously lowering barriers to participation and creating new ways to engage. However, many of these findings rely on subjects’ abilities to accurately report their Facebook use and political activity on the platform. This study combines survey responses and digital trace data from 828 American adults to determine whether subjects over- or underreport a range of political activities on Facebook, including whether they like political pages or share news links. The results show that individuals underestimate their frequency of status posting and overestimate their frequency of sharing news links on Facebook. Political interest is associated with a decrease in underreporting several political activities, while increasing the likelihood of overreporting the frequency of sharing news links. Furthermore, political interest serves a moderating effect, improving self-reports for high-volume users. The findings suggest that political interest not only predicts political activity but also shapes awareness of that activity and improves self-reports among heavy users.

The pervasive use of mobile information technologies brings new patterns of media usage, but also challenges to the measurement of media exposure. Researchers wishing to, for example, understand the nature of selective exposure on algorithmically driven platforms need to precisely attribute individuals’ exposure to specific content. Prior research has used tracking data to show that survey-based self-reports of media exposure are critically unreliable. So far, however, little effort has been invested into assessing the specific biases of tracking methods themselves. Using data from a multimethod study, we show that tracking data from mobile devices is linked to systematic distortions in self-report biases. Further inherent but unobservable sources of bias, along with potential solutions, are discussed.

With the increasing use of smartphones in web surveys, considerable efforts have been devoted to reduce the amount of screen space taken up by questions. An emerging stream of research in this area is aimed at optimizing the design elements of rating scales. One suggestion that has been made is to completely abandon verbal labels and use only numeric labels instead. This approach deliberately shifts the task of scale interpretation to the respondents and reduces the information given to them with an intention to reduce their response burden while still preserving the scale meaning. Following prior research, and by drawing on the established model of the cognitive response process, we critically tested these assumptions. Based on a web survey experiment, we found that omitting verbal labels and using only numeric labels instead pushed respondents to focus their responses on the endpoints of a rating scale. Moreover, drawing on response time paradata, we showed that their response burden was not reduced when presented with only numeric labels; quite the opposite was the case, especially when respondents answered the scale with only numeric labels for the first time, which seemed to entail additional cognitive effort. Based on our findings, we advise against using only numeric labels for rating scales in web surveys.

The collection and use of paradata is gaining in importance, especially in web surveys. From a research ethics’ perspective, respondents should be asked for their consent to the collection and use of web paradata. In this context, a positive attitude toward paradata use has been deemed to be a prerequisite for respondents’ willingness to share their paradata. The present study aimed to identify factors affecting respondents’ attitudes toward paradata use. Our findings revealed that adequately informing survey respondents about what paradata are and why they are used was an important determinant of their attitudes toward paradata use. Moreover, we found that respondents with a positive attitude toward the survey were more likely to have a favorable opinion of paradata use. Our findings suggest that a thorough understanding of the factors that contribute to a positive attitude toward paradata use provides the basis for improved paradata consent procedures, which in turn will increase rates of consent to paradata use and help attenuate the risk of consent bias in web surveys.

To collect high-quality data, survey designers aim to develop questions that each respondent can understand as intended. A critical step to this end is designing questions that minimize the respondents' burden by reducing the cognitive effort required to comprehend and answer them. One promising technique for identifying problematic survey questions is eye tracking. This article investigates the potential of eye movements and pupil dilations as indicators for evaluating survey questions. Respondents were randomly assigned to either a problematic or an improved version of six experimental questions. By analyzing fixation times, fixation counts, and pupil diameters, it was examined whether these parameters could be used to distinguish between the two versions. Identifying the improved version worked best by comparing fixation times, whereas in most cases, it was not possible to differentiate between versions on the basis of pupil data. Limitations and practical implications of the findings are discussed.

The ubiquity of digital devices and the increasing intensity of users’ interactions with them create vast amounts of digital trace data. Companies use these data to optimize their services or products, but these data are also of interest to researchers studying human behavior. As most of these data are owned by private companies and their collection requires adherence to their terms of service, research with digital trace data often entails some form of public-private partnership. Private companies and academic researchers each have their own interests, some of which are shared, while others may conflict. In this article, we explore different types of private-public partnerships for research with digital trace data. Based on general considerations and particular experiences from a research project with linked digital trace data, we propose strategies for identifying and productively negotiating both shared and conflicting interests in these relationships.

Methodological studies usually gauge response quality in narrative open-ended questions with the proportion of nonresponse, response length, response time, and number of themes mentioned by respondents. However, not all of these indicators may be comparable and appropriate for evaluating open-ended questions in a cross-national context. This study assesses the cross-national appropriateness of these indicators and their potential bias. For the analysis, we use data from two web surveys conducted in May 2014 with 2,685 respondents and in June 2014 with 2,689 respondents and compare responses from Germany, Great Britain, the United States, Mexico, and Spain. We assess open-ended responses for a variety of topics (e.g., national identity, gender attitudes, and citizenship) with these indicators and evaluate whether they arrive at similar or contradictory conclusions about response quality. We find that all indicators are potentially biased in a cross-national context due to linguistic and cultural reasons and that the bias differs in prevalence across topics. Therefore, we recommend using multiple indicators as well as items covering a range of topics when evaluating response quality in open-ended questions across countries.

In light of issues such as increasing unit nonresponse in surveys, several studies argue that social media sources such as Twitter can be used as a viable alternative. However, there are also a number of shortcomings with Twitter data such as questions about its representativeness of the wider population and the inability to validate whose data you are collecting. A useful way forward could be to combine survey and Twitter data to supplement and improve both. To do so, consent within a survey is first needed. This study explores the consent decisions in three large representative surveys of the adult British population to link Twitter data to survey responses and the impact that demographics and survey mode have on these outcomes. Findings suggest that consent rates for data linkage are relatively low, and this is in part mediated by mode, where face-to-face surveys have higher consent rates than web versions. These findings are important to understand the potential for linking Twitter and survey data but also to the consent literature generally.

Understanding how citizens keep themselves informed about current affairs is crucial for a functioning democracy. Extant research suggests that in an increasingly fragmented digital news environment, search engines and social media platforms promote more incidental, but potentially more shallow modes of engagement with news compared to the act of routinely accessing a news organization's website. In this study, we examine classic predictors of news consumption to explain the preference for three modes of news engagement in online tracking data: routine news use, news use triggered by social media, and news use as part of a general search for information. In pursuit of this aim, we make use of a unique data set that combines tracking data with survey data. Our findings show differences in predictors between preference for regular (direct) engagement, general search-driven, and social media-driven modes of news engagement. In describing behavioral differences in news consumption patterns, we demonstrate a clear need for further analysis of behavioral tracking data in relation to self-reported measures in order to further qualify differences in modes of news engagement.

The new European General Data Protection Regulation (GDPR) imposes enhanced requirements on digital data collection. This article reports from a 2018 German nationwide population-based probability app study in which participants were asked through a GDPR compliant consent process to share a series of digital trace data, including geolocation, accelerometer data, phone and text messaging logs, app usage, and access to their address books. With about 4,300 invitees and about 650 participants, we demonstrate (1) people were just as willing to share such extensive digital trace data as they were in studies with far more limited requests; (2) despite being provided more decision-related information, participants hardly differentiated between the different data requests made; and (3) once participants gave consent, they did not tend to revoke it. We also show (4) evidence for a widely-held belief that explanations regarding data collection and data usage are often not read carefully, at least not within the app itself, indicating the need for research and user experience improvement to adequately inform and protect participants. We close with suggestions to the field for creating a seal of approval from professional organizations to help the research community promote the safe use of data.

Human-centered approaches are of particular importance when analyzing urban spaces in technology-driven fields, because understanding how people perceive and react to their environments depends on several dynamic and static factors, such as traffic volume, noise, safety, urban configuration, and greenness. Analyzing and interpreting emotions against the background of environmental information can provide insights into the spatial and temporal properties of urban spaces and their influence on citizens, such as urban walkability and bikeability. In this study, we present a comprehensive mixed-methods approach to geospatial analysis that utilizes wearable sensor technology for emotion detection and combines information from sources that correct or complement each other. This includes objective data from wearable physiological sensors combined with an eDiary app, first-person perspective videos from a chest-mounted camera, and georeferenced interviews, and post-hoc surveys. Across two studies, we identified and geolocated pedestrians' and cyclists' moments of stress and relaxation in the city centers of Salzburg and Cologne. Despite open methodological questions, we conclude that mapping wearable sensor data, complemented with other sources of information - all of which are indispensable for evidence-based urban planning - offering tremendous potential for gaining useful insights into urban spaces and their impact on citizens.

Identifying inattentive respondents in self-administered surveys is a challenging goal for survey researchers. Instructed response items (IRIs) provide a measure for inattentiveness in grid questions that is easy to implement. The present article adds to the sparse research on the use and implementation of attention checks by addressing three research objectives. In a first study, we provide evidence that IRIs identify respondents who show an elevated use of straightlining, speeding, item nonresponse, inconsistent answers, and implausible statements throughout a survey. Excluding inattentive respondents, however, did not alter the results of substantive analyses. Our second study suggests that respondents' inattentiveness partially changes as the context in which they complete the survey changes. In a third study, we present experimental evidence that a mere exposure to an IRI does not negatively or positively affect response behavior within a survey. A critical discussion on using IRI attention checks concludes this article.

Filter questions are used to administer follow-up questions to eligible respondents while allowing respondents who are not eligible to skip those questions. Filter questions can be asked in either the interleafed or the grouped formats. In the interleafed format, the follow-ups are asked immediately after the filter question; in the grouped format, follow-ups are asked after the filter question block. Underreporting can occur in the interleafed format due to respondents’ desire to reduce the burden of the survey. This phenomenon is called motivated misreporting. Because smartphone surveys are more burdensome than web surveys completed on a computer or laptop, due to the smaller screen size, longer page loading times, and more distraction, we expect that motivated misreporting is more pronounced on smartphones. Furthermore, we expect that misreporting occurs not only in the filter questions themselves but also extends to data quality in the follow-up questions. We randomly assigned 3,517 respondents of a German online access panel to either the PC or the smartphone. Our results show that while both PC and smartphone respondents trigger fewer filter questions in the interleafed format than the grouped format, we did not find differences between PC and smartphone respondents regarding the number of triggered filter questions. However, smartphone respondents provide lower data quality in the follow-up questions, especially in the grouped format. We conclude with recommendations for web survey designers who intend to incorporate smartphone respondents in their surveys.

Once recruited, probability-based online panels have proven to enable high-quality and high-frequency data collection. In ever faster-paced societies and, recently, in times of pandemic lockdowns, such online survey infrastructures are invaluable to social research. In absence of email sampling frames, one way of recruiting such a panel is via postal mail. However, few studies have examined how to best approach and then transition sample members from the initial postal mail contact to the online panel registration. To fill this gap, we implemented a large-scale experiment in the recruitment of the 2018 sample of the German Internet Panel (GIP) varying panel recruitment designs in four experimental conditions: online-only, concurrent mode, online-first, and paper-first. Our results show that the online-only design delivers higher online panel registration rates than the other recruitment designs. In addition, all experimental conditions led to similarly representative samples on key socio-demographic characteristics.

Despite the importance of news exposure to political outcomes, news consumption is notoriously difficult to measure, and misreporting news exposure is common. In this study, we compare participants' news behaviors measured on a news aggregator website with their self-reported story selection immediately after exposure. We find that both individual and contextual characteristics - especially the presence of political cues in news headlines - influence reporting of news story selection. As a result, the news audience profiles differ using self-reported versus behavioral measures, creating two different pictures of news exposure. More attention is needed to improve news measurement strategies to address misreporting and to improve the accuracy of news audience profiles.

Questionnaire design is routinely guided by classic experiments on question form, wording, and context conducted decades ago. This article explores whether two question order effects (one due to the norm of evenhandedness and the other due to subtraction or perceptual contrast) appear in surveys of probability samples in the United States and 11 other countries (Canada, Denmark, Germany, Iceland, Japan, the Netherlands, Norway, Portugal, Sweden, Taiwan, and the United Kingdom; N = 25,640). Advancing theory of question order effects, we propose necessary conditions for each effect to occur, and found that the effects occurred in the nations where these necessary conditions were met. Surprisingly, the abortion question order effect even appeared in some countries in which the necessary condition was not met, suggesting that the question order effect there (and perhaps elsewhere) was not due to subtraction or perceptual contrast. The question order effects were not moderated by education. The strength of the effect due to the norm of evenhandedness was correlated with various cultural characteristics of the nations. Strong support was observed for the form-resistant correlation hypothesis.

When formulating questionnaire items, generally accepted rules include: Keeping the wording as simple as possible and avoiding double-barreled items. However, the empirical basis for these rules is sparse. The present study aimed to systematically investigate in an experimental design whether simplifying items of a personality scale and avoiding double-barreled items (i.e., items that contain multiple stimuli) markedly increases psychometric quality. Specifically, we compared the original items of the Big Five Inventory-2 - most of which are either double-barreled or can be regarded as complexly formulated - with simplified versions of the items. We tested the two versions using a large, heterogeneous sample (N = 2,234). The simplified versions did not possess better psychometric quality than their original counterparts; rather, they showed weaker factorial validity. Regarding item characteristics, reliability, and criterion validity, no substantial differences were identified between the original and simplified versions. These findings were also replicated for the subsample of lower-educated respondents, who are considered more sensitive to complex item formulations. Our study thus suggests that simplifying item wording and avoiding double-barreled items in a personality inventory does not improve the quality of a questionnaire; rather, using simpler (and consequently more vague) item formulations may even decrease factorial validity.

Attention checks detect inattentiveness by instructing respondents to perform a specific task. However, while respondents may correctly process the task, they may choose to not comply with the instructions. We investigated the issue of noncompliance in attention checks in two web surveys. In Study 1, we measured respondents’ attitudes toward attention checks and their self-reported compliance. In Study 2, we experimentally varied the reasons given to respondents for conducting the attention check. Our results showed that while most respondents understand why attention checks are conducted, a nonnegligible proportion of respondents evaluated them as controlling or annoying. Most respondents passed the attention check; however, among those who failed the test, 61% seem to have failed the task deliberately. These findings reinforce that noncompliance is a serious concern with attention check instruments. The results of our experiment showed that more respondents passed the attention check if a comprehensible reason was given.

Unlike other data collection modes, the effect of labeling rating scales on reliability and validity, as relevant aspects of measurement quality, has seldom been addressed in online surveys. In this study, verbal and numeric rating scales were compared in split-ballot online survey experiments. In the first experiment, respondents’ cognitive processes were observed by means of eye tracking, that is, determining the respondent’s fixations in different areas of the screen. In the remaining experiments, data for reliability and validity analysis were collected from a German adult sample. The results show that respondents needed more fixations and more time to endorse a category when a rating scale had numeric labels. Cross-sectional reliability was lower and some hypotheses with respect to the criterion validity could not be supported when numeric rating scales were used. In conclusion, theoretical considerations and the empirical results contradict the current broad usage of numeric scales in online surveys.

Cognitive pretesting is an essential method of piloting questionnaires and ensuring quality of survey data. Web probing has emerged as an innovative method of cognitive pretesting, especially for cross-cultural and web surveys. The order of presenting questions in cognitive pretesting can differ from the order of presentation in the later survey. Yet empirical evidence is missing whether the order of presenting survey questions influences the answers to open-ended probing questions. The present study examines the effect of question order on web probing in the United States and Germany. Results indicate that probe responses are not strongly impacted by question order. However, both content and consistency of probe responses may differ cross-culturally. Implications for cognitive pretesting are discussed.

When people use the Internet, they leave traces of their activities: blog posts, comments, articles, social media posts, etc. These traces represent behavior that psychologists can analyze. A method that makes downloading those sometimes very large datasets feasible is web scraping, which involves writing a program to automatically download specific parts of a website. The obtained data can be used to exploratorily generate new hypotheses, test existing ones, or extend existing research. The present Research Spotlight explains web scraping and discusses the possibilities, limitations as well as ethical and legal challenges associated with the approach.

Response probabilities are used in adaptive and responsive survey designs to guide data collection efforts, often with the goal of diversifying the sample composition. However, if response probabilities are also correlated with measurement error, this approach could introduce bias into survey data. This study analyzes the relationship between response probabilities and data quality in grid questions. Drawing on data from the probability-based GESIS panel, we found low propensity cases to more frequently produce item nonresponse and nondifferentiated answers than high propensity cases. However, this effect was observed only among long-time respondents, not among those who joined more recently. We caution that using adaptive or responsive techniques may increase measurement error while reducing the risk of nonresponse bias.