Search

Although polling is not irredeemably broken, changes in technology and society create challenges that, if not addressed well, can threaten the quality of election polls and other important surveys on topics such as the economy. This essay describes some of these challenges and recommends remediations to protect the integrity of all kinds of survey research, including election polls. These 12 recommendations specify ways that survey researchers, and those who use polls and other public-oriented surveys, can increase the accuracy and trustworthiness of their data and analyses. Many of these recommendations align practice with the scientific norms of transparency, clarity, and self-correction. The transparency recommendations focus on improving disclosure of factors that affect the nature and quality of survey data. The clarity recommendations call for more precise use of terms such as “representative sample” and clear description of survey attributes that can affect accuracy. The recommendation about correcting the record urges the creation of a publicly available, professionally curated archive of identified technical problems and their remedies. The paper also calls for development of better benchmarks and for additional research on the effects of panel conditioning. Finally, the authors suggest ways to help people who want to use or learn from survey research understand the strengths and limitations of surveys and distinguish legitimate and problematic uses of these methods.

Background Annual vaccination is the most effective strategy for preventing influenza. We assessed trends and demographic and access-to-care characteristics associated with place of vaccination in recent years. Methods Data from the 2014-2018 National Internet Flu Survey were analyzed to assess trends in place of early-season influenza vaccination during the 2014-15 through 2018-19 seasons. Multivariable logistic regression was conducted to identify factors independently associated with vaccination settings in the 2018-19 season. Results Among vaccinated adults, the proportion vaccinated in medical (range: 49%-53%) versus nonmedical settings (range: 47%-51%) during the 2014-15 through 2018-19 seasons were similar. Among adults aged ≥18 years vaccinated early in the 2018-19 influenza season, a doctor's office was the most common place (34.4%), followed by pharmacies or stores (32.3%), and workplaces (15.0%). Characteristics significantly associated with an increased likelihood of receipt of vaccination in nonmedical settings among adults included household income ≥$50,000, having no doctor visits since July 1, 2018, or having a doctor visit but not receiving an influenza vaccination recommendation from the medical professional. Conclusions Place of early-season influenza vaccination among adults who reported receiving influenza vaccination was stable over 5 recent seasons. Both medical and nonmedical settings were important places for influenza vaccination. Increasing access to vaccination services in medical and nonmedical settings should be considered as an important strategy for improving vaccination coverage.

The literature has no standard method for estimating the coverage of area probability segments in address-based frames. Versatility is desirable for different study needs, but standardization improves comparability. Many segment estimates are simple ratios of counts of frame addresses to control totals, or net coverage ratios. Challenges to segment ratios include geocoding error, outdated control totals, errors in the address frame, and systematic exclusion of types of addresses. We tested various net coverage ratios on segments selected for the 2015 Residential Energy Consumption Survey, and we share our results and recommendations for using net coverage ratios for estimating coverage for segments.

Researchers strive to design and implement high-quality surveys to maximize the utility of the data collected. The definitions of quality and usefulness, however, vary from survey to survey and depend on the analytic needs. Survey teams must evaluate the trade-offs of various decisions, such as when results are needed and their required level of precision, in addition to practical constraints like budget, before finalizing the design. Characteristics within the concept of fit for purpose (FfP) can provide the framework for considering the trade-offs. Furthermore, this tool can enable an evaluation of quality for the resulting estimates. Implementation of a FfP framework in this context, however, is not straightforward. In this article, we provide the reader with a glimpse of a FfP framework in action for obtaining estimates on early season influenza vaccination coverage estimates and on knowledge, attitudes, behaviors, and barriers related to influenza and influenza prevention among civilian noninstitutionalized adults aged 18 years and older in the United States. The result is the National Internet Flu Survey (NIFS), an annual, two-week internet survey sponsored by the US Centers for Disease Control and Prevention. In addition to critical design decisions, we use the established NIFS FfP framework to discuss the quality of the NIFS in meeting the intended objectives. We highlight aspects that work well and other survey traits requiring further evaluation. Differences found in comparing the NIFS to the National Flu Survey, the National Health Interview Survey, and Behavioral Risk Factor Surveillance System are discussed via their respective FfP characteristics. The findings presented here highlight the importance of the FfP framework for designing surveys, defining data quality, and providing a set a metrics used to advertise the intended use of the survey data and results.

Previous research has shown that increasing the size of incentives can increase response rates for probability-based, cross-sectional surveys. However, the effects of incentives on web panels have not been extensively studied. We sought to answer the question: What is the effect of larger, postpaid incentives on (1) response, (2) data quality, and (3) nonresponse bias for individuals in a web panel? We analyzed data from the 2015 and 2016 National Internet Flu Survey, a survey that uses the GfK KnowledgePanel® as its sampling frame. We compare panel members who received a postpaid, standard 1,000-point (the equivalent of US$1) incentive in 2015 to panelists who received a larger, 5,000-point (the equivalent of US$5) incentive in 2016. We found that larger incentives were associated with increased interview completion rates with minimal impact on data quality or bias.

While Big Data offers a potentially less expensive, less burdensome, and more timely alternative to survey data for producing a variety of statistics, it is not without error. The AAPOR Task Force on Big Data and others have called for researchers to evaluate the quality of Big Data using an approach similar to the total survey error (TSE) framework. However, differences in the construction of, access to, and overall data structure between survey data and Big Data make application of TSE difficult. In this article, we seek to develop the Total Error Framework (TEF), an extension of the TSE framework, to be (1) more inclusive and applicable to many types of Big Data, (2) comprehensive in that it considers “total” error, and (3) unified in that it allows researchers to compare errors in Big Data to errors in survey data. After outlining this framework, we then illustrate an application of TEF by comparing error in housing unit area (square footage) estimates collected in a survey (the 2015 Residential Energy Consumption Survey [RECS]) to those estimates found in three Big Data databases (Zillow.com, Acxiom, and CoreLogic).

Social media are becoming more popular as a source of data for social science researchers. These data are plentiful and offer the potential to answer new research questions at smaller geographies and for rarer subpopulations. When deciding whether to use data from social media, it is useful to learn as much as possible about the data and its source. Social media data have properties quite different from those with which many social scientists are used to working, so the assumptions often used to plan and manage a project may no longer hold. For example, social media data are so large that they may not be able to be processed on a single machine; they are in file formats with which many researchers are unfamiliar, and they require a level of data transformation and processing that has rarely been required when using more traditional data sources (e.g., survey data). Unfortunately, this type of information is often not obvious ahead of time as much of this knowledge is gained through word-of-mouth and experience. In this article, we attempt to document several challenges and opportunities encountered when working with Reddit, the self-proclaimed “front page of the Internet” and popular social media site. Specifically, we provide descriptive information about the Reddit site and its users, tips for using organic data from Reddit for social science research, some ideas for conducting a survey on Reddit, and lessons learned in merging survey responses with Reddit posts. While this article is specific to Reddit, researchers may also view it as a list of the type of information one may seek to acquire prior to conducting a project that uses any type of social media data.

As survey methods evolve, researchers require a comprehensive understanding of the error sources in their data. Comparative studies, which assess differences between the estimates from emerging survey methods and those from traditional surveys, are a popular tool for evaluating total error; however, they do not provide insight on the contributing error sources themselves. The Total Survey Error (TSE) framework is a natural fit for evaluations that examine survey error components across multiple data sources. In this article, we present a case study that demonstrates how the TSE framework can support both qualitative and quantitative evaluations comparing probability and nonprobability surveys. Our case study focuses on five internet panels that are intended to represent the US population and are used to measure health statistics. For these panels, we analyze the total survey error in two ways: (1) using a qualitative assessment that describes how panel construction and management methods may introduce error and (2) using a quantitative assessment that estimates and partitions the total error for two probability-based panels into coverage error and nonresponse error. This work can serve as a “proof of concept” for how the TSE framework may be applied to understand and compare the error structure of probability and nonprobability surveys. For those working specifically with internet panels, our findings will further provide an example of how researchers may choose the panel option best suited to their study aims and help vendors prioritize areas of improvement.

A major concern arising from ubiquitous tracking of individuals’ online activity is that algorithms may be trained to predict personal sensitive information, even for users who do not wish to reveal such information. Although previous research has shown that digital trace data can accurately predict sociodemographic characteristics, little is known about the potentials of such data to predict sensitive outcomes. Against this background, we investigate in this article whether we can accurately predict voting behavior, which is considered personal sensitive information in Germany and subject to strict privacy regulations. Using records of web browsing and mobile device usage of about 2,000 online users eligible to vote in the 2017 German federal election combined with survey data from the same individuals, we find that online activities do not predict (self-reported) voting well in this population. These findings add to the debate about users’ limited control over (inaccurate) personal information flows.

The Current Population Survey (CPS) is the source of official US labor force statistics. The wording of the CPS employment questions may not always cue respondents to include informal work in their responses, especially when providing proxy reports about other household members. In a survey experiment conducted using a sample of Amazon Mechanical Turk respondents, additional probing identified a substantial amount of informal work activity not captured by the CPS employment questions, both among those with no employment and among those categorized as employed based on answers to the CPS questions. Among respondents providing a proxy report for another household member, the share identifying additional work was systematically greater among those receiving a detailed probe that offered examples of types of informal work than among those receiving a simpler global probe. Similar differences between the effects of the detailed and the global probe were observed when respondents answered for themselves only among those who had already reported multiple jobs. The findings suggest that additional probing could improve estimates of employment and multiple job holding in the CPS and other household surveys, but that the nature of the probe is likely to be important.

Address-based sampling (ABS) refers to the use a list of addresses derived from the U.S. Postal Service’s Computerized Delivery Sequence File as a sampling frame. While most residential addresses are included on an ABS frame, it still suffers from undercoverage. Undercoverage is problematic only if the uncovered units have different attributes than covered units. In this article, we assess the risk of observing coverage bias on a variety of univariate point estimates in an in-person survey. We accomplish this task using a combination of survey data and Monte Carlo simulation models to vary the coverage rate and the target geography. For some variables, we found low risk of observing coverage bias. But, for others, the risk of observing bias was high for nearly all levels of coverage. Finally, we did not observe any trends by variable type that would give us the ability to predict a priori which variables may suffer bias.

Alumni tracking surveys are critical for evaluating and improving youth employment training programmes. However, there is little evidence about how to design these surveys, especially in less developed countries. We conducted short message service and Web surveys of alumni of a South African employment training programme. Short message service surveys returned higher response rates and more representative data than Web surveys, albeit at three times the cost. Combining modes increased data quality but may not be warranted, given the additional cost and complexity. This evidence suggests that training programmes can feasibly collect quality, actionable data on their alumni, even with constrained budgets and staff. Copyright © 2018 John Wiley & Sons, Ltd.

Background Influenza vaccination has been recommended for all persons aged ≥6 months since 2010. Methods Data from the 2016 National Internet Flu Survey were analyzed to assess provider vaccination recommendations and early influenza vaccination during the 2016–17 season among adults aged ≥18 years. Predictive marginals from a multivariable logistic regression model were used to identify factors independently associated with early vaccine uptake by provider vaccination recommendation status. Results Overall, 24.0% visited a provider who both recommended and offered influenza vaccination, 9.0% visited a provider who only recommended but did not offer, 25.1% visited a provider who neither recommended nor offered, and 41.9% did not visit a doctor from July 1 through date of interview. Adults who reported that a provider both recommended and offered vaccine had significantly higher vaccination coverage (66.6%) compared with those who reported that a provider only recommended but did not offer (48.4%), those who neither received recommendation nor offer (32.0%), and those who did not visit a doctor during the vaccination period (28.8%). Results of multivariable logistic regression indicated that having received a provider recommendation, with or without an offer for vaccination, was significantly associated with higher vaccination coverage after controlling for demographic and access-to-care factors. Conclusions Provider recommendation was significantly associated with influenza vaccination. However, overall, 67.0% of adults did not visit a doctor during the vaccination period or did visit a doctor but did not receive a provider recommendation. Evidence-based strategies such as client reminder/recall, standing orders, provider reminders, or health systems interventions in combination should be undertaken to improve provider recommendation and influenza vaccination coverage. Other factors significantly associated with a higher level of influenza vaccination included age ≥50 years, being Hispanic, having a college or higher education, having a usual place for medical care, and having public health insurance.

Background The Advisory Committee on Immunization Practices (ACIP) recommends all persons aged ≥6 months get vaccinated for influenza annually, placing particular emphasis on persons who are at increased risk for influenza-related complications and persons living with or caring for them. Methods Data from the 2016 National Internet Flu Survey (NIFS), a nationally representative, probability-based Internet panel survey of the noninstitutionalized U.S. civilian population aged ≥18 years, was used to compare influenza vaccination coverage among adults who live with household members at high-risk for complications from influenza with those who do not. Logistic regression was used to evaluate the difference in the adjusted vaccination coverage prevalence between persons living with and without high-risk household members. Results From the 2016 NIFS (n = 4,113), we estimated that 29.2% of noninstitutionalized U.S. adults had at least one household member at increased risk for influenza-related complications. Unadjusted influenza vaccination coverage was significantly higher for adults with a high-risk household member compared with those without (46.7% vs 38.6%, respectively). After adjustment for demographic and access-to-care factors, adults with high-risk household members were more likely to be vaccinated than those without (adjusted prevalence difference = 5.3 [0.3, 10.3]). Among vaccinated respondents with high-risk household members, 88.7% reported that protection of their family and close contacts was one of the reasons they were vaccinated. Conclusion Approximately half of adults living with someone at increased risk of complications from influenza did not report receiving an influenza vaccination. Vaccination reminder/recall for persons at increased risk should include reminders for their household contacts.

Surveys with samples selected from an address frame derived from US Postal Service sources are often referred to as address-based sampling (ABS) surveys. For an ABS survey that is primarily conducted by mail, web, or face-to-face, sometimes it is helpful to have a telephone number corresponding to the sample addresses for setting appointments or conducting nonresponse follow-up prompts. The usefulness of a telephone contact mode in a mixed mode ABS design depends on both the percentage of addresses for which telephone numbers can be appended (append rate or match rate) and the accuracy of the telephone numbers associated with addresses. Before planning a telephone contact as part of a mixed-mode study, the designer should know the likely effectiveness of the approach. This paper focuses primarily on append rate information, with a discussion of accuracy rates. For a single ABS frame, telephone match rates vary by geography, address type, match vendor, and by landline vs. cell telephone number. Using very large samples of addresses from a total US ABS frame, we estimated state and national telephone append rates from Marketing Systems Group's sources. The append rates are summarized here and interactively at the website http://abs.rti.org/atlas/.

As address-based sampling becomes increasingly popular for multimode surveys, researchers continue to refine data-collection best practices. While much work has been conducted to improve efficiency within a given mode, additional research is needed on how multimode designs can be optimized across modes. Previous research has not evaluated the consequences of mode sequencing on multimode mail and phone surveys, nor has significant research been conducted to evaluate mode sequencing on a variety of indicators beyond response rates. We conducted an experiment within the Racial and Ethnic Approaches to Community Health across the U.S. Risk Factor Survey (REACH U.S.) to evaluate two multimode case-flow designs: (1) phone followed by mail (phone-first) and (2) mail followed by phone (mail-first). We compared response rates, cost, timeliness, and data quality to identify differences across case-flow design. Because surveys often differ on the rarity of the target population, we also examined whether changes in the eligibility rate altered the choice of optimal case flow. Our results suggested that, on most metrics, the mail-first design was superior to the phone-first design. Compared with phone-first, mail-first achieved a higher yield rate at a lower cost with equivalent data quality. While the phone-first design initially achieved more interviews compared to the mail-first design, over time the mail-first design surpassed it and obtained the greatest number of interviews.

Survey researchers have recently begun using the U.S. Postal Service Delivery Sequence File (DSF) as a sampling frame. In addition to address, the DSF contains ancillary data regarding the status and structure of the address. These ancillary data may prove useful in further enhancing frame construction and sampling methodologies. We review three variables on the DSF: the vacancy flag, the college flag, and the drop point indicator. We conclude that the ancillary information available on the DSF may be used with caution. Despite the limitations, the flags may be used to enhance the current frame.

As researchers try to compensate for the growing undercoverage of landline RDD frames, alternative designs are increasingly being considered. One such alternative is address-based sampling (ABS). As such, much research is underway from a variety of organizations (Dekker and Murphy 2009; Link et al. 2008) to develop best practices for the use of ABS as a replacement or supplement to landline RDD frames. However, this research has lacked a fundamental component – matching telephone numbers to addresses. Because an accurate match is necessary to conduct interviews by telephone, the success (or lack thereof) in matching a telephone number to an address can dictate the data collection mode and have cost, response rate, and mode effect implications. We must therefore answer the question: What is a match? While this question may seem easy to answer – if you have an address and you have a telephone number, you have a match – it is not so simple. In this paper, we discuss different possible definitions of a “match.”

– This paper describes early findings of evaluations of the International Caries Detection and Assessment System (ICDAS) conducted by the Detroit Center for Research on Oral Health Disparities (DCR‐OHD). The lack of consistency among the contemporary criteria systems limits the comparability of outcomes measured in epidemiological and clinical studies. The ICDAS criteria were developed by an international team of caries researchers to integrate several new criteria systems into one standard system for caries detection and assessment. Using ICDAS in the DCR‐OHD cohort study, dental examiners first determined whether a clean and dry tooth surface is sound, sealed, restored, crowned, or missing. Afterwards, the examiners classified the carious status of each tooth surface using a seven‐point ordinal scale ranging from sound to extensive cavitation. Histological examination of extracted teeth found increased likelihood of carious demineralization in dentin as the ICDAS codes increased in severity. The criteria were also found to have discriminatory validity in analyses of social, behavioral and dietary factors associated with dental caries. The reliability of six examiners to classify tooth surfaces by their ICDAS carious status ranged between good to excellent (kappa coefficients ranged between 0.59 and 0.82). While further work is still needed to define caries activity, validate the criteria and their reliability in assessing dental caries on smooth surfaces, and develop a classification system for assessing preventive and restorative treatment needs, this early evaluation of the ICDAS platform has found that the system is practical; has content validity, correlational validity with histological examination of pits and fissures in extracted teeth; and discriminatory validity