Journal of survey statistics and methodology.
Journal of survey statistics and methodology.
- Washington, DC : American Association of Public Opinion, February 2021.
- 1-204 page ; 23 cm.
- V.9, No.1 .
Language Proficiency Among Respondents: Implications for Data Quality in a Longitudinal Face-To-Face Survey Alexander Wenz and others Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 73-93, https://doi.org/10.1093/jssam/smz045 Abstract When surveying immigrant populations or ethnic minority groups, it is important for survey researchers to consider that respondents might vary in their level of language proficiency. While survey translations might be offered, they are usually available for a limited number of languages, and even then, non-native speakers may not utilize questionnaires translated into their native language. This article examines the impact of language proficiency among respondents interviewed in English on survey data quality. We use data from Understanding Society: The United Kingdom Household Longitudinal Study (UKHLS) to examine five indicators of data quality, including "don't know" responding, primacy effects, straightlining in grids, nonresponse to a self-completion survey component, and change in response across survey waves. Respondents were asked whether they are native speakers of English; non-native speakers were subsequently asked to self-rate whether they have any difficulties speaking or reading English. Results suggest that non-native speakers provide lower data quality for four of the five quality indicators we examined. We find that non-native respondents have higher nonresponse rates to the self-completion section and are more likely to report change across waves, select the primary response option, and show straightlining response behavior in grids. Furthermore, primacy effects and nonresponse rates to the self-completion section vary by self-rated level of language proficiency. No significant effects were found with regard to "don't know" responding between native and non-native speakers. Optimal Response Formats for Online Surveys: Branch, Grid, or Single Item? Matthew Debell and others Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 1-24, https://doi.org/10.1093/jssam/smz039 Abstract This article reports the results of an experiment comparing branch, grid, and single-item question formats in an internet survey with a nationally representative probability sample. We compare the three formats in terms of administration time, item nonresponse, survey breakoff rates, response distribution, and criterion validity. On average, the grid format obtained the fastest answers, the single-item format was intermediate, and the branch format took the longest. Item nonresponse rates were lowest for the single-item format, intermediate for the grid, and highest for branching, but these results were not statistically significant when modeling the full experimental design. Survey breakoff rates among the formats are not statistically distinguishable. Criterion validity was weakest in the branching format, and there was no significant difference between the grid and single-item formats. This evidence indicates that the branching format is not well suited to internet data collection and that both single-item and short, well-constructed grids are better question formats. Population Size Estimation Using Multiple Respondent-Driven Sampling Surveys Brian J Kim and Mark S Handcock Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 94-120, https://doi.org/10.1093/jssam/smz055 Abstract Respondent-driven sampling (RDS) is commonly used to study hard-to-reach populations since traditional methods are unable to efficiently survey members due to the typically highly stigmatized nature of the population. The number of people in these populations is of primary global health and demographic interest and is usually hard to estimate. However, due to the nature of RDS, current methods of population size estimation are insufficient. We introduce a new method of estimating population size that uses concepts from capture-recapture methods while modeling RDS as a successive sampling process. We assess its statistical validity using information from the CDC's National HIV Behavioral Surveillance system in 2009 and 2012. Re-Examining the Middle Means Typical and the Left and Top Means First Heuristics Using Eye-Tracking Methodology Jan Karem Höhne and others Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 25-50, https://doi.org/10.1093/jssam/smz028 Abstract Web surveys are a common self-administered mode of data collection using written language to convey information. This language is usually accompanied by visual design elements, such as numbers, symbols, and graphics. As shown by previous research, such elements of survey questions can affect response behavior because respondents sometimes use interpretive heuristics, such as the "middle means typical" and the "left and top means first" heuristics when answering survey questions. In this study, we adopted the designs and survey questions of two experiments reported in Tourangeau, Couper, and Conrad (2004). One experiment varied the position of nonsubstantive response options in relation to other substantive response options and the second experiment varied the order of the response options. We implemented both experiments in an eye-tracking study. By recording respondents' eye movements, we are able to observe how they read question stems and response options and we are able to draw conclusions about the survey response process the questions initiate. This enables us to investigate the mechanisms underlying the two interpretive heuristics and to test the assumptions of Tourangeau et al. (2004) about the ways in which interpretive heuristics influence survey responding. The eye-tracking data reveal mixed results for the two interpretive heuristics. For the middle means typical heuristic, it remains somewhat unclear whether respondents seize on the conceptual or visual midpoint of a response scale when answering survey questions. For the left and top means first heuristic, we found that violations of the heuristic increase response effort in terms of eye fixations. These results are discussed in the context of the findings of the original studies. The Dynamics of "Neither Agree Nor Disagree" Answers in Attitudinal Questions Miriam Truebner Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 51-72, https://doi.org/10.1093/jssam/smz029 Abstract Attitudinal questions are an integral part of surveys in the social sciences. Previous research based on cross-sectional data has shown that both respondents' characteristics and questionnaire design can lead to higher use of midpoint responses when questions are operationalized on uneven rating scales. To further current understanding of this phenomenon, in this article we apply hybrid regression models to analyze differences between respondents but also developmental changes within respondents, allowing for a more profound interpretation of the dynamics behind midpoint responses, theoretically explained by satisficing behavior. For our midpoint analyses, we use a set of attitudinal item blocks asked in the British Household Panel Survey (BHPS) from 1991 to 2008. Respondents' reports and interviewers' judgments offer satisficing-related indicators regarding ability and motivation, enabling particularly accurate analyses of midpoint response behavior in terms of the "neither agree nor disagree" option. Results show that depending on respondents' specific characteristics, higher use of the midpoint cannot automatically be equated with nonsubstantive response behavior. Furthermore, we demonstrate that specific indicators related to low ability and motivation do not uniformly increase midpoint responses. Application of the hybrid model reveals that changes in respondents' characteristics do not affect midpoint response. We speculate that there are unobserved personality attributes that affect the propensity to midpoint response and conclude by reflecting on reasons for the increase in midpoint responses over the years. The Impact of Nonsampling Errors on Estimators of Catch from Electronic Reporting Systems S Lynne Stokes and others Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 159-184 https://doi.org/10.1093/jssam/smz042 Abstract The National Marine Fisheries Service is the agency charged with estimating the number of fish removed from US oceans by recreational anglers. Two surveys produce data for these estimates; one measures number of angler trips and the other fish caught per trip by species, geography, and time period. Both surveys collect data from people and have the usual sources of nonsampling error afflicting demographic surveys. Due to lack of accessibility to fishing sites, they also have additional sources such as undercoverage. The rare incidence of saltwater fishing among the general population makes the current method costly and time-consuming. Consequently, new ways of obtaining information to supplement or replace the current method are of interest. One is the electronic logbook (ELB); this approach allows anglers to self-report their catch using cellphones or other communication devices. Estimation of catch from these data are possible with estimators using capture-recapture methods, but new sources of nonsampling error are rising. In this article, we examine three sources of nonsampling error in estimators of catch and approximate their biasing effect. We illustrate the method by comparing the effects using data from an ELB study in the Gulf of Mexico. Tools for Selecting Working Correlation Structures When Using Weighted GEE to Model Longitudinal Survey Data Philip M Westgate and Brady T West Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 141-158, https://doi.org/10.1093/jssam/smz048 Abstract Weighted generalized estimating equations (GEEs) are popular for the marginal analysis of longitudinal survey data. This popularity is due to the ability of these estimating equations to provide consistent regression parameter estimates and corresponding standard error estimates as long as the population mean and survey weights are correctly specified. Although the data analyst must incorporate a working correlation structure within the weighted GEEs, this structure need not be correctly specified. However, accurate modeling of this structure has the potential to improve regression parameter estimation (i.e., reduce standard errors) and therefore, the selection of a working correlation structure for use within GEEs has received considerable attention in standard longitudinal data analysis settings. In this article, we describe how correlation selection criteria can be extended for use with weighted GEE in the context of analyzing longitudinal survey data. Importantly, we provide and demonstrate an R function that we have created for such analyses. Furthermore, we discuss correlation selection in the context of using existing software that does not have this explicit capability. The methods are demonstrated via the use of data from a real survey in which we are interested in the mean number of falls that elderly individuals in a specific subpopulation experience over time.
2325-0984
Language Proficiency Among Respondents: Implications for Data Quality in a Longitudinal Face-To-Face Survey Alexander Wenz and others Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 73-93, https://doi.org/10.1093/jssam/smz045 Abstract When surveying immigrant populations or ethnic minority groups, it is important for survey researchers to consider that respondents might vary in their level of language proficiency. While survey translations might be offered, they are usually available for a limited number of languages, and even then, non-native speakers may not utilize questionnaires translated into their native language. This article examines the impact of language proficiency among respondents interviewed in English on survey data quality. We use data from Understanding Society: The United Kingdom Household Longitudinal Study (UKHLS) to examine five indicators of data quality, including "don't know" responding, primacy effects, straightlining in grids, nonresponse to a self-completion survey component, and change in response across survey waves. Respondents were asked whether they are native speakers of English; non-native speakers were subsequently asked to self-rate whether they have any difficulties speaking or reading English. Results suggest that non-native speakers provide lower data quality for four of the five quality indicators we examined. We find that non-native respondents have higher nonresponse rates to the self-completion section and are more likely to report change across waves, select the primary response option, and show straightlining response behavior in grids. Furthermore, primacy effects and nonresponse rates to the self-completion section vary by self-rated level of language proficiency. No significant effects were found with regard to "don't know" responding between native and non-native speakers. Optimal Response Formats for Online Surveys: Branch, Grid, or Single Item? Matthew Debell and others Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 1-24, https://doi.org/10.1093/jssam/smz039 Abstract This article reports the results of an experiment comparing branch, grid, and single-item question formats in an internet survey with a nationally representative probability sample. We compare the three formats in terms of administration time, item nonresponse, survey breakoff rates, response distribution, and criterion validity. On average, the grid format obtained the fastest answers, the single-item format was intermediate, and the branch format took the longest. Item nonresponse rates were lowest for the single-item format, intermediate for the grid, and highest for branching, but these results were not statistically significant when modeling the full experimental design. Survey breakoff rates among the formats are not statistically distinguishable. Criterion validity was weakest in the branching format, and there was no significant difference between the grid and single-item formats. This evidence indicates that the branching format is not well suited to internet data collection and that both single-item and short, well-constructed grids are better question formats. Population Size Estimation Using Multiple Respondent-Driven Sampling Surveys Brian J Kim and Mark S Handcock Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 94-120, https://doi.org/10.1093/jssam/smz055 Abstract Respondent-driven sampling (RDS) is commonly used to study hard-to-reach populations since traditional methods are unable to efficiently survey members due to the typically highly stigmatized nature of the population. The number of people in these populations is of primary global health and demographic interest and is usually hard to estimate. However, due to the nature of RDS, current methods of population size estimation are insufficient. We introduce a new method of estimating population size that uses concepts from capture-recapture methods while modeling RDS as a successive sampling process. We assess its statistical validity using information from the CDC's National HIV Behavioral Surveillance system in 2009 and 2012. Re-Examining the Middle Means Typical and the Left and Top Means First Heuristics Using Eye-Tracking Methodology Jan Karem Höhne and others Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 25-50, https://doi.org/10.1093/jssam/smz028 Abstract Web surveys are a common self-administered mode of data collection using written language to convey information. This language is usually accompanied by visual design elements, such as numbers, symbols, and graphics. As shown by previous research, such elements of survey questions can affect response behavior because respondents sometimes use interpretive heuristics, such as the "middle means typical" and the "left and top means first" heuristics when answering survey questions. In this study, we adopted the designs and survey questions of two experiments reported in Tourangeau, Couper, and Conrad (2004). One experiment varied the position of nonsubstantive response options in relation to other substantive response options and the second experiment varied the order of the response options. We implemented both experiments in an eye-tracking study. By recording respondents' eye movements, we are able to observe how they read question stems and response options and we are able to draw conclusions about the survey response process the questions initiate. This enables us to investigate the mechanisms underlying the two interpretive heuristics and to test the assumptions of Tourangeau et al. (2004) about the ways in which interpretive heuristics influence survey responding. The eye-tracking data reveal mixed results for the two interpretive heuristics. For the middle means typical heuristic, it remains somewhat unclear whether respondents seize on the conceptual or visual midpoint of a response scale when answering survey questions. For the left and top means first heuristic, we found that violations of the heuristic increase response effort in terms of eye fixations. These results are discussed in the context of the findings of the original studies. The Dynamics of "Neither Agree Nor Disagree" Answers in Attitudinal Questions Miriam Truebner Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 51-72, https://doi.org/10.1093/jssam/smz029 Abstract Attitudinal questions are an integral part of surveys in the social sciences. Previous research based on cross-sectional data has shown that both respondents' characteristics and questionnaire design can lead to higher use of midpoint responses when questions are operationalized on uneven rating scales. To further current understanding of this phenomenon, in this article we apply hybrid regression models to analyze differences between respondents but also developmental changes within respondents, allowing for a more profound interpretation of the dynamics behind midpoint responses, theoretically explained by satisficing behavior. For our midpoint analyses, we use a set of attitudinal item blocks asked in the British Household Panel Survey (BHPS) from 1991 to 2008. Respondents' reports and interviewers' judgments offer satisficing-related indicators regarding ability and motivation, enabling particularly accurate analyses of midpoint response behavior in terms of the "neither agree nor disagree" option. Results show that depending on respondents' specific characteristics, higher use of the midpoint cannot automatically be equated with nonsubstantive response behavior. Furthermore, we demonstrate that specific indicators related to low ability and motivation do not uniformly increase midpoint responses. Application of the hybrid model reveals that changes in respondents' characteristics do not affect midpoint response. We speculate that there are unobserved personality attributes that affect the propensity to midpoint response and conclude by reflecting on reasons for the increase in midpoint responses over the years. The Impact of Nonsampling Errors on Estimators of Catch from Electronic Reporting Systems S Lynne Stokes and others Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 159-184 https://doi.org/10.1093/jssam/smz042 Abstract The National Marine Fisheries Service is the agency charged with estimating the number of fish removed from US oceans by recreational anglers. Two surveys produce data for these estimates; one measures number of angler trips and the other fish caught per trip by species, geography, and time period. Both surveys collect data from people and have the usual sources of nonsampling error afflicting demographic surveys. Due to lack of accessibility to fishing sites, they also have additional sources such as undercoverage. The rare incidence of saltwater fishing among the general population makes the current method costly and time-consuming. Consequently, new ways of obtaining information to supplement or replace the current method are of interest. One is the electronic logbook (ELB); this approach allows anglers to self-report their catch using cellphones or other communication devices. Estimation of catch from these data are possible with estimators using capture-recapture methods, but new sources of nonsampling error are rising. In this article, we examine three sources of nonsampling error in estimators of catch and approximate their biasing effect. We illustrate the method by comparing the effects using data from an ELB study in the Gulf of Mexico. Tools for Selecting Working Correlation Structures When Using Weighted GEE to Model Longitudinal Survey Data Philip M Westgate and Brady T West Journal of Survey Statistics and Methodology, Volume 9, Issue 1, February 2021, Pages 141-158, https://doi.org/10.1093/jssam/smz048 Abstract Weighted generalized estimating equations (GEEs) are popular for the marginal analysis of longitudinal survey data. This popularity is due to the ability of these estimating equations to provide consistent regression parameter estimates and corresponding standard error estimates as long as the population mean and survey weights are correctly specified. Although the data analyst must incorporate a working correlation structure within the weighted GEEs, this structure need not be correctly specified. However, accurate modeling of this structure has the potential to improve regression parameter estimation (i.e., reduce standard errors) and therefore, the selection of a working correlation structure for use within GEEs has received considerable attention in standard longitudinal data analysis settings. In this article, we describe how correlation selection criteria can be extended for use with weighted GEE in the context of analyzing longitudinal survey data. Importantly, we provide and demonstrate an R function that we have created for such analyses. Furthermore, we discuss correlation selection in the context of using existing software that does not have this explicit capability. The methods are demonstrated via the use of data from a real survey in which we are interested in the mean number of falls that elderly individuals in a specific subpopulation experience over time.
2325-0984