Evaluating the Use of a Two-Step Gender Identity Measure in the 2018 General Social Survey

Abstract

In 2018, the General Social Survey (GSS) asked some respondents for their sex assigned at birth and current gender identity, in addition to the ongoing practice of having survey interviewers code respondent sex. Between 0.44% and 0.93% of the respondents who were surveyed identified as transgender, identified with a gender that does not conventionally correspond to the sex they were assigned at birth, or identified the sex they were assigned at birth inconsistently with the interviewer's assessment of respondent sex. These results corroborate previous estimates of the transgender population size in the United States. Furthermore, the implementation of these new questions mirrors the successful inclusion of other small populations represented in the GSS, such as lesbian, gay, and bisexual people, as well as Muslims, Buddhists, and Hindus. Data on transgender and gender-nonconforming populations can be pooled together over time to assess these populations' attitudes, beliefs, behaviors, and social inequality patterns. We identified inconsistencies between interviewer-coded sex, self-reported sex, and gender identity. As with the coding of race in the GSS, interviewer-coded assessments can mismatch respondents' self-reported identification. Our findings underscore the importance of continuing to ask respondents to self-report gender identity separately from sex assigned at birth in the GSS and other surveys.

Introduction

Sexual and gender minorities remain understudied in nationally representative health and social science research (Compton 2015; Institute of Medicine 2011). In 2008, the General Social Survey (GSS) began to ask respondents about their sexual orientation in addition to existing questions on sexual experiences (Gates 2010). These methodological improvements created one of the first nationally representative samples of lesbian, gay, and bisexual (LGB) respondents, as well as heterosexual respondents who reported same-sex sexual behavior. However, data collection on gender minorities continues to be sparse, especially in survey research (Schilt and Lagos 2017). In 2018, the GSS began to ask respondents about their gender identity and sex assigned at birth at the end of its self-administered questionnaire (SAQ) (Smith and Son 2019). This has resulted in one of the first publicly available nationally representative survey samples of the United States that measures transgender identity. These new data can contribute significantly to expanding what is known about patterns and changes in social characteristics among U.S. gender minorities.

Since 1972, the GSS has collected data on opinions, attitudes, experiences, and behavior of the U.S. population in order to generate historical records of change over time (Smith et al. 2019). Across its history, the GSS has added and dropped questions in response to user requests and proposals, funding availability, and time constraints (Davis and Smith 1991). Some questions, such as those on sexual orientation, have persisted as long-term indicators of vital demographic information that may have not been considered in earlier years (Thomeer and Reczek 2016). The questions on gender identity and sex assigned at birth in the 2018 GSS could form another key demographic indicator that provides value if collected for a sustained period of time. In this study, we evaluate initial findings on gender minorities obtained from the 2018 GSS, assess the benefits of the continued use of these questions, and suggest possible improvements in data collection methodology.

The Utility of Measuring Small Populations in the GSS

In discussions proposing or presenting greater inclusion of lesbian, gay, bisexual, transgender, and queer populations in the social sciences, the relatively small size of these populations can often be a source of concern, especially in terms of sampling frames and research design (Compton 2018). Because the current sample size in the GSS is about 2,400 observations, the number of gender minorities in the sample will likely be too small to make yearly assessments about these populations. However, several questions in the GSS produce valuable information on relatively small but important populations over a sustained multiyear data collection effort. For instance, the combined proportion of respondents who are Muslims, Buddhists, Hindus, and religious “others” started at 0.8% of the GSS sample in 1973–1980 (Smith 2003) but increased to 3.34% of the sample by 2018 (Smith et al. 2019). Even if a population does not increase in terms of proportion of the population, pooling samples over a number of years can increase the number of observations in absolute terms and eventually enable statistically significant findings (Gorman et al. 2015). Aggregated samples from multiple years have informed generalizations about these small populations, such as patterns of educational attainment by Buddhists, Muslims, and Jews (Sander 2010), as well as the social integration of Muslims, Buddhists, and Hindus (Wuthnow and Hackett 2003).

Sexual minority respondents are also a relatively small population whose observations in the GSS have produced informative data and valuable findings. In the 2008 GSS, 1.8% of women identified as lesbian, and 1.5% of men identified as gay; 1.5% of women and 0.7% of men identified as bisexual. Despite the small LGB sample in the GSS, these data have been used to form generalizations about sexual minority populations, ranging from the effects of being out in the workplace (Gates 2010) to overall happiness (Thomeer and Reczek 2016) and beliefs about suicide (Blosnich et al. 2018). If the GSS continues to identify gender identity and sex assigned at birth in subsequent administrations, there will eventually be enough cases to produce high-quality population-level estimates of similar social patterns among gender minority populations. Just as more U.S. residents have come identify as lesbian, gay, and bisexual in recent cohorts (England et al. 2016; Mishel et al. 2020), the number of GSS respondents who identify as transgender and gender nonconforming may also increase in coming years.

Data and Methods

This study uses data from the 2018 administration of the GSS. In 2017, we collaborated with Kristen Schilt to submit a formal proposal to the GSS, alongside other, similar proposals related to gender identity by other researchers (Carian 2019). In these proposals, we advocated for a two-step gender measure that asks for respondent's sex as indicated on their birth certificate in one question and then asks respondents to identify their gender identity in a follow-up question. The two-step gender identity measure has shown to be a valid and cognitively feasible measure of the difference between sex assigned at birth and gender identity for both cisgender and transgender respondents (Lombardi and Banik 2016; Reisner et al. 2014; Reisner et al. 2015). Pointing to previously published studies and other data sets, we provided evidence to the members of the GSS Board of Overseers that the two-step measure was valid and could be reliable. These proposals for the two-step gender identity measure were accepted after peer review by the GSS and have now yielded one of the first publicly available nationally representative data sets that accounts for transgender identity.

These questions were included in the SAQ, which is administered to only 1,409 of 2,348 respondents (60% of the overall sample), in a section designated for sensitive questions about sexual orientation, sexual behavior, and drug use (Smith and Son 2019). The SAQ module is introduced through the following language:

At the end of this section, respondents in 2018 were asked the following question to indicate respondent sex: “What sex were you assigned at birth? (For example, on your birth certificate.)” Respondents were asked to select from three potential answers: male, female, or intersex. Of the 1,409 SAQ respondents, 1,397 provided answers to this question. Next, respondents were asked the following question about their gender identity at the time of the interview: “What is your current gender?” Respondents were asked to select from four potential answer categories—woman, man, transgender, or “a gender not listed here”—and were given space to provide a more specific gender identity. Of the 1,409 total SAQ observations, 1,401 replied to this question. Both questions had response rates of more than 99%, indicating that the questions were intelligible and accessible to respondents receiving this new module.

These two new measures can be juxtaposed with the original measure of respondent sex, in which the survey interviewer designates respondents as either male or female. This third measure, which was included in the 2018 GSS alongside the new measures, is highly sensitive to interviewer effects and inconsistencies between interviewer coding and self-identification (Riley et al. 2017). Inconsistencies between interviewer-coded and self-reported demographic characteristics are not new sites of measurement error in the GSS specific to the inclusion of transgender identity measurements. Interviewers have also coded respondents' race and ethnicity in ways that do not match the self-reported identities of respondents (Saperstein 2006). Although the new two-step measure addresses the shortcomings of the original measure, inconsistencies between self-identification and interviewer-coded sex and gender can nevertheless serve as imperfect but informative proxies for how respondents are perceived by others when the results of interviewer-coded identification are accompanied by self-reported demographic information (Lagos 2019).

Preliminary Findings

Table 1 displays the unweighted frequencies of responses to the GSS gender identity question broken down by gender identity (assessed through the question “What is your current gender?”) and sex assigned at birth (assessed through the question “What sex were you assigned at birth? [For example, on your birth certificate]”). Of the 1,397 respondents who answered the question about sex assignment at birth, 7 (0.50% of the sample) reported identifying as transgender or identifying with a gender that does not conventionally correspond to the sex they were assigned at birth. These overall frequencies are consistent with the prevalence estimates of the U.S. transgender population, at 0.4% to 0.6% of the adult population (Flores et al. 2016; Meerwijk and Sevelius 2017). Although weighted estimates from these 2018 GSS data suggest a slightly higher prevalence of about 0.85%, there is some degree of uncertainty of how weights ought to be applied to prevalence estimates of transgender populations (Henderson et al. 2019). With this in mind, we note the consistency between the raw frequencies and other population estimates simply to suggest a broad correspondence between the overall frequency of transgender and gender-nonconforming respondents in an otherwise nationally representative survey and prevailing population estimates.

In addition to identifying transgender respondents, this survey also identified a respondent who identified as a man but was assigned intersex at birth, as well as a respondent who identified with “a gender not listed here” and was assigned female at birth. Because intersex assignment at birth and identifying with “a gender not listed here” in the survey are not necessarily indicators of transgender identity, we do not include this respondent in our total count of transgender respondents. Furthermore, many, if not most, intersex people are assigned male or female at birth rather than intersex (Davis et al. 2015). As such, we urge caution in drawing conclusions from the intersex category. Accurately including intersex members of the population in surveys like the GSS would likely require a separate question about intersex status that is not tied to sex assignment at birth.

While gender identity and sex assigned at birth are quite valuable for understanding gender, external perceptions of sex have been demonstrated to have implications for overall self-rated health (Lagos 2019). Table 2 presents the unweighted frequencies of responses to the GSS gender identity question broken down by interviewer-coded respondent sex. Because GSS survey interviews are conducted in person, this measure indicates the results of an initial in-person judgment of respondents' sex. These judgments could be based on visual or auditory cues, or on names of respondents. A comparison of self-reported gender identity and interviewer-coded assessments of respondent sex reveals that 0.79% of respondents were classified by the survey interviewer inconsistently with respondents' current gender identities. Table 3 compares the unweighted survey interview ratings with sex assigned at birth. Here, 0.93% of respondents were classified by the survey interviewer inconsistently with the sex reported on their birth certificate. In both cases, the frequencies of discord between survey interview ratings and respondents' gender identities and sex assigned at birth are higher than the frequencies of respondents identifying as transgender, suggesting that survey interviewers also might misclassify respondents who are not transgender.

Discussion

In their first implementation, the additional GSS questions regarding sex assigned at birth and self-reported gender have already provided valuable information useful for identifying gender minorities in the U.S. population. Although the sample of transgender respondents in the 2018 survey is small on its own, the ability to pool small samples over a sustained multiyear period of data collection will increase the reliability and utility of the measures. Past analyses of other small groups, such as religious minority groups, suggest that this effort holds promise. Furthermore, as demonstrated in the cases of sexual minorities and many religious minority groups, the size of gender minority populations relative to the general population could grow or change significantly over time. Continuing to use these two questions will facilitate a more accurate and nuanced accounting of gender diversity in the United States and provide an invaluable account of changes in this diversity over time.

Beyond its utility in tracking and understanding gender minority populations, the cross-validation of the interviewer ratings of respondent sex with a two-step gender identity measure in the 2018 GSS also offers a major methodological insight that can help inform data collection practices on gender and reduce the risk of survey error. The two-step gender identity measure identifies instances in which the GSS interviewer classified a respondent inconsistently with the respondent's own understanding of their gender identity, as well as with a respondent's sex as recorded on their birth certificate. Before the GSS implemented the two-step gender identity measure, interviewer-rated sex was the only available measure of respondent sex or gender, implying that past classifications of respondent sex and gender reflected survey error. The two-step gender identity measure is a valuable check on interviewer effects for all respondents, whether transgender or not, given that cisgender people are also susceptible to gender misclassification (Devor 1989).

With this in mind, the mismatch between interviewer appraisals of sex and respondents' self-reports of gender and sex may serve as an informative proxy for the relationship between external gender perception and gender identity (Lagos 2019). In addition to collecting self-reported sex assigned at birth and gender identity, surveys may find ways to involve interviewers in assessing other informative perception-based respondent characteristics related to sex and gender. In 2012, GSS interviewers began coding respondents' skin tone. Although these measures are also sensitive to interviewer effects (Hannon and Defina 2016), some elements of external assessment may prove important to understanding how characteristics such as sex and gender are experienced through reflected appraisal, as has become evident with race and ethnicity (Hunter 2002; Monk 2014, 2015). Measurements of interviewer perceptions are imperfect, but this paradigm shift in measurement could acknowledge some degree of subjectivity in interviewer assumptions. Researchers can then cross-validate these interviewer perceptions with respondents' self-reports to more carefully assess the multiple dimensions of identities such as gender and race/ethnicity.

Based on our findings, we recommend that the GSS continue to field these questions to respondents in future administrations of the survey and potentially include an additional question to measure whether respondents are intersex in order to more accurately measure such populations. Furthermore, we recommend that the two-step measure be included as part of the core survey, separate from the SAQ, so that it can be asked of the complete sample and yield a larger overall number of responses, decoupled from questions on HIV/AIDS risk.

Another potential modification to improve data collection on gender identity would be to incorporate a recommendation made by the Federal Interagency Working Group on Improving Measurement of Sexual Orientation and Gender Identity in Federal Surveys. This group recommended asking respondents to confirm their selection of sexual and gender identity if they reported a gender identity that does not conventionally correspond to the sex they were assigned at birth, using the following language:

This approach ought to address many concerns about the vulnerability of this question to survey error (Smith and Son 2019). However, we recommend posing this clarifying question to all respondents, and not just to those who indicate a gender that does not traditionally correspond to the sex assigned at birth, in order to capture a more complete range of survey errors.

Decades of gender theory and empirical research on gender have established that everyone has a gender identity in one way or another that ought to be examined critically and with greater precision (Hart et al. 2019; Schilt and Westbrook 2009; West and Zimmerman 1987). Widespread implementation of the two-step method for measuring gender identity in future administrations of the GSS and other surveys would expand the capacity of survey-based research to address the salience of both sex and gender at the population level.

Acknowledgments

We thank the reviewers and editors, and thank Kristen Schilt for her vital role in our proposal for a two-step gender identity measure for the 2018 GSS. This research was supported by grant P2CHD042849 to the Population Research Center and grant T32HD007081 to the Training Program in Population Studies, awarded to the Population Research Center at The University of Texas at Austin by the Eunice Kennedy Shriver National Institute of Child Health and Human Development. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

References