A Research Note on Community Resilience Estimates: New U.S. Census Bureau Data With an Application to Excess Deaths From COVID-19

Abstract

In this research note, we describe the results of the first validation study of the U.S. Census Bureau's new Community Resilience Estimates (CRE), which uses Census microdata to develop a tract-level vulnerability index for the United States. By employing administrative microdata to link Social Security Administration mortality records to CRE, we show that CRE quartiles provide more stable predictions of COVID-19 excess deaths than single demographic categorizations such as race or age, as well as other vulnerability measures including the U.S. Centers for Disease Control and Prevention's Social Vulnerability Index (SVI) and the Federal Emergency Management Agency's National Risk Index (NRI). We also use machine learning techniques to show that CRE provides more predictive power of COVID-19 excess deaths than standard socioeconomic predictors of vulnerability such as poverty and unemployment, as well as SVI and NRI. We find that a 10-percentage-point increase in a key CRE risk measure is associated with one additional death per neighborhood during the initial outbreak of COVID-19 in the United States. We conclude that, compared with alternative measures, CRE provides a more accurate predictor of community vulnerability to a disaster such as a pandemic.

Introduction

Vulnerability, or one's ability to cope with external stressors with minimal harm, is unequally distributed across time and space (Cutter and Finch 2008). Vulnerability is also closely related to community resilience, which is the capacity of individuals and households in a community to absorb the impacts of a disaster (Masterson et al. 2014). While vulnerability and resilience are conventionally separated into different bodies of thought, they represent two sides of the same resilience coin (Summers et al. 2017). For example, individual socioeconomic factors contributing to increased vulnerability, such as quality shelter and health care access, also contribute to one's resilience to absorb the impacts of a disaster. A community's vulnerability to a disaster is thus a critical aspect of a community's resilience to a disaster. A simple measure of resilience (or vulnerability) that ranges from the most vulnerable to the most resilient is valuable to policymakers.

While there are individual socioeconomic factors, such as education, income, and occupation, that contribute to aspects of resiliency and vulnerability (Kaplan 1999; Shavers 2007), to obtain a single measure of resiliency/vulnerability, indices have generally been created by aggregating individual socioeconomic factors (Cutter 1996; Cutter et al. 2008; Cutter and Derakhshan 2018). For example, the U.S. Centers for Disease Control and Prevention's Social Vulnerability Index (SVI) and the Federal Emergency Management Agency's National Risk Index (NRI) are constructed by aggregating survey estimates. An important limitation of this approach is that these indices are developed without consideration of survey sampling error because they treat survey estimates subject to sampling error as true parameter values. So, for example, a place with an estimated poverty rate of 100% (±3%) would be treated as having a higher poverty vulnerability indicator than a place with an estimated poverty rate of 97% (±3%), despite the two places not being statistically different. When comparing indices that treat survey estimates as true parameter values, researchers find a high magnitude of uncertainty, and the precision of indices decreases as the predicted vulnerability of a community increases (Tate 2012). Despite these known problems, the construction of an index at a set geographic level based on survey estimates that are treated as true parameter values continues to be the predominant method, which can lead to inconsistent validation outcomes (Derakhshan, Blackwood et al. 2022).

Existing methods of measuring vulnerability rely on combining publicly available demographic indicators that each have their own distribution of sampling errors, and this process reduces the accuracy of estimates (Willyard et al. 2022). Robust and credible measures of vulnerability are needed (Adger 2006). As a result, the U.S. Census Bureau has launched a new data product for national agencies and local communities: the Community Resilience Estimates (CRE) Program population estimates. CRE population estimates are produced using information on individuals and households from the 2019 American Community Survey (ACS) and the Census Bureau's Population Estimates Program (PEP), and employ established U.S. Census small area modeling techniques. The CRE Program tracks how at risk every tract in the United States is to the impacts of COVID-19 and other local disasters by measuring the capacity of individuals and households to absorb, endure, and recover from the impacts of a disaster. Specifically, the CRE is based on individual- and household-level risk factors (detailed below) to produce aggregate-level (tract, county, or state) small area estimates. The CRE provides an estimate of the number of people with a specific number of risks.¹ It thus represents two major improvements over existing resiliency/vulnerability measures: (1) the CRE reduces sampling error by developing resiliency estimates directly from data, rather than by treating survey estimates subject to sampling error as true parameter values; and (2) because the CRE does not rely on already aggregated data, it maintains the ability to measure the distribution of vulnerability among individuals.² That is, two geographies may appear to have the same number of risk factors in public data, but only with microdata (like that used to construct the CRE) can we create a measure that takes an intersectional (and distributional) approach to risk. However, CRE has yet to be validated relative to other common measures of vulnerability, such as individual socioeconomic measures like poverty, or other vulnerability indices such as the SVI or NRI. This article seeks to provide such a validation test using the COVID-19 pandemic as a test disaster. The COVID-19 pandemic is a useful test case because its negative effects have been unequally distributed across geography and show a strong relationship to individual and household characteristics. However, quantifying this relationship, especially at finer geographic granularity, has been difficult with public data.

In this research note, we first review existing literature on the importance of social vulnerability and community resilience, including details on existing measures and data. Then we detail how the CRE Program develops its estimates from underlying U.S. Census Bureau microdata. Next, we develop an empirical application in which we link the CRE Program's underlying microdata to U.S. Social Security Administration microdata to test the relationship between the CRE and excess deaths from COVID-19. We show that CRE quartiles provide a more stable predictor of excess deaths than race, age, SVI, and NRI. We then use lasso machine learning to compare the predictive power of socioeconomic indicators of vulnerability and SVI, NRI, and CRE. We find that CRE provides more predictive power than these commonly used alternatives. We conclude with a brief discussion of the value of the CRE Program to community organizers and disaster management and argue that the CRE will provide a valuable resource to researchers to examine a wide variety of disasters and perhaps other applications, such as public health planning.

Alternative Measures of Social Vulnerability and Community Resilience

We briefly highlight the literature on the related concepts of resiliency and vulnerability (Folke 2006), especially in connection to disasters such as hurricanes (Finch et al. 2010; Gotham and Campanella 2011) and the COVID-19 pandemic (Kimhi et al. 2020).³ Various institutional, social, economic, infrastructural, and demographic factors have been found to affect the resilience of communities to pandemics (Suleimany et al. 2022), hurricanes (Almutairi et al. 2020), and other disasters (Cutter et al. 2008). Because individual-level microdata are not readily available, many researchers take a place-based approach to measure vulnerability and resilience by choosing a geographic level of analysis and breaking up the concept into various domains and indicators. For example, SVI is either a tract-level or county-level index made up of more than 15 indicators and four domains: socioeconomic status, household characteristics, racial and ethnic minority status, and housing type and transportation (Flanagan et al. 2011). The University of South Carolina Hazards and Vulnerability Research Institute's Baseline Resilience Indicators for Communities (BRIC) is made up of 49 indicators representing six domains measured at the county level: social, economic, community capital, institutional capacity, housing/infrastructure, and environmental (Cutter et al. 2014). NRI further expands upon SVI and BRIC by multiplying the expected annual losses for natural disasters by a community risk factor, where the community risk factor is the SVI overall percentile ranking score divided by the BRIC overall percentile ranking score. While such a measure may be less relevant in a pandemic that disproportionately affected populations rather than physical infrastructure, SVI and NRI are commonly used measures of resiliency/vulnerability (Derakhshan, Emrich et al. 2022), so we include both in our comparisons.

Indeed, indices have been shown to be valuable for community organizers, disaster managers, and researchers (Cutter et al. 2003; Sawyer et al. 2022). A review of this literature demonstrates the extent to which resilience is inequitably distributed (Peacock et al. 1997) and spatially segmented (Logan et al. 2016) by demographic groups. There is substantial agreement in the literature on how resiliency and vulnerability vary by age, gender, race, ethnicity, and socioeconomic status (Cutter et al. 2003; Elliott and Pais 2006), and these characteristics overlap with locational vulnerability (Logan et al. 2016). Therefore, existing measures such as the SVI and NRI generally include a similar selection of demography and socioeconomic variables in their construction.

While the U.S. Census Bureau's new CRE Program's selection of underlying vulnerability measures differs slightly relative to the SVI and NRI,⁴ the CRE's main improvement on these measures lies in how the survey microdata produce estimates directly rather than indirectly from already aggregated estimates with their own respective margins of error. More specifically, since many indices do not incorporate sampling error, they do not reflect the statistical intricacies of survey data in their estimates. In addition, indices that do not produce margins of error are not as practically useful as those that do. Without the production of margins of error along with estimates, a statistically significant difference between places or across time cannot be determined.

Intersectional approaches to vulnerability and resilience highlight how vulnerability is the result of different and interdependent marginalization processes (Kuran et al. 2020; Stanczyk 2020). In addition to the statistical issues associated with using estimates to develop indices, indices that rely on already aggregated publicly available data lose the ability to measure the distribution of vulnerability among individuals. That is, two geographies may appear to have the same number of risk factors in public data, but only with microdata (like that used to construct the CRE) can we create a measure that takes an intersectional (and distributional) approach to risk. Vulnerability factors are distributed nonrandomly across demography and geography (Logan et al. 2016; Peacock et al. 1997). Thus, a simple geographic count of vulnerable persons from aggregate data will miss the fact that some geographies have individuals facing multiple risk factors. An index built on disaggregated microdata can account for this distribution and thereby gain predictive power. In other words, the distribution of risk factors within a geography, rather than the simple count of risk factors within a geography, improves the estimation of local resilience.

The CRE

To implement such an index, the CRE is created by first identifying vulnerability indicators within ACS microdata. The following 10 vulnerability indicators are used in 2019 CRE: (1) households with an income-to-poverty ratio less than 130%; (2) less than one individual living in the household is aged 18–64; (3) household crowding, defined as more than 0.75 persons per room; (4) households with limited education, defined as having no one older than 16 with a high school diploma or having limited English speaking; (5) no one in the household is employed full-time year-round (but the flag is not applied if all residents of the household are aged 65 or older); (6) individual with a disability posing a constraint to significant life activity; (7) individual with no health insurance; (8) individual aged 65 or older; (9) households without a vehicle; and (10) households without broadband internet access.⁵ Of course, some expert opinion enters the choice of these 10 vulnerability indicators, but they are nonetheless common indicators of public health, vulnerability, and disaster preparedness (Adger 2006; Peacock 1997; Smith and Kington 1997; Strully 2009; Tate 2012; Willyard et al. 2022).⁶

Individuals within the ACS microdata are then described as low risk (0 vulnerability indicators), moderate risk (1–2 vulnerability indicators), or high risk (3+ vulnerability indicators). Next, using direct survey methods, tabulations for states, counties, and tracts for the number of people at low, moderate, and high risk are estimated. These direct survey estimates are then used to inform the small area model.⁷ More specifically, CRE follows an area-level approach from small area estimation: a direct survey estimate is averaged with an indirect estimate to produce a composite estimate. The average is a weighted average, and the estimates are less volatile than either of the two original estimates alone. Here, direct estimates refer to ACS estimates for the number of people at low, moderate, and high risk as described above. The indirect, or synthetic, estimates for the number of people at low, moderate, and high risk at the tract level are developed by applying modeled proportions to auxiliary population data from the U.S. Census Bureau's Population Estimates Program.⁸ Additionally, variances for direct survey estimates are smoothed using a generalized variance function (GVF).

Next, the weight given to an indirect estimate when producing the composite estimate is the ratio of the GVF variance of the direct estimate to the total variance (i.e., the sum of the GVF variance and the estimated variance of the indirect estimate). The weight for the direct estimate is the complement (i.e., one minus the weight for the indirect estimate). As a result, when survey methods are more precise, the direct survey estimate receives a greater weight; when direct survey methods are less precise, the indirect modeled estimate receives a greater weight. This allows CRE to produce reliable estimates of the number of people in each tract that are low, moderate, or high risk.⁹ Thus, in addition to improved statistical quality (found by creating the CRE directly from microdata rather than from already aggregated estimates), this final measure inherently includes information on both the number and the distribution (among residents) of vulnerability variables. The county-level and tract-level versions of the CRE are publicly available from the Census Bureau (https://www.census.gov/programs-surveys/community-resilience-estimates/data/datasets.html ).

CRE Empirical Application: Relationship With COVID-19 Excess Deaths

Data and Measures: Linking the CRE Microdata to Social Security Administration Data

We link the publicly available CRE to the restricted U.S. Social Security Administration Numident file (quarter 1 of 2021) to examine the relationship between CRE risk measures and excess deaths resulting from the COVID-19 pandemic.¹⁰ Linkages are achieved by using the restricted Master Address File Auxiliary Reference File (MAF), which we use to assign census tracts to individuals identified in the Numident.¹¹ The resulting dataset contains individual-level observations of deaths, each of which is assigned to a census tract of residence (Finlay and Genadek 2021). Thus, we can compute death counts and rates at the tract-month level and, using tract-level CRE scores, which are publicly available (https://www.census.gov/programs-surveys/community-resilience-estimates/data/datasets.html ), determine whether CRE risk scores successfully predicted excess deaths due to COVID-19 in 2020. We also link the CRE's restricted underlying microdata (Ruggles 2014) to individual deaths to compare the CRE components’ predictive capacity to the CRE itself.

Empirical Results

Stability of COVID-19 Excess Deaths Predictions

Figure 1 shows trends in mortality by tract-level CRE risk quartiles,¹² which are formed by taking quantiles of the “3+ risk factors” measure described earlier. Tracts in the highest quartile display larger responses to the COVID-19 shock than tracts in other quartiles. This larger response is most evident in the March 2020 spike, but a similar response can be seen in the December spike.

Race and age are correlates not only of mortality generally, but are also known to be important correlates of mortality during the COVID-19 pandemic (Bassett et al. 2020). Accordingly, we provide a comparison of the variation in COVID-19 mortality of these categorizations relative to the CRE to offer a simple comparison of the relative magnitude of the mortality variation across categorizations, as well as the relative consistency with age and race. The magnitude of the mortality differences across CRE risk categories is broadly comparable to the magnitude of the differences observable across race and age categories, but the CRE risk differences are more consistent across the post-COVID-19 period. Figure 2 illustrates trends in mortality by race, where race categories are assigned using Social Security records. Following the initial March 2020 COVID-19 shock, Black mortality rose to just over 170% of its pre-March mean, while White mortality rose to slightly less than 120% of its pre-March mean. Figure 3 shows trends in mortality by age, where age categories are measured using date of birth from Social Security records. The mortality of persons aged 65 or older rose to more than 125% of its pre-March mean, whereas age 0–5 mortality steadily declined over the course of 2020. Although race and age breakdowns have tremendous predictive power, because CRE measures are a summary index of risk, they more consistently differentiate groups by COVID-19 mortality response rates than these demographic measures. Specifically, we note that the CRE bins consistently stratify the trends, while the demographic groups intersect over the months in 2020 (see Figure 1).

The magnitude of the mortality differences across tract-level CRE risk categories is comparable to that of the mortality differences seen in the SVI and NRI, but the CRE risk differences are more consistent across the post-COVID-19 period. Figures 4 and 5 illustrate COVID-19 excess deaths by SVI and NRI quartiles, respectively. The SVI quartiles do not consistently predict excess deaths after the COVID-19 emergency, though it outperforms the NRI (in terms of stability of quartile excess deaths levels). The NRI's “lowest risk” quartiles appear to jump during the COVID-19 pandemic. Unlike CRE and SVI, NRI considers the value of expected annual losses of each community (in terms of “building value, agricultural value, and population value”). NRI's underperformance may be exacerbated in the context of the COVID-19 pandemic, which disproportionately affected populations rather than also affecting physical infrastructure. We take these results as descriptive evidence of CRE's relative predictive power and relative predictive stability.

Comparing the Predictive Power of CRE Using Machine Learning

To formalize the comparison of the relative predictive importance of CRE, NRI, SVI, and individual socioeconomic variables, we generated a machine learning (lasso) standardized coefficient plot, in which we tested their relative ability to predict the increase in deaths from the COVID-19 pandemic.¹³Figure 6 shows a line drawn for each coefficient that traces its value over the searched values of the penalty parameter. The predictors are entered into the model in the order of their linear regression coefficient magnitude. As a robustness check, Figure 7 shows the results when, instead of calculating the change in mortality from 2019 to 2020 at the tract-month level, we use an alternative measure of excess deaths as the dependent variable, which is a nonnegative measure of mortality responses (Lariscy et al. 2018). That is, to test for robustness of the predictive power of the CRE measures, we construct an excess death measure that takes the 2019 to 2020 tract-level difference in deaths and sets any negative values (death decreases) to zero.

Consistent with our theory-driven hypothesis and the foregoing descriptive evidence, CRE appears to have larger relative predictive power than its 10 underlying socioeconomic variables, the SVI, and the NRI. CRE has a steady rise to its final value, which is larger in magnitude than any of the alternative predictors. SVI also provides strong predictive power (though less than CRE), entering the model early and rising throughout. However, both unemployment and poverty, while entering the model later, appear to end with a larger (in magnitude) final value than SVI. NRI appears to underperform relative to SVI and even some of the more important socioeconomic variables. We offer these results as strong evidence in favor of our hypothesis that CRE provides more predictive power than alternative measures of geographic vulnerability.

Quantifying the Predictive Power of CRE on COVID-19 Excess Deaths

Table 1 quantifies this difference in response using the full values of the continuous CRE risk measure. The estimate in column 1 shows that, for an average census tract, a 1-percentage-point increase in the share of 3+ risk factors is associated with a 0.15-percentage-point (2.2%) increase in deaths from 2019 to 2020. In other words, a 10-percentage-point increase in share 3+ risk factors is associated with an increase of slightly more than one death per neighborhood from 2019 to 2020.

As with the two machine learning coefficient plots, we test an alternative measure of excess deaths. Column 2 of Table 1 shows estimates of the association between the share of 3+ risk factors and the alternative excess death measure (a nonnegative measure of mortality responses). The estimates suggest that a 1-percentage-point increase in the share of 3+ risk factors is associated with a 0.1-percentage-point (0.6%) increase in mortality. Both mortality measures reported in Table 1 suggest the same qualitative story: a 10-percentage-point increase in the share of 3+ risk factors is associated with approximately one additional death per neighborhood from 2019 to 2020.

Discussion

Consistent with previous disasters, the COVID-19 pandemic highlighted the need for enhancing the measurement of community resilience for community organizers, disaster management, and researchers. This research note develops validation tests for the U.S. Census Bureau's new Community Resilience Estimates, and we identify the advantages of developing such estimates using underlying limited-access, individual-level risk factor microdata. In particular, we emphasize the CRE's advantages of (1) using microdata directly, rather than already aggregated estimates of its components, which (2) allows for the distribution of risk factors among individuals within a geography to also provide information (and thereby enhance the accuracy) of the index. Including the distribution of risk factors within a community, rather than the simple count of risk factors within a community, improves local resilience estimation.

After describing the CRE and its research-driven development, we apply the CRE to a simple empirical description of COVID-19 mortality spatial segmentation and excess deaths. Our results highlight that the CRE (1) segments U.S. Census tracts by death rates throughout the COVID-19 pandemic more consistently than other indices, as well as single demographic criteria; (2) provides more predictive power of excess deaths than other indices, as well as single demographic or socioeconomic criteria; and (3) predicts excess deaths occurring during the COVID-19 pandemic. Specifically, we find that a 10-percentage-point increase in a key CRE risk measure is associated with one additional death per neighborhood during the initial outbreak of COVID-19. The CRE advances current measures of community resilience and will be useful for disaster preparedness, disaster response, and further research.

Acknowledgments

Any opinions and conclusions expressed herein are those of the authors and do not represent the views of the U.S. Census Bureau. All results have been reviewed to ensure that no confidential information is disclosed (U.S. Census Bureau Disclosure Review Board approval numbers CBDRB-FY22-POP001-0068 and CBDRB-FY23-SEHSD012-009).

Notes

References