Who Counts Where? COVID-19 Surveillance in Federal Countries

Abstract

Context: While the World Health Organization (WHO) has established guidance on COVID-19 surveillance, little is known about implementation of these guidelines in federations, which fragment authority across multiple levels of government. This study examines how subnational governments in federal democracies collect and report data on COVID-19 cases and mortality associated with COVID-19.

Methods: We collected data from subnational government websites in 15 federal democracies to construct indices of COVID-19 data quality. Using bivariate and multivariate regression, we analyzed the relationship between these indices and indicators of state capacity, the decentralization of resources and authority, and the quality of democratic institutions. We supplement these quantitative analyses with qualitative case studies of subnational COVID-19 data in Brazil, Spain, and the United States.

Findings: Subnational governments in federations vary in their collection of data on COVID-19 mortality, testing, hospitalization, and demographics. There are statistically significant associations (p < 0.05) between subnational data quality and key indicators of public health system capacity, fiscal decentralization, and the quality of democratic institutions. Case studies illustrate the importance of both governmental and civil-society institutions that foster accountability.

Conclusions: The quality of subnational COVID-19 surveillance data in federations depends in part on public health system capacity, fiscal decentralization, and the quality of democracy.

Public health surveillance—defined as “the ongoing, systematic collection, analysis, and interpretation of health-related data essential to planning, implementation, and evaluation of public health practice”—has undergirded the global response to the SARS-CoV-2 virus (Thacker and Berkelman 1988: 164). Since the World Health Organization (WHO 2020a, 2020b) first issued a standard case definition and recommended surveillance strategies, data on the virus have shaped public and elite decisions in profound ways, especially as the entire world has had to simultaneously align toward tracking the same novel virus. While surveillance is not a substitute for effective pharmaceutical and nonpharmaceutical interventions, its absence constitutes a major barrier to the management of infectious disease (Castillo-Salgado 2010; David, Nakouné, and Giles-Vernick 2020; Greer 2017; Groseclose and Buckeridge 2017; Lee and Thacker 2011; Pearce et al. 2020). Inadequate or inconsistent surveillance—accentuated by large numbers of asymptomatic or paucisymptomatic COVID-19 infections—has seriously hampered national efforts to control the spread of the virus (Pullano et al. 2021). Throughout the pandemic, governments' data collection failures have exacerbated existing disparities in access to care (e.g., accessibility of COVID-19 testing), care delivery (e.g., strained safety net hospitals, disparities in vaccination rates), health outcomes (e.g., high mortality and morbidity among low-income and Black, Indigenous, and People of Color communities), and other social inequities (Alberti, Lantz, and Wilkins 2020; Bambra, Lynch, and Smith 2021; Ndugga et al. 2021; Okonkwo et al. 2020).

The production of a global viral surveillance infrastructure is itself an extraordinarily complex project (Greer 2017). Effective global surveillance depends not only on international standards for disease classification but also on the development and maintenance of strong data systems at the national level (Suthar et al. 2019). Despite the WHO's rapid creation and dissemination of guidelines for standardizing global COVID-19 surveillance, a recent survey of epidemic detection and reporting capacities revealed that 77% of all countries lack the ability to collect ongoing or real-time laboratory data, while only 24% of countries possess a national system for transporting viral specimens (NTI 2019). Furthermore, COVID-19 data collection is not uniform. How COVID-19 data is collected or what exact data is recorded varies (e.g., official decisions about cause of death vary by profession and governments), making it difficult to compare between countries (Backhaus 2020; Greer et al. 2021; Greer and Elliott 2019; Karanikolos and McKee 2020; Lloyd-Sherlock et al. 2021; Naudé and Vinuesa 2020).

Adequate public health surveillance requires not only immense coordination across countries but also action across multiple agencies or jurisdictions within countries (Greer et al. 2021; Lee and Thacker 2011; Mounier-Jack and Coker 2006). The challenges of constructing national surveillance regimes may be especially visible in federal countries, which often divide the authority and capacity to produce vital statistics and the implementation of WHO surveillance guidelines across multiple levels of government (Huberfeld, Gordon, and Jones 2020). By fracturing authority for data collection, federal systems introduce the possibility of meaningful variation in the quality of COVID-19 surveillance at the subnational level. In Mexico, gaps in subnational testing capacity led to a critical underreporting of COVID-19–associated deaths; after accounting for these problems, the country's national health ministry released a revised cumulative death total that was 60% higher than previously published figures (BBC 2021). In the United States—where the federal government has played a relatively weak steering role in COVID-19 surveillance—state and local jurisdictions vary widely in their procedures for certifying cause-of-death data (Galaitsi et al. 2021; Kettl 2020). Only about half of American states are currently reporting “probable” COVID-19 deaths, or those that can be traced to COVID-19 through clinical and epidemiological evidence but lack laboratory confirmation (Koerth 2020). While the WHO acknowledges the need for adapting its case definitions depending on the “local epidemiological situation and other factors,” these subnational variations are nonetheless a barrier to global knowledge production (WHO 2020a: 1). Perhaps understandably, COVID-19 caused greater centralization of public health authority among OECD (Organisation for Economic Co-operation and Development) countries (Dougherty et al. 2020).

Yet while federations often fracture authority for the production of surveillance data, they also vary internally in ways that may lead to subnational unevenness in disease surveillance (Büthe et al. 2020; Desson et al. 2020). This article seeks to assess three potential political explanations for variation in the quality of COVID-19 data collected at the subnational level in federal systems. First, federal systems vary in the extent to which national and subnational governments devote state resources to the protection of public health (Benz and Broschek 2013; Greer et al. 2020). While some federations possess large reserves of expertise, resources, and public trust that are spread relatively uniformly across subnational territories, others may lack the skills, raw materials, or public trust necessary to execute large-scale public health surveillance (Bosancianu et al. 2020).

Second, federations vary in how they decentralize fiscal resources, administrative capacity, and political authority (Watts 2008). Decentralization may cut in multiple directions on public health surveillance. On the one hand, highly decentralized systems may create barriers to the effective coordination and standardization of data collection rules (Treisman 2007; Wibbels 2005). Yet the decentralization of resources and authority may also enable subnational governments to respond quickly to changing local conditions; assuming a sufficiently high level of subnational governing capacity, greater devolution of authority may allow subnational governments to develop the skills and capacity necessary to perform the complex tasks associated with disease surveillance (Kumar and Prakash 2017).

Finally, the quality of democratic institutions—including political rights and freedoms, free and fair elections, and a strong, independent media—varies across federations. Democracy is associated with greater life expectancy and reduced child mortality (McGuire 2010). Within federal democracies, however, the quality of democratic institutions may vary at the subnational level (Gibson 2012). This variation may affect not only citizens' leverage on how public health resources are allocated but also their ability to hold governments accountable through ensuring the production of credible information on, among other things, disease outbreaks (Berliner et al. 2021). As Greg Michener (2011: 158) puts it, “Knowledge is power, and getting rulers to give it up to the ruled remains an uphill climb.” Yet democratic institutions can give public officials stronger incentives to produce public information. In contexts with strong electoral democracy, incumbents may have an incentive to create strong surveillance systems; because they may eventually lose reelection, they may want to monitor their opponents' performance in government (Berliner 2014). The existence of strong civil society organizations at the subnational level can also help monitor officials' compliance with national health policies (Rich 2019). An independent media may be able to hold local public officials accountable for performing their role in public health surveillance (Besley and Burgess 2002).

While we have strong reasons to believe that these three factors are important in explaining how subnational disease surveillance varies across federations, there are other potentially important factors we do not analyze. First, we do not examine the effects of social structural factors on the production of surveillance data. Although ethnic diversity, economic inequality, and interpersonal trust may affect the context in which government acts to arrest a pandemic, these variables bear a less obvious relationship with the adoption of consistent COVID-19 surveillance policies at the subnational level (Bosancianu et al. 2020). Second, while the rise of populist leaders is an oft-cited reason for variation in national responses to COVID-19 and bears some relationship to governments' refusal to adopt nonpharmaceutical policy interventions (NPIs) during COVID-19, we do not examine the relationship between populism and the quality of subnational COVID-19 surveillance. Even though there are reasons to suspect that control of government by populist parties during COVID-19 is associated with greater distrust of experts and hence weaker NPIs, it seems less likely that populist strength as such has effects on the quality of public health surveillance data. This is the case because the systems and infrastructures that produce surveillance data were created prior to COVID-19 (Rinaldi and Bekker 2021). In other words, though populists may be more likely to ignore data, they may still find it difficult to inhibit its collection.

Data and Methods

The goal of our study was to better understand variation in the quality of subnational COVID-19 data reporting among federal democracies. Because surveillance is essential to public health responses to COVID-19, and because many nations have federal structures that may complicate this already cumbersome task, we wanted to understand the extent to which state capacity, decentralization, and democratic institutions affected the quality of subnational surveillance data within federations. Our analysis thus focused on the set of all countries (n = 15) classified both as federations by the Forum of Federations and as full or flawed democracies in the Economist Intelligence Unit's Democracy Index.¹

We extracted publicly available data on COVID-19 surveillance in each major subnational unit of government with its own governance structure in all 15 countries (n = 316), including all provinces, states, or territories as well as formally designated federal/capital regions (e.g., the District of Columbia, Buenos Aires, and Brussels). The data were collected from publicly available websites of subnational governments and, when necessary, national governments or nonprofit organizations, between September 15, 2020, and October 15, 2020. This allowed us to analyze a cross-section of surveillance efforts rather than change over time. Table 1 describes all federations in the data set.

Drawing on the WHO's interim guidance on COVID-19 surveillance (WHO 2020a) as well as its recent guidance on excess mortality (WHO 2020b), we extracted information on the availability of 13 pieces of data for each subnational unit of government (see table 2). Our approach to selecting indicators was deliberately minimalistic. We did not rank or rate the usability or “user friendliness” of subnational data dashboards, or the consistency of data reporting over time (Vasudevan et al. 2020). Nor did we attempt to look “behind” the numbers to recover the process by which subnational surveillance was produced (Merry 2016). Instead, we are interested in the local realities of what Paul N. Edwards (2010: 25) calls “infrastructural globalism,” or projects for “permanent, unified, world-scale, institutional technological complexes” that generate information about the entire world. As such, we focused our attention on whether—at a minimum—subnational jurisdictions are adopting basic standards for case definition and public reporting identified in WHO guidance.

Our 13 binary indicators, drawn from WHO (2020a, 2020b) guidance, can be grouped into four categories: (1) mortality information, including data on confirmed COVID-19 mortality, probable COVID-19 mortality, and excess mortality; (2) information on hospitals and health facilities, including data on hospitalizations and discharges, as well as infections and deaths among health care workers; (3) testing information, including the number of tests, the identification of test types, and information on whether the testing denominator is persons or test specimens as well as information on rates of test positivity; and (4) demographic information, including whether cases and deaths are broken down by age and sex.

Using these indicators, we constructed a quantitative codebook. After an initial training and discussion of a sample of observations, all members of the research team divided responsibilities for coding. Differences in coding approaches were resolved through consensus.

To create dependent variables that allow for cross-national comparison, we calculate the percent of subnational governments in each country where each of the 13 types of information is present. We then calculate a mean score ranging from 0 to 1 in each of the data categories to construct four component indices measuring the availability of data at the subnational level in each country: a Mortality Data Index (MDI), a Hospital/Health Facility Data Index (HDI), a Testing Data Index (TDI), and a Demographic Data Index (DDI). Finally, we calculate a mean score of the four indices to form a Subnational COVID-19 Data Quality Index (SCDQI), which ranges from 0 to 1.

We also collected data on several independent variables relevant to each of the institutional correlates described in the previous section (see table A1 in the online appendix). First, we collected data on three general indicators of state capacity, including the World Bank's measure of government effectiveness, a measure of each country's public health expenditure as a percent of gross domestic product (GDP), and the Global Health Security Index (GHSI) measure of national pandemic preparedness. Second, drawing on the GHSI database, we collected index scores of public health system capacity, including indices of real-time disease surveillance and reporting, laboratory systems, and the availability of health care facilities. Third, we collected data on three indices of fiscal, administrative, and political decentralization (Ivanyna and Shah 2014). Fourth, we collected Varieties of Democracy (V-Dem) Institute indicators of liberal democracy, free and fair subnational elections, and media independence. We also collected data on a control variable, a logged measure of per capita GDP.

Our statistical analysis examines the relationship between state capacity, decentralization, and democracy and indicators of subnational data quality. Because the dependent variables are indices of subnational data quality, we used ordinary least squares (OLS) regression with robust standard errors. We analyzed both unconditional and conditional correlations between z-standardized independent variables and each of the four component indices and the SCDQI.

We supplemented these analyses with short case studies of subnational COVID-19 surveillance in three countries that vary along our key independent variables: Brazil, Spain, and the United States. Brazil and the United States are both highly decentralized, with median scores well above the sample median on each of the three indicators, but Brazil has weaker democratic institutions and weaker state capacity than both the United States and Spain. Spain is less decentralized than the United States and Brazil, but along with the United States is above the sample median on all three indicators of state capacity and democratic institutions. The differences among Brazil, Spain, and the United States in levels of state capacity, decentralization, and democracy lead us to expect variation in surveillance outcomes. To compose these studies, the authors utilized an assortment of publicly available documents, secondary sources, and interviews.

Results

Variation in Subnational COVID-19 Data Quality

We found significant variation in the quality of subnational COVID-19 reporting across countries. As the dot plot in figure 1 shows, national scores on the Subnational COVID-19 Data Quality Index (SCDQI) range from 0 (Papua New Guinea) to 0.75 (United States). Countries known to have substantial public health capacity and infrastructure—such as the United States, Canada, Belgium, and Germany—scored higher on data quality, but none had a perfect score. The median score on the index is 0.56, with the scores for most countries falling below 0.50. So many low SCDQI scores indicate that a large percentage of subnational units in these 15 federal systems did not publicly report data on key indicators contained in the WHO guidance on COVID-19 surveillance. A closer look at scores on the component indices helps further explain this variation.

Figure 2 illustrates country scores on each of the four component indices—Mortality Data Index (MDI), Hospital Data Index (HDI), Testing Data Index (TDI), Demographic Data Index (DDI)—that make up the SCDQI. Where mortality data is concerned, country scores on the MDI ranged from 0 to 0.80. Variation in the MDI was driven largely by inconsistency in subnational reporting of probable deaths. On average, fewer than 5% of subnational units reported this type of information. Subnational governments exhibited similar variation in reporting data on hospitals and health facilities. The HDI examines such data on hospitals and health facilities, with country scores ranging from 0 to 0.75. On average, 23% of subnational governments reported data on hospital discharges, 21% reported data on COVID-19 infections among health care workers, and fewer than 10% reported data on COVID-19 mortality among health care workers. The lack of data collection on how COVID-19 impacted health care workers was notable. Country scores on the Testing Data Index exhibited especially wide variation. The TDI scores ranged from 0 to 1. On average, 50% of subnational governments reported information on the number of each type of test administered (e.g., antigen, polymerase chain reaction [PCR] test). Similarly, the average share of subnational governments that specified whether they were counting the number of people tested versus the number of tests administered was 57%. Alternatively, this means that about half of subnational governments did not differentiate between number of tests administered versus number of people tested, which could make it difficult to decipher whether individuals were being tested multiple times or whether there was broad-based testing of the entire population.

Institutional Correlates of Subnational Data Quality

We report both conditional and unconditional correlations between indicators of three institutional characteristics (state capacity, decentralization, and democratic institutions) and five indices of COVID-19 data quality (SCDQI, MDI, HDI, TDI, and DDI). As noted above, the conditional correlations incorporate controls for population size and the severity of the pandemic (measured in terms of cumulative COVID-19 deaths per million as of October 20, 2020).

Figure 3 presents bivariate relationships between each indicator and country scores on the subnational COVID-19 data quality index. Each of the indicators of state capacity (cells A–F; government effectiveness, public health expenditures, pandemic preparedness, real-time surveillance and reporting, health capacity, laboratory systems) is associated with higher scores on the index (p < 0.05). This finding confirms that adequate public health surveillance that leads to high-quality data requires an adequate public health infrastructure. Next, we report on the indicators of decentralization (cells G–I; fiscal decentralization, administrative decentralization, political decentralization). Higher levels of both fiscal and administrative decentralization (cells G and H) are significantly associated with higher scores on the SCDQI, while the association between political decentralization (cell I) and subnational COVID-19 data quality is positive, but not statistically significant. Last, indicators of the quality of democratic institutions (cells J–L; liberal democracy, free and fair subnational elections, media independence) are all positively and significantly associated with better performance on the SCDQI.

What happens when we examine both conditional and unconditional correlations across the four component indices and the SCDQI? To answer this question, table 3 summarizes the statistical relationship between variables for the three institutional characteristics and country scores on the five indices of subnational COVID-19 data quality. Cell entries represent the sign of the statistical relationship in OLS regression models estimated with robust standard errors, with and without control variables. An asterisk indicates that the relationship in both bivariate and multivariate models achieves a conventional level of statistical significance (p < 0.05). Data on estimates and p-values for each model can be found in tables A2–A6 in the online appendix.

The data here reveal three interesting patterns. First, when it comes to overall data quality (SCDQI), indicators of democratic institutions and state capacity perform well, while indicators of decentralization do not. All indicators of democratic institutions are positively and statistically significantly associated with SCDQI. Of the six indicators of state capacity, the four fine-grained measures of state capacity (e.g., health care delivery and laboratory capacity) have positive and statistically significant relationships with the SCDQI, whereas the two more generic indicators of government effectiveness and pandemic preparedness do not. By contrast, only one indicator of decentralization (fiscal decentralization) has a positively and statistically significant relationship with the SCDQI.

Second, beneath these general patterns, the data tell unique stories about the factors associated with subnational data quality on mortality, hospital utilization, and testing. Only two indicators of state capacity (real-time surveillance and reporting) and one indicator of democratic institutions (free and fair subnational elections) have consistently positive and statistically significant relationships with all three of these indices. By contrast, the influence of other key indicators appears to be limited to a single index. For example, higher levels of media independence are only significantly associated with higher-quality testing data at the subnational level. Higher levels of fiscal decentralization are only associated with higher-quality hospital data. Higher levels of health system capacity are only associated with higher-quality testing data. This suggests that there is not a single, generic explanation for variation in multiple types of subnational COVID-19 surveillance data. Rather, different aspects of state capacity may be more or less consequential for producing higher-quality subnational surveillance data.

Third, only one variable—public health expenditures as a percentage of GDP—is positively and statistically significantly associated with the reporting of COVID-19 cases and mortality by age and sex. This perhaps reflects a broader pattern of delays in the consistent reporting of demographic data on COVID–19 reporting across many countries (Dowd et al. 2020). Nevertheless, it is worth highlighting that no indicator of fiscal decentralization or democratic institutions was positively and significantly associated with increases in the reporting of demographic data.

In sum, state capacity and democratic institutions are generally associated with higher levels of subnational COVID-19 data quality. Yet understanding variation in the quality of specific types of data requires a more nuanced look at how specific types of capacity and mechanisms of democratic accountability work. Our three qualitative case studies will add context to our quantitative findings, exploring how state capacity, fiscal decentralization, and the quality of democratic institutions influenced the production of COVID-19 surveillance data at the subnational level.

Case Studies

Brazil, Spain, and the United States vary in ways that are consistent with our quantitative findings about the effects of administrative capacity, fiscal decentralization, and the quality of democratic institutions on COVID-19 surveillance data collection. In the following case studies, we illustrate the mechanisms through which each of these variables affects COVID-19 data-collection outcomes and, ultimately, national pandemic response.

Brazil

Brazil ranks in the middle on almost all subnational COVID-19 reporting indicators. On the one hand, all states in Brazil produce basic data on case and death counts. But on the other hand, states vary significantly in whether they produce more detailed data on cases, hospitals, tests, and deaths. The case analysis supports the quantitative analysis, which suggests that this middling ranking is an outcome of forces pulling in different directions. Brazil has a high degree of fiscal decentralization and a moderate degree of capacity for disease surveillance at the subnational level, but it has weak democratic institutions.

Brazil is often referred to as a case of “strong federalism” for its high levels of fiscal, administrative, and political decentralization (Ames 2002). In the public health arena, Brazil has a heavily decentralized but highly coordinated national health surveillance system (SVNS), with clearly defined roles for municipal and state-level governments. Strategic Health Surveillance Information and Response Centers (CIEVS) within municipal-level health secretariats collect daily information on cases and deaths (Magno et al. 2020). State health secretariats then aggregate and send this information daily to the national health ministry, which publishes the data on a COVID-19 webpage. The roles of municipalities and state-level governments in epidemiological surveillance are defined in the national constitution and in legislation that specifies national versus territorial responsibilities. This decentralized health-surveillance structure has ensured uniformity in basic COVID-19 reporting: all states and all municipalities collect and publish daily information on confirmed cases and deaths.

However, subnational variation in administrative capacity has limited more detailed COVID-19 reporting in some states. Beyond basic data on confirmed cases and deaths, states and municipalities vary widely in how many other COVID-19–related statistics they report, such as on hospitalizations and on testing. Weak subnational administrative capacity has also limited the reliability of even basic subnational COVID-19 data in some states. For example, many states have produced false negative COVID-19 results because of collection-quality problems such as improper handling and shipping (França et al. 2020). COVID-19 deaths are similarly estimated to be widely underreported: without confirmed test results, local doctors have been unable to mark death certificates as COVID-19 related (França et al. 2020). Conversely, states have produced misleading spikes in positive results as a result of recent expansions of PCR testing, which captures both current and past infections (interview with Wesley Cota, October 1, 2020, via Zoom). Such variation in the COVID-19 data collection seems to depend on the strength of state-level health bureaucracies. States with high-functioning public health systems such as Ceará produce relatively accurate, updated information, whereas states with failing public health systems such as Mato Grosso produce only minimal, unreliable data.

Presidential politics—arguably related to Brazil's weak democratic institutions—has further hampered national data production. The current president, Jair Bolsonaro, has denied the seriousness of the COVID-19 pandemic, firing one national health minister and forcing a second to resign in the first four months of the pandemic. More recently, President Bolsonaro appointed an interim minister who is a military general with no public health experience. These administrative upheavals delayed Brazil's release of a national protocol that adopted WHO guidelines for defining and reporting on the pandemic until well after COVID-19 had spread within Brazil. As a result of this delay, states initially adopted their own guidelines for COVID-19 reporting, which in some cases differed from those recommended by the WHO (França et al. 2020). Moreover, the Ministry of Health has required states and municipalities to collect data only on confirmed cases and deaths, without specifying the electronic format in which this data should be published. The Ministry of Health has also been inconsistent in how it makes this data available. On June 7, for example, the ministry paused reporting cumulative case and death counts, until a court order on June 9 forced the government to continue publishing this information.

At the same time, Brazil's flourishing civil society and vibrant public health community has balanced some of the negative effects of presidential politics. As a result of Brazil's prominent public health movement of the 1970s, health-focused university programs and research institutes abound in Brazil (Gibson 2019). The continued strength of civic organization around public health in Brazil has translated into a variety of initiatives to fill gaps in the national health ministry's COVID-19 data production efforts. Some of these efforts have come from within other branches of Brazil's public health system. The National Council of State Health Secretariats (CONASS) publishes daily updates to state and municipal-level data on confirmed cases and deaths. Infogripe, a project initially designed to track severe acute respiratory syndrome during the H1N1 flu epidemic, publishes open-access data on COVID-19–related cases. Several private volunteer groups such as Wesley Cota's COVID-19 group, Brasil.IO, and OpenKnowledge Brasil, have launched similar initiatives.

Spain

Our quantitative analysis suggests that Spain's highly decentralized health system has excellent data reporting capacity. However, while the range and level of detail of data reported stand out in comparative terms, Spain has experienced serious issues with respect to the usability of its COVID-19 data for controlling the spread of the virus. The reality on the ground reflects problems of state capacity, particularly in terms of rapid crisis response requiring the redirection and mobilization of governmental resources and evidence-based policy making in a decentralized polity.

On paper, Spain has a well-functioning public health surveillance system. The Center for the Coordination of Alerts and Health Emergencies (CCAES), a unit within the Spanish Ministry of Health, publishes daily updates in PDF format on the evolution of COVID-19 cases based on daily, individualized updates provided by the regional health services in each of Spain's 17 autonomous communities (ACs). The health authorities in each region (generally a specialized agency devoted to public health, including epidemiological surveillance) upload data to a dedicated website (SiViES) created in 2011 and managed by the National Epidemiology Center (CNE). Regional health authorities collect information from public hospitals, ambulatory care centers, and private health care facilities. The CNE also publishes periodic panel data using the same sources.

In practice, however, data reporting in Spain has suffered from problems of accuracy, timeliness, and insufficient detail throughout the crisis. While some regional governments have been transparent and detailed in their reporting (Castilla y León), most have underperformed (Almunia and Rey-Biel 2020). Some regions took weeks to upload any data to SiViES. When they did, they reported data in different ways, leading to important errors that took weeks to detect. For many weeks, the number of diagnostic tests administered went unreported (Llaneras 2020). To this day, data disaggregated by sex and age is not publicly available at the regional level. And with the exception of death reporting, virtually all other daily reporting continues to be restricted to PDF formats (Trias-Llimós et al. 2020). Broadly, there continues to be significant inconsistencies across regions in the reporting of COVID-19 data. Even the panel data, updated in three waves so far by the National Center for Epidemiology, is inconsistent (Andrino, Grasso, and Llaneras 2020).

It would be tempting, but wrong, to blame fiscal decentralization for these problems. Health care in Spain has been managed primarily by the 17 ACs since at least 2002. The ACs are responsible for building on national framework legislation to plan health resources, professional accreditation, quality assurance, and pricing. Health care is also financed through regions' general budgets, which gave them a great deal of policy making control. Yet what created challenges for regional health systems' data production capacity was not decentralization itself but austerity measures introduced in late 2011, which led to a reduction in overall health spending from 6.5% to 5.1% of GDP over the course of five years and the subordination of health care decisions to spending reduction targets set by the Economics and Finance Ministries (Bernal-Delgado et al. 2018). If anything, the sharp reduction in ACs' fiscal capacities helps explain why the central government suspended much of their authority with the State of Alarm declaration in March of 2020.

A far more compelling explanation for limits on surveillance capacity in Spain hinges on the role of democratic institutions. Since the return to democracy in the late 1970s, political debate in Spain has consistently been more focused on politics than policy (Subirats and Gomà 2020). The lack of consensus regarding the structure of the state itself—particularly with respect to the competencies, financing, and representation of the ACs—has consistently distracted attention from more mundane questions of policy and process that are critical for the quality of democratic institutions. These struggles have limited data sharing and stymied the Ministry of Health's efforts to coordinate policy, discouraging strong leadership on issues like public health (Artells, Peiró, and Meneu 2014). They have also generated a profound politicization of the civil service at all levels of government, devaluing expertise and discouraging civil servants from experimentation and innovation (Lapuente et al. 2018). Not surprisingly, students of Spanish policy making decry a profound lack of policy evaluation and, therefore, of learning (García-Basteiro et al. 2020). When asked to produce specific data on COVID-19, procedure-driven Spanish bureaucrats—at both the central and regional levels—have been able to do so, albeit not necessarily in the most timely and tractable fashion.

United States

The United States performs well on our quantitative indices of subnational COVID-19 data quality. This makes sense, given the country's relatively high level of state capacity, the primary role of state and local governments in setting public health policy, and the existence of civil-society institutions. Yet, especially given major reductions in state and local public health budgets during the decade preceding the pandemic, the United States has nevertheless faced challenges in coordinating the production of COVID-19 data (Weber et al. 2020). As with Spain, this likely reflects the fact that our data attempt to capture basic WHO indicators rather than data usability. The challenge of COVID-19 data collection in the United States is best exemplified by examining the implementation of the COVID-19 case definition itself.

In the United States, responsibility for maintaining and revising the national list of notifiable diseases and producing standard case definitions falls not to federal agencies but to a nongovernmental organization, the Council of State and Territorial Epidemiologists (CSTE). During COVID-19, the CSTE was able to move swiftly to adopt the WHO-approved case definition for the virus. Yet, while states have typically adhered to a “handshake agreement” to follow CSTE case-definition guidelines (e.g., during the H1N1 flu), these guidelines are essentially nonbinding (Reinhard et al. 2020). In the absence of clear subnational responsibilities, implementing these case definitions essentially depended on voluntary coordination. Months after the CSTE had adopted its case definition, nearly half of all states continued to lack data on probable cases and deaths. Failure to report probable cases and deaths is especially significant, given that testing shortages likely led to an undercount of confirmed deaths.

While states in the United States are more likely to report a broad range of data elements contained in WHO guidelines than their counterparts in other countries, weak subnational administrative capacity accounts for variation across states in their adoption of CSTE case definitions. Many local jurisdictions in the United States do not require medical examiners to formally review cause-of-death data before it is reported to state vital records' offices. In jurisdictions without adequate testing or medical expertise, death certificates often lacked clinical or epidemiological evidence relevant to a COVID-19 diagnosis, contributing to an undercount whose magnitude is still unclear (Koerth 2020). Other well-established gaps in state-level technical capacity (Weber et al. 2020) limit subnational COVID-19 reporting as well. While the Centers for Disease Control and Prevention (CDC) has attempted to standardize disease reporting through an interoperable National Electronic Disease Surveillance System (NEDSS), some states have underinvested in regular updates to the NEDSS base system, which has created challenges to real-time monitoring of the pandemic (Najmabadi and Walters 2020).

Democratic institutions, strong in some areas and weak in others, have also shaped COVID-19 data generation in the United States. As Archon Fung (2020) notes, media outlets and nonprofit organizations like the COVID Tracking Project were among the first sources of consistent information on COVID-19 testing availability and hospitalization. Through public records requests, these organizations have also incentivized the production of information from government sources. Yet gaps in democratic accountability have also stymied the production of pandemic surveillance. The Trump administration's relatively hands-off response to the pandemic did little to solve policy gaps across the states. Early in the pandemic, the administration failed to move quickly to allocate sufficient testing supplies to state, local, and territorial governments. Even when the administration did approve new antigen tests—following an initial period during which CDC-approved tests contained a faulty component rendering them ineffective—a lack of planning left states with “no standardized way to capture results” from the tests (Wan and Sun 2020). As a result, as the tests became more widely distributed, the number of inconsistencies in state testing data was expected to grow, making it increasingly difficult to geographically pinpoint new outbreaks.

Discussion and Conclusion

Our analysis aimed to understand how variation in state capacity, decentralization, and quality of democratic institutions across 15 federal democracies impacted the quality of subnational COVID-19 surveillance data. We found that there was substantial variation among federal systems even when a minimalistic set of subnational data indicators based on WHO guidance was used. Our analysis revealed that stronger democratic institutions (e.g., free and fair subnational elections) as well as specific forms of preexisting state capacity (laboratories and real-time surveillance systems) were, in general, strongly associated with higher levels of subnational surveillance data quality. With one exception, measures of fiscal, administrative, and political decentralization did not have significant effects on subnational data quality.

Beyond these generic patterns, analysis of each type of surveillance data reveals several distinctive relationships. Media independence, for example, is significantly associated with higher-quality subnational testing data, but not other forms of data. Our case studies, which point to the importance of civil society institutions' holding public agencies accountable for data on COVID-19 testing, provide partial context for this finding. But further research on the link between media independence and the incentives of public officials is warranted. Fiscal decentralization was correlated only with more consistent reporting of data on COVID-19 hospitalizations. This may be because decentralized spending increases the efficiency of public service delivery only if there are adequate political and institutional environments (i.e., autonomy of local governments, strong accountability across institutions, good governance, and strong local-level capacities) (Sow and Razafimahefa 2015). Future studies should investigate how fiscal decentralization interacts with these other institutional factors to affect COVID-19 public health surveillance and response.

As noted above, there is ample research to suggest the necessity (though not sufficiency) of disease surveillance for pandemic management. Yet it is worth pointing out that our analysis tracks variation in the collection and reporting of COVID-19 surveillance data, and a minimal set of surveillance data at that. By contrast, we do not measure or evaluate how COVID-19 surveillance data are used to shape policy. Nor does our research design assess variation in, for example, the scale of testing and contact-tracing programs at the national or subnational level. Indeed, our qualitative case studies hint at a potential disconnect between how federations help produce the basic elements of a global pandemic dataset and how policy makers use surveillance data to guide their actions in real time. For example, while subnational governments in the United States are fairly consistent in their reporting of data on COVID-19 mortality and testing, state capacities and protocols for contact tracing are extraordinarily weak compared to countries like Australia, where public health agencies—enabled by a large workforce and high levels of public trust—engaged in both forward and backward tracing of cases at multiple levels of contact (Lewis 2020).

Hence there are opportunities for future research on how policy makers employ (or fail to employ) surveillance data when developing and implementing public health interventions, including nonpharmaceutical interventions such as quarantines, requirements for physical distancing, and other mitigation measures. To be sure, there are other reasons that officials—including subnational officials—may fail to adopt appropriate health measures at the national and subnational levels, including ideological conflict, low levels of social trust, or weak capacities for coercion or public provision to support these measures (Greer et al. 2021). Nevertheless, future studies should explore how officials in countries or jurisdictions that are highly capable of reporting COVID-19 surveillance data ultimately come to regard, or disregard, this data when making decisions. Additionally, future research should identify the factors that led to the failure of both subnational and national efforts to implement effective testing and contact-tracing strategies, which not only enhance the quality of surveillance data but also have the potential to mitigate the spread of the disease (Clark, Chiao, and Amirian 2020).

Our study has several limitations. First, because we wanted to focus on the major data elements the WHO discusses in its COVID-19 surveillance guidance, we used a minimal measure of subnational COVID-19 data quality. This means we did not examine various types of data that could impact performance such as contact tracing or workplace data. Second, we did not examine how data changed over time: all data used were cross-sectional, not longitudinal. Third, we did not examine how the data were produced. While most cross-national statistics are assumed to have random errors, unrelated to political phenomena, COVID-19 has brought attention to the politicization of public health—and, more broadly, scientific—data. Our research design is not equipped to examine efforts at intentional manipulation of COVID-19 data. Last, our analysis, while simple, could have omitted one or more relevant regressors, leading to omitted variable bias in our OLS models. Still, as noted above, we have good reasons to believe that other important regressors mentioned in the literature on COVID-19 outcomes do not bear a strong relationship to the quality of surveillance, the outcome of interest in the present paper.

What we know about the global impact of COVID-19 hinges not only on the WHO's capacity to coordinate the production of surveillance data across countries but also on countries' ability to produce knowledge about the disease. In federal countries, that often means ensuring consistency in reporting across many jurisdictional boundaries. Our analysis suggests federations can and do solve this problem, but it also suggests that their ability to do so is contingent on the state's preexisting capacity for specific public health tasks as well as the quality of democratic institutions. “Seeing like a state” is hardly enough to fight a pandemic (Scott 1998). Yet the quality of the data collected by national and subnational governments will shape not only how the pandemic is managed in the present but also what we are able to learn from COVID-19 for years to come.

Acknowledgments

The authors wish to thank Liam Bower and Sarah Beck for excellent research assistance with this article. Additionally, they are grateful to Scott Greer, who provided valuable insights on the project at a crucial early stage. Finally, the authors thank the special issue editors and anonymous referees for suggestions on earlier versions of the manuscript.

Note

References