State Calculations and the Political Promise of Replication

Abstract

This article examines how and to what effect the scientific ideal and practice of replication is adopted by a Mexican federal government agency charged with measuring poverty. The commitment to replication among state poverty experts is traced to their self‐conception as democratic reformers working against cultures of state opacity associated with an authoritarian past. These experts deploy the ideal of replication as a bureaucratic ethos and the practice of replication as a public‐facing and legitimating strategy. Replication successfully performs transparency and generates trust by appealing to, and strengthening ties with, elite academic and policy actors. Ultimately, the article shows how scientific ideals and practices adopted by state actors are cast as democracy enhancing even as they produce elite closure and limit public participation.

“I do not accept the results of that survey!” Mexico's President Andrés Manuel López Obrador said during his daily morning press conference on August 6, 2020. AMLO, as the president is known, had been asked by a journalist to respond to the official poverty estimates released the day before by the National Council for the Evaluation of Social Development Policy (Coneval), an agency of the Mexican federal government dedicated to measuring poverty and evaluating social policy. According to the official calculations, between 2018, the year AMLO took office, and 2020, the number of people living in poverty had increased by almost four million. In a country of roughly 130 million people, over 40 percent of the population was now poor, with close to 10 percent categorized as “extremely poor.” Rejecting these findings, AMLO continued, “I have other data and I think that people are receiving more support. Even with the pandemic, people have enough to maintain their basic consumption and, very importantly, they have not lost faith.”

By the summer of 2020, AMLO's mañanera, the daily morning news conference during which the president would react to reporters’ questions or speak at length on a topic of his choosing, had become routine. At these conferences AMLO spoke off the cuff, and when asked about his style he boasted of rejecting prepared talking points and “analyzed ideas” in favor of being spontaneous (Beck and Iber 2022). This preference for spontaneity notwithstanding, the president's remarks about the poverty estimates were part of a well-established pattern. Since taking office in December 2018, AMLO had often rejected the Mexican state's official poverty estimates and criticized Coneval. When confronted with official poverty figures, AMLO had on various occasions insisted he had “other data,” and during his first year in office the president had even threatened to dismantle Coneval and delegate the task of poverty measurement to a different agency already under the purview of a loyalist.

AMLO's reaction to the publication of Mexico's official poverty figures was not in itself unusual. Since at least the 1970s, many Mexican presidential administrations have produced their own poverty figures by relying on their own data and calculation techniques and sometimes even their own definitions of the phenomenon.¹ Often this approach has involved rejecting not only the previous administration's data but also that produced by international development agencies or even other parts of the state. Nevertheless, at every turn, AMLO's remarks caused a swift public reaction. After the August 2020 mañanera, the national media was flooded with columns and opinion pieces extolling the virtues of Coneval and reiterating trust in the agency's work. Some commentators cataloged the merits of the methodology Coneval uses to measure poverty and the accuracy of the picture of poverty it produces. In the Twitter storm that erupted, academics, policy elites, and a handful of policymakers formed a chorus defending the agency and affirming the credibility of its poverty estimates. As one public intellectual who came out in defense of Coneval explained to me, “We defend Coneval because it has a lot of credibility within Mexican society. The poverty data are trustworthy. These are Coneval data, after all!”

Those that came to Coneval's defense praised the agency's technical merits and what they saw as its democratizing potential. In the same breath as they voiced their trust in the agency's poverty data, they reaffirmed their faith in the agency's role in democratic consolidation efforts. This sentiment was captured by one columnist who equated the attack on Coneval with a concerted effort by the president to attack the “autonomous institutions” that “protect democracy.” The public outcry following AMLO's statements thus revealed support not only for Coneval's technical work but also the technodemocratic political project in which its technical work is enmeshed—that is, a political project that combines technocratic ambitions with democratic aspirations and places faith in the capacity of expertise to bring about a more democratic future.

This article explores the success that Coneval's poverty experts have had in generating support for their work by focusing on one source of credibility on which they rely: replication.² By replication, I mean a set of ideals and practices—adopted from the field of science—designed to invite others to verify research findings and test the truth value of a calculation or claim.³ In the hands of poverty experts in Mexico, replication is not only a technique of epistemic certainty but a politically productive maneuver. As I will show, by adopting, adapting, and promoting replication, Coneval has gained allies and generated consensus among an influential, albeit limited, public that, convinced of the promise of attaining rigor and the technodemocratic goals of accountability and transparency through replication, come to the agency's defense at every challenge. Tracking Coneval's use of replication reveals how scientific ideals and practices adopted by state actors may be cast as democracy enhancing even as they produce closure and limit public participation.

Replication: Academic Antagonisms and Political Alliances

There are competing definitions of replication. What is commonly referred to as “pure replication” is an epistemic technique that involves a researcher or analyst arriving at the same findings by using the same sample population and specifications as were used in the original study (Hamermesh 2007).⁴ In practice, replication means running the same code and commands on the same data and even using the same statistical software in an attempt to verify the original results. Ideally, replication ensures epistemic certainty and contributes to scientific consensus building.

In recent years, economists and other social scientists have embraced the ideal of replication, with leading economics journals increasingly encouraging, if not requiring, authors to share their data and code and occasionally publishing replication studies (Peterson and Panofsky 2021). Replication has also become popular within evidence-based policy circles, where replication is imagined as generating better and more appropriate interventions (Brown, Cameron, and Wood 2014). For example, the International Initiative for Impact Evaluation (3ie), a research organization that promotes the use of randomized control trials in development, funds researchers to conduct replication studies of impact evaluations. Despite these efforts, the practice of replication remains rare. Replication is costly, and there are few incentives for pursuing replication studies. Issues related to data and code sharing or reproducing the original study conditions also complicate the practice (Gertler, Galiani, and Romero 2018; Peterson and Panofsky 2021).

When replication does occur, it often leads to conflict among experts and public distrust in expert findings, as evidenced by recent “crises” of replication in psychology and economics (Peterson and Panofsky 2021). As Paul Gertler and colleagues report in Nature, in economics and other social sciences, replication creates antagonism, “acrimonious debate,” and often “adversarial relationships” between original and replication researchers (Gertler, Galiani, and Romero 2018: 418). At the same time, to the degree that cases of replication failure are publicized more than cases of success, replication contributes to decreased public trust in scientists and their research (Hendricks, Kienhues, and Bromme 2020). Truth-verifying replication is especially antagonistic in its posture and, therefore, particularly prone to generating conflict (Peterson and Panofsky 2021).

In contrast, as adopted and deployed by Coneval, replication has resulted in alliances between different expert communities and trust in the work of the original (state) analysts. Coneval was founded in 2005 and in 2010 released its official method for measuring poverty, the Multidimensional Poverty Index (MPI).⁵ Since then the technocrats and civil servants who staff Coneval, as well as the methods and calculations they produce, have quickly gained a reputation for being trustworthy and accurate among a vocal group of policy elites. In a way that has surprised even its creators, the MPI has effectively replaced the use of other poverty measures for most official and many unofficial matters. Those who track and report on social policy and poverty issues, from civil society organizations to journalists and public intellectuals, have widely, and often uncritically, adopted the state agency's official measure as their own. This is a remarkable achievement considering the high levels of skepticism and mistrust in state institutions under which Coneval experts operate (Morris and Klesner 2010).

In what follows, I detail how the agency's embrace of the scientific ideal and practice of replication is key to its success. I focus on three aspects of the way replication has been adopted and adapted by Coneval. First, I track the relationship between the commitment to replication and poverty experts’ identity, self-conceptions, and self-presentation as both academics and democratic reformers working against, rather than within, the state. Second, I detail how Coneval has gone far beyond both state and academic norms of transparency, showing how the agency manages participation closely and absorbs the costs associated with replication. Third, I explore how replication moves within familiar and established expert networks, reproducing social closure and mutually beneficial ties between state and academic institutions.

Democratic Consolidation and Academics-cum-Reformers

To understand how replication became desirable and ultimately politically productive within Coneval—the Mexican state's locus of poverty expertise—we must return to the founding of the agency, its unique institutional design, and its intended purpose.

Following the historic 2000 election that ended the seven-decades-long rule of the Institutionalized Revolutionary Party (PRI), the Mexican legislature passed a raft of laws and administrative reforms to encourage transparency and accountability and purge the state of the corruption and opacity that characterized the outgoing regime. When Coneval was founded in 2005, legislators hoped that the agency would help advance these goals (Maldonado 2013; Graizbord and de Souza Leão 2022). The agency's mandate to define and measure poverty was imagined as a way to restrain the unchecked power of the executive in both the management and measurement of poverty. Poverty measurement was linked to the goals of producing a more responsive state and a more robust social development apparatus. Moreover, as part of a broader democratic consolidation effort, standardized and transparent poverty figures were cast as a tool that civil society actors might use to effectively hold policymakers accountable. In sum, poverty expertise was ascribed a set of political possibilities and became enmeshed in a technodemocratic political project.

With the goal of protecting and shoring up its political potential, legislators designated Coneval as “technically autonomous,” meaning that its civil servants and high-level administrators were to make decisions they deemed “technical” without interference. This also means that, in contrast to other agencies that remain overdetermined by political appointees and priorities, Coneval retains control over how it administers its budget and discretion over hiring and staffing decisions. Perhaps most importantly, Coneval is overseen by a “technical” advisory board composed of an interdisciplinary group of academic social scientists with expertise in social policy, statistics, or poverty who retain their faculty positions at some of Mexico's most prestigious universities. This design feature not only grants the agency a degree of legitimacy but also creates strong ties with academic institutions and direct linkages to a community of elite public intellectuals and academic experts.

Although Coneval is part of the federal government, it draws resources, capital, and a sense of identity from the academy. Indeed, the poverty experts who oversee the work of Coneval and serve as its spokespersons present themselves first and foremost as academics despite, or perhaps because of, their official roles. Consider how Agustín Escobar, a renowned Mexican anthropologist and member of the technical board of Coneval, described his role to me: “When [Coneval] began its work, we were just a group of six academic researchers and an executive director who carried out all the technical work.” Coneval experts also identify as political reformers tasked with advancing a democratic project through their technical contributions. Gonzalo Hernández Licona, the Oxford-trained economist who served as the long-time director of Coneval (2005–18), often compared the day-to-day work of the agency with the labor of “constructing democracy.”

This self-conception of academics-cum-reformers has trickled down and through Coneval. The agency's rank and file distinguish Coneval from other state agencies by referencing its unique mandate and mission. Mid-level civil servants and analysts highlight how they were drawn to the agency's technical character and “culture” of evidence, measurement, and transparency (Graizbord and de Souza Leão 2022). Many staffers I interviewed also strongly identified with the political mandate of the agency and shared a faith in the power of measurement to bring about various desirable governance outcomes.

It is within this ideational and institutional context that Coneval's commitment to replication is rooted. Agustín Escobar explains the embrace of replication as follows: “A fundamental value of academic work in Mexico and in the world is replication. This is one of the basic conditions for a work to be considered rigorous. In Coneval we retained that commitment to transparency through replication.” The appeal of replication for Escobar and his colleagues, as for scientists working within basic research institutions, lies not in practice so much as in its imagined possibilities (see Collins 1985, 1991). For Escobar and his colleagues, replication promised a chance to create a rigorous (academic) environment within the state from which they could pursue their technodemocratic mandate. In what follows, I describe how this commitment to replication is practiced.

Promoting Replication

Coneval promotes replication through a highly ritualized set of practices that go beyond the norms of transparency of other state agencies or the academic worlds that inspired them. Consider, for example, the practices of replication that accompany the yearly calculation of Mexico's poverty figures. About a week before the official poverty report is published, Coneval issues a press release announcing that the materials needed to make the official poverty calculations have been posted on its website. In the days that follow this announcement, analysts in Coneval's poverty measurement division field a flurry of phone calls and WhatsApp messages from bureaucrats, think-tank staffers, journalists, and the occasional academic asking for help as they attempt to calculate the poverty estimates in anticipation of the official report. As best and quickly as they can, these civil servants offer technical assistance, mostly by pointing callers to the technical notes and methodological documents posted on the agency's website.

Mexico's official yearly calculations are made using the nationally representative socioeconomic survey conducted by Mexico's National Institute of Statistics and Geography (INEGI). Much in the style of contemporary open governance initiatives, this large and unwieldy dataset is posted yearly in its “raw” form on Coneval and INEGI's websites. However, unlike other state agencies that perform transparency through data or document dumping, or by flaunting their technical credentials and status (Fox and Haight 2011; Matthews 2011; Richards 2021), Coneval accompanies the data it uses to estimate poverty with a vast infrastructure of materials that purportedly facilitate replication. For example, Coneval's data downloads in a zip file that includes the original survey questionnaire, documentation of the validity of each item and instrument, and a comprehensive codebook. This file also contains prewritten models, commands, and syntax to be uploaded in a statistical software program. Anticipating technical trouble, these materials offer a detailed narrative description of the steps one should take to complete the statistical analysis and technical notes explaining changes in the data collection protocol or particular issues that might lead to incompatibility with previous years’ calculations. These annual releases complement a set of materials that Coneval provides year-round: a lengthy methodological handbook that documents the construction of the MPI, including information about how the index is weighted and the formulas for calculating each of the composite indicators.

Remarkably, in addition to making its poverty knowledge infrastructures public, Coneval conducts a series of training and capacity-building events around the poverty measure. The press release for one such event I joined in 2015 describes it as an opportunity for the public to “witness how poverty figures are calculated” and ensure that “the calculation of poverty carried out by Coneval can be replicated by anyone.” On the day of the workshop, along with a group of students, academics, policy professionals, and consultants, I attended a series of seminars to learn how each of the seven indicators Coneval measures as part of the MPI are calculated. I practiced importing data into STATA, the statistical software Coneval uses to run its calculations, after the event organizers distributed a temporary license to all participants. Along with others, I ran models and commands on the dataset across various segments of the population, always checking my calculations against Coneval's official estimates. This event was just one of many. Similar in-person workshops have been hosted since 2010. In recent years, Coneval has also hosted a virtual class where participants work through material at their own pace, supported by Coneval staff who troubleshoot and answer participant queries through the course website's chat function.

These invitations to replicate the calculations conducted by Coneval are buttressed by a media and public relations strategy focused on shaping how journalists—and what the agency's communications director described as “public opinion leaders”—engage, understand, and talk about Coneval's poverty estimates and methods. This strategy includes a series of talks, seminars, and press events where the executive director and members of the board outline step-by-step procedures for calculating poverty, present an interpretation of the latest data, compare these to previous years’ estimates, and provide policy recommendations. Members of the board sit for media interviews, and staff populate the agency's YouTube channel and various social media platform feeds with engaging content, videos, and data visualizations.

In all, Coneval goes beyond academic norms of replication (e.g., asking for data and code) by making complex knowledge infrastructures public and assuming some of the costs. The state agency invests in capacity building and public relations; it targets and interpolates potential replicators; and, in effect, inculcates witnesses and potential replicators, providing instruction on the proper way of seeing and interpreting poverty data. In other words, state experts not only make replication possible, but seem to procure participation in replication. Nevertheless, the logic of replication limits participation.

Replication through a Familiar Public

Since 2012, I have tracked several controversies involving Coneval and its poverty data. These controversies have sometimes taken the form of outright rejections by politicians of the veracity of Coneval's figures, or public outcry stemming from an apparent inconsistency in reporting or measuring a particular indicator. When Coneval is challenged, op-eds appear in Mexican newspapers and on websites, and a corner of social media lights up in defense. The supporters of Coneval are familiar “public opinion leaders,” public intellectuals, and academic colleagues of the members of the advisory board. Often these supporters defend the “public” character of Coneval and its data. Consider one comment made by an economist and analyst at a prominent Mexico City think tank, in response to a perceived discrepancy in measurement across years: “The poverty data is trustworthy, and the data are public. Anyone can see that.” An academic who writes a weekly column in a national newspaper echoed this formulation. In an interview, he defended the expertise of the state agency by proclaiming it public: “What Coneval does is not part of the government or any political entity. It is all public.” For this supporter, Coneval's credibility hinged on the apparent distance between the agency and the rest of the state.

It is difficult not to notice how often its supporters directly refer to Coneval's commitment to replication in the comments, columns, and posts they produce in defense of the agency. When defenses of Coneval are rooted in replication, allies further distinguish Coneval from other parts of the state while distributing responsibility for poverty measurement onto a global public. Consider an economist who described to me why he trusted Coneval, was interested in the agency's survival, and used its estimates in his own research even when he criticized the definition of poverty the agency used: “Coneval's poverty estimates are reproducible. There's the [STATA] .do file and the data,” referring to the commands and statistical package that Coneval makes available for download or provides at workshops. After a pause, he continued, “The Secretariat of Finance and Public Credit doesn't do that for GDP growth.” A similar distinction was made by a writer and public intellectual, who in a video extolling the work of Coneval noted, “Coneval has proved itself very quickly. It is not the same to say, ‘These are data from the Secretariat of Public Education or Agriculture’ as to say, ‘these are Coneval data.’” An economist I interviewed went even further, invoking poverty data as global and internationally verified: “The multidimensional poverty measure is internationally recognized. It has been validated by experts in Mexico and internationally. The method has been replicated by experts in countries around the world. Coneval has set an example of rigor and transparency.”

While its practitioners gesture toward democratic participation (“replicated by anyone”), transparency, and openness, these controversies have evidenced how, in practice, replication moves through an already familiar and elite community of actors. Practically, replication is made possible through the extension of “knowledge infrastructures,” or materials that collect, process, and distribute data and channel attention (Hirschman 2021: 739) but demand knowledge to activate. Indeed, like with other technologies of certainty, built into replication is an idea about what makes for a capable witness and rightful public (Shapin 1994). In this case, those invited to witness and share in the work of poverty measurement and, ultimately, technodemocracy are colleagues, students, and teachers of Coneval's staff and board who populate elite academia and a carefully cultivated group of trusted journalists and organized civil society leaders. As practiced by the state experts within Coneval, replication makes use of and reproduces extant elite expert networks, social closure, and established channels that link the bureaucratic and academic fields. The fact that replication generates closure as it gestures toward transparency and openness is not solely related to the fact that, as Steven Shapin (1995: 269) argues, replication depends on “familiar others” and established “channels of familiarity.” Replication is premised on sameness and on creating like conditions, and ultimately on accepting that the original conditions are not only possible to re-create but also desirable.

The Technodemocratic Limits of Replication

When Coneval was founded and tasked with measuring and defining poverty, reformers imagined that if poverty expertise remained autonomous and protected from political pressures, it could help deliver on a number of democratizing goals. Poverty measurement was promoted as an instrument of transparency and accountability and as a means toward a more responsive and rights-guaranteeing state. The experts who oversee and staff Coneval take these technodemocratic mandates seriously, presenting the agency as a hub of technical capacity and academic knowledge at the service of progressive and positive state reform. In official statements Coneval describes its work in poverty measurement as follows: “Poverty figures are estimated using two technical elements, both public. First is [nationally representative surveys]. Second is the formula designed by Coneval, available on the agency's website. The only thing Coneval does to calculate poverty is use a formula (which is public) and apply it to the data source which is also public. In this way, anyone in the world could replicate the poverty figures, to verify that they are the same.”⁶ This official statement, just like the procedures of publicness and transparency on which Coneval's experts rely, reveals as much as it conceals (Ballestero 2012).

To be sure, state poverty experts do much more than simply apply a public formula to a public data source. By law, Coneval's experts participate in all aspects of the politics of producing poverty knowledge, including defining the phenomenon and determining the constructs and conditions of calculation. As I explored above, Coneval experts also participate in the political, sociotechnical, and cultural work of cultivating consensus while extending their own expert infrastructures and interpretive frameworks through replication. As practiced by Coneval, replication is ritualized and closely managed and goes far beyond the norms of the academic or scientific fields that inspired it. Importantly, replication makes use of channels of familiarity, strong institutional ties, and personal self-conceptions that attract a community of influential allies. In turn, these expert allies come to the agency's defense; continuing the work of replication, they defend Coneval and its data as public, verified, and trustworthy. Whether the publics they invoke actually replicate the poverty figures they defend seems not to be the point; instead, Coneval's public commitment and ritually performed invitations to replicate its calculations are what matters. Supporters are convinced by the technodemocratic promise of replication, in part, I suggest, because these ideals and practices resonate with established professional and political commitments.

The account I provide here helps elucidate how and to what effect expert practices and procedures are adopted by the state as well as how experts reproduce their capacity to inform policy and shape politics against potential challenges (Eyal 2013; Eyal and Buchholz 2010). But attention to the work of Coneval also sheds light on how technical expertise becomes enmeshed in specific political projects. In Mexico, poverty measurement is enmeshed in and enacts a technodemocratic project—that is to say, a project rooted in a powerful imaginary about the capacity of technical expertise to bring about a more democratic political order. This helps explain AMLO's distrust of the agency and dismissal of its data. When pressed by a journalist about his rejection of Coneval's official estimates during the August 2020 mañanera, the president doubled down, claiming that he had not only “other data,” but another way of conceptualizing and measuring poverty. Abandoning the language of poverty altogether, he said, “The way we measure bienestar [well-being] must be modified. I have my way of measuring. I could even boast about the results!” Read against the account I offered here, AMLO's rejection of Mexico's official poverty definition and measurement signals a refutation of the technodemocratic political project in which poverty expertise is enmeshed as well as the aspirations of poverty experts and their allies to make technodemocracy the only game in town.

I would like to thank the Institute for Advanced Study and the members of the Science and the State workshop (2020–21). A special thanks to Alondra Nelson and Charis Thompson for challenging us to think collectively about science, the state, and so much more. Thank you to Sonja van Wichelen for her leadership. This essay benefited from the generous feedback I received from Zach Griffen, Christopher Rea, and Joy Rohde as well as the editors of Public Culture. All translations from Spanish to English are my own.

Notes

References