Abstract

How does the use of evidence by policy makers differ for issues that are highly politicized compared to those that are not? Does the answer depend on whether the venue for policy making is the legislative or the executive branch? We explore these dynamics through state-level case studies of two different types of issues: what type of health insurance exchange to establish as part of the implementation of the Affordable Care Act (Idaho and Mississippi), and how to reduce infant mortality (Florida and Georgia). We highlight four sets of issues critical to understanding the use of evidence: (1) characteristics of a policy, (2) characteristics of the policy makers, (3) characteristics of the producers and disseminators of evidence, and (4) different ways evidence can be used. Barriers for academics to contribute directly to policy making change for politicized issues, particularly in the legislative branch. Under these circumstances, intermediary groups such as ideological think tanks become a trusted source of information. Policy makers themselves are a key source of evidence on less politicized issues in the executive branch. Academics wanting to inform politicized policy making need to appreciate and be comfortable with the blurry line between instrumental and rhetorical uses of evidence.

There is a rich and growing body of research on the divide between policy makers and academics. Many academics are frustrated by the barriers they face to contributing to the development of policy. Meanwhile, policy makers say either that they are overwhelmed by the amount of information available or that the research produced by academics does not answer the right questions in the right time frame (Avey and Desch 2014). How can this divide be narrowed?

John Hird (2005) assessed the state of this literature and identified a number of gaps in understanding about the use of evidence. He found that most studies provide an incomplete picture because they track the life of a single piece of research or have too narrow a conception of the ways that research can be used. He argued that a broader approach is needed that captures the importance of institutions and relationships. He also pointed out that evidence is not used in a vacuum and that it is crucial to understand political context when examining to whom policy makers look for evidence and how they use what they receive. More than ten years later, Purtle, Brownson, and Proctor (2017) argued that we still know little about how policy makers seek and use research evidence. Allen, Ruiz, and O'Rourke (2015) argued in particular that understanding how politics—specifically the politicization of an issue—affects the use of evidence by policy makers is still poorly understood and is a crucial next step in this literature.

In this article, we address these gaps by examining the use of evidence in two very different policy contexts: the development (or not) of a state-based health insurance exchange in Idaho and Mississippi, and efforts to reduce infant mortality in Florida and Georgia. The former was a highly visible and politically charged debate that involved the legislative and executive branches of a state, whereas the latter was primarily in the executive branch and outside the public's view. This approach allows us to make two types of comparisons: (1) How does the use of evidence differ for issues that are highly politicized compared to those that are not? (2) How does the use of evidence differ for a politically charged issue between the legislative and executive branches?

Our findings shed light on how context affects the use of evidence. With the literature as a guide, we organize our findings around four key aspects of the relationship between policy makers and researchers: (1) characteristics of the issue, (2) characteristics of the policy makers, (3) characteristics of who is producing and/or disseminating evidence, and (4) different ways evidence is used.

Characteristics of the Issue

Central to our research question is the notion that the nature of policies varies in key ways that may affect who is involved, what type of information is needed, and how it is used. A common typology suggests that policies differ in terms of degree of politicization and complexity (Gormley 1987). The concept of politicization can be unclear given that all policy is political. Our focus is on salience, or the degree to which the general public is aware of a particular debate (Gormley 1987; Béland, Rocco, and Waddan 2016). For example, Fowler and Gollust (2013) described a policy as more politicized or more salient when political actors are more likely to be cited in news media coverage of the issue. Wlezien (2005) clarified that, when determining the salience of an issue, we should not confuse public awareness of an issue with its actual importance.

Political salience affects who participates in policy debates and how they participate. Golden (1998) showed that nonindustry groups involved in regulatory policy are more likely to have favorable outcomes on highly salient issues. On regulatory issues that are less salient, agencies are more likely to favor the interests whose preferences most closely reflect their own (Yackee and Yackee 2006). Béland, Rocco, and Waddan (2016) showed that levels of issue salience affect whether policy makers decide to invest time and effort on an issue.

Policies also vary in complexity (Gormley 1987). Some issues are highly technical, requiring policy makers and anyone interested in contributing to the development of this policy to have a high level of specialized expertise. Other issues are relatively simple, with lower costs to participation (Meier 1991).

Salience and complexity are related concepts affecting each other. Meier (1991) described how highly complex issues have to be that much more salient to make it onto the public's radar. Even so, the terms of highly public debates are unlikely to be very technical even if an issue is complex, with stakeholders working to simplify the terms of a debate so they can move public opinion in their favor. Scholars have highly specialized knowledge and so can inform either type of policy, but whom they present it to, how they convey it, and how it is received will vary according to the public salience of the issue. The marketplace of competing ideas is likely to be more crowded in highly public debates, and the standard for what counts as evidence may be lowered.

Characteristics of the Policy Makers

Newman, Cherney, and Head (2016) suggest that, to advance our understanding of the use of evidence, we need to move beyond an overly simplistic view of a disconnect between two separate communities of policy makers and researchers. The skill and capacity of individual policy makers, along with the degree to which they are invested in an issue, have been shown to shape the level of interest in research and what they do with it (Lester 1993; Morrato, Elias, and Gericke 2007; Brownson et al. 2016; Purtle, Dodson, and Brownson 2016; Purtle, Brownson, and Proctor 2017). Two different sets of policy makers can understand and apply the same evidence differently based on differences in their background knowledge and preconceived notions about the best solutions to problems (Dobrow et al. 2006).

Policy makers in the executive and legislative branches have very different research needs. Career bureaucrats tend to have significantly higher levels of expertise than do elected officials, governors, or political appointees. As a result, they can interact with scholars and scholarly research on a more technical level. In some cases there are “pockets of expertise” in a legislature (Burns et al. 2008; Jones 2017), but in most cases legislators are generalists with specific knowledge on any given policy issue.

Another implication of this difference is that the nature of questions before elected officials differs from those considered by bureaucrats. In other words, even when working on the same topic, policy makers from the two branches are often dealing with a very different scope of the problem. Elected officials can choose the degree to which they engage on the complexity of an issue by either writing the specifics into law or setting broad guidelines and then asking an agency to fill in the details. The level of discretion given to bureaucrats varies according to key political variables, such as whether the legislative and executive branches are controlled by members of the same party (Huber and Shipan 2002).

Implied in this distinction is the observation that why an issue is on the policy agenda at the state level can vary for these two types of actors. Bureaucrats have to comply with state law and therefore have to deal with whatever issues are handed to them by their legislature or Congress. This does not mean, however, that they are simply sitting around waiting to be told what to do. Agency leaders often have significant ability—within the confines of state and federal law—to prioritize the issues they focus on and determine what to do. Our case study of infant mortality is such an example in which leaders took initiative to act rather than were reacting to a change in law.

It is also a mistake to think that state legislatures have total freedom to decide what issues they consider. The case of health insurance exchanges is an example of both the legislative and executive branches of state government needing to respond to changes in federal law.

Characteristics of the Producers and Disseminators of Information

Just as it is important to have a broad conception of who uses evidence to make policy, it is important to be clear about who is producing information. Academics at universities are just one type of researcher that may contribute evidence to inform policy making. There are also researchers at nonacademic organizations such as Mathematica and the RAND Corporation who do a mix of activities that include publishing in peer-reviewed journals and competing for government contracts to conduct studies (Brownson et al. 2006; Avey and Desch 2014; Lubienski, Scott, and DeBray 2014; Ness and Gandara 2014). Most states have nonpartisan research organizations that are creatures of the legislature and tasked with policy analysis on demand (Hird 2005). Many policy makers participate in professional associations, such as the National Association of Insurance Commissioners (Balla 2001). Stakeholder groups use evidence to advocate for specific outcomes in which they have a vested interest. Finally, think tanks serve as a middle-ground type of organization that is not quite an interest group but not quite academic.

There is a small literature on the growth of state-level think tanks, including Hird (2005), who found that Democrats at the state level were more likely to turn to think tanks than were their Republican counterparts. However, the dynamics seem to have changed in the decade since. Ness and Gandara (2014) surveyed all ideological state-level think tanks focused on education policy and found that fifty-nine of the ninety-nine such organizations across the country are conservative. They also found that the conservative think tanks are better funded, are more tightly connected to national networks, and have more robust policy activity.

The level and type of resources each group brings to a policy debate, along with the incentives that motivate them, shape how they can participate. Highly trained researchers at academic institutions have the technical expertise to contribute on complex issues but operate in a reward system that discourages involvement in politically charged issues by prioritizing publications and grants. Researchers at think tanks vary in both their level of expertise and their desire to engage on highly politicized topics. Some lean toward staying out of the fray so they can continue to be seen as politically neutral, whereas others have a higher tolerance for ideological conflict. Policy makers are sometimes confused by these differences and question the motives of researchers. This is crucial, as a number of studies show that the relationship and degree of trust significantly shape whom policy makers look to for evidence (Rich 2001; Hird 2005; Tabak et al. 2015; Brownson et al. 2016).

Complicating the picture further is the fact that the people disseminating research might not be the same people who produced it. Intermediary organizations play an important role in shaping what research makes it into a policy maker's hands and how it is spun (Jabbar et al. 2014). Morshed et al. (2017) found that advocacy organizations are more likely than legislators to be the direct consumers of academic research. Think tanks and interest groups will likely be selective about highlighting evidence that advances their agendas and downplaying evidence that does not.

Uses of Evidence

It is clear that evidence can be used in a variety of ways. One of the most commonly cited frameworks for understanding the use of evidence was developed by Carol Weiss in 1979, highlighting three different ways research is brought into policy making: (1) conceptual use, in which research provides long-term enlightenment on an issue; (2) instrumental use, in which research is applied to a specific issue during a specific instance of policy making to inform a decision; and (3) political use, in which evidence is used for tactical or symbolic purposes (Weiss 1979).

Haynes et al. (2011) reviewed the more recent literature and characterized the use of evidence in similar terms, stating that sometimes research use in policy making is (1) conceptual—not applied intentionally but permeating the policy community generally, (2) instrumental—applied to specific problems, or (3) opportunistic and rhetorical—value laden and used to advance a specific agenda. In our case studies we see examples of all three uses of evidence. Which use emerges is shaped by the interaction of the three dynamics described above: characteristics of the policy, characteristics of the policy maker, and characteristics of the producer and disseminator of information.

Methods

We used a comparative case study design to explore how policy makers used data and research to support decision making about health insurance exchanges in Idaho and Mississippi and reducing infant mortality in Florida and Georgia. We agree with Hird (2005), who argued that states are an ideal site for examining how evidence is used because they are both similar and different enough that comparisons are meaningful.

We interviewed key informants with knowledge of the policy-making process in their state, as well as experts in each field and stakeholders who were responsible for implementing the policies. Idaho and Mississippi were chosen because of similarities across key variables even though they ultimately reached different conclusions. Both are highly conservative, are led almost exclusively by Republicans, and followed a similar timeline for making their decisions. Florida and Georgia were selected based on their recent improvements in infant mortality (for more details about how we identified Florida and Georgia as states making significant improvement, see Jones and Louis 2017).

Forty-three key informant interviews were completed in this study. In Idaho and Mississippi we interviewed fourteen key informants from each state, who comprised academic researchers, state legislators, interest group representatives, reporters, and state agency leadership. We interviewed eight and seven key informants in Florida and Georgia, respectively, who comprised academic researchers, community-based program managers, interest group representatives, physicians, public health officials, and state agency leadership. We respect the privacy of our interviewees by not using identity-revealing information without their permission.

Health Insurance Exchanges

Health insurance exchanges are a key component of the coverage expansions of the Affordable Care Act (ACA). States were given a choice of developing their own exchange or ceding control to the federal government. By the time the first open enrollment began in October 2013, sixteen states had chosen to operate their own exchange, with a handful of states choosing a hybrid approach that shared responsibilities between different levels of government. States choosing to take control of their exchange had to make myriad decisions about the structure, degree of regulation, and governance.

We focus on Idaho and Mississippi. Idaho is the only state led by a Republican governor and a Republican legislature to choose to run its own exchange (Jones 2017). Mississippi is arguably the state that got the closest to establishing an exchange without doing so (Jones 2017). Both chambers of the legislature passed bills in 2011, though a conference committee could not negotiate a compromise over whether insurers were allowed to sit on the board of directors. Independently elected Republican insurance commissioner Mike Chaney took ownership of the issue after the legislature failed to act, citing the statute governing the state's high-risk pool to say that he had the authority to create an exchange without new legislation. The Mississippi insurance department established a task force of key stakeholders to study and prepare for the launch of an exchange. Information technology was developed and was in the testing phase when Republican governor Phil Bryant announced in early 2013 that he would block his state from running an exchange.

Infant Mortality

Infant mortality is a major public health problem in the United States. More than 23,000 infants died before the age of one in 2014 (United Health Foundation 2015). Using a rate per 1,000 live births as a comparison, the US infant mortality rate (IMR) is twenty-sixth among Organization for Economic and Development countries. Despite the fact that the United States spends far more money on health care than any other industrialized nation in the world, it continues to have infant mortality outcomes worse than poorer countries such as Slovakia and Hungary (MacDorman et al. 2014). A 2014 article in the Washington Post called infant mortality in the United States “a national embarrassment” (Ingraham 2014). The IMR also varies dramatically within the United States, from 4.2 deaths per 1,000 live births in Massachusetts to 9.3 in Mississippi (United Health Foundation 2015). Georgia and Florida made dramatic improvement in their IMR between 2004 and 2014 (Jones and Louis 2017).

Limitations

Similar to other exploratory studies of this nature, this study is not without limitations. Even so, our approach is consistent with what Hird (2005) and others have called for in terms of greater reliance on comparative analyses rather than single-case studies, suggesting that these limitations are an acceptable trade-off.

First, the data for each issue were collected during two different time periods. However, we believe that by collecting our information in a close proximity to the height of the policy issue, we were better able to capture key informants' true feelings about the issue. This approach also reduces recall bias. And while our interview guides may not have been exactly the same, they did have significant conceptual overlap.

Second, our data may reflect some degree of asymmetry of information bias since we were not able to speak with everyone in these four states who may have been germane to the issues. This may lessen our ability to fully understand all perspectives on the approaches that were taken. Nonetheless, our analysis suggested that we reached saturation points at each of the case sites, supporting that substantial new information was not likely to have been learned by doing additional interviews.

Finally, key informants were interviewed as reflections of their states and related processes but may introduce their own personal biases that do not necessarily reflect how the state truly functioned. Despite this possibility, there was considerable consistency across responses to similar interview questions for each policy issue.

Findings

We now turn to our four case study states to examine the use of evidence. We use the same framework from our literature review to analyze how key differences in the nature of our two issues shaped the use of evidence, with particular emphasis on the effect of politicization. We further develop this analysis to examine key differences in the characteristics of policy makers making these decisions and in the people/organizations producing and disseminating evidence. We then use Weiss's (1979) framework to discuss the different types of uses of evidence.

The Issues

Health insurance exchanges not only were politicized in their own right but also were seen as part of a highly public debate about health reform more broadly (Béland, Rocco, and Waddan 2016). The ACA was the subject of multiple Supreme Court cases and one of the most prominent issues in major elections (Cho, Deane, and Brodie 2012). Infant mortality is certainly an important issue in public health terms, but there is no evidence that it was salient as an issue to the same degree.

Multiple people in our states lamented that the exchanges became conflated with broader Obamacare politics. As former Mississippi governor Haley Barbour told us, “I do believe the exchanges would have gone more smoothly and happened earlier if people had understood that these are two entirely separate decisions, that you could have an exchange and it would have no effect on Medicaid. But you have to be an expert to understand that” (interview, January 2014).

Closely related to the degree of politicization is the fact that health insurance exchanges and infant mortality were on the policy agenda of our four states for very different reasons, thus shaping what evidence was available and how it would be used. Interest in an exchange in Idaho and Mississippi predated the ACA's passage, but the ACA forced state leaders to consider an exchange under different circumstances and with significantly greater public scrutiny than they would have otherwise. Implementation of health insurance exchanges was effectively thrust upon states, with a set of decisions and timelines imposed by federal law and overseen by federal agencies. By contrast, there was no federal statute compelling Florida and Georgia to take up the issue of infant mortality. The issue only made it onto these states' health policy agenda because of a decision by leaders to make it a priority.

These two issues differed dramatically in terms of the availability of data and evidence. Policy makers in Idaho and Mississippi had little evidence they could turn to in considering whether to establish an exchange. Massachusetts and Utah were the only states with an exchange prior to the ACA, and neither was fully compliant with the ACA's requirements. The final decision in Idaho and Mississippi on whether to run an exchange was made in early 2013. By that point more than a dozen other states had decided to establish an exchange rather than default to the federal government. But none of these exchanges went live until October 1, 2013, and so could not provide evidence on outcomes.

On the other hand, leaders in the Florida and Georgia executive branches extensively used data to design and target new infant mortality programs. In both states, “in-house” data systems were used to a greater extent for producing, analyzing, and reporting data. They had years of data to draw from to help them understand the problem and target solutions. A major difference in this case was that one of the key uses of research in these two states was to convince others that the scope of the problem merited policy attention.

The Policy Makers

The nature of the decision and the types of information needed changed dramatically in both states when the process turned from the legislature to the executive branch agencies. The decision was no longer whether or not to do an exchange—that was effectively decided for them—but how to do it. Politicians were more often described as seeking evidence to support the conclusions they had already come to, whereas bureaucrats were described as seeking evidence to inform what decision to make. Their questions were more technical. Politicians were more interested in political learning, such as understanding what their peers in other states were doing and whether they risked being punished or rewarded by the national party. The timing challenges highlighted above meant that in some ways the best information available to legislators was the political fate of candidates in other states who had supported or blocked an exchange.

What we saw in Idaho and Mississippi matches the literature, which says that legislators vary significantly in terms of their level of expertise and interest on any particular topic. This was true even for legislators with a prominent role in health policy. For example, a legislative committee chair we interviewed in Mississippi seemed to have a very poor grasp of the issue (interview, January 2014). Similarly, an Idahoan legislator who focused considerable attention on the exchange said that the eyes of his colleagues glaze over with boredom once he starts talking about an exchange (interview, December 2011). Even so, a notable element of the legislative process in Idaho was the presence of pockets of expertise (Burns et al. 2008; Jones 2017). These were legislators who had developed particular clout on health policy issues because of their professional experience as insurance brokers, physicians, or insurance company executives. Their understanding of the issue had a ripple effect as they were regularly looked to by their peers as trusted sources of information, particularly by other members within their party.

Agency officials had more expertise and different evidence needs, but they were not passive consumers of information or apolitical actors. The leaders of the insurance department in both Idaho and Mississippi were seen by others in their states as politically savvy and skilled at navigating the many competing demands inside and outside government. Idaho insurance commissioner Bill Deal had served many years in the legislature before moving to the executive branch. Mike Chaney in Mississippi was among eleven insurance commissioners in the country who are independently elected rather than appointed.

Leaders in Idaho hoped to use the later timing of their decision to their advantage to learn from the process in other states. Once legislation was passed in March 2013 they realized they would not have time to build an exchange before open enrollment began six months later. As one member of the Your Health Idaho board of directors described, “Yes we're going to be in a time crunch, but a lot of other states have done a lot of footwork that we're going to be able to use. Why reinvent the wheel?” (interview, April 2013). Multiple policy makers said that at that point they anticipated being able to use the information technology platforms developed in other states.

By contrast, decision making on infant mortality was largely driven by agency leaders. The first answer to the question of how to address infant mortality was to convince others that there was a major problem and that it could be tackled. Georgia commissioner of public health Brenda Fitzgerald, who has since been appointed director of the national Centers for Disease Control and Prevention, described in her 2016 State of the State's Health speech how she chose to prioritize infant mortality. She explained that shortly after taking office in 2013 she dug into the “America's Health Rankings” (United Health Foundation 2015). As one informant told us, “You can't have 97 priorities. So, looking at the ‘America's Health Rankings’ was really the first thing” (interview, October 2016). Fitzgerald's review of the data showed that Georgia had consistently been ranked among the bottom 25 percent of US states in infant mortality. This alarmed her as a practicing OB-GYN physician. She used these trends to highlight the need for change and began promoting this to state leaders and prominent constituents.

The legislatures of Florida and Georgia were briefly involved in infant mortality, but in ways different from how legislators addressed exchanges. In 2007 the Florida legislature passed a bill creating the Black Infant Health Practice Initiative. The bill was passed unanimously and out of the public's eye. One interviewee described initiative as “a focus explicitly on African-American infant mortality . . . used particularly to support initiatives in some of the major urban communities that had some higher infant mortality rates” (interview, October 2016). This collaborative was also specifically created to examine medical and social factors contributing to disparities in IMR and to serve as a platform for recommending local and state policy changes (REACHUP, Inc. n.d.). Rather than assert its role on the issue, however, the legislature empowered the executive branch to lead.

The Georgia legislature came into play on infant mortality only indirectly through the setting of the Department of Public Health (DPH) budget. Infant mortality was among the issues highlighted by Commissioner Fitzgerald in her arguments for increased resources. Her position was dramatically strengthened by a structural change in the state's organizational chart that elevated DPH to a cabinet-level agency rather than subsumed within another entity. This restructuring allowed Dr. Fitzgerald a more prominent voice before the legislature in negotiating for funding. Once again, though, the decision to focus on infant mortality and the details on the best way to do that were largely left to the executive branch.

The Producers and Disseminators of Evidence

Academics were rarely cited as a source of evidence on either issue. An exception was in Georgia, where a team of professors at Emory University was contracted to evaluate a program targeting infant mortality. Instead, information came to these policy makers from five main sources. First, state leaders were themselves often the primary source of evidence. This was particularly true in the executive branch compared to the legislative branch and was even more common for infant mortality than for the exchanges. In both Georgia and Florida, state-collected data was the main source of information for understanding the problem and targeting solutions.

For example, leaders at the Georgia DPH have gone to great lengths to improve the quality of their data systems. The Online Analytical Statistical Information System (OASIS) is the main publicly available mechanism for collecting, aggregating, and accessing data on infant mortality. One Georgia state leader described OASIS by saying, “It's an excellent source of data which you can go in and track. It's going to obviously be, like every other data system for birth certificates, one and a half years to two years behind, but you've got all the infant mortality by age, by area, by county. You can identify whatever you need. And that for the most part is what I use to track the data” (interview, October 2016).

Georgia has used its improved data systems to perform targeted analyses identifying “hot spots” around the state where IMRs were higher. Dr. Fitzgerald worked with researchers in the Georgia DPH to geocode counties to identify the areas with the highest IMRs. Once these counties were identified, they examined the top causes of infant mortality in those areas. Potential strategies were developed to target those specific issues in those specific parts of the state. These targeted data were used to obtain local buy-in. One informant described this process by saying, “Dr. Fitzgerald went down there and met with those OB-GYNs with their local coalition. Here is your data here. Here is what is going on here. So then they know what to do. I mean that is the difference. Public Health's role is, because when you're sitting in your office you're seeing one patient at a time, so Public Health's role is to see the landscape and then to translate that information about the landscape down to those people seeing one person at a time so they know what to look for” (interview, October 2016).

Second, interest groups were a major source of evidence in all four states. In each case, on both the issues of an exchange and infant mortality, key groups were brought together through the creation of formal entities so that state leaders could learn from key stakeholders. All four of these networks served a dual purpose of learning what stakeholder groups thought was best in an objective sense, but also what they would be willing to support. Winning buy-in was as important an objective as gathering evidence.

These formal networks took slightly different forms but in all four states were led by a leader in the executive branch. Governor Butch Otter's office in Idaho coordinated an alliance of four hundred stakeholder groups supporting the creation of a state-based health insurance exchange. Mississippi insurance commissioner Mike Chaney convened an advisory board of approximately a dozen leaders representing key stakeholders to make decisions about an exchange at monthly meetings. The Georgia DPH initiated and participated in multiple collaboratives to bring leaders together from around the state, including the Georgia Perinatal Quality Collaborative—a network of perinatal health care providers, public health leaders, and community partners—and the Georgia Maternal Mortality Review Committee, begun in 2012 to review cases of maternal deaths and make recommendations on ways to improve care. Similarly, Florida's approach was based around bringing together and empowering local leaders around the state.

Third, consultants were a major provider of information that existed only in our exchange states. Consultants played a more prominent role than academics because they could be hired to tackle targeted questions on a truncated timeline and brought a client-driven approach that compelled them to organize data in a way that was useful for policy makers. State leaders could not hire expensive consultants in the same way to work on infant mortality because they did not have the same source of federal grants.

Mississippi made a strategic choice to outsource as many functions as possible, to leverage the expertise gained by consultants working in multiple states and to keep their bureaucracy small. For example, Mississippi's Exchange Advisory Board was led by Leavitt Partners. Milliman, an international consulting firm, produced a report in 2013 that was cited by multiple people involved in Mississippi's debate. One agency leader said the procurement process itself was valuable because they even learned from many of the bidders who were not selected for a particular request for proposal.

State-level think tanks were a fourth provider of information. A prominent conservative advocacy organization operates in both Idaho and Mississippi, neither with a liberal analogue. The Idaho Freedom Foundation and the Mississippi Center for Public Policy regularly produced reports, wrote blog posts, and testified at legislative hearings in opposition to an exchange. They cited statistics and anecdotes that carried the weight of evidence to people that agreed with them. Sometimes policy makers were the indirect audience as think tank outreach targeted Tea Party activists. As a staff person in one of these conservative think tanks described, “We are ammo suppliers, for lack of better words. We provide ammunition [to the Tea Party], we provide information. We provide details and research” (interview, April 2013). Like consultants, think tanks were prominently involved on the highly politicized issue of an exchange but not on infant mortality. They sometimes produced their own information but were often more likely to interpret and disseminate information produced by others.

Finally, policy makers in all four states relied on national networks for information. Elected officials on the politically charged issue of an exchange tended to pay close attention to national networks of their peers, such as the Republican Governors Association. An adviser to the governor in one of our states discussed how they participated in weekly Republican Governors Association conference calls focused on the ACA to learn from other states and strategize. Other informants described the political information shared through these channels as among the most important factors influencing whether Mississippi in particular created an exchange.

Bureaucrats had their own networks. For example, agency officials working on an exchange in Idaho and Mississippi described learning from their peers at meetings such as those hosted by the National Academy for State Health Policy (NASHP) or on a NASHP-run e-mail list called The Exchangers. An official in Mississippi said they especially benefited from looking to the committees run by the National Association of Insurance Commissioners because “you can trust that the information has been pro'd and con'd to death” (interview, October 2012).

The federal government was an important convener on both the health insurance exchanges and infant mortality. For example, the US Department of Health and Human Services (HHS), in particular the Center for Consumer Information and Insurance Oversight (CCIIO), hosted regular meetings in person and over the phone with state leaders working on an exchange. Many agency staff from Idaho and Mississippi described the data used and key findings and lessons shared at these meetings as critical to their efforts to establish an exchange.

Similarly, the Health Resources and Services Administration, an agency of HHS, supports voluntary Collaborative Improvement and Innovation Networks (CoIINs), of which both Florida and Georgia are participants. The infant mortality CoIIN is a network comprising federal, state, and local community stakeholders from all fifty states that aims to “pull state health departments together with their partners to try and develop comprehensive strategies to address infant mortality and to promote sharing and innovation across health departments” (interview, October 2016). One state agency leader from Georgia described the value of joining the CoIIN: “Number one, focus: it gave some focus as to how we were going to approach this [high IMR]; number two, it also allowed us to basically share information with other states and find out best practices from that standpoint. . . . I think, that played a major role” (interview, October 2016). Similarly, Florida leaders described that their strategic framework for infant mortality did not exist prior to participating in the CoIIN.

An important spillover effect from participating in these formal multistate organizations is that policy makers developed strong informal networks that they could tap into whenever they needed perspective. Someone in the Mississippi Department of Insurance described attending NASHP and CCIIO meetings, saying “You started seeing the same faces in these things,” which made it easier to develop relationships (interview, November 2011). Someone in the Idaho Department of Insurance described a similar effect, where a “loose-knit group” of people has evolved into an e-mail group that regularly communicates online (interview, June 2011).

Many of these relationships transcended geography. An adviser to Governor Otter in Idaho described being in regular contact with leaders in Vermont to seek advice about working with a particular vendor. A leader in Mississippi described regularly seeking advice from leaders in Oregon. Multiple people pointed out that the insurance commissioners of Idaho and Mississippi were close friends and that they regularly shared information, particularly after Mississippi had decided to reject an exchange and Idaho was struggling to get its together.

Uses of Evidence

We observed all three types of uses of evidence described by Weiss (1979). Conceptual use is by its nature difficult to pin down because it describes ways in which evidence permeates a policy conversation generally. We observed many instances of evidence being used instrumentally, in other words, applied in a specific way to a specific problem. This tended to be more common for executive branch officials working on more technical questions, such as which part of the state has the highest IMR, or with respect to the development of information technology for an exchange.

One of the most challenging aspects of our analysis was drawing the line between when evidence use was instrumental or symbolic. However, this is more important as a theoretical insight than a methodological challenge. Policy makers are often strategic about which information they look to and how they use it. Symbolic uses of evidence—otherwise described by Weiss (1979) as opportunistic and value laden to support a specific agenda—are usually made to appear instrumental.

The degree of an issue's politicization largely shaped the extent to which information was trusted. In the politically charged fight over an insurance exchange, it often mattered more who produced evidence than what the evidence said. To some extent, this was a function of the stage of the decision-making process and who was making the decision. Bureaucrats working on an exchange were able to incorporate data from a wide variety of sources, whereas we observed a striking degree of confirmation bias among politicians. They regularly dismissed evidence that did not fit the conclusions they had already drawn while touting evidence that did. For example, a number of conservatives referenced the 2013 Milliman report to suggest a litany of negative outcomes if Obamacare components were implemented in Mississippi. The leader of a consumer advocacy organization responded that “the Milliman study is a real joke” because of unrealistic assumptions that elevated the cost to the state (interview, January 2014).

The Mississippi Milliman report actually focused on Medicaid expansion but was cited by multiple people in the context of an exchange, exemplifying the earlier point that the debates were conflated because of the intensity of Obamacare politics. This also typified confusion among many policy makers about basic facts surrounding the choice they faced, such as whether giving up state control of an exchange meant people could not receive premium tax credits, whether people would be put in a national high-risk pool, or whether consumers would have to use brokers from other states to help them buy insurance. Answers to these questions were available from the Obama administration and health policy researchers, but many conservative policy makers did not trust these sources.

On the other hand, ideologically driven think tanks and issue-driven stakeholders were selectively trusted by legislators. Legislators who wanted to believe what they heard from these conservative think tanks did not easily change their mind when presented with other sources of information. And there were lots of other types of organizations bringing their perspective to the debate, always with evidence of their own. One legislator in Idaho estimated that at least three-fourths of the lobbyists working in the 2013 session were employed to focus on the exchange (interview, May 2013). The same dynamic did not exist for infant mortality in Florida and Georgia.

Finally, policy makers were themselves producers and disseminators of information, particularly in the executive branch and on the less politicized issue of infant mortality. They are not merely consuming information that is brought to them but are strategically putting out information that advances their own agenda. Sometimes this dissemination occurs through the formal networks of stakeholders brought together to work on a given topic, and sometimes it occurs more broadly to the general public.

Conclusion

Our analysis provides substantial support for Hird's (2005) argument that it is not possible to understand how policy makers access information by tracking the trajectory of a single piece of evidence. One reason is that it is incredibly difficult—maybe impossible—in most cases to attribute final policy decisions to insights gained from any individual piece of evidence. More broadly, it is clear that evidence can be used in many different ways, and how evidence is used is shaped by key contextual variables, such as whether the policy is politicized or not, whether the policy is under consideration by the legislative or executive branch, and who is producing and disseminating this information.

It is difficult to generalize from a comparison of four states. Even so, we offer some insights from this analysis that may hold up under other circumstances. We address these points with an eye to the broad question raised at the outset of whether academics can bridge the gap with policy makers and increase the likelihood that their research is used.

First, our findings suggest that, rather than treating policy making as a mysterious black box, researchers wanting to contribute evidence need to develop a clear understanding of the structure and processes of government. Academics may be better able to contribute to policy making when an issue is less politicized and when the venue is the executive rather than the legislative branch. Their level of expertise is a closer match to that of bureaucrats than legislators. The issues tend to be more technical in the executive branch and a better fit for the types of questions academics have explored. There may be little that can be done to overcome the systemic barriers to contributing to policy development in the legislative phase of a highly political issue. The confirmation bias of legislators is such that getting high-quality peer-reviewed evidence in their hands is unlikely to sway them compared to less rigorous information from sources they trust.

The groups that had the greatest ability to inject evidence into the policy-making process were the legislators with particular expertise and interest, leaders in the executive branch, consultants, think tanks set up for rapid-cycle analysis and that were already connected to policy makers, and interest groups. Researchers wanting to disseminate their conceptual and instrumental evidence at the state level may have greater success by targeting these actors as vehicles. The same systemic barriers will likely persist, including asking the right questions at the right time and providing clear and helpful answers. But these people may be in the best position to understand, apply, and disseminate research.

Symbolic and rhetorical use of evidence was more commonly present on the highly politically charged issues and by legislators rather than executive branch officials. But legislators aside from the pockets of expertise did not seem to see this distinction. They experienced most uses of evidence as instrumental in that they believed they were applying the best evidence to the question at hand rather than making rhetorical arguments. There are few checks on the quality and rigor of evidence used in politicized debates, particularly in the legislature. There was no indication that these policy makers recognized the confirmation bias of prioritizing information obtained from sources they already trusted. Researchers may have more success sharing information in politically charged debates at the legislative level if they are willing to shift from instrumental to rhetorical types of evidence, though this may not be a comfortable or appropriate role for them.

Finally, our insights bring to the surface key dynamics in the use of evidence that should be further explored in subsequent research. One is the important role that consultants played in the development of health insurance exchanges. They were contracted by a state's executive branch to provide evidence and answers to specific questions. In our study consultants were used on the highly politicized issue of an exchange but not on the less salient issue of infant mortality. It does not necessarily follow that consultants can never be involved in politicized issues. There are also numerous instances of politically salient issues in which consultants play no role. The key factor is whether the federal government sets aside grant money for states to use in this way. More research is needed to understand the circumstances driving this decision generally.

Similarly, it will be important to better understand the role that ideological think tanks play as intermediary organizations in the production and dissemination of evidence. Do the issues drive the network of relationships between policy makers and these groups, or do the networks shape the issues? In both of our cases the issue was placed on the policy agenda by policy makers, at either the federal or state level, but this might not be true under other circumstances. More research is also needed to better understand the dynamics at play when policy makers shift from being the consumers of evidence to the producers and disseminators of information they want others to have (Rich and Weaver 1998).

Taken together, our findings suggest that the use of evidence is as complicated as the policy-making process itself. The degree to which an issue is politicized shapes the way evidence is used, but not in uniform ways. How and when research is used to inform policy depends on what the decision is to be made and who is empowered to decide.

David K. Jones is assistant professor at Boston University's School of Public Health. His research examines the politics of health reform. His recent book, Exchange Politics: Opposing Obamacare in Battleground States, examines how states made decisions about what type of health insurance exchange to establish as part of the Affordable Care Act's implementation. He is working on a new book examining the social determinants of health in the Mississippi Delta. He is a past recipient of the AcademyHealth Outstanding Dissertation Award, the AUPHA Thompson Prize for Young Investigators, and the BUSPH Excellence in Teaching Award.

dkjones@bu.edu

Christopher J. Louis is assistant professor in the Department of Health Law, Policy, and Management at the Boston University School of Public Health. His primary research interests reside in health care organization and delivery, health policy, state and federal innovation programs, and program evaluation. He earned his PhD from The Pennsylvania State University in health policy and administration. He also holds an MHA from the University of Florida and a BS in business administration from Sacred Heart University.

We gratefully acknowledge funding from the Millbank Memorial Fund for the interviews in Florida and Georgia on infant mortality, as well as the research assistance of Andrea Clark, Stephanie Ettinger de Cuba, and Kevin Griffith.

References

References
Allen
Sean T.
,
Ruiz
Monica S.
, and
O'Rourke
Allison
.
2015
. “
The Evidence Does Not Speak for Itself: The Role of Research Evidence in Shaping Policy Change for the Implementation of Publicly Funded Syringe Exchange Programs in Three US Cities
.”
International Journal of Drug Policy
26
, no.
7
:
688
95
.
Avey
Paul C.
, and
Desch
Michael C.
2014
. “
What Do Policymakers Want from Us? Results of a Survey of Current and Former Senior National Security Decision Makers
.”
International Studies Quarterly
58
, no.
2
:
227
46
.
Balla
Steven J.
2001
. “
Interstate Professional Associations and the Diffusion of Policy Innovations
.”
American Politics Research
29
, no.
3
:
221
45
.
Béland
Daniel
,
Rocco
Philip
, and
Waddan
Alex
.
2016
.
Obamacare Wars: Federalism, State, and the Affordable Care Act
.
Lawrence
:
University Press of Kansas Press
.
Brownson
Ross C.
,
Royer
Charles
,
Ewing
Reid
, and
McBride
Timothy D.
2006
. “
Researchers and Policymakers—Travelers in Parallel Universes
.”
American Journal of Preventive Medicine
30
, no.
2
:
164
72
.
Brownson
Ross C.
,
Dodson
Elizabeth A.
,
Kerner
Jon F.
, and
Moreland-Russell
Sarah
.
2016
. “
Framing Research for State Policymakers Who Place a Priority on Cancer
.”
Cancer Causes and Control
27
, no.
8
:
1035
41
.
Burns
Nancy
,
Evans
Laura
,
Gamm
Gerald
, and
McConnaughy
Corrinne
.
2008
. “
Pockets of Expertise: Institutional Capacity in Twentieth-Century State Legislatures
.”
Studies in American Political Development
22
, no.
2
:
229
46
.
Cho
Sarah
,
Deane
Claudia
, and
Brodie
Mollyann
.
2012
. “
What Issues Are Most Important to Voters This Election? The Answer Depends on the Question
.”
Kaiser Family Foundation
,
November
1
. www.kff.org/health-reform/perspective/what-issues-are-most-important-to-voters-in-this-election-the-answer-depends-on-the-question/.
Dobrow
Mark J.
,
Goel
Vivek
,
Lemieux-Charles
Louise
,
Black
Nick A.
2006
. “
The Impact of Context on Evidence Utilization: A Framework for Expert Groups Developing Health Policy Recommendations
.”
Social Science and Medicine
63
, no.
7
:
1811
24
.
Fowler
Erika
, and
Gollust
Sarah
.
2013
. “
The Content and Effect of Politicized Health Controversies
.”
Annals of the American Academy of Political and Social Science
658
, no.
1
:
155
71
.
Golden
Marissa Martino
.
1998
. “
Interest Groups in the Rule-Making Process: Whose Voices Get Heard?
Journal of Public Administration
8
, no.
2
:
245
70
.
Gormley
William T.
1987
. “
Institutional Policy Analysis: A Critical Review
.”
Journal of Policy Analysis and Management
6
, no.
2
:
153
69
.
Haynes
Abby S.
,
Gillespie
James A.
,
Derrick
Gemma E.
,
Hall
Wayne D.
,
Redman
Sally
,
Chapman
Simon
, and
Sturk
Heidi
.
2011
. “
Galvanizers, Guides, Champions, and Shields: The Many Ways That Policymakers Use Public Health Researchers
.”
Millbank Quarterly
89
, no.
4
:
564
98
.
Hird
John
.
2005
.
Power, Knowledge, and Politics
.
Washington, DC
:
Georgetown University Press
.
Huber
John D.
, and
Shipan
Charles R.
2002
.
Deliberate Discretion? The Institutional Foundations of Bureaucratic Autonomy
.
Cambridge
:
Cambridge University Press
.
Ingraham
Christopher
.
2014
. “
Our Infant Mortality Rate Is a National Embarrassment
.”
Washington Post Wonkblog
,
September
29
. www.washingtonpost.com/news/wonk/wp/2014/09/29/our-infant-mortality-rate-is-a-national-embarrassment/?utm_term=.7535c36d66e5.
Jabbar
Huriya
,
La Londe
Priya Goel
,
Debray
Elizabeth
,
Scott
Janelle
, and
Lubienski
Christopher
.
2014
. “
How Policymakers Define ‘Evidence’: The Politics of Research Use in New Orleans
.”
Policy Futures in Education
12
, no.
8
:
1013
27
.
Jones
David K.
2017
.
Exchange Politics: Opposing Obamacare in Battleground States
.
Oxford
:
Oxford University Press
.
Jones
David K.
, and
Louis
Christopher J.
2017
. “
State Population Health Strategies That Make a Difference: Reducing Infant Mortality in Georgia and Florida
.”
Millbank Memorial Fund
.
October
12
. www.milbank.org/publications/state-population-health-strategies-make-difference/.
Lester
James P.
1993
. “
The Utilization of Policy Analysis by State Agency Officials
.”
Science Communication
14
, no.
3
:
267
90
.
Lubienski
C.
,
Scott
Janelle
, and
DeBray
Elizabeth
.
2014
. “
The Politics of Research Production, Promotion, and Utilization in Educational Policy
.”
Educational Policy
28
, no.
2
:
131
44
.
MacDorman
Marian F.
,
Matthews
T. J.
,
Mohangoo
Ashna D.
, and
Zeitlin
Jennifer
.
2014
. “
International Comparisons of Infant Mortality and Related Factors: United States and Europe
.”
National Vital Statistics Report
63
, no.
5
. www.cdc.gov/nchs/data/nvsr/nvsr63/nvsr63_05.pdf.
Meier
Gerald
, ed.
1991
.
Politics and Policy Making in Developing Countries: Perspectives on the New Political Economy
.
San Francisco
:
ICU Press
.
Morrato
Elaine H.
,
Elias
Melinda
and
Gericke
Christian A.
2007
. “
Using Population-Based Routine Data for Evidence-Based Health Policy Decisions: Lessons from Three Examples of Setting and Evaluating National Health Policy in Australia, the UK and the USA
.”
Journal of Public Health
29
, no.
4
:
463
71
.
Morshed
Alexandra
,
Dodson
Elizabeth A.
,
Tabak
Rachel G.
, and
Brownson
Ross C.
2017
. “
Comparison of Research Framing Preferences and Information Use of State Legislators and Advocates Involved in Cancer Control, United States, 2012–2013
.”
Preventing Chronic Disease
14
, no.
E10
.
Ness
Eric C.
, and
Gandara
Denisa
.
2014
. “
Ideological Think Tanks in the States: An Inventory of their Prevalence, Networks, and Higher Education Policy Activity
.”
Educational Policy
28
, no.
2
:
258
80
.
Newman
Joshua
,
Cherney
Adrian A.
, and
Head
Brian W.
2016
. “
Do Policy Makers Use Academic Research? Re-examining the ‘Two Communities’ Theory of Research Utilization
.”
Public Administration Review
76
, no.
1
:
24
32
.
Purtle
Jonathan
,
Brownson
Ross C.
, and
Proctor
Enola K.
2017
. “
Infusing Science into Politics and Policy: The Importance of Legislators as an Audience in Mental Health Policy Dissemination Research
.”
Administration and Policy in Mental Health and Mental Health Services Research
44
, no.
2
:
160
63
.
Purtle
Jonathan
,
Dodson
Elizabeth A.
, and
Brownson
Ross C.
2016
. “
Uses of Research Evidence by State Legislators Who Prioritize Behavioral Health Issues
.”
Psychiatric Services
67
, no.
2
:
1355
61
.
REACHUP, Inc.
n.d. “
Black Infant Health Practice Initiative
.” www.reachupincorporated.org/bihpi (accessed on
April
20
,
2017
).
Rich
Andrew
.
2001
. “
The Politics of Expertise in Congress and the News Media
.”
Social Science Quarterly
82
, no.
3
:
583
601
.
Rich
Andrew
, and
Weaver
Kenneth
.
1998
. “
Advocates and Analysts: Think Tanks and the Politicization of Expertise in Washington
.” In
Interest Group Politics
,
5th
ed., edited by
Cigler
Allan
and
Loomis
Burdett
,
235
53
.
Washington, DC
:
Congressional Quarterly
.
Tabak
Rachel G.
,
Eyler
Amy A.
,
Dodson
Elizabeth A.
,
Brownson
Ross C.
2015
. “
Accessing Evidence to Inform Public Health Policy: A Study to Enhance Advocacy
Public Health
129
, no.
6
:
698
704
.
United Health Foundation
.
2015
. “
America's Health Rankings
.” www.americashealthrankings.org/explore/2015-annual-report/measure/IMR/state/ALL.
Weiss
Carol
.
1979
. “
The Many Meanings of Research Utilization
.”
Public Administration Review
39
, no.
5
:
426
31
.
Wlezien
Christopher
.
2005
. “
On the Salience of Political Issues: The Problem with ‘Most Important Problem.’
Electoral Studies
24
:
555
79
.
Yackee
Jason Webb
, and
Yackee
Susan Webb
.
2006
. “
A Bias toward Business? Assessing Interest Group Influence on US Bureaucracy
.”
Journal of Politics
68
, no.
1
:
128
39
.