Congress created the Patient Centered Outcomes Research Institute (PCORI) to fund comparative effectiveness research without encroaching on health care decision making in the private sector. This study asked if the organization's design is sufficient to insulate it from the hostile political environment that accompanied past comparative effectiveness research efforts. Data for the study came from key informant interviews, stakeholder interviews, content analysis of public comments, congressional hearings, and media and Internet content about PCORI. Drawing on theoretical frameworks of interest group behavior, the study assessed current and potential future stakeholder activity directed toward PCORI. The study found that PCORI's leadership has successfully mobilized patients and researchers in support of its mission. However, patient groups tend to mobilize within rather than across disease categories, limiting the collective impact these groups might have. Moreover, PCORI's success in including the patient voice in every stage of the research process has created only diffuse support for the organization. A lack of “practice-changing” findings—likely the result of the organization's interest group environment—leaves PCORI open to the criticism of ineffectiveness.
The 2010 health care reform in the United States achieved a laudable goal of significantly expanding health insurance coverage (Cohen and Martinez 2014; McMorrow et al. 2015; Sommers et al. 2013). At the same time, the Affordable Care Act (ACA) contained fewer mechanisms to address the problem of rising health care costs. Though other Organisation of Economic Co-operation and Development (OECD) countries employ a variety of policy tools to manage costs—for example, global budgets, state-owned hospitals, salaried doctors (Chalkidou et al. 2009)—most are unpopular in the United States given the American aversion to government regulation of the health care sector. One such mechanism, comparative effectiveness research (CER), which can be used to incentivize the use of treatments proven to be effective, did make its way into the ACA in the form of the Patient Centered Outcomes Research Institute (PCORI) (Ashton and Wray 2013; Sorenson, Gusmano, and Oliver 2014).
Though CER has received bipartisan support in the past (Ashton and Wray 2013; Wilensky 2006), and its current incarnation is designed to prevent the government from shaping physician decision making, PCORI's viability rests on its ability to mobilize stakeholders who can defend it against hostile provider associations and industry groups who view federally supported CER as laying the foundation for government rationing of health care. Using theoretical frameworks from political science to predict the relationships among agency performance, statutory authority, and stakeholder dynamics, we analyzed current stakeholder views of and activities surrounding PCORI among patient groups and professional and trade associations to assess the likelihood that PCORI can survive the difficult politics of CER in the US context. Though PCORI has no formal regulatory power and is designed to be a classically “distributive” organization, we found evidence that the agency is experiencing “regulatory” politics. Moreover, the organization has not been able to avoid the classic conundrum of CER politics. Pursuit of CER studies that have the potential to guide physicians and patients away from high-cost/low-value care is likely to mobilize powerful stakeholders against the organization. At the same time, steering clear of such cost drivers opens the organization to arguments that it is wasteful and has not contributed to a federal health research portfolio that is better managed by more established agencies.
Comparative Effectiveness Research in the US Context
The contentious politics that surround efforts to conduct CER date back to the 1970s, when the US government, having recently launched Medicare and Medicaid, began to contend with rising health care costs. Around the same time, health services researchers uncovered troubling dynamics in how health care is delivered in the United States. First, Wennberg and Gittelsohn (1973) found that physicians exhibited significant variation in the treatments they prescribed for a given condition. Moreover, these variations were apparent even when physicians were treating comparable patients. Subsequent research found that professional networks better explained the so-called small area practice variations than did patient characteristics or supply of medical services (Corallo et al. 2014; McPherson et al. 1982; Roos 1989; Wennberg 1984). This research suggests that a physician's approach in selecting treatments, while shaped by a complex set of factors (Eisenberg 1985), is significantly influenced by what high-status physicians in his or her professional network would do when confronted with a similar patient and diagnosis. If high-status physicians were choosing the most effective treatments, then this follow-the-leader approach to practicing medicine would work well. But the degree of practice variation suggests that, if some of these professional networks were delivering the best treatments, many others were not. While health services researchers were delving deeper into the causes of practice variations, the Office of Technology Assessment (Office of Technology Assessment 1978) produced a report arguing that medical technologies were not adequately assessed for effectiveness before enjoying widespread use. Moreover, the report argued that medical procedures were held to an even lower standard, where “only 10 to 20 percent of all procedures currently used in medical practice have been shown to be efficacious by controlled trial” (7).
Though medical societies bristled at the claims that doctors may not be providing the best available treatments (Patashnik, Gerber, and Dowling 2017), members of Congress along with health policy entrepreneurs began to push for government programs that would carry out a health technology assessment (HTA). What followed were a string of initiatives designed to distinguish between more and less effective treatments and technologies (table 1). To varying degrees, the sponsors of these programs intended the findings to be used by the Health Care Financing Administration (later Centers for Medicare and Medicaid Services [CMS]) to decrease public spending for high-cost, low-value interventions. While the initiatives passed with bipartisan support, most were short-lived and dismantled in response to lobbying on the part of medical societies and trade associations (Cotter 2009; Eisenberg and Zarin 2002; Gray, Gusmano, and Collins 2003; Perry 1982; Sorenson, Gusmano, and Oliver 2014).
These repeated legislative efforts demonstrate that it is difficult to sustain publicly funded HTA or CER in the United States. Existing scholarship notes two likely points of failure for such programs. First, if HTA/CER points to treatments in widespread use that are ineffective, groups with a financial stake in those treatments will mobilize to dismantle the CER-producing agency. A second issue arises when a timid agency provides little value for the dollars it spends by issuing vague or weak guidelines. Though current scholarship does not link these outcomes, they are an obvious Scylla and Charybdis of any HTA/CER organization: steering clear of the big ticket items to avoid mobilizing hostile stakeholders can cause agencies to produce lackluster results. Adding to this difficult balancing act are concerns that HTA/CER agencies are duplicating the work of other, more established agencies. Though it is clear that medical practice is proceeding without sufficient evidence (Institute of Medicine 2009), how health care, health services, and basic health research are divided among agencies like the US Food and Drug Administration (FDA), Agency for Healthcare Research and Quality (AHRQ), and National Institutes of Health (NIH) is neither fixed nor completely clear. Thus, newer programs, even when they are designed to fill a gap in current research, may struggle to establish unique identities and counter charges of duplication.
The creation of PCORI illustrates that, in spite of multiple past failures, the promise of CER continues to have congressional backers. At the same time, PCORI's structure suggests that its authors were especially conscious of—or perhaps subject to—the political liabilities of past HTA and CER efforts. To guard against the use of PCORI-funded studies to guide reimbursement decision making, CMS is barred from using PCORI-funded research as the sole basis of a reimbursement decision. Moreover, PCORI is not housed within Department of Health and Human Services or, in fact, in the federal government at all. Thus, the head of PCORI is not a presidential appointee but is selected by the PCORI board, whose membership is heavily constrained by its authorizing legislation. Those under contract to produce a CER study for PCORI are not permitted to include cost in their studies, so that results are presented only in terms of effectiveness. Unlike many of its counterparts in OECD countries (Chalkidou et al. 2009), PCORI's findings are not linked to decision making about public-sector spending on health care.
Finally, naming the organization “Patient Centered” sends a strong signal about which sectors of society the organization should serve. One might see this as a symbolic effort to avoid capture by either the medical products industry or the clinical research communities that might bend PCORI toward their respective interests. However, it was the medical products industry that insisted on the language in the first place (Ashton and Wray 2013: 199). Naming the organization “Patient Centered” may have been an attempt at “astroturfing” whereby the medical products industry, which provides financial support to a significant proportion of patient groups (McCoy et al. 2017), might shape PCORI's agenda and process without the appearance of having done so. Alternatively, the medical products industry may feel that anchoring the organization to patient demands would ensure few if any limits on access to care. The next section presents our theoretical framework and poses four possible stakeholder scenarios that are possible as PCORI undertakes its CER mission.
In the classic pluralist view of the American political system, one assumes that interest groups form around shared goals and, when effective, are able to translate shared interests into favorable policy outcomes. While this model captures a real dynamic in American politics, scholars have also identified the opposite mechanism: a policy or program, once enacted, will create predictable patterns of advocacy either in support of or opposition to that program—in this case, policies create politics. Theodore Lowi argued that policies tended to fall into three categories, distributive, regulatory, and redistributive, and that each type was associated with a distinct type of interest group mobilization and locus of congressional activity (Lowi 1964, 1972). James Q. Wilson articulated a similar framework for understanding the political constraints that executive branch agencies face, comprising four ideal types associated with the mix of costs and benefits that an agency's policies produce (Wilson 1989).
According to Lowi's framework, the least competitive political environments accompany policies that produce concentrated benefits with distributed costs and are characterized by legislative pork.1 However, agencies also experience similar politics when their primary mission is to distribute federal dollars. NIH is an example of an agency that experiences distributive politics (or, in Wilson's terminology, client politics). NIH uses general revenues to fund basic research that distributes dollars to researchers, primarily in academic and medical settings. Groups that feel excluded by NIH funding can organize and argue for inclusion without having to displace existing research endeavors, given that such additions are not zero sum.2 Although there may be groups that want more from NIH and even groups that want the agency to do something different—for example, devoting a larger share of its budget to applied rather than basic research—few if any of these groups have mobilized against the agency. Moreover, it is rare for stakeholders in this environment to be pitted against one another.
Contrast the NIH with an agency like the Environmental Protection Agency (EPA), which experiences what Lowi calls “regulatory politics” (Wilson's “interest group politics”). EPA's environmental protection mission is articulated in statutes designed to curtail the negative externalities many industries impose on the environment. Thus, stalling or blocking its rule-making efforts is a primary goal of well-organized trade associations representing the industries subject to its regulations. While most agencies that produce concentrated costs and distributed benefits have no substantial interest group support for their missions, the EPA can rely on environmental groups to counter the claims of industry. Long-standing public interest organizations like the Natural Resources Defense Council and Environmental Defense push the agency to meet the public health goals articulated in its statutes. While the EPA has been able to function between the allegations of too much and too little regulation, neither set of mobilized groups approaches the agency in the more cooperative way that is characteristic of the NIH and its stakeholders. Agencies with regulatory functions that do not have well-organized public interest organizations holding the agency's proverbial feet to the fire are more likely to fall prey to capture by the regulated industry. Important for this study is the distinction between distributive politics and regulatory politics in shaping the environment in which an agency attempts to carry out its mission.
Though the view that policies create politics is widely accepted, scholars differ in their conceptions of the degree to which actors planning or implementing policies can alter the circumstances surrounding implementation. Scholars also differ in their views about agency capacity to perform given a set of political constraints (table 2). Lowi's (1964, 1972) model suggests that the relationship between policies and politics is largely fixed. Wilson (1989), however, injected the idea of entrepreneurial behavior on the part of agency civil servants as a way to stimulate mobilizations around diffuse benefits. Similarly, Daniel Carpenter (2001, 2010) argues that agencies, by demonstrating capacity, can build external networks of support to insulate themselves from political attack. Eric Patashnik (2008) locates the seeds of success of new reforms in the authorizing legislation itself, rather than in the capacity of policy implementers. Patashnik argues that legislation that can produce novel political economies in support of new reforms is more likely to stabilize reforms going forward. In a very different model, Terry Moe (1989) argues that those seeking to pass contested legislation will often saddle implementing agencies with structural constraints that limit their potential for effective performance. According to this perspective, contested political environments should produce structural outcomes that make it harder for bureaucrats to demonstrate competence.
These theories exploring relationships between the content of a given policy and the political environment likely to emerge around that policy's implementation raise important questions regarding whether policy designers and/or implementers facing hostile stakeholder environments can increase the chances that a policy reform will take hold. CER, when it is able to distinguish among more and less effective treatment options—its stated intent—is likely to alter patterns of treatment. One should therefore expect corporate and professional interests to mobilize to protect themselves against CER findings that would put them on the losing side of that equation. The history of public-sector efforts to produce CER certainly provides evidence of such mobilizations (Chalkidou et al. 2009; Gray, Gusmano, and Collins 2003; Sorenson, Gusmano, and Oliver 2014). The gamble in creating PCORI is that three aspects of its design—its independent status, its lack of formal regulatory authority, and its explicit emphasis on responding to patients' interests—will place it on more stable political footing than its CER predecessors by decreasing its potential to deliver concentrated costs and by increasing the chances that PCORI will galvanize patient groups to support the agency as if it were delivering concentrated benefits to patients. At the same time, there is no guarantee that efforts to prevent the organization from having a regulatory effect—something meant to be protective—will not damage the organization when it comes to an assessment of its impact.
This article examines four possible trajectories for PCORI and analyzes available data to assess the extent to which stakeholder mobilizations are following one of several predicted patterns. A general-interest PCORI that responds to patient interests and involves patients in the range of CER decision making might emerge if patient groups act collectively to support the organization and can provide effective mobilization against threatened professional or industry groups. A second scenario is one where patient advocates and researchers who embrace patient-centered research find common cause and form patient-researcher coalitions that routinely work with PCORI and advocate for the organization during important episodes of congressional oversight. The third scenario involves weak patient group participation compared with researchers or professional and/or trade associations. This could have two outcomes: (a) traditional researchers are able to assert their preferences over those of patient advocates, allowing PCORI to lose its distinctive claim to serving patients as primary stakeholders,3 or (b) drug and device manufactures and/or provider associations are able to keep lucrative but comparatively ineffective treatments off PCORI's agenda in the first place so that patients continue to have low information regarding widespread but ineffective treatments. In the first outcome, what might be called “weak capture,” patient groups may be less motivated to defend the organization than they would be if they felt they had more than a symbolic role in setting the organization's research agenda. In the other outcome, “strong capture,” the agency may be subject to criticism of wastefulness or ineffectiveness. In the fourth scenario, “hostile mobilization,” either professional groups or industry, acting to counter loss of market share, are able to portray the research funded by PCORI as biased or flawed and the organization as wasteful and/or a federal encroachment on the private sector.
These four scenarios elucidate the difficult political terrain surrounding PCORI. Mobilizing supporters in broad coalitions that go beyond narrow disease identities is not a given. Avoiding capture by traditional health research stakeholders who are already well mobilized in support of long-standing federal research programs, for example, may be difficult. Finally, producing results that steer patients away from popular yet ineffective treatments—something necessary for PCORI to demonstrate its impact—is likely to spur opposition, whereas failing to produce such results runs the risk of appearing wasteful or redundant.
The goal of this research was to identify stakeholder interests in PCORI and analyze their actual and potential mobilization efforts in pursuit of those interests. In general, this research asked whether the balance of stakeholder resources directed at the organization is on the side of its supporters or its detractors. While we did not seek to answer that question definitively, we assessed both the level of interest and resources on the part of groups that have expressed interest in PCORI or are likely to express interest in the future. This research goal suggests that one can observe multiple interest groups or interest group sectors over time. Yet most research on interest group behavior relies on proxy data of multiple groups or participant observation of a single organization. Thus, this research drew data from a variety of sources to generate a clearer picture of PCORI's political environment.
Though the original plan, with approval from the UC Berkeley Committee for the Protection of Human Subjects, was to collect substantial data about PCORI's stakeholder environment from PCORI staff, a senior PCORI staff member canceled an initial set of scheduled interviews and barred PCORI staff from participating in the research project.4 As a next step, the lead investigator recruited key informants who, though knowledgeable about the organization, were not PCORI employees. The lead investigator conducted five interviews by telephone between July 2013 and April 2015 (see appendix B).
A second set of interviews were conducted with PCORI stakeholders in 2017. Participants were recruited (N = 10) using contact information from letters submitted as part of public comment to PCORI and using a public list of stakeholders attending PCORI's 2016 stakeholder meeting. The research team interviewed a representative from each organization who agreed to be interviewed and continued recruitment until at least one organization interviewee from one group from each sector identified in prior research was included (patient organizations, academic researchers, professional associations, and industry groups).5 For both sets of interviews, the interviewer took detailed notes during the interview and corrected these notes immediately following the interview to fill in details as necessary. The research team reviewed interview transcripts to identify important themes and then coded the transcripts to capture each theme/code systematically in every interview.
Public Comment and PCORI Governance
In its first two years of operation, PCORI solicited public comments on three topics: PCORI's working definition of patient-centered outcomes research (comments accepted July–September 2011), the methodological standards that PCORI would set out for those seeking PCORI funding (January–February 2012), and PCORI's national priorities for research and its research agenda (comments accepted January–March 2012). Table 3 includes descriptions of each data set, including the number of total commenters, the number of comments sampled for analysis, and whether the identity of the commenter was included as part of the public data.
The research team conducted content analysis on a random sample of comments from each data set, with a goal of coding at least 20% of the comments in each set. The research team designed codebooks for each using an iterative process that began with expected categories of significance. The codebooks were revised as needed to remove codes that produced little or no variation and to pick up patterns that the research team had not anticipated. Once the codebook was finalized, the research team, which included the principal investigator and one to two research assistants, selected a new random sample of comments to code. The principal investigator coded a random sample of 30% of the comments from the working definition data set and achieved intercoder reliability scores for each variable ranging from 0.77 to 1. Efforts to generate sufficiently high intercoder reliability scores for comments submitted for PCORI national research priorities and research agenda did not achieve intercoder reliability scores above 0.77. To address this, comments were coded in teams of two or more researchers who assigned codes for each of the randomly selected comments. Public comments submitted in response to PCORI's research agenda and methodology contained information on each commenter's identity and achieved intercoder reliability scores from 0.85 to 0.90. Table 3 summarizes the types of data collected from each set of public comments and associated intercoder reliability scores where applicable.
Data were also collected on the membership of PCORI's board, the types of stakeholders who make public comments at PCORI board meetings, and the membership of PCORI's advisory panels across stakeholder categories (table 4). Board members and those who made comments during board meetings were coded using the same coding scheme applied to the public comments. Advisory panel membership data are publicly available through PCORI's website and correspond with the categories used for the other data sets to ensure comparability.
To gain additional insights into stakeholder interest and mobilization around PCORI, the research team coded a sample of congressional hearings that contained one or more references to PCORI. Data were collected from the ProQuest legislative database by searching for “Patient Centered Outcomes Research Institute” in all categories, including full text after January 1, 2010. This search, conducted in February 2015, returned 77 hearings. Half the hearings were coded to capture the witness type and the nature of the discussion about PCORI.6
Internet, Social, and Other Media
Because groups concerned with PCORI might neither appear before Congress nor submit public comments, the research team sought access to coverage of PCORI activities in the trade association press for the biopharmaceutical industry, specifically via a trial subscription to the Pink Sheet. Though the goal was to code a random sample of stories, the terms of trial subscriptions changed during our analysis, leaving the research team with access only to story titles and abstracts. Reported here are the topics found in the analysis of the Pink Sheet coverage of PCORI activities from January 2011 to December 2013. However, because of loss of access to the data, the distribution of themes across the relevant articles could not be calculated.
In an effort to capture perspectives about PCORI that may not have been articulated in its formal comment periods, the research team turned to Internet and social media. General searches for discussions of PCORI on Facebook and Twitter were dominated by PCORI output and did not generate much insight into stakeholder activities. We also conducted a general Internet search for web content mentioning “PCORI.” Because search results produced a very positive or neutral image of PCORI, we conducted more targeted searches to find out where controversy about PCORI might exist. To that end, the team searched for web content containing the words “PCORI” and “ration” and those containing the words “PCORI” and “winners and losers.” Table 5 summarizes the respective search strings, number of search results, and number of websites coded.
In this section, we present findings from the two distinct phases of data collection, 2013–15 and 2017. In the first section, we draw from the initial phase of data collection to explore, respectively, how stakeholders view the organization and the extent to which they appear to be active in trying to shape PCORI directly and/or have lobbied Congress about PCORI. We begin by presenting data from public comments to generate a picture of the types of groups active during PCORI's early days. Next, we weave together insights from the public comment data with other relevant data to create a more complete picture of views and likely mobilization from three distinct types of stakeholders: (a) patients and patient organizations, (b) those hoping to secure PCORI research grants, and (c) stakeholders who have expressed concerns about PCORI's regulatory potential. The second section presents data from the 2017 interviews. By dividing up the findings this way, we are able to highlight where the more recent data suggest an evolution in stakeholder perspective and/or mobilization.
Data from Public Comments
In its first two years (2011–12), PCORI solicited three sets of public comments: (a) a working definition of patient-centered outcomes research, (b) its proposed research agenda, and (c) its draft methodology. Commenter identity is not available for comments submitted in response to PCORI's definition and commenters self-identify only half the time when responding to PCORI's proposed research agenda. Thus, we cannot make direct comparisons among participants in these three opportunities for comment. However, several notable sources of variation are worth considering. Comments made in response to PCORI's definition are overwhelmingly critical, with concerns about state intrusion into the private sector or other critical comments making up over three-quarters of the coded comments (table 3). Many of these commenters argued that PCORI should not exist. Only 13% of the coded comments expressed support for PCORI's overall goals. Seven percent appeared to support PCORI's goals but expressed an interest in the need for PCORI to give greater emphasis on patient involvement or patient goals. While we do not have data on commenter identity, many comments exhibited low information regarding PCORI's structure and function. This suggests a higher rate of lay participation compared to the other two sets of public comments.
Because only half of the commenters self-identified when responding to PCORI's research agenda, we can say little of the overall makeup of commenters. Among those that did self-identify, most were professionals (professional associations, 46%; researchers, 14%; table 3). Patient advocates comprised 16% of the self-identified population. The two most frequent types of comment offered in response to PCORI's research agenda were (a) emphasis on a specific type of specialty or disease (27%) and (b) the need for transparency, access, or shared decision making (19%). While there were some negative comments—for example, 1% of commenters suggested that PCORI would be used to ration care—most of the comments appeared to accept PCORI as legitimate and were phrased constructively toward shaping PCORI's focus (table 3). These comments also display a much higher level of information about PCORI's structure and about CER than comments submitted in response to PCORI's working definition. When comparing comments across self-identified commenters, the heavy emphasis on access by patient groups and industry stands out (fig. 1). Also notable is the fact that specific advocacy (where a group asks PCORI to consider a specific disease, disability, or group) is the first- or second-ranked comment for three of the four types of stakeholders. It ranks as the most common type of comment made by professional associations. Advocacy shares top billing with access and dissemination for researchers: their comments are evenly divided among those three concerns. It is the second most common type of comment for patients. Among industry comments, specific advocacy comes in as the third most common type (dissemination and NIH duplication get similar scores), falling well behind the predominant access comments and technical comments.
Commenters responding to PCORI's draft methodology self-identified at a rate of 99% (table 3). Most comments came from researchers (38%), industry (19%), and professional associations (15%). Care should be taken in comparing the relative distributions of self-identified commenters across the two comment periods, given that only 50% of commenters self-identify when responding to PCORI's research agenda. At the same time, the increase in rates of researcher and industry comments responding to PCORI's methodology is plausible given the more technical subject matter associated with CER methodology (table 3). It is likely that the drop in lay participation rate when comparing research agenda and methods comments is much steeper than these data suggest. This would be the case if lay participants are a significant proportion of those who do not self-identify when commenting on PCORI's research agenda.
The tenor of comments submitted in response to PCORI's methodology also displays high information about PCORI and CER. In fact, the plurality of coded methodology comments offered specific technical advice about how to conduct CER (36%). Issues of transparency, access, and shared decision making are the second most frequent type of comment (19%). While some commenters are critical of some aspect of PCORI's proposed methodology (11%), the overall tenor of comments is constructive and geared toward improving PCORI's ability to fill gaps in CER (table 3). Unlike comments submitted in response to PCORI's working definition, these comments almost never question the legitimacy of PCORI as a publicly funded organization.
Mobilization of Patient Advocates
Data concerning PCORI's governance and rates of participation on the part of stakeholders in PCORI outlets for public participation indicate that patient groups are a consistent but rarely dominant presence. Patient advocates represent 14% of PCORI's board membership (table 4) and 38% of the membership on PCORI advisory panels. Public comments offered during PCORI board meetings between January 2011 and May 2013 show that patient advocates make up almost a third of all commenters, while professionals (researchers and/or clinicians) make up 45% and industry stakeholders make up 6% (table 4).
Data collected from public comments submitted to PCORI tell a similar story: patient advocates participate without dominating. They comprise 16% of the self-identified stakeholders commenting on PCORI's research agenda and 7% of those commenting on PCORI's draft methodology (table 3). In spite of being outnumbered by other types of stakeholders, the most frequent argument made by patient advocates in response to PCORI's research agenda is the need to include patients in PCORI decision making (fig. 1). Thus, patient advocates appear to be supporting PCORI's mission rather than trying to redirect it.
Though patient advocates might increase their influence by acting collectively, current evidence suggests that patient advocacy organizations tend to pursue disease-specific rather than general patient goals (Keller and Packel 2014). When asked about the extent of cross-disease activism on the part of patient advocates, one key informant argued that patient advocates who participate in PCORI's public meetings face social pressure to think beyond the specific diseases, stating, “You're not going to argue that your disease-specific issue is most important when someone suffering from a different disease is sitting right next to you” (interview 1). At the same time, two other key informants felt that disease identity drove most of the advocacy directed at PCORI from patients and patient advocates (interviews 2 and 3). One key informant reported that one was more likely to see coalitions of patient advocates and researchers aligned around a single disease than coalitions of patients representing many diseases (interview 3). Another quipped, “Ask a different patient group [what PCORI should be doing], get a different answer” (interview 2). In addition, the limited number of patient groups that have testified about PCORI before Congress have focused on the disease-specific benefits those groups expect PCORI to produce (Chard 2014; Summar 2014).
While the public comment data show that patient groups are participating at lower rates than other types of stakeholders and do not appear to be forming cross-disease coalitions, perspectives offered by key informants suggest that patient advocates have a discernible influence on PCORI decision making (interviews 1–3). Two key informants referenced PCORI's goal of making patient-centered research the new standard for health outcomes research (interviews 1 and 4). Moreover, actors close to the organization report that the most important early decision made by the board was the choice between two competing visions of the organization: a patients-first versus a science-first orientation. After a period of debate, the board formed a consensus around the patients-first approach (interview 2). Given the large representation of academic researchers on the PCORI board (43%; table 4), one might not have predicted this outcome. Moreover, there is evidence that the patients-first principle is substantive rather than symbolic. One key informant reported that, during review of research proposals submitted to PCORI, scores were much less consistent across reviewers than is the case for NIH, owing to the substantive role given to lay participants on PCORI review panels. This key informant also shared that the PCORI review process was leading to a lack of “meritorious” proposals in the area of treatment options because patient reviewers found many proposals to be insufficiently patient centered (interview 5). Contrast this with Kleinman's (2000) finding that the NIH review process allows patient representatives to comment only on proposals already deemed meritorious by scientific experts.
Patient group influence is also evident in PCORI's funding initiative for CER on the diagnosis and treatment of hepatitis C. Describing the decision process in a PCORI blog post, director Joe Selby indicated that patients moved the organization away from its initial plan to fund a randomized control trial that would compare immediate treatment with close monitoring upon diagnosis. Though most stakeholders endorsed a PCORI-funded randomized controlled trial, Selby explains that patient advocates rejected the proposal. Patient advocates felt that enough evidence exists for immediate treatment and thought proceeding with a randomized controlled trial would be unethical. Thus, PCORI decided to fund studies that analyzed existing variation in medical practice rather than calling for random assignment of trial participants (Selby 2014). Though patient advocates may be participating in lower numbers and with fewer resources than other types of stakeholders, they are active participants who are taken seriously by the PCORI leadership.
Stakeholders Interested in PCORI Research Funding (Distributive Politics)
In light of stagnant and declining NIH budgets since 2003 (Johnson 2013), it would be no surprise if medical research communities were interested in PCORI's grant-making potential. In fact, the data on PCORI governance and stakeholder participation in PCORI public forums show that researchers are a consistent and sometimes a numerically dominant presence. Researchers (primarily academics) make up nearly a quarter (24%) of the membership of PCORI's board, outnumbering every other type of stakeholder (table 4). In addition, professionals offer a plurality (45%) of the comments made during PCORI board meetings (table 4). Researchers also make up 23% of PCORI's advisory panel membership and account for 30% of the membership on three of the four panels. Professionals, either as individuals (clinicians or researchers) or as represented by professional associations, offer a majority of the public comments for whom the commenter is identified. Approximately 60% of the comments responding both to the proposed research agenda and methodology come from these three categories (table 3).
Evidence of distributive politics appears strongly in comments submitted in response to PCORI's research agenda. The most prevalent type of comment (27%) is to ask that PCORI not overlook a specific disease or specialty in allocating its research dollars (table 3). “Specific advocacy” comments are the most frequent comment coded for professional associations, one of researchers' top three comments (advocacy, access, and dissemination make up 21% each of the coded comments by researchers) and patient groups' second most frequent comment (fig. 1). Lobbying for a “piece of the pie” is what one would predict given that PCORI's research agenda, once set, will shape the flow of future research dollars.
There is a notable shift in comments submitted in response to PCORI's draft methodology. Here, technical comments about CER methods are the most prevalent (table 3). Though specific advocacy is only the fourth-ranked comment in response to PCORI methods (table 3), it is the second most frequent comment made by professional associations (fig. 2). Thus, while most commenters responding to PCORI's draft methods engage in a more technical discussion, many professional associations use the comment period as an opportunity to continue to lobby for a piece of the pie.
In addition to mobilizing groups who are typical stakeholders in the health research space (professional associations, researchers, and industry), PCORI may stimulate new coalitions of supporters around its effort to create two types of open-access disease registries under the heading of PCORI's “national patient-centered clinical research network,” PCORnet. The first type of registry is called a clinical data research network. These involve two or more health care delivery systems that, in partnership, create integrated data platforms that can be used to conduct randomized comparative effectiveness studies.
The second type of registry encouraged by PCORnet is the patient-powered research network (PPRN). These networks match patient populations with researchers around the goal of studying the most effective treatments for a given condition or set of conditions. However, unlike traditionally funded clinical research, PPRNs expect that patients and patient advocates will be actively involved in the selection of interventions to be studied, in the research design, and in the structures and protocols involved in collecting data from patients. Taken together, these disease registries may create coalitions of providers that would otherwise lack the resources to produce in-house CER (clinical data research network) or coalitions between researchers and patients who collaborate through PPRNs. Since PCORnet is in its early years, current analysis of the distributive mobilizations surrounding PCORI may underestimate the size and complexity of groups that could become PCORI's active supporters.
Concerns about Reimbursement and Rationing among Stakeholders (Regulatory Politics)
The evidence presented above supports the argument that many of PCORI's stakeholders are engaging in classically distributive politics and participating to increase their chances for getting a piece of the pie. Because the organization is prevented from funding research that takes cost into account and the federal government is barred from using PCORI-generated research as the sole basis for setting coverage and reimbursement policy, one might expect that PCORI would experience only distributive stakeholder politics. However, in spite of the fact that PCORI was designed as a nongovernmental organization with no formal regulatory power, PCORI continues to be subject to regulatory politics. While some of the stakeholders expressing such views do so without understanding PCORI's design and legislative mandate, many stakeholders demonstrate a solid grasp of PCORI's structure and mandate and yet also view PCORI as at least quasi regulatory.
Content analysis of the public comments made in response to PCORI's working definition show that only 13% of the submitted comments were supportive (table 3). Many of the negative comments did not respond specifically to the working definition and instead seemed to reflect a negative view of the ACA and of the potential for government intervention in the private sector, particularly intrusion into the doctor-patient relationship. A typical example reads, “I think it is very irrational to think that a bunch of bureaucrats who know nothing about me should have ANY control over my health care decisions!” Such comments indicate that commenters do not understand PCORI's mandate and do not view PCORI as an organization that funds and disseminates research. They also often characterize PCORI as one of the ACA's alleged “death panels.” Though commenter identity was not included in the database of comments submitted in response to PCORI's definition, such comments are consistent with populist sentiment against the ACA in general and PCORI specifically (Kaiser Family Foundation 2013; Ungar 2013).
A second type of criticism offered regarding PCORI's definition was much more specific to CER. These comments argued that, while attractive in theory, CER will not guide physicians when it comes to selecting treatment for an individual patient. According to this logic, CER generates only population-level averages that do not provide guidance for any specific patient's treatment. This line of argument is a centerpiece of the Partnership to Improve Patient Care, an umbrella organization formed in March 2009 in response to CER funding provided through American Recovery and Reinvestment Act (Mundy 2009). The group's web page claims that “[CER] can be misapplied in ways that unintentionally undermine patient access to care and physician-patient decision-making. That's because CER results typically are based on broad population averages that don't reflect the differences in needs of individual patients.”7 The National Pharmaceutical Council (2013) also characterizes CER results as generating findings based on population averages that are too general for guiding individual patients.
Attacks on the value of CER are not necessarily evidence of regulatory politics, and there is some evidence that individuals close to the organization are not particularly concerned about this line of criticism. Key informants who were asked about stakeholder concerns about the usefulness of CER in selecting treatments for individual patients tended to dismiss those concerns. One characterized that as a straw man argument. Instead, the interviewee argued that CER has a long and justified tradition of research and that most doctors accept that evidence about treatment outcomes is of value (interview 4). Another key informant argued that CER is finely tuned to be relevant to decision making among specific types of patients such that effectiveness of a given course of treatment is defined in terms of the specific subgroups for whom that treatment has been shown to be effective (interview 3).
While criticism of CER is not proof of regulatory politics emerging around the organization, it is quite distinct from the types of complaints that surface among those competing for PCORI research dollars.8 Moreover, there is evidence of specific concern about the potential for PCORI to influence reimbursement decision making. These concerns emerge in public comments, in press coverage of PCORI, and in congressional hearings. About 5% of the public comments made in response to PCORI's definition and research agenda mention costs; however, many of these argue that PCORI should include cost in its analysis. A more consistently negative view comes in response to PCORI's draft methodology, where several industry commenters requested that all mention of cost be removed from the draft methodology. Additionally, several of these letters express concern about the report's use of value-of-information analysis to set research priorities. Manufacturers note that value-of-information has been used by the United Kingdom to inform research priorities, whose results are then used in reimbursement and coverage decisions. These letters offer recommendations that would limit the ability to use PCORI-funded research to make population-level assessments of specific treatments by preventing PCORI's methodology committee from engaging in priority setting in the first place or by limiting its reliance on value-of-information analysis.
Concerns about regulatory impacts also appear in the general press coverage of PCORI and in trade association coverage of the organization. A content analysis of targeted Internet searches on PCORI returns a number of websites that address the potential for PCORI to ration care by limiting access to costly treatments. Some of the websites raise the topic in order to dismiss the notion that PCORI has the power to ration care or set reimbursement policy and portray PCORI in a positive (41%) or a neutral (15%) light (table 5). Forty-four percent of the coded articles reinforce the idea that PCORI has regulatory power and will ration care (table 5). Some of these argue that, in spite of its lack of formal regulatory power, its research will ultimately have this effect.
It is important to note that conducting a general search on “PCORI” will not lead the searcher to discussions of PCORI as a regulatory or quasi-regulatory actor. General search returns are dominated by sites containing technical instructions for organizations required to pay their share of the excise tax funding PCORI or neutral presentations of the organization (58%) and by content portraying the organization in a positive light (38%) (table 5), much of which comes from PCORI itself. At the same time, Internet content portraying PCORI as regulatory may be reaching an audience already inclined to view PCORI negatively, especially if those audiences are learning about PCORI through trusted blogs and media sources rather than conducting a general search on the organization's name. Given the tendency for individuals to seek content that reinforces rather than challenges their political beliefs (Stroud 2007), it is possible that members of the public who view PCORI negatively are having those views reinforced, in spite of the fact that online portrayals of PCORI as regulatory are far less common than neutral or positive portrayals of the organization.
Data drawn from review of a popular trade association press for biotech and pharmaceutical companies called the Pink Sheet also contains evidence of regulatory concerns. A search on “PCORI” in the Pink Sheet archive returned 85 articles mentioning PCORI from the time of its formation via passage of the ACA (March 2010) through December 2013. Content analysis of these articles showed that the Pink Sheet contained almost no coverage of information on PCORI funding opportunities, suggesting that these trade groups were not interested in competing for PCORI grants or, possibly, were receiving information about grant opportunities via other channels. Instead, the Pink Sheet content focused primarily on issues of PCORI governance and access to PCORI decision making. For example, the Pink Sheet reported on industry concerns that PCORI included no corporate representatives on its methodology committee while noting the dominant number of academics on that panel (Twachtman 2012a). In another instance, the Pink Sheet covered efforts on the part of a biotech lobbying organization to remove language from the PCORI draft methodology report referring to the role of costs in affecting patient outcomes and objecting to the view that industry-funded research might be viewed by many as biased (Steinke 2012). While the coverage of PCORI in the Pink Sheet contains a great deal of overlap with public comments from manufacturers submitted to PCORI in response to its draft methods, one issue was found only in the Pink Sheet: one article addressed concerns that PCORI's research approach and methods might alter regulatory standards outside of PCORI (Twachtman 2012b).
Finally, members of Congress who discuss PCORI during congressional hearings are much more likely to view the organization as having a negative regulatory impact than a positive, market-correcting one. Comments include the perspective that PCORI represents an undue burden on taxpayers (Latta 2010), questions about PCORI's independence from the Department of Health and Human Services (Upton 2011), concerns about CER being used to deny care to individuals (HCA 2011), concerns about the likely regulatory burden on the drug and device industry (Burgess 2011), and a general concern about PCORI's potential to ration health care (Roberts 2015). Senator Sander Levin (D-MI) is the only member of Congress in the coded hearings to portray PCORI in a positive light (Goolsbee 2011).
In keeping with the negative view expressed by most members of Congress who discuss PCORI in the hearings process, the draft House appropriations bill for fiscal year 2016 included language to rescind PCORI's budget of $100 million. This draft bill also cut the budget of the Center for Medicare and Medicaid Innovation and eliminated the AHRQ (Raths 2015; HCA 2015).9 The proposed cuts to PCORI's budget did not materialize when Congress passed the Bipartisan Budget Act of 2015, which increased spending by $112 billion to avoid the caps that come with sequestration.10 This suggests that the draft appropriations bill was driven, at least in part, by sequestration politics and did not arise only out of hostility to CER. At the same time, even under the threat of sequestration, the draft House bill proposed to increase the budget of the NIH (Raths 2015). This points to the vulnerability of lesser known agencies whose missions are not well understood by the electorate and who lack well-established reputations and/or highly mobilized and supportive stakeholders.
Data from Stakeholder Interviews
In this subsection, we present findings from 10 interviews conducted in 2017 with PCORI stakeholders. While we reached a reasonable cross section of PCORI stakeholders (see appendix B), one cannot assume that these groups are representative of the larger population of PCORI stakeholders. Information in this section is presented with that limitation in mind. In spite of our caution about generalizing from these interviews, one theme was present in so many interviews that we present it with high confidence: almost all of the interview subjects (8 of 10), with no prompting from the interviewer, praised the organization for the impact it has had on getting patients more involved in clinical research.11 In fact, the interviews give the impression that this is one of the organization's most important contributions thus far. The following quotations illustrate how interviewees discussed PCORI's contribution to patient-centered medical research:
I have been involved in epi studies in the past, in CBPR [community-based participatory research]. It's always seemed like the community was subordinate to the scientist. In PCORI, it really seemed like the patient was a genuine participant in the research. [They] really treated the patients as an equal member in the research. (interview 15)
I am very supportive, and my organization is supportive of PCORI succeeding in achieving what they set out to do . . . engaging patients and showing the research world a patient-centered manner of doing research. That is very valuable. We recognize that. (interview 8)
PCORI demands more than NIH in terms of dissemination, sustainability, and patient involvement in research. PCORI keeps you moving. It seems like PCORI is holding the researcher's feet to the fire. I can write a bed and breakfast guide to [study region redacted for confidentiality]. You have to meet [the community] on their terms. You have to go there to make it work. (interview 14)
The federal government has been talking about this for 20 years without a real change in how they were doing their research. I think PCORI has really transformed that. But by creating pressure on NIH to shift direction a little bit. That has been a gradual transition. But they [PCORI] have really operationalized it. (interview 6)
We have a positive view of PCORI. We have learned from them about being more patient-focused. FDA and CMS are also doing that [increasing the role of patients], so we want to be aligned in those goals. (interview 10)
Some interviewees talked about this achievement in general terms, for example, PCORI “changed the model” or “moved the needle” toward more patient involvement in research. However, in four interviews the interviewee drew specific comparisons with other agencies to argue that PCORI's efforts were more substantive and/or served as a model for other agencies to follow. In addition to the two examples above where interviewees indicated that PCORI gives patients more voice than does NIH,12 another interviewee observed, “FDA has had patients serve on review panels. It's not until product is already at the FDA and then there is one patient. They are surrounded by experts and intimidated. But PCORI has really brought them to the beginning of the process. Before even knowing the research questions” (interview 13).
Table 6 summarizes several variables from the interviews that shed light on PCORI's interest group environment. These data indicate that many groups were brought into PCORI's orbit because of its funding potential. Six of 10 interviewees were grantees, and 4 of these said that their interactions with PCORI were limited to their role as grantees. Three interviewees expressed no interest in PCORI as a funding agency. These interviewees explained their interest in terms of a desire to shape the organization's agenda and/or decision making. Three other interviewees indicated that they were interested both in opportunities to shape PCORI's processes and in pursuing funding.
Interviewees were asked to reflect on how well PCORI was balancing stakeholder views (table 6). Many interviewees needed clarification on this question because they did not immediately see tensions among stakeholders. Such responses are consistent with a distributive political environment where stakeholders do not see themselves as participating in a zero-sum game (Lowi 1964, 1972). When asked to consider the need to balance competing stakeholder interests, four interviewees mentioned tensions between patients and researchers (interviews 6, 8, 9, 12). A health-oriented nongovernmental organization put it this way: “The other thing I am aware of is the patient-centered thing. I have colleagues that have participated in grant review process and have talked about the patient stakeholder input for that and, as grand a vision as it is to include patients in the scientific process, they didn't feel that all the patient stakeholders were well-versed in scientific methodologies, which was an issue” (interview 8).
Three interviewees raised the potential for tensions to arise around questions of value and coverage (interviews 8, 12, and 13). One interviewee implicitly set up a tension between those paying for care and those selling treatments. At one point, the interviewee clarified that CER as a concept was acceptable only if costs were not taken into account: “We wanted them not to focus on cost but on comparative-effectiveness. We want to understand how their research would be used. But wanted to make sure there were parameters and guardrails around it” (interview 13). At another point, however, this interviewee cast PCORI's future in terms of its ability to identify ineffective care, arguing that “the rubber is going to hit the road when the CER results come out. Then we will be able to see what the return on investment has been. So, the payers have to see whether it will be worth it for them to pick up the tab after the trust fund runs out.” Interesting in this interview was the way in which the interviewee cast patient interests as aligned with those profiting from current treatment patterns, but later aligned patients more with payers, saying, “We [patients] don't want to pay for what's not working.” A second interviewee expressed the concern that PCORI has been captured by medical manufacturers, saying:
I kind of feel like, the patient advocacy groups that may be PhRMA [Pharmaceutical Research and Manufacturers of America] funded. Those groups may be overrepresented. . . . We are wary of the influence of PhRMA on those groups. I don't think they [PCORI] have handled it poorly. But they have to deal with those types of groups, and I worry about the role of PhRMA. They [PhRMA] had someone there [on the board or advisory board] at the outset with industry conflicts of interest. (interview 12)
Without explicitly laying out who the winners and losers might be, this interviewee implied that PCORI needs to guard against capture by the industry.
Table 7 summarizes interviewee views about PCORI's strengths and weaknesses. While a handful of interviewees were positive about PCORI across the board, most provided a somewhat mixed review of the organization. Many gave the organization high marks for ensuring that patients would have a substantive role in PCORI funded research. Other strengths included its talented staff and the funding PCORI has to support CER. At the same time, three interviewees characterized the organization as slow to disseminate its research, and two faulted the organization for a lack of major findings thus far. While two interviewees praised the organization for not addressing costs (interviews 10 and 13), three interviewees criticized the organization for its avoidance of costs (interviews 8, 9, and 12). The following quotations illustrate the sentiments we heard from interviewees:
Is it doing a good job? I don't know if it is doing a good job. It is a complicated organization. They have a grand vision; as well they should, for what they want to deliver. But I'm not sure the way they are going about it is actually succeeding. Both in the research community as well as in the general media. There is a perception that they are wasting money, or have funds that are not distributed. You have to jump through so many hoops to get it [funding]. If you are going to piss a bunch of people off to obtain your goal, I don't think that serves any purpose. And I know getting PCORI funding is increasingly competitive and people still apply. It's just a lot of work. (interview 8)
They haven't yet turned out practice-changing studies, but they have changed the way research is done. I will give them a passing grade in that they have made an impact. But they have had less success, so far, on changing practice of medicine. (interview 11)
The people there are talented . . . talented, thoughtful. They want to do the right thing. As individuals, they are all very dedicated and want to improve things for patients. The biggest barrier they have is not being able to address costs when that's what is in the news. They are prohibited from even mentioning the word cost in your [sic] grant applications. That is not their fault, but that makes them look irrelevant. (interview 12)
In addition to giving PCORI a mixed review in terms of its accomplishments, many who were grant/contract recipients indicated that PCORI's rules for grant recipients were far too burdensome (interviews 6, 8, 9, and 12).
Out of nine interviewees that discussed whether they would lobby on behalf of PCORI in the future (table 6), three indicated that they would (interviews 7, 9, and 14). Two interviewees were more guarded and said that they planned to monitor future PCORI activities and decide, at the time, whether to provide input (interviews 10 and 13). Two indicated that, although supportive of PCORI, it was not a priority for their respective organizations with respect to the time each spends on Capitol Hill (interviews 6 and 11).13 One of the patient organization interviewees reflected, “We've benefited but I'm not sure it would be a priority for us” (interview 6). Another patient organization interviewee put it this way:
If I were to rank PCORI, it would be 20th. It is not big organizationally in terms of our priorities. It is not one we are spending a lot of time to engage in. We try to make sure the patient voice is ever-present. That is essentially what PCORI is trying to do. So we have a similarly aligned policy interest. There is some overlap, but we probably won't heavily engage in lobbying on behalf of them. I would say we are quietly appreciative. (interview 11)
Among the four possible stakeholder scenarios laid out above, the first posits a well-mobilized group of patient advocates who can vouch for PCORI's unique contribution to patient interests in CER. Patient advocates are active participants in PCORI governance, and their involvement shapes PCORI decision making in measurable ways. Moreover, stakeholders were almost unanimous in expressing the view that PCORI was changing the standard for patient involvement in clinical research and that this was valuable.
The emphasis by PCORI's leadership on putting patients first is an important expression of how PCORI understands its congressional mandate. Moreover, this emphasis mitigates two potential vulnerabilities it faces in competing for future congressional funding: (a) differentiation from more seasoned research agencies like NIH or AHRQ and (b) creating a stakeholder community that regards PCORI as providing significant benefits. The emphasis on involving patients in every stage of the CER research process allows PCORI supporters to argue that PCORI is changing the standard of medical research so that it better serves the needs of patients rather than being driven by the career goals of researchers. That emphasis also represents an entrepreneurial effort to mobilize a group of stakeholders who could act to defend the broadly distributed benefits PCORI aims to provide—the very supporters interest group theory predicts would be weak or absent.
However, while PCORI's internal decision making has appeared to level the playing field so that patient advocates have more than a symbolic role in shaping PCORI research, there is little indication that patients are mobilizing in a way that would support PCORI in future budget battles. Key informants report that patient advocacy remains disease specific. Corroborating this view are data from congressional hearings in which patient advocates testify about the benefits PCORI brings to that group's specific membership rather than to patients in general. When members of Congress hear about the more general, public benefits of PCORI, that message comes from researchers and civil servants. A lack of cross-disease collective action on the part of patient advocates may mean that patient mobilization will be too narrow to counter future attacks on the organization. Even the idea that patient groups would advocate for PCORI based on specific benefits that each group received is undermined: interviewed stakeholders indicated that PCORI did not rank highly among their lobbying priorities.
The second scenario involves cooperative mobilizations between researchers and patients. Given that PCORI funding depends on research projects that meaningfully involve patients, the very nature of the grants submitted to PCORI ensures that patients and researchers must work collaboratively. PCORnet also creates opportunities for providers and patients to work together on generating data that increase the capacity for research on specific diseases or in specific settings. PCORnet, to the extent that it can institutionalize the creation and sharing of treatment and outcomes data, has the potential to give rise to a novel political economy that will act to sustain these entities. However, to the extent that there is cooperation between researchers and patients, this activity appears to be forming within rather than across disease categories. In addition, stakeholder interviews also pointed to some tensions arising between researchers and patients around the central role that patients are giving in shaping PCORI research. While patients and researchers are collaborating, PCORI's norm-changing efforts with respect to clinical research might weaken its ties to its primary distributive audience: the clinical research community.
The “weak capture” outcome of the third scenario has little evidence to support it. Patient involvement in scoring grant applications and the rejection of a randomized controlled trial approach in generating evidence for treatment of hepatitis C show that there are instances where patients' views are weighed heavily in PCORI decision making. The consensus view among interviewed stakeholders that PCORI is changing the status of patients in clinical research also suggests that researchers have not captured the organization. Researchers represent a plurality on many of PCORI's governing bodies, yet the organization's strong emphasis on including the patient voice appears to be guarding against capture by researchers. While some public commenters worried about PCORI becoming “just another NIH,” those familiar with both organizations argue for PCORI's unique role in funding health-related research.
Harder to rule out is the potential that PCORI is subject to “strong capture” by industry groups—the other potential outcome of the third scenario. In spite of PCORI's careful institutional design and its fundamental lack of regulatory power, regulatory politics are also present in the organization's environment. Industry groups' public comments raise clear regulatory concerns. To date, industry groups have not argued that PCORI should not exist. Instead, they have worked to ensure they are well represented in PCORI decision making and that costs will not be considered in PCORI-funded research. Only one interviewee expressed direct concern about industry influence over PCORI, mentioning both a conflict of interest stemming from industry representation on PCORI's board and influence exerted via industry-funded patient groups.14 More common among the 2017 interviews was the concern expressed about the lack of significant results coming from PCORI-funded research. The fourth scenario, “hostile mobilization,” cannot be assessed in the absence of practice-changing findings. As long as PCORI avoids funding research that would indicate a widespread treatment modality was ineffective, as in the case of performing surgery to treat lower back pain (Sorenson, Gusmano, and Oliver 2014), one would not expect professional or trade associations to lobby for an end to federally funded CER.
The stakeholder interviews, when taken together, generate an impression of stakeholders who view PCORI as a lead organization in redefining the role of patients in clinical research. At the same time, groups interviewed did not indicate that PCORI was a high-priority organization in terms of their lobbying efforts. These interviews also demonstrate that stakeholders were hard pressed to point to practice-changing outcomes. Many attributed this to the fact that the organization had to start from scratch and would need more time to have an impact. At the same time, the comfort that industry groups expressed with PCORI's performance and its responsiveness to industry concerns suggests that the lack of impact might have its source not in the organization's youth but in organizational capture.
Some stakeholders have also expressed regulatory concerns through their websites and by organizing conferences that frame CER in a negative light. While key informants argued that this view of CER is outdated and overlooks a long, established tradition of effective research improving doctors' ability to deliver effective care, the perception of PCORI by vocal members of Congress more closely matches the perspective offered by these critical stakeholders. Moreover, populist sentiment—especially pronounced in PCORI's early days—that sees CER as an intrusion into the private realm of doctor-patient relationship has the potential to add grassroots fuel to future controversies about PCORI-funded research. It is worth noting that past federal efforts at CER were defunded without the additional political weight of such populist mobilizations.
PCORI emerged only after several failed attempts to create a new, publicly funded version of CER in the United States and represents a carefully crafted compromise that funds CER research while limiting its potential impacts, particularly with respect to Medicare reimbursement. The result is a distinctly American approach to CER. While PCORI's budget comes from the federal government's ability to levy taxes, and the organization is subject to congressional oversight, PCORI sits outside of the federal government and has a board that mixes public- and private-sector members, including representatives of the pharmaceutical industry. Moreover, the lion's share of PCORI-funded research is carried out in the private sector. Perhaps most important, the application of PCORI findings is designed to take place entirely in the private sector. Patients and doctors with access to PCORI findings can use those findings in their own decision making, as can private-sector payers.
In addition to PCORI's careful design, PCORI leadership made several decisions displaying considerable insight into the organization's political vulnerabilities: elevating patients in organizational decision making and establishing PCORnet. At the close of 2014, PCORI had overcome substantial political odds to become an active participant among federal sponsors of medical research. Given the partisan hostility to the ACA and the populist mobilization against “death panels” associated with PCORI's CER mission, many, including officials within the Obama administration, expressed surprise at the organization's ability to set a research agenda and initiate funding after starting essentially from scratch (interview 2). PCORI's patients-first commitment increases the potential that PCORI's contribution will look different from other health research funders, like NIH and AHRQ.
As much as this provides evidence of able leadership at PCORI, the organization is far from achieving stable footing. Most Americans trust their doctors and are not aware of the problem that PCORI was created to solve: over half of medical treatment is given without sufficient evidence (Institute of Medicine 2009). With a disengaged public and only weakly mobilized patient groups, PCORI-funded researchers are its most likely allies when it comes up for renewal. In its 2015 review of the organization, the General Accounting Office (2015) noted that PCORI will unlikely be able to show effects from its research in terms of reducing practice variation or changing health care delivery. The General Accounting Office indicated that this is not a weakness of the organization but a characteristic of the time it takes to produce research findings that might change practice. This will leave PCORI vulnerable to arguments of government waste, burdensome taxation, and intrusion into market and professional domains. Practice-changing results could alter this list of concerns. Yet, as we have seen with past HTA/CER efforts, practice-changing studies are likely to mobilize antagonistic professional and/or trade groups.
The PCORI case suggests that the challenges of establishing credibility in politically contested environments are formidable. New organizations may be especially vulnerable. Though PCORI is a particularly American approach to CER, relying heavily on the private sector to conduct research and respond to its findings, persistent concerns about its potential regulatory impacts may prevent PCORI from reaching its full potential in generating a more comprehensive evidence base for delivering health care. Should PCORI be terminated, future policy options for generating CER in the United States are not obvious. One strategy would be to house a new CER effort within the NIH, capitalizing on that agency's strong reputation and benign distributive political environment to protect a CER program. Given the active lobbying of the medical products industry regarding CER legislation, it is not clear that establishing CER within NIH would surmount likely legislative hurdles. Another policy approach would be to create greater incentives for private-sector use of existing CER. Organizations like the Department of Veterans Affairs and Kaiser Permanente Family Foundation, because of their respective settings and structures, already produce in-house CER. CER efforts in OECD countries may add to the current evidence base. Nudge policies to encourage providers to explore the outputs of these organizations have the potential to increase the evidence base in medical treatment without stirring the politics of a centralized, publicly funded CER program.
The challenges faced by PCORI do not invalidate the idea that reforms can create new and supportive political economies or the idea that entrepreneurial behavior on the part of an organization's leaders can create active networks of mobilized supporters. Because political science scholars are divided about the ability to reshape political economies to enable general interest reforms, more studies of the relationship between policy environment, agency leadership, and capacity are needed to articulate the conditions under which public organizations can successfully manage difficult stakeholder environments. This is especially important where policies are intended to produce broad societal benefits that may not spur strong stakeholder mobilizations.
This research received funding from a UC Berkeley Faculty Research grant and from the Robert Wood Johnson Foundation Scholars in Health Policy Research Program. We thank Steve Shortell and March Schlesinger for support and valuable feedback. We also thank anonymous reviewers at JHPPL whose comments led to a stronger analysis. Most important, we want to thank everyone who agreed to be interviewed as part of this study.
Legislative pork refers to appropriations for local spending that are created specifically to bring federal funds to home districts rather than serving some larger policy goal.
Lowi argues, from a theoretical perspective, that distributive politics are not zero sum in that legislators can always increase an agency's budget authority to accommodate new groups. Rachel Best (2012) provides empirical support for Lowi's contention, arguing that patient mobilizations targeting NIH had the effect of increasing NIH spending across diseases rather than increasing the gains of some at the expense of others.
We seek to distinguish between health care researchers accustomed to the NIH model of research that is dominated by experts and involves patients at the margins (Kleinman 2000: 141–42) and researchers who embrace the idea of patient-centered research where experts and patients work much more collaboratively throughout the process of research from articulating research questions and designing studies all the way through to disseminating research findings.
Data collection through interview was reviewed and approved as exempt by the University of California Office for the Protection of Human Subjects. Researchers recruited interviewees with a letter describing the research and a consent form indicating that confidentiality would be protected and that individual names would not be used in publications or presentations stemming from the research.
Appendix A provides a more complete account of recruitment efforts for the stakeholder interviews. Appendix B provides a numbered list of interviews that distinguishes between the key informant and stakeholder interviews.
We coded every other hearing in the returned sample under the assumption that chronological order of hearings does not correlate with patterns in witness lists or with perspectives offered. Because the entire period was included, this sampling approach would capture any shifts in perspective over time.
This passage can be found on the group's website under “Issues”: www.pipcpatients.org/patient-centered-outcomes-research.html (accessed June 30, 2016).
The next section and table 7 give an overview of the criticism that grantees have leveled at the organization: that it is too slow in getting grant money out the door, that its rules for grantees are too cumbersome, and that it has generated uninteresting research questions.
The draft bill, which has no bill number since it was a draft, can be accessed at appropriations.house.gov/uploadedfiles/bills-114hr-sc-ap-fy2016-laborhhs-subcommitteedraft.pdf (accessed June 28, 2016). The Senate appropriations draft bill did not include cuts to PCORI and cut AHRQ's budget by only 35% instead of proposing to eliminate the agency.
AHRQ's budget, however, was cut from $334 million in fiscal 2015 to $280 million for fiscal 2016. Figures are from www.hhs.gov/about/budget/fy2017/budget-in-brief/ahrq/index.html (accessed June 28, 2016).
See appendix C for a list of questions included in the interview guide.
The view that the patient role in NIH decision making is limited is echoed by Kleinman's (2000) research. He found that the patient perspective is incorporated too late in the process, after experts have approved a subset of proposals, to have much influence on outcomes.
Two organizations indicated that they did not lobby (interviews 3 and 10), and one interview did not include a discussion of future lobbying related to PCORI (interview 7).
McCoy et al. (2017) found that as many as 83% of patient groups receive funding from drug, device, and biotechnology companies and have industry executives serving on their governing boards.
Appendix A: Stakeholder Interview Methodology
Early comments on PCORI activities provided insight into PCORI's stakeholder landscape, but also indicated that this landscape, at least at the outset, was unstable. Based on reviewer comments on an earlier draft of this article, the authors decided to interview current stakeholders in 2017 to try to capture PCORI's more recent stakeholder landscape. The decision to try to conduct interviews stemmed from the fact that more recent opportunities for public comment yielded a comparatively small number of commenters. For example, in response to PCORI's “2016 New and Revised Methodology Standards,” 16 organizations/individuals submitted public comments, compared with 128 individuals or groups that submitted comments in response to PCORI's draft methodology. Since the drop in participation might indicate either that stakeholders are disillusioned and no longer engaged or that they are satisfied with PCORI's process and progress, we interviewed stakeholders in an effort to develop a clearer picture of current stakeholder views.
While interviews can be a rich source of data, recruitment is very difficult given that PCORI stakeholders are not tightly networked. This limited our ability to conduct a snowball sample where, after securing the first several interviews, one can recruit using within-network referrals. In this case, all of our recruitments were cold contacts and yielded a low response rate. Our initial approach was to contact a random selection of participants who attended PCORI's 2016 stakeholder meeting. This approach failed for two reasons. First, the list of attendees included only participant names and institutions. In most cases, academic researchers were the only ones for whom Internet searches provided us with actual contact information. This introduced significant bias in whom we were able to contact from the 2016 stakeholder meeting. Second, academic researchers were very hard to recruit into the study. Most either did not respond or declined to participate.
Our second approach was to review past public comments and look for actual submitted letters since these contained footers with contact information of the letter writer. Using the combined approaches, we were able to recruit ten stakeholders representing a cross section of types of PCORI stakeholders. The interviews include at least one stakeholder from all of the types of stakeholders discussed in the theoretical framework (patients, researchers, professional associations, and the medical products industry). At the same time, with the exception of patient groups, three of which were interviewed, the cross section represents very little depth in any given sector. The limits of the sample size are taken into consideration in reporting and interpreting results from stakeholder interviews.
Appendix B: List of Interviews
Key Informant Interviews, 2013–2015
1. July 2013
2. October 2014
3. January 2015 a
4. January 2015 b
5. April 2015
Stakeholder Interviews, 2017
6. Patient group—parents of children with disease/disability—Transcript ID #2201
7. Professional association—Transcript ID #5501
8. Nongovernmental organization—Transcript ID #8801
9. Provider organization—Transcript ID #3301
10. Trade association—Transcript ID #4401
11. Patient group—broad class of diseases—Transcript ID #2203
12. Patient group—patients in general, no specific disease—Transcript ID #2202
13. Patient/industry group—Transcript ID #9901
14. Academic research institution—Transcript ID #1101
15. Government organization—Transcript ID #6601
Appendix C: PCORI Stakeholders Interview Questionnaire
How often have you/your organization lobbied PCORI or collaborated with PCORI since it began in 2011?
What are the main issues that brought you/your organization in contact with PCORI?
Have those issues changed over time? If so, how have they changed?
Thinking about you/your organization's interactions with PCORI, has PCORI been responsive to your concerns and interests?
Have you/your organization lobbied Congress about PCORI? If so, how often and what was the nature of the concerns you/your organization brought to Congress?
Thinking about the number of types of interest groups that have tried to shape PCORI's mission, research portfolio, and methods, how well do you think PCORI has balanced competing stakeholder demands?
Given PCORI's mandate, to fund patient-centered outcomes research without attention to costs, how well do you think the organization has performed in its first 5 years?
Do you think PCORI deserves to be reauthorized? Why or why not?
How supportive are you/your organization of the idea of publicly funded comparative effectiveness research, in general?
To your knowledge, have you/your organization participated in past federal efforts to fund and conduct comparative effectiveness research? If so, can you tell me a little bit about those efforts?
If Congress maintains PCORI and its funding, are you/your organization likely to continue to work with PCORI in the future?
Overall, how would you characterize your (or your organization's) view of PCORI?
Thinking about staff time, organizational resources, and direct lobbying expenditures, how much money has your organization spent on lobbying related to PCORI?