The California Health Policy Research Program — supporting Policy Making Through Evidence and Responsive Research

Abstract

This article explores the creation, design, and execution of a university-based collaboration to provide responsive research and evidence to a group of diverse health care, labor, and consumer stakeholders through convening a funded series of deliberative meetings, research briefs, peer-reviewed journal articles, ad hoc data analyses, and policy analyses. Funded by the California Endowment, the California Health Policy Research Program was created by researchers at the University of California, Berkeley Center for Labor Research and Education, and the UCLA Center for Health Policy Research. The collaboration not only allowed new research and analyses to be used by stakeholders and policy makers in decision making but also allowed university researchers to receive input on the important health policy issues of the day. The guidance of stakeholders in the research and policy analysis process was vital in driving meaningful results during an important time in health policy making in California. The manuscript discusses lessons learned in building relationships with stakeholders; meeting research and analytic needs; engaging stakeholders and policy makers; building capacity for quick-turnaround data collection and analysis, dissemination and publication; and maintaining the collaboration.

Introduction

Most academic health policy researchers really care about expanding access to needed services and improving the quality of health care in the most efficient and equitable way possible. Many do research with a hope that their findings will make a difference. Yet these hopes are often dashed by basic practical considerations. First, research is often time-consuming and may not respond to the short time frames within which policy makers must make decisions. Second, for their own promotion, scholars must publish in academic journals, which tend to have long publication timelines and are not always easy for policy makers and the public to digest. Third, because scholars and policy practitioners do not typically communicate, there is often a mismatch in the questions posed by policy makers and research conducted by academics.

In 2007, California's efforts to enact statewide health reform led to unusual scholar-practitioner collaboration. Funded by the California Endowment, UC Berkeley and UCLA researchers began collaborating on a new, stakeholder-driven model for research, which was designed to shape policy rather than understand its impacts after the fact. Because key stakeholders were willing to provide guidance and input to prioritize research needs and identify gaps in the evidence base, researchers were ultimately able to provide information that has helped policy makers and stakeholders understand policy impacts. This collaboration has proved fruitful for all parties — policy makers, stakeholders, and researchers: policy makers and invested stakeholders are receiving more useful and timely information, while researchers can genuinely feel that their work is making a difference.

The purpose of this essay is to provide more detailed information about this five-year-old collaboration with the hope that similar models may be replicated elsewhere around the country. In addition to describing the collaborative process, we also suggest ways in which other health services and health policy researchers can capitalize on practitioner insights and input to shape their own research to address the important health policy issues of the day. We begin this discussion below by considering what we know to date about how research and evidence is currently used by policy makers. We use this knowledge to construct how a built collaboration can overcome common obstacles toward conducting policy-relevant research and policy makers being able to capitalize on useful research.

Given the important role of states in operating Medicaid and Children's Health Insurance Programs (CHIP), the ongoing implementation of the Affordable Care Act, and innovation occurring in local health care systems, it is vital to have researchers who understand the dynamics of the policy debates, the need for information, and the ability to deliver responsive research to help shape those policy decisions. We propose that collaborative research teams, informed by partners working in the interface between politics and health policy, can be very helpful in advancing discourse and providing evidence to drive policy decisions.

Background: The Use of Evidence in Policy Making

The use of research or social science evidence by legislators to make rational choices between a set of policy options appears to be a fiction (Caplan 1979: 459–60). Although research into social issues and evaluation of social programs has increased since World War II, it is still apparent that there is a disconnect between the production of research and policy analysis for consumption by policy makers, and the actual use of the research and expert analysis to make rational decisions by comparing multiple policy options (Shulock 1999: 227–28; Weiss et al. 2008: 30–31). The divide between social scientists and policy makers is so vast, according to the conventional wisdom, that “social scientists and policy makers live in separate worlds with different and often conflicting values, different reward systems, and different languages” (Caplan 1979: 459).

If the gap is insurmountable and rational choice theory does not apply to the policy-making process, researchers and policy analysts have two potential choices: (1) stop doing research and policy analysis, as it will go unused, or (2) conduct research and policy analysis from a different perspective, by taking into consideration the needs and goals of policy makers and stakeholders in the policy process. This type of knowledge utilization and collaboration could partly bridge the gap between social science and policy makers when it comes to macro-level policy decisions (Caplan 1979: 466–67).

Other experts in agenda setting and policy making suggest that legislators are “bombarded” with too much information supplied by interest groups, think tanks, professors, and researchers within the policy-making process. The real issue then becomes, how do policy makers and their staffs interpret and prioritize that information (Jones and Baumgartner 2004: ii–v)? While not as damning as the idea that empirical research and evidence are not useful to policy making, it is still a daunting task to make sure that evidence is appreciated and used in making decisions. However, Caplan (1979: 461) warns that quantity of interaction is no substitute for quality and that social scientists interacting with policy makers must consider ideology and values in addition to technical expertise.

Ideology and values are important considerations, especially in the context of modern policy making, where choices are constrained by political realities and the evidence available. Other researchers have examined the use of evidence in decision making more generally, rather than in policy making. Weiss (1980) postulated that in large organizations, decisions do not spring from systematic investigations. Instead, research provides the context for decision makers to make decisions. There is a dependence on researchers to provide evidence, but often it goes to providing broader knowledge that informs decisions (Caplan 1979: 463–65; Weiss 1980). Unfortunately for researchers who want to influence policy, Weiss and others also found that among congressional staffers, the “enlightenment” use of research to inform one's worldview is less frequent, and legislators commonly use research to support preexisting beliefs and positions. So research is important, but not to guide decision making (Jenkins-Smith and Weimer 1985; Weiss 1989: 424–25).

Other researchers found that evidence played a role in policy making, but that its impact and use varied depending on legislative timing and environment. For example, in the early stages of policy making (i.e., writing legislation, exploring policy options to author a bill), research will be consulted more often and could play a substantive role in shaping policy (Mooney 1991: 445–55; Whiteman 1985: 294–311). One might argue that this formative role of research in the policy-making process allows researchers to be policy entrepreneurs rather than policy analysts (Kingdon 2011: 122–24, 179–82, 204–5). In high-pressure or high-conflict circumstances, there is support for the idea that researchers are viewed as outsiders in the policy process and that their work may be used to strategically advocate for preconceived policy positions, as most evidence to support decisions is coming from insiders, such as congressional staff, colleagues, and interest groups (Mooney 1991).

While the typical “objective” policy analysis may not be used to compare two choices and determine the appropriate course of action, decisions are still being made based on existing knowledge and perception. Researchers and analysts who want to contribute in the arena have to figure out a way to be involved, stay informed, stay relevant, and serve a purpose in guiding policy makers to easy-to-understand, insightful evidence that may shape decisions and provide needed information.

Background: Developing Collaborations with Stakeholders

While the literature on policy analysis and the use of research does not support the idea that research meaningfully contributes to the policy-making process, it is apparent that there are opportunities to influence health policy. In some cases, opportunities are found early in the formation of legislation as a policy entrepreneur (Kingdon 2011: 122–24, 179–82, 204–5) or in contributing to the knowledge base that policy makers use to make decisions (Weiss 1980). In others, researchers and analysts need to develop relationships and participate in coalitions that move them from “outsider” status to “insider” status (Florio, Behrmann, and Goltz 1979: 67; Mooney 1991). While insiders who have built relationships with legislators from think tanks, trade associations, agencies, and lobbying firms may be trusted sources of information, there is certainly a concern about their independence and perceived objectivity (Florio, Behrmann, and Goltz 1979).

The California Health Policy Research Program (CHPRP) was designed to contribute to decisions by making useful information available and engaging in responsive research to a diverse group of stakeholders and policy makers in California. Caplan's (1979) early work on the “Two Communities” theory and bridging the gap between social scientists and policy makers helps us to understand why collaborating with stakeholders directly on research needs could alter the use of analysis and research. However, it is apparent that in addition to technical expertise, researchers have to confront ideological and value-based conflicts, develop relationships, and establish trust in order to make contributions to policy debates and provide useful information (Caplan 1979: 459–61; Florio, Behrmann, and Goltz 1979). How researchers confront those challenges can significantly alter their ability to engage in rigorous, independent research and analysis that shapes policy decisions.

Forming a Responsive Research Collaboration

Several failed attempts at health reform in California dating back to 2003 facilitated cooperation among stakeholders interested in evidence to support policy decisions that would improve access to high-quality health care in the state. In 2003, prior to the enactment of Senate Bill 2 (SB 2),¹ an advisory group of stakeholders and researchers was convened to discuss the proposed pay-or-play bill being considered by the legislature. While the bill passed and was signed into law by former governor Gray Davis, it was narrowly defeated via referendum during Governor Arnold Schwarzenegger's first term. In 2007, Governor Schwarzenegger and democratic legislative leadership (Nuñez and Perata) both proposed health reform bills designed to improve access to health insurance coverage. The Schwarzenegger/Nuñez plan called for an employer pay-or-play provision, an individual mandate, an expansion of Medi-Cal (California's Medicaid program), and tax credits to buy private insurance through a health insurance exchange (similar to the Massachusetts health reform package that had passed one year earlier). During the policy debates in 2007 and after this bipartisan health care reform effort failed in 2008, it became clear that timely, California-specific data and research would be needed to shape future health policy decisions.

The California Endowment funded UCLA's Center for Health Policy Research, UC Berkeley's Center for Labor Research and Education, and the California Budget Project (CBP) to work on issues that would be relevant to the policy debate occurring over the summer of 2007 in Sacramento. UCLA wrote two conceptual policy pieces on the role of the safety net and the issue of the undocumented uninsured in the state (Roby, Kominski, and Cameron 2007; Yang and Wallace 2007). In addition, UCLA was asked to partner with both UC Berkeley and CBP on two policy briefs exploring the out-of-pocket premium costs for families purchasing individual market coverage. These analyses used Medical Expenditure Panel Survey (MEPS) and California Health Interview Survey (CHIS) data to analyze the affordability of coverage for individuals and families under the proposed law (Carroll et al. 2007; Jacobs et al. 2007. We were also asked to provide CHIS data on the undocumented to Dr. Gruber at MIT to assist in his micro-simulation modeling of the potential impacts of the law. During this same time period, UC Berkeley produced briefs on the employer spending on health care and the impact on businesses of the proposed employer health mandates (Jacobs, Ronconi, and Graham-Squire 2007).

While the different analyses were informative, we did not receive substantial feedback on how extensively these reports and briefs were used in the policy debate over the law. The Legislative Analyst's Office (LAO) cited our work on the insurance tax credits and the business impacts of the employer requirement in their analysis, and we know that the micro-simulation modeling was done partly using our data. However, the LAO used Survey of Income Program Participation (SIPP) data to do its own modeling of the cost of the health care reform package in California, and issued a very high estimate even though SIPP is not designed to produce state estimates (State Health Access Data Assistance Center 2013: 1).

Shortly after the failure of California's 2007 health reform effort, the California Endowment and other advocates decided that it would be appropriate to build the capacity to engage in responsive research and the development of a micro-simulation model specific to California, in the event that health reform returned to the state or national agenda. Because of our previous work for the Endowment using MEPS and CHIS to develop estimates, UC Berkeley and UCLA were invited to partner on an expanded proposal to establish a comprehensive health reform–focused research and modeling project to provide actionable evidence to guide policy decisions. The proposal included substantial funding to bring a group of experts together to advise the UC Berkeley and UCLA researchers on policy developments, provide feedback on micro-simulation estimates, and generally guide the research agenda for the group.

Main Components of the CHPRP

The most important component of the program is the stakeholder advisory board. An engaged and influential group of California health policy practitioners, they contribute valuable experience and expertise to help the research team navigate state and federal law, regulations, prioritize projects, and providing feedback. Notably, state agency and legislative staff are not on the advisory board membership. The decision not to have policy makers or public agencies on the board helped to create a space where stakeholders could talk openly about issues and feel free to ask questions without worrying about the political ramifications.

While the stakeholder advisory board members actively propose information they need to know to engage policy makers in discussions about specific policy decisions, they are careful not to exert any undue influence over the researchers to obtain a specific result. As Caplan (1979) appropriately pointed out, our interactions with the consumers of our research are limited to high quality, rather than high quantity. This approach ensures that we are targeting the most important research projects independently, based on the insights of the advisory group.

Perhaps the passage of health reform helped to target the requests around specific issues, but there has not been significant conflict over priorities for research. This agreement is partly shaped by the composition of our advisory board — there is quite a bit of commonality between the members when it comes to belief in equitable access to care and the pursuit of universal health coverage. While there is certainly disagreement about how to achieve those goals, all of the board members generally support the Affordable Care Act. While not monolithic in terms of policy actions, priorities, and opinions, the common views held by the board members help the researchers to avoid the battle over ideology and values brought up by Caplan (1979) in his discussion of engaging with policy makers. There are certainly disagreements over specific policy issues, like the Basic Health Program option. However, both sides generally agree that the analysis is needed and trust the researchers to objectively analyze the options and provide important evidence that may alter or confirm their beliefs on the issue.

Access to data and capacity to analyze those data are instrumental to responsive research and timely analysis. Spending time acquiring and learning about new data sources can be expensive and cause delays. Putting together a team of academic collaborators with lots of expertise in various software packages, data sources, and policy areas is very helpful in setting up a collaborative research project. The largest component of the California Endowment's funding support is dedicated to salary support for participating researchers. While not all researchers need or receive funding through the CHPRP, those conducting data analyses and policy research obtain some salary coverage to support that work from the grant. Each year, the grant proposal contains broad objectives, with room for decision making over the course of the year related to tasks, focus areas, and deliverables. For example, the grant proposals list the number of research briefs produced under the grant. However, only half of the briefs to be published in the next year will have a prespecified topic or focus. The remainder of the funding supports travel for advisory group members and researchers to attend the deliberative meetings convened at or near UC Berkeley, the actual cost of convening the meetings, and subcontracts to other partners, such as actuarial firms or non-UC researchers, who provide supplemental data or information that informs our projects. The advisory board members themselves are unpaid.

Our collaboration is not only cross-campus but also cross-discipline. The diverse set of skills is helpful in managing and carrying out the multiple projects we juggle. From a data perspective, California is lucky to have its own state Health Interview Survey (CHIS), administrative data from the Office of Statewide Health Planning and Development (OSHPD) on inpatient hospital discharges, ER visits, primary care and hospital utilization and financials, and a variety of other provider-level data sources. We also benefit from the California-specific Employer Health Benefits Survey,² which provides information on employer behavior and insurance offerings available to California workers. However, in states with limited local- or state-level data, the American Community Survey or Current Population Survey, combined with the national Employer Health Benefits Survey can be used in planning and analysis.

Finally, the dissemination of responsive research findings are key to success in influencing decisions by stakeholders, legislators, and agency leadership. In the case of the CHPRP, only about half of the analyses we do ends up in a published form (policy brief, policy report, or journal article). Because of time constraints and the need for evidence to reach a broad policy audience, many of the publications are internally peer-reviewed based on the editorial policies of the collaborating organizations.³ Much of the work we do is captured in PowerPoint slides, Excel tables, brief memos to legislative members at the state or federal level, testimony to the Exchange Board or legislative committees, or infographics.⁴ Because the advisory board members are made up of labor, consumer, and health care advocacy and stakeholder groups, they are well connected to state officials and legislators. We are often contacted by state offices because of referrals from the advisory board, so that our responsive research projects are not only targeted to stakeholders sitting on the advisory board but also inform other decisions made by advocates and by county and state policy makers directly.

Lessons Learned

In the process of developing, operating, and maintaining the CHPRP, we have learned several lessons that will be important to universities, their policy-making partners, and health policy stakeholders in creating their own interdisciplinary, cross-organization partnerships.

Meet people where they are

The policy issues targeted for research and analysis came out of stakeholder input. The participating researchers knew of various activities in state and federal policy making, but were not always aware of the importance of specific pieces of information or what other policy options were being considered. The advisory group more often than not told the researchers what the most salient issues were, and the research agenda came out of agreement by the stakeholders around what was needed and feasible. We were required to figure out how to measure or quantify the policy issue and get evidence to them in a timely way. The magnitude or direction of the finding did not matter as much as having the information.

Build capacity

Although resource dependent, one extremely important factor behind the ability to execute quick-turnaround, actionable research is the acquisition of staff, data, and knowledge about working with various data sources. This also allows you to accurately estimate what you can do and on what timeline. It is also important to budget support for publications. Because many of them are not going to end up in peer-reviewed journals, editing, design, and dissemination are important aspects to consider.

Do not overpromise

Stakeholders and policy makers need an answer yesterday. However, they also understand the information they need is not readily available, so they are looking for the best that you can do on their timeline. Sometimes that means making assumptions and being transparent about them, so the consumer of the information can make a decision based on its utility and accuracy.

Be flexible about publications

The currency of academia tends to be peer-reviewed publications. Participating in this type of collaboration often results in findings that need to be disseminated in alternative ways, typically through white papers, memoranda, policy briefs, or longer policy or research reports. The analyses and clearly written, actionable recommendations contained in those manuscripts are important to driving policy decisions, but may not translate to peer-reviewed publications because of time constraints and early dissemination priorities.

Develop ongoing ways to get information

In our case, after several years of two in-person deliberative meetings, we were able to expand to two in-person meetings per year (four hours each) and two conference call meetings per year (two hours each). These deliberative meetings provided the opportunity to hear from the advisory board members about the pressing policy issues of the day, which varied from decisions being made in the formation of the Affordable Care Act (ACA) (2009–10) to regulations being issued by the Department of the Treasury in response to the ACA (2011–12). Having state and federal policy experts at the table providing updates on the interpretation of the law, and new state legislation being considered, and advising the researchers on what questions needed to be answered to inform the debate, was instrumental in developing and executing timely and appropriate research projects. Over the years, the stakeholder advisory group has grown to a broader swath of stakeholder groups in the state, including organizations aligned with providers, immigration reform advocates, and small business.

Conclusions

Engaging in responsive research that helps to shape policy decisions and answer questions from stakeholders, advocates, policy makers, and the public is a rewarding endeavor. Developing the capacity to do this applied research is a challenge, but the time is right for state policy makers and health care stakeholders to make decisions based on evidence. The structure of the Affordable Care Act allows for states to make independent decisions (with federal support) on a variety of issues, from the design of their State Innovation Model grants to the structure and rules for operating a Health Insurance Exchange, or whether and how to expand Medicaid. Researchers in universities can benefit from this interest in their data, analyses, and research, as long as it is communicated in an effective way and is based on the needs of stakeholders and policy makers. Still, the quality of the research is not enough, and a successful collaboration will require time, resources, strong personal relationships, trust, and careful planning.

We would like to thank the California Endowment, specifically Robert Phillips (now of the Sierra Health Foundation) and Richard Figueroa, for supporting the creation of the CHPRP. We also thank our current and former colleagues at UC Berkeley (Dave Graham-Squire, Miranda Dietz, Laurel Lucia, Elizabeth Lytle, Korey Capozza, Brent Fulton, and Will Dow), UCSF (Janet Coffman and Annette Gardner), University of Chicago (Jon Gabel), UCSD (Rick Kronick), and UCLA (Greg Watson, Xiao Chen, Nigel Lo, Alla Bronshteyn, Jack Needleman, Michelle Keller, Shana Lavarreda, Nadereh Pourat, Daphna Gans, Rick Brown, and Christina Kinane) for their contributions to the CHPRP. Most importantly, we thank our incredible stakeholder advisory board for providing the honest guidance and insight necessary to engage in this type of collaboration.

Notes

References