“Getting to the Table”: Changing Ideas about Public and Patient Involvement in Canadian Drug Assessment

Abstract

Context: Involving patients and the public in health policy may contribute to legitimacy and accountability. However, tensions may arise between paradigms of scientific-evidence-based decision making and new ideas valuing inclusivity and patient experience when evaluating and allocating health resources. This article asks whether 10 years of experience with public and patient involvement in Canadian drug assessment has affected participants' ideas about how it works.

Methods: The author surveyed the ideas of participants in the drug assessment process (members of expert committees, officials, and patient groups) as described in reports and hearings in 2005, 2007, and 2012 and conducted interviews in 2014 and 2016.

Findings: The author found some consensus across groups of participants regarding the broad goals of health technology assessment (HTA) and the validity of some form of public and patient involvement. There were also important areas of disagreement and uncertainty about how public and patient involvement should be used in drug assessment and how much impact it has on deliberations and recommendations. Overall, uncertainly about the specific role for public and patient involvement in HTA limits the potential for ideational change among participants.

Conclusions: These findings have implications for evaluation of public and patient involvement, the way we understand ideational change, and practical questions of communicating health resource decisions.

Who should be involved in health policy decisions? What types of evidence are relevant when making choices about how to allocate scarce health resources? How should expert knowledge be balanced with lived experiences of illness? These questions and others about how to include patients and the public in decision making are long-standing in health policy (Charles and DeMaio 1993; Florin and Dixon 2004). Various methods of public and patient involvement are now implemented in health systems as diverse as Brazil, Britain, the Netherlands, and Canada (de Freitas 2017). The United States has for a number of years been experimenting with new ways to include patients and the public in decision-making processes from drug development to product labeling (Basch et al. 2015; Perfetto et al. 2015). Introducing new perspectives in health policy decision making can challenge existing paradigms for evaluating and allocating health resources. Participants' ability to adapt their ideas about evidence, expertise, and objectivity may be a key factor in whether more inclusive decision-making processes actually produce greater legitimacy and accountability. However, to date there have been very few attempts to measure such ideational change.

This article examines ideational change among participants in two Canadian drug assessment processes, which make recommendations about which drugs should be reimbursed by various federal and provincial public drug insurance plans. The province of Ontario has used cost-effectiveness analysis as part of drug assessment since the early 1990s. In 2003, the Common Drug Review (CDR) was created to apply cost-effectiveness analysis at the pan-Canadian level. These analyses are types of health technology assessment (HTA), a way of systematically evaluating the costs and benefits of drugs, devices, and procedures (Goodman 2014).

Both Ontario's Committee to Evaluate Drugs and the pan-Canadian CDR began to adopt public and patient involvement measures in 2006. The available evidence suggests that public and patient involvement was adopted mainly to address legitimacy goals: participants were divided about the legitimacy of drug assessment, and public and patient involvement was one way to change ideas and improve legitimacy. However, public and patient involvement is sometimes seen to be in tension with other ideas about objectivity and scientific evidence in HTA. Introducing public and patient involvement to HTA means introducing practices and ideas that challenge the work and worldview of some existing participants: How do they incorporate these ideas or adapt their own over time?¹ How do the ideas of new participants (patient groups and lay members of drug assessment committees) change as they gain experience with the process?

We currently do not have a good basis for measuring this type of ideational change, much less understanding the conditions under which it might occur. This research asks whether the ideas participants hold about HTA and public and patient involvement have changed since 2006, and whether the processes are meeting their goals in the eyes of their participants. It does so using unique longitudinal evidence from published surveys and reports, parliamentary hearings, and interviews I conducted about how different groups of participants (technical and lay members of drug advisory committees, agency and ministry officials, representatives of patient groups) understand and evaluate drug assessment in Canada.

This research responds to calls in the health policy literature for more evaluation of public and patient involvement processes (Abelson and Gauvin 2006; Jabbar and Abelson 2011). There is a robust literature that seeks to identify goals for public and patient involvement (Gauvin, Abelson, and Lavis 2014; OHTAC Public Engagement Subcommittee 2015) and to identify and in some cases apply criteria by which processes may be evaluated (Abelson et al. 2015; Berglas et al. 2016; Burton 2009; Emery, Mulder, and Frewer 2015; Mitton et al. 2006; Rosenberg-Yunger and Bayoumi 2017; Rowe and Frewer 2000). However, the research presented here is among the first to evaluate public and patient involvement processes over time, and to do so based on participants' own views rather than external criteria.

This research also contributes to literature on ideas and public policy. It is interpretive rather than causal and sees describing ideational change as a necessary precursor to explaining ideational change more rigorously (Béland 2010; Berman 2012; Daigneault 2014). By ideas I mean a set of cognitive and normative beliefs and perceptions regarding the goals of HTA, the role of public and patient involvement in drug assessment, and the particular strengths and weaknesses of the process as the participants experience it.

Although the content of these ideas varies by type of participant (and perhaps over time), they can be organized into a general framework to structure comparisons. First, participants have ideas about goals for HTA: what purpose it serves and what it should achieve. Abelson et al. (2007: 40) distinguish between process-oriented goals, which aim to “improve the legitimacy of decision-making” and achieve “fair and transparent processes,” and instrumental goals, which aim “to inform policy decisions” and to find “meaningful ways to gather input from relevant patient groups and publics and make better quality decisions.”

Ducey et al. (2015: 14) have found that technical experts tend to have “moral commitments” to a particular view of objectivity and evidence (see also Lehoux et al. 2009: 2003). Ducey et al. use the concept of moral economy from Daston (1995: 23), referring to “specific constellations of emotions and values” that affect “how scientists at a given time and place dignify some objects of study . . . and trust some kinds of evidence and reject other sorts.” These insights about the value-laden nature of certain scientific ideas suggest that for some participants goals related to improved decisions may be inextricably linked to process-related goals: meeting certain standards for objectivity and good evidence is necessary for the process to be legitimate. It is also important to note the possibility that participants will hold competing conceptions of legitimacy. Is a decision-making process legitimate because it considers input from the full range of actors affected by the decision, or because it adheres to a moral view of scientific objectivity, which might exclude certain actors precisely because of their interest in the decision?

This article describes goals as primarily related to legitimate processes (legitimacy goals) or improved decisions (instrumental goals), but these categories still potentially encompass different interpretations and expressions of legitimacy or improved decisions. Legitimacy may be linked to representation, for example, by allowing all affected constituencies to participate in a deliberative process (Warren 2009). Representation may be limited to symbolic inclusion, or it may involve empowerment, where new kinds of participants are able to influence decisions (Johnson 2009). Legitimacy may also be about accountability (who can hold decision makers responsible and how) and/or transparency (can stakeholders see and understand how decisions are made?). Improved decisions may involve providing context for clinical data and accounting for health outcomes that are valued by patients (Staley and Doherty 2016), as well as considering information about unmet needs, quality of life, or the often cited but generally undefined “social values” (Abelson et al. 2013). Alternatively, in this research some participants report thinking decisions would be improved by information about differences in how patients respond to a given therapy, which could potentially be provided by patients themselves. This points to a challenge in defining and measuring what constitutes meaningful public and patient involvement that has been reflected in other recent studies of Canadian drug assessment (Rozmovits et al. 2018) and is discussed further below. In this article I try to specify how various participants interpret the two broad categories of goals but acknowledge the potential for crossover and for vagueness in how participants themselves describe goals.

The second component of participants' ideas about HTA involves their evaluations of whether and how goals are being met. Identifying strengths of the process provides an opportunity for participants to reflect on how these strengths contribute to their goals for HTA and public and patient involvement. Challenges may be related to achieving legitimacy goals, instrumental goals, or both, and groups may differ in whether they see these challenges as “fixable” or related to fundamental tensions between public and patient involvement and HTA. One measure of the degree to which public and patient involvement meets instrumental goals deserves special mention, because it derives from my pilot interviews: a respondent suggested asking other respondents to recall instances when, in their view, public and patient involvement changed the recommendation of the drug advisory committee. This is a narrow view of impact that omits important nuances in the role of public and patient involvement (Berglas et al. 2016; Li et al. 2015; Staley and Doherty 2016), and interviewees' responses sometimes reflect that. Nonetheless, it does capture one straightforward aspect of impact that matters to a variety of participants. Taken together, these ideas about the goals, strengths, and challenges of HTA provide a reasonably comprehensive way to describe how participants understand the process, as well as a framework for making comparisons both among groups and over time.

Background

Canada was an early adopter of formal cost-effectiveness analysis as part of its drug assessment process. This type of analysis compares the economic cost of a drug to its therapeutic benefits relative to other drugs when making listing decisions. At both the provincial and national levels, the adoption of cost-effectiveness analysis was motivated primarily by a desire for more consistent, evidence-based recommendations, the possibility of achieving better value for money in drug reimbursement, and more harmonization in assessments and listing decisions (Boothe 2016). Communication to the public or patients was not initially a major concern, nor was involving the public or patients in the assessment process (Ekos Research Associates Inc. 2005: 4).

This began to change in the early 2000s. At that time, representatives of patient groups (and the pharmaceutical industry, which helps fund many of these groups)² expressed significant frustrations with what they saw as a time-consuming and opaque drug assessment process. In the 2007 hearings of the parliamentary Standing Committee on Health (2007a: April 30), about 7 months after the first public members joined a drug assessment committee, a representative of the Colorectal Cancer Association of Canada noted, “The perception of patients is that there is a greater emphasis on cost containment rather than on ensuring patient access to important medications,” and a representative of Canadian Breast Cancer Network said, “The actual criteria upon which decisions are based are not apparent to the average Canadian.”

In 2006, Ontario's Committee to Evaluate Drugs and the national Canadian Drug Expert Committee, which produces the CDR's recommendations,³ both added lay members—either members of the public or patient representatives—to their committees of clinicians, pharmacists, health economists, and ethicists. The Ontario committee added two patient members, who must be “living with a chronic disease . . . or acting as a caregiver to such an individual [and have] knowledge of the health care system” (Ontario Public Appointments Secretariat 2013). The Canadian committee added two public members whose responsibility is “to bring a lay perspective” to the committee (CADTH 2017a) and who cannot be a member of a patient advocacy group, as Ontario patient members typically are.

Initially, the role of lay members was not clearly defined. One early lay member reported that “nobody really understood our role” (LAY14), and two interviewees commented that the role of public and patient members has evolved over time (LAY14, TEC10). However, it appears that, from the beginning, lay members of both committees sought to collect and interpret information from patient groups, and this was formalized in 2010 when both committees adopted a process for submissions from patient advocacy groups. As of 2016, the process at both committees is for patient groups representing a condition to complete a template with their input on each new drug relevant to their condition when it is considered for reimbursement. This submission is then included in the committee's packet of information for a given drug. The lay members' role on both committees is to summarize and present patient submissions at committee meetings, to participate in deliberation with technical members of the committee, and to vote on recommendations.

Methods

This research asks to what extent have participants' ideas about public and patient involvement in drug assessment changed since these processes were introduced. The early years of public and patient involvement are captured by five reports by consultants and a parliamentary committee (Ekos Research Associates Inc. 2005; Lague and SECOR Consultants Ltd. 2012; SECOR Consultants Ltd. 2011, 2012; Standing Committee on Health 2007b). These reports represent all publicly available evaluations of the CDR since its inception. To capture more recent views, I conducted interviews with HTA participants. I conducted 2 interviews in 2014 as part of a pilot study and conducted 13 additional interviews in 2016.

The published reports included the views of committee members, expert reviewers (who prepare the clinical and economic reports that form the basis of committee deliberations) and agency officials, representatives of patient groups, and industry associations. The reports contain both direct quotes and summary statements of groups' views on various topics. Transcripts of the parliamentary hearings conducted by the Standing Committee on Health during its 2007 study were also used. While these reports and transcripts are not a direct substitute for interviews conducted by a single researcher over time, my interviews were designed to address themes similar to those in the reports' surveys and interviews. This makes the information as comparable as possible, and the evidence from the reports is coded and analyzed in the same way as the new interview transcripts.

My interviewees included nine current or former members of the Canadian and Ontario committees (six technical members and three lay members), three officials at the Canadian Agency for Drugs and Technologies in Health (CADTH) and the Ontario Ministry of Health and Long Term Care, and three representatives of patient groups (see table 1). This means I have longitudinal evidence for patient advocacy groups, agency officials, and technical members of the committees. Lay members were not identified in the 2005–12 reports, so I can provide evidence of their views only from the 2014–16 interviews, but because their ideas differ from both technical members and patient groups, I include them here. This article discusses participants' perspectives on the differing roles of patient versus public members but, for confidentiality reasons, groups all patient and public member interviewees together as “lay members.” This does not occlude potentially important aspects of the analysis, because I found that lay members (both patient and public members) have views that are similar to one another and quite distinct from the views of patient advocates who are not members of a drug advisory committee.

The pool of potential committee member interviewees is relatively small: the Ontario committee has 10 technical members and 2 patient members, and the Canadian committee has 12 technical members and 2 public members. The pool of officials and patient groups is larger: as of 2015, 114 separate patient groups made submissions to the CDR (CADTH 2015). The patient groups interviewed included a large health charity with extensive experience making submissions, a small volunteer-run association with extensive experience, and a second small volunteer-run association that had recently made its first submission. This variety gives more confidence that points of agreement among the patient groups are reasonably representative of the broader patient community.

The 2005–12 reports did not survey or interview participants in the Ontario process, but the overlap between technical members of the two committees and the similarity in their views in the 2016 interviews suggest that the two groups of experts likely had similar views earlier in the process as well. Interviews with participants in the Ontario process allowed the study to achieve variation in type of lay members who sit on the committees (patient members on the Ontario committee, public members on the Canadian committee). As discussed below, the potentially differing roles of patient versus public members is currently being debated in Canadian drug assessment and is an important area for future research. Industry representatives were excluded because public and patient involvement is more directly concerned with legitimacy in the eyes of patients and members of the public, but given the links between industry and patient groups, future research will include these respondents.

The reports and interview transcripts were coded using NVivo 11 software. The study used an open coding method to identify concepts and common themes. As the transcripts were read and reread, topics key to participants' understanding of HTA and public and patient involvement began to emerge, and analytic memos were developed to help clarify patterns among themes and across types of participants. This allowed me to compare participants' perspectives on different aspects of public and patient involvement both across time and across types of participants and to ask such questions as whether patient groups' perspectives on the challenges of public and patient involvement had changed over time and whether they currently differed from the perspectives of committee members.

Findings

Canadian drug assessment in the early 2000s was marked by disagreement between technical experts and patient groups about the legitimacy and value of HTA. However, in 2016 all categories of participants identified evaluating the clinical and cost-effectiveness of drugs as a key goal for drug assessment. They said drug assessment processes should provide drug plans with the “best information available with respect to potential cost-effectiveness issues and [should be] informed by patient concerns” (OFF02), should determine the “benefit and cost-effectiveness of drugs” (PAT11), and should consider this relative to other drugs currently on the market (TEC10).

Goals for public and patient involvement are less clear. The 2005 and 2007 reports evaluating the CDR contained no explicit discussion of what public and patient involvement should achieve. The lack of clarity around goals for public and patient involvement continued after its implementation. A key finding of a 2012 study was that “patients, committee members, stakeholders, manufacturers, and CADTH staff are not fully aligned on the objective of the patient input process” (Lague and SECOR Consultants Ltd. 2012: 2).

One important example of this ongoing lack of alignment on goals for public and patient involvement is the lack of clarity about the potentially different roles of public and patient members. Public members appear to have been added to the Canadian committee primarily as a response to demands for inclusion by patient groups (MacPhail and Shea 2017: 50) and a reluctance on the part of experts in agencies responsible to involve patients directly on the committee (Standing Committee on Health 2007a: June 6 [John Wright]). Patient members were added to the Ontario committee as part of Ontario's Transparent Drug Systems for Patients Act (bill 102), which aimed to “strengthen . . . transparency by giving patients a role in drug listing decisions” (Drug System Secretariat 2006).

It is unclear why the Ontario committee and other national committees, like the pan-Canadian Oncology Drug Review, chose to appoint patient members instead of public members, and in fact, the distinction between “public” and “patient” members appears to have been something of an afterthought at this time. However, the different roles of patient and public members are currently a subject of intense deliberation among researchers, agencies, and members of patient communities (Menon et al. 2017). In 2016, lay members of both committees reflected on the potentially differing roles of patient and public members:

The role of patient versus public members was discussed at meetings of lay members from a variety of pan-Canadian and provincial HTA committees at the CADTH symposiums in 2016 and 2017. Members discussed the need for greater clarity in these differing roles and the potential tensions between social and patient values.⁴ Some interviewees from other groups of participants comment that the distinction may be artificial:

Other interviewees, however, saw the roles as having distinct strengths and challenges:

There is some evidence of ideational change here. “Patient” and “public” member positions may have been created without a good deal of consideration for the different contributions they could make, but many interviewees now see a distinction. Unlike many ideas about HTA discussed below, interviewees' views on the potentially differing roles of patient versus public members do not appear to be linked to their position as lay members, technical members, or officials. There is some literature on differences between patient members and public members as they relate to HTA and to public and patient involvement in standard settings (Knaapen and Lehoux 2016; Menon and Stafinski 2011), but it remains an important area for research. In particular, future research should investigate what makes ideas about lay member roles potentially different from other ideas about the overall goals of public and patient involvement, an issue that is beyond the scope of this article.

In the remainder of the article I describe the ideas held by different types of HTA participants over time, in terms of their perceptions of the goals, strengths, and challenges of the process. I then discuss what these findings mean for our understanding of changing ideas about HTA and conclude with directions for future research.

Officials

When public and patient involvement was introduced, CADTH officials reported thinking that the CDR was already a fair, transparent, and accountable process (Ekos Research Associates Inc. 2005; Standing Committee on Health 2007a). The CEO of CADTH described the CDR as setting “new standards of transparency for drug reimbursement in Canada and abroad” and addressed criticism that the CDR was a covert method of cost control: “Cost-effectiveness is considered only once improved health outcomes have been demonstrated” (Standing Committee on Health 2007a: April 25). CADTH officials and officials in the federal, provincial, and territorial governments who fund CADTH discussed a need for improved communication with the public and greater transparency but noted that transparency should be “a two-way street” (Standing Committee on Health 2007a: June 6 [John Wright]), pointing to the pharmaceutical industry's insistence on keeping parts of its submissions confidential and the need for patient groups to disclose financial ties to industry (Ekos Research Associates Inc. 2005; Standing Committee on Health 2007a).

In 2016, CADTH and Ontario Ministry of Health respondents remained quite positive about the work of HTA. Their goals for HTA were mainly focused on improved decisions:

When asked about goals specific to public and patient involvement, officials mentioned goals related to both legitimate processes and improved decisions. Some respondents distinguished between these goals, suggesting they see both symbolic and substantive aspects of public and patient involvement. Even within individual responses, a number of versions of legitimacy were sometimes cited, for example, accountability and an “all affected” representational principle.

Officials identified strengths that were closely aligned with their stated goals for public and patient involvement of increasing democratic legitimacy and helping produce better, more robust decisions. One interviewee said, “It's helped to increase the openness or transparency of the processes . . . because people can see that the relevant people are involved, in the process itself, and that the decision process itself is one that they can trust, and it's fair” (OFF02). Another noted that public and patient involvement provides “some degree of democratic accountability to the drug review process, and . . . I think that the patient evidence piece also provides some context to the clinical findings from trials” (OFF01).

Officials also commented on the ways dissemination of information about drug listing recommendations is linked to legitimacy. Some noted that CADTH was using new forms of information dissemination, such as feedback letters to submitting patient groups, with good success, but all felt that the agency or ministry could do more to communicate its reasons for recommendations to patient groups, clinicians, and the public (OFF01, OFF02, OFF03).

In 2016, officials noted challenges related to both instrumental and legitimacy goals. Instrumental challenges included providing sufficient training and resources for lay members and patient groups, as well as the variable quality of patient submissions. An official acknowledged patient groups' frustration with the template for patient submissions, noting that “it has a lot of instructions and technical detail. . . . We felt that's sort of alienating to the patient groups as well, because we just want them to be relatively plain speaking, and so the questions ought to reflect that expectation” (OFF02). The official continued, suggesting that a lack of clarity in how instrumental goals should be met made it difficult for patient groups to know how their input was used, a key element of increased transparency:

The official explained this surprising lack of clear expectations for patient submissions as a product of how the original template for submissions was created, saying “the questions were posed and framed when they weren't really sure how patient engagement was going to work and what information was really necessary” (OFF02). Another official commented that defining a high-quality submission was difficult because in some instances a submission may be fairly “raw,” or lack sophisticated methods for collecting patient views, but still provide “that insight into their life, and that was the bit that we're missing for that review” (OFF03). The official continued, saying that sometimes submissions that are more sophisticated or developed do not ultimately influence the recommendation, because a given negative recommendation may be based on a lack of strong clinical trial data. The official concluded, “I don't feel that . . . the quality of patient input is something that's letting down the process” (OFF03) but that many factors contribute to whether a submission is influential in given recommendation.

A third official identified uncertainty about how patient input should be incorporated into reviews, arguing that “there isn't any current gold standard of how incorporation of patient evidence into the drug assessment process should occur” (OFF01). All three officials noted that measuring the impact of patient input on committee recommendations was quite difficult and was an area the ministry/agency needed to improve or was working to improve.

As I discuss below, concerns about what should go into a submission and how it should be used were echoed by both technical members of drug advisory committees and representatives of patient groups. All officials noted the burdens that submissions placed on patient groups and the challenges of ensuring that submissions include relevant, high-quality information (OFF02, OFF03, OFF01).

Officials had a positive view of HTA and its ability to meet its goals in the initial reports and over time have adopted goals for public and patient involvement related to both legitimate processes and improved decisions. This group expressed some reservations about public and patient involvement in 2005 and 2007, and these reservations were much less apparent in 2016. Their ideas can be summarized as believing that public and patient involvement is an opportunity to make a good process of HTA better, even while they acknowledge the need for improvement in how public and patient involvement is carried out and how these efforts are communicated to patients and the public.

Technical Members of Drug Advisory Committees

The 2005 and 2007 reports grouped technical members' views of HTA with those of officials, so they were reported to think that the existing drug assessment process was rigorous, timely, and transparent (Ekos Research Associates Inc. 2005; Standing Committee on Health 2007b). In the 2007 hearings, Andreas Laupacis, a former chair of the Canadian committee, also expressed concerns about transparency expectations extending to all participants in the HTA process (including pharmaceutical manufacturers), saying, “To my knowledge, the CDR is the only drug reimbursement committee in Canada to make the reasons for its recommendations publicly available on its website. For this degree of transparency, the CDR deserves credit. While I fully support calls for greater transparency in the CDR process, the fact is that greater transparency is needed in the whole drug evaluation system” (Standing Committee on Health 2007a: May 14).

In 2012, “most members [of the Canadian committee] believe[d] that the concept of having patient submissions is important” (Lague and SECOR Consultants Ltd. 2012: 12). However, technical members also expressed concerns about the objectivity of patient input, saying it was “inherently subjective and biased” and risked being “another selling avenue for companies” (Lague and SECOR 2012: 12), reflecting concerns about the funds many patient groups receive from the pharmaceutical industry.

In 2016, technical members tended to focus more on instrumental goals than legitimacy goals. Some technical members emphasized the potential for public and patient involvement to improve decisions by providing “context” for evidence and helping prioritize health outcomes that are important to patients (TEC06, TEC09, TEC07), which is consistent with experts' responses in a recent study of England's National Institute for Health and Care Excellence (NICE) (Staley and Doherty 2016). Interviewees describe the role of patient input as follows:

There was some disagreement among technical members about what information patient submissions should provide. A number of technical members wanted information about what it was like to live with a given disease and insights into what patients viewed as critical unmet needs—gaps or failures of current therapies (TEC05, TEC06, TEC07). For example, one technical member wanted patient input to explain “what . . . you find is the biggest inconvenience about your disease, what . . . you worry about, what would you look for new therapies to provide” (TEC06). However, other technical members wanted information about patients' experience with the drug under review, for example, “Start with the question, which is: which of you have used the drug? . . . What's your experience? And if the answer is none of them, well, I don't know what value the report has” (TEC10). Therefore, some technical members argued for new data and analysis on differences in patient responses to a therapy as a way patient input could contribute to better decisions, although they also acknowledged this type of data would be difficult for patient advocacy groups to provide (TEC10, TEC05).

Some interviewees also acknowledged legitimacy goals for public and patient involvement in HTA. One suggested that, although technical members tended to focus on the potential for public and patient involvement to improve decisions by providing context for clinical evidence, the real reason public and patient involvement was adopted was to increase HTA's legitimacy: “The role of patients . . . if you're an expert on the committee, you'll say this is to provide an added dimension that is necessary for policy making, to understand the experience and patient values. So what do patients value, and what are their experiences? So that we can understand, we can contextualize evidence. But I mean . . . my feeling is it's actually to legitimize a liberal democratic process” (TEC07).

Others noted the democratic value of public and patient involvement in both symbolic and substantive terms:

Regarding strengths in meeting instrumental goals, all technical committee members interviewed noted that public and patient involvement had some influence on deliberations and/or recommendations, although their interpretation of the significance of this influence varied. Three technical members could think of specific examples where patient input changed the recommendation, noting this was an infrequent occurrence. Technical members noted that patient input may be most influential through its potential to confirm the committee's assessment about the clinical value of a drug.

As noted above, asking interviewees if they can recall instances where public and patient involvement resulted in a different recommendation than might have been expected with only technical input is a blunt instrument for measuring impact. This is demonstrated by the fact that even technical members who could not recall a specific instance when public and patient involvement changed a recommendation were reluctant to dismiss its influence. One interviewee noted that, although they could not recall “a decision that would have been different had the patient component not been there . . . the caveat to that is the clinical review and the systematic review that is done, the clinical review is influenced by the public report, as far as what outcomes are valued” (TEC09).

Although technical members expressed mainly instrumental goals for public and patient involvement, their ideas about the strengths of public and patient involvement also included increased legitimacy. One technical member noted, “The biggest success is an acknowledgement that social values . . . are important to decision making [and that] an important contributor to determining those social values must be from the public voice” (TEC09). The technical member quoted above regarding the “optics” value of having a public or patient member (TEC05) explicitly saw this as a tool for increased accountability.

Technical members' ideas about the challenges posed by public and patient involvement were similar in 2012 and 2016. Compared to officials, technical members were more likely to identify challenges as fundamental conflicts between their conception of objectivity and evidence and the involvement of patients or the public. Objectivity of patient submissions was the most frequently mentioned challenge among this group (TEC06, TEC10, TEC05, TEC07). Four technical members expressed concerns that patient submissions were unduly influenced by pharmaceutical industry funding. A committee member said that “the patient submission information often comes from an advocacy group, that are potentially funded by drug companies, so you can see that they're biased. Or, they're not funded enough to give us information that we are looking for,” noting the difficulties of conducting a robust survey with limited resources (TEC05). Another noted that industry influence could negate precisely the type of information the committee hoped to get from patient input. “Frequently, the submissions we get have clearly got industry input in them, and those are really the most worrying because it really doesn't help us in any way, knowing what is really important to the patients and what's not” (TEC06).

Although this interviewee noted this type of problem occurred “frequently,” an official described it as “not a huge issue,” owing to the conflict of interest disclosures required on the patient input form (OFF01). Other respondents did not comment on how frequently they observed problematic submissions. Two technical members cautioned that potential conflicts of interest were not restricted to patient groups, noting that clinicians who advised the committee also had to disclose any financial ties to industry (TEC05, TEC10).

Five technical members also raised concerns that the colloquial or anecdotal information provided by patient input was at odds with or was difficult to integrate into a rigorous, evidence-based process (TEC05, TEC06, TEC07, TEC09, TEC10). This is consistent with other recent research on drug assessment (Rozmovits et al. 2018) and is distinct from the question of conflicts of interest. Even without industry funding, patient input is necessarily different from the type of information provided by clinical trials. One member noted that some colleagues who come from “an evidence-based medicine paradigm . . . feel that [patient input] is not evidence that needs to be considered.” That interviewee continued, “To me, anecdotal evidence is still, you know . . . it's not ideal, but it tells us a lot” about which outcomes matter to patients (TEC06).

There were concerns that the quality of the evidence presented by patient submissions was poor, or insufficiently critical, although interviewees acknowledged that groups likely would need more resources to improve their submissions (TEC07, TEC05), and one commented that a way to grade patient input would be of value, similar to the way clinical trial data is graded (TEC09).

As a group, technical members appear to have experienced limited change to their ideas about HTA. They identified a variety of challenges of public and patient involvement in HTA but throughout the study period were particularly focused on objectivity, expressing concerns about the role of “unscientific” patient input that may be perceived as tainted by industry funding. However, in 2016, some technical members also acknowledged an alternate view of legitimacy, linked to inclusivity and broader practices for gathering and assessing evidence rather than a particular moral-scientific view of objectivity. This, along with fairly general acceptance of the instrumental role of “best-case scenario” public and patient involvement in improving decisions, leads me to conclude that there have been some shifts in technical members' ideas. Ideas about the goals, strengths, and challenges of public and patient involvement in HTA are not entirely coherent within the group or even within individual interviewees, so this is likely an ongoing process.

Lay Members of Drug Advisory Committees

Patient or public members, or lay members, did not exist in 2005, and in 2007 and 2012 their views were not recorded separately from technical committee members. Even though their ideas cannot be traced over time, it is useful to compare their current ideas about the goals, strengths, and challenges of public and patient involvement in HTA to the ideas of other participants because of their potential to act as a bridge between patients and technical experts.

Like technical members, lay members discussed improved decisions as a goal for public and patient involvement, specifically by providing an opportunity for the consideration of social values and for getting patient input on unmet needs. Lay members said they have a role “in assessing patient needs in relationship to . . . public values” (LAY15) and ensuring that patient priorities are identified and determining whether these needs are reflected in the information from clinical trials (LAY16).

Lay members' ideas about the strengths of the public and patient involvement process in 2016 were closely aligned with those of officials. Lay members mentioned specific improvements to the patient submission process and the benefits of lay member and patient participation in the annual CADTH Symposium starting in 2015 (LAY16, LAY15). One member described the strengths of public and patient involvement in HTA as follows, citing goals related to both legitimacy and improved decisions (eventually) through improved clinical trials:

Another lay member noted that having a formal process for public and patient involvement was a strength. I heard this comment from interviewees in all categories of participants, and it supports a focus on legitimacy goals: “Just involving patients in HTA is really important. Like having them understand what HTA is and letting them voice their opinions, I think is a success of it. I think there needs to be even more transparency, but I think that that part of having a formal process, where people can give their opinion and gather information and have it presented before a committee, is of vital importance” (LAY16).

Two lay members could think of specific examples where patient input changed the committee's recommendation: the lay member votes were influenced by patient input, and their votes were decisive (LAY14, LAY15). One also noted that a negative recommendation did not mean that patient input was discounted: “I think the patient groups sometimes, understandably, assume that if the recommendation is what they see as a negative recommendation, that we didn't pay attention to the submission or we were unsympathetic, or something. And I can say that's not the case at all” (LAY15).

The third lay member noted that, although the impact of public and patient involvement on HTA decisions was difficult to measure, they were confident that the relevant agencies were working in good faith to improve the process:

Lay committee members echoed many of the challenges identified by technical members and patient groups. They emphasized challenges regarding the quality of patient submissions and the need to measure the impact of public and patient involvement and provide feedback about its impact to patient groups (LAY15, LAY16). They also had specific concerns about their own roles on the committees. One lay member reported a concern that “there's not buy-in from all the [committee] members” regarding the value of patient input and lay members, although the member reflected that there may have been some increase in the level of acceptance over time (LAY16). An early lay member noted that “physicians found it difficult to appreciate our role and evidence. Some were very open, but some dismissed it right away” (LAY14). Another lay member noted that, from the beginning of their tenure, technical members of the committee were “were all helpful, welcoming” (LAY15). Technical members of one committee (both long-standing and relatively new) commented that lay members were “fully integrated” into the committee (TEC10, TEC09).

Lay members do appear to act as a bridge between other groups of HTA participants, acknowledging the concerns that both technical members and patients have with public and patient involvement in HTA, and sharing some of them. Lay members are most similar to officials when it comes to their buy-in to the process: although they recognize areas for improvement, they express strong support for both instrumental and legitimacy goals and note ways the current process is helping achieve these goals.

Patient Groups

The challenges expressed by patient advocacy groups (often in concert with the pharmaceutical industry) in 2005 and 2007 are quite general and reflect groups' lack of experience with drug assessment and the newness of the public and patient involvement process. Challenges concerned both the legitimacy of the process, which was seen as too opaque, and the substantive outputs, which patient groups viewed as punitive and missing key patient-relevant information. Patient groups felt that the CDR delayed access to drugs through longer listing times, which the authors of the 2005 consultancy report note is not supported by evidence (Ekos Research Associates Inc. 2005). Patient groups highlighted the need to consider quality-of-life data in addition to clinical outcomes and called for committee deliberations to be more transparent by opening meetings to the public and/or publication of committee minutes. Patient groups thought the CDR was biased against orphan drugs, wanted to know the identity of CDR reviewers, and wanted more information about CDR's reasons for recommendations (Ekos Research Associates Inc. 2005; Standing Committee on Health 2007a, 2007b).

In 2012, after a few years of experience with public and patient involvement, some of the challenges were still present. Patient groups wanted to know reviewer identity and thought the patient submission process placed too much emphasis on the potential conflict of interest posed by the funding they receive from the pharmaceutical industry. Groups also noted they had insufficient training and resources to prepare submissions and said they needed more notice of upcoming submissions and more time to complete them (Lague and SECOR 2012).

In 2016, patient groups acknowledged the goal for HTA to determine clinical and cost-effectiveness (PAT11, PAT13), although one interviewee continued, “It [a drug] might not stack up in a population, but about being clinically effective, but from a patient perspective, if it helps one or two patients . . . that haven't responded to everything else out there, and there are patients like that, then it's a really difficult thing” (PAT13). This suggests that patients and committee members might judge outputs differently: a scientifically “good” recommendation is one that uses high-quality evidence to assess clinical and economic evidence at a population level, but patients are also concerned about choice of therapies and individual impacts that HTA processes are not necessarily designed to capture.

One patient group saw public and patient involvement, in the best-case scenario, as meeting legitimacy goals by allowing patients to gain representation influence in decision-making bodies.

This interviewee described the group's goal for its submissions to HTA committees, saying they aim to provide “a sort of medical perspective” on what it is like to live with a condition while also providing “a couple examples or a couple of descriptions from patients, so people have a sense of the weight and significance at a very personal level” (PAT11). Other groups expressed similar ideas about instrumental goals for public and patient involvement—what they thought they were “supposed to” contribute to improve decisions (PAT13). However, patient group interviewees also expressed uncertainty about the most effective information to include in their submissions and about how that information was used. This echoes the uncertainty or disagreement on the part of officials and members of drug advisory committees about the information patient input should include.

Two patient groups reported struggling to include quotes from patients who had experience with the drug under review, as they understood it to be an important part of their submission. One expressed concern that “if we somehow missed saying the right word, or the right . . . the right thing, then it wasn't taken as having any value” (PAT12). Another noted that

When asked to reflect on the strengths of the existing process for public and patient involvement, all patient groups observed that having a formal process for patient and public involvement was a strength (PAT11, PAT13, PAT12). One patient representative noted that symbolic inclusion was a good start but that empowerment might be more difficult, saying, “It's a success getting to the table . . . but I think moving beyond that tokenism to making sure it's really meaningful, and impactful, is another” (PAT13).

Patient interviewees identified challenges related to legitimacy goals, saying there was not enough communication and feedback during the submission process. This suggests problems with both transparency and accountability. One described it as “a monologue not a dialogue” (PAT11). Another commented that there “wasn't an opportunity to rebut or debate or discuss” (PAT12). Two patient interviewees felt there was a need to improve patient participation so it was not “just a PR exercise” or “tokenism,” although they were unsure exactly how this should be done (PAT11, PAT13).

All three patient groups reported being unsure how their input was being used, whether it influenced recommendations, and whether it was worth the time to submit (PAT11, PAT13, PAT12), suggesting challenges regarding both the legitimacy of the process and output. For example, one interviewee said,

Interviewees identified other challenges related to substantive goals. They noted that preparing patient submissions was time-consuming and costly, they did not feel they had the necessary support, and they were frustrated by the need to repeat information, for example, about what it is like to live with a condition. This was acknowledged by officials, and CADTH introduced a new template for patient submissions in June 2017 (CADTH 2017b), although it is too early to see how it affects patient groups. One interviewee from a smaller patient group noted:

Like for technical members, conflicts of interest and issues around objectivity were a key concern for patient groups, although they took different positions than committee members on these issues. One patient interviewee noted hearing of (but not experiencing) pressure from pharmaceutical manufacturers on smaller patient groups to make submissions and worried that there was an opportunity for industry to manipulate public and patient involvement (PAT11). Other interviewees (both of whom are volunteers at smaller patient groups) expressed frustration with the concerns about their potential conflicts of interest, noting that if they were not funded by industry, they would need money from government or CADTH, and that industry did not assist in preparing their submissions, other than making contacts to get patient quotes about the treatment (PAT13, PAT12).

During the study period, patient groups' ideas about the goals and legitimacy of HTA seem to have changed. In the earliest reports, patient groups rejected the principle of cost-effectiveness analysis and saw it as a punitive method for cost control. In the 2016 interviews, patient groups acknowledged the need for an assessment of clinical and cost-effectiveness, even though some also noted the importance of choice, particularly for conditions where individuals' response to different therapies may be unpredictable. Patient groups reported valuing a formal process for public and patient involvement but viewed it as a starting point rather than a complete solution to the transparency and accountability problems identified in the earlier reports. The types of challenges that patient groups identify have also changed over time as groups gain a more sophisticated understanding of HTA. Many of these are framed as challenges that could be ameliorated with more or different communication and feedback. Other challenges may be fundamental: most patient groups need industry funding to carry out their activities, including submissions to drug advisory committees, and this has a negative effect on their legitimacy in the eyes of some committee members.

Discussion and Conclusions

This research asks whether more than 10 years of experience with patient and public involvement in Canadian drug assessment has changed participants' ideas about the goals of HTA and public and patient involvement and affected their perceptions of whether these goals are being achieved. I found that the degree and content of ideational change varied by participant type. There was some consensus across groups of participants regarding the broad goals of HTA and the validity of some form of public and patient involvement. However, a key finding is that there are important areas of disagreement and uncertainty about how public and patient involvement should be used in the drug assessment process and how much impact it has on deliberations and recommendations. This uncertainly limits the potential for ideational change among participants.

Perhaps the most significant change in ideas came in patient groups' acceptance of the principles of HTA over time and the growing sophistication of their concerns about how public and patient involvement was carried out. In 2016, patient groups did not necessarily identify fewer challenges with drug assessment, but they identified different challenges than patient groups did in 2005 and 2007. The changes in the challenges identified may reflect a learning process as public and patient involvement has allowed patient groups a closer view of drug assessment, but they also demonstrate key concerns among patients about how their input is being used and how it is valued next to clinical and economic evidence.

Technical members of drug assessment committees experienced less change in ideas than patient groups and perhaps other experts who serve as agency or ministry officials. Technical members identified challenges related to objectivity throughout the study period and in 2016 added new concerns about the quality of patient submissions relative to the goal of improving decisions. There are two areas where some technical members appear to have adopted new ideas compared to the earliest reports: the acknowledgment that public and patient input could contribute to improved decisions, and the acknowledgment of an alternate view of legitimacy that considers inclusivity of participants in addition to objectivity of evidence. There was no consensus among technical members on the importance of these two issues, and additional research is required to better understand the extent and depth of ideational change among technical members.

The interviews show some ideas in common across participant groups. The key points of consensus were about the value of having a formal process for public and patient involvement and the potential for certain types of public and patient involvement to contribute to instrumental goals and make decisions better. There was less agreement about how exactly public and patient involvement should accomplish these goals, as participants across groups expressed uncertainty about how to best involve patients and the public and how to best use the input they provide. This contestation points to the importance of additional research on participants' views of legitimacy, particularly the degree to which patient input can or should adhere to scientific norms of objectivity and methodological rigor.

This study contributes to the literature on public and patient involvement in HTA, first, by complicating the categorization of goals for public and patient involvement. Abelson et al. (2007) categorize goals as process-oriented or instrumental, and their work elaborates on earlier research emphasizing acceptance and process criteria (Rowe and Frewer 2000). This research has found that groups of participants, and even individuals within a group, may define and interpret these broad goals differently. This suggests the need for a more granular approach to understanding participants' own goals that can account for the possibility that some participants will have incompatible versions of what initially seems to be a similar goal.

The second contribution is to demonstrate the value of a somewhat novel approach to evaluating public and patient involvement in HTA, based on participants' own criteria rather than a predetermined external framework. This approach can provide new insights into participants' values, and it may create opportunities for expanding comparative research on HTA processes. If processes can be evaluated based on how their participants view them, it may be possible to transcend more specific rubrics and make additional generalized hypotheses about the factors that contribute to “successful” public and patient involvement.

This study shows that for many HTA participants ideational change comes after policy change, if it occurs at all, and that the process of adapting to new ideas introduced by a policy change is uneven across and even within different groups of participants. This contrasts the finding in some ideational literature that ideational change will precede and cause policy change or be almost simultaneous with policy change (Jacobs 2009; Schmidt 2011). These findings prompt a number of questions for research into ideas about public and patient involvement in HTA that may also apply to citizen involvement in other aspects of health policy and in other jurisdictions. Future research should consider particular examples of ideational change in order to understand how and why they occur.

This type of causal research should be designed to capture further detail on the differences in ideational change across groups. Does the mechanism of ideational change differ among groups of participants? How do participants' training and background affect their ideas about public and patient involvement in HTA? This research found that patient groups are not fully integrated into the norms and processes of HTA. Additional research should focus on the types and amounts of involvement that produce ideational change, particularly related to legitimacy goals. It should also consider the more fundamental tension of achieving legitimacy when the outcomes of the process are viewed negatively, as may often be the case when a drug is not recommended for reimbursement.

The nature of the challenges for public and patient involvement identified by participants in 2016 suggests that there are still important barriers to achieving a drug assessment process that enjoys a high degree of legitimacy with all participants and affected groups and makes the best possible use of all sources of information when making its recommendations. The first 10 years of experience with public and patient involvement has led to some new ideas among participants. Patient groups increased their knowledge of the drug assessment process, which appears to have produced some legitimacy gains, but remain unsure how and whether they impact the process. Technical members acknowledge legitimacy goals and appear to have a better understanding of the potential for patient input to improve decisions. They also have mixed views on the degree to which this achieved in practical terms and often appear to maintain a view of evidence that conflicts with the types of information that can be gathered through existing modes of public and patient involvement.

Participants agree on the value of having a formal process for public and patient involvement, even as they disagree about key questions, such as the conflicts posed by industry funding for patient groups or, more fundamentally, the way successful public and patient involvement should be defined. As questions about how and whether to involve patients and members of the public become more pressing in other aspects of health policy, the Canadian experience may provide valuable lessons about the ability of such involvement to contribute to the legitimacy and effectiveness of health policy decision making.

Acknowledgments

This research was funded by a McMaster Arts Research Board Standard Research Grant and a Social Sciences and Humanities Research Council Insight Development Grant. Many thanks to the drug advisory committee members, officials, and members of patient groups who generously shared their time and insights. I am grateful for research assistance from Matthew Han, Danielle Blab, and Spencer Naylor. I received helpful comments from Ariel Ducey, Julia Abelson, Éric Montpetit, Max Cameron and the Canadian-Comparative Workshop at the University of British Columbia, Catherine Frost, Daniel Béland, and the journal's two anonymous reviewers.

Notes

References