Remedial Action Plans continue to be the principal program to operationalize an ecosystem approach to the restoration of degraded locations across the Laurentian Great Lakes called Areas of Concern. Initiated in 1985, the progress of Remedial Action Plans on balance has been slow and disappointing. The Remedial Action Plan program has been continued following revisions to the Great Lakes Water Quality Agreement in 2012 despite very little systematic inspection of its strengths and limitations. Further, the 2012 Agreement calls for a “nearshore framework” with no clarity on the process for understanding place-based governance methods as developed under these Remedial Action Plans. In this context, we conducted a three-round anonymous online Policy Delphi study involving several dozen experts in the development and implementation of Remedial Action Plans from across the Great Lakes basin within government, industry, academia and civil society. Round 1 collected their direct knowledge of the strengths and limitations of Remedial Action Plans. We distilled that knowledge and asked study participants in Round 2 to further reflect on what worked and what did not work in their experience as Remedial Action Plan practitioners. We found an expected diversity of opinion on what ails the program in Round 2, but an unexpected consensus on the desire to move forward with seven governance options that emerged and were ranked by participants in Round 3. Rankings also indicated a consensus that the options were somewhat feasible and likely to succeed as enhancements to the current governance of Remedial Action Plans. Importantly, the results relate to both the structure and attributes of these collaborative processes, and we therefore stress the need to focus on the predominant tendencies and characteristics that underline Remedial Action Plan processes. These findings will have broad significance for evolving place-based nearshore restoration strategies in the Great Lakes and elsewhere as such programs are initiated.

Introduction

In 1985, the International Joint Commission’s (IJC) Great Lakes Water Quality Board identified 42 degraded harbours, embayments, and connecting channels in the Great Lakes as Areas of Concern (AOCs; a 43rd was identified in 1991). AOCs were defined by the measurable chemical, physical or biological ‘impairment’ of 14 ‘beneficial uses’ (BUIs). The federal, provincial, and state governments committed to developing and implementing Remedial Action Plans (RAPs) as locally-designed approaches to stakeholder engagement and priority-setting to remediate BUIs and remove the AOC designation. This commitment to RAPs was codified in the 1987 Great Lakes Water Quality Agreement (the Agreement), which stipulated that RAPs were to “embody a systematic and comprehensive ecosystem approach to restoring and protecting beneficial uses” in AOCs, the process for which is described in detail by others in this volume (see also Botts and Muldoon, 2005; Muldoon, 2012; Krantzberg, 2012).

The original 1972 Agreement between Canada and the United States was based on a rational analytic understanding of contamination problems as well documented (to the extent that they were known) and technical solutions as relatively straightforward (Bulkley et al., 1989; Regier et al., 1999). Where this traditional technocratic approach to environmental problem solving assumes primacy for technical information and the role of the technological expert (Miller, 1985), it was recognized that a more comprehensive approach to restoring the Great Lakes would be required (NRC-RSC, 1985; Vallentyne and Beeton, 1988; Botts and Muldoon, 2005).

By introducing a collaborative conceptual framework for RAPs, the 1987 Agreement provided a mechanism by which the local character and variation in problems and possible solutions could be reflected in management and action. This was remarkable in that participation in environmental governance was extended to all stakeholders, including governments, industry and other private interests, civil society and indigenous groups, and academics and individual citizens.

The development of RAPs generated initial excitement and optimism from three principal sources: the explicit inclusion of multiple and non-technical stakeholders in decision making, the accountability involved in specifying responsibilities for action, and the oversight function of the IJC to review and evaluate progress towards the objectives of each RAP (Hartig and Vallentyne, 1989). And yet, “slow progress” on RAPs was a “major disappointment almost from the beginning” (Botts and Muldoon, 2005). Although Colborn et al. (1990) stated that RAPs would be a “major test of the ecosystem approach,” and that the 1990s would be a “crucial period” in that regard, the partial or complete withdrawal of federal or subnational funding for RAPs in both Canada and the United States during that decade contributed to a lack of implementation and a loss of public enthusiasm (IJC, 2003; Krantzberg, 2002, 2003; Botts and Muldoon, 2005; OPAC, 2006). These circumstances were exacerbated by a decline in the effectiveness of IJC oversight (Botts and Muldoon, 2005) and a general leadership malaise characterized in part by a governmental “reluctance” to accept responsibility for fulfilling obligations under the Agreement (GAO, 2003; also OAG, 2001; GAO, 2009). The subsequent U.S. Great Lakes Restoration Initiative created in 2010 has had a very positive impact on the removal of BUIs in U.S. AOCs (U.S. Great Lakes Interagency Task Force, 2017), demonstrating that funding commitments contribute to RAP program successes but, again, are only one of many relevant factors leading to progress or impediments.

There was significant effort during the first decade of RAPs to determine the key characteristics that would promote their success (e.g. Caldwell, 1988; Hartig and Thomas, 1988; Hartig and Vallentyne, 1989; Hartig and Zarull, 1992b; Landre and Knuth, 1993; MacKenzie, 1993, 1997; Gurtner-Zimmerman, 1995; Hartig et al., 1995; Krantzberg and Houghton, 1996; Kellogg, 1998; Krantzberg, 1998). But our extensive search of the literature for subsequent insights on RAPs demonstrates a precipitous decline in interest in RAP evaluation during the second and third decades of the RAP program. The most recent IJC (2003) status report on progress in RAP implementation and the restoration of beneficial uses concluded that key challenges to further progress include insufficient financial commitments, unclear accountability and responsibility, undefined restoration priorities and targets, and a lack of adequate monitoring—effectively, the entire program. Further, independent federal auditing agencies have more recently been critical of the inability of both federal governments to adequately coordinate and assess AOC-related activities and outcomes (OAG, 2008; GAO, 2009), and even the 2006–2007 governmental Agreement Review concluded that many of the 1987 Agreement’s provisions were outdated, the roles of various orders of government and the public were not clear, and accountability was weak (ARC, 2007). In this context, the IJC (2009) proposed the development of a framework to more broadly encompass the ecological problems that occur in the nearshore zone of the Great Lakes (where AOCs occur) and link those problems more closely to watershed management initiatives. This was manifest in the 2012 Agreement’s new Annex 2 on lakewide management that calls for an integrated nearshore framework to be implemented collaboratively for each Great Lake.

But what is current reality with respect to those initial sources of enthusiasm for RAPs? And what policy learning could inform a new framework for governance of the nearshore zone? Understanding such critical collaborative interactions is a key to policy learning for more successful management planning and more effective decision making (Bulkley et al. 1989; Manno and Krantzberg, 2008; Curtin and Parker, 2014; Whitting 2017), and could inform new approaches to RAPs and other policy developments specific to restoration of the nearshore zone of the Great Lakes and their connecting waterways (see Kalafatis et al., 2015). However, a formal audit and appraisal with respect to the RAP program’s development and implementation does not exist. To better benefit and learn from experiences with the RAP program, we conducted an inquiry that probed the knowledge base of a cross section of technical and non-technical experts within and beyond government, each with significant first-hand experience with RAP development and/or implementation. The goal of this study was to collect expert knowledge of factors that helped or hindered the success of the development and implementation of RAPs, to distil a set of lessons learned from that collective knowledge, and to translate those lessons into policy options that could inform the future design of placed-based, nearshore restoration commitments and governance models.

Methods

We used the Policy Delphi technique to systematically collect and characterize broad expert knowledge of the strengths and limitations of RAP development and implementation from study participants across the Great Lakes basin. We ultimately used that expert knowledge to formulate policy options that could improve RAPs (and other place-based governance models for rehabilitation of nearshore environments) and asked study participants to rank those options. Figure 1 illustrates the framework of our Policy Delphi study and includes its key elements, themes and outcomes. A more detailed account of our process of inquiry can be found in McLaughlin (2011) and McLaughlin and Krantzberg (in press).

The Delphi qualitative research technique was initially devised to make effective use of informed intuitive judgment to generate long-range consensual forecasts on technological issues (Dalkey and Helmer, 1963). In its simplest form, the technique is a systematic solicitation and collation of anonymous expert opinion through a series of designed questionnaires called “rounds” interspersed with structured feedback (Linstone and Turoff, 1975). Its iterative nature allows participants to reflect on that feedback prior to providing subsequent opinion, and therefore in a sense ‘collaborate’ with other participants.

We used the Policy Delphi technique, a variation of the standard Delphi, as a forum for ideas designed to generate the fullest and most substantive range of views and potential resolutions to a policy issue (Turoff, 1970, 1975; de Loë, 1995; Franklin and Hart, 2007). The Policy Delphi is a decision-facilitation tool used to address a broadening set of objectives, such as to explore or expose underlying assumptions or perceptions (e.g. Collins et al., 2009), to perform a post-hoc policy implementation analysis (e.g. Buck et al., 1993), or to develop a range of possible strategic options or priorities (e.g. de Loë, 1995). The objectives and outcomes of our study aligned with each of these uses of the Policy Delphi technique, each of which is intended to embed a breadth of expertise throughout the research process (Gabb et al., 2006). The Policy Delphi technique allowed us to organize very broad and well-informed insights into a systematic knowledge base to determine the strengths and limitations of RAPs empirically and to shape governance options for future consideration (see Lubell et al., 2005).

Participants

Participants were invited from Ontario and each U.S. Great Lakes state via email. Each invitee was an expert in the development and/or implementation of RAPs from government, industry, academia, and civil society. An expert with respect to a Delphi study is anyone deemed to have special knowledge based on experience with the subject matter and issues being examined. In our study, expert participants were individual stakeholders or representatives of stakeholder groups with direct and substantive experience in the development and/or implementation of RAPs. Participants were advised that our purpose was to both generate a critical analysis of factors that helped or hindered the success of the development and implementation of RAPs, and inform the future design of placed-based, nearshore restoration commitments and governance structures under a renegotiated Agreement.

We invited a group that was both broadly representative of multiple jurisdictions and sectors, and manageable within the construct of the Delphi survey tool. Participation was anonymous and the identity of participants was hidden from all involved, including ourselves, in order that participants could feel free to share their experiences and opinions without reservation. We used SurveyMonkey to facilitate data gathering. The numbers, percentages and affiliations of participants in this study are provided in Table 1.

Round 1, Strengths and Limitations of RAPs

Round 1 asked four open-ended questions to generate textual data describing characteristic strengths and limitations of RAPs as a governance process experienced by the expert participants (Figure 1). Participants were asked to describe how the development and/or implementation of RAPs, in their experience, helped or hindered:

  • Collaboration and shared responsibility for management and/or decisions.

  • Partnerships and trust.

  • Access to and sharing of information.

  • Opportunities for learning from each other.

These four themes framed Round 1 and were broadly representative of principal topics of consideration in published research on the RAP program (e.g. Hartig and Law, 1994), collaborative management (e.g. Sabatier et al., 2005; Plummer and Armitage, 2007a, 2007b), publications of the IJC (e.g. IJC, 2006a, 2006b), and recent public discussions regarding the future of the Agreement, particularly RAPs (e.g. ARC, 2007).

Round 2, Affective Factors

Guided by qualitative techniques for the analytical categorization of textual data, we applied a structural coding procedure to the combined contributions of Round 1 (6931 words in 218 separate responses; Saldaña, 2009; also Richards 2009). Structural coding applies a content-based conceptual phrase representing a topic of inquiry to a segment of text that relates to a specific research question used to initiate and frame the response (Saldaña, 2009). Using this process we generated 17 unique representative codes, and selected four code types that occurred widely for both development and implementation, that varied widely in the direction (positive or negative) and magnitude (mild or emphatic) of the responses, and that would be ‘actionable’ within the Great Lakes policy regime (Figure 1). Participants were asked to suggest improvements to what worked or remedies for what did not work with respect to the following four significant determinants of RAP program outcomes as defined by Round 1:

  • Command of the process, referring to the control of the process; to direct, guide, lead, manage and oversee the process; to provide a mandate and an atmosphere that encourages and validates participation; to take and share responsibility for decisions and direction.

  • Coordination of the process, referring to mechanisms to organize, facilitate or integrate, as in people, activities, information, research, expertise and ideas.

  • Design of the process, referring to characteristics of institutional approaches, arrangements, contexts, structures, models, or frameworks; including specifics such as rules, workplans, documentation and monitoring; and including qualities such as consistency and transparency.

  • Scope of the process, referring to the extent or range of points of view, issues under consideration, and operations, particularly with respect to ideas, objectives and the mandate of the process.

Round 3, Ranking Remedies

We applied the same structural coding procedure to the combined contributions of Round 2 (2849 words in 94 separate responses; Saldaña, 2009; also Richards 2009), generating seven categorizations of expert suggestions on how to improve what worked and/or to remedy what did not work with respect to RAP development and implementation (Figure 1). In effect, these seven remedies are suggested to strengthen or address shortcomings in those four factors determining RAP outcomes (i.e. the command, coordination, design, and scope of the process). These seven categories were phrased in the form of potential ‘actionable’ policy options:

  1. Ensure government coordination involves senior personnel trained and experienced in the mediation of group processes and able to navigate political arenas.

  2. Ensure continuity of government coordination (meaning that coordination roles do not go unfilled for long periods of time, and that those roles are assigned adequate and dedicated resources and time).

  3. Create stakeholder agreements and implementation workplans with assigned responsibilities, timetables, deliverables, and explicit criteria for engaging new stakeholders, ideas, and issues.

  4. More closely link institutional arrangements and workplans to legislative and regulatory instruments.

  5. Require delisting endpoints that reflect environmental quality to ensure that the scope of RAPs remain focused on required interventions.

  6. Directly link science and monitoring to policy needs regarding restoration of beneficial uses.

  7. Provide an overarching strategic RAP development and implementation framework that enables local flexibility.

Participants were asked to rank the desirability, feasibility, and likelihood for success of each option using a seven-point Likert scale. The scale permitted ratings of individual belief as somewhat, very, or highly desirable, feasible, or likely to succeed (and that we assigned 5, 6, or 7 respectively) or somewhat, very, or highly undesirable, unfeasible, or unlikely to succeed (and that we assigned 3, 2, or 1 respectively). A rank of 4 was permitted and was explained to mean that the participant thought the idea had merit but declined to suggest the direction of their belief.

A desirable idea was defined as one that participants believed would be advisable and worthwhile pursuing. A feasible idea was defined as one that participants believed would receive governmental, political and socioeconomic consideration and support in a policy venue such as RAPs. An idea was defined as likely to succeed where participants believed that there would be positive governance outcomes if the idea was implemented.

Results

Our three-round Policy Delphi study invited 69 individual RAP stakeholders or representatives of RAP stakeholder groups from across the Great Lakes basin (Group A) to contribute expert knowledge they gained from their direct and substantive experience in RAP development and/or implementation. Group A had participation rates of 41% (Round 1), 21% (Round 2), and 47% (Round 3). This is within what Needham and de Loë (1990) describe as the Policy Delphi’s critical participation threshold of a maximum of 50 participants in an expert sample and a minimum size of 10. We invited participation in Round 3 from an additional 38 people with expertise similar to that of Group A (Group B) simply to increase the number of rankings of governance options, although the low participation rate of Group B (13%) brought the overall rate of Round 3 participation down to 36%. Participation among Canadians and Americans was relatively even across rounds. Similarly, participation was diverse across sectors and no single sector dominated the responses (Table 1).

Figure 2 illustrates the arithmetic means and mean deviations of the combined ranks for each option separately and of all rankings combined on a seven-point Likert scale of desirability, feasibility, and likelihood for success. The principal results of our study are (1) that each of the seven policy options for reforming the governance of place-based restoration efforts in the Great Lakes is considered “highly” desirable (i.e. advisable and worthwhile pursuing) by the broad cross section of participants in our study with relatively little variation, and (2) that there is a general consensus that all of the options generally are both “somewhat” feasible (i.e. there would be governmental, political, and/or socioeconomic receptivity to their consideration and support) and “somewhat” likely to succeed, meaning that participants believed that there would be some positive governance outcomes if the idea were implemented.

Discussion

It was recognized from the outset of the RAP program that broad stakeholder inclusion and their diverse objectives represented a significant and challenging departure from the traditions of Great Lakes governance that had narrowly defined problems and internalized information and decision making (e.g. Hartig and Thomas, 1988; Hartig et al., 1991). Moreover, it was recognized as equally significant that an ecosystem approach to the RAP program would also require managerial talents and decision-making processes dramatically more sophisticated than traditional norms if RAPs were to achieve shared and interdependent technical and social goals (e.g. Hartig and Zarull, 1992a). We sought to characterize successful RAP governance from the direct experience and collective knowledge of our study participants, and assess the degree to which RAPs could better embody those characteristics.

The tension between technical and social norms underlying the development of RAPs is observed in debates about policy reform that can be preoccupied with changes to the structure of governance in trying to devise rules and procedures to expand participation and increase legitimacy (Hooghe and Marks, 2003). Many of the strengths and limitations of RAPs described in Round 1 reflected this debate. However, study participants in Round 2 focused as much on attributes of governance such as responsibilities, trust, partnerships and social learning, emphasizing qualities and characteristics that can accompany rules and procedures. Taken together as a framework for approaching governance reform, the seven policy options developed in Round 3 form the basis for addressing both the structures and attributes of RAPs and evolving nearshore zone governance models under the 2012 Agreement.

While careful attention to the expert oversight of stakeholder engagement (Option 1) is an important consideration in maximizing the return on such exercises (both for the agencies and the public), the quality and characteristics of ongoing facilitation of long-term processes (Option 2) is equally important to ensure positive, resilient program outcomes (Westley, 1995, 2002; Miller, 1999; Manno and Krantzberg, 2008; McLaughlin and Krantzberg, 2011, 2012).

Options 3 and 4 relate to linking shared and defined responsibilities more directly to RAP outcomes. Such issues of accountability have been contentious over the life of the Agreement (e.g. NRC-RSC, 1985; Munton, 1988; IJC, 2006c). Option 3 results from the ongoing inability of responsible agencies to properly account for the collective effort expended on rehabilitation as described by our participants. Option 4 originated from a variety of U.S. experiences with the legal system being a positive, catalytic element in RAP processes, although closer links between workplans and legal tools was considered among the least feasible.

Options 5 and 6 could provide learning opportunities for policy with more closely linked information feedbacks between specific RAP activities and delisting objectives (Option 5) and more generally between science and management (Option 6). A principal criticism of RAPs has been a tendency for the scope of issues and related activities to extend well beyond what was initially intended and agreed upon. Not surprisingly, our data reveal a desire to restrict the administrative or geographic boundaries of RAPs (Option 5), although this desire was not shared by those who argue that RAPs should explicitly include the broader watershed context for the receiving waters in which the BUIs occur, and possibly also new causes of impairments. These views represent two distinct perspectives for RAP-related nearshore restoration and watershed management: one based on narrowly delineated local problems with well-defined endpoints representing measurable goals, a second with explicit connection to the ways in which receiving waters are impacted by social factors that extend upstream through related watersheds. The directions in which RAPs and future nearshore programs evolve are not necessarily opposed or mutually exclusive—in fact, they could be nested one within the other in ways that are complementary. Our point is that choices going forward must make maximum use of available lessons from experience in the nearshore to date and should be used to determine the features and extent of those choices.

The social aspects of RAPs share a complexity with the ecological relationships involved, and both types of complexity have a high degree of connectivity and interdependence (Lee et al., 1982). Despite the initial enthusiasm for the inclusive decision making of the RAP model – an innovative aspect of their early development – a lack of definition for RAPs has also introduced uncertainties that have compromised success. One source of uncertainty is caused by the difficulty in determining cause and effect. For example, what do we know and how much can we know about what is going on in the ecosystem. To what extent are water quality improvements possible? How much of that uncertainty is an inevitable characteristic of multiple and overlapping environmental problems? How much is also a result of the inevitable overlapping human perspectives present in a participatory decision forum? Do scientists and lay stakeholders share the same understanding on multiple issues, for example? Other uncertainties involved in RAP participation have included cost uncertainty (e.g. who is going to pay, and will it be more than projected?), uncertainty that trust and other elements of social cohesion (such as shared responsibilities and accountability) will last throughout the process, and the uncertainty of trying to reconcile the many different ways to arrive at valuations of nature (e.g. prioritizing one beneficial use over another, and how to determine the endpoint where restoration is sufficient). And, of course, the uncertainty that all the effort for any given stakeholder will be worth it in the end. Learning to better cope with ecological and sociological uncertainties through RAP processes will demand greater integration of science and policy in future RAP governance models and correspondence of fixed endpoints (e.g. a RAP’s delisting objectives) with the variously unpredictable nature of the systems that they measure or represent (Option 6; McLaughlin, 2012).

Finally, Option 7 represents a reaction to the inability of the federal governments to adequately coordinate and assess AOC-related activities and outcomes. The 1987 Agreement introduced RAPs with a minimum of imposed structure in order that they be locally-designed with the flexibility to adjust to local conditions. But a broader vision for RAPs with a rigorous programmatic foundation has been absent, a condition which remains in the 2012 Agreement. According to Botts and Muldoon (2005), the Great Lakes regime generally lacked a plan for institutional development following adoption of the 1987 Agreement. Arguably, an absence of an overarching RAP strategy is still reflected in our results.

The three rounds of our Policy Delphi form a pathway from reflective and evaluative characterizations of RAPs to specific actions for policy change. Taken together they are a framework for both planning and assessing stakeholder-driven water management collaborations. Our primary goal was to learn from experience in order to improve RAP program outcomes and inform new nearshore initiatives under the 2012 Agreement. From a practical perspective, these seven policy options mirror other typologies of water management being developed to improve policy processes and implementation across a diversity of social-ecological systems with respect to the influence and integration of actors and knowledge systems (e.g. Carlsson and Berkes, 2005; Armitage et al., 2007; Plummer and Armitage, 2007a; Taljaard et al., 2013; Moore et al., 2014). Our analyses explicitly relates to preconditions and best practices required for effective stakeholder engagement (McLaughlin and Krantzberg, 2011; Susskind, 2013) and the importance of social learning and system resilience (McLaughlin and Krantzberg, 2012; Gerlak, 2014).

Conclusions

From the outset of the Great Lakes RAP program it was recognized that for RAPs to be inclusive and consensual, an approach to water governance and environmental problem-solving very different from traditional management would be required. From the beginning, the IJC (1984) warned that “there are limits to what technical and scientific programs can accomplish when fundamental elements are not only technological but also societal and attitudinal. As technological and scientific limitations on progress become more apparent, the challenge becomes increasingly one of engaging public support for the new approaches and programs that are needed.”

Conceptually, RAPs challenged governments and other stakeholders to transform management. The RAP model, it was thought, would amount to “ecological democracy” (Hartig and Zarull, 1992a). In some ways RAPs did both, and we now have more than three decades of experience on which to move forward. Our study fills a need for an introspective investigation of the strengths and limitations of the RAP program and the choices that decision-makers can make to improve those processes (see Burton et al., 2002). Many people have deeply held opinions on what has helped or hindered RAPs based on their direct experience with the program, but very little formal empirical data on what has worked and not worked has ever been generated. Moreover, the RAP program has never, to the best of our knowledge, received a comprehensive post-audit appraisal of the program’s performance.

Our study provides a framework upon which to improve RAPs and inform the development of an inclusive, responsive, and effective future nearshore governance framework. The nearshore framework being finalized under the Agreement’s Annex 2 calls for both monitoring and governance elements. Since RAPs operate in the nearshore, consider the land-water interface, and are stakeholder driven (or could be), we believe these lessons from thirty years of RAP development and implementation are highly relevant to transposition to nearshore geographically-focused initiatives.

We found a diversity of opinion on several characteristics of RAPs that would be expected from such a pluralistic group of study participants. But we also found an unanticipated level of consensual enthusiasm for the options for reforming RAPs and developing other models of Great Lakes governance. We conclude that this consensus ultimately stems from those same early sources of enthusiasm mentioned in our introduction, that RAPs would hold the promise of:

  • effective inclusion of stakeholders in decision making,

  • shared responsibilities and appropriate accountability, and

  • actions sensitive to feedback and respectful of both ecological and social processes.

The essence of our body of evidence, the motives behind the ranked policy options, and the optimism for collaborative and place-based Great Lakes management can therefore be summed in three principles: inclusion, learning, and accountability. Learning can be improved when participation is inclusive and accoutable, because “learning is a political process [and is] as much about interaction and leadership as information” (Whitting, 2017; also McLaughlin, 2012). We conclude that these principles, and the pathway provided by the research we have conducted, are a basis for better public engagement, better integration of science and policy, and better shared outcomes across the Great Lakes.

The enthusiasm that we found suggests that the Great Lakes RAP community would welcome and possibly embrace renewed attention and renewed approaches to the governance of RAPs and other related nearshore initiatives. That there remains enthusiasm for RAPs after so many years and so much frustration should embolden those with the ability to influence such change. One way of improving the likelihood that at least some of our findings are of benefit to practitioners now and in future, will be to share these results with the Great Lakes Executive Committee, including its Annex 1 and Annex 2 subcommittees. In addition, communication with the IJC Water Quality Board and Commissioners will further broaden this important discussion. We will also share these findings with RAP practitioners at Great Lakes gatherings in the future, and we invite readers to do the same.

Acknowledgements

Thank you to Karen Szala-Meneok and Johnny Saldaña for improving the study design, and to Robert Stewart and two anonymous referees for their thorough review of earlier versions of the manuscript. And many, many thanks to the dozens of RAP veterans across the Great Lakes who shared their invaluable experience and knowledge – you have our tremendous gratitude.

References

ARC (Agreement Review Committee)
,
2007
. Report to the Great Lakes Binational Executive Committee: Volumes 1, 2, and 3.
Armitage, D. , Berkes, F. , Doubleday, N. ,
2007
.
Adaptive Co-Management: Collaboration, Learning, and Multi-Level Governance
.
UBC Press
,
Vancouver
.
Botts, L. , Muldoon, P. ,
2005
.
Evolution of the Great Lakes Water Quality Agreement
.
Michigan State University Press
,
Michigan
.
Buck, A.J. , Gross, M. , Hakim, S. , Weinblatt, J. ,
1993
.
Using the Delphi process to analyze social policy implementation: A post hoc case from vocational rehabilitation
.
Policy Sciences
26
,
271
288
.
Bulkley, J.W. , Donahue, M.J. , Regier, H.A. ,
1989
. The Great Lakes Water Quality Agreements: how to assess progress toward a goal of ecosystem integrity. In: C.G. Gunnerson (Ed.),
Post Audits of Environmental Programs and Projects
, pp.
27
42
.
Proceedings of a session sponsored by the Environmental Impact Analysis Research Council of the American Society of Civil Engineers in conjunction with the ASCE Convention in New Orleans
,
Louisiana
, 11 Oct 1989.
Burton, I. , Huq, S. , Lim, B. , Pilifosova, O. , Schipper, E.L. ,
2002
.
From impacts assessment to adaptation priorities: the shaping of adaptation policies
.
Climate Policy
2
,
145
159
.
Caldwell, L.K. ,
1988
. Implementing an ecological systems approach to basin-wide management. In: L K. Caldwell (Ed.),
Perspectives on Ecosystem Management for the Great Lakes: A Reader
, pp.
1
29
.
State University of New York Press
,
NY
.
Carlsson, L. , Berkes, F. ,
2005
.
Co-management: concepts and methodological implications
.
Journal of Environmental Management
75
,
65
76
.
Colborn, T.E. , Davidson, A. , Green, S.N. , Hodge, R.A. , Jackson, C.I. , Liroff, R.A. ,
1990
.
Great Lakes
,
Great Legacy? The Conservation Foundation and the Institute for Research on Public Policy
,
Halifax, N.S
.
Collins, J. , Hanlon, A. , More, S.J. , Wall, P.G. , Duggan, V. ,
2009
.
Policy Delphi with vignette methodology as a tool to evaluate the perception of equine welfare
.
The Veterinary Journal
181
,
63
69
.
Curtin, C.G. , Parker, J.P. ,
2014
.
Foundations of resilience thinking
.
Conservation Biology
28
,
912
923
.
Dalkey, N. , Helmer, O. ,
1963
.
An experimental application of the Delphi method to the use of experts
.
Management Science
9
,
458
467
.
de Loë, R. ,
1995
.
Exploring complex policy questions using the policy Delphi
.
Applied Geography
15
,
53
68
.
Franklin, K.K. , Hart, J.K. ,
2007
.
Idea generation and exploration: benefits and limitations of the policy Delphi method
.
Innovation in Higher Education
31
,
237
246
.
Gabb, J. , Balen, R. , Gibbs, G. , Hall, C. , Teal, A. ,
2006
.
Victoria Climbié inquiry data corpus project: using the Delphi method in multidisciplinary child protection research
.
British Journal of Social Work
36
,
577
596
.
GAO (United States General Accounting Office
),
2003
. Great Lakes: an overall strategy and indicators for measuring progress are needed to better achieve restoration goals. Report GAO-03-515.
GAO (United States General Accounting Office
),
2009
. EPA needs a cohesive plan to clean up the Great Lakes Areas of Concern. Report 09-P-0231.
Gerlak, A.K. ,
2014
.
Policy interactions in human–landscape systems
.
Environmental Management
53
,
67
75
.
Gurtner-Zimmerman, A .,
1995
.
A mid-term review of Remedial Action Plans: difficulties with translating comprehensive planning into comprehensive action
.
Journal of Great Lakes Research
21
,
234
247
.
Hartig, J.H. , Law, N .,
1994
.
Institutional frameworks to direct development and implementation of Great Lakes Remedial Action Plans
.
Environmental Management
18
,
855
864
.
Hartig, J.H. , Thomas, R.L .,
1988
.
Development of plans to restore degraded areas in the Great Lakes
.
Environmental Management
12
,
327
47
.
Hartig, J.H. , Vallentyne, J.R .,
1989
.
Use of an ecosystem approach to restore degraded areas of the Great Lakes
.
Ambio
18
,
423
428
.
Hartig, J.H. , Zarull, M.A .,
1992a
. A Great Lakes mission. In: J.H. Hartig , M.A. Zarull (Eds.),
Under RAPs: Toward Grassroots Ecological Democracy in the Great Lakes Basin
, pp.
5
35
.
University of Michigan Press
.
Hartig, J.H. , Zarull, M.A .,
1992b
. Keystones for success. In: J.H. Hartig , M.A. Zarull (Eds.),
Under RAPs: Toward Grassroots Ecological Democracy in the Great Lakes Basin
, pp.
263
279
.
University of Michigan Press
.
Hartig, J.H. , Lovett-Doust, L. , Zarull, M.A. , Leppard, S. , New, L.A. , Skavroneck, S. , Eder, T. , Coape-Arnold, T. , Daniel, G .,
1991
.
Overcoming obstacles in Great Lakes remedial actions plans
.
International Environmental Affairs
3
,
91
107
.
Hartig, J.H. , Law, N.L. , Epstein, D. , Fuller, K. , Letterhos, J. , Krantzberg, G .,
1995
.
Capacity-building for restoring degraded areas in the Great Lakes
.
International Journal of Sustainable Development & World Ecology
2
,
1
10
.
Hartig, J.H., Zarull, M.A., Reynoldson, T.B., Mikol, G., Harris, V.A., Randall, R.G., Cairns, V.W.,
1997
.
Quantifying targets for rehabilitating degraded areas of the Great Lakes
.
Environmental Management
21
,
713
723
.
Hartig, J.H. , Zarull, M.A. , Law, N.L. ,
1998
.
An ecosystem approach to Great Lakes management: practical steps
.
Journal of Great Lakes Research
24
,
739
750
.
Hooghe, L. , Marks, G .,
2003
.
Unraveling the central state, but how? Types of multi-level governance
.
American Political Science Review
97
,
233
243
.
IJC (International Joint Commission)
, Second Biennial Report.
1984
.
IJC (International Joint Commission)
,
2003
. Status of restoration activities in Great Lakes areas of concern: a special report.
IJC (International Joint Commission
),
2006a
. Advice to Governments on their Review of the Great Lakes Water Quality Agreement: A Special Report to the Governments of Canada and the United States.
IJC (International Joint Commission),
2006b
. Priorities 2003-2005: Priorities and Progress under the Great Lakes Water Quality Agreement.
IJC (International Joint Commission)
,
2006c
. Thirteenth Biennial Report.
IJC (International Joint Commission)
,
2009
. Work group report on nearshore framework.
Kalafatis, S.E. , Campbell, M. , Fathers, F. , Laurent, K.L. , Friedman, K.B. , Krantzberg, G. , Scavia, D. , Creed, I.F .,
2015
.
Out of control: how we failed to adapt and suffered the consequences
.
Journal of Great Lakes Research
41
(
S1
),
20
29
.
Kellogg, W.A .,
1998
.
Adopting an ecosystem approach: local variability in Remedial Action Planning
.
Society & Natural Resources
11
,
465
483
.
Krantzberg, G .,
1998
.
After more than ten years of effort, are Remedial Action Plans making a difference?
Journal of Great Lakes Research
24
,
485
486
.
Krantzberg, G .,
2002
.
Restoring active participation
.
Journal of Great Lakes Research
28
,
491
492
.
Krantzberg, G .,
2003
.
Keeping remedial action plans on target: lessons learned from Collingwood Harbour
.
Journal of Great Lakes Research
29
,
641
651
.
Krantzberg, G .,
2012
. The Remedial Action Plan program, historical and contemporary overview. In: Grover, I. , Krantzberg, G. , (Eds.),
Great Lakes: Lessons in Participatory Governance
, pp.
245
256
.
CRC Press
.
Krantzberg, G. , Houghton, E .,
1996
.
The Remedial Action Plan that led to the cleanup and delisting of Collingwood Harbour as an Area of Concern
.
Journal of Great Lakes Research
22
,
469
483
.
Landre, B.K. , Knuth, B.A .,
1993
.
The role of agency goals and local context in Great Lakes water resources public involvement programs
.
Environmental Management
17
,
153
165
.
Lee, B.J. , Regier, H.A. , Rapport, D.J .,
1982
.
Ten ecosystem approaches to the planning and management of the Great Lakes
.
Journal of Great Lakes Research
8
,
505
519
.
Linstone, H.A. , Turoff, M. , (Eds.).,
1975
.
The Delphi Method: Techniques and Applications
.
Addison-Wesley Publishing
,
Boston, MA
.
Lubell, M. , Sabatier, P.A. , Vedlitz, A. , Focht, W. , Trachtenberg, Z. , Matlock, M .,
2005
. Conclusions and recommendations. In: P.A. Sabatier , W. Focht , M. Lubell , Z. Trachtenberg , A. Vedlitz , M. Matlock (Eds.),
Swimming Upstream: Collaborative Approaches to Watershed Management
, pp.
261
296
.
MIT Press
,
Cambridge, MA
.
MacKenzie, S.H .,
1993
.
Ecosystem management in the Great Lakes: some observations from three RAP sites
.
Journal of Great Lakes Research
19
,
136
144
.
MacKenzie, S.H .,
1997
.
Toward integrated resource management: lessons about the ecosystem approach from the Laurentian Great Lakes
.
Environmental Management
21
,
173
183
.
Manno, J. , Krantzberg, G .,
2008
. Rediscovering and revitalizing the Great Lakes governance. In: K. Bosselmann , R. Engel , P. Taylor (Eds.),
Governance for Sustainability Issues, Challenges, Successes
, pp.
159
170
. IUCN Environmental Policy and Law Paper No. 70.
McLaughlin, C .,
2011
.
On the Evolution of the Great Lakes Water Quality Agreement: From Pathological Management to Adaptive Governance? Sketches of Learning From Experience
. PhD Dissertation, McMaster University. Hamilton, Ontario, Canada.
McLaughlin, C .,
2012
. Innovations in decision making. In: I. Grover , G. Krantzberg (Eds.),
Great Lakes: Lessons in Participatory Governance
, pp.
67
104
.
CRC Press
,
FL
.
McLaughlin, C. , Krantzberg, G .,
2011
.
An appraisal of policy implementation deficits in the Great Lakes
.
Journal of Great Lakes Research
37
,
390
396
.
McLaughlin, C. , Krantzberg, G .,
2012
.
An appraisal of management pathologies in the Great Lakes
.
Science of the Total Environment
416
,
40
47
.
Miller, A .,
1985
.
Ideology and environmental risk management
.
The Environmentalist
5
,
21
30
.
Miller, A .,
1999
.
Environmental Problem Solving: Psychosocial Barriers to Adaptive Change
.
Springer, NY
.
Moore, M.-L. , von der Porten, S. , Plummer, R. , Brandes, O. , Baird, J .,
2014
.
Water policy reform and innovation: a systematic review
.
Environmental Science & Policy
38
,
263
271
.
Muldoon, P .,
2012
. Governance in the Great Lakes: a regime in transition. In: I. Grover , G. Krantzberg (Eds.),
Great Lakes: Lessons in Participatory Governance
, pp.
44
66
.
CRC Press
.
Munton, D .,
1988
. Toward a more accountable process: the Royal Society-National Research Council Report. In: I. Grover, G. Krantzberg (Eds.),
Perspectives on Ecosystem Management for the Great Lakes: A Reader
, pp.
299
317
.
State University of New York Press
.
Needham, R.D. , de Loë, R.C .,
1990
.
The Policy Delphi: purpose, structure, and application
.
The Canadian Geographer
34
,
133
142
.
NRC-RSC (National Research Council of the United States and The Royal Society of Canada).
1985
.
The Great Lakes Water Quality Agreement: An Evolving Instrument For Ecosystem Management
.
National Academy Press, Washington
.
OAG (Office of the Auditor General of Canada)
.,
2001
. A legacy worth protecting: charting a sustainable course in the Great Lakes and St. Lawrence River basin; Chapter 1 of the Report of the Commissioner of the Environment and Sustainable Development. Public Works and Government Services Canada, Ottawa, ON.
OAG (Office of the Auditor General of Canada)
.,
2008
. Areas of Concern in the Great Lakes Basin; Chapter 7 of the Status Report of the Commissioner of the Environment and Sustainable Development. Public Works and Government Services Canada, Ottawa, ON.
OPAC (Ontario Public Advisory Council).
2006
.
Great Lakes Hotspots: Ontario Citizens Speak Up: Highlights
.
Ontario Public Advisory Council
.
Plummer, R. , Armitage, D.R .,
2007a
.
A resilience-based framework for evaluating adaptive co-management: linking ecology, economics and society in a complex world
.
Ecological Economics
61
,
62
74
.
Plummer, R. , Armitage, D.R .,
2007b
.
Charting the new territory of adaptive co-management: a Delphi study
.
Ecology and Society
12
(
2
), art10.
Regier, H.A. , Jones, M.L. , Addis, J. , Donahue, M .,
1999
. Great Lakes–St. Lawrence River basin assessments. In: K.N. Johnson , F.J. Swanson , M. Herring , S. Greene , (Eds.),
Bioregional Assessments: Science at the Crossroads of Management and Policy
, pp.
133
165
.
Island Press
,
Washington, D.C
.
Richards, L .,
2009
.
Handling Qualitative Data: A Practical Guide
. Second edition.
Sage Publications
,
California
.
Sabatier, P.A. , Focht, W. , Lubell, M. , Trachtenberg, Z. , Vedlitz, A. , Matlock, M. , (Eds).
2005
.
Swimming Upstream: Collaborative Approaches to Watershed Management
.
MIT Press, MA
.
Saldaña, J .,
2009
.
The Coding Manual for Qualitative Researchers
.
Sage Publications, CA
.
Susskind, L .,
2013
.
Water and democracy: new roles for civil society in water governance
.
International Journal of Water Resources Development
29
,
666
677
.
Taljaard, S. , Slinger, J.H. , van der Merwe, J .,
2013
.
Dual adaptive cycles in implementing integrated coastal management
.
Ocean & Coastal Management
84
,
23
30
.
Turoff, M .,
1970
.
The design of a policy Delphi
.
Technological Forecasting and Social Change
2
,
149
171
.
Turoff, M .,
1975
. The policy Delphi. In: H.A. Linstone , M. Turoff (Eds.),
The Delphi Method: Techniques and Applications
, pp.
84
100
.
Addison-Wesley
,
Boston, MA
.
U.S. Great Lakes Interagency Task Force
,
2017
. Great Lakes Restoration Initiative Report to Congress and the President.
Vallentyne, J.R. , Beeton, A.M .,
1988
.
The ecosystem approach to managing human uses and abuses of natural resources in the Great Lakes basin
.
Environmental Conservation
15
,
58
62
.
Westley, F .,
1995
. Governing design: the management of social systems and ecosystems management. In: L.H. Gunderson, C.S. Holling, S.S. Light (Eds.),
Barriers and Bridges to the Renewal of Regional Ecosystems
, pp.
391
427
.
Columbia University Press, NY
.
Westley, F .,
2002
. The devil is in the dynamics: adaptive management on the front lines. In: L.H. Gunderson, C.S. Holling (Eds.),
Panarchy: Understanding Transformations in Human and Natural Systems
, pp.
333
360
. Island Press, Washington, DC.
Witting, A .,
2017
.
Insights from ‘policy learning’ on how to enhance the use of evidence by policymakers
.
Palgrave Communications
3
, art49.