Evaluation of the Regional Electricity Cooperation and Strategic Infrastructure (RECSI) Initiative

Audit and Evaluation Branch
Natural Resources Canada

July 2019

Table of Contents

• List of Acronyms
• Executive Summary
• Introduction
• Program Information
• Evaluation Objectives and Methods
• Evaluation Limitations and Mitigation Strategies
• Findings
  • Relevance
  • Performance – Effectiveness
  • Performance – Efficiency
• Conclusions
• Appendix 1: Evaluation Team
• Appendix 2: Reference List

List of Acronyms

AEB

Audit and Evaluation Branch

AESO

Alberta Electricity System Operator

CIB

Canada Infrastructure Bank

DRF

Departmental Results Framework

G&C

Grants and Contributions

GHG

Greenhouse Gas

IBAs

Integrated Bilateral Agreements

NRCan

Natural Resources Canada

O&M

Operation and maintenance

PAA

Program Alignment Architecture

PCF

Pan-Canadian Framework on Clean Growth and Climate Change

RECSI

Regional Electricity Cooperation and Strategic Infrastructure

Executive Summary

About the Evaluation

This report presents the findings, conclusions, and lessons learned from the evaluation of Natural Resources Canada’s (NRCan’s) Regional Electricity Cooperation and Infrastructure (RECSI) Initiative, which was delivered from 2016-17 through 2017-18. The objective of this two-year Initaitive was to identify least cost options for transitioning fossil fuel-reliant regions in Canada to renewable energy through regional electricity cooperation and strategic infrastructure. Actual expenditures for the Initiative over this period were approximately $2 million.

The objective of the evaluation was to assess the relevance, effectiveness and efficiency of the the RECSI Initiative, with a focus on the extent to which it was designed and delivered in a way that would facilitate the achievement of, or contribution to, the intended outcomes.

What the Evaluation Found

Overall, the evaluation found the RECSI Initiative to have been relevant. There is a need and role for NRCan to facilitate cooperation between provinces/territories, particularly in natural resource regional planning exercises. Although provinces/territories have jurisdiction over the development of their natural resources, the federal government plays a legitimate and appropriate role when it intervenes to facilitate collaboration between provinces/ territories on a regional basis. Through initiatives like RECSI, for instance, provinces/territories are encouraged to share information and consider opportunities at a regional level, beyond their own boundaries, while also contributing to federal priorities and commitments.

The RECSI Initiative successfully produced its planned outputs, namely, Atlantic and Western region dialogues, and knowledge products from these dialogues (e.g., lists of potential projects, technical studies, and summaries for policy makers). The evaluation confirmed the Initiative directly contributed to the three intended immediate outcomes: collaboration among provinces/territories in the area of regional electricity planning; agreement among provinces/territories on scenarios/actions for modelling/testing; and increased technical and policy knowledge among provinces/ territories, utilities, and the federal government.

External interviewees identified the quality and capacity of NRCan employees delivering the Initiative as a key internal success factor. The collaborative design of the Initiative and financial support from NRCan for the studies also enabled the Initiative to achieve the intended immediate outcomes.

There is no evidence to suggest the RECSI Initiative was not delivered efficiently. However, because there is no alternative for comparison no definitive conclusion can be made on whether it is the most efficient way of producing the intended outputs and achieving the intended immediate outcomes. The evaluation confirmed the Initiative was largely implemented as planned. It also deteremined that it is reasonable to expect an intiative designed to facilitate collaboration and conduct studies to contribute to longer-term outcomes (i.e., investments on clean energy or transmission/integration projects, identifying potential options or solutions to reduce reliance on fossil fuel-fired electricity).

Actual expenditures for the Initiative were approximately 78% of planned expenditures, suggesting a good use of resources. Moreover, no wholly different designs were identified; rather interviewees suggested some adjustments that could be made. For example, engaging provincial/territorial and industry stakeholders at the planning phase, broadening the scope to include secondary benefits (e.g., electrifying industry), using academics/consortiums to conduct the studies, and so on. Stakeholders also generally agreed that the collaborative approach and use of external experts to conduct the studies were key strengths in the design of the Initiative.

Lessons Learned

The evaluation confirmed that NRCan should continue to deliver programs/initiatives focused on working with, and facilitating cooperation between, provinces/territories, and industry (if applicable), particularly in regional natural resource planning exercises. Within this context, three lessons learned were identified that could be considered in the design of future interventions focused on collaboration and dialogue with stakeholders:

1. These programs/initiatives should clearly address or explain possible approaches or suggestions (if any) for support or implementation of projects/actions by provinces/territories following the end of the initiative.
2. Implementation, including of key activities and outputs (e.g., as identified in a logic model), should be systematically monitored to identify if the initiative is on track to meet its intended outcomes or if course correction is needed.
3. Future initiatives designed to faciliate collaboration and/or the conducting of studies, particularly related to regional natural resource planning, could consider some adjustments to the design. For example, including stakeholders early in the process to identify needs and inform the direction and scope of the initiative, clarifying roles and responsibilities, and ensuring objectives and intended outcomes are clear at the outset.

Introduction

This report presents the findings, conclusions, and lessons learned from the evaluation of Natural Resources Canada’s (NRCan’s) Regional Electricity Cooperation and Strategic Infrastructure (RECSI) Initiative. NRCan’s Audit and Evaluation Branch (AEB) undertook the evaluation between May 2018 and January 2019. It was conducted in accordance with the Treasury Board Policy on Results (2016) and to conform with a commitment in a February 2016 Treasury Board streamlined attestation.

Program Information

The RECSI Initiative was delivered from 2016-17 through 2017-18, with actual expenditures of approximately $2 million. It was part of NRCan’s green infrastructure programming delivered as part of Phase I of the Government of Canada’s Investing in Canada Plan, which has a focus on addressing climate change and supporting the transition to a low-carbon economy built on growth.

The objective of the RECSI Initiative was to identify least cost options for transitioning fossil fuel-reliant regions in Canada to renewable energy through regional electricity cooperation and strategic infrastructure. To deliver on this, NRCan undertook two key activities as part of the Initiative:

1. Convening and facilitating eastern and western regional dialogues involving provinces and their utilities: to identify potential projects that could reduce greenhouse gas (GHG) emissions, reduce reliance on fossil fuels, and increase critical investments in hydroelectricity and inter-provincial transmission infrastructure.
2. Commissioning studies: to assess proposed projects, and rank them based on the production cost of electricity to GHG reduction potential.

As part of this Initiative, NRCan:

• Provided leadership in convening and facilitating the regional dialogues;
• Managed delivery of the Initiative;
• Promoted and supported provinces/territories in their collaborative activities throughout the implementation of the Initiative;
• Participated on/chaired senior management level Steering Committees and working level Technical Advisory Committees for each dialogue; and
• Administered contribution agreements to regional organizations to deliver/lead the dialogues and studies.

Figure 1 presents the logic model developed for the purpose of this evaluation. The development of this logic model was informed by RECSI Initiative documents and consultation with Initiative stakeholders.

Figure 1: RECSI Initiative Logic Model

Text version

Activities

• Convene and facilitate dialogues with stakeholders from provinces/ territories and utilities
• Commission studies with stakeholders from provinces/ territories and their utilities

Outputs

• 2 dialogues^{Figure footnote *}
• Knowledge products^{Figure footnote **}

Immediate Outcomes

• Collaboration among provinces/ territories in the area of regional electricity planning
• Agreement among provinces/ territories on investigating scenarios/ actions that increase the use of renewable energy and increase grid integration (where applicable)
• Increased knowledge (technical and policy) among provinces/territories, utilities, and the federal government about actions that could increase the use of renewable electrical energy and increase grid integration (where applicable)

Intermediate Outcome

• Investments by provinces/territories, utilities, and/or the federal government in the construction of large clean energy projects and transmission/ integration projects to deliver that clean energy

Ultimate Outcome

• Reduced reliance on fossil fuel-fired electricity across Canada

Evaluation Objectives and Methods

The objective of the evaluation was to assess the relevance, effectiveness, and efficiency of the RECSI Initiative, with a focus on the extent to which it was designed and delivered in a way that would facilitate the achievement of, or contribution to, the intended outcomes (as identified in Figure 1). Because the evaluation focused on implementation rather than assessing outcome achievement or impact, the assessment of effectiveness emphasized the following:

• Determining the extent to which the activities and outputs facilitated (or will facilitate) the achievement of the intended immediate outcomes;
• Identifying factors impacting (positively or negatively) the achievement of all intended outcomes;
• Identifying whether any modifications were made during implementation, why, and what impact (if any) these had on the achievement of the intended outcomes; and
• Determining the extent to which the design of the Initiative could reasonably be expected to contribute to the higher-level intended outcomes.

The evaluation used two key lines of evidence, consisting primarily of qualitative data:

Evaluation Limitations and Mitigation Strategies

The evaluation used mixed methods and multiple lines of evidence in order to mitigate against any limitations associated with individual methods. These enabled the triangulation of evidence across sources of information to identify valid findings and conclusions relative to the evaluation questions. Nevertheless, the following limitation should be considered when reviewing the findings from this evaluation:

• Provinces/territories and their utilities that did not participate in the RECSI Initiative (i.e., Ontario and Quebec) were not interviewed as part of this evaluation. As a result, their perspectives on the relevance and effectiveness of NRCan involvement in an initiative like this, and regional electricity planning more generally, are unknown. Interviews with participating stakeholders explored whether appropriate stakeholders were engaged in the Initiative and if any key groups were missing.

Findings

Relevance

To what extent is there an ongoing need for the federal government to engage in regional electricity planning exercises like the RECSI Initiative?

In November 2016, as part of the Pan-Canadian Framework on Clean Growth and Climate Change (PCF), the Government of Canada announced regulations to take effect in 2030 to phase out coal-fired electricity. The PCF also identified specific actions intended to “transform regional electricity systems… [including,] increasing the amount of electricity generated from renewable and low-emitting resources [and] connecting clean power with places that need it” (Government of Canada, 2016). To prepare for these regulations and respond to the actions identified in the PCF, provinces/ territories, especially those reliant on fossil fuels, need to identify and implement alternative approaches to electricity.

Although some provinces/territories (e.g., British Columbia, Manitoba, Yukon, Ontario, Quebec, Prince Edward Island, Newfoundland and Labrador) already produce the majority of their electricity from non-emitting or renewable resources, others (e.g., Alberta, Saskatchewan, New Brunswick, Nova Scotia) need to increase this. In some cases, provinces/territories that already produce electricity using non-emitting or renewable resources have an abundance of electricity supply that could potentially be shared (e.g., Quebec); whereas other provinces/territories may not have a stable source to easily enable them to switch from coal-fired electricity (e.g., New Brunswick, Nova Scotia). Regional electricity planning, through an initiative like RECSI, provides opportunities for provinces/territories within the same region to identify projects that could benefit the region and/or to learn from each other.

The evaluation identified that because electricity generation falls within the jurisdiction of provinces/territories, the federal government has limited ability to take action beyond regulations or encouraging collaboration or dialogue amongst provinces/ territories. To that end, most interviewees identified that the involvement of the federal government through the RECSI Initiative encouraged provinces/territories to look beyond their own borders to cooperate and share information and knowledge at a regional level. Initiative documents confirm that one of the key elements of interventions such as RECSI is aligning key stakeholders (i.e., provincial/territorial government representatives and utilities) in different provinces/territories within a region to identify regional differences and opportunities for each jurisdiction.

Through the RECSI Initiative, NRCan provided funding to engage consultants to conduct regional studies examining potential projects in the Atlantic and Western regions to help meet the objective of transitioning away from coal-fired electricity. Some interviewees, particularly RECSI Initiative management and staff and provincial/ territorial representatives, identified that this support was needed to facilitate regional electricity planning studies; with at least one noting that provinces/territories may not have been able to fund these studies otherwise.

Consensus among interviewees was that the dialogues and studies conducted as part of the RECSI Initiative responded well to their needs and expectations related to regional electricity planning. Most notably, the Initiative was effectively able to bring together provinces/territories in two regions (Atlantic and Western) to collaborate and share information from a regional perspective. In particular, the networking opportunity through these dialogues helped provinces/territories in these regions:

• Gain an understanding of challenges and priorities of other provinces/territories;
• Discuss electricity and infrastructure planning at a regional inter-provincial level; and
• Build trust between provincial/territorial governments and their utility organizations.

Some external interviewees, however, identified some needs to which the Initiative did not respond, or that it could have better addressed. For example:

• Focusing primarily on electricity generation rather than transmission in the Atlantic region was not useful to provide options to all Atlantic provinces;
• Insufficient attention was given to project implementation once they were identified through the Initiative, including information related to cost-sharing and benefit-sharing; and
• The Initiative did not look at how to share the geographic diversity of renewable resources.

Nevertheless, the majority of interviewees believed the need for federal government involvement would continue in the future. Specifically, interviewees noted there would be a need for the federal government to continue to provide leadership to push provincial/territorial governments on electricity infrastructure, and to facilitate regional information sharing and collaboration.

To what extent do the objectives of initiatives like RECSI align with federal government and NRCan priorities?

Alignment with Federal Government Priorities

As noted previously, the RECSI Initiative was part of Phase 1 of the Government of Canada’s Investing in Canada Plan. Thus, it aligns with federal government priorities related to addressing climate change and supporting the transition to a low-carbon economy. In particular, this Initiative helped fossil fuel-reliant provinces/ territories identify opportunities to transition to renewable energy.

Budgets 2016 and 2017, as well as Canada’s PCF, also identify the importance of investments in the electricity sector. For instance, Budget 2016 acknowledges the need to identify electricity infrastructure projects that have the potential to achieve significant GHG reductions and help inform future investments in electricity infrastructure that will maximize both economic and environmental benefits.

The current government has also prioritized collaboration between the federal government and provinces/territories. Additionally, as part of the 2016 Vancouver Declaration First Ministers committed to “foster[ing] dialogue and development of regional plans for clean electricity transmission to reduce emissions” (Office of the Prime Minister, 2016). Initiatives like RECSI directly support this priority because they are designed specifically to facilitate cooperation on electricity resources through regional dialogues that involve the federal government, provinces/territories and their utilities.

Lastly, the RECSI Initiative supports Canada’s international commitments, specifically those made through the 2015 Paris Agreement in which Canada committed to a 30% reduction in GHG emissions by 2030 compared to 2005 levels. The government has announced regulations to be implemented in 2030 that phase out coal-generated electricity, which will help contribute to this commitment. Initiatives like RECSI help provinces/territories identify projects or infrastructure for investments that might have the greatest economic and environmental benefits.

Alignment with NRCan Priorities

In both 2015 and 2018, NRCan Ministerial Mandate Letters included the priority for the Minister and department to work closely with provinces/territories (Office of the Prime Minister, 2015 and 2018).

Moreover, since its implementation in 2016-17, the Initiative has directly supported NRCan priorities, as demonstrated through its inclusion in the Program Alignment Architecture (PAA) contributing to departmental strategic outcome 2 “Natural Resource Sectors and Consumers are Environmentally Responsible”, and more recently, through its contribution to NRCan’s Electricity Resources Program in the Departmental Results Framework (DRF). As part of this Program, the RECSI Initiative contributes to the departmental result “Canada’s natural resources are sustainable”, by strengthening cooperation and dialogue with provinces/territories and their utilities to identify opportunities to decrease GHG emissions from Canada’s electricity infrastructure projects (NRCan, 2017).

Is there a legitimate and appropriate role for the federal government to convene and facilitate cooperation among provinces/territories and their utilities to identify least-cost options for transitioning fossil fuel-reliant regions in Canada to renewable energy through regional electricity cooperation and strategic infrastructure?

Constitutionally, provinces/territories have jurisdiction over the development of their natural resources and energy infrastructure. However, under the Department of Natural Resources Act (1994, c. 41), NRCan has a mandate to plan activities that promote cooperation with provincial/territorial governments in relation to the development of Canada’s natural resources, including the generation of electricity. In addition, the Canadian Environmental Protection Act (1999) confirms it is the responsibility of the federal government to protect the environment from any adverse effects from the release of pollutants (e.g., air pollution from GHG emissions). Consistent with these, the PCF specifically indicates that federal and provincial/territorial governments should work together to “facilitate, invest in, and increase the use of clean electricity across Canada” (Government of Canada, 2016), which is a direct objective of the RECSI Initiative.

The evaluation also confirmed the appropriateness of NRCan’s involvement in initiatives like RECSI. Prior to this Initiative, for instance, there had been little regional cooperation on electricity infrastructure, particularly among western provinces/territories. The exception was the Atlantic Energy Gateway Initiative (delivered by NRCan from 2009-10 through 2012-13), which facilitated collaboration and cooperation among eastern provinces to develop the Atlantic renewable energy sector.

Leadership by the federal government in bringing together provinces/territories to collaborate can help reduce barriers related to regional electricity cooperation. Documents from the RECSI Initiative identified that, historically, there have been siloes within provinces/territories, with self-sufficiency being a key priority. Interviewees indicated regional electricity planning would likely not have happened without the RECSI Initiative, which suggests this priority may not have changed without NRCan’s involvement. Because provinces/territories have different resources, needs, and interests related to electricity production and generation, the perceived benefits of regional electricity cooperation and infrastructure investments differ from one province/territory to the next.

Furthermore, RECSI Initiative documents revealed that some provinces already have established and highly integrated electricity transmission relationships with the United States but do not have those same relationships with other Canadian provinces/territories. Documents suggested this is primarily because of the geography of Canada, but also because of investments from the United States into the development of the infrastructure. As a result, it could be that not all provinces have the same economic incentive to cooperate with other Canadian provinces/territories. Through initiatives like RECSI, the federal government can encourage cooperation among provinces/territories by bringing them together to share information and learn about the different challenges each faces, thereby facilitating regional energy planning to contribute to a federal policy objective.

Lessons Learned

Lesson Learned 1. Programs/initiatives focused on working with, and facilitating cooperation between, provinces/territories, and industry (if applicable), particularly in regional natural resource planning exercises, should clearly address or explain possible approaches or suggestions (if any) for support or implementation of projects/actions by provinces/territories following the end of the initiative.

Performance – Effectiveness

To what extent did activities of the Initiative result in the planned outputs?

At the planning phase, the RECSI Initiative intended to establish up to three regional dialogues. RECSI Initiative documents revealed that the priority was to have an Atlantic and Western region dialogue, with the potential of also including a Central region dialogue. Following discussions with provincial/territorial counterparts (e.g., through a 2016 Energy and Mines Ministers Conference and subsequent communication between NRCan and provinces/territories), the Initiative supported two dialogues and studies. One dialogue and study was in the Atlantic region, including government and utility representatives from New Brunswick, Nova Scotia, Prince Edward Island, and Newfoundland and Labrador. The second was in the Western region, including government and utility representatives from British Columbia, Alberta, Saskatchewan, Manitoba, and the Northwest Territories. Ontario and Quebec, which would have made up a Central region dialogue, produce the majority of their electricity from renewable or non-emitting sources, so it was reasonable to proceed with the two priority dialogues and studies.

The document review and interviews confirmed that each dialogue identified different scenarios to test through the regional studies to be led by expert consultants contracted specifically for this purpose. Based on these dialogues, the consultants prepared technical reports (one for each region), including modelling details, results, and other detailed information for each of the scenarios or projects tested. NRCan then worked with participating provinces/ territories and utilities to publish summaries for policy makers, again, one for the Atlantic region and another for the Western region.

By facilitating dialogues in the Atlantic and Western regions, provinces/territories in these regions were able to identify potential electricity infrastructure projects that would contribute to the following PCF objectives: (1) to connect clean power to the provinces/territories that need it; and (2) for fossil fuel-reliant provinces/territories to identify alternatives. For example, the Western Region Summary for Policy Makers revealed that they tested 25 scenarios and identified that six of these could reduce GHG emissions and result in overall electricity production cost savings (NRCan, 2018b).

Similarly, the Atlantic Region Summary for Policy Makers indicated that this region tested five scenarios against a business as usual scenario; however, they did not identify specific projects. Rather, the Atlantic region dialogue and study found that each of the five scenarios tested “will require significant investment, both from public and private sources” as there are significant costs associated with each scenario (NRCan, 2018a). Thus, for this region, additional examination and consideration is required before identifying specific projects.

Still, interviewees generally agreed that these dialogues and studies likely would not have happened without support from the RECSI Initiative, or at least would not have happened to the same degree. Interviewees also found the dialogues and studies to be helpful and informative. As noted previously, provinces/territories have their own priorities, often focused within their own borders; this Initiative provided a forum for, and encouragement to, look outside of their individual boundaries, network, learn from, and collaborate with others at a regional level.

To what extent do the design and delivery of the Initiative facilitate the achievement of its intended immediate outcomes?

As identified in the logic model (presented previously in Figure 1), the RECSI Initiative intended to achieve the following three immediate outcomes:

1. Collaboration among provinces/territories in the area of regional electricity planning;
2. Agreement among provinces/territories on investigating scenarios/actions that increase the use of renewable energy and increase grid integration (where applicable); and
3. Increased technical and policy knowledge among provinces/ territories, utilities, and the federal government about actions that could increase the use of renewable electrical energy and increase grid integration (where applicable).

The presumption was that if NRCan convened and facilitated regional dialogues, and provided financial support for studies, there would by an incentive for provincial/territorial governments and their utilities to engage in regional electricity planning. By providing financial support through contribution agreements to a lead participant in each dialogue (Nova Scotia Power for the Atlantic dialogue, and Alberta Electricity System Operator (AESO) for the Western dialogue), this also gave ownership to regional participants, which was expected to increase the willingness of provinces/territories and utilities to share information.

Collaboration among provinces/territories in the area of regional electricity planning

Because NRCan designed the RECSI Initiative explicitly to be collaborative, it directly contributed to collaboration among provinces/territories, including their respective utilities, on regional electricity planning. Documents and interviews confirmed this contribution as they identify that there was at least one representative from each applicable provincial/territorial government and provincial/territorial utility engaged in each of the regional dialogues and studies. This included senior representatives participating on the Steering Committees for the Initiative, and working level representatives participating on the Technical Advisory Committees. Nearly all interviewees identified the collaborative design of the RECSI Initiative (i.e., NRCan facilitating and convening the two dialogues) as key to its success. While a few interviewees suggested some collaboration might have happened without NRCan involvement, they agreed that it would not have happened to the same extent or with the same results.

To achieve this outcome, NRCan had to ensure it included the appropriate stakeholders in each dialogue and study, and stakeholders had to be willing to participate actively in these dialogues and studies (e.g., share information openly). To that end, the Initiative included provincial/territorial governments and utilities with responsibility for power system planning within their respective province/territory. In most provinces/territories, these utilities are Crown Corporations; however, in Alberta, there is a deregulated energy market with many active investor-owned utilities. NRCan made the decision to include the AESO rather than multiple investor-owned utility organizations. Interviews and documents revealed the reasons for this included:

• AESO is an independent operator, not representing a specific interest, that is able to provide information and data required for regional electricity planning purposes; and
• There was a need to limit the number of participants in order for the dialogues and studies to be feasible and result in actionable (rather than visionary) projects for examination in the studies.

For similar reasons, other industry or consumer stakeholders were also not included in either of the regional dialogues and studies. Almost all interviewees agreed with these exclusions, noting that including more stakeholders and interests would have negatively affected the ability of the Initiative to produce its outputs and achieve the intended immediate outcomes. Importantly, all stakeholders (participants and non-participants) had the opportunity to take part in multiple webinars to discuss the Initiative, including the results of each dialogue and study.

Interviewees involved in each of the regional dialogues and studies agreed that the right stakeholders participated in the Initiative with most interviewees indicating that the dialogues facilitated knowledge/information sharing, such as different perspectives, contexts, needs, and challenges. Interviewees generally agreed that this knowledge/information sharing would not have happened, at least not to the same extent, without the RECSI Initiative.

Agreement among provinces/territories on scenarios/actions

Because provincial/territorial government and utility representatives were willing to share information through their participation on either the Steering Committees or Technical Advisory Committees, participants in each regional dialogue were able to agree on which scenarios/actions to test through the studies. Many external interviewees as well as some RECSI Initiative management and staff interviewees identified that NRCan’s financial support was needed for these studies to be undertaken, with most indicating provinces/territories would have been unlikely or unable to undertake them otherwise.

As identified by many interviewees, the value-added of the collaboration (described above) was that the RECSI Initiative provided a forum for provinces/territories and utilities to share technical knowledge that facilitated the design of the studies. More specifically, it enabled participants to comfortably share information, generate and assimilate data sets, develop baseline cases, and examine projects on a regional basis rather than for an individual province/territory.

It is clear that the RECSI Initiative achieved this outcome because both regions conducted studies assessing environmental and economic impacts of various scenarios/actions based on jointly developed models, and produced technical reports and summaries for policy makers. Furthermore, documents and interviews revealed that work has been continuing in both regions related to some of the scenarios/actions identified through this Initiative. In particular, the Atlantic Region Summary for Policy Makers noted that NRCan has been continuing to work with the Atlantic region in two areas:

1. Regional Hybrid Portfolio: “…investigating an appropriate amount of dispatchable firm generation capacity, and/or storage and demand-side management systems, and transmission reinforcement to support more sources of variable renewable generation for further GHG reductions, or to replace incremental coal-fired capacity retirement”; and
2. Second transmission interconnection between Nova Scotia and New Brunswick: “…by undertaking a cost-benefit analysis…[that] could inform how costs could be allocated between the provinces and how federal funding could advance the project” (NRCan, 2018a).

Similarly, the Western Region Summary for Policy Makers revealed that “NRCan will continue to work with SaskPower and Manitoba Hydro to explore further the benefits of increased intertie capacity…” associated with three options investigated for a new intertie between Manitoba and Saskatchewan (NRCan, 2018b).

Increased technical and policy knowledge among provinces/territories, utilities, and the federal government

As a result of the collaboration and the examination of different scenarios/action, the evaluation confirmed there was an increase in both technical and policy knowledge among all stakeholders, including provinces/territories, utilities, and the federal government. By looking at electricity projects from a regional perspective, and with participants openly sharing information, all stakeholders gained knowledge to some degree. For example, all four utilities interviewed as part of the evaluation reported that the dialogues and studies enabled all stakeholders to improve their understanding of infrastructure, project options, problems or challenges, and approaches used by other provinces/territories.

According to the Atlantic and Western Region Summaries for Policy Makers, this increased technical and policy knowledge included information about the different needs or opportunities of the different provinces/territories within the region. Examples of this include understanding that:

• There are legacy policies (provincial and federal) that have resulted in a diverse supply of electricity generation in the Atlantic region;
• New Brunswick and Nova Scotia are already reducing GHG emissions even without a full transition away from coal-fired electricity;
• Even though Newfoundland and Labrador have discovered significant amounts of gas, the infrastructure remains undeveloped;
• Both Alberta and Saskatchewan rely on coal-fired electricity generation and plan to transition to a more renewable mix of electricity generation; and
• Significant hydropower resources exist in British Columbia and Manitoba that could be used to help Alberta and Saskatchewan transition away from fossil fuel-fired electricity.

As noted previously, each of the studies also identified scenarios/ projects and assessed the environmental and economic impact these might have. Provincial/territorial and utility interviewees confirmed the knowledge gained through the collaboration and studies is information they did not previously have and they expect it will be useful to future discussions and decisions. For example, through the Western region dialogue and study it was discovered that:

• Interprovincial transmission projects would reduce GHG emissions by between 0.5 megatonnes and 1.2 megatonnes per year;
• Electrification of LNG and upstream natural gas production has the potential for significant GHG emission reductions; and
• There are a number of project types between Alberta and Saskatchewan that would reduce GHG emissions for both provinces. (NRCan, 2018b)

Similarly, through the Atlantic region dialogue and study it was found that:

• There could be greater renewable energy generation and use in the Atlantic region if transmission interconnections between Nova Scotia and New Brunswick was increased; and
• Significant GHG emissions reductions are possible with coordinated regional action. (NRCan, 2018a)

Have there been any unintended outcomes (positive or negative) as a result of the implementation of the Initiative?

Although the RECSI Initiative intended for provinces/territories and their utilities to collaborate, provincial/territorial and utility interviewees confirmed that the networks and relationships they built through the Initiative were continuing even now that the Initiative had ended. For example, participants could now easily communicate with representatives from utilities and other provinces/territories who also participated in the Initiative because they had contacts with whom they had built relationships. Neither the documents nor interviews identified any negative unintended impacts resulting from the implementation of the Initiative.

What internal and external factors impact the Initiative’s ability to respond to the intended need and achieve the intended outcomes? How and to what extent?

Many external interviewees identified a key internal factor that influenced the success of the RECSI Initiative, namely the high quality of the NRCan employees delivering the Initiative. Without being asked about this specifically, interviewees repeatedly highlighted that these employees kept the Initiative on track and effectively managed to get busy stakeholders from provinces/ territories and utilities to participate in meetings/teleconferences and provide the information and data necessary to support the studies.

In addition, the evaluation revealed a number of external factors that had the potential to either positively or negatively impact the ability of the RECSI Initiative to respond to the needs of stakeholders (as discussed under Relevance) and achieve (or contribute to) the intended immediate, intermediate, and ultimate outcomes (as identified in Figure 1). These external factors fall into three categories: (1) policy infrastructure; (2) provincial/territorial culture or practice; and (3) industry and consumer buy-in. Each of these categories, and the specific factors, are discussed below.

Policy Infrastructure

The evaluation identified three factors related to policy infrastructure that influence the relevance and effectiveness of an initiative such as RECSI, namely:

• Federal priorities, policies, and regulations;
• Willingness of provinces/territories and their utilities to participate and engage in regional electricity planning; and
• Funding/financial support for infrastructure projects.

In the case of the RECSI Initiative, the federal government had priorities, policies, and regulations that facilitated the achievement of the intended outcomes of this Initiative. This included, the PCF, the government focus on collaborating with provinces/territories, and the introduction of new regulations to phase out coal-fired electricity in 2030. As noted previously, Initiative documents suggested that there are individual priorities and geographic challenges that present some barriers to regional cooperation on electricity planning and infrastructure. Having federal priorities, policies, and regulations in place that include or implicate provinces /territories may create an incentive for provinces/territories to collaborate on a regional basis to share lessons and information, and to identify potential opportunities to help each other meet provincial/territorial needs and respond to federal regulations, policies and priorities. This may be particularly important when there are provincial/territorial mandates that do not explicitly consider regional impacts; that is, impacts beyond the individual borders of a single province/territory.

Notwithstanding these federal priorities, policies, and regulations, because the development of natural resources, including in relation to electricity, falls within the jurisdiction of provinces/territories, a federal government initiative like RECSI can only be effective if provinces/territories are willing to participate and engage actively. The evaluation revealed that provinces/territories in both the Atlantic and Western regions were interested in participating.

However, some interviewees indicated that even though provinces/ territories collaborated in these dialogues and studies, more investigation is needed related to cost and benefit sharing. For example, if one province/territory will benefit significantly more than another, there is a need to define the incentive for the other province/territory to contribute to a project (e.g., investing in infrastructure to export cleaner electricity to another province/territory). As noted previously (under Relevance), the studies and dialogues within the RECSI Initiative did not explicitly address this issue. While this did not negatively affect the achievement of the immediate level outcomes, it could hinder the ability to contribute to the intermediate and ultimate outcomes of investments in infrastructure and reduced reliance on fossil fuel-fired electricity.

Related to this is the capacity to fund or provide financial support for new electricity infrastructure. The RECSI Initiative never intended to provide financial support beyond facilitating the dialogues and studies. As such, even though potential projects have been identified for both the Atlantic and Western regions, it is still not clear if there is capacity elsewhere to invest in the identified infrastructure projects. In particular, in the Atlantic region, the Summary for Policy Makers revealed that all scenarios tested would require significant investments from both the public and private sectors. Some external interviewees reiterated this point, noting that provinces may not be able to develop the infrastructure alone. As noted above, if this capacity does not exist, intermediate and ultimate outcomes related to investments and reduced reliance on fossil fuel-fired electricity will be harder to achieve.

Provincial/Territorial Culture or Practice

Documents and interviews confirmed two provincial/territorial culture or practice related factors that could influence the relevance and effectiveness of the RECSI Initiative. These include:

• Openness of provinces/territories with current integrated electricity transmission relationships with the United States to diversify across Canada (i.e., East-West) instead of, or in addition to, these North-South relationships; and
• Political relationships between federal and provincial/ territorial governments and/or between one province/ territory and another.

As discussed previously (under Relevance), documents revealed that some provinces/territories have established relationships with the United States pertaining to electricity transmission. This might create a disincentive for them to collaborate with Canadian provinces/territories, particularly given the lack of East-West integration and geographical barriers that exist within Canada. Nevertheless, the evaluation confirmed active participation of provinces/territories in both the Atlantic and Western regions in the RECSI Initiative, indicating there might be some interest and willingness to explore these options further. Results of the interviews with all stakeholders suggest that this interest in collaboration may not have happened without an Initiative like RECSI; and without maintaining the momentum from this, it is possible that these opportunities may not continue and the intermediate and ultimate outcomes may not be realized.

A second culture or practice factor relates to political relationships, either between the federal government and provinces/territories, or between one province/territory and another. Concerning the Western region, for example, some interviewees noted potential tensions between Western provinces and the federal government on some energy-related issues. RECSI Initiative documents also identified potential challenges between British Columbia and Alberta, primarily because of the different energy markets, with a crown corporation in British Columbia and investor-owned utilities in Alberta. In the Atlantic region, on the other hand, documents and interviews flagged the tensions between Quebec and some Atlantic provinces, stemming from the relationship between Hydro Quebec and Newfoundland and Labrador related to historical energy infrastructure projects and transmission agreements. At the time of this evaluation, interviewees suggested that challenges with these relationships are not hindering their progress on regional electricity cooperation.

Industry and Consumer Buy-in

The third category of potential factors that could influence the ability of the RECSI Initiative to be relevant and effective relates to buy-in by industry and consumers. More specifically:

• Inclusion of appropriate stakeholders in the Initiative; and
• Support from consumers as ratepayers.

Whether the appropriate stakeholders were included in the Initiative was discussed previously, in particular noting the explicit decision to exclude investor-owned utilities from Alberta in the Western dialogue and study as well as other potential industry interests in both regions. As noted earlier in the report, nearly all interviewees agreed it was appropriate to exclude these stakeholders; that including them would have negatively affected the success of the dialogues and studies. Although stakeholders were excluded from the regional dialogues and studies, they were invited to participate in multiple webinars related to them, including on the results.

In addition to this, a couple of internal (RECSI Initiative management and staff) and external interviewees from the Atlantic region dialogue and study did note that not having Quebec involved resulted in some gaps. Although it was possible for the Atlantic region study to include some technical information pertaining to Quebec, there was no information about different provincial policies that might have been helpful to the study.

Some interviewees, particularly in the Atlantic region dialogue and study raised the need for support or acceptance from consumers as both tax and ratepayers to invest in alternative or new electricity projects. RECSI Initiative documents also pointed towards this factor, suggesting there could be political pressure from ratepayers related to the financial cost of developing electricity infrastructure projects. The risk is that, because of the financial investment required, ratepayers, who may already be paying high rates for electricity, would feel the costs. While this has not impeded the achievement of the immediate outcomes, if consumers/ratepayers do not support the investment, provinces/territories may not be inclined to proceed with the projects identified through the dialogues and studies.

Lessons Learned

There are no lessons learned associated with this section.

Performance – Efficiency

To what extent was the Initiative implemented as planned?

1. If there were deviations, how and why did these occur and what was the impact (positive or negative) on the implementation of the Initiative?
2. Did the deviations change the extent to which the activities, outputs, and intended outcomes could reasonably be expected to contribute to the higher-level outcomes? If so, how? What impact (if any) did this have on the ability to respond to the identified need for the Initiative?

RECSI Initiative documents identified that the Initiative was designed to be utility-driven, using a bottom-up approach (i.e., managed by NRCan, but not directed by them). The regional dialogues, and participants in them, were to be confirmed in April/May 2016, with all work (including technical reports and summaries for policy makers) completed by March 31, 2018.

Although the implementation of the Initiative did not seem to be systematically monitored, the document review provided no indication that the Initiative was not implemented as planned. Similarly, most of the internal and external interviewees felt the RECSI Initiative proceeded as planned. A few interviewees, representing provincial/territorial and utility representatives, identified some unexpected delays in the process, notably the time required to get non-disclosure agreements in place at the outset and to finalize reports at the end of the process. For the most part, these interviewees did not believe these delays had a negative impact on the Initiative and/or the ability to proceed with projects afterwards. One interviewee representing a utility, however, reported that the delays meant information from the studies was not available in time to inform internal projects (e.g., intertie restorations), which resulted in some provinces/territories losing interest in the projects being examined through the studies.

Although the RECSI Initiative was never intended to provide funding to infrastructure projects resulting from the studies, RECSI Initiative management and staff had expected that provinces/territories could receive funding through the following:

• Canada Infrastructure Bank (CIB), a federal Crown Corporation designed to invest in new revenue-generating infrastructure projects; or
• Integrated Bilateral Agreements (IBAs) negotiated through green infrastructure funding, which allocate funds to provinces/territories to potentially fund “…the construction of electricity transmission lines to bring clean power to a neighbouring province or territory…” (NRCan, 2018a and 2018b).

Budget 2017 indicated that strategic electricity grid inter-connections would be eligible for investments from the CIB. However, RECSI Initiative documents revealed that as the CIB was implemented its criteria became a potential barrier to the funding of these projects because provinces/territories might not want to rely on private capital or might not have market rules consistent with the ideas being supported by the CIB. IBAs continue to be a funding option, and at least one provincial/territorial interviewee indicated the results of the RECSI Initiative study would help inform their business proposal to access this funding.

Many internal interviewees (RECSI Initiative management and staff) identified lack of funding for projects as a potential issue for the success of the Initiative, indicating that some participants may have expected there to be funding. However, most external interviewees understood that funding projects following the studies was beyond the scope of the Initiative. Nevertheless, as discussed previously, some external interviewees did indicate it would have been helpful for the RECSI Initiative to have clearly explained potential options for implementing projects afterwards, even if the actual support and/or implementation of projects was outside of its scope.

As discussed (under Performance – Effectiveness), the RECSI Initiative successfully produced the intended outputs and achieved the intended immediate outcomes, primarily as a result of the design of the Initiative (i.e., collaborative approach, NRCan funding of studies). From an efficiency perspective, the evaluation also assessed the extent to which this design could reasonably be expected to contribute to the intermediate and ultimate outcomes. As identified in Figure 1, the Initiative intended to contribute to the following intermediate and ultimate outcomes:

• Investments by provinces/territories, utilities, and/or the federal government in the construction of large clean energy projects and transmission/integration projects to deliver that clean energy (intermediate outcome); and
• Reduced reliance on fossil fuel-fired electricity across Canada.

To assess this, interviewees were asked to identify the likelihood that the RECSI Initiative would result in these outcomes. They were also asked to identify the likelihood that these outcomes could be (or would have been) achieved without NRCan involvement, and without the RECSI Initiative in particular.^{Footnote 1} For both scenarios, interviewees were also asked to identify the potential size of the impact. Though not a definitive assessment of whether an initiative like RECSI can result in these outcomes, the analysis does provide some indication of whether it was reasonable to expect an initiative designed to facilitate collaboration and conduct studies to contribute to outcomes such as these longer-term outcomes.

Both internal and external interviewees expect that the RECSI Initiative will contribute to investments by provinces/territories, utilities, and/or the federal government in the construction of large clean energy projects and transmission/integration projects to deliver that clean energy (intermediate outcome). Analysis of responses to the questions described above revealed that internal stakeholders expect there is a 30% greater chance that these investments will be made because of the Initiative, and external stakeholders think there is a 24% greater chance of this.

Notably, participants from the Western region expect a 44% greater chance of these investments being made because of the Initiative compared to Atlantic region participants who expect only a 19% greater chance of this. This likely reflects the determination in the Atlantic region that the projects would have significant costs associated with them. Regardless, the RECSI Initiative is expected to contribute to these investments because of the information and knowledge produced about potential projects through the collaboration and studies – information and knowledge that was not previously available and would not have been available without the Initiative (as identified earlier in this report).

That being said, many of the factors identified previously could still affect whether or not these investments are made. For instance, in the Atlantic region where the study made it clear that large investments will be needed for any of the projects, and that the province benefitting the most from the projects may not be the one incurring the greatest cost, there may not be an incentive to invest in large clean energy and/or transmission integration projects. Whereas, in the Western region, where projects with positive environmental and economic impacts were identified through the studies, there may be more of a justification to make investments, for example.

In terms of the ultimate outcome, although both internal and external stakeholders believe that the RECSI Initiative will contribute to a reduced reliance on fossil fuel-fired electricity use across Canada, both groups of interviewees indicated this outcome will be achieved even if the Initiative did not exist because of the ECCC coal regulations that take effect in 2030. Nevertheless, external stakeholders, in particular, noted that an initiative like RECSI can help provinces/territories identify potential projects or solutions that could help them meet the requirements of new regulations; including regional projects that may not have been considered previously, which may help them meet the requirements of new regulations in more efficient and effective ways.

To what extent are the design and delivery of the Initiative the most efficient means of producing the intended outputs and achieving the intended immediate outcomes?

Over the two years the RECSI Initiative was delivered (2016-17 to 2017-18), expenditures were anticipated to be approximately $2.6 million, including $1.04 million in grants and contributions (G&C) funding (approximately 40% of planned expenditures). As Table 1 illustrates, the actual expenditures were approximately $2.0 million, including about $0.92 million in G&C funding (approximately 45% of actual expenditures). Actual expenditures accounted for about 78% of planned expenditures.

Table 1: Planned and Actual Expenditures for the RECSI Initiative ($ million), 2016-17 to 2017-18

Expenditure Type	2016-17		2017-18		Total
	Planned	Actual	Planned	Actual	Planned	Actual
Operation and Maintenance (O&M)	0.33	0.06	0.28	0.06	0.61	0.13
SalariesTable footnote *	0.34	0.30	0.35	0.43	0.69	0.73
OverheadTable footnote **	0.13	0.13	0.12	0.12	0.25	0.25
G&C	0.20	0.10	0.84	0.82	1.04	0.92
Total	1.00	0.59	1.60	1.44	2.60	2.00

As Table 1 shows, actual expenditures only exceeded planned expenditures in the case of salaries. Expenditures were higher on salary because in 2017-18 there were four (4) FTEs allocated to the Initiative, rather than the planned three (3).

The outputs produced by the RECSI Initiative (i.e., dialogues and knowledge products) were produced using approximately $0.9 million ($0.5 million for the Western region, and $0.4 million for the Atlantic region). As identified previously, stakeholders felt these outputs were important to achieving the immediate outcomes, in particular, increasing collaboration, and increasing knowledge.

Nearly all external interviewees expressed satisfaction with the Initiative overall, particularly the collaboration on the studies and knowledge gained through the process and with results of the studies. Nevertheless, internal and external stakeholders suggested some tweaks to the Initiative that could improve its processes, effectiveness, and/or efficiency, but there were no wholly different or alternative designs proposed. The suggested tweaks included:

• Engaging stakeholders (provinces/territories, utilities) at the planning phase of the Initiative to conduct resource planning, discuss regional electricity planning and the needs of the different provinces/territories, and use this to inform the direction of the Initiative (noted by both internal and external interviewees);
• Broaden the scope to include other areas, for example, secondary benefits such as electrifying industry, distributive generation, increasing capacity to implement renewable energy sources, or focusing on opportunities between pairs of provinces/territories rather than a whole region (noted by both internal and external interviewees);
• Using academic associations/consortiums to conduct the studies rather than contractors (noted by internal interviewees);
• Reducing the bureaucracy such as, having NRCan hire the contractors directly rather than through a third party delivery agent, flexible funding arrangements that are not based on fiscal year ends (noted by external interviewees);
• Clarifying objectives and outcomes, including what funding (if any) is available for projects (even if not included as part of the Initiative) (noted by external interviewees); and
• Clarifying roles and responsibilities, particularly in terms of steering committee decision-making, and communicating with contractors during the conduct of the studies (noted by external interviewees).

Some of these suggestions would likely improve the efficiency of an Initiative like RECSI, particularly in terms of timeliness, but not necessarily in terms of resource use.

Notwithstanding these suggested adjustments, a couple of strengths were identified by internal and external interviewees, including: the collaborative design (discussed previously); and having external parties conduct the studies as this prevented issues about ownership of the product and the information.

Lessons Learned

Lesson Learned 2. Implementation, including of activities and outputs (e.g., as identified in a logic model), should be systematically monitored to identify if the initiative is on track to meet its intended outcomes or if course correction is needed.

Lesson Learned 3. Future initiatives designed to facilitate collaboration and/or the conducting of studies, particularly related to regional natural resource planning, could consider some adjustments to the design. For example, including stakeholders early in the process to identify needs and inform the direction and scope of the initiative, clarifying roles and responsibilities, and ensuring objectives and intended outcomes are clear at the outset.

Conclusions

Relevance

There is a need and role for NRCan to facilitate cooperation between provinces/territories particularly in regional natural resource planning exercises. Even though provinces/territories have jurisdiction over the development of their natural resources, the federal government plays a legitimate and appropriate role when it intervenes to support provinces/territories in collaborating on a regional basis. Through initiatives like RECSI, for instance, provinces/ territories are encouraged to share information and consider opportunities on a regional basis, while also contributing to federal priorities and commitments.

Performance – Effectiveness

The RECSI Initiative successfully resulted in its planned outputs, namely, regional dialogues (Atlantic and Western) and knowledge products from these dialogues (lists of potential projects, technical studies, and summaries for policy makers). Through interviews and a document review, the evaluation confirmed the Initiative directly contributed to the three intended immediate outcomes: collaboration among provinces/territories in the area of regional electricity planning; agreement among provinces/territories on scenarios/actions; and increased technical and policy knowledge among provinces/territories, utilities, and the federal government.

External interviewees identified the quality and capacity of NRCan employees delivering the Initiative as a key internal success factor. The evaluation also found that the collaborative design of the Initiative and financial support from NRCan for the studies enabled the Initiative to achieve the intended immediate outcomes, with interview results suggesting this collaboration and the studies likely would not have happened otherwise.

For the most part, the conditions are in place to facilitate the Initiative’s contribution to the intended intermediate and ultimate outcomes. The exception relates to the capacity of provinces/ territories and/or other stakeholders to fund projects identified through the regional dialogues/studies, and whether consumers/ ratepayers will support these investments even if it means an increase in electricity rates.

Performance – Efficiency

There is no evidence to suggest the RECSI Initiative was not delivered efficiently. However, because there was no alternative for comparison, no definitive conclusion can be made on whether it is the most efficient design to produce the outputs and achieve the outcomes. The evaluation did confirm the Initiative was largely implemented as planned. It also determined that it is reasonable to expect an initiative designed to facilitate collaboration and conduct studies to contribute to longer-term outcomes (i.e., investments in clean energy or transmission/ integration projects, and identifying potential options or solutions to reduce reliance of fossil fuel-fired electricity).

Actual expenditures for the Initiative were approximately 78% of planned expenditures, suggesting a good use of resources. Moreover, no wholly different designs were identified; rather interviewees suggested only some adjustments that could be made. Stakeholders also generally agreed that the collaborative approach and use of external experts to conduct the studies were key strengths in the design of the Initiative.

Appendix 1: Evaluation Team

• Christian Asselin, Chief Audit and Evaluation Executive (CAEE)
• David Ash, Senior Advisor to the CAEE
• Mary Kay Lamarche, Senior Evaluation Manager
• Lauren Omson, Junior Evaluation Analyst
• Odilia Maessen, Evaluation Officer

Appendix 2: Reference List

• Government of Canada. (2016). Pan-Canadian framework on clean growth and climate change (PCF).
• Natural Resources Canada (NRCan). (2017). 2017-18 departmental plan.
• NRCan. (2018a). Atlantic region summary for policy makers.
• NRCan. (2018b). Western region summary for policy makers.
• Office of the Prime Minister. (2015). Minister of natural resources mandate letter [Archived].
• Office of the Prime Minister. (2016). Communiqué of Canada’s First Ministers.
• Office of the Prime Minister. (2018). Minister of natural resources mandate letter.