Evaluation of the Energy Innovation Program

Audit and Evaluation Branch
Natural Resources Canada
July 8, 2019

Table of Contents

• ACRONYMS
• EXECUTIVE SUMMARY
• INTRODUCTION
• FINDINGS
  • Relevance
  • Performance – Effectiveness
  • Performance – Efficiency and Economy
• CONCLUSIONS AND RECOMMENDATIONS
• APPENDIX 1: ECOEII / EIP LOGIC MODEL

ACRONYMS

ACOA

Atlantic Canada Opportunities Agency

ADM

Assistant Deputy Minister

APD

Assistant Program Director

CAA

Clean Air Agenda

CC & CCS

Clean Coal and Carbon Capture and Storage

CEF

Clean Energy Fund

CEIP

Clean Energy Innovation Program

CMSS

Corporate Management and Services Sector

CRL

Commercial Readiness Level

CTS

Clean Transportation System

DEEP

Centre for Digital Entrepreneurship and Economic Performance

DG

Director General

DOE

Department of Energy

ECCC

Environment and Climate Change Canada

ecoEII

ecoENERGY Innovation Initiative

EEBC

Energy Efficient Buildings and Communities

EERE

Office of Energy Efficiency and Renewable Energy

EIP

Energy Innovation Program

EV

Electric Vehicle

EVID

Electric Vehicle Infrastructure Demonstration

FAA

Financial Administration Act

GHG

Greenhouse Gas

IETS

Innovation and Energy Technology Sector

IRAP

Industrial Research Assistance Program

NG

New Generation

NRC

National Research Council of Canada

NRCan

Natural Resources Canada

NRTEE

National Round Table on the Environment and Economy

NZE

Net Zero Energy

OEE

Office of Energy Efficiency

OERD

Office of Energy Research and Development

OGCT

Oil and Gas Clean Technology

PCF

Pan-Canadian Framework

PERD

Program of Energy Research and Development

PRL

Policy Readiness Levels

PV

Photovoltaic

R & D

Research and Development

RD & D

Research, development and demonstration

S & T

Science and Technology

SDG

Sustainable Development Goals

SG

Smart Grid

SME

Small and Medium-sized Enterprise

TRL

Technology Readiness Level

UOG

Unconventional Oil and Gas

VOC

Volatile Organic Compound

WD

Western Economic Diversification Canada

EXECUTIVE SUMMARY

About the Evaluation

The purpose of this evaluation is to assess the relevance and performance of the Energy Innovation Program (EIP), including the last three years of the ecoENERGY Innovation Initiative (ecoEII).The assessment of effectiveness includes consideration of the extent to which the current EIP is designed and delivered in a way that should facilitate or optimize outcome achievement. The evaluation covers the period of 2014-15 to 2017-18 and up to June 2018. These research, development and demonstration programs include funding for contribution agreements and CanmetENERGY laboratories. They cover a broad range of technology areas: electric vehicles, renewables, smart gird, clean oil and gas, and buildings.

The evaluation of the ecoEII and EIP programs is taking place within a broad context of both an evolving Pan-Canadian Framework to address climate change initiatives and Sustainable Development Goals and a major review and potential revision of Government of Canada business innovation program support. There are several other recent programs and initiatives designed to improve horizontal coordination and to develop and implement novel approaches to manage innovation programming. As such, the focus of this study has been to address key past results in terms of the implications for programming going forward.

What the Evaluation Found

The findings of this evaluation indicate that there is a need for federal government support to energy innovation with respect to both energy supply (oil and gas production, power plants, and electric grids) and demand, (buildings, transportation, industry) to address clean energy innovation. The development and adoption of affordable clean energy solutions is considered an important means of addressing climate change issues. Evaluation evidence indicates that government assistance is needed to reduce the risks associated with the high costs and uncertainty of developing and demonstrating new clean technologies. The marketplace on its own is unlikely to invest in sufficient levels to meet Government of Canada commitments to reduce GHG reductions. EIP is consistent with Canada’s commitments to reduce GHG emissions and is an important component of the Pan-Canadian Framework.

Science and technology advancement, on its own, is unlikely to address environmental and competitiveness needs, and should be part of a broader and integrated effort, which includes an appropriate regulatory and policy environment. EIP funding fits directly as part of a Pan Canadian Framework in which the regulatory environment is complemented by directly funded assistance for technology advancement (e.g. EIP). However, even in the absence of a favourable regulatory environment, evaluation evidence indicates that advancing clean energy science and technology is appropriate given the long time-frame required to address technology gaps and the need for science to inform policy, decision-making and regulatory issues.

Optimizing the impact of clean energy innovation also requires cooperation and coordination among federal departments and agencies as well as provincial and territorial governments. At the federal level, efforts are underway for a more coordinated ‘whole of government’ approach to innovation, which includes initiatives such as the Clean Growth Hub. Results of these efforts are expected to be realized over time and are not within the scope of this evaluation. Evaluation evidence suggested good cooperation between provinces/territories and NRCan in several technology areas. Documents also indicate that NRCan’s approach to energy innovation is fairly broad – engaging a variety of producers, adopters, regulators and other stakeholders.

In general, the role of NRCan in conducting and managing energy research, development and demonstration (RD&D) is considered by stakeholders to be appropriate given the Department’s legislated mandate, expertise and long-standing experience in delivering these types of programs. Past funded NRCan energy innovation programs and the Federal CanmetENERGY laboratories have created a tradition of trust which has resulted in NRCan acting as a ‘lead’ or catalyst for others to invest – among both government agencies and private investors. However, the recent creation of several new innovation programs dispersed among several departments and agencies has contributed to confusion among external stakeholders as to NRCan’s role vis-à-vis other federal government players delivering energy innovation programs.

Overall, the evaluation found that the energy innovation programming was reaching key stakeholder and target groups: industry, academia, regulators and provinces/territories. The program has also incorporated several changes to enhance its reach to small and medium sized companies.

Evaluation evidence indicates that outcomes pertinent to collaborations, leveraging, codes and standards, and technical progress have met or exceeded reasonable historical expectations. Indeed, the evidence suggests that NRCan plays an important role not just in its direct support of technology advancement, but also in terms of enabling this advancement through stakeholder engagement, building stakeholder capacity to conduct research, supporting codes and standards development, and generating knowledge to inform policy development and decision-making.

While evidence of longer-term outcomes, including commercial adoption, environmental benefits (e.g. GHG reductions) and economic impacts is limited, several technology areas demonstrate promise such as electric vehicles, renewables, buildings and reducing diesel use in the North. Moreover, as indicated by interviewees and as supported by the literature, the timeline from early stage research to commercialization is typically long and therefore it is too early for the realization of long-term results. Additionally, case study evidence illustrates how consistent and long-term support is a key factor in advancing innovation. For more advanced clean energy technologies, external factors such as low oil prices and unfavourable regulatory environments have hindered more widespread adoption.

The literature on energy innovation indicates that the process is unpredictable and does not progress in a linear fashion from early stage research to commercialization. Indeed, knowledge created during the various stages of innovation can be used to inform thinking and decisions in previous or next stages of the process. Therefore, measures of success for an innovation program have to consider not only how technologies themselves advance, but also how the knowledge that is generated through the RD&D process is used.

The program has a good performance measurement system that tracks a variety of outputs and outcomes related to knowledge, codes and standards development and technology advancement. However, there were some gaps in outcome measurement. For example, there were no systematic measures of knowledge uptake and use, which means program managers did not get consistent or complete feedback on influence of EIP projects. In addition, the follow-on reporting format and structure for contribution recipients has not generated the most relevant data.

The evaluation found good indications that both ecoEII and EIP were well-managed programs. For example, several lessons learned from both ecoEII have been incorporated into EIP going forward. Improvements to proposal solicitation, evaluation and selection over time were noted as well as certain design changes. Some challenges persist - such as the periodic 'call' approach used, funding rules and program timelines (which contributes to lapsing and reprofiling of funds), and repayability clauses. Some of these issues are related to Treasury Board processes pertaining to sunset funding. Moreover, EIP has recently received ongoing funding, which is expected to improve NRCan capacity to respond to these issues.

Interviews and workload records suggested that the workload involved with higher than anticipated letters of interest, applications and projects may have produced a time and effort burden on staff causing less attention to elements that were not urgent but still important – such as performance measurement and monitoring, reflection, information dissemination, and strategy development.

Document and interview evidence show several positive elements of strategic planning such as regular stakeholder consultations and environmental and technology scans. However, there were some reported issues, which were attributed in large part to a lack of ongoing funding for energy innovation, a situation that has recently changed with ongoing funding allocated to energy innovation. Strategic planning was reported to be particularly problematic during the start-up of EIP (2016). The 2016 Budget for EIP, which allocated funding to three program areas, did not according to interviews, fully reflect OERD’s long-term needs. Evaluation evidence also suggests that an enhanced strategic planning approach, one in which more systematically incorporates consideration of policy and systems readiness is required, particularly for more advanced technologies. The evaluation also found that planning documents sometimes contained unrealistic, unclear or inadequate performance targets.

Recommendations

Recommendations have been made to address the findings and lessons learned from the review of ecoEII and EIP over the 2014-15 to 2017-18 time period with a view to improving NRCan energy innovation programming going forward. In general the evaluation found that EIP is a well managed, relevant and effective program requiring some enhancements to its planning, communications, design, and performance measurement system.

Recommendations	Management Action Plan	Responsible Manager / Target Date
1: In light of new core funding for EIP, OERD should strengthen its strategic planning process:	Management agrees Actions:
Consider a broader systems based approach to planning for each technology area	1a. In 2018-19 OERD developed a new strategic planning framework centred around objective-based missions that focus RD&D activities to address specific technology gaps to meet government of Canada priorities using a systems based approach. OERD and IETS labs have already used the framework to develop their integrated 3-year work plan (2019-20 to 2022-23) for internal research and development programs. Currently the framework is informing the planning and design of the relaunch of Energy Innovation Program in 2019-20.	DG, OERD Completed March 31, 2019
Set clear and realistic strategy, targets and milestones	1b. As part of the new strategic planning framework, work plans include a clear delineation of how each project contributes to one or more missions, as well as, targets and milestones that will enable tracking of results.	DG, OERD March 31, 2020
2: OERD should communicate the role of EIP to external stakeholders to clarify its role	Management agrees. Actions:
	2. OERD is currently preparing to launch the revamped Energy Innovation Program during FY 2019-20. As part of the relaunch, OERD is working with NRCan’s Communications and Portfolio Sector to develop a new communications plan and suite of products and tools to help clarify the objectives, scope and focus of EIP for external and internal stakeholders.	DG, OERD March 31, 2020
	OERD will continue to collaborate with its federal government partners, particularly the Clean Growth Hub, to ensure that intended stakeholders and target groups for EIP and other OERD administered programs are able to find, understand, and have the information they need to apply for the appropriate program.	DG, OERD March 31, 2020
3: OERD should further enhance elements of its program design and delivery as follows:	Management agrees Actions:
Consider new, more flexible models of funding intake, especially given more stability in funding allocations.	3a. OERD has taken steps to integrate new, flexible funding models and calls for proposals into its program design. The launch of the revamped Energy Innovation Program will incorporate effective practices and lessons from OERD programs, such as Clean Growth Program and Impact Canada Clean Tech Initiative. These could include accelerated approval processes through trusted partnerships; enhanced collaboration through the Science and Technology Assistance for Cleantech (STAC) model, and/or innovation prizes and challenges.	DG, OERD March 31, 2020
Working with Treasury Board to eliminate contribution repayment clauses and to identify and implement strategies to minimize funding lapses and reprofiling of funds.	3b. OERD strongly concurs with the recommendation to remove the repayability requirement. OERD will continue to pursue this course of action, working with NRCan’s Corporate Management and Services Sector and the Treasury Board Secretariat regarding the elimination of the contribution repayment clause from OERD programs. Furthermore, OERD will continue to make representation to Finance regarding appropriate funding profiles for innovation programs (i.e. less funding in start up years).	DG, OERD March 31, 2020
4: OERD should improve its performance measurement system:	Management agrees. Actions:	Phase I: DG, OERD June, 2020
Strengthen its performance measurement processes (e.g. metrics, data collection, analysis) to ensure systematic monitoring and reporting of other aspects of effectiveness (e.g. codes and standards, uptake of information products, intellectual property, etc.)	4a. OERD will take a phased approach to enhance its performance measurement processes. Phase I will include a review of performance metrics and the development of new tools and methods for data collection and analysis. By June 2020, OERD will review its performance indicators and metrics as part of a broader NRCan and Innovation and Energy Technology Sector exercise to review performance measurement frameworks (please see 4b). Additionally, in Phase II, which will occur over the next two years, OERD will develop, test and implement methods and tools (2-4) to measure uptake and diffusion of IP, information and influence on codes and standards, as well as, measure the long-term outcomes of OERD investments in energy innovation programs. OERD is also exploring opportunities to automate data collection methods and analysis that will enable more systematic monitoring and reporting of performance information.	Phase II: DG, OERD March 31, 2021
Streamline reporting requirements (e.g. identifying and monitoring the most critical information, using appropriate format and structure for follow-on contribution recipient reporting)	4b. OERD is working with other branches in the Innovation and Energy Technology Sector and relevant groups within NRCan to review performance measurement frameworks and reporting requirements (ex. Departmental Results Framework, Performance Information Profiles) in an effort to reduce and streamline reporting requirements. Additionally, OERD is streamlining reporting requirements for proponents as part of the new strategic framework. OERD is also revising its structure for follow-on recipient reporting. Further, the automation of data collection and adoption of new performance measurement methods and tools will improve the quality of performance data, while reducing the reporting burden for proponents.	DG, OERD March 31, 2020

INTRODUCTION

Program Information

This report represents an evaluation of the Energy Innovation Program (EIP) including the last three years of the ecoENERGY Innovation Initiative (ecoEII) based on a review of program documents, project files and supporting literature as well as select interviews with NRCan staff and sector- case studies.

This evaluation covers two research, development and demonstration (RD&D) programs – the ecoENERGY Innovation Initiative (ecoEII) and the subsequent Energy Innovation Program that builds upon the ecoEII. The delivery of ecoEII and EIP involves a combination of non-federal government organizations, such as technology producers, industry, and academia, and internal R&D conducted by federal Canmet laboratories. The majority of the funding is allocated to external organizations via contribution agreements.

The ecoENERGY Innovation Initiative funded under the Clean Air Agenda (CAA), which sunset on March 31, 2016, was designed to improve environmental performance and economic competitiveness and productivity in Canada’s energy sector. In particular, it intended to address the need for reducing the risk and cost of new, clean energy technologies by supporting efforts to bring ‘yet-to-be-proven’ technologies from the conceptual stage to the ‘ready-to-be-tested’ stage of development. Note that ecoEII was extended for one year (until March 2017) to facilitate project completion, although no new funding was allocated to that year.

The ecoEII supports activities related to knowledge and technology development in five technology areas:

• energy efficiency in buildings and communities, industry and transportation;
• clean electricity and renewables;
• bio-energy;
• electrification of transportation; and
• unconventional oil and gas, e.g., shale gas, oil sands.

The Energy Innovation Program was launched in April 2016 as an interim two-year renewal, pending development of the Pan Canadian Framework (PCF). This interim renewal had three components: Clean Energy Technology Innovation (CETI) ($82.4M) (e.g. renewables, smart grid, reducing diesel use in northern and remote communities; reducing GHGs in the building sector, and carbon capture, use and storage; Oil and Gas Clean Technology (OGCT) ($50M); and Electric Vehicle Infrastructure Demonstration (EVID), Phase 1 ($46.1M).

In Budget 2017 EIP was renewed, receiving a total of $211.6M over four years and then ongoing funding of $52.9m per year. The EIP supports research, development and demonstration (RD&D) of clean energy technologies to produce and use energy more cleanly and efficiently. The program aims to deliver a coordinated suite of RD&D activities to address specific innovation needs in PCF sector mitigation strategies for electricity, built environment, transportation and industry in order to meet Canada’s 2030 GHG target, and provide deeper, ongoing GHG reductions post-2030.

Evaluation Objectives, Scope and Methodology

The objective of this evaluation was to assess the relevance (i.e., continued need and appropriateness of federal role) and performance (i.e., effectiveness, efficiency and economy) of the EIP, including the last three years of the ecoEII. The assessment of effectiveness includes consideration of the extent to which the current EIP is designed and delivered in a way that should facilitate or optimize outcome achievement.

The evaluation covers the Department’s direct program spending on the ecoEII and EIP programs over the period of 2014-15 through 2017-18 and up to June 2018 and $265M in expenditures. The emphasis of this report is to draw from (i) past ecoEII performance as established from general document reviews, interviews and case study work; and (ii) information on the recent roll-out of the EIP to address important design and delivery lessons.

While the evaluation covers both contribution funding and internal ecoEII and EIP Federal laboratory R&D, most of the evaluation’s emphasis is on the contribution funding due to its higher materiality.

The evaluation includes a mix of approaches, including an outcome / impact assessment of the ecoEII (2014-15 to 2016-17) and a formative / implementation assessment of the EIP (2016-17 to 2017-18, and up to June 2018). The evaluation utilized the following methods:

• Sixty-seven (67) interviews with internal (28) and external stakeholders (39). These interviews included more general key informant interviews and interviews conducted as part of the case studies.
• Case studies covering the following sectors/themes:
  • Buildings (focus on Net Zero Energy)
  • Smart Grid (focus on grid integration of Renewable Energy Technologies)
  • Oil and Gas
  • Transportation (Electric Vehicles)
  • Renewables (Alternative Energy – Marine, Wind, Geothermal, Solar)
  • Northern and Remote Communities
• ecoEII and EIP program document review (over 300 documents)
• 30 randomly selected ecoEII projects (25 external and 5 internal laboratory projects) in which available completion/project status reports and outcome reports were reviewed. A stratified, random selection process was used in which projects were randomly selected from each of the ecoEII program components.

The case studies are based on a specific theme or technology area (as opposed to one or two specific projects). To assess progress over a longer timeframe and to examine the connections of ecoEII to its predecessor program – Clean Energy Fund (CEF) – previous related CEF projects were also included in the case studies. While each case study assessed several ecoEII and EIP projects (covered a total of 22 ecoEII and EIP projects), they also included broader sector-based document and literature reviews, project documents as well as interviews with a range of stakeholders including those that had broader sector level knowledge. Case study themes were selected in accordance with the key components / technology areas of the ecoEII and EIP programs.

Challenges and Mitigation Strategies

A key challenge in conducting this evaluation was ensuring coverage of the broad range of technology areas and program components contained within both ecoEII and EIP. The theme-based case study approach was utilized to optimize coverage of these components. In addition, the document review covered the key program planning and performance documents for each technology area. Finally, a file review of ecoEII projects (that were not part of the case studies) were conducted to ensure coverage was sufficiently broad.

FINDINGS

Relevance

To what extent is there an ongoing need for the EIP?

Ongoing Need

The EIP has the key objective of reducing GHG emissions in the four key priority areas identified in the Pan-Canadian Framework: electricity, built environment (e.g. buildings), transportation and industry, which are amongst the highest GHG emitters in Canada (National Inventory Report, 1990-2016, ECCC). Various international agencies such as the International Energy Agency (IEA) and The International Renewable Energy Agency (IRENA) note that innovation will continue to play a crucial role in decarbonising the energy sector. The literature on clean energy show technology and innovation gaps exist for energy supply (oil and gas production, power plants, and electric grids) and demand, (buildings, transportation, industry) (International Energy Agency, Tracking Clean Energy Innovation Progress, 2018).

A review of documents (e.g. Expert Panel on the State of Science and Technology and Industrial Research in Canada, 2018; Energy Innovation Roundtables Report, 2014) and interview evidence indicate that the marketplace on its own will not invest at sufficient levels to meet Pan Canadian Framework goals due to the high capital investments and long timelines required to advance technology. Furthermore, end-use demand for clean energy is not sufficient at present. Therefore, government assistance is needed to support high risk, long-term efforts to advance energy innovation.

Energy RD&D as part of overall strategy

Interviews, case studies and documents indicate that RD&D, as part of a coordinated strategy, is needed to optimally address environmental issues. NRCan planning documents, quoting international research reports, note that, given the Government of Canada's recent orientation to address commitments related to reduced GHGs and other goals, a Pan Canadian Framework would advise that carbon pricing should be complemented by directly funded assistance for technology advancement to be able to affect desired change. EIP funding fits directly as part of this Government of Canada strategy.

Using a comprehensive approach, NRCan’s energy innovation programming aims to move the system forward. The document review and case studies show that NRCan has engaged a variety of stakeholders – regulators, industry and provincial and territorial governments. It supports various innovation activities – technology advancement, improved information for decision-makers, and codes and standards development. NRCan has also made efforts to better coordinate innovation programming, within the department, through the newer Clean Growth Program, launched in 2017-18, which covers RD&D in three sectors: energy, mining and forestry.

Case study and document evidence suggest good cooperation between provinces/territories and NRCan in several technology areas. At the federal level, efforts are underway for a more coordinated ‘whole of government’ approach to innovation, which includes initiatives such as the recently launched Clean Growth Hub. Results of these efforts are expected to be realized over time and are not within the scope of this evaluation.

Both internal and external interviewees noted that even without the supporting regulatory environment, clean energy technology RD&D, given that it is a long-term effort, continues to be needed to address technology gaps and to inform regulatory and policy issues. The literature review also shows that the relative importance of technology push (RD&D) and market pull (regulations, policy, codes and standards) varies with a number of factors such as the characteristics of industry, and the nature of interactions and networks. Technology push tends to be more important in early stages and demand pull in later stages. Moreover, incremental innovations may depend on demand pull while far-reaching innovations may require a more concerted technology push approach.

Needs by Technology Area / Sector

Case studies conducted across theme areas (buildings, smart grid, clean oil and gas, electric vehicles, north, and renewables) examined the relevance and performance of ecoEII and EIP in broad areas. The case studies generally conclude that there is an ongoing need for innovation support to these areas. They support federal policy objectives related to reduced GHG emissions, increased economic activity and improved energy security. These technology areas vary in terms of the stages of technology readiness, types of stakeholders, and regulatory environment suggesting the need for a customized approach to program design.

The case studies (e.g. buildings, electric vehicles, smart grid) also highlight the important role that codes and standards play in technology development and adoption. For example, coordinated standards and regulations to advance smart grid development are needed across provinces, territories and internationally to support grid integration and emerging international market opportunities for Canadian smart grid technology companies. As noted in the case of buildings research and past evaluation studies, codes and standards can increase homebuilder, developer and buyer awareness of (and confidence in) new energy efficient products and construction methods.

High subscription rate for ecoEII and EIP

Historically there has been a high subscription rate for energy R&D programs such as those managed by IETS (PERD, CEF, ecoEnergy Technology Initiative, and ecoEII). The high subscription rate for these programs is not necessarily indicative of need. Interview and document evidence indicate that there may be a number of factors influencing the high subscription rate for these programs. However, according to interviews and project documentation there were a substantial number of qualified unfunded EIP (e.g. 2017 EIP) proposals.

OERD launched the ecoEII Demonstration Program and External R&D Program in August 2011. In response, 238 Letters of Interest (LOIs) for demonstration projects and 437 LOIs for R&D were received. Table 1 below illustrates the ecoEII review process for 2011. As noted in the ecoEII 2011-12 Annual Report, the volume of applications received was unprecedented for OERD.

Table 1: ecoEII Review Process

	ecoEII Demonstration	ecoEII External R&D
Letters of Interest	238	437 (149 invited to submit proposal)
Full Project Proposals ($ Value)	131 ($878.6M)	112 ($178 M)
Funded Projects ($ Value)	23 ($59 M)	44 ($58.5 M)
Subscription Rate ($ requested: $ available)	15:1	3:1

Source: ecoENERGY Innovation Initiative Contribution Annual Report 2011-12

The Clean Energy Innovation Program (CEIP), a component of EIP, call for proposals in 2016 generated 346 proposals requesting approximately $700 million in funding. A total of 35 projects with a total funding envelope of $25 million were funded. The distribution of proposals by technology area is shown in Table 2 below. It illustrates that the highest number of proposals (and levels of funding requested) came from the renewable, smart grid and storage systems and buildings technology areas. It is also interesting to note that 25% of proposals for methane and volatile organic compounds (VOC) reduction were successful (few proposals but high success rate) vs. about 5% for renewables and smart grid.

Table 2: Distribution of 2016 EIP- Clean Energy Innovation External Proposals by Technology Area

Technology Area	Proposal Count	Funding Requested (million)	Number Funded	Funding Allocated (million)
Renewable, smart grid and storage systems	117	$213.2	10	$12.4
Buildings (Net Zero Energy communities, commercial buildings, housing and building design tools)	84	$162.0	8	$9.6
Industry emission reductions	43	$109.7	2	$3.9
Carbon Capture, use and storage	36	$92.2	5	$9.4
Methane and Volatile Organic Compounds (VOC) reduction	37	$57.1	10	$8.0
Northern (reducing diesel use in remote communities)	29	$77.2	2	$4.2
Total	346	$711.4	37	$47.4

Note: Total funding envelope was $49.1 million. The ~$1.6M variance reflects a reallocation of funds to other areas (e.g. EVID) to support program needs. Source: EIP Master Tracking Sheet (OERD) May 4, 2018. Energy Innovation Program – Clean Energy Innovation. ADM Committee on Energy S&T. January 18, 2017.

To what extent is EIP consistent with federal priorities?

EIP is consistent with Canada’s commitments to reduce GHG emissions. Canada ratified the Paris Agreement on Climate Change in October 2016, reiterating a commitment to reduce GHG emissions by 30% from 2005 by 2030. To meet the 2030 target, additional mitigation measures are needed including efforts to develop and demonstrate clean technologies that reduce greenhouse gas (GHG) emissions from energy production and use, which account for about 80% of Canada’s emissions (according to NRCan planning documents) and in 2016, following consultations, First Ministers adopted the Pan-Canadian Framework (PCF) on Clean Growth and Climate Change, on December 9, 2016.

EIP represents an important element, in conjunction with other strategies (e.g. regulatory), to reduce GHG emissions. These connect to the broader PFC established in 2016. The PFC has four main pillars: pricing carbon pollution; complementary measures to further reduce emissions across the economy; measures to adapt to the impacts of climate change and build resilience; and actions to accelerate innovation, support clean technology, and create jobs.

To what extent is the role of the federal government/ NRCan appropriate?

In general, the role of NRCan in conducting and managing energy research, development and demonstration (RD&D) is considered by stakeholders to be appropriate given NRCan’s expertise and experience in delivering these types of programs. A history of NRCan clean energy funded programs for energy innovation have created a legacy of trust which select case studies have shown has resulted in NRCan acting as a lead for others to invest – among both government agencies and private investors. In addition, document review, interviews and project case analysis suggest that NRCan has built up the required credibility and expertise to play an ‘honest broker’ role at a vital Pan-Canadian level.

The recent creation of several new innovation programs dispersed among several departments and agencies was noted by some external stakeholders to have contributed to confusion as to NRCan’s role vis-à-vis other federal government players delivering energy innovation programs.

The Natural Resources Act gives the Minister various responsibilities related to the sustainable development of natural resources, including development and promotion of scientific technologies, and the Energy Efficiency Act provides NRCan with authority to conduct research that promotes efficient use of energy and the use of alternative energy sources.

According to case study and interview evidence, government has an important role to play in reducing the risk associated with innovative technologies, including: supporting early-stage R&D and technology demonstrations that enable further de-risking and market uptake. A review of the international literature suggests that the involvement of national governments in the development and evolution of energy-related systems including RD&D, which include a full spectrum from established institutions to start-up technology companies is an important function.

The EIP components provide a mechanism to demonstrate federal leadership in areas pertaining to the Canadian Energy Strategy (CES) (2015). Under the CES, federal, provincial and territorial governments have agreed to collaborate on technology and innovation, including in the areas of small-scale electricity generation, energy storage, electrification of transport, renewable fuels, carbon capture use and storage and reducing reliance on diesel in remote communities.

Although there is no “one best model” for government intervention, analysis indicates that R&D tax credits often do not incent incremental innovation. Indirect R&D incentives are generally non-discriminatory and market-led, direct support may be better suited to targeting specific priorities and addressing market failures, such as combatting climate change and stimulating early-stage innovation (Drummond et al., 2015). Some studies indicate that tax incentives tend to support what would have happened anyway. Therefore, it is more effective for governments to directly support targeted technological advances (e.g., Mazzucato 2013). Furthermore, according to analysts (e.g. Bonvillian and Weiss Technological Innovation in Legacy Sectors, 2015) energy is an area which needs innovation, but has in it traditional sectors which tend to have accumulated unwarranted subsidies and price structures that ignore, for example, environmental externality costs.

Canada as compared to most other OECD countries has in the past spent less on direct support for R&D. Nearly 90% of government investments that support business expenditure on R&D in Canada are indirect, in contrast to 30% for the OECD countries as a whole (OECD 2015). Moreover, Canadian R&D as a share of gross domestic product (GDP) steadily declined in Canada from 2001 to 2015. It is also one of the few OECD countries that had no growth in total national R&D expenditures between 2006 to 2015 (Expert Panel on the State of Science and Technology and Industrial Research and Development in Canada, 2018). While declining business R&D expenditures are a key driver of this trend, government R&D expenditures also declined over the same period. Since 2016, however, Canadian government investments in RD&D investments have increased significantly in support of Mission Innovation commitments.

Performance – Effectiveness

To what extent did the ecoEII achieve its intended outcomes?

ecoEII Performance and Outcomes

The ecoEII and EIP Results Logic Model is shown in Appendix 1. Measures of program reach and collaborative outputs/products include program support for innovative RD&D projects (e.g., contribution agreements, leverage from collaborators) and information products to support knowledge and technology sharing (e.g., technical reports, presentations, codes and standards, patents).

Expected immediate outcomes are:

• Increased investment by stakeholders in energy RD&D;
• Increased collaboration between academia, industry and the public sector for RD&D of energy technologies; and
• Increased access to scientific and technical knowledge related to energy technologies.

Expected intermediate outcomes are:

• R&D projects move emerging technologies closer to commercial readiness;
• Increased use and adoption of clean energy (limited evidence to date); and
• R&D informs policy, regulations, codes and standards.

Expected Long-term outcomes:

• Environmental benefits from technology and/or new policies, codes and standards; and
• Economic benefits from clean energy technology RD&D projects.

Table 3 shows early impacts for case studies (focused on specific technology or theme areas).

Table 3: Case Study Outcomes

Outcomes	Buildings (Net Zero Energy)	Smart Grid	Electric Vehicle	Northern Communities	Marine Renewable
Immediate Outcomes
Increased investment
Increased collaboration
Increased access to scientific and technical knowledge*	P	P	P	P	P
Intermediate Outcomes
R&D projects move emerging technologies closer to commercial readiness					P
Increased use and adoption of clean energy	P	P	P	P	P
R&D informs policy, regulations, codes and standards				P
Ultimate Outcomes
Environmental benefits from technology and / or new policies, codes and standards at sector level	P	P	–	P	–
Economic benefits from clean energy technology RD&D	P	P	–	–	–

Key:
= Cases clearly demonstrated
P = Partially demonstrated (for intermediate and ultimate outcome levels, partial achievement indicates good progress at the project level, but not broader impact)
? = Not clear from evidence
– = Not demonstrated

*All of the increased access to scientific and technical knowledge noted partial due to limited tracking (i.e. Outputs are counted but not influence.)

Effective Innovation

A review of the literature on successful energy innovation indicates that there is a significant time lag from early stage research (e.g. Nemet, G.F., Zipperer, V., and Kraus, M., Energy Policy, 2018). Moreover, the innovation process is unpredictable and does not progress in a linear fashion to commercialization. Indeed, as depicted in Figure 1, knowledge created during the various stages of innovation can be used to inform thinking and decisions with respect to further research. For example, a demonstration project can highlight the need for earlier stage research. Innovation is also cumulative - the more the knowledge base grows, the better the capacity to address barriers to technology adoption (e.g. high costs of technology). Therefore, measures of success for an innovation program have to consider not only how technologies themselves advance, but also how the knowledge generated through the RD&D process is shared and used. In addition, there are a number of enabling factors, (that are part of ecoEII and EIP) such as collaboration networks, capacity building of key stakeholders to conduct and use research and to make informed decisions, and codes and standards development, which are critical to support the advancement of innovation.

Figure 1: Iterative Innovation Process

Text version

Figure 1 shows the iterative innovation process. The steps of this process are as follows: research, development, demonstration, deployment, and commercialization. The graphic shows arrows going forward and backwards from research to commercialization and from commercialization to research. For example a demonstration project may show that more research is needed in a specific area. This is in contrast to the more traditional view of innovation, which has innovation progressing in a linear fashion from research to commercialization. The graphic also shows that technology push is stronger at the research and development stages and that market pull is stronger at the deployment and commercialization stages.

Immediate Outcomes

Increased Collaboration

Effective collaboration is widely viewed as an important condition for successful innovation (International Renewable Energy Agency. Renewable Energy Innovation Policy: Success Criteria and Strategies). Case study and document evidence indicate overall good progress with respect to enhanced collaboration. Note that the evaluation focused on the adequacy of collaboration (e.g. engagement with appropriate groups of stakeholders such as the private sector) rather than on whether collaboration has ‘increased’ – something that is difficult to measure without adequate baselines. In addition, it may not be the most meaningful measure of effective collaboration.

Over 230 collaborations were in place across the 72 ecoEII external RD&D projects, for an average of 3 per project (Source: ecoEII 2017-18 Annual Report). OERD program and project managers noted that this number may be underestimated as it was calculated using the highest number of collaborations in a single year (i.e., it is not sum of unique collaborators per project). Academic institutions led 36% of the ecoEII external R&D projects, followed by the private sector (30%) and industry associations and networks (26%). A review of ecoEII collaborators list (from Project Status Reports) shows a diverse group of stakeholders for external RD&D: industry, industry and standards associations, utilities, provincial and territorial governments, non-government organizations (NGOs) and international organizations.

Table 4 provides a summary of the program reach and indicates that the program is engaging various target groups. The table illustrates that industry and academia are the most frequent type of collaborators with respect to R&D and not surprisingly, the private sector is the most common type with respect to demonstration projects.

Table 4: Distribution of ecoEII Projects, by Lead Organization

	Private Sector	Utility	Academic Institution	Industry Assoc. / Research Networks	Communities	Other	Total
External R&D	16	1	19	14Footnote 1	1	2	53
Demonstrations	13	2		1	1 (Province)		17
Total	29	3	19	15	2	2	70

Source: ecoEII Project Summaries, NRCan website.

Industry involvement in EIP appears to be significant as shown in Table 5. For the three EIP components, private sector companies led the majority (70%) of projects. The EIP Master Tracking spreadsheet provides a count of the number of collaborators per project (showing an average of 3.2 collaborators per project) but does not provide information on the type of collaborators.

Table 5: Distribution of EIP Projects, by Lead Organization

	Private Sector	Utility	Academic Institution	Industry Assoc / Research / Networks	Communities	Total
Electric Vehicle Infrastructure Demonstration	7	1	1		1	10
Oil and Gas	6			2		8
Clean Energy Investment Program (CEIP)	24	2	4	2	4	36
Total	37	3	5	4	5	54

Source: Innovation and Clean Growth Contribution Programs – Midyear Report 2017-18. OERD.

Case studies and interview findings reveal the extent and importance of collaboration in each clean energy technology area supported. For areas such as marine tidal wave energy development, smart grid technologies and electric vehicles cases revealed strong collaborative networks among different communities which have helped to establish important science in these areas, as well as codes and standards.

External interviewees reported new collaborations among sector players (e.g., private sector technology providers, utilities, end-users) and improved cooperation and consultations with stakeholders, including community groups (Indigenous and others), provincial governments, and codes and standards organizations. Case study and file review evidence also show appropriate and effective collaborations within each of the projects reviewed.

There is case study evidence that the EIP has adapted its offerings to reach potential users. For example, small and medium-sized enterprises (SMEs) who may have been under-represented in earlier programs appear to be better represented in current processes. Case studies involving small companies suggested that their efforts had been better received in more recent iterations. For example, funding for Front End Engineering and Design (FEED) studies helped new program proponents (SMEs) access funding for the required up-front project design issues such as permitting, community consultations (especially important in the north) site selection, and consideration of local knowledge. Case studies showed that FEED studies allowed SMEs to be funded during early project development stages, i.e., support for activities required before developing and submitting a proposal for a demonstration project.

The program approach to built environment projects encouraged multi-disciplinary project teams. Owens Corning Canada’s Net Zero Home (NZE) Home demonstration project involved a team of experts in the fields of building technology, building science, solar technology, and energy efficiency, each with experience mentoring production builders to adopt voluntary energy efficiency/renewable energy measures.

NZEH buildABILITY Corporation was the lead consultant managing the project, with participation by five building science and technical consulting companies. Five homebuilders in four provinces (Alberta, Ontario, Quebec and Nova Scotia) built a total of 26 net zero energy and net zero energy ready homes in four communities, the largest NZE community demonstration in Canada to date. In addition, according to interviewees, Owens Corning Canada’s NZE Home demonstration project contributed to the development of the ad-hoc Canadian Home Builder’s Association New Zero Energy Housing Council. The project’s participating builders and partners were founding and sponsor members of the Council whose mission it is to help launch the first private Net Zero Energy certification for housing.

Increased Investment and Leverage

Between 2011-12 and 2016-17 ecoEII leveraged significant investment from external sources. Overall, for every dollar (contribution funding to recipients) provided, $1.50 was provided by others, with industry providing almost half of the overall funding. As shown in Figure 2, OERD contributed 30% of total demonstration project costs, with industry participants providing the majority of funding at 60% (10% provided by universities, other government sources, etc.). For every $1 of ecoEII investment in demonstration projects the program leveraged $2.32 from other sources. This is slightly higher than ecoEII’s predecessor program, the Clean Energy Fund, in which for every $1 of CEF funding for smaller demonstration projects, an additional $2.10 was invested in the projects by others.

Figure 2 – ecoEII Demonstration Project Funding by Source

Text version

Figure 2 is a pie chart that shows the sources of funding for ecoEII demonstration projects. OERD, NRCan provided $40.1 million, other federal departments and provincial / territorial governments provided $11.5 million, and industry provided the largest portion of funding - $80.3 million. For every $1 NRCan spent, others (industry, government) provided $2.32.

With respect to external R&D projects OERD contributed 50% of total costs and industry 38%. External funders provided $69.2 million for a leverage of 1:1 (Figure 3). Other includes universities and NGOs.

Figure 3 – ecoEII External R&D Project Funding by Source

Text version

Figure 3 is a pie chart that shows the sources of funding for ecoEII external research and development projects. In this case OERD, NRCan provided half of the overall funding which amounted to $68.9 million. Industry provided $52.4 million, other federal government and provincial / territorial governments provided $6.6 million, and universities provided $8.2 million. Non-governmental organizations and others provided $2.1 million. Therefore, for every dollar NRCan provided other stakeholders provided one dollar.

Note the term ‘leverage’ infers that NRCan (Government of Canada) money induces investment from other governments and the private sector. While there has been no independent validation that it was NRCan’s ‘first move’ investment that induced others to invest, case studies suggested that NRCan support has played a particularly important role in leading the way. For example, NRCan has played a leadership role in the following circumstances:

• where there are dominating risk averse actors (e.g. smart grid) as well as special conditions affecting system capacity (e.g. North); and
• in supporting codes and standards development (codes and standards are significant enabler of adoption of innovation).

In addition, the strong technical assessment and credibility of NRCan energy innovation program review and acceptance has enabled other groups with less technical capacity to provide funding in several cases. For example, several regional developments agencies have followed ecoEII or EIP funding with their own support to help create hubs or centres of expertise in certain relevant regions.

A number of external respondents pointed to the fact that the Federal Government initiation and consistent support in some areas has led to support by provincial governments. This pattern was confirmed through case studies conducted in the sector areas. For example, the AddEnergie demonstration project (EV charging stations and IT management system) received additional support by the province of Quebec through the Bureau de l’efficacite et de l’innovation energetique (TechnoClimat). The South Coast British Columbia Transportation Authority (TransLink) project received substantial funding in addition to EIP through the Greater Vancouver Regional Fund. Marine energy centres of excellence have been assisted by regional development agencies on both coasts subsequent to ecoEII funding.

RD&D Government-Industry-Academia Capacity Building

ecoEII and EIP have supported the development of new capacity for RD&D. These programs led to the development of critical technology development platforms, new research facilities and equipment. Early and on-going support for these platforms enhances the capacity of industry and others to develop clean energy technologies (e.g. components, systems integration). For example, prior to 2011 there was little funding or capacity for smart grid RD&D in Canada. According to case study evidence, beginning with the Clean Energy Fund (predecessor to ecoEII) and continuing with ecoEII, NRCan investments led to increased awareness, understanding and capacity for smart grid technology development in Canada (both private and public sector). A demand response test bench at NRCan’s Varennes Canmet lab was developed to support future proof of concept and standardization.

In addition, ecoEII projects have also supported high quality personnel (HQP) development. According to ecoEII performance database (2011-12 to 2015-16), 904 academic HQP were supported through internal R&D (275 academic HQP) and external RD&D (629). Moreover, 397 professional HQP were supported during the timeframe (348 through Federal laboratory R&D and 49 professional HQP through external RD&D).

Increased access to scientific and technical knowledge

Overall, the program has produced numerous scientific and technical products as shown in Table 6. Moreover, a project document review (randomly selected) indicated that information, knowledge and data was a key output for many ecoEII projects (e.g. monitoring data for Net Zero Energy and energy efficient buildings, properties of oil sands tailings, purity guidelines for demonstration carbon capture and storage (CCS) projects, Source books for Net Zero Energy Solar Buildings). However, the uptake, use or impact of these types of knowledge products was not systematically tracked. Therefore, the extent of access and use of many of these products cannot be assessed.

By the end 2016-17, as indicated in Table 6, the ecoEII G&C Program had generated 1,790 scientific knowledge products (or an average of 25 products per project), one web page per project and up to five NRCan news stories per project.

Table 6: Summary of ecoEII Knowledge Products, by Theme

Knowledge Products Generation	Patents	Licenses	Codes & Standards	Peer-reviewed Publications	Presentations	Technical Reports
Total ecoEII	21	5	25	59	330	204
R&D	17	3	23	54	253	184
Demonstrations	4	2	2	5	77	20
Bioenergy	-	-	-	8	21	30
Clean Coal &Carbon Capture and Storage	6	1	-	31	76	49
Energy Efficient Buildings and Communities	2	-	-	3	20	13
Renewables	6	2	13	17	100	54
Smart Grid	10	1	3	3	91	33
Unconventional Oil and Gas	-	-	-	7	37	34
Transportation	1	1	9	14	16	19

Source: ecoEII Project Summary Report, OERD. July 2016.

ecoEII introduced a requirement for a public report and a summary fact sheet for each project. These documents are posted online (NRCan website and some companies provide links from their corporate websites) and are a key element of the program’s dissemination activities. Case studies and document review indicate that project results were typically shared through publications and other mechanisms such as conferences. The case studies highlighted some best practice examples of information sharing and dissemination. For example, the knowledge transfer component of the Owens Corning Net Zero Energy (NZE) housing project was an integral and funded part of the project plan (one of the primary project objectives was to act as a platform for other builders to promote the adoption of NZE housing concepts).

Nevertheless, information dissemination is one area where several project proponents and NRCan representatives indicated that greater efforts are needed to promote and share findings with relevant stakeholders. Across all technology areas, there is little to no funding allocated to projects directly, or to OERD, to support results sharing and dissemination.

Advancements in Technology

ecoEII began using Technology Readiness Levels (TRLs) as a performance measure in 2013-14. The Technology Readiness Level scale is a standard methodology is a scale from 1 to 9 that helps to categorize a technology’s maturity. TRL stages 1-3 correspond with earlier stage research, TRL stages 4-6 with development and stages 7-9 with scaling-up and demonstration of a proven commercialization-ready technology (TRL 7-9). TRL statistics are available for 46 of the 72 ecoEII projects. Note that the TRL scale is not relevant to all projects. Several ecoEII projects encompass related science activities (RSAs) and such as knowledge generation to support codes and standards or policy development. Moreover, as illustrated by the file review, knowledge generation is a critical part of RD&D projects.

By the end of the program, ecoEII reported a weighted average of 0.82 TRLs advanced per year. According to program documentation, the unweighted average TRL’s advanced for the entire program was 2.34. The majority of projects funded under ecoEII started in TRL 1 – 4 or as Related Science Activity (RSA). By the end of the program, the majority of projects were in TRLs 3, 5 and 6. Data also showed that projects starting between TRL’s 1-3 were on average able to advance by more TRL’s than those started at TRL 4-7, indicating that later stages of RD&D takes more time (as is typical with R&D according to the literature review) than earlier stages as shown in Figure 4.

Figure 4: Average advancement of ecoEII Technology Readiness Levels

On average demonstration projects started at a TRL of 4.4 and advanced approximately 2 TRL levels prior to project completion. External R&D projects began at a lower, average TRL level (3) and also advanced, on average, by 2 TRL levels (to 5.1) by the end of the project.

For ecoEII, proponents provided estimates of their project’s TRL at the beginning of the project, throughout the project and at project completion. This process was continued with EIP, though according to interviews – a closer NRCan review has been maintained on the validity of TRL changes. A review of TRL assessments in selected ecoEII cases versus project descriptions and progress reports shows that there was sometimes a significant disconnect between reported TRLs and actual project progress. Reported level of confidence by NRCan and other delivery partners in these TRL assessments is relatively low.

While TRLs show technology advancement of NRCan projects, it does not capture progress of subsequent projects financed by other sources. Several project proponents noted that ecoEII projects helped to obtain funding from other sources to further scale-up their technologies. For example, CO2 Solutions Inc., a company that received funding from ecoEII to demonstrate a patented process for carbon capture, subsequently obtained financing from SDTC and the province of Quebec for large pre-commercial and demonstration projects.

Increased use and adoption of clean energy

Increased adoption of clean energy requires a convergence of technology supply and demand, which is affected by a number of factors including costs, perceived risk (which can be mitigated by codes and standards), regulations, etc. While there were few examples of widespread adoption of the clean energy technologies supported by ecoEII; case studies and interviews revealed some promising evidence of future use.

The smart grid case study highlights the complex environment in which smart grid advancement occurs. A key aspect of a smart grid is automation intended to achieve one or more of the following benefits - increase grid resilience, environmental performance, and operational efficiencies (Canadian Electricity Association). Provinces and territories have responsibility for electricity grids, and each jurisdiction has different levels of interest in, and support for, smart grid and related systems research (e.g., charging stations, storage). Each province or territory approach differs according to its assets and need, but the more active regions (e.g. Ontario, B.C.) are responding to economic drivers coupled with quality of service and environmental drivers. A number of factors contribute to the differences among provinces including energy mix (hydro, oil, gas, coal) and grid ownership.

Smart meters are used to replace the traditional approach of manual meter reading and their deployment is generally regarded as the first stage of smart grid development. In 2014, approximately 67% of meters were smart meters as compared to 2018 when about 82% of meters in Canada were classified as smart meters (Smart Grid in Canada Reports, 2014 and 2018). This upward trend is directly the result of provincial smart meter programs. However, interview and document evidence suggest that ecoEII internal and external RD&D funding have played both an enabling role and a more direct role in smart grid development as a whole. CanmetENERGY energy through the establishment of networks (for example CanmetENERGY established the Canadian Smart Grid Action Network to connect national stakeholders) and ecoEII smart grid demonstrations which have focused on electric vehicle integration, microgrid installations, and grid monitoring and automation.

Industry interviewees commented on the impact of early, ecoEII microgrid projects on the level of understanding and interest in smart grid applications. All industry interviewees reported an increased interest in smart grid / microgrid technologies over the past 8 to 10 years. In some cases, the interest is international. For example, according to interview findings, Hatch’s (industry contribution recipient) small scale, isolated power systems expertise and equipment are of interest to remote communities in the Caribbean, Indonesia and Africa.

In the building sector, the focus was on advancing the commoditization of net zero energy homes (e.g. buildings that can be constructed using commercially available technologies, which therefore limits the need for additional standards-related activity). Technologies were limited to pre-engineered products and systems (e.g. versus custom mechanical systems), which helped to ensure that the technologies could be more accurately modelled and provided assurance with regard to their long-term performance in the Canadian climate. ecoEII project results demonstrated that an R-2000 home can achieve net zero energy performance using off-the-shelf technologies and methods. The project demonstrated the technical feasibility of building Net Zero Energy Housing Communities (with 26 buildings across Ontario, Quebec, Nova Scotia, and Alberta).

Canada generates its majority share of electricity from renewable sources, mostly hydro generation. Approximately 60% of Canadian electricity came from hydropower in 2015, typically from large facilities with reservoirs.

Grid connected and overall renewable energy capacity has experienced growth over the past 10 years, although the total installed capacity remains low. Wind power capacity in Canada increased twenty-fold between 2005 to 2015, with installations across the country. In 2018, the total installed wind capacity was approximately 12.7 GW. It is estimated that about 1 GW of wind capacity is interconnected to the distribution system (CanWEA and internal NRCan reporting, 2018). Solar is a small component of Canada’s electricity (where it represented 5% of total capacity in 2015) and virtually all capacity is installed in Ontario, due to its past incentive program.

The intermittency of wind and solar generation has been a key technical challenge for widespread adoption. A number of ecoEII projects were directed towards this challenge (e.g. RD&D for grid connected battery storage, smart grid technologies, demonstration of EV grid infrastructure, energy storage systems for tidal generation, ecoEII internal R&D for built environment thermal storage, etc.).

Other renewable technologies, such as offshore wind, tidal power and geothermal energy, have not experienced significant uptake in Canada, but have demonstrated potential. Offshore wind projects are proposed on Canada’s west and east coasts, and a 20 MW tidal power facility already exists in Nova Scotia. The Pan Canadian Wind Integration Study – the largest integrated wind study in North America as of its completion and supported by a contribution of almost $1.8 million by ecoEII - showed that wind energy generation systems are showing increasing potential to play a significant role in future electricity grid management. Geothermal projects are also being considered in isolated northern communities, which according to document and interview evidence would benefit from combined heat and power. An EIP demonstration project, located in Saskatchewan, has been launched (project is underway and completed first geothermal test well in 2019) and is expected to be Canada’s first geothermal power plant.

With respect to electric vehicles, case study evidence suggests that several ecoEII projects have provided a foundation for continuing efforts in the demonstration and deployment of electric vehicle infrastructure. According to external interviewees and documents reviewed, ecoEII projects (e.g. Electric Mobility Adoption and Prediction study, ADDEnergie’s EV infrastructure demonstration) contributed to the following: an increased understanding of needs of charging station owners and drivers; identification of areas where further code development is required; fostering of relationships between utilities and other stakeholders; and improved management and IT systems for electric vehicle charging station networks.

ecoEII demonstration projects resulted in the installation of 1200 Level 2 chargers^{Footnote 1} (700 in Quebec, 100 in Ontario and 400 in BC) and 60 fast charge stations (Level 3). The ecoEII demonstration project, 2011 to 2016, (AddÉnergie Technologies Inc. proponent) resulted in the installation of 600 Level 2 chargers and 5 fast charge stations serving diverse market segments. Charging stations were installed in a variety of locations including shopping centres and residential areas. The installation of this infrastructure aimed to demonstrate the Charging Station Central Network Management System, a cloud based software system designed to manage the charging stations.

However, interviewees noted that for at the public chargers have been underutilized and they typically have not been recovering their costs as anticipated. However, an increased Return on investment (ROI) is expected by project proponents as the demand for electric vehicles continues to increase.

The underutilization of electric vehicle charging infrastructure is not unique to Canada. Other jurisdictions also grapple with the ‘chicken or egg’ dilemma: To demonstrate and install infrastructure to increase demand or to build infrastructure in response to demand. The International Energy Agency (IEA) EV Global Outlook Report (2018) notes that even though the frequency of use of publicly accessible charging infrastructure is fairly low in comparison with private installations, public chargers are an important component of the EV charging infrastructure. The IEA advocates a variety of policies including the direct investment for increasing the availability of public EV charging infrastructure to facilitate the adoption of electric vehicles.

The Electric Vehicle Infrastructure Demonstration (EVID) Program, Phase 1, funded 10 EIP demonstration projects aimed at addressing barriers to the implementation of charging infrastructure in multi-unit residential buildings, along the Trans-Canada Highway, in urban centers and at workplaces. A few projects consisted of installation of overhead charging stations in several municipalities to allow for displacement of diesel buses with electric buses and to improve interoperability of electric bus charging infrastructure. The TransLink project (Vancouver) is the farthest along and in 2018 a two and a half year integration trial of fast-charging buses and charging stations was launched. Four bus and equipment manufacturers involved in the project have agreed to innovate and re-design their systems to establish an interoperable model for electric bus charging infrastructure. Data will be collected on the different technologies to better plan the transition towards a zero-emission bus fleet.

The EVID program (Phase 1 and 2) was launched as part of the Green Infrastructure stream of the Investing in Canada Plan (IICP), a broader federal initiative focused on rehabilitation and modernization of various types of infrastructure in Canada. The objective of the EVID program is to fund demonstrations of next generation and innovative electric vehicle charging infrastructure projects to accelerate market entry and address hurdles to the deployment of EV charging infrastructure.

Some external key informants expressed concerns that the EVID program signaled a migration away from basic R&D for battery development when there are remaining research gaps in energy storage technology (e.g. advanced chemistry batteries). While the ongoing funding for EIP supports clean energy R&D and small scale demonstrations in transportation/vehicle technologies^{Footnote 2}, this finding highlights EIP’s important role in supporting early and later stage RD&D that is responsive to challenges, opportunities and innovation gaps in vehicle technologies, as part of its strategic planning exercises.

R&D informs policy, regulations, codes and standards.

The deployment of clean energy technologies often face significant non-technical barriers. Federal support for codes and standards development is important for addressing these barriers and are critical to support the deployment of technologies. In established technology areas existing policies, codes and standards provide a framework for the development or amendments needed to support the deployment of new technologies or further use of existing technologies to meet more stringent performance targets (e.g., GHG emission reductions, increased use of renewable energy, etc.).

Document and case study evidence show that ecoEII R&D is being used to inform codes and standards development across many technology areas. According to the ecoEII database 25 codes and standards have been informed by ecoEII R&D. However, the impact of R&D on policy and regulations is not consistently tracked so the extent of achievement of this outcome with respect to informing policies and decision-making could not be fully assessed.

ecoEII external and internal Canmet lab work is contributing to new Code for existing building alterations by 2022 and more stringent model codes by 2020. Adoption by Provinces and Territories of NZE ready codes is targeted for 2030. NRCan Canmet laboratory was done with respect to IEA SHC task 40 – Towards Net-Zero Energy solar Buildings. The labs also participate in the IEA Large Solar Systems Group and the Heat Pump Program. In addition to EIP funding, the federal government allocated $62 million over 8 years through the Green Infrastructure Program to meet this goal. The Owens Corning project involved the entire building supply chain and the lessons learned and performance data from monitoring energy use in the demonstration buildings will support the development of a NZE ready building code.

The Canadian Standards Association Group ecoEII project delivered a series of safety standards and protocols related to the supply, charging and storage of electricity for electric vehicles. This work helped to inform the 2015 and 2018 versions of the Canadian Electric Code, which is expected to result in safer use of electric vehicle equipment. Project documentation also indicated better alignment of the Canadian Electric Code with the U.S National Electrical Code.

A marine sector – test centre for codes and standards was developed and Nova Scotia has announced regulations and legislation based on the results of an ecoEII project. According to case interviews – ecoEII and EIP funded projects led the way to establishing world class codes and standards.

Cost-effective methods and standards for handling tailings and standards for methane gas flaring risks helped to inform policy and standards, clearly attributable to both internal and external ecoEII and EIP work with the oil and gas sector according to interviews and documents.

The results of the trials at the Low Carbon Fuel Demonstration Pilot Plant (Lafarge project proponent) were used as the basis for applications to the Ontario Ministry of the Environment and Climate Change for the permanent use of low carbon fuels at the Bath plant and could be used by other Cement Industry companies. Ontario Regulation Alternative Low-Carbon Fuels (ALCF) Reg. 79/15 was influenced by research, modelling, and testing of low carbon fuels at Lafarge’s Bath Cement Plant.

Environmental / Economic benefits from technology, new policies, codes and standards

While the program is progressing towards achievement of its long-term outcomes, to date there are limited widespread environmental and economic benefits from ecoEII and EIP, although economic and environmental benefits have been observed at the project level. Given the stage of technology development in the targeted sectors, all lines of inquiry confirmed that it is too soon to see any longer-term, sustained environmental and economic benefits. Moreover, there are significant external factors (e.g. low price of oil) that are impeding widespread adoption of more advanced technologies. The evaluation, however, did identify cases of economic and environmental benefits at the project level and several cases, based on documents and interviews, with good potential to advance.

The smart grid and electric vehicles case studies show that advancement is occurring with the emergence or growth of new companies such as Opus One, Tugliq, and AddÉnergie as well as the development of new operating groups within existing companies such as Hatch Ltd (for microgrid controllers). Tugliq was created to manage the ecoEII project at the Raglan mine site in Northern Quebec. The company continues to operate and pursue remote community smart grid / microgrid projects. Towards the end of the ecoEII project (2016), AddÉnergie was selected to provide charging solutions to the province of Quebec’s Electric Circuit Project, a major endeavor, which is expected to result in the addition of 1,500 public charging stations. Project proponents indicated that the ecoEII projects helped build their capacity and improved their credibility making it easier to solicit other collaborators and market their innovations and expertise.

Based on interview and document review findings, remote communities across the north stand to benefit from smart grid projects. Also Hatch Ltd. (an industry ecoEII contribution recipient that developed a controller for remote microgrids) has identified replication potential in 20 large mining sites in Northern Ontario, as well as offshore export opportunities. Hatch continues to improve the HμGrid design and pursue opportunities to commercialize and export the controller.  As a result, this project may contribute to Canadian competitiveness by supporting the development of products that can be used worldwide in similar hybrid renewable energy-diesel installations. Adoption of this technology should have impacts on energy costs, GHG emissions, and air quality in northern and remote communities that are reliant on diesel generation. The next steps for Hatch Ltd. include commercialization efforts (including export markets), pilot project implementation in select Canadian communities, and continued R&D to further develop the technology.

While Canada has not met its targets to date with respect to electric vehicle stock, there are good indications that there will be strong expected growth in the Electric Vehicle North American market. Canada is well below its target with respect to electric vehicle stock (500,000 electric vehicles on the road by 2018) as compared to the approximately 50,000 electric vehicles on the road. With respect to the proportion of electric cars sold relative to internal combustion cars, Canada ranks ninth in the critical metric of market share—with only a 0.59% share in 2017 (Clean Energy Canada Report EVs, 2017).

However, there have been small and steady increases in sales with more significant increases occurring in 2016, 2017, and 2018. Electric car sales in Canada increased from approximately 7,000 in 2015 to 17,000 in 2017 (International Energy Agency (IEA), EV Global Outlook, 2018). A number of organizations (for example, IEA, OPEC, Bloomberg) are forecasting significant electric vehicle growth (including North America) in the next 10 to 20 years. According to IEA’s Global Electric Vehicle Outlook Report (2018), electric vehicles, worldwide, will grow from 3 million to 125 by 2030. One of the reasons for this growth is the substantial decrease in costs of batteries with continuing decreases expected (J.P. Morgan. Driving into 2025: The Future of Electric Vehicles).

Some projects reviewed as part of the case studies also reported environmental and GHG reductions. The Glencore Raglan Mine Smart Grid pilot demonstration project began operation in January 2016. The 3 MW wind turbine achieved 97.3% availability and displaced 3.4 million litres of diesel over an 18-month period and reduced GHG emissions by an estimated 5.8 kilo tonnes. According to an industry stakeholder, the storage system / technology was the most significant clean technology adopted during this project. Storage and grid control technologies are the key determinants of renewable or alternative energy projects.

There were some project examples (e.g. Wasdell Falls Hydro Power Project and Tugliq’s Glencore RAGLAN mine renewable electricity smart-grid pilot demonstration, AddÉnergie Technologies’ commercial demonstration of a management system for electric vehicle charging stations) of job creation outcomes (temporarily for construction of the demonstration projects as well as more sustained employment for ongoing operations).

The Low Carbon Fuel Demonstration Pilot Plant (Lafarge project proponent) reported 70% GHG reductions in their site operations (estimate based on a Life Cycle Assessment - proponent entered values and calculated values using various conversion factors based on Queen’s University research). The objective of this project was to demonstrate the viability of lower carbon fuels, as a partial replacement (up to 10%) for the coal/petroleum coke mix used at Lafarge’s Bath, Ontario plant.  This Project built on the previous use of purpose-grown biomass at the plant but used more economical low carbon fuels, namely waste materials such as railway ties, construction and demolition waste, shingles, and paper sludge. The technology has been adopted at the Lafarge Exshaw plant to date. According to an academic interviewee, if this technology is fully deployed in other spin off projects, the Canadian industry can reduce its CO2 emissions by over 1 million tonnes annually.

External Factors influencing achievement of ecoEII intended outcomes

The following evidence and observations have been gleaned from a review of documents, including past reviews, post-mortem lessons learned sessions, interviews and case study findings. Please note that a discussion of internal factors will be included in the Efficiency section of the report. Technology specific context is important. Case Studies have shown that different sectors face different weightings of external and to some extent internal factors (discussed in a later section). They also vary in development times – depending on the technology life cycle maturity.

Economic factors directly affect investment: The price of fossil fuel has not only directly impacted oil sands development, but has also made alternative energy sources less competitive. Figure 5 illustrates the magnitude of the problem and the unfortunate timing for many ecoEII projects wherein projects which were approved under oil prices of $100 per barrel fell by more than half by the time the projects were coming to completion and commercialization was being contemplated. The timelines for three oil and gas case study projects as indicated in Figure 2 (Tetraseis, CO2 Solutions and Saltworks) have been shown here to illustrate. However the price drop for fossil fuels was also noted as affecting alternative energy investments (since oil became relative cheap as an energy source).

Figure 5: Oil Price Impact

Text version

Figure 5 shows the price of oil from 2010 to 2018. Oil and Gas external eco-EII cases examined were started when oil prices were over $100 per barrel (2011-12). These projects were completed when oil prices were less than half that price (around 2016). The same can be said for essentially all the other ecoEII cases which occurred over similar time periods.

A text box reads: Oil and Gas external eco-EII cases examined were started when oil prices were over $100 per barrel. They were completed when oil prices were less than half that price. The same can be said for essentially all the other ecoEII cases as well.

• Political factors affect concerted policies:   These factors are closely linked to economic factors. The politics of Canadian federalism is such that there must be consistent F/P/T support to move innovations – especially through the final stages of readiness where systems are more ‘fixed’. For example, smart grid development and electric vehicles have advanced more in those provinces with a supportive market and policy environment.
• Regulations: The extent to which existing federal, provincial or municipal requirements influence Canadian companies (or individual end-users) decisions to invest in clean energy technologies were found to factor into adoption.  For the Oil and Gas sector, expected clean water and effluent requirements were not imposed, making both Saltworks and CO₂ Solutions cleaning technologies much less compelling to potential buyers and users. The absence of a Clean Fuel Standard was also noted to hinder production and sales of next generation biofuels.
• Stakeholder / intermediary / end-user capacity: In the Tugliq energy project in the north, the company benefited from the engineering and operating expertise and capacity of a local mining company.  This was essential for project implementation.
• Looking forward at the EIP and judging from case evidence, the following external factors are likely to particularly influence the EIP design and / or success:
• Evolving policy environment: The new Pan Canadian Framework puts a greater emphasis on buildings, transportation and northern and remote community support. The PCF relies, as described, on a ‘whole of government approach’.
• Evolving program environment: The introduction of clean growth, energy and innovation challenges and the harmonization of programs in 2017 may have an influence on OERD Program delivery in terms of helping the program to link to a broader outcomes vision. On the other hand, what is perceived by external stakeholder to be a confusing array of federal energy innovations programs, may negatively affect potential.
• Stage of technology development: Some technologies, which had been supported in the past by NRCan R&D and small-scale demo programs, are now at the large-scale demonstration and deployment stage. This implies the need for different types of support, timelines, funding profiles, partnerships/collaborations, etc.

Performance – Efficiency and Economy

To what extent is the program design (e.g., strategic planning, contribution agreement design etc.) the most efficient and economic means of producing the intended outputs and achieving the intended outcomes?

Several past assessments of ecoEII performance were reviewed – including an assessment by KPMG in 2014.). As concluded in that review, and more recently completed central agency reviews (e.g., internal and Auditor General audits) the OERD delivery process has been recognized as essentially sound, with appropriate, systems procedures, and controls – albeit in some need for improvement in information management and reporting.

Internal Factors influencing Performance

Internal factors are related to strategic planning, program design and delivery, and financial restrictions. For the purposes of this report, these will be considered internal to the Government of Canada. The following factors emerged from both documents reviewed, case studies and interviews and unless otherwise noted, occurred across different technology areas.

Consistent, long-term approach

Case study, interview and document evidence suggest a number of internal factors that have positively contributed to efficiency and effectiveness. For example, the case studies highlight how ecoEII research links to and builds upon the previous program – the Clean Energy Fund (CEF). NRCan’s consistent and long-term support has also been a positive influencing factor - which according to interview and document evidence, is necessary to facilitate the strengthening of collaborations, codes and standards development and technology advancements. Figure 4 below shows the timelines of NRCan’s ecoEII key projects and activities related to marine tidal achievements. The marine case study highlights that NRCan Clean Energy Fund and ecoEII support has informed codes and standards; increased capacity (e.g. a hub for innovation in wave energy in Canada, contributed to testing and research infrastructure) to conduct relevant tidal power research. This in turn has contributed to (along with provincial incentives) a number of marine technology deployments (many in progress) in the Bay of Fundy.

Several projects reviewed highlight the need for patient capital and a long-term perspective regarding project impacts. For example, an early Borealis Geopower FEED study (2011), according to NRCan and industry stakeholders, contributed to several important interim outcomes that benefited the EIP funded Sustainaville demonstration project, which commenced in 2017. The demonstration geothermal turbines are expected to provide geothermal power to greenhouses, hot pools and commercial buildings in Valemont, B.C. The initial FEED study is also expected to support other future geothermal power projects (e.g., issuing process for drilling permits, development of geothermal project review and approval processes).

Figure 6: Marine Tidal Timelines and Achievements – Broad NRCan Approach

Program Continuous Improvement Practices

Several lessons learned from both ecoEII and the Clean Energy Fund have been incorporated into EIP going forward. Improvements to contribution agreement assistance, proposal solicitation, evaluation and selection over time were noted as well as certain design changes.

While the evaluation could not fully assess the impact of more recent EIP and other program changes, case study evidence provides some indications of impact with respect to improved EIP contribution assistance and capacity building activities. For example, these improvements helped to mitigate challenges associated with remote and northern community projects.

Lessons learned / observations regarding contribution agreement assistance include the following:

• Improved Strategies for developing and supporting EIP northern and remote community projects: There are significant inherent difficulties to working in the North. The need for special capacity considerations is apparent. The case study of Tuglic, a Northern project, showed how these challenges have been mitigated by the extra time, attention and flexibility taken by project managers to support a sustainable energy system in a remote community. Other strategies to better support these types of projects include greater support for project proponents at the proposal stage; and, the addition of program elements to address capacity building in subsequent programs (e.g., Energy Innovation Program, Green Infrastructure Program).
• Graduated assistance and development: The early ecoEII projects were incorporated into EIP; for example, there is more graduated applications through the use of online tools and letters of interest, more funding for capacity building, support for proposal writing, flexible reporting (i.e. ability to report through the interview process).
• Moves towards better integration of areas: The results of a recent OEE-DOE study were integrated into the Green Infrastructure buildings deployment program (the GI program recognizes the codes and standards work done by OEE, along with OERD’s RD&D activities).
• Clean Growth Hub: The Hub aims to improve access and service to companies and researchers navigating federal clean technology programs and to strengthen coordination and tracking of results.

Improvements to proposal solicitation, evaluation and selection over time were noted through interviews and document reviews including the following points:

• OERD introduced an Expression of Interest (EOI) phase in the ecoEII R&D and demonstration programs. This has been used to help refine proposals.
• New to EIP, OERD will provide a feedback process at the proposal stage. This step is intended to balance the playing field for smaller companies, new applicants and those unfamiliar with federal funding programs.
• Technology area working groups is now using external academic reviewers.
• Within NRCan there are ongoing efforts to write more targeted calls for proposals. Documents show that the results of this effort are starting to show in more recently launched programs including Green Infrastructure and Impact Canada.
• Front End Engineering and Design Studies are being used to help determine project scope and requirements with respect to technology, costs, time and regulatory requirements.
• Project Standby Lists allowing increased flexibility to fund projects which may had initially failed to receive funding in the application process.
• Implementation of the Technology Readiness Level scale to assess technology advancement of projects.

Funding Rules and Program Timelines

As is the case with most technology development / RD&D programs, the time required for proponents to assemble a project team, complete a rigorous project plan and finalize a project budget coupled with the time required by NRCan to review project proposals is not built into the front end of most programs. This front-end delay can mean projects are not approved for many months. A number of interviewees commented on the negative impact that funding rules have on technology development program implementation. This was especially relevant in the case of the early EIP program (which included three one to two-year programs) and in the first year of ecoEII. This caused uncertainty as well as a rush to deliver – putting particular burden on smaller applicants. For example, in 2015-16 the ecoEII program was extended for one year (with no additional funding) to allow recipients to complete their projects. At the end of FY2015/16 only 12 (of 72) projects had been completed. By the end of 2016-17, 72% of ecoEII projects had completed all project activities, submitted their final reports, and project completion holdbacks were released; however, 20 projects remained uncompleted. (They were completed in 2017-18.) Continuing to track and monitor these projects’ outputs and reports places additional demands on OERD staff (and according to ecoEII Annual Report is not accounted for by Program-level resource allocations).

In 2016-17 short-term funding was approved for EIP. This funding created some process constraints – as staff struggled to accommodate large unexpected submission calls. An internal post mortem review suggested that the 2016 call had generated too many unqualified submissions. In addition, program staff in some cases, given the timelines, had to reallocate funds in short order – risking suboptimal investments and reach in 2016-17. Interview evidence suggests that tight commitment restrictions provide insufficient flexibility – given the inevitable delays in spending experienced in innovation projects - can lead to funding lapses or the need for shifting funding to manage the resulting budget issues.

Overall planned expenditures (G&Cs, O&M and salaries and employee benefit plan) for ecoEII and EIP from 2014-15 to 2017-18 were $265.8M. Total actual expenditures for this period were $268.6M. Figure 4 shows planned and actual expenditures from 2014-15. 2017-18 shows a particularly large variance in planned vs actual funds – approximately $17M. A large part of this variance is the result of the Electric Vehicle Initiative Demonstration (EVID) program. It saw the need to re-profile $13 million (in 2018-19) of the program’s budget as potential recipients were unable to launch projects in time to meet the program schedule. The higher ecoEII actual expenditures as compared to planned expenditures (2014-15 and 2015-16) reflects some reprofiling of funds to those years. In the early years (2012-13) ecoEII lapsed $23.7M (not reprofiled) given that RD&D projects took longer to negotiate than had been accounted for in planning phases of the program.

Figure 7: Planned vs. Actual ecoEII and EIP Expenditures

Text version

Figure 7 shows the planned and actual expenditures for ecoEII and EIP from 2014-15 to 2017-18. IN 2014-15 the planned expenditures for ecoEII were $55.68 million and the actual expenditures were $68.27. In 2015-16 the planned expenditures for ecoEII were $42.39 million and the actual expenditures were $46.21. In 2016-17 the planned expenditures for EIP were $34.46 million and the actual expenditures were $37.88 million. In 2017-18 the planned expenditures for EIP were $133.28 million and the actual expenditures were $116.27 million.

Four-year program design and periodic ‘call’ for interest

In fast developing technology sectors, rather than using a periodic call for proposals every four years, several interviewees (NRCan managers), in alignment with the approach used by DOE, suggested that it would be more efficient and effective to select projects on a continual (annual basis).^{Footnote 3} This approach, they contend, would give NRCan the ability to respond more efficiently to the quickly developing technologies and emerging needs that characterize the energy innovation sector. In addition, several NRCan interviewees suggested that an annual project review process would help address capacity issues with respect to project selection.

Repayability Clauses

According to program managers and documents, the time spent by OERD to review company financial reporting, vet the accuracy of the data and determine repayability is warranted as disproportionate to potential financial benefits. While specific time reporting has not been captured, the process flowcharting of these calculations and the assertions of all officers who have been involved in this process, when compared to the amount of actual repayments garnered, suggests that the burden is unreasonably high. Interviews also suggested that specific applicants have walked away from funding negotiations because of the arduous repayability determination components.

During the last round of EIP negotiations with central agencies OERD argued against including repayability requirements, noting that these program obligations are not consistent (in the majority of cases) with the type of R&D projects eligible for funding (i.e., stage of technology development, expected time to market, etc.), end up costing NRCan time and money in terms of tracking, and negatively affects relations with industry clients. It is notable that other technology development programs such as the Industrial Research Assistance Program (IRAP) does not require repayability.

Collaboration Challenges – Internal NRCan laboratories and external stakeholders

Interviews and documentary evidence suggest that funding allocation rules had significantly restricted collaborations between NRCan labs and external stakeholders. According to consultations with IETS, these collaboration barriers have recently been reduced through engagement with TB officials and other NRCan representatives.

NRCan Capacity for Proposal Review and Project Selection, Support for Up-front Delivery and Information Dissemination

Several program staff interviews and supporting evidence in documents indicate there is insufficient capacity for a number of activities including for proposal review and project selection. The number of proposals submitted for these programs placed an increasing burden on internal capacity. Several NRCan interviewees noted that workload demands left less time for strategy development, project monitoring and dissemination activities.

The Horizontal Framework for Clean Growth and Climate Change Activity Attestation Form notes 13 Scientific and Technical FTEs; 6 Finance and Admin and 7 Energy Innovation Policy FTEs. The number of programs and projects, the need to monitor projects five years past completion and to leverage the program investment by disseminating project findings as broadly as possible place significant demands on a limited number of staff. The letters of interest – already considered large at 238 (ecoEII 2011) and 346 (proposals in 2016) have now gone to over 700 under Clean Growth in 2018. Further, as shown in Table 9, the caseload in terms of management of externally funded projects has more than quadrupled between 2015 to 2018.

Table 9: FTE vs. the Growth in Number of Projects Managed

FISCAL YEAR	OERD FTE's*		External Funding (Vote 10)		Total # new external projects		Total # of existing external projects being managed		Sum of new and existing external projects	Project Reviews
2015-16	53		56,326		46		34		80
2016-17	56		65,992		10		78		88
2017-18	71		107,745		54		113		167
2018-19 (as of Oct. 25, 2018)	88		142,091		191		183		374	Total # of existing projects (183**)

*Excludes IETS labs, OGDs and students
** includes grants that haven’t been closed, repayable and non-repayable projects requiring monitoring and reporting (revenue and outcome reports) and projects that we are still funding from previous programs.

Analysis points:

1) Agreements are also getting financially more versatile/complex, with various holdback criteria and terms, amendments, innovations, etc. which increases workload and processing time
2) The reporting burden on multiple fronts is increasing; more detailed information under very tight timelines is sought consistently by numerous parties which puts further pressure on staff and management

Internal laboratory reliance on non-core funding

The reduced core funding of internal laboratories over the time period under review, as documented by a 2015 audit and other NRCan documents, has created uncertainty about funding in some laboratory-connected researchers. NRCan documents show that in 2014-15, CANMET energy labs operated with 24% of their overall budget covered by stable A-Base funding. Fifty-five percent (55%) of overall funding is allocated on a competitive basis through the Program for Energy Research and Development (33%) (PERD) and C-Base programs such as the ecoENERGY Innovation Initiative (22%). Twenty-one percent (21%) of funding comes from Vote Net Revenue - performing research services for non-federal organizations such as the private sector.

While there is some variability across labs, A-Base (or core) funding generally is only able to cover a small portion of core operating expenses (e.g. operation and maintenance, core salaries for lab management and operations). This was noted in documents and interviews to hinder planning, particularly HR planning, and retaining talent. In addition, in the view of some the inadequacy of core funding creates a risk of bias in project selection towards discretionary projects under ecoEII and EIP being chosen to support existing laboratory staff research interests as opposed to priority sector needs. The detailed analysis to confirm this perception is beyond the scope of this study, however interviews suggest that this resourcing factor may have caused both uncertainty and some biases in the system. Recent process adjustments such as the appointment of fairness monitors may reduce this perception.

In addition, OERD is implementing (2018-19) a new approach to streamline federal R&D programs (including PERD and EIP internal laboratory research), which includes a revamping of the governance and funding structure and provides stable, annual funding for CanmetENERGY labs (NRCan documentation). Interview and document evidence suggests that better alignment of internal laboratory work with strategic priorities is required. As part of the revised approach mentioned above, an integrated 3-year work planning process aligned with strategic priorities is being implemented.

Strategic Planning

NRCan interviewees and document review identified a number of good strategic planning practices. These include regular external stakeholder consultation, environmental and technology scans (e.g. Renewable Energy Technology Scan for Canada, 2015). However, NRCan interviewees noted that the absence of ongoing funding, while no longer the case, served to impede long-term strategic planning. Challenges pertaining to the 2016 EIP program planning process, were noted by interviews and as part of the document review. The 2016 Budget for EIP which allocated funding to three program areas did not, according to interviews, fully reflect / align with OERD’s long-term needs (i.e., funding allocations were not commensurate with needs or program timelines). Program managers suggested that a greater percentage of overall funding should have been given to ongoing programs (making it possible to design and deliver to meet longer-term R&D needs).

In general, both documents reviewed and interviews suggested that NRCan facilitated roadmaps provided a strong opportunity for NRCan programs to be relevant to Canadian sector needs. For the planning periods of EIP fewer (as compared to ecoEII) NRCan guided program-sector specific road maps were assembled. Interview and case study evidence suggests that industry led Roadmaps do not tend to reflect broader interests and a more balanced analysis of the sector.

While recent consultations with smart grid stakeholders have informed NRCan’s current activities, there is no overarching Smart Grid Framework that identifies technology, policy, regulatory issues or gaps and shows how these could be addressed through federal R&D, F/P/T funding programs or by industry. A majority of smart grid interviewees (public and private sector), believe that a current, comprehensive roadmap, that takes into consideration the specific provincial / territorial needs, priorities and existing grid characteristics could be used to target and prioritize future project selection.

An assessment of NRCan EIP planned results and targets (as contained in foundational NRCan documents and Portfolio Strategic Plans) revealed that there is a need for more meaningful and clear targets. For example, a comparison of EIP targets and United States, Department of Energy showed that the US targets were much more specific in terms of targeted outcome results. According to interview and document evidence, some of the sector roadmaps contain significantly overestimated planned results and targets.

New thinking related to energy policy suggests the need for a strengthened strategic planning process which includes a broader view of energy sector actors and systems that includes an assessment of commercial, policy or systems readiness, particularly when technologies are at more advanced stages of the continuum. Practically speaking this broader approach cannot be fully achieved without the cooperation and coordination of other federal government departments and agencies. There are a number of initiatives, not within the scope of this evaluation, currently underway, aimed at improving horizontal coordination of innovation programming at the federal level. Nevertheless, NRCan should consider more fully integrating a broader systems approach into its planning to the extent possible.

To what extent does the EIP collect performance information that supports the determination of effectiveness, efficiency and economy?

Interviews with OERD managers and project proponents identified the following issues concerning the extent to which performance information will support the determination of effectiveness, efficiency and economy:

• Scope: TRLs provide a measure of the rate of technology development / supply. To realize the longer-term program outcomes of economic and environmental benefits (e.g., GHG reductions) also requires market acceptance and commercialization. Measures of other factors, including the extent to which government policies, codes and standards, regulations and incentives support market development are also needed.

• Data quality and reliability: Case files reviewed and the overall project control spreadsheets (source: Cumulative ecoEII Master Expected Funding January 24) were compared to information found in project files and in some cases significant information products were not tracked / recorded and therefore these results were underestimated.

As previously mentioned, the TRLs, at least with respect to ecoEII, have reliability issues. Project proponents have provided an annual assessment of TRL progress throughout their project. In some cases OERD program managers validated the assessments, but in many cases – according to interviews - they did not. Looking at project-level reports, there are examples of technologies that achieved (limited) commercialization even though their TRL ratings were ‘middle of the spectrum’. This suggests under-rating. In other cases there are examples of extreme jumps (of 6 or 8 levels) which seem to have occurred in comparatively short time frames, suggesting over-rating.

• Data availability: NRCan has required project proponents to report back to NRCan for a period of five years following project completion, providing both financial data (required to assess repayability obligations) as well as outcomes / impacts attributed to the program. Compliance has been modest and the format and structure of the reporting (e.g. requiring written responses) has not generated the most useful information. According to interview evidence, the effort to improve reporting by implementing a small hold-back (1% per year for each of the five years) did not affect the number of reports submitted.

Conclusions and Recommendations

Conclusions

While ecoEII was generally successful, it, along with the start-up of EIP, faced transitional times in terms of government priorities, the innovation assistance landscape and the economic circumstances facing all targeted segments. Evaluation evidence showed that the objectives of the renewed EIP have been revised to reflect shifts in energy and economic policy and the energy innovation environment.

Case studies, interviews, and documents indicate that program reach and collaborations, leveraging, codes and standards informed by research, and technical progress have met or exceeded reasonable historical expectations. Indeed, the evidence suggests that NRCan plays an important role not just in its direct support of research and technology advancement, but also in terms of its enabling role – engaging with a variety of stakeholders, increasing capacity of key stakeholders to conduct research, providing technical input into codes and standards development, and generating knowledge to support policy and decision-making.

The evaluation found that EIP is generally a well-managed program, but that some enhancements are needed going forward. These include the following:

• There is a need to for a nuanced and broader approach to planning by engaging a broad range of stakeholders appropriate to each technology area, considering more fully policy and systems’ levels of readiness and by including clear yet realistic outcome targeting. There is no equivalent to the technology readiness level for policy or systems readiness being actively used by OERD at this time – though they are under development);
• Communicate the role of EIP to external stakeholders. New funding for clean energy among a variety of federal departments and agencies has contributed to stakeholder confusion as to the role of NRCan and others with respect to clean energy programming;
• Gaps in monitoring the take-up and use of knowledge and information; and
• Federal funding and collaboration restrictions with respect to internal R&D.

Recommendations

In light of these observations and conclusions, the following recommendations are made with a view to improving NRCan energy innovation programming going forward:

1. In light of new core funding for EIP, OERD should improve its strategic planning process:
   1. Consider a broader systems based approach to planning for each technology area
   2. Establish clear and meaningful strategy, targets and milestones
2. OERD should further communicate the role of EIP to external stakeholders to clarify its role.
3. OERD should further enhance elements of its program design/delivery as follows:
   1. Consider new, more flexible models of funding intake, especially given more stability in funding allocations.
   2. Work with Treasury Board to eliminate contribution repayment and to identify and implement strategies to minimize funding lapses and reprofiling
4. OERD should improve its performance measurement system:
   1. Strengthen its performance measurement processes (e.g. metrics, data collection, analysis) to ensure systematic monitoring and reporting of other aspects of effectiveness (e.g. codes and standards, uptake of information products, intellectual property, etc.)
   2. Streamline reporting requirements (e.g. identifying and monitoring the most critical information, using appropriate format and structure for follow-on contribution recipient reporting)

Appendix 1: ecoEII / EIP Logic Model

Text version

Appendix 1 shows the logic model for ecoEII and EIP. The logic model starts with the inputs: Federal funds; personnel; equipment and supplies; physical facilities; and recipient resources (e.g. leveraged funds). It then progresses to the programs’ activities which are as follows: program management, conduct efficient, effective, transparent project selection processes, establish energy research and development and demonstration initiatives and collaborations, encourage broad participation of ideas and technology proposals for clean energy innovation, and dissemination of clean energy knowledge and technology transfer products. The outputs are as follows: Innovative energy research, development and demonstration projects; and information products to support knowledge and technology. The immediate outcomes are as follows: Increased investment by stakeholders in energy RD&D projects, increased collaboration between academia, industry and the public sector, increased access to scientific and technical knowledge. The intermediate outcomes are as follows: Research, development and demonstration projects move emerging technologies closer to commercial readiness, increased use and adoption of clean technology, and research, development and demonstration from clean technologies informs policies, regulations, codes and standards. The ultimate outcomes are as follows: Environmental benefits from technology and/or new policies, codes and standards at sector level, and Economic benefits from clean energy technology RD&D projects.

---