Evaluation of the Electric Vehicle Infrastructure Demonstration (EVID) Program

Audit and Evaluation Branch
Natural Resources Canada
November 1, 2023

Table of Contents

• List of Acronyms
• Executive Summary
• Introduction
  • Program Profile
  • Evaluation Objectives, Methods, and Limitations
• What We Found: Relevance
  • Is the EVID Program focusing on relevant needs and priorities within the changing context of the program’s operating environment?
• What We Found: Effectiveness
  • To what extent has the EVID Program achieved its intended immediate outcome?
  • To what extent has the EVID Program achieved its intended immediate outcome?
  • To what extent has the EVID Program achieved its intended immediate outcome?
• What We Found: Efficiency & Economy
  • To what extent has the EVID Program been implemented and delivered as planned?
  • Is the EVID Program’s design an efficient and economical means of achieving the intended outcomes?
  • Does the EVID Program’s performance measurement strategy support the determination of program performance?
  • Are there any best practices, lessons learned, and improvements required related to the design of the program?
• Conclusion
• Appendix A: Evaluation Matrix
• Appendix B: Evaluation Team

List of Acronyms

Executive Summary

This report presents the findings, conclusions, and recommendations from the evaluation of the Electric Vehicle Infrastructure Demonstration (EVID) program. This evaluation responds to a commitment to the Treasury Board and adheres to section 42.1 of the Financial Administration Act.

The EVID program is a transfer payments program that supports the demonstration of next-generation and innovative electric vehicle (EV) charging and lower-carbon (LC) refuelling infrastructure. The program supports demonstration projects that seek to address technical and non-technical barriers and gaps to installing, operating, and managing next-generation and innovative EV charging and LC refuelling technologies and solutions. The program was allocated a total of $76.1 M over a total of six fiscal years (2016-17 to 2021-22). Contribution agreements were to be monitored until expected completion in 2023-24. During the evaluation period, the program approved 33 projects.

The approach and methodology used for the evaluation followed the TB Policy on Results (2016) and related Standards on Evaluation. The evaluation method included a document review, a literature review, and key informant interviews. The evaluation focused on the following issues from fiscal years 2016-17 to 2021-22:

1. The EVID program’s ability to appropriately address the evolving needs and priorities of the EV charging and LC refuelling sector (relevance).
2. The program’s achievement of the short-term outcomes, as well as trends towards the achievement of the longer-term outcomes (effectiveness).
3. The program’s capacity to operate as planned to achieve the intended outcomes, given the rapidly evolving context and the resources allocated (efficiency and economy).

What the Evaluation Found

Overall, the evaluation found that the EVID program was relevant, effective, efficient, and economical. Areas for improvement were identified and additional details regarding these findings as well as recommendations to management to remediate these issues are outlined in the report.

Relevance: The EVID program contributes to meeting stakeholder needs in the rapidly evolving ZEV sector in Canada. EV charging and LC refuelling infrastructure is a critical enabler in ZEV adoption. As Canada moves towards a low-carbon future, the needs and priorities in this area continue to exist. The program also aligns with the Government of Canada and Natural Resources Canada’s priorities, frameworks, and strategies. Federal support is essential for demonstrating next-generation and innovative EV charging and LC refuelling infrastructure. Without it, progress in electrification and decarbonisation in the transportation sector would be more technologically fractured and slower. Canada is one of the few countries that support the demonstration of technologies and solutions in this area.

Effectiveness: The EVID program has been effective in making progress towards its short-term outcomes, producing its planned outputs and meeting its immediate outcome. In particular, the program has selected 33 demonstration projects that are generating technologies, solutions, and knowledge to target key barriers and gaps in the sector. Vendor innovation, capital assets, learning and transferable skills, and valuable relationships have been formed by performing project tasks. While progress towards the immediate outcome is promising, it is still too early to assess and conclude on the program’s achievement of its intermediate and ultimate outcomes. Nonetheless, stakeholders generally perceived the program to be on the right track based on the data, technologies, and solutions generated thus far. The evaluation identified unintended outcomes associated with the ZEV sector. However, no negative unintended outcomes were attributed to the program. Equity, diversity, and inclusion (EDI) are important in the ZEV sector as not all groups have equitable access to ZEVs, ZEV infrastructure, and the associated benefits. While not initially designed for EDI, the program integrates EDI considerations in its operations and decisions, including the support of several projects that address EDI principles.

Efficiency and economy: The implementation and delivery of the EVID program were influenced by several internal factors and external factors, resulting in some deviation from the planned design. The program’s timeline and funding structure were the main challenges. The dynamic program context led to lower-than-planned spending in the first three years and annual reprofiling exercises. The COVID-19 pandemic caused numerous amendments due to delays in timelines and increased funding needs. While the program had limited control over these factors, program staff tried to mitigate them to the extent possible by being creative and adaptive. Despite these challenges, the program has largely remained successful. The program design is generally efficient and cost-effective, considering its risk tolerance level. Although the program is supported by on existing resources in the OERD, the program has been able to pivot and adapt to the evolving program operating context. The evaluation identified several strengths and limitations, with the lack of flexibility in the federal funding structure noted by key informants as the most significant limitation. While the program’s performance framework and indicators provide a basic understanding of program performance, the existing reporting requirements lack sufficient detail to provide a comprehensive understanding of the program’s impacts. The OERD is working on solutions to effectively measure the program outcomes. Proponents did not perceive the project reporting requirements as burdensome. However, they suggested that the program could clarify the requirements of the reporting templates and consider sensitivities when reporting on knowledge products. Additional suggestions for improvements, as well as lessons learned and best practices, are further explained in the report. Notably, proponents suggested that a recurring platform to facilitate discussion and collaboration among proponents and key stakeholders would add value to the program.

Recommendations and Management Response and Action Plan

Introduction

This report presents the findings, conclusions, and recommendations from the evaluation of Natural Resources Canada’s (NRCan’s) Electric Vehicle Infrastructure Demonstration (EVID) program. The Audit and Evaluation Branch (AEB) conducted this evaluation as part of the NRCan Integrated Audit and Evaluation Plan 2021-26. This evaluation responds to a commitment to the Treasury Board (TB) and adheres to section 42.1 of the Financial Administration Act.

Program Profile

Program Context and Rationale

The Intergovernmental Panel on Climate Change has concluded that achieving net-zero greenhouse gas (GHG) emissions by 2050 is key to keeping the rise in the global mean temperature to 1.5°C above pre-industrial levels (2014). For Canada to meet the 2030 target and net-zero GHG emissions by 2050, additional mitigation measures are required at all levels of government, as well as by the private sector and individual Canadians. In 2015, the Government of Canada (GC) committed to reducing GHG emissions by 30% from 2005 levels by 2030, based on the Paris Agreement, ratified in October 2016. The Pan-Canadian Framework on Clean Growth and Climate Change (PCF) proposed specific federal mitigation and adaptation measures to meet the 2030 target. In 2021, Canada updated the nationally determined contribution under the Paris Agreement to reduce emissions to 40-45% reduction by 2030, increasing Canada’s level of ambition in reducing GHG emissions.

The transportation sector is one of Canada’s largest sources of GHG emissions. The increased adoption of zero-emission vehicles (ZEVs, which include battery electric, plug-in hybrid, or hydrogen vehicles that have the potential to produce no tailpipe emissions) is critical to decarbonising the transportation sector and transitioning to a low-carbon future in Canada. A key enabler to greater ZEV penetration is the availability of next-generation and innovative electric vehicle (EV) charging and lower-carbon (LC) refuelling technologies and solutions to support the establishment of sufficient infrastructure.

Although ZEV technologies and solutions are commercially available, there are technical and non-technical barriers and gaps (e.g., cold weather-related charging efficiency, applications for public transit, etc.) to their adoption. EV charging and LC refuelling infrastructure is a young and emerging industry where technologies and solutions continue to move along the innovation or product lifecycle continuum. EV charging and LC refuelling infrastructure should be viewed in the context of a complex system that marries hardware, payment, communication, energy management, and other systems. As projects progress, these individual systems and their technological readiness levels (TRL)^{Footnote 1} may not evolve in parallel as solutions are validated and tested.

Program Description

The EVID program is a transfer payments program that supports the demonstration of next-generation and innovative EV charging and LC refuelling infrastructure. Demonstration projects span the country and partner with various proponents (e.g., industry, municipal governments, etc.). The projects seek to address technical and non-technical barriers and gaps to installing, operating, and managing next-generation and innovative EV charging and LC refuelling technologies and solutions.

The EVID program along with the broader suite of energy innovation and clean growth research, development and demonstration (RD&D) programs at NRCan work together to drive innovative energy technologies and solutions to commercialisation, as well as to promote the adoption of these technologies and solutions to provide environmental and economic benefits. The program is sunsetting in fiscal year 2023-24, with most projects ending by March 31, 2023. The EVID program is currently managed under the Energy Innovation and Clean Technology Program in NRCan’s Program Inventory by the Energy Efficiency Technology Sector, specifically the Energy S&T Programs section of the Office of Energy Research and Development (OERD). The program is under NRCan’s Core Responsibility 2: Innovative and Sustainable Natural Resources Development.

Program Expected Results

The EVID program’s logic model and the related theory of change (TOC) were developed by the AEB for the purpose of the evaluation and discussed with the OERD during the evaluation planning process (see Figure 1). The program logic model and TOC were reviewed and approved by the OERD. The evaluation team developed several performance indicators based on the logic model and TOC that were not specified in the foundational documents; therefore, these indicators do not have a related performance target. Some performance indicators that were from the foundational documents have a pre-existing performance target, which is referenced where relevant in the evaluation report. The performance indicators used for this evaluation can be found in Appendix A.

Figure 1: Program logic model

Text version

Figure one: Program logic model, shows the EVID program logic model that displays the relationships amongst inputs, activities, outputs, immediate outcome, intermediate outcomes, and the ultimate outcome. An arrow points from “input” to “activity”, from “activity” to “output”, from “output” to “immediate outcome”, from “immediate outcome” to “intermediate outcome”, and from “intermediate outcome” to “ultimate outcome”.

The inputs are: Federal Funds, Personnel, and Recipient resources (e.g., leveraged funds).

The activities are:

• Identifying initial barriers and gaps in the lower-carbon transportation sector.
• Program management and priority setting, comprising:
  • Call for Expressions of Interest and proposal;
  • Project selection, due diligence, contribution agreement approval;
  • Academic, industry, and public sector engagement; and
  • Portfolio management (e.g., payments, reporting, assessment of outcomes, etc.).

There are three outputs. The first output is “academic, industry, and public sector collaborations for the demonstration of innovative energy technologies and solutions”. An arrow from the first output points to the second output, “supported innovative energy demonstration projects through investment by stakeholders (e.g., contribution agreements, memoranda of understanding, leverage from collaborators, etc.)”. The second output points to the third output, “accessible scientific and technical information/knowledge products (e.g., technical reports, journals, gap analysis, etc.) and events (e.g., workshops, conferences, presentations, etc.) to support knowledge transfer”.

The immediate outcome is “technologies, solutions, and knowledge target specific barriers and gaps to the uptake and adoption of EV charging and LC refuelling infrastructure”.

There are three intermediate outcomes (post-program sunset, 2024). The first two intermediate outcomes are:

• EVID projects move innovative technologies and solutions closer to commercial readiness.           
• Results and learning from EVID projects complement the development of codes & standards.

An arrow from each of the first two intermediate outcomes points to the second intermediate outcome: EVID projects address targeted barriers and gaps to the uptake and adoption of EV charging and LC refuelling infrastructure.

Finally, the ultimate outcome (post-program sunset, 2024) is “environmental and economic benefits resulting from technologies and solutions, as well as evolving codes and standards, advancing towards the commercialisation of LC transportation in the long run”.

The TOC articulates that the EVID program addresses barriers and gaps related to next-generation and innovative EV charging and LC refuelling infrastructure through three core features:

As next-generation and innovative EV charging and LC refuelling infrastructure is being deployed, while being supported by evolving codes and standards, the adoption of ZEVs increases. This shift begins to stimulate benefits to the environment and economy. Given that the development and deployment of new technologies and solutions often require multiple components, decades of effort, numerous research and multiple proponents, the environmental and economic benefits are often only observable several years in the future. The literature shows that the time range from invention to widespread deployment could take between 20 and 70 years. ^{Footnote 2}Therefore, the EVID program’s results and progress to date must be interpreted in this context.

The EVID program contributes to federal leadership actions in addressing climate change and the urgent need for action at all levels. The program promotes clean growth in the transportation sector. The EVID program itself does not have any GHG targets or related GHG reporting requirements because it is an enabler for reducing GHG emissions. The EVID program along with the broader suite of energy innovation and clean growth RD&D programs create future opportunities for the sector to grow, which leads to broader systemic changes in the S&T ecosystem.

Key Stakeholders and Target Groups

Eligible recipients for funding include legal entities validly incorporated or registered in Canada: utilities; companies; industry associations; research associations; standards organisations; Indigenous and community groups; Canadian academic institutions; and provincial, territorial, regional and municipal governments and their departments and agencies.

Program Governance

Key decision-making bodies include the Interdepartmental Director General (DG) Committee on Energy S&T and its related DG S&T Sub-Committee, Technical Review Committee (TRC), and Program Management Team.

• The Interdepartmental DG Committee on Energy S&T and the DG S&T Sub-Committee are co-chaired by the DG, OERD, Innovation & Energy Technology Sector and DG, Energy Resources Branch (ERB), Energy Sector. The committees provide oversight and advice to the OERD to support and guide program delivery, including endorsing the TRC’s recommendations on the demonstration projects to be funded. The Interdepartmental DG Committee in Energy S&T Sub-Committee includes DGs from Environment & Climate Change Canada, and Innovation, Science & Economic Development Canada. Other members that could be engaged for specific objectives include NRCan’s CanmetENERGY-Ottawa and CanmetENERGY-Varennes, Indigenous Services Canada, the National Research Council of Canada, and Transport Canada.
• The TRC is chaired by an Assistant Program Director or otherwise appointed staff of the OERD or ERB. The committee advises the Program Management Team, including proposal review and technical matters related to their expertise. The TRC is comprised of the OERD and other S&T advisors, senior scientific, technical and policy experts from science-based departments, and external stakeholders from academia, provincial governments, civil society or industry.
• The Program Management Team is co-chaired by the Assistant Program Directors or otherwise appointed lead(s) for the program from the OERD and ERB. The team is tasked with the design and delivery of the program, including activities for program management and priority setting. It includes staff members of the OERD and other branches that support program design and delivery.

Program Resources

Phase I and Phase II were allocated $46.1 M over two years (2016-17 to 2017-18) and $30 M over four years (2018-19 to 2021-22) respectively. Both phases received an incremental funding of $1.3 M for their duration to support a full-time employee and operating costs. Contribution Agreements (CAs) were to be monitored in the intervening years until expected completion in 2021-22, which was then extended to 2023-24 to match the project spending profile provided to NRCan by proponents. During the evaluation period, the EVID program approved 33 projects with CAs for a total of over $140 M in NRCan and leveraged project value. If a contribution to a demonstration project leads to a profit, the funding recipient will be required to repay the transfer payment.

Evaluation Objectives, Methods, and Limitations

The following key aspects of the program operating context were considered as part of the evaluation design:

• The ZEV sector is rapidly evolving, as its technologies and knowledge are constantly changing, resulting in changing needs and priorities. Hence, the EVID program constantly needs to adapt to the evolving context. There is a need to examine whether the program has selected appropriate demonstration projects to address the needs and priorities, especially in the face of resource constraints.
• S&T RD&D projects are not conducted with expectations of a 100% success rate. The timeline to accomplish what the EVID program intends to do is challenging. To date, the program has begun to observe some short-term outcomes from Phase I. In general, long-term outcomes are difficult to capture given that the program has an indirect influence on GHG reductions for the transportation sector.

The evaluation examined the relevance, effectiveness, efficiency, and economy of the EVID program from fiscal years 2016-17 to 2021-22. Accordingly, the evaluation focused on:

1. The EVID program’s ability to appropriately address the evolving needs and priorities of the EV and LC refuelling sector (relevance).
2. The EVID program’s achievement of the short-term outcomes, as well as trends towards the achievement of the longer-term outcomes (effectiveness).
3. The EVID program’s capacity to operate as planned to achieve the intended outcomes, given the rapidly evolving context and the resources allocated (efficiency and economy).

The approach and methodology used for the evaluation followed the TB Policy on Results (2016) and related Standards on Evaluation. The evaluation used three lines of evidence (Figure 2) to answer the evaluation questions (EQ; see Appendix A).

Figure 2: Lines of Evidence

Text version

Figure two, Lines of Evidence, is an infographic showing the evaluation’s three lines of evidence.

The first column describes the document review. A document review was conducted to provide an overall understanding of the program, reported results, efficiency, and economy. The document review also provided information on performance, governance, priorities and planning, resource allocation, decision-making processes, and operational structures and challenges. Foundational and strategic documents, project and program files, and reports on program results were reviewed.

The second column describes the literature review. The literature review provided an understanding of relevance and performance in a wide context. It also helped identify best practices, lessons learned, and improvements required to support the design, implementation, and delivery of similar programming in the future. Three topics were included in the literature review:

• Canada’s role and comparison of its approach with select countries in supporting the demonstration of EV charging and LC refuelling infrastructure.
• The unintended outcomes related to EV charging and LC refuelling infrastructure.
• EDI factors in EV charging and LC refuelling infrastructure.

Key documents and articles generated from academia, the federal government, and other national and international jurisdictions obtained from Internet searches and key informants were reviewed.

The third column describes the key informant interviews. Key informant interviews were used to answer evaluation questions about relevance and performance. A total of 15 interviews were conducted via video conferencing with 5 program staff members, 3 TRC members, and 7 proponents. Proponents were selected using maximum variation selection, capturing a range of perspectives from key areas or priorities to the extent possible. Although all stakeholder groups were asked questions related to relevance, effectiveness, efficiency, and economy, interview questions were tailored to the stakeholder group. For instance, interviews with proponents focused on their experiences with the EVID program and the results generated from their demonstration projects.

Evaluation Limitations

Although the evaluation was designed to collect data using multiple lines of evidence to enhance the reliability of results and validity of findings, the following limitations should be considered when reviewing the evaluation findings.

• Although sufficient documentation was available to inform the evaluation, some information that would have been useful to inform the analysis was unavailable at the time of the evaluation. For example, performance reporting from all proponents was not yet available at the time of analysis. This limitation was mitigated to the extent possible by interviews with proponents.
• Due to consultation fatigue and limited availability, not all proponents were available to participate in the interviews. This limitation was mitigated using maximum variation selection to capture the wide range of perspectives possible and leveraging the EVID program’s existing relationship with proponents to contact the proponents. The evaluation team also ensured that the interviews were targeted (i.e., only asked questions that were relevant to proponents).
• Due to the EVID program’s operating context, the evaluation of the program’s effectiveness was limited to the intended outputs and immediate outcome. There are numerous external factors (e.g., the success of the demonstration projects) that affect the program’s achievement of the intermediate outcomes and ultimate outcomes; the program is mostly an enabler. To mitigate this limitation, the evaluation focused on the extent to which the achievement of the intended outputs and immediate outcome would facilitate progress towards the long-term outcomes.
• TRC members had a limited view of the EVID program because of the nature of their involvement. TRC members were mostly involved in the proposal evaluation stage. Therefore, their viewpoints were mostly limited to a high-level view of the program’s relevance, results, and activities (e.g., how proposals were evaluated). Accordingly, findings from the TRC members were analysed and interpreted with this context in mind.

What We Found: Relevance

Is the EVID Program focusing on relevant needs and priorities within the changing context of the program’s operating environment?

The EVID program aligns with and responds to stakeholders’ needs.

Evidence showed that EV charging and LC refuelling infrastructure is a critical enabling mechanism to accelerate the deployment of ZEVs, which will contribute to overall GHG reductions from the transportation sector. The ZEV sector is still developing in Canada; EV charging and LC refuelling infrastructure RD&D in this sector is occurring at a rapid pace. Key informants from all stakeholder groups stated that the needs and priorities continue to exist because the sector has not yet matured sufficiently to support Canada’s decarbonisation and transition to a low-carbon future. The EVID program is perceived to have a niche because it addresses Canadian needs and priorities that cannot be demonstrated elsewhere, such as demonstrating technologies and solutions in cold climates, partnering with Indigenous communities, and ensuring pan-Canadian interoperability. The program works with provinces and territories, such as by collaborating with the Government of Yukon to demonstrate the all-season performance of public charging stations, enabling a strong alignment between industry and all levels of government, increasing learning in the areas of interoperability and communication standards as well as addressing unique needs (e.g., northern jurisdictions). Key informants emphasized that the work with neighbouring jurisdictions is required to fill gaps for travelling between or through jurisdictions.

Key informants affirmed that progress in this area would be more challenging and slower without federal support, such as through the EVID program. Indeed, the evaluation team observed that the introduction of the EVID program as a “kick-start” was well-timed to be positioned for growth in the sector and was complementary to other initiatives and programs (e.g., vehicle purchase incentives). The electrification of the sector needed additional complementary measures, including EV purchase incentives, regulatory measures, etc., to occur at the same time to provide proponents with the confidence that the long-term viability of the sector was assured and that investments would be worthwhile. These other complementary measures have been initiated at the federal and provincial levels incrementally over the past 8 years.

The EVID program aligns with and responds to GC and NRCan’s priorities, frameworks, and strategies.

Canada has several targets and priorities to reduce GHG emissions from the transportation sector. For example, the PCF, Clean Energy Ministerial’s Electric Vehicles Initiative, Mission Innovation, and the International Energy Agency (IEA)’s Hybrid and Electric Vehicle Technology Collaboration Programmes.

Examples of GC targets and priorities

Key informants from all stakeholder categories affirmed that the program facilitates Canada’s progress to meet these national and international targets and commitments involving electrification and decarbonisation in the transportation sector and in general. Through Budget 2016 and 2017, federal investment in the transportation sector supported Canadian leadership in transportation electrification and decarbonisation. In particular, the support for the demonstration of technologies and solutions fulfilled a core federal role in addressing innovation gaps and accelerating the development of higher-performance lower-cost green infrastructure technologies and solutions. The document review and interviews demonstrated that the federal government and NRCan have a role in supporting the demonstration of technologies and solutions for EV charging and LC refuelling. For instance, key informants perceived that the federal government plays a unique and an important leadership role in this area, especially in facilitating the interoperability of EV charging and LC refuelling infrastructure across the country. They explained that the sector would be more technologically fractured due to the competitive nature of the market, which would likely develop systems that were proprietary and not interoperable. The literature review showed that Canada is one of the few countries that supports the demonstration of innovative EV charging and LC refuelling infrastructure.

What We Found: Effectiveness

To what extent has the EVID Program achieved its intended outputs?

The EVID program has a diverse list of key stakeholders.

Evidence revealed that the EVID program has exceeded its performance targets for the number and type of stakeholders, having achieved an average of at least one partner signed per contribution agreement by 2018-19 and at least one Federal/Provincial/Territorial collaborative project by 2018 19. Collaborations among stakeholders provide support for project proponents and knowledge transfer, acting as mechanisms to address the barriers and gaps in the sector. Many demonstration projects are co-funded by provincial or territorial stakeholders. Most proponents are working with several partners, such as provincial utilities, technology solutions providers (e.g., engineering firms, equipment suppliers, etc.), EV manufacturers, other levels of government, transit agencies, etc. Private industry firms and companies are the primary project proponents and partners. At least one of the demonstration projects involves Indigenous communities. Projects by stakeholder type are illustrated in Figure 3.

Figure 3: Demonstration projects by stakeholder type

Text version

Figure three, Demonstration projects by stakeholder type, is a pie chart showing the percentage of EVID demonstration projects by stakeholder type: Provincial or territorial stakeholder projects, 3%; municipal or transit, 9%; education, 12%; utility (including private), 24%; and industry, 52%.

Scientific and technical knowledge products and events are generated either by the OERD or the supported demonstration projects.

Evidence showed that the EVID program projects have contributed towards scientific and technical knowledge products and events generated. Document review and interviews indicated that products and events are generated either directly by the OERD or indirectly by the demonstration projects supported by the program. The products generated include research papers, white papers and discussion papers, social media posts and discussions, and contributions to national and international standards (including work with the IEA and the Canadian Standards Association).

Notable events include the OERD’s EVID Symposium and Consultation (2021). Symposium attendees rated the event as “very good” and noted that events like this are important because they allowed them to share challenges and lessons learned related to their projects, as well as brainstorm opportunities and solutions to further advance their projects. Some projects produced more scientific and knowledge products than others partly due to the nature of the projects. However, it is important to note that according to proponents, not all projects can produce knowledge products either due to the nature of the project components or intellectual property concerns.

Document review and interviews indicated that information about the demonstration projects is shared through media, events (e.g., regional, national and international), and the NRCan website. Proponents also share information about the demonstration projects through their networks (e.g., utilities). One proponent reported that media items on their high-visibility project, reported as the world’s first automated electric-vehicle parking system, reached and engaged a large audience. Calls and emails are still coming in as a result, demonstrating that information products have engaged a wide audience and the products generated are influencing information uptake. Sharing of knowledge and results also facilitates the replication of the technologies or solutions (e.g., solutions for MURBs), thus advancing the sector as a whole.

Formal and informal collaborations among relevant cohorts were established.

Evidence indicated that the EVID program has met its performance target for degree and type of collaboration, by maintaining a varied and active portfolio. Document review and interviews showed that every demonstration project has some type of formal and informal collaborations among different cohorts, such as utilities, academia, government, vendors and more. Collaborations between project proponents and other stakeholders range from formal collaboration (e.g., through memoranda of understanding) to informal or relatively unstructured collaboration. The program also connects proponents with similar interests to facilitate information sharing. All proponents expressed positive views on the collaboration that was occurring. They also stated that the program could do more, such as by creating a forum, similar to the EVID Symposium and Consultation, for proponents to share information with other proponents and key stakeholders on a recurring basis.

To what extent has the EVID Program achieved its intended immediate outcome?

The EVID program is perceived to be effective in targeting specific barriers and gaps in the sector.

Evidence suggests that the EVID program is effective in identifying barriers and gaps related to EV charging and LC refuelling infrastructure [e.g., multi-unit residential buildings (MURBs), hydrogen refuelling, etc.]. Program staff explained that the program conducts activities (e.g., stakeholder engagement and literature review) during planning, implementation, and delivery to ensure that suitable demonstration projects are selected to address the identified barriers and gaps. Furthermore, the program intermittently updates the barriers and gaps through a continuous learning approach (e.g., the OERD updated the call for proposals based on the last call). TRC members, who are from other government departments, provided further support that the program is effective in identifying barriers and gaps. They stated that the program seems to align with key pieces that are moving in the field of electrification and decarbonisation in the transportation sector based on their knowledge of the sector and their observations of which demonstration projects are selected for funding. Evidence revealed that the EVID program exceeded its performance target, which was the approval of 4 to 8 demonstration projects in Phase I and 5 to 8 projects in Phase II, for the number of projects. Phase I included 10 projects with CAs, and Phase II included 23 projects with CAs. During the evaluation, there were 14 ongoing projects and 19 completed projects. Key areas or priorities that were included are illustrated in Figure 4. Some of the projects fall into more than one category. Program staff explained that at the start of the program, the EV market was a young and emerging sector that grew over time, which resulted in more applications than expected around Phase II. The program had to make adjustments to meet the growing needs and evolving nature of the sector. The program was extended from Phase I to Phase II with additional funding.

Figure 4: Program priority areas

Text version

Figure four: Program priority areas, is an infographic showing the EVID program priority areas: urban environment, including MURBs; two-way energy transfer; EV charging, including autonomous vehicles; freight transportation; workplaces and fleets; hydrogen refuelling; public transit; charging networks; and electrical grid.

Vendor innovation was generated because of the EVID program.

The EVID program’s TOC articulates that through support from the program, proponents can provide money to vendors and other partners to satisfy demonstration project tasks. Vendors and partners in turn have to innovate to deliver the project tasks. Evidence showed that the EVID program has enabled the generation of various degrees and types of vendor innovation. The projects have advanced knowledge and TRL to some extent. Based on a review of the program documents and project files as well as interviews with proponents, the program’s project selection encompasses a range of technologies, solutions, and targeted areas of development. For example:

• The innovation was a novel charging solution that could work for human-operated vehicles and self-driving vehicles. The project was reported by CBC News to be the world’s first integrated automated EV parking system.
• Vendor innovation for a transit project included breaking down competitive barriers, wherein EV and electric bus manufacturers would supply charging solutions that did not offer interoperability with other systems. The project was successful in creating charging points that could charge buses from multiple manufacturers.
• The innovation generated involved high voltage chargers and self-cooled cables that could withstand Canada’s very high and very low-temperature extremes without producing excessive waste heat. Solutions for on-street charging were also developed. The project successfully demonstrated that the charging technologies work well without significant downtime in the extreme cold of the Canadian north.

Capital assets were acquired because of the EVID program.

The EVID program’s TOC articulates that through the EVID program’s support, proponents have the resources and capacity to acquire capital assets for long-term use in their business or organization to provide operational benefits. Evidence demonstrated that the program has enabled proponents to acquire various degrees and types of capital assets. EV charging and hydrogen refuelling infrastructure are the most acquired or highest-value capital assets. Proponents mostly acquired or assembled EV chargers and charging systems, many of which were installed to conduct project-related testing. Select examples provided by interviews and program documentation were:

• The infrastructure acquired included eight high-density low-footprint parking towers with autonomous charging nodes and robotics to help plug-in vehicles.
• Highly technical physical assets for fully interoperable charging of electric buses from most of the major manufacturers with Canadian operations that were not deployed anywhere else. The transit agency hoped to be able to extend the life of the equipment to 15 or 18 years, instead of just 10, by using a strong maintenance program.
• Several charging points are now in operation in the North, connecting areas that did not have any public EV charging points previously.

Learning and transferable skills were gained because of the EVID program.

The EVID program’s TOC articulates that through the EVID program’s support, proponents can learn new knowledge and skills by performing project tasks; learning and transferable skills are constantly evolving as the innovation landscape is dynamic. Evidence indicated that the program has enabled the attainment of various degrees and types of learning and transferable skills among NRCan staff, proponents and partners. The skills gained vary by demonstration project, mainly as a result of learning by doing in driving forward project work. Select examples provided by interviews and program documentation were:

• Learning has been acquired in hardware and software design and development, including for mechanical, electrical and computer technicians.
• To use new equipment, it was necessary to utilize new learning processes and training methods for staff and work with original equipment manufacturers (OEMs) to develop training. The training was given to maintenance staff, millwrights, electricians, and first responders from the municipality (including high voltage safety training).
• An organisation’s team trained several students by involving them in the project.
• Knowledge and learning have been obtained on equipment operation and charger performance.
• Learning has occurred for local tradespeople and the private sector that have participated in installing the charging units.
• The ability to test the Open Charge Point Protocol (OCPP) has been a critical innovation, benefiting spin-off companies from the partners involved. The OCPP is a standardized communication protocol that enables interoperability and communication between open EV chargers and charging station management systems.

Valuable relationships were formed because of the EVID program.

The EVID program’s TOC articulates that coordination and communication among proponents, partners, and relevant cohorts help to form valuable relationships. These relationships are important because solving the barriers and gaps in this sector involves multiple key areas (e.g., government, energy providers, network operators, etc.). Evidence showed that the program has enabled the formation of various degrees and types of valuable relationships (e.g., business alliances, engagement with collaborative networks, etc.). At the time of the interviews, program staff and proponents stated that many of these relationships were ongoing as the demonstration of the technology and solutions continue. Proponents explained that relationships with clients, partners, and other interested or key stakeholders (e.g., utilities) are important and useful. Select examples provided by interviews and program documentation were:

• Valuable working relationships have been formed with peers in the transit area across North America and internationally, including with the American Transit Association in the United States, Europe, and Asia.
• Valuable relationships have been formed with the First Nations involved in a project, and these Indigenous partners have become strong advocates for electric mobility.
• A proponent credited the EVID project for getting manufacturers to work together and agree on the OCPP, which is now included in all municipal procurements. The OCPP allows all vehicle and charging performance information to be stored in a single repository that was not possible previously.
• Valuable relationships are ongoing with Canadian universities, international partners, the Canadian Standards Association, and technical solutions providers and suppliers.

To what extent has the achievement of the immediate outcome facilitated the achievement of the longer-term outcomes?

The EVID program’s progress towards the immediate outcome may facilitate the achievement of the longer-term outcomes.

Evidence suggests that the EVID program’s progress towards the immediate outcome is facilitating the achievement of the intermediate outcomes and ultimate outcome. Program staff and TRC members agreed that the program is progressing towards the intermediate outcomes and ultimate outcomes. However, it is still too early to assess or demonstrate these outcomes because not all demonstration projects have been completed and in operation for a sufficient period. Phase I projects are still in the early outcome reporting phase, and Phase II projects are in various implementation stages.

Program staff and TRC members agreed that the EVID program is supporting demonstration projects that are contributing to the next stage of innovation in key areas and priorities (as identified in the applicant guide). Based on their observations of the program’s activities and results to date, they believed that the projects are facilitating the advancement of the sector. However, it is important to note that there is still a timing gap in knowing whether the projects will actually be successful. Additionally, program staff noted that not all barriers and gaps in the sector can be addressed by the program alone because of the size and scope of the challenges in the sector. The program is perceived to be a relatively small program and an enabler within a complex S&T ecosystem.

Program staff perceived that the projects funded by the EVID program are headed in the right direction concerning commercialisation based on the current information generated by the demonstration projects and information gathered by the program to date. However, they cautioned that the program does not have complete results yet. Based on the document review, the evaluation found that it is still too early to conclude or generalize an average TRL increase. Many of the projects are continuing and have not submitted the final reporting that will contain the final TRL values. That said, about four of the presumably high-capacity proponents (e.g., utilities and educational institutions) have shown four-point increases (e.g., 4 to 8) with one industry project reporting it started at a 5 and went to 9. It is worth noting that all proponent reporting is based on self-assessment and an assessment by an expert S&T verifier does not happen until the end of the project. Figure 5 depicts the nine TRLs; the program requires projects to have a TRL of at least 5 at the beginning of the project and must achieve a TRL of at least 8 by the end of the project.

Figure 5: Technology readiness levels^{Footnote 3}

Text version

Figure five: Technology readiness levels, is an infographic showing technology readiness levels. Arrows point from Level 1 to Level 9.

• Level 1: Basic principles of concept are observed and reported.
• Level 2: Technology concept and/or application formulated.
• Level 3: Analytical and experimental critical function and/or proof of concept.
• Level 4: Component and/or validation in a laboratory environment.
• Level 5: Component and/or validation in a simulated environment.
• Level 6: System/subsystem model or prototype demonstration in a simulated environment.
• Level 7: Prototype ready for demonstration in an appropriate operational environment.
• Level 8: Actual technology completed and qualified through tests and demonstrations.
• Level 9: Actual technology proven through successful deployment in an operational setting.

Source: Innovation, Science and Economic Development Canada. Technology readiness levels.

Program staff perceived that the EVID program’s results and learning complement the development of codes and standards. They explained that the program learns about codes and standards through the demonstration projects, and the results and learning may help inform the sector. For instance, in the case of at least one demonstration project, a proponent highlighted that they were approached by the Canadian Standards Association for input as a result of their project work. Based on the document review, the evaluation found that reporting is too inconsistent at this stage to conclude on this intermediate outcome.

Program staff and TRC members agreed that the EVID program is likely to contribute to environmental and economic benefits by advancing towards the commercialisation of LC transportation in the long run. They explained that EV charging and LC refuelling infrastructure advancement goes together with the adoption of ZEVs, helping Canada progress towards its ZEV targets and commitments in the most effective ways. However, they emphasized that the EVID program is mostly an enabler and that achieving the ultimate outcome depends on a range of factors outside the program’s direct control (see the program’s TOC in a previous section). Based on the information being generated by the demonstration projects to date, program staff and TRC members were positive that the program is moving in the right direction. The evaluation team observed that every project was assessed from both environmental and economic perspectives during selection to ensure that the program is well positioned to contribute to the ultimate outcome. The projects are expected to produce outputs that will contribute to the expected benefits once they are complete, and the technologies and solutions are operational.

The evaluation did not find any negative unintended outcomes that could be directly attributed to the program.

Program staff and proponents could not identify any unintended outcomes directly attributed to the EVID program. However, one proponent mentioned that their project has helped to develop standardized transit-industry-specific performance indicators for the charging infrastructure and vehicles that will allow the industry to make comparisons and standardize potential GHG reductions, which could be considered as a positive unintended outcome.

The literature review found several unintended outcomes that are often associated with ZEVs and EV charging and LC refuelling infrastructure (see Figure 6). It is important to emphasise that these are general potential impacts of the transition to ZEVs, which cannot be attributed to the EVID program based on the evidence from the evaluation. Therefore, these are mostly issues to be considered for future transportation-related programs. Furthermore, addressing these issues is outside the scope of the EVID program. Although there is a potential for negative unintended outcomes because of the transition to ZEVs, there are concrete efforts by the Government of Canada (including NRCan) to mitigate many of these broader potential negative unintended outcomes. The evaluation team also observed that some demonstration projects supported by the EVID program (e.g., EV public transit, grid integration, load management, etc.) may address some correlated negative unintended outcomes related to the automobile industry.

Figure 6: Potential unintended outcomes of the shift to ZEVs from the literature review

Text version

Figure six: Potential unintended outcomes of the shift to ZEVs from the literature review, is an infographic showing potential unintended outcomes of the shift to ZEVs from the literature review.

Positive unintended outcomes:

• EVs can act as dynamic energy storage devices and can create an energy buffer.
• Reduced noise pollution.
• Use of EV charging and LC refuelling infrastructure located in a community may support the community’s economy.

Negative unintended outcomes:

• Uncontrolled charging may negatively impact distribution networks of electrical grids.
• Social and environmental concerns (e.g., pollution impacting nearby Indigenous communities) from mining for battery components.
• Negative impacts on communities that are historically underrepresented or underserved in the transportation sector  (see EQ8).
• Environmental impacts related to the automobile industry (e.g., carbon emissions from EV productions, land use of vehicles, etc.).

To what extent has the EVID Program considered equity, diversity and inclusion (EDI) factors?

EDI factors are important considerations in the ZEV sector, and some of these factors are being addressed by the EVID program.

Evidence suggests that EDI considerations are pertinent in the ZEV sector. The literature review showed that not all groups have equitable access to ZEVs and ZEV infrastructure. Underinvestment in EV charging and LC refuelling infrastructure in disadvantaged groups raises the risk that these groups will miss out on the benefits of transport electrification and decarbonisation. The lack of sufficient access to EV charging and LC refuelling infrastructure tends to disproportionately affect communities with a high proportion of low- to middle-income households and people of colour, exacerbating the already present racial, financial and cumulative social disparities in ZEV adoption. The lack of access to EV charging and LC refuelling infrastructure in these communities is in part due to housing conditions. For example, MURBs in disadvantaged neighbourhoods tend to have few resources for EV charging and LC refuelling infrastructure. A report from the Pembina Institute stated that building owners of existing MURBs are unlikely to see EV charging station investment as a priority without significant financial incentives.

Furthermore, existing underdeveloped electricity, transportation, or communication infrastructure in disadvantaged groups may result in biased and inequitable siting decisions for future EV charging and LC refuelling infrastructure. For instance, rural, remote and northern communities, which tend to have a high population of Indigenous persons, face unique barriers to electrification using clean, reliable, and renewable sources. Many of these communities are not connected to an electricity grid that is reliable and robust and is powered by diesel generators. These conditions limit energy availability and security. A reliable and robust electricity grid is needed to establish a dependable EV charging and LC refuelling infrastructure that can cater to growing a potential ZEV market in these communities. Additionally, a report from the Pembina Institute mentioned that youth is one group that may look to governments to provide affordable low-carbon alternatives including electric transit options.

The foundational documents indicated that there are no factors identified in the planned design and delivery of the EVID program that would differentially impact gender groups, nor would it reinforce the existing gender imbalance in these sectors. Although there is differential access or impacts of ZEVs in general (e.g., some groups cannot afford to adopt advanced technology vehicles), it is not in the scope of the EVID program to significantly correct this disparity. Furthermore, understanding the interplay of EDI factors in the clean technology sector is made difficult by a lack of disaggregated data, and that is not a limitation dealt with by only the EVID program. ^{Footnote 4} To date, NRCan programs have experienced limitations in their ability to collect EDI data for GBA+ due to privacy and legal requirements that contributed to data gaps in NRCan programs in general.

When the EVID program was first developed in 2016-17, there was not the same level of emphasis on EDI or GBA+ as there is now because GBA+ and EDI were not widely operationalized policies at the time. The foundational documents revealed that although the EVID program was not designed to address EDI factors when it was first launched in Phase I, the program has incorporated EDI considerations in Phase II. Program documentation and interviews revealed that during implementation, demonstration project proposals that address EDI factors received additional consideration (e.g., prioritising projects in the North with Indigenous communities as partners).

According to program staff, there are funded demonstration projects that are addressing issues and barriers that are related to equity to some extent. This is in terms of access for underrepresented groups, regionality (e.g., Northern Canada, urban versus rural), housing type and cost and payment options, based on issues flagged by the literature. The evaluation team observed that the overall project selection represented EDI principles. Notably, some of the demonstration projects supported by the EVID program are specifically designed to address some of the aforementioned EDI factors. Examples of projects that consider EDI factors include:

• Project #1015 Yukon Government is a northern project that involves a First Nation community, Carcross Tagish First Nation. This project monitors the successes and challenges of operating EV fast chargers in all seasons, including in cold temperatures. The results of the project will encourage other communities to move forward with the electrification of transportation and expand EV infrastructure in Northern communities.
• Electrification of municipal transit projects (e.g., Brampton Transit and Toronto Transit Commission) made technologies and solutions accessible to a representative range of Canadians that use public transit. These projects did not limit impacts to only those Canadians that can afford to drive a ZEV.
• Some projects selected (e.g., S2E) focused on charging solutions for MURBs or for curb-side charging, which will have impacts on urban or suburban residents in higher-density population centres. The technology is not likely to reach lower-income MURBs at present.
• Making EV charging affordable was cited by more than one proponent as an equity factor. For one northern project, electricity and charging were offered free of charge.

Proponents interviewed represented a mix of organisations that did not have an official EDI policy, those that had one, and those that relied on policies from industry associations (e.g., The Canadian Urban Transit Research and Innovation Consortium). Proponents considered EDI factors from the perspective of project teams having sufficient diversity. They mentioned that their organisations’ internal project teams were created without discrimination, using mostly internal EDI mechanisms.

What We Found: Efficiency & Economy

To what extent has the EVID Program been implemented and delivered as planned?

The evaluation identified several factors influencing the EVID program’s implementation and delivery.

Evidence demonstrated that the EVID program’s implementation and delivery were affected by several internal factors and external factors (see Figure 7); therefore, the program was not implemented completely as planned. The internal factors were compounded by the external factors. The external factors affected the shifts in proponents’ timelines, which in turn affected the internal operation of the program due to the need to accommodate proponents. Interviews and document review showed that the program’s timeline was the most significant difference between the planned versus actual design. The timeline for Phase I (2016-17) was extended an additional four years (i.e., Phase II from 2018-19 to 2022-23); Phase II of the program was then extended to 2024 based on the same reasons.

Figure 7: Internal factors and external factors impacting the EVID program

Text version

Figure seven: Internal factors and external factors impacting the EVID program, is an infographic showing the internal factors and external factors impacting the EVID program.

Internal factors:

• The aggressive nature of the planned timelines for implementation and delivery.
• Conducting reprofiling exercises to deal with the lack of flexibility in the funding structure.
• The depth and amplitude of the work required more effort and resources than expected.

External factors:

• Changes in the sector, as the sector was just emerging at the beginning of the program.
• The lengthy ARLU process, necessitated by proponents needing funding extensions.
• Policy changes at the provincial-, territorial- and municipal- level.
• The COVID-19 pandemic.
• Inability of some of the demonstration projects to proceed within the originally envisaged scope or in all of the planned activity areas (See Figure 9).
• Innovation is inherently risky; some failure is expected.
• The evolving nature of codes and standards can be both an opportunity and a limitation depending on the demonstration projects.

An arrow points from “external factors” to “internal factors”.

Program staff stated that the original program timeline was a significant challenge because it was too short. The first call for proposals did not generate an adequate number of proposals, due in part to the novelty of ZEVs and the required infrastructure. They explained that the project solicitation to implementation process can take up to two years to complete (e.g., due diligence process), and then proponents require up to two years to ramp up spending. This resulted in less than planned spending in the first three years (see Figure 8). Planned and actual expenditures peaked in 2017-18, which is attributable to the ramp-up of project spending by the proponents. As of 2021-22, the program underspent by approximately $15M from the originally budgeted amounts.

Figure 8: EVID Budget Over Evaluation Period (2016-17 to 2021-22)

Text version

Combination chart showing EVID Budget over evaluation period, 2016-17 to 2021-22.

Spending ($) by fiscal year for original budgets, post ARLU budgets, and actual expenditures.

	2016-17	2017-18	2018-19	2019-20	2020-21	2021-22	TOTAL
Original Budgets	9.47 M	36.06 M	10.82 M	10.94 M	4.48 M	3.53 M	75.31 M
Post ARLU Budgets	9.47 M	22.26 M	4.74 M	15.57 M	4.04 M	6.34 M	62.43 M
Actual Expenditures	1.20 M	22.02 M	4.79 M	10.29 M	9.60 M	12.26 M	60.16 M

A textbox within the chart states, “The COVID-19 pandemic started to impact the program at the beginning of 2020.”

Program staff also identified the lack of flexibility in the funding structure, such allocated funds that must be spent in the year in which they were budgeted, as a significant challenge. Document review showed that program funds had to be transferred from one year to another to address the lack of flexibility in the funding structure. Program staff stated that the reprofiling exercises were time and resource intensive. The program was already limited in its planned resources (i.e., two-term employees for Phase I and one full-time employee for Phase II). Funding reprofiling was necessary each year to accommodate shifts to proponents’ timelines due to the rapidly evolving and challenging nature of the program context. Concerning annual variances, program staff cited challenges related to the time needed for proponents to initiate the demonstration projects once CAs are signed, the time to order and receive goods, the time to complete civic works, and critical path project constraints. In consultation with proponents, the program attempts to forecast project spending on an ongoing basis, but actual spending often varies from these forecasted amounts.

Program staff cited the cancellation of co-funding on numerous green energy projects due to the changes in the Government of Ontario in 2018 as a significant challenge for many proponents. This impacted numerous proponents by either delaying their projects to seek new partners or cancelling their projects altogether. Proponents perceived the EVID program as the only reliable source of financing for the magnitude needed for largely pre-commercial project activities. Starting in 2020, the COVID-19 pandemic contributed to the program’s underspending because it impacted the ability of several demonstration projects to spend the funds allocated in the CA during the affected years. Program staff recalled that the pandemic resulted in many amendments due to delays in timelines and more funding needs (e.g., non-availability of material, equipment and human resources, supply chain issues, increased costs, difficulties for contractors or workers to access project sites, etc.). Figure 9 shows additional factors that impacted project performance to some extent.

Figure 9: Factors impacting proponents’ demonstration projects

Text version

Figure nine: Factors impacting proponents’ demonstration projects, is an infographic showing the factors impacting proponents’ demonstration projects.

Positive factors:

• The EVID program as a reliable source of financing.

Negative factors:

• The lack of reliable leveraged funds from other sources.
• The COVID-19 pandemic.
• Lengthy delays in negotiations with partners (e.g., time for partners to consult with their legal divisions on changes and updates).
• Municipal planning and other regulations (e.g., do not generally facilitate new charging solutions).
• Presence of interprovincial barriers (e.g., it is easier to sell to the United States market than to different provinces within Canada).
• Laws related to provincial and territorial utilities acts (e.g., the sale of electricity is usually restricted to regional utilities).
• Extra workload in having to meet higher Canadian standards (e.g., welding).

Program staff explained that although the program had little control over the factors impacting program performance, they tried to mitigate these factors by being creative and adaptive within allowable parameters. Program staff initiated additional calls for proposals to ensure that there was an adequate number of proposals. They also created a secondary list of pre-screened and pre-approved projects as backup, leveraged existing resources, and streamlined activities to maintain internal efficiency. However, there were times when staff members had to work long hours to maintain efficiency. Additionally, the evaluation team confirmed that some of the risks cited in foundational documents were accurate (e.g., the risk of failing projects and the rapidly evolving nature of the industry). However, the evaluation could not conclude on the effectiveness of any of the mitigation measures. During the COVID-19 pandemic, the program staff tried to maintain good communications with proponents (e.g., sending emails to proponents to ask them how the pandemic had impacted them to try to better understand proponent needs) and meet proponent needs by being flexible [e.g., giving proponents the opportunity to change the funding profiles via the Annual Reference Level Updates (ARLUs)]. Program staff affirmed that despite these challenges, the program’s success to date was largely unaffected because the program was able to adjust to the circumstances.

Is the EVID Program’s design an efficient and economical means of achieving the intended outcomes?

The EVID program design generally supports the program’s activities, outputs, and outcomes.

Evidence revealed that the EVID program design generally supports the program’s activities, outputs, and outcomes. The document review revealed that program activities, outputs, and outcomes are enabled using tools for administration, portfolio management, calls for proposals, project selection, and communications. Program staff stated that the program design is generally efficient and economical for its risk tolerance level. They explained that the processes used by the program (e.g., the extensive due diligence process) are necessary to ensure that federal funds are protected and that there are opportunities to course-correct at any point. Some staff members noted that certain interim processes could be streamlined (e.g., decision-making process to get the CAs signed). Program staff provided several examples of the strengths and limitations of the program design (see Figure 10). They cited the lack of flexibility in how allocated funds must be spent in the year in which they were budgeted as the most significant limitation of the program design.

Figure 10: Strengths and limitations of the program design

Text version

Figure ten: Strengths and limitations of the program design, is an infographic showing the strengths and limitations of the program design.

Strengths:

• The EVID program leverages the common OERD infrastructure and processes (including the matrix team design), which is perceived by program staff to be an efficient model that is learning and improving continuously.
• The program model is robust and effective in leveraging expertise (e.g., a diverse group of stakeholders) with opportunities and processes to validate and evolve the program’s directions.
• The process is open and transparent as program and project information are available publicly, and feedback is provided to proponents.

Limitations:

• There is a lack of flexibility in the funding structure, which is further strained by delays and challenges that are common in RD&D. When amendments are required, the lack of flexibility affects the efficiency of the EVID program and its proponents. NRCan's funding cycles are not perfectly aligned with proponents’ timelines.
• The planned resources are limited because there is only one FTE allocated. During implementation and delivery, the program is supported by existing resources in the OERD to maintain efficiency.

Does the EVID Program’s performance measurement strategy support the determination of program performance?

Although the EVID program has a performance measurement approach in place, there are opportunities for improvement.

The EVID program’s expected results and performance measurement approaches are articulated in its foundational documents. In 2020, the OERD consolidated the EVID program and other OERD transfer payment programs under the Energy Innovation and Clean Technology Performance Information Profile.

Evidence indicated that the EVID program has a performance measurement approach in place, which was developed in early 2017 to reflect requirements specified in the TB Policy on Results. The document review found that the program has several trackers and reporting dashboards; however, tracking progress against expectations could be improved. For instance, some of the information is not up to date, and from an evaluation perspective, does not provide consistent or standardized metrics that can be used to assess the program’s ongoing performance. Although the evaluation team was able to use the demonstration project files to determine the status of many of the projects, it was challenging for the team to use the files to determine the project status for the complete list of projects and overall program performance to date. It is also unclear how the program uses the project files received from proponents (e.g., progress reports on performance and project execution) to manage the program on an ongoing basis. Program staff explained that the frequency of progress reporting varies by demonstration project and that some information may only be due in the final project reporting. This approach may limit the opportunity for the program to course correct.

Program staff stated that the EVID program’s performance framework and indicators are useful because it provides a knowledge base on program and project performance. Examples of information currently collected from proponents include a self-assessment of the advancement of the technologies and solutions (e.g., TRL), number of highly qualified personnel trained (HQP), financial information on EVID-eligible expenses, and percentage of the project completed against the project plan (i.e., milestone approach). However, they remarked that the program’s performance measurement approach could be enhanced because the existing program reporting requirements cannot provide all the information of interest to inform on the program’s impacts. They expressed the need to better understand the program’s impacts; for instance, how the demonstration projects have progressed over time when it comes to social, environmental and economic benefits, and how much investment by the OERD has contributed to the success of a technology or solution. The OERD is aware of the issues related to the challenges in effectively measuring the long-term outcomes of its programs and is currently working to find solutions.

Proponents stated that project reporting was not onerous in general and that they understood the need for accountability in federal expenditures. There was a consensus that the EVID program succeeded in its selection of performance information and reporting requirements. When proponents were asked to provide suggestions for improvement, opinions were divided on what indicators would be most useful. Suggestions included economic measures, people-driven qualitative indicators, indicators that are tailored to the stakeholders (e.g., where people live including climate, geography, and other socio-economic factors), project-specific data on usage (e.g., EV charging and LC refuelling and how often special parking is used that could include consumer feedback), and how many ZEVs are in an area (e.g., EV density). Proponents also suggested that the program could clarify the specifics of what is required in the reporting templates (e.g., what type of information the program is looking for, how to report on the HQP, etc.). They mentioned that some assumptions had to be made about what the program is looking for because openness in the reporting templates created some level of uncertainty on what the program needs to know (e.g., the TRL scale can be subjective in the way it is applied). They also suggested that given the ultimate goal of enabling GHG reductions, there could be additional clarity on how NRCan views or calculates potential reductions, especially at the proposal stage. Proponents also noted several sensitivities when reporting on knowledge products as some project components are not amenable to producing related knowledge products (e.g., software-based solutions). In some cases, the project cannot publish sensitive intellectual property or partners may not want to consent to have the information published, especially while testing is ongoing.

Are there any best practices, lessons learned, and improvements required related to the design of the program?

The evaluation identified several lessons learned, best practices, and opportunities for improvement.

Evidence indicated that several lessons learned, best practices, and suggestions for improvement could be applied to similar future programming. Program staff stated that a key lesson learned is that the actual implementation and delivery of the EVID program required more resources than anticipated because of the growing needs and evolving nature of the sector (e.g., onerous administrative duties from the increased number of applications and project amendments). Program staff also stated that a program like this could benefit from more flexible financial budgeting. Another key lesson learned is that information sharing across different stakeholders with similar interests can be challenging at times due to the broad range of proponents and partners. At least one proponent cited that experience gained during the COVID-19 pandemic proved that virtual collaboration works.

Program staff shared several best practices. Most of these best practices were discovered during implementation and delivery to mitigate the internal factors and external factors impacting program performance:

• To address proponents’ concerns with the funding profiles, the program staff kept the proponents informed and educated about the ARLU process and risks before proceeding with the ARLU.
• The EVID program conducted webinars to inform proponents about the rules and processes for federal funding.
• The program uses ongoing feedback and learning to continuously improve the efficiency of its internal processes.

TRC members highlighted several features of the EVID program that are best practices that could be applied to similar, future programming. For example, NRCan could continue to collaborate with other government departments and agencies and other organisations that are heavily involved in ZEVs. NRCan could also continue to ensure that the results and learning from projects are disseminated to other relevant organisations and the public.

In terms of alternative program design, program staff suggested that the EVID program could consider a flexible funding model (e.g., allows funds to be carried from year-to-year), an annual call for proposals, and a budget timeframe that spans beyond the 4-year funding cycle. Proponents also mentioned the importance of continuing flexible programming in a future iteration of the program. They also provided suggestions for additional program elements, including an online inter-proponent discussion and collaboration forum, a Canadian EV infrastructure charging testing centre, or an online tool for decision support and charger selection. Notably, proponents raised the value of having a recurring platform that facilitates discussion and collaboration among proponents and key stakeholders, which could include a digital sharing session with other attendees (e.g., breakout meetings) to discuss successes and challenges with others operating in a similar area. Proponents believe such a platform would add value to the program by helping to advance the demonstration projects, and ultimately the sector, by allowing stakeholders to work collectively to find common solutions and identify innovative opportunities. It is worth noting that several proponents cited the EVID Symposium and Consultation that can be used as a model for a recurring platform. TRC members suggested that the program could enhance its focus on more challenging areas related to EV charging and LC refuelling infrastructure (e.g., hydrogen in MHDVs and other sectors). They also suggested that it may be beneficial to get reviewers that are external to the GC to participate in the TRC.

Conclusion

The evaluation focused on the EVID program’s ability to appropriately address the evolving needs and priorities of the EV and LC refuelling sector (relevance), the program’s achievement of the short-term outcomes and trends towards the achievement of the longer-term outcomes (effectiveness), and the program’s capacity to operate as planned to achieve the intended outcomes in the rapidly evolving context using the resources allocated (efficiency and economy).

Overall, the evaluation found that the program is relevant and will continue to be relevant as Canada works towards its targets and commitments to clean growth and climate change. The program has also achieved meaningful progress and achievement while operating efficiently and economically.

• Relevance: All lines of evidence suggest that the EVID program is relevant to the needs and priorities of stakeholders. Although the program is operating within an evolving context, the program has been able to keep pace with the sector. Demonstration projects are reported to be working on next-generation and innovative EV charging and LC refuelling technologies and solutions that are addressing key areas, which are expected to enable greater ZEV uptake in the future. There is a continued need for federal support in the demonstration and development of EV charging and LC refuelling infrastructure. The GC has a critical role in electrification and decarbonisation in the transportation sector in Canada.
• Effectiveness: All lines of evidence suggest that the EVID program is effective in achieving the intended outputs and immediate outcome. The program has enabled key capacity-building mechanisms to accelerate the deployment of next-generation and innovative EV charging and LC refuelling infrastructure. The program has exceeded all of its performance targets that were articulated in the foundational documents. However, more time is required to observe the program’s long-term outcomes due to the nature of RD&D and the sector. The program’s current progress suggests that the program is on track towards the achievement of the intermediate outcomes and ultimate outcomes.
• Efficiency and economy: All lines of evidence suggest that the EVID program has attained operational efficiency and economy. The program has generally been implemented and delivered as planned. The program’s implementation and delivery were affected by several internal factors and external factors. However, the program was able to mitigate some of these factors and adapt to the evolving context. Therefore, the program’s overall performance was unaffected. The program also supports several demonstration projects that consider EDI factors. The evaluation identified lessons learned, best practices and opportunities for improvement that could be applied to similar programming in the future. The program’s performance measurement approach could be enhanced so that the program can measure and track progress during the RD&D process towards solving specific challenges, gaps, and barriers along a range of innovation pathways, which will eventually lead to economic, social and environmental results for Canadians.

Appendix A: Evaluation Matrix

Appendix B: Evaluation Team

Michel Gould, Chief Audit and Evaluation Executive
Stephanie Kalt, Director of Evaluation
Marc Riopel PhD, Evaluation Manager
Edmund Wolfe, Senior Evaluation Analyst
An Gie Yong PhD, Evaluation Officer

The evaluation team would like to acknowledge those individuals who contributed to this project, particularly program representatives and members of the Evaluation Working Group who provided insights and comments as part of the evaluation.