Evaluation of the Energy Efficiency Program

Audit and Evaluation Branch
Natural Resources Canada
November 5, 2020

Table of Contents

• Acronyms
• Executive Summary
• Introduction
  • Program information
  • Evaluation objective, methods and limitations
• Findings
  • Relevance
  • Performance - Effectiveness
  • Performance - Efficiency
• Conclusion
• Appendix 1: Logic model
• Appendix 2: Detailed evaluation methodology

List of acronyms

AEB

Audit and Evaluation Branch

BEERS

Better Energy Efficiency Reporting System

BID

Buildings and Industry Division

CIPEC

Canadian Industry Partnership for Energy Conservation

CSPA

Canadian Steel Producers Association

DPAD

Demand Policy and Analysis Division

ECCC

Environment and Climate Change Canada

ED

Equipment Division

EEP

Energy Efficiency Program

EMMC

Energy and Mines Ministers’ Conference

Gs&Cs

Grants and Contributions

GHG

Greenhouse Gas

HD

Housing Division

IEA

International Energy Agency

ISO

International Organization for Standardization

M

Million

NBC

National Building Code of Canada

NECB

National Energy Code of Canada for Buildings

NRCan

Natural Resources Canada

OEE

Office of Energy Efficiency

PCF

Pan-Canadian Framework

PIP

Performance Information Profile

PJ

Petajoules

PTs

Provinces and Territories

SIU

Social Innovation Unit

Executive Summary

About the Evaluation

The purpose of this evaluation was to assess the relevance and performance (i.e., effectiveness, efficiency and economy) of the Energy Efficiency Program (EEP), in accordance with the Treasury Board (TB) Policy on Results. This evaluation covered all program areas included in the EEP, as well as related work of the Social Innovation Unit (SIU). The evaluation covered EEP expenditures from 2014-15 to 2018-19.

The evaluation was completed by Goss Gilroy Inc. (GGI) for Natural Resources Canada’s (NRCan) Evaluation Division. Data collection was substantively completed from January to June 2019.

What the Evaluation Found

Relevance

There is a continued need for the EEP. Energy efficiency offers a powerful demand-side management option to reduce greenhouse gas (GHG) emissions and meet Canada’s climate change commitments and targets. It also reduces energy costs, supports innovation, and increases competitiveness. While Canada has seen significant energy efficiency improvements over the years, its economy is still very energy-intensive. There remain persistent barriers to the adoption of energy-efficient policies, behaviours, and products. The EEP’s continuous engagement with targeted residential, commercial and industrial sectors is needed to achieve market transformation and behavioural changes towards energy efficiency.

Effectiveness

With the exception of the Energy Efficiency Regulations, 2016, the program’s design relies on voluntary adoption of the products and practices it supports or promotes. There is evidence that the EEP contributes to increased awareness of energy-efficient solutions and labels among Canadian consumers, homeowners, building owners or managers, and industry. The Program also works to build capacity (knowledge, skills and abilities) of specific groups to make use of these tools. The OEE’s new SIU contributes to its ability to modernize its product offering to respond to the changing needs of these stakeholders through user-centric design and experimentation.

The EEP also engages regularly with provinces and territories (PTs), local governments, other federal departments and agencies, and industry on many aspects of this programming, including efforts to harmonize and implement codes and standards. As a result, most jurisdictions generally support federal propositions on equipment regulations. With the exception of Nunavut, all PTs have adopted or adapted the energy efficiency requirements of the National Building Code of Canada (or developed their own building codes to a comparable level). Six provinces have also adopted a version of the National Energy Code of Canada for Buildings. However, there are opportunities for NRCan to improve key target groups’ understanding of the benchmark used by the EnerGuide Rating System for houses.

These initiatives have shown positive results in increased adoption of energy efficiency solutions and practices. More than 99% of the product models reported to NRCan under the authority of the Energy Efficiency Regulations, 2016, are compliant, and data indicates that consumers seek out and are positively influenced by the ENERGY STAR^® label for these products. Adoption of ENERGY STAR and EnerGuide label for homes have exceeded the Program’s targets. An estimated 27.6% of Canada’s commercial floor space is now covered by the ENERGY STAR Portfolio Manager tool for buildings. As costs remains a real or perceived barrier to the adoption of energy-efficient solutions across target groups, the EEP’s products and services are critical to building a business case for energy efficiency. To this end, the EEP also engages directly with industry sectors to assist them in their efforts to reduce energy consumption and provides funding to help industry increase its capacity to identify and adopt energy efficiency projects and practices. However, while the Program also deems implementation of energy management systems by industry to be on track, the evaluation found many opportunities to increase engagement and participation within various industry sectors.

In 2016, TB introduced a new Policy on Results that has caused shifts in performance measurement for all NRCan programs. Overall, while data suggests that most of the EEP’s program areas have shown year-to-year improvements, the revision of the Program’s indicators impeded the evaluation’s ability to assess the Program’s progress towards its ultimate outcomes for improved energy efficiency (i.e., petajoules of energy saved and megatonnes of GHG avoided) over the period of evaluation. The lack of interim targets in some program areas makes it a further challenge to determine the extent to which the program is on track. Regardless, evidence indicates that progress to date still leaves a significant gap in required reductions to be achieved by 2030. Various factors affect energy use and the performance of the Program, many of which are outside the Program’s control. Importantly, energy efficiency is an area of shared jurisdiction with the PTs. Success in achieving the EEP’s ultimate outcomes requires strong collaborations and significant action by the Program’s partners.

Efficiency

Stakeholders generally perceived the EEP to be well resourced and efficient. The Program’s resources were slightly underspent (8%) for the period of evaluation, partly resulting from a delay in start up for some Grants and Contributions (Gs&Cs). This situation was largely corrected in 2017-18. While interviewees described effective coordination mechanisms within the OEE, they noted some internal challenges with project management, decision-making processes and corporate services. Overall, the evaluation found few opportunities to increase efficiencies.

Recommendations and Management Action Plan

In response to rising energy demand and related GHG emissions, NRCan’s Low Carbon Energy Sector (LCES) is taking action to accelerate uptake of energy efficiency to promote and expand tools discussed in this evaluation, like the EnerGuide label and industrial efficiency programs, which are critical to meeting 2030 and 2050 targets and more immediately, new mandate commitments for greener homes and buildings for the broader energy transition.

Recommendations resulting from the evaluation and related management response and action plans are as indicated in the following table.

Recommendations	Management Response and Action Plans
Recommendation 1: The ADM of LCES should engage with key target groups to ensure that there is a good understanding of how to interpret and use the EnerGuide Rating System, especially with respect to rating methodology.	Agreed. The LCES concurs with this finding, and the OEE is implementing a plan to ensure different user groups can effectively interpret the EnerGuide Rating. Two actions have already been completed that respond to this recommendation:   Between November and December 2018, OEE engaged approximately 30,000 Canadians in British Columbia, Ontario, and Newfoundland and Labrador through the Carrot Rewards mobile app to evaluate how effectively the EnerGuide rating communicates energy performance. As a result, updates were made to the EnerGuide Homeowner Information Sheet to help homeowners and other users better understand the rating and associated methodology. The OEE engaged with PTs as part of a Labelling and Energy Use Disclosure working group, leading to the development and launch of a new website in 2019 that provides access to the results of EnerGuide evaluations and helps homeowners interpret and use the Rating System. LCES-OEE will engage with partners, other departments, PTs, municipalities and with Canadians through the model building code consultative process (both national and provincial) and the federal-provincial-territorial Labelling and Energy Use Disclosure working group to build their understanding of how to use EnerGuide. This will ensure that EnerGuide continues to effectively support regional home energy labelling and incentive programs as we promote related federal commitments. Due Date: December 2020.
Recommendation 2: The ADM of LCES should review and improve the EEP’s support to industry to ensure that companies with different levels of energy management capacity are engaged and effectively supported to improve their energy efficiency and achieve related GHG emission reductions. Applying a user-centric approach, this review could include engagement with small- and medium-sized companies to better understand how to best respond to their particular needs.	Agreed. LCES concurs with this recommendation. The OEE will build and expand on existing approaches in consultation with the LCES-OEE stakeholder community, including small- and medium-sized enterprises by developing new user-centric engagement content, including webinars, social media, videos, and cohort-based training. This includes:   Work with the Canadian Industry Partnership for Energy Conservation (CIPEC) on a plan for industrial engagement, with a targeted focus on industrial de-carbonization and sector-specific needs. Work with PTs through committees of the Energy and Mines Minister Conference (EMMC) to develop a long-term vision and associated action plan on decarbonizing industry sectors while improving their competitiveness. Expand its programs and tools to support companies with a range of energy management capacity. For example: Expand the ENERGY STAR for Industry Challenge program and its Energy Management Guidelines (launched in 2018 and 2019 respectively); Launch the ISO 50001 Ready program in 2020 to support companies with less energy management capacity to follow a self-paced, step-by-step process without requiring significant investment in consultants and equipment; and Deliver the new Energy Manager Program launched in summer 2019 to support organizations with different levels of energy management capacity (in those provinces that are eligible). Due Date: September 2021.
Recommendation 3: The ADM of LCES should establish indicators against which to assess the EEP’s interim progress towards ultimate outcomes in all program areas. These indicators should be supported by robust performance measurement methodologies. Where they do not already exist, targets or performance thresholds should be set for an interim date (e.g., 2023) to allow NRCan management to effectively assess progress. To the extent possible, performance indicators should be supported by methodology that enables benchmarking against past performance. If performance data indicates that the Program is off-track, it will need to either review its activities in order to meet its targets or review the targets if these are deemed unrealistic.	Agreed. LCES concurs with this recommendation. OEE is working to review and improve the methodologies used by its Divisions and to set targets and calculate annual energy savings performance. This review will inform the program’s next steps on establishing indicators against which to assess interim progress. Due Date: December 2020. LCES-OEE has also recently established the OEE Planning and Reporting Network (OEEPRN), with representatives from all OEE Divisions. This network is a venue to discuss improvements to performance measurement and target setting across the OEE, and to link to departmental and sectoral initiatives such as the Performance Information Profiles operationalization exercise and the NRCan DataHub project. Both initiatives, led by NRCan’s Strategic Policy and Innovation Sector, improve the information available to management to better facilitate program performance. By March 2021, the OEEPRN will make recommendations to LCES-OEE Management for improving how LCES-OEE tracks progress of the EEP against ultimate outcomes. Implementation of accepted recommendations will give LCES-OEE more effective tools to report on progress toward, and support the achievement of Canada’s 2030 and 2050 targets. Due Date: March 2021.

Introduction

This report presents the findings, conclusions and recommendations of the evaluation of the Energy Efficiency Program (EEP). This evaluation was conducted as specified in Natural Resources Canada’s (NRCan) Joint Audit and Evaluation Plan 2018-2021, and in accordance with the Treasury Board Policy on Results (2016). The evaluation covered the EEP’s performance and expenditures from 2014-15 to 2018-19. Total expenditures over this period were $120.2 million (M). The last evaluation to cover this area was the Evaluation of the Office of Energy Efficiency, published in 2015.

Program information

As defined in NRCan’s 2018 Departmental Results Framework (DRF), the EEP (Program 2.10) contributes to Core Responsibility 2: Innovative Solutions and Sustainable Natural Resource Development. The Program is managed by NRCan’s Office of Energy Efficiency (OEE).

The objective of the EEP is to encourage and enable the implementation of cleaner and more energy-efficient technologies, products and practices. The Program directly benefits Canadians (including the federal government) by reducing their energy consumption, lowering energy bills, and decreasing greenhouse gas (GHG) emissions. As the majority of the EEP’s programming is voluntary, the achievement of EEP expected outcomes is dependent on the actions of stakeholders. It includes the following four program areas:

1) Equipment Standards and Labelling: The objective of this area is to reduce energy use and GHG emissions from consumer, commercial, and industrial products. This program works to eliminate the worst energy performers and accelerate the introduction of more energy-efficient products in Canada’s equipment stock. It does this by working with federal and provincial partners to introduce, enforce and raise energy efficiency standards for a wide range of products and by labelling energy efficiency information for certain products through the Energy Efficiency Regulations, 2016, (the only regulatory component of the EEP). The program area also promotes energy-efficient products through the voluntary ENERGY STAR® initiative in Canada, and through the use of the EnerGuide energy disclosure label for products (Box 1). This program area is managed by the OEE’s Equipment Division (ED).

Box 1: Energy Efficiency Labelling Programs

 

 

2) Housing: The objective of this area is to reduce energy use in the housing sector by improving the efficiency of the housing stock (e.g., retrofit of existing homes and construction of energy-efficient new homes). Activities include the development and maintenance of a home energy labelling tool (EnerGuide Rating System) and high-performance standards (ENERGY STAR for New Homes and R-2000), licensing service organizations, disseminating information to industry and home owners, and collaborating with provinces, territories (PTs) and utilities to develop related energy efficiency programs and encourage the adoption of codes and standards. This program area is managed by the OEE’s Housing Division (HD).

3) Buildings: The objective of this area is to reduce energy consumption and GHG emissions in the building sector by improving the efficiency of the building stock, including commercial, institutional and multifamily residential buildings. This area provides information and benchmarking tools to building managers to improve energy performance of new and existing commercial and institutional buildings in Canada. This program area supports the National Research Council (NRC) in the development and implementation of the National Energy Code of Canada for Buildings (NECB), and collaborates with PTs on their adoption and implementation of NECB. It complements this energy code work by encouraging market uptake of existing building commissioning as well as recognition of industry leaders in the design and operation of near- and net-zero energy buildings. It provides related training, tools and information materials for use by stakeholders and partners including industry associations, utilities and building managers, including delivery of the ENERGY STAR Portfolio Manager benchmarking tool in Canada and the new ENERGY STAR building certification program. It also facilitates retrofit projects in buildings owned by the federal government through information, training, technical advice and financing mechanisms (Greening Government Services). This program area is managed by the OEE’s Buildings and Industry Division (BID).

4) Industry: This area works to advance the energy efficiency of Canada’s industrial sector by accelerating energy-saving investments and the exchange of best practices information within Canada's industrial sector. It does this by developing energy management and benchmarking tools and training, cost-sharing energy management projects with industrial facilities and other partners, and expanding the Canadian Industry Partnership for Energy Conservation (CIPEC), an industry-government network for sharing energy efficiency information and practices. This program area is directly supporting the implementation of the ISO 50001 Energy Management Systems Standard and the ENERGY STAR for Industry program in Canada, and has expanded CIPEC offerings to include webinars on energy efficiency practices. It also leads an international initiative, the CEM Energy Management Working Group, where the 18 member countries share resources to promote ISO 50001. This program area is managed by the OEE’s BID.

Since 2016, the work of the OEE has also been supported by the Social innovation Unit (SIU). The SIU builds relationships and capacities for energy efficiency policy and service innovation; generates evidence and collective learning by co-creating and testing insights and interventions (e.g. through prototyping and experimentation); and scales learning and impacts by implementing what works through service redesigns and new policy directions.

Evaluation objective, methods, and limitations

The purpose of the evaluation was to assess the relevance and performance (i.e., effectiveness, efficiency and economy) of the EEP (Program 2.10 in NRCan’s DRF), in accordance with the Treasury Board Policy on Results. This evaluation covered all program areas included in the EEP (i.e., Equipment Standards and Labelling, Housing, Buildings, and Industry), as well as related work of the SIU. The evaluation covered EEP expenditures from 2014-15 to 2018-19. During this period, NRCan replaced its previous Program Alignment Architecture (PAA) with the new DRF. With the exception of the OEE’s work in the vehicle program area, the current EEP aligns to the previous Sub-Program 2.1.4 (Energy Efficiency) in NRCan’s PAA. The OEE’s work within the vehicle program area is now part of a separate Program (Low Carbon Transportation) and was thus excluded from this evaluation. Funding provided through the EEP to the Green Municipal Fund was also excluded from the evaluation as it is scheduled to be subject to a separate horizontal evaluation.

The evaluation was completed by Goss Gilroy Inc. (GGI) for NRCan’s Evaluation Division. Data collection was substantively completed from January to June 2019.

Evaluation Questions

Evaluation Questions

Relevance
Is there a continued need for the EEP?
Effectiveness
To what extent is the EEP (and its components) achieving its immediate and intermediate outcomes? To what extent is the EEP (and its components) achieving its ultimate outcomes? To what extent have there been unintended outcomes (positive or negative) resulting from the EEP? What factors have facilitated or hindered the achievement of program outcomes?
Efficiency
Have program resources been used efficiently? Are there alternative ways of delivering the EEP to achieve similar outcomes?

Evaluation Methods

The evaluation questions and methods were developed in consultation with an evaluation working group composed of senior management and managers of each OEE Division with EEP responsibilities. In particular, the scope and focus of case studies were validated with the working group. The evaluation also took into account the results of the last evaluation and calibrated the methodology accordingly. Feedback on preliminary findings shared with the evaluation working group was considered in this report.

The following lines of evidence were used to answer the above evaluation questions:

Evaluation Limitations

In general, the EEP is highly data- and target-driven and program staff facilitated access to documentation, performance data and stakeholder information. However, the evaluation experienced some challenges resulting in limitations for a limited number of aspects. These limitations and associated mitigation actions implemented by the evaluation team are presented in Appendix 2. Overall, these limitations were successfully mitigated to produce reliable results for all lines of evidence.

Findings

Relevance

Is there a continued need for the EEP?

Legitimate Role for Federal Government

Evidence indicates that there is a necessary role for the federal government in supporting energy efficiency. Although Canada has seen significant energy efficiency improvements over the years, documentation indicates that there remain “persistent barriers” to uptake, including limited awareness of existing solutions, unleveraged opportunities for collaboration, and capacity issues, resulting in a need for energy efficiency programming. While all levels of government (federal, provincial and municipal) engage in such programming, the federal Energy Efficiency Act gives the Governor in Council the mandate to make regulations prescribing energy efficiency standards for energy-using products. It also gives the Minister of Natural Resources the mandate to support the amendment and enforcement of these regulations.

Key informant interview findings also showed that NRCan is well positioned to continue to play a role in increasing energy efficiency. NRCan is seen as a source of reliable, neutral, science-based information for consumers and stakeholders. Given that adoption of energy efficiency solutions by stakeholders is mainly voluntary in Canada, the EEP’s continuous engagement with the residential, commercial and industrial sectors is needed to achieve market transformation and behavioural changes towards energy efficiency. Overall, interviewees did not feel that the EEP overlapped with any other program.

Environmental Needs and Federal Environmental Priorities

All lines of evidence confirm there is an important and continued need for energy efficiency programming to achieve greater reduction of GHG emissions, in the context of responding to climate change. NRCan’s Energy Fact Book (2018-19) indicates that 78% of GHG emissions from human activity are from the production and consumption of energy. Canada has the most energy-intensive economy per capita among International Energy Agency (IEA) member countries. The IEA puts energy production and use at the center of global action to tackle climate change, and deems energy efficiency to be one of the most cost-efficient and quickest ways to reduce GHG emissions worldwide^{Footnote 1}.

Addressing climate change through reducing GHG emissions is a stated priority of the federal government. Canada has set a target under the Paris Agreement (2015) to reduce GHG emissions by 30% from 2005 levels by 2030. In 2017, the Government of Canada also set a new target to reduce its own GHG emissions by 40% by 2030, and by 80% by 2050 (relative to 2005 levels), as part of its Greening Government Strategy^{Footnote 2}. The recent Generation Energy dialogues and consultations confirmed that energy efficiency is also important for Canadians and is a central piece to achieve Canada’s transition to a low-carbon economy^{Footnote 3}.

Program documentation indicates that the EEP evolved during the evaluation period to place more emphasis on GHG impacts. It emphasizes the need for energy efficiency programming as an important component of Canada’s strategies to mitigate climate change, achieve clean growth, and drive innovation towards the de-carbonization of the Canadian economy. The Program’s activities and outputs are aligned with:

• The Pan-Canadian Framework on Clean Growth and Climate Change (PCF) – A 2016 federal-provincial-territorial plan to enable clean economic growth, reduce GHG emissions, and build resilience to a changing climate. The PCF sets Canada on a path to meet its target under the Paris Agreement. Some of the most important commitments in that plan relate to energy efficiency measures, including: developing increasingly stringent model building codes; working towards required labelling of building energy use; and renewed support for industrial efficiency. By improving its energy efficiency at a 2% annual rate by 2025 and 3% by 2030, Canada would achieve at least a third of its Paris Agreement targets.
• Recommendations made by the Generation Energy Council, namely with regard to expanding energy labelling, and supporting new and improved building codes^{Footnote 4}.

Economic needs

Canada’s overall energy use represents a major cost for Canadians and a driver for energy efficiency. Canada’s per capita energy use is one of the highest in the world, with energy bills representing the fifth largest Canadian household expense^{Footnote 5}. In 2016, 13% of Canada’s energy demand was residential (Figure 1). Survey findings (2013) have confirmed that Canadians are looking for ways to improve energy efficiency in their households, mostly to generate cost savings and to protect the environment^{Footnote 6}.

Investing in energy efficiency also produces net benefits to the Canadian economy by adding jobs, increasing Canada’s GDP, and producing energy-related savings for Canadian households and industries. Documentation further indicates that action to address climate change (including energy efficiency programming) offers an opportunity to innovate and to take a leadership position in the low-carbon economy. The 2016 PCF speaks of a growing global market for low-carbon goods and services worth over $5.8 trillion and offering opportunities to all sectors and regions of Canada. By contrast, there is also a predicted cost to inaction. Climate change and related extreme weather events are major disruptive factors for many economic sectors.

Key informant interview findings also showed the importance of Canada’s participation in the development of international standards that can influence or impact the competitiveness of Canadian firms and organizations, both domestically and globally. NRCan is well positioned to continue to convene Canadian participation in the International Organization for Standardization (ISO) committees, such as the ISO 50001, composed of a range of experts from industry, consultants, and various levels of governments.

Recommendations

None. There are no recommendations made related to the relevance of the Program.

Performance - Effectiveness

To what extent is the EEP (and its components) achieving its expected immediate and intermediate outcomes?

The EEP is expected to lead to a number of immediate and intermediate results, progressing from building awareness, engagement and capacity, to adoption of energy-efficient solutions and practices, to ultimately achieve energy savings in target sectors (see logic model in Appendix 1). This section analyses the extent to which these results were achieved by each program area. For the purposes of this evaluation, engagement is defined as concrete actions by the EEP to allow stakeholders to partake in various activities carried out or supported by the EEP. Capacity is defined as the ability of stakeholders to adopt technology and practices, including knowledge and best practices of energy efficiency standards, codes, labels and available technologies.

Equipment Standards and Labelling

Awareness of energy-efficient solutions. The ED of OEE is responsible for regulating the amount of energy a product uses, as well as how its energy use is reported. The Division administers both the EnerGuide energy disclosure label as well as the voluntary ENERGY STAR label for products (Box 2). Interviewees from the Division explained that regulations are implemented once the market is ready and able to comply with the new energy efficiency requirements. The ED works with the manufacturing sector to keep stakeholders informed about regulations, and collects their views on what regulations would be appropriate. In doing so, it tries to find a balance between regulating and ensuring industry is ready for upcoming regulatory changes. In 2016-17 and 2017-18, the Program reported that 95% of regulatees were aware of their obligations to comply with new/updated regulations for equipment. This was confirmed by case study findings, through which one appliance association representative explained that NRCan has continued to improve the regulatory process by giving their membership more lead-time to adapt to forthcoming changes.

Data indicates that target groups, including consumers and manufacturers, are aware of the ENERGY STAR and EnerGuide labels. In 2018, a public survey commissioned by NRCan reported a high level of visual awareness of the ENERGY STAR label amongst Canadians, particularly for home appliances^{Footnote 7}. While the survey found a lower level of visual awareness of the EnerGuide label, in both cases this result exceeds the Program’s targets with respect to awareness (Table 1).

Table 1: Awareness of Energy Efficiency Labels

Table 1: Awareness of Energy Efficiency Labels

Label	Program Target	Actual Awareness
ENERGY STAR	85% by 2021	88% in 2018
EnerGuide	50% by 2021	65% in 2018

Source: ENERGY STAR^® and EnerGuide Labelling Awareness Survey 2018

Engagement. The evaluation found that the ED’s collaboration and coordination efforts are aimed at harmonizing codes and regulations across PTs. The EEP engages with PTs (and other stakeholders) to give them opportunities to express concerns and share their views regarding upcoming regulations. The Energy and Mines Ministers’ Conference (EMMC) and the Canadian Advisory Council on Energy Efficiency each have a federal-provincial-territorial (FPT) working group focused on equipment regulations for energy efficiency. In support of activities under EMMC, in 2017 and 2018, the EEP coordinated workshops with PTs and stakeholders to develop aspirational goals regarding energy performance for residential windows, space heating and water heating equipment, with the goal of improving energy performance in the building sector. This ultimately led to the development of the Market Transformation Roadmap for Energy Efficient Equipment in the Building Sector, endorsed by FPT Energy Ministers in 2018^{Footnote 8}. Governments have committed to work collaboratively with industry and other stakeholders to implement activities in support of the roadmap. The ED also holds quarterly calls with Canadian utilities on promotion, compliance and enforcement of ENERGY STAR brand use.

Target groups have capacity to adopt energy efficiency technology, products and practices. The ED does not have immediate objectives related to this program impact area. Interviewees explained that consumers’ adoption of standards based on voluntary programs, like ENERGY STAR, depends on the capacity of stakeholders to invest in and install energy-efficient equipment. At present, the Program does not maintain data to assess this capacity.

Responsiveness to changes affecting regulatory and market environments. According to key informants, challenges associated with the adoption of energy-efficient products include the availability, accessibility, awareness, affordability and acceptance of these products. The work leading to the Market Transformation Roadmap is an example of how the Program is sensitive to market pressures, at least with respect to windows and heating technologies. However, interviewees noted that Canadian regulations cannot always reach transactions in the context of online shopping, or when Canadians buy directly from other jurisdictions. According to one interviewee, this trade is not high volume at this point but may increase.

Adoption and implementation of energy-efficient technologies, products and practices in Canada. Generally speaking, the adoption of technologies, products and practices will differ between regulated products and voluntary measures such as ENERGY STAR labels. While regulations can enforce the widespread application of certain product standards, voluntary measures depend strongly on stakeholders’ willingness to implement them. Performance data (2017-18) shows that practically all (>99%) energy-using products reported to NRCan under the authority of the Energy Efficiency Regulations, 2016 comply with the energy efficiency standards, exceeding program targets (95% by March 2021).

While the EEP planned to evaluate the market penetration of some product categories in 2018-19, recent estimates of the penetration of high efficiency equipment in the Canadian market were not available at the time of evaluation. However, there is evidence that many Canadians are sensitive to labelling and that the labels are positively affecting their consumer behaviors. Results of the 2018 public survey found that 81% of consumers who purchased certified products were influenced by the ENERGY STAR symbol. This exceeds the Program’s target to have 70% of Canadians using the label in decision-making by 2021.

PTs adopt and/or adapt increasingly stringent energy codes and regulate energy use. Provinces generally support federal propositions on equipment regulations and will adapt their standards to meet federal ones, according to interview findings. Provinces also routinely engage in field-testing of higher efficiency equipment. NRCan interviewees noted that six PTs also regulate products. The EEP works with provinces to harmonize federal regulations with provincial systems.

Housing

Awareness of energy-efficient solutions. Energy-efficient solutions promoted by the HD include the R-2000 and ENERGY STAR for New Homes building standards, and the EnerGuide Rating System (Box 3). Evidence indicates that provinces, municipalities, and home builders are all highly aware of these labels and NRCan’s underlying tools and standards. Over 60 external housing programs, mostly from provincial and local governments, rely on EnerGuide or ENERGY STAR to operate. They are often used as a step in the application process and/or as a condition to obtain a grant. The following are examples of such programs:

• The City of Toronto’s Home Energy Loan Program uses ENERGY STAR as the basis for low-interest loans (up to $75,000) to cover the cost of home energy improvements.
• The Vancouver Heritage Foundation’s Heritage Energy Retrofit Grant Pilot Program uses ENERGY STAR as the basis for an incentive of up to $5,000 per heritage house.

A recent public survey (2018) shows a lower level of awareness of ENERGY STAR for New Homes among the Canadian public; only 31% of Canadians are aware that homes, in addition to products, can also be ENERGY STAR-certified. This is actually a decrease from the level of awareness reported in 2016 (36%). However, this may be explained by the sporadic nature with which Canadians engage with the new housing market (about 250,000 new homes are built in Canada per year).

Engagement. Engagement with other jurisdictions, including PTs and municipalities is critical to the work of the HD. The Division engages with these jurisdictions and other stakeholders on a regular basis to promote or collaborate on programs or regulations that relate to the energy efficiency of homes, such as the BC Energy Step Code (Box 4). This includes provision of data on the use of ENERGY STAR for New Homes and the EnerGuide Rating System within their jurisdictions to allow PTs and municipalities to develop and monitor the progress of their policies and regulations.

The HD also supports NRCan and other departments such as Crown-Indigenous Relations and Northern Affairs Canada, Indigenous Services Canada, Environment and Climate Change Canada (ECCC), the Canadian Mortgage and Housing Corporation (CMHC) in their programs, including those focused on northern, remote, and Indigenous communities, by providing policy expertise and technical support on residential energy efficiency. For example, it has supported the National Research Council (NRC) to develop a technical guide for housing construction in northern and remote areas that will include measures to increase energy efficiency and address other priorities, such as adaptation to climate change. The Division also engages with communities by funding projects to improve energy efficiency in northern, remote, and Indigenous communities. From 2016-2019, it made investments in Indigenous communities in the Northwest Territories, including pilots for community energy planning and retrofits of community buildings. From 2019-2021, funds will be provided to help design energy-efficient and affordable housing models for mineworkers in Nunavut, and to support housing energy efficiency retrofits in Indigenous communities in BC and Alberta.

A number of interviewees mentioned that there could be greater engagement between the HD and the construction sector. Currently, the Division works closely with the Canadian Home Builders’ Association to support their ongoing capacity building efforts, including the development and implementation of their Net Zero Energy brand, which is built on the foundation provided by the EnerGuide Rating System. In addition, ongoing HD programming will be providing dedicated contribution agreement funding to support the market transformation towards high-performance construction and renovation. More engagement would allow the EEP to increase its reach, and in some cases, identify additional opportunities to improve products like ENERGY STAR for New Homes and the EnerGuide Rating System.

Responsiveness to changes affecting regulatory and market environments. The OEE works to ensure that the EnerGuide Rating System, ENERGY STAR for New Homes and R-2000 meet current client needs. For instance, although EnerGuide was first developed as a stand-alone design tool, it evolved considerably over time to support new applications, as is evidenced by its use as part of a variety of partner-led programs. This demand is expected to continue to rise as the EnerGuide tool and NRCan’s energy-efficient construction standards are increasingly used for regulatory purposes by provincial and municipal governments, such as mandatory energy use labelling and ‘net-zero energy ready’ building code compliance. Demand to access both raw and aggregate NRCan housing data is also likely to grow as municipalities and provincial governments look for ways to monitor progress of their energy efficiency policies.

This increased demand requires thinking about new ways to support the use of data by industry, provinces and municipalities. Recognizing this, the OEE is currently developing a strategy to facilitate access to EnerGuide data directly by users such as PTs, utilities, and homeowners using modern information technology approaches (e.g., cloud-based, open access applications). Updating the EnerGuide process requires reviewing the IT component and reducing the non-scalable, labour-intensive parts of the process (e.g., time consuming data collection).

Meeting client needs also requires clarity in communication and interpretation of data. The evaluation found that many stakeholders may not fully understand the reference house approach used in the EnerGuide Rating System. EnerGuide assesses the performance of a home in part by comparing it against what the rating would be if the home had been built to the requirements of the NBC, referred to on the label as a “typical new house”. This reference house approach is consistent with the approach used to develop new model codes that target net-zero energy ready performance. However, NRCan representatives indicated that this reference house is often mistaken by stakeholders to represent the rating of an actual average or typical home. This confusion may explain why many of the evaluation’s interviewees (industry, non-governmental organizations [NGOs] and municipal governments) perceived that this comparison is not always suitable for their needs. For example, some municipal representatives perceived that the EnerGuide ratings are not the best indicators to assess progress against their desired long-term targets (based on net-zero homes). External interviewees also felt that NRCan could have more activities around the promotion of net-zero home construction. This is unexpected, as the EnerGuide Rating System approach to compare performance to a reference house is currently being used as the basis for performance targets under the new NBC net-zero energy ready step codes. In addition, program representatives indicated that this approach was recommended to NRCan as part of a broad stakeholder consultation (2010-2012) that included PTs, and industry representatives.

Target groups have capacity to adopt energy efficiency technology, products and practices. The HD set a program target of 85,000 homeowners and buyers “to embrace energy efficiency practices, homes and technology” by March 31, 2018. According to program data, this capacity target was exceeded; more than 160,000 homes received ENERGY STAR or EnerGuide home labels from 2016-2018 (Table 2).

Table 2: ENERGY STAR and EnerGuide Label for Homes: Performance against Targets

Table 2: ENERGY STAR and EnerGuide Label for Homes: Performance against Targets

Targets	Actual Performance
22,000 labels issued by 2014-2015	83,112 labels (2015)
85,000 labels issued between 2016 and 2018	162,066 labels (2018)

Source: NRCan Program data (BEERS Reporting System).

Evidence indicates that stakeholders appreciate the flexibility of ENERGY STAR for New Homes to allow builders to choose how they want to meet an overall performance target (20% better than code). ENERGY STAR also allows those that build multiple homes of the same design to benefit from an economy of scale by applying the same construction standard without the need to evaluate the energy efficiency of each home individually. EnerGuide also supports the ENERGY STAR for New Homes program. However, EnerGuide’s other applications, such as testing the energy efficiency of individual existing homes, are relatively expensive. The hiring of an Energy Advisor to conduct EnerGuide assessments creates a barrier to its adoption. While some representatives of the construction industry have thus promoted the idea of a simpler, lower cost EnerGuide, the evaluation found that its current design is well adapted for existing homes and new homes with unique designs.

Adoption and implementation of energy-efficient technologies, products and practices in Canada. Performance data shows that the adoption rates of ENERGY STAR and EnerGuide have exceeded targets (Table 2). Performance data shows that energy efficiency savings derived by home improvements incentivized by programs implemented by provinces and utilities have been significant; participating homeowners that had undertaken retrofits reported average savings of 24.8% by 2014-15 and 22.6% by 2015-16 (exceeding targets of 20%). EnerGuide provides an important support to these external programs, providing a means to calculate and report on impacts. Particularly given success to date, the evaluation found that future targets in this area should be more ambitious and aligned with broader climate change goals.

All PTs (with the exception of Nunavut) have adopted or adapted the energy efficiency requirements of the current NBC – or developed their own building codes to a comparable level. Through the PCF (2016), FPT governments have committed to collaborate, develop and adopt increasingly stringent model building codes, starting in 2020. The goal is to have PTs adopt model building codes that are net-zero energy ready by 2030. Achieving this goal will ensure that our future building stock has a much lower carbon footprint. Work is underway to support PTs and to advance the code development process, which is led by NRC. By encouraging the use of new technologies and practices before they are required under new model codes, labels such as ENERGY STAR and R-2000 and related initiatives are helping to prepare the building sector for these more stringent codes.

Buildings

Awareness of energy-efficient solutions. Evidence suggests that when collected and shared, building performance information becomes a powerful dataset that can drive significant emissions reductions, energy efficiencies and cost savings. However, these improvements are only made possible when access to reliable building performance data is provided to owners, occupants, and legislators. The OEE’s BID oversees the implementation of multiple tools and services to encourage and assist owners and managers of buildings to increase the energy efficiency of their buildings (including commercial, institutional and multi-family residential buildings), including the ENERGY STAR Portfolio Manager and associated training (Box 5). Specifically, Energy STAR Portfolio Manager can be used by building managers to benchmark the performance of buildings. It is used by other provincial and municipal programs (e.g. Ontario’s Energy and Water Reporting and Benchmarking for buildings, Richmond’s Building Challenge, Ontario’s Race to Reduce, etc.) and by NGOs or private sector certification programs for buildings, such as the widely used BOMA BEST and Leadership in Energy and Environmental Design (LEED). ENERGY STAR Portfolio Manager is also used by the Canada Green Building Council (CaGBC)’s Disclosure Challenge, a publicly-available, interactive, online mapping tool hosted by CaGBC to display building energy use data and demonstrate the value of data transparency.

 

The evaluation found that awareness of the ENERGY STAR Portfolio Manager is high: a recent survey of building sector specialists indicated that 78% were aware of the ENERGY STAR Portfolio Manager^{Footnote 9}.

Engagement. The evaluation found that the Program has conducted extensive engagement activities with PTs, municipal governments and the private sector. For example, performance data indicates that nine PTs are actively participating in (NECB) working groups. The BID also engages with the private sector to encourage the adoption of high-efficiency products and measures. For instance, OEE has a contribution agreement with BOMA Canada (the industry association that represents building owners and managers) to support their Net Zero Challenge award program that recognizes industry leaders and showcases successes. This Division has also established the High-performance Building Community of Practice whose 100+ members participate regularly on WebEx events and share their experiences.

One external interviewee expressed that NRCan should be sharing its expertise on energy efficiency more broadly with other federal departments and other public organizations to help them improve the energy efficiency of their building infrastructure. The evaluation found that, as part of the Federal Buildings Initiative (FBI), BID facilitates energy efficiency retrofit projects in buildings owned and managed by the federal government. The ENERGY STAR Portfolio Manager is a key tool proposed by NRCan to help federal departments achieve their energy and GHG goals. NRCan has also provided training for federal building specialists to help them in their efforts to reduce the energy consumption of the buildings under their responsibility, and helps federal organizations enter into third-party energy performance contracts to implement energy efficiency improvements in federally owned and operated facilities across Canada. According to performance data (2017-18), 40 federal organizations (and sub-units with delegated energy management authorities) had been provided with information, tools and services to improve the energy performance of federal buildings, against a target of 70 by 2022. The Program has recently set a target to achieve 750 terajoules/ year in energy savings from federal buildings by 2030. While projects are ongoing, it is too soon to report on results.

Target groups have capacity to adopt energy efficiency technology, products and practices. NRCan has made efforts to support the adoption of ENERGY STAR Portfolio Manager by implementing Canadian technical content and providing technical documentation and webinars and information sessions to users. As part of the FBI, a total of 286 person-days of training on energy management have been delivered to federal building managers. In addition, NRCan produces and maintains tools and training materials to support PTs in the adoption and compliance of the codes in their jurisdictions, including the development of a building energy modelling software application (CANQUEST) for building designers to facilitate the code compliance process.

All external interviewees perceived ENERGY STAR Portfolio Manager as meeting their benchmarking needs, allowing them to collect information that is integral for monitoring their buildings’ energy use. Some individuals further explained that the tool provides the information needed to inform cost reduction strategies. For example, in one school district, electricity and water represent 80% of their carbon footprint; having accurate information regarding those two components is important for decision-making related to cost and energy reductions. Users who have adopted the tool for external programming – supporting others’ use of the tool for programs that promote energy efficiency – highlighted that the tool provides the appropriate information needed to assess how buildings are faring and also emphasized the tool’s comparative feature as meeting an explicit need. For example, in the case of energy efficiency competitions, the ability to compare buildings is at the center of decisions as to which buildings will receive awards.

However, interviewees noted that use of the tool requires having on-site capacity (i.e., staff) to enter data on performance and leverage data for the adoption of energy efficiency solutions. This on-site capacity is not always in place, which acts as a barrier to building owners and managers in making full use of the tool.

Adoption and implementation of energy-efficient technologies, products and practices in Canada. Results of a recent survey (2017) of industry and building specialists indicate that 33% reported using the ENERGY STAR Portfolio Manager tool. Of those who are not using the ENERGY STAR Portfolio Manager tool, 85% reported that they are very (28%) or somewhat (57%) interested in this benchmarking tool. Survey results also suggest an opportunity for the EEP to fill a particular gap, as 59% of the buildings specialists who mentioned they are interested in ENERGY STAR Portfolio Manager are not using other programs to benchmark the energy consumption of their buildings.

The Program is generally on track to meet its targets with respect to the coverage of the ENERGY STAR Portfolio Manager. Performance data indicates this tool was used in more than 20,000 buildings across Canada in 2018 (against a target of 22,800 buildings for 2021 and 40,000 for 2026). According to the 2017-18 performance reports, 234 million square metres (MSM) of commercial and institutional building floor space is now registered in the ENERGY STAR Portfolio Manager (target is 300 MSM for 2021). This represents an estimated 27.6% of Canada’s commercial floor space. An analysis of the ENERGY STAR Portfolio Manager account activity provided by the Program also indicates an average growth rate of 40 new accounts per month.

Specific to the FBI, data indicates that the Program has helped the Department of National Defence implement energy and GHG reduction projects at three of its bases (against a target of 10 bases by 2021). The BID also helped three federal government departments develop audits and energy management plans (against a target of 20 by 2021).

PTs adopt and/or adapt increasingly stringent energy codes and regulate energy use. The EEP’s activities include engaging with PTs on adopting energy-efficient codes and regulations for buildings. Overall, results show that the Program is on track to meet its targets in this area. As shown in Table 3, six provincial jurisdictions have now adopted a version of the NECB. This exceeds the EEP’s target (i.e., five PTs by 2021). Performance data also shows that a total of 41.6% of new building floor space in Canada was covered by the 2015 NECB in 2017, against a target of 50% by March 31, 2021.

Table 3: Jurisdictions Enforcing the National Energy Code of Canada for Buildings

Table 3a: Jurisdictions Enforcing the National Energy Code of Canada for Buildings

Jurisdiction	Version of NECB Code Enforced
BC	NECB 2015
Alberta	NECB 2017
Saskatchewan	NECB 2017
Manitoba	NECB 2011
Ontario	NECB 2017 (equivalent)*
Nova Scotia	NECB 2017

* Note: Ontario does not adopt the NECB but OBC-SB-10 is now equivalent to the requirements of NECB 2017.

Industry

Awareness of energy-efficient solutions. The BID also works to advance energy efficiency by accelerating energy-saving investments and the exchange of best practices information within Canada's industrial sector. The Division supports the CIPEC and directly supports the implementation of the ISO 50001 Energy Management Systems Standard (Box 6) and the ENERGY STAR for Industry program in Canada. Industrial facilities located in Canada can earn ENERGY STAR certification and display the ENERGY STAR symbol, similar to those seen on appliances and electronics in the marketplace.

In 2017, CIPEC included 2,386 facilities leaders and over 50 trade associations in 20 industrial sectors. The EEP supports the organization of CIPEC meetings, develops content for CIPEC’s website, and supports its social media presence (Twitter and LinkedIn) and newsletter (“Heads Up CIPEC”). The number of subscribers to the newsletter is estimated to be over 5,000.

The BID also organizes CIPEC’s Energy Summit every two years, which includes presentations from expert speakers, energy-management workshops, networking and awards. Most industry energy specialists are aware of CIPEC and ISO standards: a survey of industry energy specialists (2017) shows that 68% of respondents were aware of the CIPEC and 76% were aware of ISO 50001. About 68% were also aware of the ENERGY STAR for Industry label.

The EEP also promotes the use of other relevant awareness tools, such as Dollars to $ense (D2$) energy management workshops provided by the Canadian Institute for Energy Training. Since 1997, over 30,000 representatives of industrial, commercial and institutional organizations from across Canada have enrolled in these workshops. The BID also provided support to develop an employee awareness program toolkit to help engage industry staff in energy management initiatives, and build an energy-saving culture in the workplace.

Engagement. Through CIPEC, BID’s Industrial Partnerships unit conducts outreach and engagement activities with 18 industrial sectors under its Industrial Energy Management Program. The Industrial Partnerships unit develops and coordinates action plans for each sector, including updates on past engagement activities and responses to ongoing issues (e.g., United States (US) tariffs on steel and aluminum). The evaluation’s case study on the steel sector provides a more in-depth description of the effectiveness of the EEP’s engagement with one industrial sector (Box 7).

Overall, there is an opportunity for BID to enhance its engagement activities and reach towards various industrial sectors. According to interview findings, there is an opportunity for more involvement on the part of senior-level management officials on CIPEC, from both the federal government and industry. There is also an opportunity to increase the reach of CIPEC. BID has recognized this gap and is examining ways to enhance its presence on social media. According to internal interviewees, the Division is now considering learning more about the way stakeholders are accessing and consuming information, in order to adjust its communication products.

Responsiveness to changes affecting regulatory and market environments. Evidence indicates that the BID monitors market environment changes and establishes actions plans for key industry sectors. For example, a survey administered to companies in the steel sector in response to the recent trade issues with the US identified barriers faced by the steel industry that hinder uptake of energy efficiency projects. These barriers include competition with other priorities for capital within the organization, lack of data on return on investment (ROI) required to build a business case for energy efficiency, and a lack of competent staff to support energy efficiency project development and management. Interview findings suggest these challenges are similar to those faced by other Canadian industries. However, survey results also point to different needs specific to the steel sector’s energy efficiency training and support. As a result, BID is working to develop a more customized offering for the steel sector based on the availability of energy efficiency resources.

Target groups have capacity to adopt energy efficiency technology, products and practices. Through its Industrial Energy Management Program, the BID provides funding to help industry increase its capacity (knowledge, skills and abilities) to identify and adopt energy efficiency projects and practices such as the implementation of ISO 50001 and Energy Management Information Systems. Specifically, project funding supports industrial energy managers’ capacity in developing convincing business cases for energy efficiency projects as a cost-saving measure. This funding was expanded in 2017-2018, enabling the Program to support 21 energy efficiency projects in that fiscal year.

The EEP encourages the uptake of ISO 50001 and implementation of other industrial energy management projects by promoting available tools and services, such as D2$ energy management workshops and RETScreen (Box 8). However, the evaluation found that a number of challenges and opportunities remain to be addressed. In a survey administered by the BID, CIPEC leaders identified the availability of competent energy efficiency staff as a key barrier to energy efficiency. Case study findings suggest that while larger companies in the steel sector generally have the capacity to adopt energy-efficient standards and systems, smaller companies may not. They may also lack energy management champions to influence the adoption of solutions. Case study interview respondents also pointed to the need among small- and medium-sized enterprises for direct support from NRCan experts and advisors for the following:

• Identification of energy savings and energy efficiency project opportunities;
• Demonstration of ROI for identified energy efficiency project opportunities;
• Development of business cases for energy improvement; and
• General assistance during project implementation and follow-up.

In addition, since the reporting of GHG emissions to ECCC is now mandatory from prescribed facilities, industry representatives interviewed asked for more support from the federal government to better integrate technical considerations around GHG emissions reduction into business cases for energy efficiency projects. They also identified an opportunity for the Program to broaden its capacity building activities to more sectors and increase its focus on regions outside of Ontario, Quebec and BC.

Adoption and implementation of energy-efficient technologies, products and practices in Canada. Performance data indicates that BID is on track to meet its targets with respect to the adoption of energy efficiency systems and standards for industry^{Footnote 10}. As of 2016-2017, the Program deemed the percentage of industrial energy use managed through ISO 50001 and ENERGY STAR to be on track. In 2017-18, internal reports indicate that industry had implemented 36 ISO 50001-certified energy management systems. This places the percentage of industrial energy use managed through ISO 50001 at 1.1% (against a target of 1.6% by 2021).

Social Innovation

Created in 2016, the SIU is located within the Demand Policy and Analysis Division (DPAD) of the OEE. The SIU is dedicated to: developing innovative ways to achieve EEP outcomes; engaging with stakeholders and partners through preliminary research and data collection; co-developing interventions; testing; and prototyping supporting implementation of concepts and ideas. The SIU helps the OEE conduct environmental scans and to modernize its product offerings. It contributes to the OEE’s ability to respond to changes through user-centric design and experimentation.

While some are ongoing, a number of SIU projects have already led to improvements to product design and delivery. For example, the SIU joined efforts with the City of Edmonton and the Alberta Co-Lab to improve the EnerGuide for homes. The project involved a small ethnographic study of the energy audit process with 24 participants, a co-design phase with the study’s participants and project partners to identify insights, and the testing of prototypes. This information led HD to make changes to the EnerGuide report.

The SIU also worked with the OEE’s Divisions to plan and implement Carrot Rewards campaigns. The Carrot Rewards App, developed through support by several government and non-for-profit organisations, rewards positive behaviours through loyalty points. Over the past 3 years, collaboration with the OEE has allowed for the creation of an offer stream on energy efficiency. Besides allowing the OEE to leverage its specialized technical knowledge, use of the Carrot Rewards App allowed the OEE to reach a large pool of Canadians in three provinces (over one million users in BC, Ontario and Newfoundland and Labrador). The first set of projects (campaigns and offers) focused on assessing consumer knowledge and awareness of energy efficiency and OEE tools like ENERGY STAR and EnerGuide. This resulted in an increase in views and engagements on OEE outreach platforms and content (e.g., social media). Analytics indicate that an increased awareness and targeted information resulting from SIU Carrot Rewards collaborative projects are positively impacting consumer behavior; an estimated 18,000 Carrot Rewards App users entered their postal code on the OEE website to search for a service organization delivering EnerGuide evaluations for existing homes following an SIU campaign or offer.

The SIU has also provided funding support for innovative projects and holistic approaches to advancing energy efficiency in Canada. For example, the SIU funded a national NGO (Quality Urban Energy Systems of Tomorrow - QUEST) to kick-start the development of an Energy Efficiency Scorecard for Smart Communities of all sizes (including three Indigenous communities). Evidence indicates the scorecard will be the first tool of its kind to evaluate how multiple community stakeholders are contributing to Smart Energy Communities. The scorecard is expected to result in: a standard set of key performance indicators that define the pathways to Smart Energy Communities; methodology to evaluate progress; a Scorecard Report; and a web-based dashboard containing a roadmap for action for the featured communities to advance along the pathway to becoming a Smart Energy Community. The SIU is currently supporting the ongoing testing and refinement of the scorecard, including its potential use in Indigenous communities.

According to interviewees, having a team dedicated to social innovation within the OEE is needed and useful because the OEE's Divisions otherwise have limited time and capacity to pursue important social innovation initiatives. Staff from the HD felt they acquired new knowledge from their collaboration with the SIU and were now better equipped to incorporate elements of user-centred design into their own activities. NRCan interviewees notes that the SIU is able to build collaborations and has the resources and time to develop relationships with different groups to the benefit of teams within the division. In supporting external partners’ initiatives, the SIU also helps fill gaps in understanding and contributes to ambitious projects that can further generate insights and impact towards greater energy efficiency.

Looking forward, the SIU targets having 100% of social innovation projects include prototyping and testing of interventions. The baseline for this measure is 57% (2016-17). In 2017-18, the SIU reached 75% of its target.

Recommendations

Recommendation 1: The ADM of the Low Carbon Energy Sector (LCES) should engage with key target groups to ensure that there is a good understanding of how to interpret and use the EnerGuide Rating System, especially with respect to rating methodology.

Recommendation 2: The ADM of LCES should review and improve the EEP’s support to industry to ensure that companies with different levels of energy management capacity are engaged and effectively supported to improve their energy efficiency and achieve related GHG emission reductions. Applying a user-centric approach, this review could include engagement with small- and medium-sized companies to better understand how to best respond to their particular needs.

Ultimate Outcomes

Improved energy efficiency in target sectors

The EEP’s ultimate outcome is to improve energy efficiency in target sectors (i.e., housing, consumer, commercial and industrial products, buildings, and industry). Over the period of evaluation, the OEE measured its performance for this outcome by estimating the Program’s impact in terms of PJ of energy saved. To the extent that GHG reductions co-relate to energy savings, the EEP also measures its performance in terms of Mt of GHGs avoided from energy use (by converting the PJs of energy saved into Mt of GHGs avoided).

Since 1990, the OEE has produced the annual report Energy Efficiency Trends in Canada. Linked to the performance of Canada’s economy as a whole, this report shows continuous improvement in energy efficiency (Figure 2). While these results cannot be directly attributed to the work of the EEP (e.g., also includes gains in transportation efficiency), internal performance reporting provides supporting evidence that the work of the EEP has contributed to these results.

Figure 2. Secondary energy use with and without energy efficiency improvements (1990-2016)^{Footnote 11}

Source: Natural Resources Canada, 2018. Energy Efficiency Trends in Canada 1990–2016.

Text Version

Comparison of secondary energy use (PJ) in Canada with and without energy efficiency improvements from 1990-2016. In 2016, this difference equated to 2089.9 PJ of energy saved with a related 112.1 Mt of GHG avoided. Annual amounts of energy use with and without energy efficiency improvements are presented below.

Figure 2. Secondary energy use with and without energy efficiency improvements (1990-2016)

Year	Energy use without energy efficiency improvements (PJ)	Energy use with energy efficiency improvements (PJ)
1990	6957.22	6957.22
1991	6961.87	6844.08
1992	7143.97	6942.33
1993	7320.95	7066.63
1994	7600.41	7334.29
1995	7961.08	7547.07
1996	8178.43	7678.75
1997	8385.55	7799.66
1998	8361.73	7614.97
1999	8719.58	7801.46
2000	9046.08	8090.49
2001	8924.72	7848.21
2002	9276.41	8111.73
2003	9428.50	8291.54
2004	9635.92	8525.68
2005	9741.26	8453.12
2006	9625.89	8329.79
2007	10006.40	8733.90
2008	9898.91	8594.55
2009	9550.09	8312.94
2010	9835.44	8501.02
2011	10033.45	8748.66
2012	10071.07	8705.16
2013	10405.43	8881.33
2014	10692.24	8961.44
2015	10739.03	8932.24
2016	10876.31	8786.43

As part of the implementation of NRCan’s DRF that started in 2016-17, the Program adopted a new target and the method by which ultimate outcomes are calculated. The target came into effect on April 1, 2018. The Program has actual results to report against this target from 2017-2018. Rather than a positive five-year trend, its current target is to achieve an annual energy savings of 600 PJ in 2030. The EEP estimates that this would equate to close to 42 Mt of GHG avoided in 2030^{Footnote 12}. Table 3 presents this target broken down by program area and the most recent reported results (2017-18).

Table 3: Energy Efficiency Targets and Reported Results

Table 3b: Energy Efficiency Targets and Reported Results

Program Area	Energy Savings (PJ/yr)			GHG Avoided (Mt/yr)
	Target		Actual Result  (2017-18)	Target		Actual Result  (2017-18)
	March 31, 2021	March 31, 2030		March 31, 2021	March 31, 2030
Buildings		132.6	3.18	[Estimates of GHG avoided are derived from data on energy savings (PJ). Given recent changes in methodology, calculations of GHG avoided in 2017-18 had not yet been completed at the time of writing.]
Equipment	42.1	219.5	10.13
Equip. Regulations	15.5	153.1	0.37
ENERGY STAR	26.6	66.4	9.76
Housing		133.7	3.14
Industry	17.9	117 PJ	3.5
Total EEP	≥63.6	602.8	19.95	n/a	n/a	n/a

Source: OEE Self-Reported Performance (BEERS Database)

As of 2017-18, the OEE estimated total annual energy savings across all program areas of about 20 PJ. Results to date reflect the ramping up of efforts against the new target since 2016-17. The Program spent the first two years laying the groundwork for delivery; a lag in results achieved is expected as secondary energy users take time to adopt the new codes, standards, regulations and certifications currently being developed by the Program.

While data suggests that most program areas have achieved year-to-year improvements, the revision of indicators creates issues for data continuity that prevent comparison by program area over the full period of evaluation. The lack of interim targets for some program areas makes it a challenge to know the extent to which the Program is on track. Related data on GHG avoided was also not available for most program areas at the time of evaluation. Regardless, it is clear that overall performance to date still leaves a significant gap in required energy savings to be achieved by 2030.

Representatives for the Equipment program area have also expressed concerns about the capability of meeting the performance threshold due to methodology used to calculate performance, as well as other factors. Progress towards the target for PJ savings associated with the Energy Efficiency Regulations, 2016 was developed using a different model and data than what was used to develop the target. Further, policy decisions were taken during the regulatory development process for amendments to the Energy Efficiency Regulations, 2016 which led to different standard stringencies, timelines, and product categories, than originally planned. These and other factors will affect this program area’s ability to meet the overall 2030 PJ target.

The Program’s revised target is ambitious and aligned to commitments in the PCF. Efficiency measures currently account for about a third of the reductions in GHG emissions planned under the Framework. However, the specific target set by the Program assumes that PTs adopt or adapt codes, standards, and regulations as per the schedule agreed to in the Framework. The plan in place to achieve the ultimate outcome thus relies heavily on the voluntary actions of PTs and other stakeholders, which necessitates effective collaboration. Interview respondents noted that recent political changes may put the achievement of the EEP’s ultimate targets at risk. As noted in a following section, these are just some of the numerous factors that may facilitate or hinder the achievement of Program outcomes. Given this, it will be important to use sound methodology to closely track interim progress. If the Program is found to be off-track, it will need to either review its activities in order for the Program to meet its targets or review the targets if these are deemed unrealistic.

Recommendations

Recommendations 3: The ADM of LCES should establish indicators against which to assess the EEP’s interim progress towards ultimate outcomes in all program areas. These indicators should be supported by robust performance measurement methodologies. Where they do not already exist, targets or performance thresholds should be set for an interim date (e.g., 2023) to allow NRCan management to effectively assess progress. To the extent possible, performance indicators should be supported by methodology that enables benchmarking against past performance. If performance data indicates that the Program is off-track, it will need to either review its activities in order to meet its targets or review the targets if these are deemed unrealistic.

To what extent have there been unintended outcomes resulting from the Energy Efficiency Program?

Unintended social and economic benefits (or co-benefits)

In the context of this evaluation, unintended outcomes are defined as outcomes that have not been identified as intended outcomes in the Program’s logic model. For the EEP, we found that this includes a number of co-benefits intended by purposeful action. Most notably, the OEE promotes cost savings as part of the business case for energy efficiency measures to increase adoption by target stakeholders. Savings are derived from avoided energy costs for consumers and businesses. These benefits are also referenced in some performance reports. For example, the OEE anticipated that by improving energy performance in 24,000 buildings, the 2011 version of the national building code would lead to $70M in cost savings for building owners in 2016. Since April 2015, the ENERGY STAR Portfolio Manager benchmarking tool has generated an estimated $26M per year in energy cost savings for building owners, managers and operators. The Program reports annual cost savings of $2M per facility for businesses adopting the ISO 50001 Energy Management Systems Standard.

There are also other unintended economic impacts to be considered. As part of the EnerGuide for Homes process, energy advisors test the energy efficiency of Canadian homes and calculate the energy savings that would be generated based on proposed energy efficiency upgrades. While it is not an explicit objective of the Program, these upgrades and renovations also involve an economic impact on the Canadian economy, i.e., to the construction industry that undertakes these renovations (construction and buildings materials). The evaluation team assessed the economic impacts of EnerGuide for Homes using program data on specific renovations implemented following an assessment (Table 4).

Table 4: Estimated Contribution of EnerGuide to the Canadian Construction Industry (2014-2019)

Table 4: Estimated Contribution of EnerGuide to the Canadian Construction Industry (2014-2019)

Component	Number of Homes Reporting Renovations after EnerGuide Assessment	Value of Renovations (estimated)
Windows and Doors Replacements	31,004	$496 million
Space Heating	122,939	$615 million
Attic upgrades	57,063	$85 million
Total		$1.196 billion

Source: Calculations based on EnerGuide data

Based on conservative assumptions, the resulting annual averages (spread evenly over the period of 5.08 years) amount to $235.3M per year in renovation expenditures. Homeowners recover most of these renovation costs at the moment of house resale^{Footnote 13}. There is no empirical data on the extent to which these impacts are directly attributable to the EnerGuide assessment. For example, many provincial programs also contribute to these outcomes by using EnerGuide in their program administration. However, if only half of the homeowners decide to renovate their homes as a result of the EnerGuide assessment (e.g., half would have renovated their home whether they had used EnerGuide or not), this would still equate to more than $100M per year of unintended, incremental economic impacts.

Documentation indicates that improved standards for homes and buildings also contribute to healthier indoor environments by reducing cold drafts and indoor pollutants. The Program also contributes to the long-term de-carbonization of the Canadian economy by contributing to a reduced reliance on fossil fuels. While program performance data does not calculate the social cost savings of GHG avoided associated with the reductions, data from various Canadian and American sources suggests that the social cost savings of GHG in 2020 can be estimated at $40 per tonne of CO². On this basis, social cost savings associated with the GHG avoided would amount to approximately $200M avoided between 2014 and 2019.

What factors have facilitated or hindered the achievement of program outcomes?

Interview findings indicated a variety of factors that affect energy use and EEP programming. Canada has energy-intensive industries, a vast geography, high standards of living and minimal constraints on space occupation, all of which drive greater energy consumption. Energy use is also impacted by weather conditions. Canada has a generally cold climate which increases energy usage (heating being energy-intensive). Extreme hot or cold weather can also drastically increase the heating and cooling demands, which can counterbalance the energy and GHG reduction impacts of the Program in a given year: annual building energy use and associated GHG emissions can vary by as much as 5% from year to year due to extreme hot or cold weather. These conditions can impact the extent to which Canada would be able to achieve energy savings. Volatile energy prices also affect the adoption of energy-efficient products and practices, e.g., a peak in oil prices can act as an additional incentive for energy efficiency, while a drop in energy prices can have the opposite effect.

Previous evaluations have noted that OEE relationships and strong collaborations have been a key success factor of energy efficiency programming, providing support and reinforcing desired energy-efficient behaviours. Decisions and policies in various jurisdictions can have a negative or positive impact on program results. This is the case for many products that are regulated at the provincial (and municipal) level. Adoption of building codes and NRCan energy efficiency tools (e.g., EnerGuide) by PTs have a direct impact on the performance of the EEP. Program results are also likely to be impacted by shifts in government priorities, either at the federal or provincial levels, as OEE also has limited control over other initiatives that support energy efficiency. If other federal or provincial programs targeting energy efficiency are downscaled or eliminated, this can remove existing incentives and supports, which in turn will impact the ultimate outcomes of the Program in terms of energy efficiency gains. Changes in environmental and energy or industrial policies, for instance, can have an impact on energy efficiency programming and on the adoption of energy-efficient practices and products by stakeholders.

Furthermore, ENERGY STAR is a program and brand owned by the US EPA. Although the EEP considers a complete loss of this program to be unlikely, decisions made by the Government of the US could impact ENERGY STAR in Canada. Interviewees noted that some efforts (e.g., harmonization of energy norms between the two jurisdictions) have been impacted by changes in government.

The adoption of energy efficiency in households, the private sector, and public organizations is also influenced by the availability of human and financial resources to implement energy-efficient products and practices, which in turn can be dependent on current economic conditions. More subjective elements such as management buy-in and vision, as well as consumer preferences and tastes also influence adoption of new products and practices. For example, consumers often make equipment purchases reactively (i.e., after a product ceases to work) and can therefore be deterred by the higher upfront costs of energy-efficient products when making a replacement purchase.

Fast-paced technological changes can also impact energy use, either driving a greater demand, or providing new and more effective energy efficiency methods and products. The commercialization processes of new and more efficient technology (e.g., gas absorption heat pumps) can be lengthy and can face impediments such as those related to demonstrating performance and/or economic payback. The development of new technologies like 3-D printing and artificial intelligence can disrupt consumption patterns and can make regulating products more difficult, but also presents new opportunity areas for energy efficiency developments that the EEP will have to incorporate into its programming.

Finally, the diversity of Canadian society means that the Program’s reach or impact may differ among specific populations. Questioned on this factor, interviewees felt there were additional avenues to explore in this regard, pertaining to:

• Gender balance in the energy efficiency sector (referring namely to the Equal by 30 campaign which highlights benefits of gender diversity for clean energy innovation);
• Considerations for low-income Canadians and the affordability of energy-efficient products and improvements like retrofits;
• Working with Indigenous groups (collaboratively with other departments, such as the CMHC) to improve program impacts in remote regions and benefits for communities; and
• Considerations for particular challenges of communities and organizations in the North.

Performance - Efficiency

Have program resources been used efficiently?

Between 2014-15 and 2017-18, the EEP’s expenditures totalled just over $120M^{Footnote 14}. The vast majority (91%) of these expenditures were directed to operating costs (salaries and Operations and Maintenance), with an average employment of 143 full-time equivalents (FTEs). The average amount distributed as Gs&Cs was about $2.7M per year, increasing with the ramping up of a new Gs&Cs program for industry in 2017-18. In 2016, the OEE secured A-base funding for the Program. This is perceived as a positive shift that should provide greater stability and make hiring easier.

Overall, interviewees perceived program resources to be used efficiently. Over the period of evaluation, the EEP’s resources were underspent by 7.8%. As shown in Table 5, the Program underspent relative to its budget during the first three years of the evaluation period. Among other reasons, this was the result of a delay in start-up for some Gs&Cs. This situation was largely corrected in 2017-18. Financial data for 2018-19 was not available at the time of evaluation.

Table 5: Energy Efficiency Program Financial Profile, 2014-15 to 2017-18^a

Table 5: Energy Efficiency Program Financial Profile, 2014-15 to 2017-18a

Budget (Planned)	2014-2015	2015-2016	2016-2017	2017-2018b	Total
Total Vote 1 and EBPs	$30,693,544	$38,634,312	$23,856,193	$24,054,120	$117,238,169
Total Vote 10	$3,634,000	$3,259,150	$2,543,800	$3,733,300	$13,170,250
Total Budget	$34,327,544	$41,893,462	$26,399,993	$27,787,419	$130,408,418
Expenditures (Actuals)	2014-2015	2015-2016	2016-2017	2017-2018	Total
Total Vote 1 and EBPs	$30,095,732	$32,601,736	$21,933,774	$24,908,061	$109,539,303
Total Vote 10	$2,635,493	$2,127,757	$2,235,854	$3,689,513	$10,688,617
Total Expenditures	$32,731,225	$34,729,493	$24,169,628	$28,597,574	$120,227,920
Difference	$1,596,319	$7,163,969	$2,230,365	-$810,155	$10,180,498
Difference (%)	4.65%	17.10%	8.45%	-2.92%	7.81%
Notes: a – Excludes PSPC accommodation premiums b – Total for Vote 10 in 2017-18 excludes $62,500,000 transferred to the Federation of Canadian Municipalities for the Green Municipal Fund. This Fund was excluded from the scope of this evaluation.

Source: Financial Data provided by EEP.

The EEP’s funding is divided among the OEE Divisions responsible for each program area, with some minimal inputs from the DPAD related to the SIU (see break-down by Division in Figure 3). The different Divisions involved in delivering the EEP have distinct, complementary mandates and operate in parallel. Internal coordination is thus an important program enabler and an efficiency factor in avoiding overlap, maximizing resources, and combining efforts where relevant. Coordination is not always required on a daily basis, but occurs through formal as well as ad-hoc, issue-specific mechanisms.

Figure 3: Overview of EEP Expenditures by OEE Division (2014-15 to 2017-18)^{Footnote 15}

Source: Compiled from data provided by EEP.

Text Version

Overview of EEP Expenditures by OEE Division from 2014-15 to 2017-18. Text version below.

Figure 3: Overview of EEP Expenditures by OEE Division (2014-15 to 2017-18)

Division	Expenditures (Actuals) by Fiscal Year
	2014-2015	2015-2016	2016-2017	2017-2018
Equipment	$7,381,217	$7,044,995	$7,609,929	$8,495,379
Housing	$9,664,590	$16,896,667	$8,635,704	$10,487,280
Buildings	$6,276,086	$6,440,993	n/a	n/a
Industry	$9,409,332	$4,346,839	n/a	n/a
Buildings and Industry	n/a	n/a	$7,923,996	$9,614,915

Source: Compiled from data provided by EEP.

While interviewees described these coordination mechanisms as effective, they also identified an opportunity for greater coordination between the Divisions regarding regulations for equipment that affected the sectors under their responsibility (e.g., windows used in housing). Also, a few internal interviewees described challenges related to project management and indicated that the EEP sometimes suffered from “analysis paralysis” as multiple groups are tasked with discussing an issue. In their view, having multiple people involved in decision-making sometimes results in inefficiencies, with additional time and resources consumed through many rounds of discussions.

More broadly, interviewees noted internal challenges with corporate functions (e.g., Procurement, Human Resources, Information Management and Information Technology, etc.). These included capacity issues that hinder the Program from effectively using its resources.

The evaluation noted some specific efficiency challenges related to industry. Interviewees noted that while Gs&Cs are necessary to initiate industry sectors’ investments in energy efficiency projects, their implementation is fairly labour intensive given the large numbers of low-value contributions and related results. Furthermore, the Industrial Energy Management Program aims to engage with 18 industrial sectors and has had a stable workforce of 22 FTEs. Senior Industry Officers of Industrial Partnerships are each responsible for the engagement with up to seven industrial sectors and for the management of contribution agreements in these sectors, which may ultimately limit the time and resources allocated to each sector. Given that the industry’s interest in energy efficiency is highly dependant on these interactions, the evaluation found there may be a need to deliver more proactive engagement activities. The Program could increase resources available for engagement with industry sectors by outsourcing some activities (e.g., organization of Energy Summit every two years).

The evaluation also found instances where the EEP worked towards gaining efficiencies for both itself and users. For example, the HD proceeded with a LEAN analysis of the EnerGuide process to identify opportunities to shorten the overall length of time between an evaluation and the homeowner receiving the evaluation report, which can be as long as 61 days for some applications. It is expected that major efficiencies can be gained, particularly through better use of information technologies and by outsourcing some activities associated with EnerGuide.

Are there alternative ways of delivering the EEP to achieve similar outcomes?

When asked about alternative ways to deliver the EEP, interviewees were of the opinion that the Program is solid as it is but could benefit from some improvements, or some considerations around its role and influence. The following comments were noted through the lines of evidence:

• There is an opportunity to increase the EEP’s policy capacity and find better ways to engage Canadians and to meet their needs. The technological and environmental contexts are complex and there is an opportunity for more policy capacity to ensure that the tools provided by each program area (e.g., energy efficiency labels, benchmarking software) are well aligned with the needs of users and synchronized with the latest trends in interface technologies (for online delivery). There is also a need for policy capacity to set and adjust program targets as the context changes.
• There is an opportunity for more engagement with stakeholders, including industry, provincial/territorial government and community stakeholders in developing energy-efficient building/facility solutions specific to the northern conditions.
• There is also an opportunity for more engagement with industry stakeholders to increase awareness and support innovation to improve the energy efficiency of their operations. This activity is especially relevant in the context of the federal carbon pollution pricing system that will encourage industry to reduce their GHGs.
• There is also an opportunity for EEP to leverage best practices from other countries in implementing similar programs (e.g., ENERGY STAR program) and developing policies for adoption of energy efficiency solution by industries.

Recommendations

None. Areas of improvement identified in this section are reflected in previous recommendations.

Conclusion

Relevance

There is a continued need for the EEP. Energy efficiency offers a powerful demand-side management option to reduce GHG emissions and meet Canada’s climate change commitments and targets. It also reduces energy costs, supports innovation, and increases competitiveness. While Canada has seen significant energy efficiency improvements over the years, its economy is still very energy-intensive. There remain persistent barriers to the adoption of energy-efficient policies, behaviours, and products. The EEP’s continuous engagement with targeted residential, commercial and industrial sectors is needed to achieve market transformation and behavioural changes towards energy efficiency.

Effectiveness

With the exception of the Energy Efficiency Regulations, 2016, the Program’s design relies on voluntary adoption of the products and practices it supports or promotes. There is evidence that the EEP contributes to increased awareness of energy-efficient solutions and labels among Canadian consumers, homeowners, building owners or managers, and industry. The Program also works to build capacity (knowledge, skills and abilities) of specific groups to make use of these tools. The OEE’s new SIU contributes to its ability to modernize its product offering to respond to the changing needs of these stakeholders through user-centric design and experimentation.

The EEP also engages regularly with PTs, local governments, other federal departments and agencies, and industry on many aspects of this programming, including efforts to harmonize and implement codes and standards. As a result, most jurisdictions generally support federal propositions on equipment regulations. With the exception of Nunavut, all PTs have adopted or adapted the energy efficiency requirements of the NBC (or developed their own building codes to a comparable level). Six provinces have also adopted a version of the NECB. However, there are opportunities for NRCan to improve key target groups’ understanding of the benchmark used by the EnerGuide Rating System for houses.

These initiatives have shown positive results in increased adoption of energy efficiency solutions and practices. More than 99% of the energy-using products reported to NRCan under the authority of the Energy Efficiency Regulations, 2016 are compliant. Data indicates that consumers seek out and are positively influenced by energy efficiency labels for these products. Adoption of ENERGY STAR and EnerGuide label for homes have exceeded the Program’s targets. An estimated 27.6% of Canada’s commercial floor space is now covered by the ENERGY STAR Portfolio Manager tool for buildings. As costs remains a real or perceived barrier to the adoption of energy-efficient solutions across target groups, the EEP’s products and services are critical to building a business case for energy efficiency. To this end, the EEP also engages directly with industry sectors to assist them in their efforts to reduce energy consumption and provides funding to help industry increase its capacity to identify and adopt energy efficiency projects and practices. However, while the Program also deems implementation of energy management systems by industry to be on track, the evaluation found many opportunities to increase engagement and participation within various industry sectors.

Overall, while data suggests that most of the EEP’s program areas have shown year-to-year improvements, the revision of the Program’s indicators impeded the evaluation’s ability to assess the Program’s progress towards its ultimate outcomes for improved energy efficiency (i.e., PJs of energy saved and Mt of GHG avoided) over the period of evaluation. The lack of interim targets in some program areas makes it a further challenge to determine the extent to which the Program is on track. Regardless, evidence indicates that progress to date still leaves a significant gap in required reductions to be achieved by 2030. Various factors affect energy use and the performance of the Program, many of which are outside the Program’s control. Importantly, energy efficiency is an area of shared jurisdiction with the PTs. Success in achieving the EEP’s ultimate outcomes requires strong collaborations and significant action by the Program’s partners.

Efficiency

Stakeholders generally perceived the EEP to be well resourced and efficient. The Program’s resources were slightly underspent (8%) for the period of evaluation, partly resulting from a delay in start up for some Gs&Cs. This situation was largely corrected in 2017-18. While interviewees described effective coordination mechanisms within the OEE, they noted some internal challenges with project management, decision-making processes and corporate services. Overall, the evaluation found few opportunities to increase efficiencies.

Appendix 1: Logic Model

The figure below represents the overall logic model for the EEP. It should be noted that each sector within the EEP has its own logic model which roughly matches the general one, based on the specifics of each Division.

Figure 1: EEP Logic Model

ULTIMATE OUTCOME

Improved energy efficiency in target sectors

 

 

INTERMEDIATE OUTCOME

Adoption and implementation of energy-efficient technologies, products and practices in Canada

P/Ts adopt and/or adapt increasingly stringent energy codes and regulate energy use

 

 

IMMEDIATE OUTCOMES

Target groups are aware of energy-efficient solutions

Target groups have capacity to adopt energy efficiency technology, products, and practices

Target groups are engaged in improving energy efficiency

 

 

OUTPUTS

Partnerships, Memoranda of understanding , and collaborative arrangements

Information tools, products and services on energy efficiency

Standards, certifications, product specification, and benchmarking tools

Tools to enable mandatory labelling; conformity assessments

Model national energy codes for residential, institutional and commercial buildings

Grant and Contribution agreements

 

 

ACTIVITIES

Develop and maintain collaborative networks

Capacity (knowledge, skills and ability) building activities

Develop and maintain standards, regulations, certifications and benchmarking tools

Administer labelling and compliance programs

Develop model energy codes for residential, commercial and institutional buildings

Provide funds

Source: Logic Model from the EEP Performance Information Profile (PIP).

Text Version

The figure presents the logic model for the EEP. The program’s activities lead to its outputs, and outputs lead to achievement of immediate, intermediate and ultimate outcomes. These specific activities, outputs and outcomes of this logic model are as follows:

Activities:

• Develop and maintain collaborative networks
• Capacity (knowledge, skills and ability) building activities
• Develop and maintain standards, regulations, certifications and benchmarking tools
• Administer labelling and compliance programs
• Develop model energy codes for residential, commercial and institutional buildings
• Provide funds

Outputs:

• Partnerships, Memoranda of understanding , and collaborative arrangements
• Information tools, products and services on energy efficiency
• Standards, certifications, product specification, and benchmarking tools
• Tools to enable mandatory labelling; conformity assessments
• Model national energy codes for residential, institutional and commercial buildings
• Grant and Contribution agreements

Immediate Outcomes:

• Target groups are aware of energy-efficient solutions
• Target groups have capacity to adopt energy efficiency technology, products, and practices
• Target groups are engaged in improving energy efficiency

Intermediate Outcomes:

• Adoption and implementation of energy efficiency technologies, products and practices in Canada
• PTs adopt and/or adapt increasingly stringent energy codes and regulate energy use

Ultimate Outcome:

• Improved energy efficiency in target sectors

Appendix 2: Detailed evaluation methodology

The evaluation questions and methods were developed in consultation with an evaluation working group composed of senior management and managers of each OEE Division with EEP responsibilities. In particular, the scope and focus of case studies were validated with the working group. The evaluation also took into account the results of the last evaluation and calibrated the methodology accordingly. Feedback on preliminary findings shared with the evaluation working group was considered in this report.

The evaluation was completed by GGI for NRCan’s Evaluation Division. Data collection was substantively completed from January to June 2019.

The following lines of evidence were used to answer evaluation questions:

1. Program document and data review: Documentation from over 85 sources (including relevant websites) were reviewed. This includes additional documents obtained as a result of interviews. Extracts from NRCan’s Gs&Cs database (AMI administrative data) and compilation of the OEE’s Industry Project database (result data) were also reviewed.
2. Interviews: A total of 50 interviewees were consulted for this evaluation:
   1. Strategic interviews: Strategic interviews with 14 program and project managers. Strategic interviews were conducted early in the process to help inform key evaluation issues and the design for the selection and conduct of impact focused case studies and targeted database review/economic analysis to capture and report on expected and unexpected impacts.
   2. Case study interviews: Interviews with 38 internal and external key informants were conducted on case study areas. The number of interviews varied between and 4-14 key informant interviews per case: Housing (six interviews), Equipment (four interviews), Buildings (seven interviews), Industry (seven interviews) and the SIU (14 interviews).
   3. Senior interviews: Interviews senior managers from NRCan’s EEP Program (six) and other government departments (OGD) (two). These senior interviews were conducted later in the evaluation process to further explore themes and issues previously identified with the help of other lines of evidence.
3. Case studies: Five case studies were completed. The cases were selected based on their reported success and potential for lessons learned following a review of documentation and consultations with Program staff. Data for each case was collected through a targeted document and literature review, secondary data analysis, and key informant interviews (see above for details). The purpose of the case studies was to illustrate the rationale and performance (effectiveness and impact) of the OEE in each program area, through the following specific aspects or products:
   • Equipment. Gas Furnaces, with a specific focus on how the ED works to eliminate the worst energy performers and accelerate the introduction of more energy-efficient products in Canada’s gas furnace stock.
   • Housing. EnerGuide Rating System, with a specific focus on how the HD works to facilitate the use and the application of the energy performance ratings labels to new homes.
   • Buildings. ENERGY STAR Portfolio Manager Tool, with a specific focus on how the BID works to make the building sector more energy efficient.
   • Industry. Steel Sector, with a specific focus on how the BID engages the steel sector towards the implementation of energy efficiency practices and initiatives.
   • Social Innovation Unit. This case examined the SIU’s approach through five projects. It focused on how the
     SIU: 1. Builds relationships and capacities for energy efficiency policy and service innovation; 2. Generates evidence and collective learning by co-creating and testing insights and interventions (e.g. through prototyping and experimentation); and 3. Scales learning and impacts by implementing what works through service redesigns and new policy directions.
4. Secondary data analysis: Targeted supplementary analyses were also conducted. This included a secondary impact analysis looking at energy and GHG outcomes to validate EEP ultimate outcomes, and a secondary economic impact analysis focused on unintended benefits for the economy (defined in the context of this evaluation as outcomes that have not been identified as intended outcomes in the Program’s logic model).

Evaluation limitations

In general, the EEP is highly data-driven and program staff facilitated access to documentation, performance data and stakeholder information. However, the evaluation experienced some challenges resulting in limitations. These limitations and associated mitigation actions implemented by the evaluation team are described below.

Mitigation actions(s): GGI examined the complete annual performance story and used specific years to illustrate particular outcomes, but used data for the latest year available (2017-18) to speak about the achievements to date that may be instrumental for the next five years. Some additional data to inform performance in 2018-19 was also collected through interviews and document review. Overall, GGI deems that the impact of this limitation is low given the high level of quality of the data provided up to 2018.

Mitigation actions(s): GGI conducted a targeted database review and an economic analysis with alternative source data (Statistic Canada) to compare with program performance data. The impact of this limitation is low as the EEP performance data is generally in line with the data collected from Statistics Canada by GGI. Variances were identified where relevant in technical reports produced for each line of evidence.

Mitigation actions(s): To limit issues of confidentiality, GGI obtained information from the Industry Project Tracker aggregated by province and by predefined industry sectors. The impact of this limitation is deemed medium as the content of this source is unknown.

1. Program data availability and variability: Financial and performance reporting data for the latest fiscal year (2018-19) was not available at the time of writing for this evaluation; in most cases, financial and performance data used in this report is for 2017-18. Also, as a result of program restructuring in 2016-17, the Program performance indicators and associated targets for each program area changed during the evaluation period. As a result, it was not possible to assess the extent to which the Program and its areas achieved targets for the whole evaluation period.
2. Program performance data assessment: GGI did not assess the quality of program performance data, and in most cases relied on summary tables prepared by the Program. However, checks were conducted to ensure the data provided was coherent with other sources and the EEP databases have been audited during the evaluation period. The OEE has had an external consulting firm review the methodology for its impact estimation plans.
3. Limited perspective of external stakeholders in some areas: The anticipated number of interviews with external stakeholders was lower than planned. GGI experienced difficulties in reaching and booking interviews with the following stakeholder groups on the following topics:

   Mitigation actions(s): GGI reviewed relevant web sites to complement the perspective of this stakeholders group. Impact of this limitation is deemed low.

   Mitigation actions(s): GGI reviewed industry web sites and EEP industry action plans to complement the perspective of this stakeholders group. The impact of this limitation is deemed medium.

   Mitigation actions(s): GGI cross-referenced evidence from the document and data reviews and interview data to complement the perspectives of these stakeholders groups. GGI also conducted additional searches on the Internet to obtain information about provincial and municipal programs that use NRCan tools. The impact of this limitation is deemed low.

   • OGDs on interdepartmental coordination: Only two (2) representatives from OGDs agreed to be interviewed as part of the key informant interviews.
   • Industry on engagement and results: All the industry representatives interviewed for this evaluation were consulted in the context of the case study on the steel sector. One interviewee presented the industry perspective as a whole because of its involvement in CIPEC governance and activities.
   • PTs stakeholders on housing standards, tools and adoption: Few representatives from PTs agreed to be interviewed as part of the stakeholder interviews and case study interviews.
4. Limited access to industry information: Considered as confidential, results information from the Industry Project Tracker (e.g., energy consumption numbers, energy savings numbers) were not shared at the nominal level with the evaluation team.