Audit of the delivery of ecoENERGY Efficiency Program (AU1509)

TABLE OF CONTENTS

• EXECUTIVE SUMMARY
  • Introduction
  • Strengths
  • Areas for Improvement
  • Internal Audit Conclusion and Opinion
  • Statement of Conformance
• INTRODUCTION
  • Audit Purpose and Objectives
  • Audit Considerations
  • Scope
  • Approach and Methodology
  • Criteria
• FINDINGS AND RECOMMENDATIONS
  • Governance, Human Resources, and Roles and Responsibilities of Oversight Bodies
  • Communication Processes
  • Performance Measurement and Reporting
• APPENDIX A – AUDIT CRITERIA

EXECUTIVE SUMMARY

INTRODUCTION

In 2007, the federal government introduced the Clean Air Agenda (CAA) as a fundamental component of Canada’s broader efforts to address the challenges of climate change and air pollution, with a view to building a clean and healthy environment for all Canadians. These challenges are being addressed through regulatory initiatives in the industrial, transportation, consumer, and commercial sectors, and a range of program measures to complement these regulatory initiatives. The Office of Energy Efficiency (OEE), within Natural Resources Canada’s (NRCan’s) Energy Sector, is responsible for delivering the ecoENERGY Efficiency Program as one of seven federal programs addressing the CAA. The Program is designed to promote reductions in Green House Gas (GHG) emissions and air pollutants and to address climate change.

The objective of the audit was to assess the overall adequacy of the control framework in place to deliver the ecoENERGY Efficiency Program including compliance with government policies. Specifically, the audit assessed whether the program contained the following key management components: an established and sufficient governance process including human resource management and succession planning; established business processes for the management of stakeholder engagement; and established program planning, monitoring, and reporting activities to inform management decision-making. The scope of the audit included program activities from fiscal years 2011-12 to 2014-15, focusing on regulations; codes and standards; and, information and tools. 

The Audit of the Delivery of ecoENERGY Efficiency Program at NRCan was approved by the Deputy Minister as part of the Risk-Based Audit Plan for fiscal year 2014-15.

STRENGTHS

An adequate governance structure and management practices exist for the delivery of the ecoENERGY Efficiency Program. The Program benefits from a particularly robust performance measurement, monitoring, and reporting function. Additionally, a current, comprehensive succession plan is in place.

AREAS FOR IMPROVEMENT

There exist opportunities to update the Strategic Human Resource Plan and the Strategic Communications Framework, better coordinate the delivery of program specific information between the OEE’s divisions, and establish a central repository for all third party agreements.

INTERNAL AUDIT CONCLUSION AND OPINION

In my opinion, overall, NRCan has an adequate control framework in place to deliver the ecoENERGY Efficiency Program, including compliance with government policies. Opportunities for improvements exist to update the Strategic Human Resource Plan and the Strategic Communications Framework; better coordinate the delivery of program specific information between the OEE’s divisions; and strengthen documentation related to third party agreements.

STATEMENT OF CONFORMANCE

In my professional judgement as Chief Audit Executive, the audit conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of the Quality Assurance and Improvement Program.

Christian Asselin, CPA, CA, CMA, CFE
Chief Audit Executive
June 4, 2015

ACKNOWLEDGEMENTS

The audit team would like to thank those individuals who contributed to this project and, particularly employees who provided insights and comments as part of this audit.

INTRODUCTION

The mandate of Natural Resources Canada (NRCan) is to enhance the responsible development and use of Canada’s natural resources, including energy. Authority to deliver on these responsibilities was established through the Energy Efficiency Act (1992) and the Department of Natural Resources Act (1994). The Office of Energy Efficiency (OEE) has been delivering an evolving suite of programs to address these responsibilities for more than twenty years.

In 2007, the federal government introduced the Clean Air Agenda (CAA) as a fundamental component of Canada’s broader efforts to address the challenges of climate change and air pollution, with a view to building a clean and healthy environment for all Canadians. These challenges are being addressed through regulatory initiatives in the industrial, transportation, consumer, and commercial sectors, and a range of program measures to complement these regulatory initiatives. The OEE, within NRCan’s Energy Sector, is responsible for delivering the ecoENERGY Efficiency Program as one of seven federal programs addressing the CAA. The Program is designed to promote reductions in Green House Gas (GHG) emissions and air pollutants and to address climate change.

In the past, the Program has had a greater emphasis on financial incentives to achieve program objectives. These included grants and rebates to businesses, homeowners, and consumers. Program funding was subsequently renewed in 2011 consisting of $39 million annually for a total of $195 million over 5 years. The 2011 program renewal has evolved from providing financial incentives towards a stronger emphasis on regulation, codes and standards, and the provision of information and tools to Canadians. Due to the integrated nature of the North American marketplace, where appropriate, the program aims to align with approaches adopted within the U.S. The expected result of the program is to increase energy efficiency thereby reducing GHG emissions by 3.6 to 4.4 megatonnes (Mt) annually by the end of the 2015-16 program year.

Text version

The ecoENERGY Efficiency Budget for 2013-14 is allocated to program areas as follows:

• 30% in Housing;
• 24% in Equipment;
• 16% in Vehicles;
• 13% in Buildings;
• 13% in Industry;
• 3% in PWGSC Accommodation Costs; and
• 1% in Alternative Transportation Fuels.

Program initiatives and policy outcomes are framed around the following 5 key areas:

1. Support energy market diversification with the promotion of products, services and technologies;
2. Lower energy costs in communities, including northern, remote and aboriginal;
3. Help Canadian consumers and businesses save money;
4. Improve industry’s productivity and competitiveness; and
5. Build public confidence in energy development by improving environmental performance and reducing energy waste.

These outcomes are delivered through five components of the ecoENERGY Efficiency Program. The following is a brief overview of these program components:

1. ecoENERGY Efficiency for Housing supports the delivery and update of the EnerGuide Rating System, the national home energy labelling and evaluation platform used by provinces as the foundation for their housing programs. In addition, it provides new levels of energy performance in housing codes and standards by incorporating home energy requirements into the National Building Code and establishing the next generation of standards for premium home labelling.
2. ecoENERGY Efficiency for Buildings supports the development and implementation of energy codes, benchmarking tools, and training and information materials to improve the energy efficiency of commercial and institutional buildings in Canada.
3. ecoENERGY Efficiency for Industry works closely with industry to encourage the sector to invest in, develop, and use methods and industrial processes that are more energy efficient. This is accomplished through government collaboration with industry associations, technical experts, utility providers, and other key stakeholders.
4. ecoENERGY Efficiency for Vehicles aims to raise awareness regarding the impact of vehicle choice and operating habits on fuel efficiency and the environment through education, decision making tools, and benchmarking.
5. ecoENERGY Efficiency for Equipment Standards and Labelling aims to accelerate the introduction of more energy efficient products in Canada’s equipment stock. This initiative is accomplished through energy efficiency regulations governing minimum energy performance standards for energy using products and products that affect energy use. This component administers the ENERGY STAR labelling and promotion program to promote energy efficient products and help ready the market for future regulatory action.

Prior audits of programs delivered through the Office of Energy Efficiency have included a strong emphasis on grants and contributions. The scope for this audit excluded grants and contributions, reflecting the program’s evolution towards focusing on regulations, codes and standards, and information and tools. The audit of the delivery of ecoENERGY Efficiency Program was included in the Department’s Risk-Based Audit Plan, approved by the Deputy Minister on February 27, 2014.

AUDIT PURPOSE AND OBJECTIVES

The objective of the audit was to assess the overall adequacy of the control framework in place to deliver the ecoENERGY Efficiency Program including compliance with government policies.

Specifically, the audit assessed whether:

• Governance processes were established for the ecoENERGY Efficiency Program;
• Business processes were in place to manage stakeholder engagement; and
• Program planning, monitoring, and reporting activities were established and operating as intended to inform management decision-making.

AUDIT CONSIDERATIONS

The audit was identified as part of the Risk-Based Audit Plan for 2014-15.

A summary of the key underlying potential risks that were identified during the planning phase of the audit included:

• The risks associated with maintaining a sufficient and representative workforce with the appropriate experience and skill-mix; 
• The risk of ineffective partnerships, agreements, and communications with key stakeholders adversely affecting the Department’s ability to achieve program objectives;
• The risk associated with insufficient or ineffective business processes, protocols, and tools; and
• The risks associated with inadequate information and reporting processes which may negatively impact decision-making.

SCOPE

The scope of the audit included an examination of ecoENERGY Efficiency program activities from fiscal years 2011-12 to 2014-15. Prior program evaluations had concluded on program alignment and performance. The audit team reviewed these evaluations and took their findings into consideration when defining the coverage and scope of this audit project. Grants and contributions have been scoped out of the audit to reflect the latest program shift away from financial incentives to regulations, codes and standards, and information and tools.

APPROACH AND METHODOLOGY

The audit methodology was based on Treasury Board (TB) Policy on Internal Audit and Government of Canada Internal Audit Standards and included:

1. Interviews with key personnel with respect to the management of ecoENERGY Efficiency activities;
2. Review of key documents including the TB Submission and amendments, policies, directives, and guidance; and
3. A detailed examination of information and documentation pertaining to program planning, monitoring, performance, and reporting.

CRITERIA

FINDINGS AND RECOMMENDATIONS

GOVERNANCE, HUMAN RESOURCES, AND ROLES AND RESPONSIBILITIES OF OVERSIGHT BODIES

Summary Finding

The ecoENERGY Efficiency Program has an adequate governance structure in place, with clearly defined and documented roles, responsibilities, and accountabilities. Program-level succession planning is in place; however, the Strategic Human Resource Plan for the OEE has not been updated in a timely manner.

Supporting Observations

The audit sought to determine whether the program has sound, established governance processes in place. Such processes ensure clarity of roles, responsibilities, and accountabilities among stakeholders, and that appropriate committees and reporting structures exist to provide effective oversight on program activities. The audit also examined the program’s human resources management activities to ensure that a comprehensive strategy was in place to manage current and future resourcing requirements.

Governance and Oversight

Primary oversight of the ecoENERGY Efficiency Program is accomplished through regular weekly meetings held by the Director General, OEE and the directors of each division within the branch. The audit found that this reporting structure is functioning as intended, that the meetings are attended by appropriate management, and that they inform decision-making regarding program performance and progress towards established objectives. For example, during the weekly meetings, each Director provides the Director General with program related updates of broader OEE interest and relevance. Additionally, performance targets are discussed on a semi-annual basis, and updated appropriately.

At the Sector-level, there are three additional committees with representation from the OEE; the Senior Management Committee, the Extended Management Committee, and the Human Resources and Management Oversight Committee:

1. The Senior Management Committee is a Director General-level committee chaired by the Assistant Deputy Minister, Energy Sector (ADM ES). This committee provides a forum for information-sharing among Energy Sector senior management.
2. The Extended Management Committee is chaired by the ADM ES. The committee includes all sector executives and provides a forum for the dissemination of corporate initiatives, branch-level management updates and the identification of emerging issues affecting the Energy Sector.  
3. The Human Resources and Management Oversight Committee is chaired by the Director General, Energy Policy Branch (DG EPB). This committee monitors the Sector’s human resource planning, staffing, and reporting activities.

The audit found these committees sit regularly, are attended by appropriate management representatives, and provide sufficient oversight and governance for the ecoENERGY Efficiency Program. Furthermore, program management provide the DG OEE with performance updates on an ongoing basis.

Human Resource and Strategic Planning

The audit sought to determine if adequate human resources management was being performed to address program delivery needs. The audit focused on strategic human resource planning and succession planning. Strategic human resource planning identifies current needs and capacity as well as anticipated needs and formulates strategies to address identified resource gaps and related risks. Succession planning identifies specific personnel whose expertise may be difficult to replace through recruitment or training and development, and develops strategies to measure and mitigate the risks associated with the potential loss of these resources. Mitigating strategies include internal talent management, training, incremental transfer of responsibilities, and external recruitment.

The OEE has completed a robust succession plan that identifies the current human resource capacity of the ecoENERGY Efficiency Program. This plan includes the identification of key personnel, risks associated with departures, and corresponding strategies to manage those risks.

With regards to human resource planning, the OEE last completed a comprehensive Strategic HR Plan for 2010-2012 and contributed to the 2013-14 Energy Sector Strategic HR Plan completed in 2013-14. Although management has advised us that the OEE has all components required to fulfill their HR Planning needs, their Strategic HR Plan is not current. An updated plan is a prerequisite to identifying required human resource expertise and informing recruitment, retention, and development strategies for current and future program delivery.

RISK AND IMPACT

Ongoing delivery of the ecoENERGY Efficiency Program is contingent upon the quality of the human resources within the Office of Energy Efficiency, including effective coordination with the Energy Sector. The ecoENERGY Efficiency Program has evolved and continues to adapt to an environment of changing constraints, directives, and challenges. The development and maintenance of a strategic human resource plan that is responsive to ongoing and emerging challenges is a prerequisite for optimal program delivery.

RECOMMENDATION

1. It is recommended that the Director General of the Office of Energy Efficiency update the Strategic Human Resource Plan. The plan should be maintained through periodic updates to address changes in both capacity and program needs.

MANAGEMENT RESPONSE AND ACTION PLAN

Management agrees.

In response to recommendation 1, the Office of Energy Efficiency has begun updating its Strategic Human Resource Plan to ensure a comprehensive strategy was in place to manage current and future resourcing requirements.

Position responsible: Director General of the Office of Energy Efficiency

Timing: November 30, 2015

COMMUNICATION PROCESSES

Summary Finding

The audit identified adequate business processes to manage stakeholder engagements through communication and formal agreements. Opportunities exist to improve the planning of communication activities and to create a central repository for third party agreements.

Supporting Observations

The audit sought to determine if the ecoENERGY Efficiency Program had established processes to facilitate ongoing communication with stakeholders and program delivery partners. The audit included a review of the Energy Efficiency Strategic Communication Framework, as well as division-level communication activities. Internal processes were assessed to determine the timeliness and accuracy of communication with program stakeholders. Agreements with third party stakeholders were examined to ensure that they were current, enforceable and that roles, responsibilities, obligations and timelines were clearly defined.

Communication with stakeholders

The ecoENERGY Efficiency Program employs a range of regulatory initiatives and program measures in the industrial, transportation, consumer, and commercial sectors. Ongoing communication with stakeholders from all of these sectors is critical for facilitating stakeholder engagement and successful program delivery. Stakeholders range from industry partners who develop energy efficient products (such as automobile and appliance manufacturers) to end-user consumers who purchase and use energy efficient products (such as home and automobile owners).

The Energy Efficiency Strategic Communications Framework is an instrument developed to assist the coordination of the department’s overall communications regarding energy efficiency. The Framework incorporates initiatives underway within the Energy Sector’s OEE and other Sectors within NRCan. It outlines an overall communications approach including proposed communications products and activities. This framework was developed at a strategic level, and is not specific to any one individual program. In contrast, division-level communication activities include tools and strategies used to support program delivery for the five components of the ecoENERGY Efficiency Program. These activities include the use of many communication channels to distribute program specific information including publications, conferences and workshops, media partnerships, websites, and social media.

The Public Affairs and Portfolio Management Sector (PAPMS) provides guidance to the OEE on its communication plans and activities, and coordinates proactive media outreach and partnerships. In 2012, PAPMS developed the Energy Efficiency Strategic Communications Framework to assist communication activities of the OEE for program years 2012-2014. Since the development of this framework, the communication function of the ecoENERGY Efficiency Program has experienced additional constraints resulting from government wide changes in guidance and directives. In 2013, the Treasury Board of Canada Secretariat issued new Guidance on Publishing and a new Directive on Travel, Hospitality, Conferences and Events Expenditures. The Guidance on Publishing places restrictions on the printing of communications products, including publications, in high volume. Departments are to publish communications products in print-ready formats and make publications available online. The Directive on Travel, Hospitality, Conference and Event Expenditures places restrictions on travel expenditures and encourages departments to use virtual presence and remote meeting solutions instead. Consequently, the use of print publications and attendance at conferences and events has been significantly reduced. The Strategic Communications Framework was created prior to these changes and has not been updated to address Treasury Board Secretariat’s new guidance and directive.

The audit also examined operational communication products from the Equipment, Housing, Building, and Industry and Transportation Divisions within the OEE. Operational communication activities performed by OEE personnel include processes for the planning, approval, and execution of outreach products. These processes ensure program information is communicated to stakeholders in an accurate and timely manner. As noted previously in this report, the audit found that adequate governance structures were in place to facilitate communication between divisions; however, opportunities remain to further coordinate the delivery of communication products between the OEE divisions. Improved coordination of these activities may result in opportunities to identify potential efficiencies and minimize duplication of efforts.

Communication Tools and Guidance

With the reduced use of print publications, the OEE delivers information regarding its programs and initiatives primarily through its webpages and interactive web tools such as databases and calculators. Program information on the webpages and interactive web tools are easily accessible. There is an established process in place for managing and implementing web publishing which includes support from both the corporate web publishing team and the Communications Services Branch. Most requests are processed, approved, and published within two weeks. Overall, information regarding program changes and ongoing updates are incorporated within web-tools and online guidance in an accurate and timely manner. 

Third Party Agreements

The OEE collaborates with third party stakeholders for the development, delivery, and promotion of ecoENERGY Efficiency initiatives. These collaborations are formalized through Memoranda of Understanding, Letters of Agreement, Net Vote Revenue Agreements, and other types of agreements with industry associations, provincial energy providers, academia, and other government departments. The audit examined eight of these agreements and found that roles, responsibilities, deliverables, and effective dates were clearly defined. In two instances, the OEE provided unsigned agreements and was unable to locate enforceable agreements signed by authorized departmental representatives.

RISK AND IMPACT

Ongoing engagement with stakeholders through communication and formal agreements is critical to the delivery of the ecoENERGY Efficiency Program. Leveraging existing division-level management discussions to improve coordination of communication activities between divisions could assist in the identification of efficiencies in the processes that may further strengthen timely program delivery. Compiling third party agreements in a central repository would allow for the periodic review of all agreements to ensure third parties continue to honor their obligations to the program.

RECOMMENDATIONS

2.a.  It is recommended that the Director General of the Office of Energy Efficiency (OEE) review and update the Strategic Communications Framework to ensure that it is relevant and reflects the current environment in which the Office of Energy Efficiency operates.

2.b.  It is recommended that the Director General of the Office of Energy Efficiency, leverage existing OEE management meetings to improve coordination between divisions related to the delivery of communication products such as publications, conferences and social media.

3.  It is recommended that the Director General of the Office of Energy Efficiency ensure that all agreements with third parties are signed by the appropriate representatives and the original documents are centrally retained and periodically reviewed.

MANAGEMENT RESPONSE AND ACTION PLAN

Management agrees.

In response to recommendation 2a, the OEE is in the process of updating its Strategic Communications Plan to ensure it reflects recent changes to the operating environment.

Position responsible: Director General of the Office of Energy Efficiency

Timing: November 30, 2015

In response to recommendation 2b, the OEE will work towards leveraging existing OEE management meetings to further coordinate the timing and delivery of communications activities and products within the OEE. This will include a periodic review of upcoming initiatives and coordination to identify potential efficiencies.

Position responsible: Director General of the Office of Energy Efficiency

Timing: On a quarterly basis beginning June 2015 at the OEE management meetings

Management agrees.

In response to recommendation 3, the OEE will ensure that all agreements with third parties are periodically reviewed and that signed copies are centrally retained within GCDOCS. 

Position responsible: Director General of the Office of Energy Efficiency

Timing: September 30, 2015

PERFORMANCE MEASUREMENT AND REPORTING

Summary Finding

The OEE has established a robust Performance Measurement and Risk Strategy (PMRS) for the ecoENERGY Efficiency Program. The PMRS defines the strategy for measuring, monitoring, and reporting program results. Division and branch management regularly review results and monitors progress against program objectives. This information is used in planning to establish ongoing targets and to inform both tactical and strategic adjustments to further the long-term achievement of program objectives.

Supporting Observations

Effective performance measurement and monitoring is a key component of program delivery. It provides the necessary information to enable informed decision-making by senior management and assists in maintaining the sound stewardship of program funds and activities.

Monitoring and reporting of program performance against PMRS targets is done within the Better Energy Efficiency Reporting System. The System records and tracks the progress of program activities on an ongoing basis. This database is a branch-wide electronic performance reporting system that captures information on outputs, intermediate outcomes, and longer-term outcomes in an effort to track commitments. Program information and targets are entered into the system by program managers at the beginning of each year and results are entered at both mid-year review time and at year-end, and are presented to senior management. Included within these performance measurement indicators are program participant compliance activities such as marketplace audits for compliance with labeling and marking requirements, and product performance testing. These functions are performed by NRCan personnel, contract service providers, and certified testing laboratories.

In addition to serving to help manage the branch’s annual performance plans, the database allows for long-term performance tracking. Divisions regularly review results in the System to monitor progress on program activities. This information is used to establish ongoing targets and to make strategic adjustments, in order to ensure the long-term achievement of program objectives. The audit found that there are relevant established metrics and targets in place to evaluate ongoing program performance and inform management decision-making.

APPENDIX A – AUDIT CRITERIA

This appendix details the specific audit criteria related to each of the objectives of the audit along with the source of the criteria (see * below).

The objective of the audit was to assess the overall adequacy of the control framework in place to deliver the ecoENERGY Efficiency Program including compliance with government policies.

Specifically, the audit assessed whether:

• Governance processes are established for the ecoENERGY Efficiency Programs;
• The ecoENERGY Efficiency programs have business processes in place to manage stakeholder engagements; and
• Program planning, monitoring, and reporting activities are established and operating as intended to inform management decision-making.

The following audit criteria were used to conduct the audit:

Audit Sub-Objectives	Audit Criteria
Sub-Objective 1: GOVERNANCE To determine whether governance processes are established for the ecoENERGY Efficiency Programs.	1.1 Management provides oversight to enable the ongoing achievement of program objectives.
	1.2 A Human Resource Plan is developed, and succession planning performed, to address capacity and meet current and future program objectives.
	1.3 Roles, responsibilities, and accountabilities are clearly defined and documented.
Sub-Objective 2: COMMUNICATION PROCESSES To determine whether the ecoENERGY Efficiency Program have business processes in place to manage stakeholder engagements.	2.1 Information regarding program changes and updates are communicated to stakeholders in an accurate and timely manner.
	2.2 Information regarding program changes and updates are incorporated within tools and guidance in an accurate and timely manner.
	2.3 Third party collaborations are managed through formal agreements.
Sub-Objective 3: PERFORMANCE REPORTING To determine whether program planning, monitoring, and reporting activities are established and operating as intended to inform management decision-making.	3.1 Planning, monitoring, and reporting processes are in place and are operating as intended.
	3.2 There are relevant established metrics and targets in place to evaluate ongoing program performance and inform management decision-making.

* The criteria have been developed from the key controls set out in the Treasury Board of Canada Secretariat’s Core Management Controls, Management Accountability Framework and relevant associated policies, procedures and directives.