Summary of the Joint Audit and Evaluation of the Impact Canada Initiative – Clean Technology Stream
About the Program
The Impact Canada Initiative (ICI) is a whole-of-government effort, led by the Privy Council Office (PCO), which promotes the use of innovative funding models designed to help departments accelerate the adoption of outcomes-based approaches to deliver results to Canadians. Natural Resources Canada (NRCan)’s Clean Technology Stream (CTS) is one of the first program streams developed under this new initiative. Led by the Energy Technology Sector (ETS), the objective of the program is to co-develop and launch a series of prize challenges to achieve breakthroughs in clean technology innovation. Since 2017, the program has launched five challenges and one Indigenous innovation initiative. Each of these has its own mission and unique set of goals, performance targets, incentive structures, timelines, collaborators, participants, outreach strategies, and evaluation criteria. This program was allocated $75 million (M) from 2017-18 to 2020-21 (extended to 2021-22 in response to COVID-19).
What the Engagement Found
Relevance
The engagement found that the ICI-CTS is relevant. The program aligns with federal government and NRCan priorities, roles and responsibilities and each of the six selected focus areas meet key criteria that were set out for their prioritization. More broadly, the ICI-CTS supports mandate commitments to increase experimentation, and is structured to give more flexibility to proponents to propose innovative solutions to problems for which solutions are not apparent.
Governance
The engagement found governance structures, processes and mechanisms for ICI-CTS were defined, established and operating effectively. Differences in key intradepartmental players’ interpretations of the ICI-CTS’ Terms and Conditions in the initial stages of program design and development resulting from the unfamiliar policy context have since been addressed with an ongoing collaborative approach.
Processes and Controls
Overall, we found that processes and controls are in place to support compliance with relevant departmental guidance and the Treasury Board (TB) Policy on Transfer Payments. Key financial operational controls have been designed and are operating effectively most of the time. However, there was an opportunity to improve the retention of documentation related to key decision processes. Proactive disclosure was not always performed within the required reporting period. The program’s process for tracking, managing and resolving recipient audit recommendations issued is not clear.
Program Design
Overall, we found that the design of focus areas under the ICI-CTS was consistent with principles of Impact Canada and best practices in challenge models. While communications strategies could have been more innovative and better resourced, the program attracted a sufficient number of quality proposals.
Stakeholder engagement in co-design was recognized as a key strength for the program. The program was responsive to co-design feedback and, while there was some room for improvement, the resulting funding flexibilities and non-financial proponent supports built into the design of each focus area facilitated participation from diverse solvers. NRCan also made good use of collaborators to co-develop and/or deliver activities under the ICI-CTS.
Measuring Expected and Unexpected Outcomes
Impact Canada uses innovative approaches that necessitate new methods and strategies for measuring program outcomes and impacts. While a notional performance framework was included with program approvals, NRCan is still working on its own and with PCO to enhance this framework and to develop challenge-specific outcomes and related methodologies. Nevertheless, available evidence suggests that the program has made some progress on each of its immediate outcomes – increased collaboration and investment in clean tech, mobilization of new talent (including in underrepresented groups), and enhanced public awareness.
It is too soon to report on progress towards intermediate and ultimate outcomes. However, a key difference in the design of challenge-based models is that if the result is not achieved, the funding is not awarded. We found progress at least insofar as finalists are progressing towards outcomes specific to their project; projects are achieving expected project milestones.
Lessons Learned
The challenge model is a new tool that provides an alternative to traditional funding programs, but decisions on when it is appropriate to use this model should be carefully considered in light of the context of the outcomes to be achieved. NRCan has gained a lot of experimental learning by testing innovations in each focus area – we expect NRCan to also apply lessons learned from the ICI-CTS in its own design of future prize challenges, whether under the umbrella of Impact Canada or as part of a separate initiative.
ABOUT THE ENGAGEMENT
The joint engagement examines the design and delivery of the ICI-CTS from its announcement in Budget 2017 to March 2021, with updates reflecting the impact of COVID-19. The objectives of the joint engagement were:
- To assess the extent to which the program’s design and delivery facilitates the effective and efficient achievement of immediate results and complies with relevant authorities.
- To identify and document lessons learned from the experimental approaches under the Impact Canada – Clean Technology Stream that can be applied to future interventions.
The scope of the engagement includes an examination of outcomes for the program stream as a whole and the activities and outputs for each of its six focus areas. While recognizing the importance of the outcomes targeted by each challenge and initiative, the joint engagement focused on expected results for the program as a whole. Although it considers their role in facilitating design and delivery by NRCan, the engagement’s scope excludes the performance of PCO and any other external collaborators.
NRCan’s Audit and Evaluation Branch (AEB) conducted this engagement in accordance with the TB Policy on Results (2016) and the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing and the Government of Canada’s Policy on Internal Audit. Details on specific engagement questions, methods and limitations are found in the full report.
In future, NRCan’s AEB will look for evidence that lessons learned have been considered at least in the proposed design of relevant programs and activities. Key lessons learned include:
Design |
|
Delivery |
|
Impact |
|
Recommendations | Management Response and Action Plan |
---|---|
|
Management Agrees. The Office of Energy Research and Development (OERD) will further formalize and standardize its processes for documenting lessons learned from Impact Canada Cleantech Challenges. OERD will also explore options and develop a plan for sharing lessons from Impact Canada Cleantech with NRCan senior management and other sectors. Following the awarding of prizes and program wrap up, OERD will develop a lessons learned summary document, which will be updated as lessons emerge. OERD will continue to share lessons with the PCO Impact and Innovation Unit to support the development of “Challenge Resource Materials” that are shared and presented to other federal departments and the public (e.g. PCO case studies, PCO Challenge Guide and PCO advice on challenge juries). OERD will also continue with its practice of giving presentations on Impact Canada within NRCan and to other departments upon request. Position Responsible: DG, OERD on behalf of ADM, ETS Date to achieve:
|
|
Management agrees. OERD will update and finalize the core Impact Canada Cleantech performance measurement strategy, including:
Position responsible: DG OERD on behalf of ADM,ETS Date to achieve:
|
|
Management agrees. OERD will strengthen its risk-based recipient audit plan and methodology to ensure that it better documents and tracks the status and results of audits at the Branch (OERD) level, rather than Division level. OERD has already undertaken a review of its risk-based audit processes and updated its methods for planning, selecting and monitoring audits to ensure that all OERD programs implement NRCan’s Guide on Recipient Auditing in a consistent manner (ex. consistent sampling method). OERD piloted its updated risk-based recipient audit planning and selection method in Q2 2021. OERD is currently refining methods, and developing new guidance documents and tools for monitoring OERD’s annual audit plan. Implementation of processes and tools for monitoring audit recommendations and compliance will occur in Q3&Q4 2021-22. This will include optimising the use of NRCan’s AMI database for tracking audit results, and proponent’s audit history. Position responsible: DG, OERD on behalf of ADM, ETS Date to achieve: March 31, 2022 |
Page details
- Date modified: