Joint Audit and Evaluation of the Impact Canada Initiative – Clean Technology Stream

Presented to the Departmental Audit Committee (DAC) January 21, 2022
Presented to the Performance Measurement, Evaluation and Experimentation Committee
(PMEEC) November 24, 2021

List of Acronyms

ADM

Assistant Deputy Minister

AEB

Audit and Evaluation Branch

ARLU

Annual Reference Level Update

BEIS

UK Department for Business, Energy and Industrial Strategy

CEMI

Centre for Mining Innovation

CI!

Crush It! Challenge

CoE

NRCan’s Centre of Expertise on Grants and Contributions

CPS

Communications and Portfolio Sector

CTF

Charging the Future

CTS

Clean Technology Stream

ETS

Energy Technology Sector

FAA

Financial Administration Act

Gs&Cs

Grants and Contributions

GARDN

Green Aviation Research & Development Network

GHG

Greenhouse Gas

ICE

Indigenous Clean Energy Social Enterprise

ICI

Impact Canada Initiative

ICI-CTS

Impact Canada Initiative - Clean Technology Stream

IIA Standards

International Standards for the Professional Practice of Internal Auditing

IODI

Indigenous Off-Diesel Initiative

LMS

Lands and Minerals Sector

MaRS

MaRS Discovery District

NRCan

Natural Resources Canada

OERD

Office of Energy Research and Development

PCF

Pan-Canadian Framework (on Clean Growth and Climate Change)

PCO

Privy Council Office

PCO COE

Privy Council Office Centre of Expertise for the Impact Canada Initiative

PF

Power Forward

RD&D

Research, Development and Demonstration

REDI

Reconciliation, Equity, Diversity and Inclusion

SMEs

Small and Medium Sized Enterprises

SAF

Sustainable Aviation Fuel

STAC

Science and Technology Assistance for Cleantech

STL

Sky’s the Limit

TB

Treasury Board (of Canada)

TRL

Technology Readiness Level

Ts&Cs

Terms and Conditions

WICT

Women in Clean Tech

Table of Contents

• Executive Summary
• Introduction
• Engagement Objectives and Methods
• What We Found – Relevance
• What We Found - Governance
• What We Found – Design
• What We Found – Processes and Controls
• What We Found – Expected and Unexpected Outcomes
• What We Found – Factors Influencing Program Delivery
• What We Found – Lessons Learned
• Conclusion
• Appendix A: Engagement Sub-Objectives and Criteria
• Appendix B: Joint Engagement Team

Executive Summary

About the Engagement

This report presents the findings, conclusions, and recommendations from the joint audit and evaluation of the Impact Canada Initiative (ICI) – Clean Technology Stream (CTS). Announced in Budget 2017, the ICI is a whole-of-government effort, led by the Privy Council Office (PCO), designed to help departments accelerate the adoption of outcomes-based approaches to deliver results to Canadians. The ICI promotes the flexible use of grants and contributions through a range of innovative funding models, including prize challenges. Natural Resources Canada (NRCan)’s Clean Technology Stream, one of the first program streams developed under this new initiative, has launched six selected focus areas – i.e., the Crush It! Challenge, Charging the Future Challenge, Power Forward Challenge, Sky’s the Limit Challenge, Women in Clean Tech Challenge, and Indigenous Off-Diesel Initiative.

The joint engagement examines the design and delivery of the ICI-CTS from its announcement in Budget 2017 to March 2021, with updates reflecting the impact of COVID-19 (extending the program to 2021-22 and offering additional funding supports). The objectives of the joint engagement are:

• To assess the extent to which the program’s design and delivery facilitates the effective and efficient achievement of immediate results and complies with relevant authorities.
• To identify and document lessons learned from the experimental approaches under the Impact Canada – Clean Technology Stream that can be applied to future interventions.

NRCan’s Audit and Evaluation Branch (AEB) conducted this engagement in accordance with the Treasury Board (TB) Policy on Results (2016) and the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing and the Government of Canada’s Policy on Internal Audit.

What the Engagement Found

Relevance

We found that the ICI-CTS is relevant. The program aligns with federal government and NRCan priorities, roles and responsibilities to support innovations in clean technology related to Canada’s natural resource sectors that tackle Canada’s most pressing environmental challenges and create economic benefits. Each of the ICI-CTS’ six selected focus areas meet key criteria that were set out for their prioritization. The narrow focus of focus area sets this apart from other funding programs, with the suite of clean tech challenges acting as vehicles to enhance the communication of priorities in the Government of Canada’s environmental agenda.

More broadly, the ICI-CTS supports mandate commitments to increase experimentation, and is structured to give more flexibility to proponents to propose innovative solutions. It enables the Government of Canada to test innovations and generate evidence of which efforts work best to create greater public value in areas of high priority. The ICI-CTS provided an opportunity to test innovative ways to address problems for which solutions are not apparent.

Governance

Overall, we found that governance structures, processes and mechanisms for ICI-CTS were defined, established and operating effectively. Roles, responsibilities and accountabilities were clear, including delineation between the responsibilities of PCO and NRCan, and any areas of complementary and overarching responsibilities. However, the unfamiliar policy context resulted in difficulties in the initial stages of program design and development, with fundamentally different interpretations of the ICI-CTS Terms and Conditions (Ts&Cs) among key intradepartmental players. These differences have since been addressed with an ongoing collaborative approach.

Internal reporting mechanisms were also found to be established and functioning as intended at all levels of program management. These reporting mechanisms (both formal and informal) enabled key communication on the program’s progression to senior management at intervals that supported timely reporting across all phases of design and implementation.

Design

Challenges have a set of features meant to attract new talent and new ideas, and accelerate progress towards solving problems that matter to people. As a result, challenges are designed, promoted and delivered differently than the types of funding programs that governments and their stakeholders are accustomed to running and participating in. Many focus areas also incorporated numerous experiments within this broader experiment, for example, testing new models of Indigenous participation or new ways of working with international partners.

We examined how and for whom the program’s experimental tools are effective in facilitating the achievement of intended outputs and outcomes, with a focus on five elements common to all focus areas, i.e.: inclusive innovation (co-design), communications and outreach, collaborative arrangements, flexible funding mechanisms, and additional (non-financial) proponent supports. These elements are critical to the success of challenges. Overall, we found that the design of focus areas under the ICI-CTS was consistent with principles of Impact Canada and best practices in challenge models.

Stakeholder engagement in co-design was recognized as a key strength for the program. This allowed NRCan to target key issues while scaling the challenge to what would be achievable by solvers, including consideration of specific barriers for the problem area. The program was responsive to this feedback and, while there was some room for improvement, the resulting funding flexibilities and non-financial proponent supports built into the design of each focus area facilitated participation from diverse solvers (project proponents and potential collaborators or investors).

To attract the interest of solvers there was also effort dedicated to the development and delivery of communication strategies. While this could have been more innovative and better resourced, the program did attract a sufficient number of quality proposals.

NRCan also made good use of collaborators to co-develop and/or deliver activities under the ICI-CTS. There is evidence that the collaborators selected were both appropriate and effective in facilitating and/or accelerating delivery of the program. In particular, working with Indigenous collaborators on the Indigenous Off-Diesel Initiative (one of the six focus areas) was perceived as a positive move towards a new model of collaborative (rather than Crown-led) program delivery.

Processes and Controls

Overall, we found that processes and controls are in place to support compliance with relevant departmental guidance and the TB Policy on Transfer Payments. We also found that key financial operational controls (refer to Appendix A) have been designed and implemented and are operating effectively most of the time.

There is an opportunity to improve the retention of documentation related to key decision processes, particularly for focus areas that used a collaborator as an intermediary. Further, while all files sampled were noted to be signed by the appropriate individual with a valid financial signing authority and adhered to the required stacking provisions, the release of proactive disclosure information was not always performed within the required reporting period.

The program’s identification of auditable agreements was found to align with NRCan policies and procedures on the recipient-audit process, with consideration to the inherent risk ratings of the agreements. However, while the program uses corporate systems to capture audit related information (i.e., status, findings and recommendations), its process for tracking, managing and resolving recipient audit recommendations issued is not clear.

Expected and Unexpected Outcomes

We examined the extent to which the ICI-CTS is making progress towards program-level immediate outcomes identified in its draft logic model, including increased collaboration and investment by stakeholders in the clean tech ecosystem, increased awareness and participation in clean tech (i.e., mobilization of “new talent”), and increased public awareness of clean tech issues among Canadians. There are a number of internal and external factors influencing program delivery. Within its sphere of influence, the program has been responsive to mitigate or address barriers to delivery.

Impact Canada uses innovative approaches that necessitate new methods and strategies for measuring program outcomes and impacts. While a notional performance framework was included with program approvals, NRCan is still working on its own and with PCO to enhance this framework and to develop challenge-specific outcomes and related methodologies. Difficulties in the design and application of performance metrics still need to be resolved. While there is evidence that the design and delivery of each focus area considered and worked to achieve the program’s immediate outcomes, complications in performance measurement mean that calculation of results against ICI-CTS indicators is not possible given data available.

Nevertheless, available evidence suggests that progress has been made in each area. There is strong evidence to suggest that finalists are developing their own formal and informal collaborative arrangements among diverse stakeholders. Most finalists’ projects have also leveraged investments, at least to some extent, and further leveraging across all focus areas could reasonably be expected as projects continue to advance. While the vast majority of finalists are not new to federal funding programs, a large number were SMEs or start-up businesses. Two focus areas were also specifically designed to support outcomes for underrepresented groups, i.e., to attract and maximize the participation of women and Indigenous communities. Evidence indicates that the program has been successful in mobilizing new talent in these areas. Evidence related to enhanced public awareness indicates that the ICI-CTS performed relatively well on social media compared to most government content. Communications beyond NRCan’s accounts may also have been extensive. However, finalists noted that there was less opportunity for or emphasis on communications promoting their progress than they expected when they decided to apply (in part due to restrictions imposed by COVID-19).

Our engagement was not designed to assess progress towards intermediate and ultimate outcomes, as it is too soon to indicate whether these will be achieved. However, a key difference in the design of challenge-based models is that if the result is not achieved, the funding is not awarded. We found progress at least insofar as finalists are progressing towards outcomes specific to their project; projects are achieving expected project milestones. Progress towards outcomes was also reported by solvers who are not selected as finalists, with many continuing to advance their solutions, albeit at a slower pace or to a more limited extent. The extent to which this can be attributed to Impact Canada is unclear.

The issues targeted by each focus area will not be fully addressed by the ICI-CTS, nor are they intended to be. There is no one, “silver bullet” solution to these complex problems. While there is evidence of viable pathways to scale success for each focus area, this requires dedicated efforts that extend beyond the scope and timelines of the ICI-CTS but which may be accelerated as a result of the program. However, the ICI-CTS is meant to help advance and accelerate solutions. The challenge model is also such that there is no guarantee that anyone will win the final prize. However, even for the one area of the ICI-CTS where the challenge closed without the prize being awarded (i.e., no producer met all the criteria before the submission deadline), there were experimental learnings gained; this should not be viewed as a failure on the part of the program. Moving forward, this intelligence is available to inform perspectives for other programs and actors and may also accelerate the design or implementation of parallel initiatives in related focus areas.

Lessons Learned

Overall, interviewees perceived the challenge model as effective and appropriate but there was no consensus on whether this model should be replicated in the future. We found the challenge model to be a new tool that provides an alternative to traditional funding programs, but decisions on when it is appropriate to use this model should be carefully considered in light of the context of the outcomes to be achieved.  

NRCan has also gained a lot of experimental learning by testing additional innovations in each focus area. Best practices and lessons learned are highlighted where relevant throughout this report. Future use of design elements that were found to be strengths of the ICI-CTS (e.g., co-design, supports to remove barriers for underrepresented groups, etc.) need not be limited to challenge models.

Lessons learned by NRCan related to systemic and departmental barriers to the use of flexible mechanisms have already fed into guidance released by PCO, and should help accelerate future use of related models by NRCan and impart new perspectives to corporate practices. We expect NRCan to also apply lessons learned from the ICI-CTS in its own design of future prize challenges, whether under the umbrella of Impact Canada or as part of a separate initiative. In future, NRCan’s AEB will look for evidence that these lessons learned have been considered at least in the proposed design of relevant programs and activities. Key lessons learned include:

Recommendations and Management Response

Introduction

This report presents the findings, conclusions, and recommendations from the joint audit and evaluation of the Impact Canada Initiative (ICI) – Clean Technology Stream (CTS). The engagement examines the design and delivery of the program from its announcement in Budget 2017 to March 2021, with updates reflecting the impact of COVID-19. The Audit and Evaluation Branch (AEB) of Natural Resources Canada (NRCan) conducted this engagement in accordance with the Treasury Board (TB) Policy on Results (2016) and the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing and the Government of Canada’s Policy on Internal Audit.

Program Information

Announced in Budget 2017, the Impact Canada Initiative (ICI) is a whole-of-government effort, led by the Privy Council Office (PCO), that is designed to help departments accelerate the adoption of outcomes-based approaches to deliver results to Canadians. The ICI’s outcomes-based approach is intended as a new way of opening up the problem-solving process to innovators who can bring fresh perspectives and new ideas. The ICI promotes the flexible use of grants and contributions through a range of innovative funding models, including prize challenges. These approaches can be supplemented with additional funding supports to recipients and/or third parties for project elements such as training, technical assistance, business planning, and evaluation. The Government of Canada has developed a common set of Terms and Conditions (Ts&Cs), housed within PCO, as a horizontal authority for all departments to pursue this outcomes-based programming.

Announced as part of the ICI in Budget 2017, NRCan’s Clean Technology Stream (CTS) is one of the first program streams developed under this new initiative. The objective of the ICI-CTS is to co-develop and launch a series of prize challenges to achieve breakthroughs in clean tech innovation, addressing climate change and environmental impacts while harnessing new economic opportunities. Challenges are designed, marketed and delivered differently from the types of funding programs that governments and their stakeholders are accustomed to running and participating in. They provide incentives (both financial and non-financial) to encourage a broad range of innovators to tackle problems where solutions are not apparent, or current responses are not achieving the desired results. Challenges may be structured to use:

• Pure prizes where specific technical criteria are clearly defined at the outset and must be met before prizes are awarded;
• Stage-gated approaches where challenge participants receive incentives at different stages of the challenge to build capacity or narrow the field of applicants before selecting winning projects; and/or
• Competitive accelerators where prize funding is layered into intensive and time-limited business supports to propel cohorts of early stage innovators towards achieving particular outcomes.

Impact Canada also creates space for Indigenous innovation initiatives. These initiatives borrow elements from challenge-based approaches but rely less on the competitive aspects of the challenge model, and emphasize instead the importance of Indigenous leadership and collaboration with and among communities.¹ This latter approach is not explicitly recognized in the program’s foundational documents but rather emerged as a result of NRCan and other federal departments’ engagement with Indigenous peoples during delivery of the ICI.

Under the authorities of the ICI, specific parameters of each clean tech challenge were to be developed, co-designed and refined by NRCan through extensive stakeholder engagement before launch and challenge delivery. Two focus areas were committed to as part of program approvals, i.e., one focused on sustainable aviation fuels and another on supporting Indigenous communities to reduce their reliance on diesel. The program has since launched five challenges and one initiative (see Figure 1). Each of these has its own mission and unique set of goals, performance targets, incentive structures, timelines, collaborators, participants, outreach strategies, and evaluation criteria.

Impact Canada’s umbrella Ts&Cs include an Annex specific to the CTS. Budget 2017 allocated NRCan $75 million (M) over two years for its delivery. This was subsequently extended to four years prior to program launch, from 2017-18 to 2020-21. The program’s funding authorities notionally distributed the funding between grants ($50.8 M) and contributions ($16.9 M), with the remainder earmarked for program administration. The actual funding profile was to be determined once challenge design was complete. Distribution of planned expenditures assumed challenge design would be completed in Summer/Fall 2017, with a full suite of challenges launched or ready to launch by Winter 2018.

Figure 1: Profile of ICI-CTS Focus Areas

In September 2020, the Minister of Natural Resources approved a program change to remove the expiry date of the Clean Tech Program Stream Annex in response to COVID-19. Funding was extended to 2021-22. Additional program modifications made in response to the pandemic (Box 1) and are reflected in the profile of focus areas presented in Figure 1.

Box 1: Program Response to COVID-19, by Focus Area

Governance Structure and Program Accountabilities

The ICI is a whole-of-government mechanism. PCO has established a Centre of Expertise (COE) within its Impact and Innovation Unit to support the design of program streams and individual departmental projects across a range of challenge areas. While projects launched under the ICI-CTS program stream are expected to follow PCO’s Impact Canada Ts&Cs, NRCan has accountability for delivery of the ICI-CTS.

Within NRCan, the ICI-CTS falls under the responsibility of the ADM of the Energy Technology Sector (ETS). The specific parameters of each challenge were developed by ETS’ Office of Energy Research and Development (OERD), in collaboration with other sectors as required. For example, NRCan’s Lands and Minerals Sector (LMS) led the design and delivery of the Crush It! challenge. NRCan’s Deputy Minister was responsible for ultimate approval of challenge design prior to implementation.

While PCO does not manage program funding or oversee project implementation, the PCO COE may still provide advisory support to NRCan, as requested. NRCan is expected to make use of a government-wide interactive website (developed by PCO as a core component of the ICI) to both inform Canadians on the challenges launched and as the portal to process applications.⁴ NRCan is also required to inform the PCO COE on the design of its challenges and the outcomes achieved through individual projects so that PCO can track overall outcomes achieved under the ICI structure.

Expected Results

The Impact Canada – Clean Technology Stream is part of the Energy Innovation and Clean Technology Program under NRCan’s Core Responsibility for ‘Innovative and Sustainable Natural Resources Development’ in the Departmental Results Framework.

The ICI-CTS’ performance measurement framework is still in development. PCO has developed a logic model and narrative that presents (in a general way) a basic theory of what Impact Canada expects to achieve with its suite of challenges, including a broad objective to accelerate the use of innovative and experimental approaches across the Government of Canada.⁵ Building off this theory, Figure 2 presents a (draft) logic model developed by NRCan’s OERD specific to the ICI-CTS. This model nests the outcomes targeted by each challenge and initiative (indicated by green boxes in the figure) within the expected results for the program stream as a whole (blue boxes).

Expected results of the ICI-CTS are aligned with other NRCan energy RD&D programs, with a focus on advancing clean technology (clean tech) closer to commercial readiness to improve the environmental and economic performance of Canada’s clean tech and natural resource sectors. The ICI-CTS is also expected to broaden the reach of NRCan’s traditional clean tech programming, by increasing public awareness of clean tech problems or issues, stimulating innovation ecosystems by building non-traditional collaborations, and mobilizing new talent or solvers. Challenges were intended to be inclusive in order to attract and encourage participation from different sectors. To this end, the list of eligible recipients is broad and can include incorporated and unincorporated groups, private sector and not-for-profit organizations, other governments and other levels of government, Indigenous organizations, academia, and individuals, with the range of eligible recipients narrowed as appropriate for individual challenges based on desired outcomes.

Collaboration and partnership-building are also meant to be a core feature to the design and delivery of each challenge. Depending on the focus area, collaborators have been used as delivery agents, to provide challenge participants with a range of supports, to build connections within the innovation ecosystem and/or to draw the public’s attention to a challenge.

While recognizing the importance of the outcomes targeted by each challenge and initiative, particularly to the relevance of the program, this joint engagement is focused on expected results for the ICI-CTS as a whole. This includes an expectation for experimental learning from the design and implementation of the challenges and initiative as an outcome that overarches all expected results for the program.

Figure 2: (Draft) NRCan Logic Model for the ICI-CTS

Engagement Objectives and Methods

The AEB included a commitment to conduct this joint engagement in its Integrated Audit and Evaluation Plan 2019-2024. The AEB identified the need for this engagement through its risk-based planning process and in response to a TB commitment to complete an evaluation. The engagement also meets requirements for evaluation of ongoing grant and contribution (Gs&Cs) programs under section 42.1 of the Financial Administration Act (FAA) and the Treasury Board (TB) Policy on Results. The Evaluation Division led this engagement, in collaboration with the Audit Operations Division. Where relevant, the approach for this joint engagement also followed the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing (IIA Standards) and the TB Policy on Internal Audit.

The objectives of the joint engagement are:

• To assess the extent to which the program’s design and delivery facilitates the effective and efficient achievement of immediate results and complies with relevant authorities.
• To identify and document lessons learned from the experimental approaches under the Impact Canada – Clean Technology Stream that can be applied to future interventions.

The original scope of the engagement included NRCan’s design and delivery of the Impact Canada – Clean Technology Stream from 2017-18 to 2019-20, including outcomes for the program stream as a whole and the activities and outputs for each of its six focus areas. This was expanded in March 2020 to enable the engagement to consider the impact of COVID-19 on delivery of the program stream. While we consider the role of PCO in facilitating design and delivery by NRCan, the engagement’s scope excludes the performance of PCO and that of international collaborators in the Power Forward Challenge.

Engagement Questions and Sub-Objectives

Table 1 identifies the questions and related sub-objectives to be addressed in this joint engagement. Appendix A summarizes the detailed criteria applied in assessing these sub-objectives.

Table 1: Engagement Questions and Sub-Objectives

Engagement Methods

The joint engagement limits the duplication of effort that would be required to conduct separate audit and evaluation projects. The engagement was carried out in a manner that ensures the team’s neutrality and objectivity, as per respective professional requirements and standards, and ensures that observations and conclusions are evidence-based.

The approach and methodology used in this engagement followed the TB Policy on Results, including related Standards on Evaluation, the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing (IIA Standards), and the TB Policy on Internal Audit. These standards and policies required that the project be planned and performed in such a way as to obtain reasonable assurance that engagement objectives are achieved. The engagement included tests considered necessary to provide such assurance.

The joint engagement used five data collection methods (see Table 2), designed to ensure validity of the information collected and allow for the triangulation of evidence.

Table 2: Engagement Methods

Engagement Limitations and Considerations

The engagement used multiple lines of evidence in order to mitigate against any limitations associated with individual methods. This enabled the triangulation of evidence across sources of information to identify valid findings and conclusions relative to the engagement’s questions. Nevertheless, the following limitations should be considered when reviewing the findings from this engagement:

Timing of Evaluation. An evaluation of the ICI-CTS was required to be completed at this time to comply with the TB Policy on Results and the Financial Administration Act. However, it is really too soon to report on the impact of this program; measurement of program outcomes will require tracking the trajectory of results over the long-term.

COVID-19. COVID-19 had a significant impact on the design and delivery of both the program and the joint engagement. The conduct (data collection) phase of the joint engagement was delayed by several months to allow time for program collaborators, stakeholders and participants to adjust to the pandemic. The program has also effected important design changes in response to pandemic-related delays and financial pressures (e.g., to the structure of payments and timelines for project completion). Related decisions were not all finalized and/or communicated at time of data collection. Pressures related to COVID-19 also limited the ability of some key informants to participate in the engagement.

Document Review. The document review, file and database analysis were constrained by the information that is collected and maintained by OERD. Documentation required for review was generally accessible but varies considerably in quality and consistency by focus area. Given timing of data collection, mid-project reports to inform our analysis of progress towards outcomes were not yet available for all focus areas. Funding agreements were reviewed in consideration of NRCan’s responsibilities for their delivery, and limited work was performed in assessing areas of responsibility where delivery agents were used.

Analysis of internal reporting mechanisms included a review of reporting documentation beginning as early as challenge design and development to challenge launch, which spanned the period from April 2017 through to December 2018. Documentation related to one challenge (CTF) was not included in this material due to its development and launch in a later period.

Key Informants. Respondents for interviews and the engagement’s surveys were not all familiar with some elements of design and implementation. In some cases, the respondent was not necessarily the same as the applicant for a specific project and so could not comment on early activities and outputs in program delivery (e.g., effectiveness of initial outreach or innovations in the application process). The time elapsed between co-design or application process and the conduct of the engagement also meant some respondents could not recall all details or make any specific recommendations for improvement. Companies receiving services from the Centre for Mining Innovation (CEMI), a collaborator on Crush It!, were not included among respondents for this engagement and we lack their perspectives on CEMI’s appropriateness or effectiveness.

Survey. The overall survey response rate of 24.6% is considered adequate for purposes of this engagement. Response rates by focus area range from 20.4% for the IODI to 50% for STL. However, the total number of applicants and thus survey respondents is not divided evenly among focus areas. The relatively small sample for some specific focus area makes it difficult to report disaggregated data on a challenge-by-challenge basis.

Additional limitations and considerations are identified where relevant throughout this report.

What We Found – Relevance

The ICI-CTS aligns with federal government and NRCan priorities, roles and responsibilities.

In 2015, a Ministerial mandate letter directed the President of the Treasury Board to “work with […] colleagues to ensure that they are devoting a fixed percentage of program funds to experimenting with new approaches to existing problems and measuring the impact of their programs”. Budget 2017 announced the Impact Canada Initiative and identified this as a mechanism by which this mandate commitment could be achieved. The ICI is perceived to fill an important gap in the Government of Canada’s programming toolkit by establishing a mechanism to work with and support departments to employ innovative funding models. Interviewees agreed that the program is structured in such as way as to be more flexible and less administratively burdensome for proponents, giving more flexibility to propose innovative solutions.

The Impact Canada – Clean Tech Stream is more specifically aligned with federal government and NRCan priorities for clean tech. In 2015, the Minister of Natural Resources was mandated to invest in clean tech producers in our natural resource sectors so that they can tackle Canada’s most pressing environmental challenges and create more opportunities for Canadian workers. Canada also made a Mission Innovation pledge to double its investments in clean energy and clean tech research and development by 2019-20 and encourage private sector investment in early-stage clean energy innovation companies in Canada. In Budget 2017, the federal government committed more than $2B to numerous initiatives (including the ICI-CTS) aimed to boost the growth of Canada’s clean tech sector. More recent Ministerial mandate letters (2019) and Throne Speeches (2019) continue to prioritize positioning Canada as a global leader in clean tech.

From the perspective of program representatives, the narrow focus of each challenge or initiative sets this apart from other funding programs, with the suite of focus areas acting as vehicles to enhance the communication of priorities in the Government of Canada’s environmental agenda. For example, four of the challenges and the initiative specifically target environmental benefits related to energy efficiency and greenhouse gas (GHG) mitigation. The WICT challenge is targeted more broadly at environmental benefits, but aligned with priorities to increase gender equity and the participation of women in the Canadian economy and clean tech. The IODI is aligned to priorities for both reconciliation and building the resilience of Indigenous communities, including a mandate commitment (2015) to support the transition of Indigenous communities from reliance on diesel-fueled power to clean, renewable and reliable energy by 2030.

The Impact Canada – Clean Technology Stream gave priority to the selection of relevant challenges and initiatives.

During program design, an analysis of best practices determined that running between four and six clean tech challenges was optimal given the program’s funding profile. However, only two focus areas were identified in program approvals – i.e., supporting Indigenous communities to reduce their reliance on diesel and a sustainable aviation fuels challenge to reduce GHG emissions in aviation – with the remainder to be prioritized based on a specific set of criteria. The criteria and our observations on related program alignment are as follows:

• Breakthrough Solutions. The program was expected to focus on unlocking transformative (i.e., not incremental), high-impact solutions to complex and persistent problems in the clean tech sector. We found that the program has a clear definition of what would constitute a “breakthrough”; its approach brings focused attention to barriers and opportunities specific to each problem area without specifying how the result is to be achieved, leaving space for innovation to occur. There is evidence that the transformations targeted were well researched in the design phase. The objectives of the challenges and initiative are diverse but are all expected to provide innovative solutions to disrupt the market and/or create knock-on or ripple effects that will boost the entire ecosystem in their problem area (Table 3). While the IODI is unlikely to spur growth in new clean technologies, it presents a breakthrough in ensuring that traditional knowledge and consideration of systemic inequalities are at the center of program design and delivery. Recognizing the federal government’s responsibility to respect the right of Indigenous peoples to self-determination, the design of the IODI relies on communities to develop their own solutions.

Table 3: Issues Addressed by ICI-CTS Focus Areas

• High Potential for Impact and Benefits to Canadians. The program was expected to prioritize focus areas with a strong potential for reducing environmental impacts and improving economic performance, as well as for increasing the participation of women and Indigenous peoples. We found that all focus areas were designed to target environmental benefits. In some cases, targeted benefits have been quantified (e.g., project proponents for Crush It! are challenged to generate a step-change reduction equal to or better than a 20% reduction in the energy consumed by crushing and grinding when compared to a pre-established baseline). Challenges were also selected in areas where there are significant opportunities for economic benefit and market growth. Technical benefits would be derived from the solutions themselves as well as from potential ripple effects across the entire value chain or that create new value chains, and are expected to amplify the environmental and economic benefits. Early or accelerated commercialization of clean technologies would also provide a competitive advantage in the global market. Unlike the other challenges, the main benefits of the IODI are expected to be at the community level. Similarly, the support to six finalists for WICT on its own is not expected to significantly impact the gender balance in the clean tech sector. However, in both cases a knock-on effect would be achieved if others are inspired to take action. The clean tech sector as a whole should also ultimately benefit from increasing participation of diverse groups (e.g., women, Indigenous peoples) and new solvers in clean tech.
• High Degree of Complementarity. The program was expected to prioritize focus areas that would have a high degree of complementarity with existing government and non-government efforts in the given problem space. We found that the focus areas selected meet this criterion. Overall, we found that the design & delivery of the ICI-CTS fills a perceived gap in dedicated support for high-risk, high-reward clean tech solutions. The focus areas support priorities that are also addressed through other clean tech programs, but each fills an identified gap without creating duplication. While the identification of follow-on strategies is outside the scope of the program, there is also evidence of viable pathways to scale successes for each focus area (e.g., clearly identified opportunities for handoff to other government programs, venture capital, etc.).
• Ability to Leverage Collaborators and Resources. The program was expected to prioritize focus areas with the ability to leverage collaborators and resources to help design, deliver or promote participation in the prize or challenge, including by acting as a conduit to important stakeholder groups, providing technical expertise, or by contribution of additional resources (financial or in-kind). There is evidence that the opportunities to leverage relevant collaborators to support and accelerate results were an important consideration incorporated into the design of each focus area. NRCan also expected the ICI-CTS to encourage proactive engagement and collaboration with traditional and non-traditional stakeholders to enhance Canada’s innovation ecosystem. The challenges are thus designed to provide a unique platform for networking and forming collaborative arrangements, highlight promising research, development and demonstration (RD&D) ideas and help the ecosystem overcome barriers.

What We Found - Governance

Roles, Responsibilities and Accountabilities within the ICI-CTS

The governance structure for the management of the ICI-CTS at NRCan has been defined in authoritative documentation that describes the roles, responsibilities and accountabilities of NRCan, a key federal partner for the Impact Canada Initiative. The ICI-CTS is supported through the organizational structure of ETS’ OERD, responsible for the Program’s delivery. The authoritative document reviewed refers to a well-defined program framework that would leverage OERD’s existing governance structure, given its experience with transfer payment administration, and identifies resources for ensuring the successful delivery of the Program, with defined roles and responsibilities.

The complementary and overarching roles, responsibilities and authorities of PCO’s COE on Impact Canada, as well as its interactions with federal partner departments operating as part of this initiative were also described, with clear delineation across the two governance structures. Documents recognize that PCO’s authority would not extend to direct program management or funding but instead, assist with overseeing and enabling the departments’ approach to innovative programming. NRCan would operate using the Ts&Cs of PCO; however, NRCan would remain the authority for the direct management and implementation of the ICI-CTS. PCO would primarily be responsible for tracking progress on the approach of the ICI from an overall perspective, as well as general outcomes.

At NRCan, key roles and responsibilities for Gs&Cs are shared at the corporate and sectoral levels. The NRCan Guide on Transfer Payments identifies key players for the administration and management of Gs&Cs, including Program Staff, the Center of Expertise on Grants and Contributions (CoE on Gs&Cs), Legal Services, the Finance and Procurement Branch (FPB), Delegated Authorities, and the Transfer Payment Review Committee (TPRC), which has since been replaced by the Finance and Real Property Committee (FRPC). Overall, the roles, responsibilities and accountabilities of key intradepartmental players for the management of Gs&Cs programs were found to be defined and communicated through the departmental intranet website and key guideline and procedural documents and well understood within the Department.

Across the organization, the governance structure and processes in place for the management of Gs&Cs were found to facilitate the involvement of key intradepartmental players in situations where required, or requested. In the case of the ICI-CTS, and as with all new Gs&Cs programs at NRCan, these structures necessitated the involvement of Legal Services and the CoE on Gs&Cs in performing oversight and review activities of the development of Ts&Cs for the ICI-CTS and subsequent funding agreements. We found that early engagement involving NRCan’s key players for Gs&Cs had taken place, which played an integral role throughout the program’s design and development.

Subsequently, the program faced some difficulties during the initial stages of its design and delivery attributed to systemic and departmental barriers. Specifically, it was noted that there were often opposing perspectives in interactions between the program and corporate functions. The ICI-CTS was looking to challenge status quo and experiment with innovative ways of doing business, whereas NRCan’s corporate functions are responsible for enforcing policies, providing guidance, and identifying and containing areas of perceived risk or concern. Many of the challenges faced were found to have resulted from a lack of communication of and familiarization with the policy context and authorities offered through PCO’s Ts&Cs. This led to fundamentally different views in the interpretation of the program’s Ts&Cs, and the intersection of these viewpoints among corporate colleagues and the program led to resistance during the earlier phases of challenge design and development. However, since that time, the differences between various key players have been addressed with an ongoing collaborative approach.

Internal Reporting Mechanisms

Through an examination of various reports provided to senior management on the progress of the Impact Canada Initiative, we found that formal reporting structures were in place at all levels of management. The formats of these documents include briefing notes, presentation decks, speaking points, and corresponding project documentation, provided to the Director General (DG), Assistant Deputy Minister (ADM), Deputy Minister (DM), and the Minister’s Office at NRCan, in addition to several other departmental executive-level senior management committees, where relevant. The frequency of these internal reports to senior management were found to take place at intervals that supported timely and informed decision-making by executive management. Given the high profile nature of the ICI, many of these internal reporting documents were shared in order to provide updates, and to obtain Ministerial endorsement/approval relating to the proposed challenge design, structure and format for funding.

We reviewed reporting documentation beginning as early as challenge design and development to challenge launch, which spanned the period from April 2017 through to December 2018.

We also obtained reporting documentation from the implementation phase of the challenges, in the form of Memorandums to the ADM, which presented key project-related information. These reports provide key information on applications received, assessed, eligibility, evaluation and ultimately, a recommendation for approval. The interval of these reports varied as they depended on the each of the challenges unique structure and timelines.

Through the review of the documentation provided to senior executive management, we found that the information provided contained sufficient detail to inform the future phases of challenges that were, at this time, developed and launched. Overall, the team concluded that information provided to senior management contained sufficient detail to inform the future phases of challenges.

What We Found – Design

Challenges have a set of inherent features meant to attract new talent and new ideas, and accelerate progress towards solving problems that matter to people. As a result, challenges are designed, promoted and delivered differently than the types of funding programs that governments and their stakeholders are accustomed to running and participating in. Overall, we found that the design of focus areas under the ICI-CTS was consistent with principles of Impact Canada and best practices in challenge models. Many focus areas also incorporated numerous experiments within this broader experiment, for example, testing new models of Indigenous participation (IODI) or new ways of working with international partners (Power Forward). We examined how and for whom the program’s experimental tools are effective in facilitating the achievement of intended outputs and outcomes, with a focus on five elements common to all focus areas, i.e.: inclusive innovation (co-design), communications and outreach, collaborative arrangements, flexible funding mechanisms, and additional (non-financial) proponent supports.

Inclusive Innovation (Co-Design)

The first key steps in any challenge are to understand, identify and define the specific problem that the challenge will seek to address and set the design parameters that will support its most effective delivery. Literature supports the benefits of inclusive innovation (co-design) in supporting this process. Co-design processes brings together experienced, diverse groups of internal and external stakeholders to apply their collective knowledge and form synergistic programming. Engaging with stakeholders is critical for identifying and understanding the issue, and selecting and defining a specific problem that will form the basis of the challenge. Stakeholders can also confirm that a challenge model is feasible and appropriate to seek solutions to the problem and that the financial and non-financial supports to be used are designed to meet the needs of proponents and end-users, thus helping to achieve outcomes faster and at lower total cost. Engagement of diverse stakeholders during the co-design process also increases network connections and strengthens problem-solving communities, creating further opportunities for collaboration that can support long-term innovation.

We found that all focus areas undertook a co-design process. NRCan used its own existing networks and those of its collaborators to reach out to a wide range of stakeholders in the relevant ecosystems, including traditional and non-traditional stakeholders. The timeline and specific approach to co-design varied significantly by challenge, with mechanisms used to engage stakeholders that included consultations, webinars, design workshops and surveys. However, each followed an iterative process that allowed the challenge concept to be validated more than once by groups of key stakeholders, with final approval by NRCan senior management.

Interviewees familiar with the co-design process agreed this was among the ICI-CTS’ main strengths. It allowed NRCan to refine the design to target key issues but also scale the challenge based on given design parameters (time and funds that could be allocated by NRCan) to what would be achievable by solvers, including consideration of specific barriers and pathways for the problem area. There is a risk in co-design of being so responsive to feedback that people propose and vote on ideas that are not practical or feasible. NRCan was perceived to have successfully managed stakeholder expectations in this area. While consensus could not always be achieved, there is evidence that NRCan shifted design of challenges as a result of co-design input. For example, the IODI shifted its design from a “challenge” to an “initiative” with feedback from Indigenous stakeholders that competition between communities as part of the off-diesel program was not in line with their values. Extensive engagement also shifted its design from a technology to a capacity building focus. 

Co-design also brought visibility to the challenges, as an early signal to key stakeholder groups that a challenge would be launched. For most challenges, there is evidence that the co-design process also resulted in furthering collaboration among players in the ecosystem. Given the international nature of Power Forward, inclusion of high profile events in co-design was important to not only collect stakeholders’ insights to support challenge design but also to start building partnerships critical to challenge delivery. By extension, given the small size of the nascent innovation ecosystems, there is also a risk of unfairly advantaging stakeholders engaged in co-design who may later also become proponents. There is evidence that NRCan considered and mitigated this risk.

Interviewees did not perceive significant barriers to participation in co-design. For IODI and WICT, there was a clear focus on co-design on inclusivity – i.e., who benefits from the innovation, who participates in the process of innovation, and who manages the outcomes of innovation. This means that co-design did not stop at challenge launch. These focus areas continue to be co-designed with input from finalists to ensure that capacity building elements and other supports meet their specific needs. Community engagement and support is critical to advance projects under the IODI, ensuring that those participating have a say in the design of projects that will impact their community. Both WICT and IODI also use an external jury that reflects the participants in the challenge itself. For IODI, many respondents viewed this co-design process as a positive step towards Indigenous self-determination.

Communications and Outreach Strategies

Literature supports the importance of communications as a core element in challenge design. Recognizing that this would be a key determinant of the overall success of the program, NRCan’s program team allocated about 12% of its initial operating (O&M) budget to communications.

There is evidence that the ICI-CTS program worked closely with NRCan’s Communications and Portfolio Sector (CPS) to develop and apply a front-end communications and outreach strategy for each challenge and initiative. The objectives of these communications were to promote visibility of the focus areas to potential solvers, looking to attract more non-traditional solvers and small-scale innovators than previous programs, and to gain exposure for participants and collaborators. While the specific strategy varied by focus area, there is evidence of a wide range of media and approaches used to achieve communications objectives. Each challenge and initiative worked with collaborators to promote awareness of the program. This external engagement was important to maximize reach given the nascent or disperse nature of targeted ecosystems and the objective to attract non-traditional solvers (who by definition are not part of NRCan’s existing network). Participation in major conferences and events helped garner interest, attracting attention with high-profile participants like the Prime Minister of Canada and Margaret Atwood. Focus areas also made use of PCO’s centralized website, with four of the challenges using this as a key communications platform (WICT and IODI relied more heavily on their collaborators’ websites).

Survey respondents most commonly reported hearing about an Impact Canada challenge via either an innovation, clean tech or industry network and/or from a friend, colleague or employer (see Figure 3). Similarly, the majority of finalists interviewed stressed the importance of NRCan’s collaborators and general word-of-mouth in notifying them of the opportunity to participate. These results varied significantly by challenge. For example, survey respondents for CTF were the only respondent group for whom the Impact Canada website was identified as the most common means to learn about the challenge (37.5%). WICT was the only challenge for which social media was reported to be an important source of information (30%).

Figure 3: How did the ICI-CTS’ applicants and semi-finalists hear about Impact Canada?

Interviewees generally agreed that these communications were effective in attracting “enough” of the “right people” to apply (based on the number of quality proposals). Regardless, given its importance to challenge delivery, program staff perceived that this could have been better resourced (e.g., a dedicated communications resource within the OERD team and/or more support from CPS). In their view, capacity within NRCan’s CPS following the launch of focus areas was a major factor limiting the program’s delivery of communication products and activities. Given the importance of community support to the IODI, some Clean Energy Champions also suggested that culturally appropriate (or translated) speaking points may have aided in explaining this initiative to their community. Otherwise, NRCan’s approach to the IODI was reported to be appropriately adjusted for Indigenous participants with consideration for culturally appropriate communications.

We also examined the design and delivery of communications that would give profile to the solvers and the issues addressed by their solutions as they progress through the challenge, gaining exposure that could help attract potential collaborators or investors. Survey responses suggest that outreach and communications was a motivational factor in some respondents’ decision to participate in the ICI-CTS, though by far not the most important (ranked 10 out of 15 choices). Again, there is variability among focus areas, with the importance of exposure ranked lower as a motivation for participation in the IODI.

There is evidence of some strategies in place for communications to promote solvers. To date, participant-generated content (e.g., presentations at pitch events and showcase videos to be shared on social media) has been the most important feature. In 2020, a social media strategy was also developed aimed at ramping up the promotion of finalists as challenges move towards the selection of winners. However, with exception of WICT (where communications are managed by MaRS), finalists were generally less positive about communications on program progress than for program launch noting that there was less opportunity or emphasis on this than they expected when they first applied to participate. While noting that delivery of communications benefitted from the collaboration of PCO, some interviewees perceived this to have made the communications process more bureaucratic and less in line with innovative communications observed in benchmark challenges. Interviewees also suggested that the PCO platform could have been used more effectively and updated more frequently to better promote the challenges.

From March 2020 to date, there have been limitations on PCO’s communications and for publication on NRCan’s social media due to COVID-19. Conference participation and other planned events have also been reduced and shifted to a virtual context. Of the six focus areas, CTF was the last to be launched in October 2019. COVID-19 was thus perceived to have the greatest impact on communications for this challenge, delaying the announcement of finalists for a couple of months. Finalists noted that this was not optimal as it removed some of the “bang” from the announcement and impeded their ability to advertise this information to attract collaborators and investors at a critical stage.

Collaborative Arrangements

Literature indicates that collaborating with various players in the innovation ecosystem to co-develop and deliver activities in challenge models can spark breakthrough innovation and solve difficult technical challenges faster and at lower total cost. To this end, Impact Canada supports the ability of federal organizations to collaborate with expert intermediaries (and/or an international partner in the case of Power Forward) to build the capacity of innovators and communities, to support challenge delivery, or to deliver challenges (or specific components).

All of the ICI-CTS’ focus areas engaged collaborators, either formally or informally, to co-develop and deliver activities. The number and role of collaborators varied significantly by challenge and initiative, activities supported can generally be broken into five broad categories:

1. Outreach. All focus areas worked with one or more collaborators to deliver strategies to increase NRCan’s reach in attracting proponents. This was particularly important given the nascent and/or fragmented nature of the related innovation ecosystems. Collaborators were reported to play an important role in raising awareness of the challenges for applicants. Overall, 16.3% of survey respondents identified “the website of a collaborator in program delivery” as a key means by which they first heard about the challenge.
2. Networking and Ecosystem Development. Many focus areas worked with one or more collaborators to facilitate proponents’ partnership building efforts, including where the development of consortiums was required pre-application. This support was particularly important for Power Forward given the tight timelines (5 months) available to build international partnerships pre-application.
3. Facilitating Scale-up of Solutions. Many focus areas engaged collaborators to provide services that would assist proponents in scaling-up their ideas by building capacity for commercialization and business development and/or by identifying and helping to remove related barriers. For example, NRCan engaged the Pembina Institute on the IODI to provide policy support intended to address barriers to community-driven clean energy projects both for participating communities and others who may later face similar barriers. Similarly, NRCan engaged the Centre for Mining Innovation (CEMI) on Crush It! to undertake outreach and engagement that would enhance Canada’s mineral processing innovation ecosystem by mobilizing new and existing solver communities, building new linkages across the innovation landscape, and setting the stage for the commercialization and adoption of emerging solutions. To enhance diversity of problem solvers, this included offering commercialization and business development services to support scaling-up the ideas of clean tech innovators in mineral processing to clients that were not participants of Crush It!
4. Delivery Agent. Two focus areas used collaborators as agents in delivery. The IODI is being delivered with the Indigenous Clean Energy (ICE) Social Enterprise (see Box 2). NRCan engaged MaRS, a pioneering organization in the field of business acceleration for innovators, as the lead delivery agent for WICT. NRCan gave these two collaborators important roles in delivery of the application process, including supporting development of evaluation criteria and identification of jurors, and the development and delivery of capacity building. Interviewees perceived the engagement of these non-government collaborators as delivery agents as allowing challenges to be more nimble than they otherwise may be if delivered by the federal government.
5. Co-Delivery Agent (Partner). NRCan established a formal international partnership for co-delivery of Power Forward with the UK Department for Business, Energy and Industrial Strategy (BEIS), with responsibility for funding of proponents and provision of awards shared between the two jurisdictions.

There is evidence that NRCan selected collaborators based on the alignment of their mandate, their expertise and established credibility, and/or the reach of their existing networks. Overall, program representatives and finalists perceived that the collaborators engaged to support delivery were both appropriate and effective. In particular, working with ICE and an all-Indigenous jury on the IODI was perceived as a positive move towards new model of collaborative (rather than Crown-led) program delivery.

Box 2: Indigenous Clean Energy Social Enterprise (ICE)

Flexible Funding Mechanisms

The Impact Canada Initiative’s Terms and Conditions for Grants and Contributions (Ts&Cs) are built in a flexible manner to support departments in their experimentation with prizes, challenges, micro-funding, and outcomes-based and innovative programming approaches using grants and/or contributions. Under the ICI, challenges take place within the same basic structures and rules as other funding mechanisms, but have distinct features that differentiate them from traditional programs (see Table 4). This may include instances where recipients are not known in advance and no agreement is signed with a recipient in advance of achieving the outcomes set out in the prize or challenge.

Literature indicates that there is no precise science to setting the value of a financial prize, but it should be proportionate to the investment innovators are asked to make in developing a solution as well as the potential market value of the new product or service. The prize does not need to cover the entire cost of the development but it does need to cover at least some of the risks challenge participants assume. The amount and distribution of funding devoted to the challenge model must be large and secure enough to shape priorities and influence concrete action. A stage-gate approach, incorporating shorter-term success milestones can facilitate interim progress and collaboration between solvers to develop better quality prototypes or solutions.

Table 4: Comparing Challenges to Regular Government Funding Mechanisms

Source: Adapted from PCO, Impact Canada Challenge Guide (2021)

We found that the design and delivery of flexible funding mechanisms under the ICI-CTS was consistent with these best practices and the principles of flexible funding under Impact Canada. With a uniform emphasis on payments based on outcomes, focus areas used a mix of micro-grants, grants and contribution agreements. These funding decisions were informed by co-design and through consultations with industry experts. For at least two challenges (PF and WICT), there is evidence that the design of funding mechanisms was also developed with consideration and comparison to other government-led funding programs and/or inspiration from benchmark accelerator programs (e.g., Cyclotron Road fellowship program at the Lawrence Berkeley National Lab in California) and signal checking against other prizes. Funding from NRCan was not sufficient to fully advance technologies, nor was it intended to be. This enhanced participants’ commitment to the success of the project as, with exception of IODI, their organization(s) still had to assume some financial risk. While projects needed to be scaled to the supports available, funding for the ICI-CTS was generally perceived to be more generous than that available from other government programs and/or to be well directed to fund a specific gap not met by other programs.

STL’s Cross-Canada Flight Challenge was the only pure prize challenge. Most other focus areas (including STL’s Green Aviation Fuels Innovation Competition) used a stage-gated approach, with each stage building on the previous and rewarding proponents for advancing their solution. Program representatives perceived this to lower the risk related to each funding agreement, as payments were split between milestones. Perhaps more importantly, the design of this staged funding gave explicit consideration to how funding could mitigate barriers to increase participation from diverse problem solvers and help them advance in this problem space, regardless of the final outcomes of each finalist. For example, the funding structure for Crush It! helped level the playing field by awarding selected small-scale innovators $10K each to help prepare and present their ideas to challenge jury. Similarly, NRCan and BEIS offered support to small and medium sized enterprises (SMEs) wishing to participate in Power Forward to enable consortium-building and project development activities, which could include feasibility studies. The WICT challenge gave finalists an annual stipend and the IODI paid Clean Energy Champions a salary to allow them to dedicate their time to their project. For IODI, staged funding was also essential to address a need within communities for up-front investment to reach the initiative’s objectives. Funding for the IODI was also adjusted during implementation in response to feedback from Champions and delivery collaborators, with a decision to issue micro-grants of $25K in Phase 2 to address the issue of funding from Phase 1 contribution agreements flowing more slowly than expected.

There is evidence that this design strategy was effective in attracting solvers; participants across all focus areas perceived financial supports as among the most important motivational factor in their decision to participate.⁷ This is particularly true for staged funding (including stipends and salaries), with close to 94% of survey respondents identifying this as very important or critical to their participation. By contrast, the final prize award was only identified as an important motivation by 80% of respondents. This is consistent with feedback during interviews, where finalists explained that their decision to participate relied more on how far they believed their solution could advance with the certainty of staged funding than a final prize that they were uncertain to win.

The design and delivery of the financial supports was also positively viewed by all interviewees, including finalists. In general, interviewees appreciated the flexibility that the outcomes-focused funding model gave to the proponents to determine how funds should be allocated specific to their project. This approach was also perceived to lower the reporting burden for proponents (compared to receipts-based reporting), which allowed solvers to spend less time on administration and more time on advancing their solution. However, milestone-based payments were also a new approach for both NRCan and proponents, and respondents in both groups reported some difficulty in determining how to define and report against relevant milestones. This took additional up-front effort to resolve. While the majority of interviewees were positive, there were also mixed views on the extent to which in practice the timing of milestone-based payments more effectively supported participation of SMEs; some final recipients noted that there could be more flexibility in this area to meet the particular cash flow needs of each solver. Some finalists also noted a possible barrier to participation from academia, as the financial practices of most academic institutions do not allow for payments based on milestones. The extent to which this is an issue would depend on the nature of the specific challenge.  

Across all focus areas, most finalists (particularly SMEs and start-ups) noted that projects would likely not have been able to advance or would not have advanced as rapidly without these funding supports. At the same time, we found that the pathways to scale success in all problem areas requires investment which exceeds the amounts that could be provided through the ICI-CTS’ prize awards. This is consistent with the theory of challenge-based models and program expectations, i.e., the funding provided by the ICI-CTS was not expected to be sufficient to bring technology towards full commercialization; it was intended to accelerate the process and help to leverage follow-on investments. Given resources available, the development of follow on strategies for participants to continue to build on their success after the final prize award is outside the scope of the program. To date, only WICT and IODI have secured and communicated a clear follow-on strategy. Even so, while Champions for the IODI were clear that this is not meant to be a competitive challenge, at the time of data collection there was still a lack of clarity around how the final $9M would be allocated among communities. Some interviewees perceived this to be increasing feelings of competitiveness as the IODI moved closer to the final deadline.

Finalists across all focus areas perceived the absence of clear follow-on strategies to be a gap. However, many proponents also perceived the level of funding provided to advance their idea as a “stamp of approval” in their solution by NRCan that they could use to entice further investment or collaboration into their solution, thus accelerating results. The program is also designed to increase intelligence (knowledge and application) of the state of technology and of the innovation ecosystem. Moving forward, this intelligence is available to inform perspectives for other programs and actors and may also accelerate the design or implementation of parallel initiatives in related focus areas.

Proponent Supports

Literature indicates that it is a best practice for challenges to incorporate supports that build capacity within participating organizations and mobilize further investment. A prize structure that offers non-financial support such as support and advice from experts, access to testing facilities, networking opportunities, and exposure to investors can be particularly useful for challenges that require innovators to develop new knowledge and capacities.

Similar to financial supports, we found that the design and delivery of non-financial supports under the ICI-CTS was consistent with these best practices and the broader principles of the Impact Canada Initiative. The design of these supports was tailored to the needs of each focus area, as informed by co-design and through consultations with industry experts. However, perceptions of the effectiveness of non-financial proponent supports varied by focus area (Table 5).

Survey results indicate that, where applicable, non-financial proponent supports were much less important for motivating participation than financial supports. Across all challenges, finalists indicated that the level of support required depended on the existing maturity or experience of the proponent organization and the Technology Readiness Level (TRL) of their solution (Box 3). Participants in WICT and the IODI perceived non-financial supports to be somewhat more important in motivating their participation, particularly access to experts, mentors and capacity building (and in the case of WICT, business incubation support).

Box 3: Technology Readiness Level (TRL)

Overall, there was also a significant difference in perspective among finalists and semi-finalists; semi-finalists were generally much less satisfied with non-financial supports.⁸ Satisfaction in supports for networking and outreach was higher than satisfaction for capacity building and technical support. However, no respondents identified difficulties in accessing any of these non-financial proponent supports.

Table 5: Perspectives on the Effectiveness of Non-Financial Proponent Supports

As indicated in Table 5, one of the innovations being tested by the ICI-CTS was the use of federal labs to support challenge delivery (included as a support for WICT). Providing access to these resources presents an opportunity to mobilize new players in the clean tech ecosystem as federal labs have more expertise and equipment than some smaller firms or consortia may have access to on their own. For WICT, the use of labs as a design feature included implicit assumptions that: (1) finalists supported by the challenge required the support of federal labs; and (2) there was a research centre whose equipment and expertise would be a good match for each finalist. Overall, interviewees agreed that there was value to finalists in accessing federal lab support and that this helped speed up technology advancement. However, the specific mechanism used to secure and fund this support federal lab support was new for NRCan, i.e., Science and Technology Assistance for Cleantech (STAC) model (Box 4). Finalists’ projects are also very diverse, addressing solutions in very different clean tech sectors and stages of business development (TRL). Respondents reported that it was thus difficult to match and negotiate agreements with each distinct lab given Canada’s lack of centralized lab structure, with finalists perceiving that there could have been better or more coordination between program staff and federal labs in the lab matching process. There is also evidence that it took time to resolve issues related to administrative procedures and finalize funding agreements with each lab. As a result, some STAC collaborations started early and ended early in the program; whereas, others started late and will end late in the program (e.g., the finalists were announced in September 2018, but one lab agreement was not signed until February 2020).

Box 4: Science and Technology Assistance for Cleantech (STAC)

We also found some evidence of a greater demand for access to federal labs. In absence of this as part of challenge design, a small number of proponents in STL and CTF independently engaged federal labs as partners to support their work on a fee-for-services basis (not supported by money from NRCan). While not an issue created by ICI-CTS, some indicated this as a potential barrier to mobilization of new talent as private industry does not always know how to access services from federal labs, or even what equipment or expertise may be available.

What We Found – Processes and Controls

Selection Process

Pre-determined assessment criteria were generally found to have been finalized and communicated prior to challenge launch across all focus areas. The criteria were found to contain clear and detailed eligibility and assessment information, which was publically communicated to interested parties in advance through online publication, supported by Ministerial announcements. However, we did identify the following opportunities for improvement for focus areas that used a delivery agent as part of the assessment process:

• For the IODI, the majority of the eligibility and assessment criteria were developed in advance of challenge launch. Selection criteria for the first phase of this Initiative, led by ICE, were found to have been finalized after launch but prior to any selection/evaluation process. The finalized assessment criteria did not deviate significantly from the eligibility criteria previously developed. Further, consistent with co-design principles, information on the nature of the eligibility criteria were shared and discussed with potential applicants through engagement sessions before they were finalized.
• For WICT, where assessment criteria were developed by MaRS, there was an opportunity to clarify how these criteria were to be applied to applicant’s proposals as part of the assessment process.

For each focus area, we found an evaluation committee and/or Jury was established to review and assess applications. However, a complete and finalized listing of committee membership was not always available in documentation provided. Although a listing of committee members is not a requirement, it is an overall good practice as it helps to ensure information is available, documented, and complete.

We also noted that documentation was generally sufficient to satisfy selected proponent’s assessment against the selection criteria of the individual challenge or initiative; however, there were a few instances where this documentation was missing or incomplete. For Crush It!, an opportunity was noted to improve the availability of documentation surrounding the decision-making process that takes place through consensus meetings. For WICT, there was an opportunity for NRCan to improve its retention of documentation related to the process completed by MaRS as an external delivery agent.

Funding Agreement Compliance

The NRCan Center of Expertise (CoE) on Grants and Contributions (Gs&Cs) is responsible for providing guidance and advice related to Gs&Cs within the following areas: Treasury Board submissions, program, risk, and agreement management, policy and procedures, service standards, financial systems, and recipient audit and evaluation. In fulfilling its mandate, the CoE’s roles and responsibilities extend to the development of tools, guides, checklists and other sources of information to assist program managers in administering their transfer payment program in addition to providing training to all employees on the management of Gs&Cs.

NRCan has developed documentation, tools, templates and processes to assist program managers when launching new programs and drafting contribution agreements. The CoE established the NRCan Guide on Grants and Contributions, a resource that outlines policy instruments, processes, templates and tools, on-line training and expert advice. The CoE supports programs by ensuring staff have the tools, advice and direction in the management of transfer payment programs and the use of public funds. The NRCan Guide on Grants and Contributions contains several resources to help guide program managers in their development of grant and contribution agreements, including tools/templates for generating and amending grant and contribution agreements. The contribution and grant agreement generators, along with their schedules, provide users with a standard NRCan approved template for use when drafting agreements.

A sample of funding agreements were selected and reviewed in order to determine whether they were developed in accordance with TB and NRCan policies, directives and guidelines. The agreements examined included representation across the six focus areas under the ICI-CTS, and included a selection of both grant and contribution agreements. Overall, we found that the majority of selected agreements reviewed satisfied the requirements for their development. The program consulted with corporate functions (i.e. Legal and CoE) to assist with the review of select agreements and obtain the necessary endorsement, where required.

Risk-Based Accountability Structure

Recipient audits serve as component of the continuum of activities that should be used for ongoing monitoring activities by program staff. The CoE on Gs&Cs has developed an NRCan Guide on Recipient Auditing (last updated February 2020) to provide guidance to program managers on their roles and responsibilities for recipient auditing and a Recipient and Project Risk Management Model (RPRMM) in order to assess the risk of projects. According to the RPRMM, recipients having an overall high-risk level may be subject to mandatory recipient audits, with a random selection of recipient audits for recipients that have low-to-medium risk levels.

Overall, we determined that the OERD has developed and implemented a broad, branch-level risk-based recipient audit plan, which incorporates the selection of projects through the ICI-CTS and follows the CoE’s RPRMM. This plan is updated annually through a call-out exercise, whereby project managers consider factors throughout an agreement’s administration, including new or emerging risks, perceived issues, or concerns that may arise, for possible inclusion. The risk-based recipient audit plan was found to provide a snapshot of both active and upcoming audits, and, includes a selection of recipients for audit consideration identified as part of mandatory random probability sampling for moderate-to-low risk recipients.

The engagement team also confirmed that recipients selected for audit are captured within NRCan’s Agreements Module Interface (AMI), a database containing departmental information on Gs&Cs agreements. The AMI system allows users to track recipients selected for audit, the status/completion of the audit, findings, and lessons learned, however, the information captured is minimal in nature. Given that completed audits and their recommendations do not appear to be formally tracked and monitored as part of OERD’s recipient audit plan, or through other developed mechanisms, it is not clear how recommendations are managed and resolved. In the absence of a process to track and monitor audit recommendations, the Program’s ability to consider the results of past audits and apply lessons learned to future projects may be limited. In addition, while the recipient audit plan considers recipients for mandatory random probability sampling, we were not able to determine the methodology and selection process to inform the random selection of these recipients.

Payment Compliance

The engagement team reviewed a sample of 25 funding agreements across the six focus areas to determine whether funding agreements comply with relevant FAA requirements. The findings of this examination are outlined in Table 6.

Table 6: Compliance of Payments with FAA Requirements

Proactive Disclosure

The Access to Information Act (ATIP) requires the public disclosure of information related to grants and contributions with a value over $25,000 (as well as those Gs&Cs under $25,000 that are amended with a value exceeding $25,000) to be published in electronic form. The Treasury Board of Canada Secretariat (TBS) Guideline on the Reporting of Grants and Contribution Awards supports the proactive disclosure information requirements under the ATIP and provides guidelines in the identification, collection, publication and monitoring of Gs&Cs data.

Although the requirement for the Government of Canada to proactively disclose the awarding of Gs&Cs information is not new, Bill C-58, which received Royal Assent in 2019, included amendments to the ATIP  (Section 87) to require that Gs&Cs information be reported within 30 days after the reporting quarter in which the funding agreement is entered into. For all 25 sampled Gs&Cs agreements examined, proactive disclosures were found to be verified for accuracy prior to publishing, received approval for reporting, and information related to these Gs&Cs agreements were all publically disclosed. However, for 12 of 25 sampled agreements, the proactive disclosure was not performed within its respective reporting period, as required by TBS. The NRCan CoE on Gs&Cs monitors the status and timelines of Gs&Cs information to be disclosed, and has implemented a call-out process reminding Programs of upcoming cut-off dates. In instances where information is not available by the quarterly due date, it is escalated and reporting to senior management and posted in the following quarterly batch.

Conflict of Interest

The NRCan Guide to Transfer Payments contains high-level guidance related to the requirements for the selection of members responsible for the review and evaluation of applicants’ proposals. Required practices include ensuring the evaluation committee is established prior to reviewing applicant proposals, includes three or more persons, and that its members are separate and independent. The onus is on Sectors to determine whether real or perceived conflict of interest (COI) considerations exist.

Through our review of documentation of the selection/evaluation committees and jury members, evidence was available to support program consideration for COI across all focus areas. We found COI considerations were included and documented in various forms, including individual COI attestation forms, or consideration for COI through the selection/evaluation committees’ or juries’ terms of reference. Overall, it was evident that the selection/evaluation committee and juries (where applicable) included consideration for COI among its members.

What We Found – Expected and Unexpected Outcomes

We examined the extent to which the ICI-CTS is making progress towards program-level immediate outcomes identified in its draft logic model (Figure 2). These include:

• Increased collaboration in key technology challenges and projects, as evidenced by the development of collaborative networks and partnerships.
• Increased investment by stakeholders in the clean tech ecosystem, as evidenced by the ratio of funding leveraged.
• Increased awareness and participation in clean tech, as evidenced by an increase in non-traditional proponents and collaborators (i.e., mobilization of “new talent”).
• Increased public awareness of clean tech issues among Canadians, as evidenced by the reach of communications.

As described in earlier sections of this report, there is evidence that the design and delivery of each focus area considered and worked to achieve these immediate outcomes. While the calculation of results against most NRCan indicators is not possible given data available to this engagement, evidence suggests that the ICI-CTS is making progress in each area.

Performance Measurement

Impact Canada uses innovative approaches that necessitate new methods and strategies for measuring program outcomes and impacts. Literature indicates that key differences in the design of challenge-based models also extend to performance measurement. By design, challenges are outcomes based initiatives, wherein key performance indicators are the challenge assessment criteria. As a result, some of the incremental and proxy indicators that are typically used to assess progress of traditional programs are not necessary. If the result is not achieved, the funding is not awarded. 

This does not mean that additional performance metrics are not required. Consistent with its broader objective to accelerate the use of innovative and experimental approaches across the Government of Canada, the ICI seeks to understand how the challenge approach itself adds value as a public sector instrument (as compared to traditional ways of doing business). According to PCO, there is a need to generate better evidence that the perceived risks of running a challenge, and the level of investment that is required to run them, is justified by the achievement of better outcomes. Ongoing monitoring of performance against key metrics is also important to accountability, ensuring that senior management receives comprehensive updates of the program’s performance as it progresses.

In 2017, funding authorities outlined a notional performance framework for the Impact Canada Initiative as a whole, as well as specific outcomes and targets for the Clean Technology Stream. Each focus area was also to have its own unique set of goals and performance targets, with performance information to be collected on an ongoing challenge-by-challenge basis. Given the broad range of possible pathways that challenges and individual projects could take, OERD indicated that work on these methods could only start once it had a clearer indication of the specific innovation pathways and solutions to be pursued.

In 2019, PCO released a first set of guidelines for impact measurement in support of its work under Impact Canada.⁹ In 2020, PCO released a logic model that presents a basic theory of what Impact Canada expects to achieve with its broad suite of challenges.¹⁰ The outcomes in this logic model are consistent with the original framework approved in 2017. These include:

1. Immediate outcomes related to the challenge process – i.e., enhanced public awareness, mobilization of new talent, and building of collaborative networks and collaborative arrangements;
2. Intermediate outcomes related to development of sustainable business models – i.e., increased investment, enhanced skill and capacity, and development of innovative products, services, or technologies; and
3. Ultimate outcomes linked to socioeconomic and environmental benefits for Canadians.

NRCan developed a draft logic model for the ICI-CTS in 2019-20 (Figure 2). In March 2021, OERD indicated that it still working separately and with PCO to enhance the related performance measurement strategy, including additional indicators, methods, data sources and potential counterfactuals, with an emphasis on methods for tracking results post-program (other than self-reports). OERD is also still working towards developing more challenge- and solution- specific outcomes and methodologies that will allow for more precise and sensitive monitoring of results following the announcement of grand prize awards. Currently, OERD is working with others (e.g., PCO, Statistics Canada, and the University of Alberta) to explore methodologies to measure the outcome of programs, including use of quasi-experimental methods that combine Impact Canada program data with administrative / statistical data to assess key outcomes for challenge participants against comparator groups. PCO is also collaborating with Statistics Canada to conduct a horizontal impact assessment of challenges across the federal government, using quantitative and qualitative data. However, program managers noted that overall progress in this area has been delayed due to resource constraints and COVID-19, further complicated by the lack of existing methods for effectively tracking and measuring intermediate-and long-term effects of prize-based challenges.

Within this context, program managers for the ICI-CTS are aware that numerous difficulties in the design and application of performance metrics still need to be resolved. The engagement found that these include but are not limited to:

• Lack of Program Comparators. A key question for the ICI-CTS is whether the challenge model worked better than known alternatives. However, for most focus areas, there are no bundles of projects in core programming against which to make a relevant comparison. Where comparison may be possible, results may be skewed by the design of OERD’s core programs. For example, prior or concurrent OERD funding calls often intentionally support the same technology or proponent at different stages of innovation, and therefore may have higher results by default. Other programs (e.g., NRCan’s Energy Innovation Program) often use very targeted funding calls that may limit the pool of applicants, thus limiting their relevance as comparators for proposed metrics such as the number of applicants per dollar of NRCan funding. The ICI-CTS’ challenges also do not have the same matched funding or (with exception of consortia-based challenges) collaboration requirements as traditional programs, which could also skew comparisons.
• Definitional Issues. Performance metrics include some basic terms that still require better definition. For example, who is included in the definition of “new talent”? Does this include just the main applicant or all project partners? In what ways does an organization need to collaborate in order to be considered a “collaborator”? Do inputs from growing consortia count as leveraged investments?
• Expected Results Expected to Evolve. Metrics related to business model outcomes such as investment and collaborative arrangements are expected to evolve over time and are subject to increase or decrease. Baseline data on related metrics was collected with applications, but progress towards outcomes requires frequent updating throughout delivery and for an extended period post-challenge award. It may take several years to be able to detect any changes in outcomes as compared to counterfactuals. This makes it significantly more challenging to track and benchmark progress.
• Limitations in Project Reporting.  There are ongoing reporting obligations for semi-finalists and finalists designed by the challenges. We observed that the templates for milestone reports were set up to collect data that would track progress but, for reports that were on file, the nature and quality of the information collected varied significantly – for some focus areas, reports provided few details that could be used to support performance measurement, with some sections left blank.¹¹ Project reports may also be insufficient for collecting data on longer-term outcomes or spin-off results, particularly if these results come from solvers that were not semi-finalists or finalists. The nature of the prize-based challenges is such that OERD cannot mandate follow-on reporting as it can with traditional funded projects. Quasi-experimental tracking methods do not rely on project reports but will likely require access to business or charitable registration numbers that were not consistently collected as part of applications.
• Gender-Based Analysis Plus (GBA+). There is no indication of gender specific metrics and data collection in place (other than as it relates to challenge eligibility for IODI and WICT). However, program staff indicated that both OERD and PCO are working with Statistics Canada to collect gender, equity, diversity and inclusion data at the firm level.
• Measuring the Effectiveness of Failure. NRCan’s draft logic model for the ICI-CTS focuses both on the results to be achieved by each focus area and the use of innovations to develop solutions to specific problems. As with traditional programs, the theory of change presented by this logic model is oriented towards the achievement of outcomes (i.e., success). However, in a challenge model, failure to arrive at a solution and award the final prize may also be a perfectly acceptable experimental outcome. How can challenge-based models best capture the effectiveness of this failure?

While NRCan has collected some data as part of applications and project reports, as noted above, the nature and level of detail of this data is inconsistent across focus areas and contains information gaps. Raw data has not been subject to analysis to date. These gaps limit what the engagement can conclude related to progress on immediate outcomes.

Increased Collaboration

Literature suggests that a key benefit of challenge models derives from strengthening the problem-solving community to encourage closer collaboration and long-term innovation. The ability to increase collaborative networks and partnerships in the program’s focus areas was identified as critical given the nascent state of the related innovation ecosystems in Canada, and the ICI-CTS created opportunities to build networks and partnerships among their communities of solvers (and in some cases end users such as commercial airlines) as an integral part of the process.

In the ICI-CTS’ notional performance framework (2017), NRCan targeted year-over-year progressive increase in the number of partnerships developed and an increase in the number of partners per dollar spent than comparable bundles of projects under core programming for each challenge. Calculation of results against these indicators is not possible given data available. However, there is strong evidence to suggest that finalists are developing their own formal and informal partnerships among diverse stakeholders. The extent and scope of these collaborative arrangements varies by challenge and by project.

Many finalists highlighted that they established new relationships across the ecosystem that would not have occurred or would have occurred more slowly in the absence of the ICI-CTS. Factors contributing to their success in this outcome area included networking supports and increased visibility. Finalists in consortia-based challenges (PF and STL) indicated that the requirement to form collaborative arrangements was beneficial but also presents some risks (e.g., ongoing partner commitment). Finalists in WICT and the IODI also noted the importance of support for networking among peers in their cohort.

NRCan also undertook significant efforts to identify, engage and connect players during the co-design and application phases. For example, data indicates that semi-finalists in Power Forward collectively established approximately 186 new connections (prospective partners) as a result of networking events in January 2019 and 70 new connections in June 2019. However, in contrast to results for finalists, less than half of all non-finalists surveyed indicated their collaborative networks or partnerships increased as a result of participation (Figure 4). Of survey respondents who indicated that they had formed new connections (n=28), only 13.2% indicated that NRCan or a collaborator had played an important role in facilitating these connections. These results were more or less consistent across focus areas.

Figure 4: Change in collaboration resulting from ICI-CTS reported by survey respondents

For the purpose of performance measurement, NRCan defined “partners” as project participants other than NRCan or the applicant organization. However, many challenges also engaged collaborators to continue to identify and build ecosystem connections throughout challenge implementation. This includes NRCan’s support to GARDN to create an online platform to facilitate networking on sustainable aviation fuel, and to CEMI to build new linkages across Canada’s mineral processing innovation ecosystem. The result is better mapping or networking of the innovation ecosystem for these focus areas. There is also evidence that external collaborators and federal players are now better connected. These connections can now be leveraged by NRCan and other organizations to advance future initiatives.

Increased Investment

As noted earlier in this report, funding from NRCan through the ICI-CTS was not sufficient to fully advance technologies, nor was it intended to be. Further investment in clean tech solutions is required to continue to advance solutions, regardless of the final prize. Literature indicates that challenges should incorporate supports that mobilize further investment and generate new opportunities for solvers. The design of the ICI-CTS’ focus areas incorporated opportunities to connect and build relationships with investors (supports varied by challenge).

In the ICI-CTS’ notional performance framework (2017), NRCan expected NRCan program funding for each challenge to be leveraged by participants at a ratio of more than 1:1 (including financial and in-kind investments) and an overall ratio of investments higher than that observed in core programming. However, none of the focus areas included a requirement to obtain matched or leveraged funding as a criterion for application. Rather, consistent with literature on prize-based challenges, NRCan anticipates that participants will accrue investments and in-kind support as projects advance, via networking and collaboration events, increased publicity, etc. While still supportive of Indigenous communities sourcing additional investments, NRCan has also explicitly excluded the IODI from its targets in this area.

Again, calculation of results against these indicators is not possible given data available. However, there is evidence indicating that most finalists’ projects have leveraged such investments, at least to some extent. For example, one finalist reported having raised over $5M from government and private sector sources since joining WICT. Evidence suggests that participants in PF and STL had generally made the least progress in this area, with finalists indicating that it was a challenge to attract investment in these areas at the technology’s current level of maturity (TRL) or given other market and regulatory barriers. However, further leveraging across all focus areas could reasonably be expected as projects continue to advance.

The nature of a challenge is also such that the majority will not win. Funding under the ICI-CTS was available to support a limited number of participants. Program representatives indicated that they forwarded the more promising proposals of some non-finalists to other potential funders, such as via the Clean Growth Hub.¹² Among all non-finalists and semi-finalists surveyed, only 7.8% indicated that their proposal was referred to another program or initiative. Nevertheless, among respondents indicating that they had continued to advance their project, 55.2% indicated that they had secured new sources of investment (financial or in-kind) since applying for the challenge.

Most non-finalists and finalists could not directly attribute increased project investment to Impact Canada (e.g., some had previously or concurrently applied to other funding programs) but finalists acknowledged that their success in this area was facilitated by the reputational “stamp of approval” gained by participation in the program. Finalists perceived being selected among finalists as validation of project potential that serves as a positive signal for investors and customers, particularly for prospective investors that were not familiar with the innovative technology being advanced by the project.

Mobilization of New Talent

Literature indicates challenges are strategically generally run in an open and transparent manner to encourage broad participation among diverse actors, in order to produce a wide range of potential solutions. Inclusive innovation invites diverse groups of people outside the existing or established innovation ecosystem to participate in solving public-facing problems, on the assumption that this will also bring new perspectives, skills, ideas, resources, etc. Extra provisions are required to draw out and engage traditionally underrepresented solvers. Challenges feature low barriers to entry to help entice entrants with lower capacity or those who may otherwise be uninterested in the burden of traditional application and reporting processes associated with many government grant and contribution programs.

There is evidence that the design and delivery of all focus areas considered and worked to attract new talent and address barriers to participation of new and/or smaller players, including in some cases a specific focus on underrepresented groups (e.g., women and Indigenous people). NRCan engaged with collaborators to facilitate this outcome, many of whom were also new to collaboration with NRCan or have taken on new or expanded roles in supporting the innovation ecosystem as a result of the ICI-CTS.

Beyond involving a range of stakeholders in program design, innovations in the application process were perceived by program representatives and finalists as facilitating the participation of new talent. Many smaller organizations and underrepresented groups can experience barriers to application writing, such as a lack of experience or human resource capacity given the time burden required to develop detailed proposals. Indigenous communities may face additional barriers for applications such as internet bandwidth (particularly in remote areas), language barriers, and competing priorities during culturally important periods of the year (hunting, fishing, and celebrations). Timing of application may also be important for some female entrepreneurs, where intake periods can compete with priorities such as school holidays. Overall, there was clear design consideration for how application requirements could be streamlined to reduce burden for applicants, with the application process tailored to the specific design and target audience of each focus area. All applicants had access to information (e.g., process maps and applicant guides) and support as they worked on their applications to ensure project readiness. Selection processes for many focus areas also included features such as applicant interviews and pitch events where semi-finalists or a short-list of applicants had the opportunity to directly explain their proposal to the jury or selection panel.

The majority of proponents reported a positive impression of the application process, but satisfaction of finalists and non-finalists differed for some aspects of the process (Table 7).

Table 7: Comparison of Finalists’ and Non-Finalists’ Perspective on the Innovative Application Process

Both finalists and non-finalists noted that the cost to putting together an application makes it important to ensure that the application process is scaled to reflect (1) the value of each challenge’s financial supports and (2) the likelihood of being selected as a semi-finalist or finalist. While application was generally recognized to be less burdensome than other funding programs, some respondents indicated that the level of effort required to complete an application under the ICI-CTS (including pitch presentation where applicable) may have still posed a barrier for applicants who are unable to commit significant time or funding to the application process. This may be a particular barrier for early stage applicants, i.e., those with less experience or a lower level of base funding. Three focus areas did provide funds to help capacity stretched organizations (SMEs, small-scale innovators and Indigenous communities) prepare their proposal. Respondents suggested that this sort of support could have been extended to more challenges or participants, or be made available earlier in the application process, but this may not always be appropriate given types of solvers or feasible given timelines. Some also suggested that the ICI-CTS could have mitigated barriers by providing more information on what to expect in the application process, e.g., improved application templates (including examples of information required).

Respondents for the IODI (participants and non-participants) did not perceive any significant barriers to the application process, other than in some cases the availability of their own time. Interviewees identified a capacity issue with not having enough qualified people in their communities to focus or advise on completing applications, which means that they often have to rely on external consultants to complete funding applications. They perceived that these difficulties were largely addressed by the simplicity of the application process and support from program staff for the IODI.

In the ICI-CTS’ notional performance framework (2017), NRCan expected that increased awareness and participation in clean tech would be demonstrated by a achieving a higher number of applicants per dollar of NRCan funding and a higher proportion of “unconventional” applicants than for comparable funding calls under core programming in the same technology areas. Unlike increased collaboration and increased investment, the program did not expect to achieve its targets for the mobilization of new talent until 2028. Our engagement found early evidence that new talent has been mobilized by the ICI-CTS but related results are currently difficult to quantify.

NRCan has defined “unconventional applicants” as those who do not typically apply for NRCan clean tech programs. The vast majority of finalists interviewed were able to compare the application against their experience with other federal programming, which indicates that these finalists are not new to federal funding programs. Similarly, only 16.7% of surveys respondents applying on challenges indicated that they had not previously applied for funding from the Government of Canada. However, that they applied does not mean that they received funding. Previous funding experience may have also been for a different solution or in a different sector.

More importantly, we found the ICI-CTS’ definition of new talent may be too narrow. There are many other ways that “new talent” and “non-traditional proponents” can be defined. Interviews and survey data provide some proxy indicators on the extent to which new talent has been mobilized to date in other ways:

• New Field or Sector. In total, 20.4% of survey respondents indicated that they had applied to participate in a focus area that represented a new field or sector for them or their organization. All of these responses were related to CI! and WICT. By contrast, the majority of respondents for STL and PF indicated that their organization was already working on a solution to the issue addressed by the challenge at the time of application. These results are similar to responses provided by finalists during interviews. However, while some interviewees for STL had previous experience working with biofuels, they clarified that the challenge has pushed them to consider more seriously SAF as part of their business. Similarly, Power Forward had a focus on systems integration, meaning that by definition the solutions would already need to be in progress.
• New / Smaller Players. A large number of finalist organizations were SMEs or start-up businesses. Statistics Canada’s Survey on Financing and Growth of Small and Medium Enterprises (2017) offers the first-ever snapshot of the financing and growth activities of clean tech small and medium-sized enterprises (SMEs) in Canada.¹³ Results reveal that the profile of SMEs operating in the clean tech sector were larger and older than all SMEs, were more likely to seek financing, and their financing requests were generally approved. This may also explain why so many applicants and finalists were not new to federal funding programs, and supports the idea that this narrow definition of new talent may not be the best indicator.
• Underrepresented Groups. Two focus areas were specifically designed to support outcomes for underrepresented groups; the processes and capacity support built into WICT and IODI helped attract and maximize the participation of women and Indigenous communities. Evidence indicates that they have both been successful in mobilizing new talent. Efforts to spotlight these participants’ clean tech and clean energy projects are designed to further inspire participation within these target groups.
  More generally, data from Statistics Canada’s survey indicates that ownership of clean tech SMEs tended to be less diverse than that of all SMEs. In 2017, this survey found that clean tech SMEs tended to be majority owned by men (81%), compared with 64% of all SMEs. Clean tech SMEs were also less likely to be majority owned by visible minorities, Indigenous peoples and people with a disability. Similarly, 18% of clean tech SMEs had primary decision makers who were born outside of Canada, compared with 25% of all SMEs. The flexible and inclusive design of challenges under the ICI-CTS provide a greater opportunity to address gaps from the broader lens of reconciliation, equity, diversity and inclusion (REDI). However, as noted above the program has not systematically collected data that would enable analysis of GBA+ consideration. Related data collected by our engagement is also limited; it provides a profile for survey and interview respondents but not the organization to which these individuals belong.

Enhanced Public Awareness

Enhanced awareness among communities of potential solvers is a critical early success factor for challenges, important to both attracting participants and influencing market demand. Over time, challenges can also have the effect of communicating government priorities and better engaging the general public on the broader policy issue(s) at hand. This can help generate greater public demand or social licence for the challenge host to more effectively deliver on its core mandate in the future.

Since 2017, the program’s social media strategy has focused on the promotion of specific milestones in the challenges (e.g., challenge launch, announcing finalists, challenge deadlines, etc.) and the ad hoc amplification of challenge finalists’ content. Finalists have been invited to proactively share links to online content showcasing relevant activities (e.g., project progress) that NRCan could echo on the PCO website and its own social media platforms (Twitter, Facebook, and LinkedIn). Where applicable, videos included in posts were also submitted separately as deliverables required by the finalists’ contribution agreements (mid-project videos to report on progress). In 2020, NRCan developed a further social media strategy aimed at ramping up the promotion of challenges and finalists as they move towards the selection of winners.

In the ICI-CTS’ notional performance framework (2017), NRCan planned to measure increased public awareness of clean tech issues among Canadians by the year-over-year progressive increase in social media by channel. Program managers indicated that while this indicator may make sense at the PCO level, this is not a realistic expectation for the ICI-CTS. This program’s focus areas will have natural increases and decreases in attention related to major announcements.

There was no data provided to the engagement that would indicates a year-over-year trend in public awareness. However, there is some evidence of data being collected. In 2020, NRCan’s CPS produced a social media analytics report for the program, which indicates the reach of content across focus areas and social media platforms. This analysis shows that posts related the ICI-CTS performed relatively well in terms of both reach and engagement rate, particularly when compared to the average for government content (Table 8). The results specific to each focus area differ, with the highest engagement rates for WICT and STL and the lowest engagement rates for Crush It! (perhaps due to the very technical nature of solutions). However, the results provided in the report do not cover communications beyond NRCan’s accounts, which may have been extensive. While data is not available across all focus areas, data reported by MaRS indicates significant reach for some communications (e.g., 37,000 social media impressions related to posts for PF and 670,000 related to WICT). One finalist for Power Forward also reported 7,000+ views on their recent video post on Facebook.

Table 8: Cumulative Reach of NRCan Social Media Communications, May 2018-April 2021 (All Challenges)

Source: Impact Canada Social Media Analytics Report (NRCan CPS)

There is also anecdotal evidence of traditional media coverage and interest in NRCan’s challenges and finalists’ projects but a detailed media analysis was not completed by the engagement.

Unexpected Outcomes

Our engagement also sought to identify whether any additional or unexpected outcomes (positive or negative) have resulted from the ICI-CTS. For the purposes of our engagement, this would include any results other than those targeted in the performance framework. Our analysis in this area is somewhat limited as it is challenging to identify unexpected outcomes when the ICI-CTS’ performance measurement framework is still in development, meaning that all expected outcomes have not been clearly identified and documented.

Based on evidence available, the engagement did not identify any negative unexpected results from the ICI-CTS. However, we did note a few positive achievements. For example:

• Across all focus areas, semi-finalists and finalists indicated that their projects were generally advancing faster than they expected when they decided to apply. While not entirely unexpected, several non-finalists also indicated that the challenge had motivated them to better organize or advance their proposed solution; two of these individuals indicated that they had since received funding support through programs in other jurisdictions.
• For WICT, the “cohort effect” was highlighted across interviews as a positive unexpected outcome. The cohort model of the WICT challenge has helped the six finalists develop a mutually supportive peer-to-peer network, despite being engaged in a prize competition. This unexpected outcome was perceived to have accelerated the pace at which the finalists’ projects are progressing.
• Particularly for the IODI and WICT, interviewees identified personal benefits of participation, e.g., that participation has helped boost their confidence, leadership abilities, and provided new skills that they can now expand to opportunities in other areas.
• Lab representatives indicated having benefited from the opportunity to learn about the current state of research across a range of clean tech projects and where they are best placed to support further development.

Our engagement was not designed to assess progress towards intermediate and ultimate outcomes, as it is too soon to indicate whether these will be achieved. Nevertheless, we found progress at least insofar as finalists are progressing towards outcomes specific to their project; projects are achieving expected project milestones. The key exception is STL’s Cross-Canada Flight Challenge, where the challenge has closed without the pure prize being awarded as no producer met all the criteria before the submission deadline (January 1, 2021). However, this should be viewed as a lesson learned rather than a failure on the part of the program (see Box 5). Across all focus areas, finalists indicated that Impact Canada had contributed to the advancement of their project. The majority indicated that they would not have been able to move their solution forward without this support. While about 20% of finalists interviewed for any given focus area indicated that they would still have advanced their projects in the absence of the ICI-CTS, they nevertheless believed that the pace of advancement would have been slower without this support.

Box 5: Lessons Learned from the STL Cross-Canada Flight Challenge

Marking a key difference from traditional programs, the results logic for the ICI-CTS suggests that progress towards outcomes should also be observed in solvers who are not selected as finalists, albeit to a more limited extent. The nature of a challenge is such that the majority will not win the grand prize, however the challenge process might generate new opportunities for the solvers that would not have otherwise transpired. Non-winners may develop fruitful partnerships, promising prototypes, new business processes, or attract other investments in their activities that all go on to improve socioeconomic or environmental outcomes in their own ways. This is all part of the impact of a well-designed challenge. In response to the engagement’s survey, the majority of semi-finalists and unsuccessful applicants indicated that they are continuing to advance their project (60.5%) or different opportunities in a related area (19.8%). Semi-finalists in this group indicated that they continue to advance at or beyond the pace they expected to achieve when they decided to apply. The extent to which these results can be attributed to Impact Canada is unclear given available evidence.

What We Found – Factors Influencing Program Delivery

We examined the extent to which the ICI-CTS appropriately considers the internal and external factors that could impact its ability to effectively or efficiently achieve its intended outputs and outcomes. We found that the program has identified and developed approaches to respond to internal and external factors that impact (or are expected to impact) its delivery. These factors, together with additional factors identified by our engagement, are outlined in Table 9. Some of these factors, such as increased flexibility in decision-making, had a positive influence on implementation. The use of a program stream (versus individual challenges) enabled NRCan to build internal capacity, including a team of dedicated human resources, and provided more flexibility to adapt along the way. Other factors, such as COVID-19, had a more negative influence. We found that within its sphere of influence, the program has been responsive to mitigate or address barriers to delivery.

Table 9: Factors Influencing Program Delivery

What We Found – Lessons Learned

What are the conditions required to accelerate the use of the Impact Canada – Clean Technology Stream’s innovative approaches to future interventions? Our engagement sought to identify best practices and lessons learned through the design and delivery of the program’s prize challenges and experimental design features that could support the positive evolution of government systems to enable greater innovation. There are various lessons learned highlighted throughout this report. This section focuses on those lessons that are either overarching or not covered in previous sections.

Perspectives on Challenge Model

Interviewees generally perceived the challenge model as effective and appropriate given the objectives of the specific challenges that were launched under the ICI-CTS, with the exception of the IODI (where design ultimately recognized that a prize competition was inappropriate). Besides the flexibility, visibility and capacity boost provided by financial and non-financial supports, finalists expressed strong appreciation for the accelerated timelines for decision-making that were a feature of challenge delivery. However, there was no consensus on whether this model should be replicated in the future. It is a new tool that provides an alternative to traditional funding programs, but decisions on when it is appropriate to use this model should be carefully considered in light of the context of the outcomes to be achieved. PCO’s Impact Canada Challenge Guide (2021) highlights that a challenge may not be the optimal instrument choice if:

• The solution to the problem is evident, currently in the market, and/or the best choice for the solution provider is clear;
• The primary goal is to provide core funding or ongoing support to build capacity to meet program objectives over a longer term;
• The ultimate objective is to procure a solution exclusively for the Government of Canada’s own use;
• The subject matter is particularly sensitive for stakeholders and an element of competition would detract from solving the problem; or
• There are regulatory or legislative barriers to the uptake of solutions.

While the above are common examples, they are not hard rules. For example, Innovation, Science and Economic Development Canada (ISED)’s Innovative Solutions Canada is a competitive, challenge-based research and development program designed to solve internal departmental operational issues and/or to fill a gap in the marketplace, with federal departments as the intended first (but not necessarily only) customers.

It is also important to understand the needs and interests of potential solvers. For example, WICT finalists gave more importance to business incubation support than to the competitive aspects of the challenge model. Respondents across challenges also noted that emerging clean tech is an inherently risky field, and that the use of milestone payments may be best suited to where there is a relatively high degree of certainty on how things will roll out. Otherwise, flexibility is needed to ensure that emerging developments (including technological developments or changes in international standards) can be considered in (re)negotiation of the milestone schedule.

Challenge Guidance

While design of the ICI-CTS’ focus areas captured lessons learned from benchmark challenges at the formative stage, at the outset of the program there was limited guidance from the Government of Canada to draw from to inform design and implementation. As a result, NRCan had to learn by doing, which created some difficulties and delays. However, NRCan now has a number of lessons learned that could help accelerate future use of related models by NRCan (and other government departments) and bring new perspectives to corporate practices. NRCan’s lessons learned have already fed into recent guidance released by PCO as a resource to guide future programming. For example, recognition of Indigenous innovation initiatives in PCO’s Impact Canada Challenge Guide (2021) emerged as a result of NRCan and other federal departments’ engagement with Indigenous peoples during delivery of the ICI.

Specific lessons learned (mostly compiled by NRCan program representatives and supplemented by evaluation) include:

• Ensuring Early Engagement on Co-Design. Co-creation is a foundational element to the challenge model but two focus areas (support for SAF and to move Indigenous communities off-diesel) were committed to in funding authorities before the interventions were designed. As noted above, engagement with stakeholders in the co-design phase led to the prize challenge model being deemed inappropriate for the IODI. If possible, future interventions should undertake engagement and/or consultation in advance of committing to a focus area.
• Working with the Fiscal Framework. The Government of Canada’s fiscal framework is not built for nimble programming. As shown in Figure 5, the program’s notional funding allocation by fiscal year was determined in funding authorities at the outset of the program, but the actual funding profile was designed with subsequent stakeholder engagement in the co-design phase. Further amendments were made in response to COVID-19. While there was relatively limited deficit or lapse of funds in any given fiscal year, realignment of resources across fiscal years required extensive use of the Annual Reference Level Update (ARLU) reprofile mechanism, which was reported by program managers to be cumbersome and resource-intensive. If possible, including a carry-forward provision in funding authorities could simplify use of flexibilities in allocating funds across fiscal years.

Figure 5: Evolution of Expenditure Profile of ICI-CTS (Before and After Co-Design)

• Managing Uncertainty. NRCan experienced difficulties in managing uncertainty related to the ICI-CTS’ innovative funding agreements. This includes uncertainty as to whether the pure prize grant for STL would be issued, with the potential for a resulting $1M in lapsed funds at end of program. This also includes the difficulties in managing milestone-based contribution agreements where payments are released once results are achieved. While the program must earmark these funds in its budget as per the notional timelines in the agreement, this creates a risk of lapsed funds if milestones are not achieved or are delayed (the latter resulting in a need to find funds in the next fiscal year). Again, if possible, including a carry-forward provision in funding authorities could simplify use of flexibilities in allocating funds across fiscal years. The accelerated timelines of a challenge model also provide less flexibility to deal with unexpected events (e.g., COVID-19, replacement of participants). To the extent possible, it is important to have contingency plans to deal with the unexpected.
• Overcoming Cultural Resistance. While encouraged by the principles of the Impact Canada Initiative to implement innovative practices, the program was often met with resistance when proposing to do things differently because it did not align with conventional practices. This frame of thinking is common across the government, and made it difficult for the program to reach a consensus with corporate colleagues. A key lesson learned is that collaboration with corporate services is crucial when implementing a new, innovative program. Corporate services should be engaged early, as part of the program team, to enable them to have the time to really understand the program and its objectives. Senior management involvement is also required help manage expectations for the program’s implementation, while ensuring that senior managers are also aware of the limitations imposed by the broader policy framework. However, innovative new mechanisms - e.g., flexible milestone agreements, use of contribution agreements (vs. procurement) to develop collaborative arrangements, etc. – have now been tested and demonstrated to facilitate solutions and build connections in the innovation ecosystem in ways that advance the objectives of the challenges. Experimental learnings from the ICI-CTS should be used to impart new perspectives, modernize methods, and maximize efficiency in existing corporate processes.
• Understanding Limits to Co-creation. There are policies and directives in existing frameworks (TB Policy and Directives) that limit the extent to which federal programs can collaborate with external entities, including limitations in how federal labs can collaborate with proponents. For example, the Government of Canada cannot directly benefit or be seen to benefit from the outputs of a contribution agreement or carry out any work that would suggest an employee/employer relationship. This means that NRCan cannot work hand-in-hand with co-creation organizations in the same way as may be done in external challenge models; NRCan cannot receive direct advice from collaborators and cannot direct collaborators in their activities. However, program representatives for the ICI-CTS have found that with adjustments (e.g., by using the term “collaborator” instead of “partner”, attending engagement sessions hosted by collaborators that focus on broader context), the program can achieve results similar to those intended in co-creation. While collaborators should always be carefully selected through research and engagement to determine who is best positioned to support results and benefits to Canadians, small collaborators require funding since they do not have the capacity to cover their resources but may also want ownership. Program representatives perceive the ideal collaborator in this context as one who does not need funds from the program to support the initiative (i.e., they see enough value in it to use their own resources).
• Staff Skillsets. Collectively, the ICI-CTS program team was perceived to have the right mix of skillsets (e.g., project management, content expertise, etc.). However, implementation has required OERD to rely on internal resources beyond staff dedicated to the ICI-CTS; the level of resources necessary to deliver a new program model should not be underestimated. Consideration should also be given to ensuring that specialized skillsets are available to dedicated to aspects of project management that pose greater complexities than in a traditional Gs&Cs program, such as delivery on innovative communications objectives, and performance measurement and impact assessment.

Building on Progress

For most focus areas under the ICI-CTS, one solution will win the grand prize. However, the important issues targeted by each challenge and initiative (Table 3) cannot be fully addressed by the challenge, nor are they intended to be. There is no one, “silver bullet” solution to these complex problems. The solutions stimulated by these challenges may not be sustainable without further intervention. Interviewees indicated ongoing programming is necessary to maintain momentum, promote and/or accelerate adoption of solutions, both within and outside the Government of Canada. Suggestions varied by challenge but finalists and collaborators did not necessarily think that further funding was the answer – or at least not the only answer. Additional proposals included support for market development, policy and regulatory changes. Within OERD, it is understood that addressing barriers beyond technology is often required. These barriers are identified in OERD by applying its Innovation System Analysis Tool and associated methodology. We found that NRCan has learned a lot about the ecosystems through delivery of the ICI-CTS – how they work, where there are gaps, who the key players are – that can now be used to build on success and make further progress in addressing these areas as government priorities.

Replicating Success

NRCan has also learned a lot from its other design experiments applied in each focus area (the “experiments within an experiment”). The challenge model may not be the only design element that NRCan would want to replicate. For example, the co-design process added significant value to target key outcomes, including consideration of specific barriers for the problem area. Federal research centres may have a larger role to play in supporting RD&D. Programs and supports for new talent are important to increase participation of new solvers (e.g., SMEs) and can remove barriers for underrepresented groups. The testimonials from finalists illustrate the importance of targeted opportunities like WICT and the IODI, specifically for women and Indigenous peoples. Other elements we found to be best practices or real strengths in the ICI-CTS are highlighted throughout this report.

Conclusion

We found that the ICI-CTS is relevant. The program aligns with federal government and NRCan priorities, roles and responsibilities to support innovations in clean technology, supports mandate commitments to increase experimentation, and is structured to give more flexibility to proponents to propose innovative solutions.

Overall, we found that governance structures, processes and mechanisms for ICI-CTS were defined, established and operating effectively, including internal reporting mechanisms. Processes and controls are also in place to support compliance with relevant departmental guidance and the TB Policy on Transfer Payments, and are operating effectively most of the time.

The ICI-CTS provided an opportunity to test innovative ways to address problems for which solutions are not apparent, and generate evidence of which efforts work best to create greater public value in areas of high priority. Many of the ICI-CTS’ focus areas also incorporated numerous experiments within this broader experiment, for example, testing new models of Indigenous participation or new ways of working with international partners.

We found that the design of focus areas under the ICI-CTS was consistent with principles of Impact Canada and best practices in challenge models. Stakeholder engagement in co-design was recognized as a key strength for the program. This allowed NRCan to target key issues while scaling the challenge to what would be achievable by solvers, including consideration of specific barriers for the problem area. The program was responsive to this feedback and, while there was some room for improvement, the resulting funding flexibilities and non-financial proponent supports built into the design of each focus area facilitated participation from diverse solvers. While communications could have been more innovative and better resourced, the program attracted a sufficient number of quality proposals.

We examined the extent to which the ICI-CTS is making progress towards program-level immediate outcomes identified in its draft logic model, including increased collaboration and investment by stakeholders in the clean tech ecosystem, increased awareness and participation in clean tech (i.e., mobilization of “new talent”), and increased public awareness of clean tech issues among Canadians. While available evidence suggests that progress has been made in each area, the ICI-CTS uses innovative approaches that necessitate new methods and strategies for measuring program outcomes and impacts. A key difference in the design of challenge-based models is that if the result is not achieved, the funding is not awarded. We found progress at least insofar as finalists are progressing towards outcomes specific to their project; projects are achieving expected project milestones. Regardless, difficulties in the design and application of performance metrics still need to be resolved. There is a need to generate better evidence that the perceived risks of running a challenge, and the level of investment that is required to run them, is justified by the achievement of better outcomes.

The issues targeted by each focus area will not be fully addressed by the ICI-CTS, nor are they intended to be. There is no one, “silver bullet” solution to these complex problems. While there is evidence of viable pathways to scale success for each focus area, this requires dedicated efforts that extend beyond the scope and timelines of the ICI-CTS. However, the program is intended to help advance and accelerate solutions, both as a direct impact of the solutions proposed in each focus area and by applying the intelligence gained during implementation to inform other programs and actors, which may in turn also accelerate the design or implementation of parallel initiatives in related focus areas.

Overall, we found the challenge model to be a new tool that provides an alternative to traditional funding programs, but decisions on when it is appropriate to use this model should be carefully considered in light of the context of the outcomes to be achieved. We expect NRCan to continue to apply lessons learned from the ICI-CTS in its own design of future prize challenges and other initiatives, whether under the umbrella of Impact Canada or as part of a separate initiative. Future use of design elements that were found to be strengths of the ICI-CTS (e.g., co-design) need not be limited to challenge models.

Appendix A: Engagement Sub-Objectives and Criteria

The objectives of this engagement are:

• To assess the extent to which the program’s design and delivery facilitates the effective and efficient achievement of immediate results and complies with relevant authorities.
• To identify and document lessons learned from the experimental approaches under the Impact Canada – Clean Technology Stream that can be applied to future interventions.

The following table identifies the questions addressed by the engagement, as well as the sub-objectives and criteria used to conduct the engagement:

Appendix B: Joint Engagement Team

Chief Audit and Evaluation Executive

Michel Gould

Evaluation Team

David Ash – Senior Advisor, Evaluation
Stephanie Kalt – Evaluation Manager
Christiane Ngo Manguelle, PhD – Senior Evaluator
Edmund Wolfe – Senior Evaluator
Harpreet Sahota – PARDP
Aishwarya Babu - PARDP

Audit Team

Linda Jones – Senior Director, Audit Operations
Katherine Venus – Audit Manager
Raj Brar – Senior Auditor
Nour Al Youssef - Auditor

Footnotes