Language selection

Search


Joint Audit and Evaluation of the Impact Canada Initiative – Clean Technology Stream

Presented to the Departmental Audit Committee (DAC) January 21, 2022
Presented to the Performance Measurement, Evaluation and Experimentation Committee
(PMEEC) November 24, 2021

List of Acronyms

ADM
Assistant Deputy Minister
AEB
Audit and Evaluation Branch
ARLU
Annual Reference Level Update
BEIS
UK Department for Business, Energy and Industrial Strategy
CEMI
Centre for Mining Innovation
CI!
Crush It! Challenge
CoE
NRCan’s Centre of Expertise on Grants and Contributions
CPS
Communications and Portfolio Sector
CTF
Charging the Future
CTS
Clean Technology Stream
ETS
Energy Technology Sector
FAA
Financial Administration Act
Gs&Cs
Grants and Contributions
GARDN
Green Aviation Research & Development Network
GHG
Greenhouse Gas
ICE
Indigenous Clean Energy Social Enterprise
ICI
Impact Canada Initiative
ICI-CTS
Impact Canada Initiative - Clean Technology Stream
IIA Standards
International Standards for the Professional Practice of Internal Auditing
IODI
Indigenous Off-Diesel Initiative
LMS
Lands and Minerals Sector
MaRS
MaRS Discovery District
NRCan
Natural Resources Canada
OERD
Office of Energy Research and Development
PCF
Pan-Canadian Framework (on Clean Growth and Climate Change)
PCO
Privy Council Office
PCO COE
Privy Council Office Centre of Expertise for the Impact Canada Initiative
PF
Power Forward
RD&D
Research, Development and Demonstration
REDI
Reconciliation, Equity, Diversity and Inclusion
SMEs
Small and Medium Sized Enterprises
SAF
Sustainable Aviation Fuel
STAC
Science and Technology Assistance for Cleantech
STL
Sky’s the Limit
TB
Treasury Board (of Canada)
TRL
Technology Readiness Level
Ts&Cs
Terms and Conditions
WICT
Women in Clean Tech

Table of Contents

Executive Summary

About the Engagement

This report presents the findings, conclusions, and recommendations from the joint audit and evaluation of the Impact Canada Initiative (ICI) – Clean Technology Stream (CTS). Announced in Budget 2017, the ICI is a whole-of-government effort, led by the Privy Council Office (PCO), designed to help departments accelerate the adoption of outcomes-based approaches to deliver results to Canadians. The ICI promotes the flexible use of grants and contributions through a range of innovative funding models, including prize challenges. Natural Resources Canada (NRCan)’s Clean Technology Stream, one of the first program streams developed under this new initiative, has launched six selected focus areas – i.e., the Crush It! Challenge, Charging the Future Challenge, Power Forward Challenge, Sky’s the Limit Challenge, Women in Clean Tech Challenge, and Indigenous Off-Diesel Initiative.

The joint engagement examines the design and delivery of the ICI-CTS from its announcement in Budget 2017 to March 2021, with updates reflecting the impact of COVID-19 (extending the program to 2021-22 and offering additional funding supports). The objectives of the joint engagement are:

  • To assess the extent to which the program’s design and delivery facilitates the effective and efficient achievement of immediate results and complies with relevant authorities.
  • To identify and document lessons learned from the experimental approaches under the Impact Canada – Clean Technology Stream that can be applied to future interventions.

NRCan’s Audit and Evaluation Branch (AEB) conducted this engagement in accordance with the Treasury Board (TB) Policy on Results (2016) and the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing and the Government of Canada’s Policy on Internal Audit.

What the Engagement Found

Relevance

We found that the ICI-CTS is relevant. The program aligns with federal government and NRCan priorities, roles and responsibilities to support innovations in clean technology related to Canada’s natural resource sectors that tackle Canada’s most pressing environmental challenges and create economic benefits. Each of the ICI-CTS’ six selected focus areas meet key criteria that were set out for their prioritization. The narrow focus of focus area sets this apart from other funding programs, with the suite of clean tech challenges acting as vehicles to enhance the communication of priorities in the Government of Canada’s environmental agenda.

More broadly, the ICI-CTS supports mandate commitments to increase experimentation, and is structured to give more flexibility to proponents to propose innovative solutions. It enables the Government of Canada to test innovations and generate evidence of which efforts work best to create greater public value in areas of high priority. The ICI-CTS provided an opportunity to test innovative ways to address problems for which solutions are not apparent.

Governance

Overall, we found that governance structures, processes and mechanisms for ICI-CTS were defined, established and operating effectively. Roles, responsibilities and accountabilities were clear, including delineation between the responsibilities of PCO and NRCan, and any areas of complementary and overarching responsibilities. However, the unfamiliar policy context resulted in difficulties in the initial stages of program design and development, with fundamentally different interpretations of the ICI-CTS Terms and Conditions (Ts&Cs) among key intradepartmental players. These differences have since been addressed with an ongoing collaborative approach.

Internal reporting mechanisms were also found to be established and functioning as intended at all levels of program management. These reporting mechanisms (both formal and informal) enabled key communication on the program’s progression to senior management at intervals that supported timely reporting across all phases of design and implementation.

Design

Challenges have a set of features meant to attract new talent and new ideas, and accelerate progress towards solving problems that matter to people. As a result, challenges are designed, promoted and delivered differently than the types of funding programs that governments and their stakeholders are accustomed to running and participating in. Many focus areas also incorporated numerous experiments within this broader experiment, for example, testing new models of Indigenous participation or new ways of working with international partners.

We examined how and for whom the program’s experimental tools are effective in facilitating the achievement of intended outputs and outcomes, with a focus on five elements common to all focus areas, i.e.: inclusive innovation (co-design), communications and outreach, collaborative arrangements, flexible funding mechanisms, and additional (non-financial) proponent supports. These elements are critical to the success of challenges. Overall, we found that the design of focus areas under the ICI-CTS was consistent with principles of Impact Canada and best practices in challenge models.

Stakeholder engagement in co-design was recognized as a key strength for the program. This allowed NRCan to target key issues while scaling the challenge to what would be achievable by solvers, including consideration of specific barriers for the problem area. The program was responsive to this feedback and, while there was some room for improvement, the resulting funding flexibilities and non-financial proponent supports built into the design of each focus area facilitated participation from diverse solvers (project proponents and potential collaborators or investors).

To attract the interest of solvers there was also effort dedicated to the development and delivery of communication strategies. While this could have been more innovative and better resourced, the program did attract a sufficient number of quality proposals.

NRCan also made good use of collaborators to co-develop and/or deliver activities under the ICI-CTS. There is evidence that the collaborators selected were both appropriate and effective in facilitating and/or accelerating delivery of the program. In particular, working with Indigenous collaborators on the Indigenous Off-Diesel Initiative (one of the six focus areas) was perceived as a positive move towards a new model of collaborative (rather than Crown-led) program delivery.

Processes and Controls

Overall, we found that processes and controls are in place to support compliance with relevant departmental guidance and the TB Policy on Transfer Payments. We also found that key financial operational controls (refer to Appendix A) have been designed and implemented and are operating effectively most of the time.

There is an opportunity to improve the retention of documentation related to key decision processes, particularly for focus areas that used a collaborator as an intermediary. Further, while all files sampled were noted to be signed by the appropriate individual with a valid financial signing authority and adhered to the required stacking provisions, the release of proactive disclosure information was not always performed within the required reporting period.

The program’s identification of auditable agreements was found to align with NRCan policies and procedures on the recipient-audit process, with consideration to the inherent risk ratings of the agreements. However, while the program uses corporate systems to capture audit related information (i.e., status, findings and recommendations), its process for tracking, managing and resolving recipient audit recommendations issued is not clear.

Expected and Unexpected Outcomes

We examined the extent to which the ICI-CTS is making progress towards program-level immediate outcomes identified in its draft logic model, including increased collaboration and investment by stakeholders in the clean tech ecosystem, increased awareness and participation in clean tech (i.e., mobilization of “new talent”), and increased public awareness of clean tech issues among Canadians. There are a number of internal and external factors influencing program delivery. Within its sphere of influence, the program has been responsive to mitigate or address barriers to delivery.

Impact Canada uses innovative approaches that necessitate new methods and strategies for measuring program outcomes and impacts. While a notional performance framework was included with program approvals, NRCan is still working on its own and with PCO to enhance this framework and to develop challenge-specific outcomes and related methodologies. Difficulties in the design and application of performance metrics still need to be resolved. While there is evidence that the design and delivery of each focus area considered and worked to achieve the program’s immediate outcomes, complications in performance measurement mean that calculation of results against ICI-CTS indicators is not possible given data available.

Nevertheless, available evidence suggests that progress has been made in each area. There is strong evidence to suggest that finalists are developing their own formal and informal collaborative arrangements among diverse stakeholders. Most finalists’ projects have also leveraged investments, at least to some extent, and further leveraging across all focus areas could reasonably be expected as projects continue to advance. While the vast majority of finalists are not new to federal funding programs, a large number were SMEs or start-up businesses. Two focus areas were also specifically designed to support outcomes for underrepresented groups, i.e., to attract and maximize the participation of women and Indigenous communities. Evidence indicates that the program has been successful in mobilizing new talent in these areas. Evidence related to enhanced public awareness indicates that the ICI-CTS performed relatively well on social media compared to most government content. Communications beyond NRCan’s accounts may also have been extensive. However, finalists noted that there was less opportunity for or emphasis on communications promoting their progress than they expected when they decided to apply (in part due to restrictions imposed by COVID-19).

Our engagement was not designed to assess progress towards intermediate and ultimate outcomes, as it is too soon to indicate whether these will be achieved. However, a key difference in the design of challenge-based models is that if the result is not achieved, the funding is not awarded. We found progress at least insofar as finalists are progressing towards outcomes specific to their project; projects are achieving expected project milestones. Progress towards outcomes was also reported by solvers who are not selected as finalists, with many continuing to advance their solutions, albeit at a slower pace or to a more limited extent. The extent to which this can be attributed to Impact Canada is unclear.

The issues targeted by each focus area will not be fully addressed by the ICI-CTS, nor are they intended to be. There is no one, “silver bullet” solution to these complex problems. While there is evidence of viable pathways to scale success for each focus area, this requires dedicated efforts that extend beyond the scope and timelines of the ICI-CTS but which may be accelerated as a result of the program. However, the ICI-CTS is meant to help advance and accelerate solutions. The challenge model is also such that there is no guarantee that anyone will win the final prize. However, even for the one area of the ICI-CTS where the challenge closed without the prize being awarded (i.e., no producer met all the criteria before the submission deadline), there were experimental learnings gained; this should not be viewed as a failure on the part of the program. Moving forward, this intelligence is available to inform perspectives for other programs and actors and may also accelerate the design or implementation of parallel initiatives in related focus areas.

Lessons Learned

Overall, interviewees perceived the challenge model as effective and appropriate but there was no consensus on whether this model should be replicated in the future. We found the challenge model to be a new tool that provides an alternative to traditional funding programs, but decisions on when it is appropriate to use this model should be carefully considered in light of the context of the outcomes to be achieved.  

NRCan has also gained a lot of experimental learning by testing additional innovations in each focus area. Best practices and lessons learned are highlighted where relevant throughout this report. Future use of design elements that were found to be strengths of the ICI-CTS (e.g., co-design, supports to remove barriers for underrepresented groups, etc.) need not be limited to challenge models.

Lessons learned by NRCan related to systemic and departmental barriers to the use of flexible mechanisms have already fed into guidance released by PCO, and should help accelerate future use of related models by NRCan and impart new perspectives to corporate practices. We expect NRCan to also apply lessons learned from the ICI-CTS in its own design of future prize challenges, whether under the umbrella of Impact Canada or as part of a separate initiative. In future, NRCan’s AEB will look for evidence that these lessons learned have been considered at least in the proposed design of relevant programs and activities. Key lessons learned include:

Design
  • Engage on co-design of the prize challenge as early as possible in the design phase. If possible, future interventions should undertake engagement and/or consultation in advance of committing to a focus area.
  • Allow for the maximum possible flexibility in the fiscal funding arrangements, including contingency plans to deal with the unexpected. If possible, include a carry-forward provision in funding authorities to simplify use of flexibilities in allocating funds across fiscal years.
  • Engage corporate services early to enable them to fully understand the program and its objectives.
  • Ensure that senior management is closely engaged help overcome resistance to innovative programming, and that expectations for the program’s implementation are clear and aligned to the intended design and delivery of the initiatives proposed.
  • Recognizing the limits to co-creation with external entities, carefully select collaborators based on who is best positioned to support results and benefits to Canadians and ensure that their role is clearly defined, understood, and does not introduce a real or perceived conflict of interest.
Delivery
  • Ensure that dedicated resources are available (within the program team and/or corporate services) to fully deliver on innovative communications objectives as a core element in challenge design.
  • Develop clear selection criteria and clearly communicate to applicants how they will be assessed against these pre-determined criteria, and ensure that documentation of decisions related to selection of applicant proposals is complete. 
  • Ensure that in cases where intermediaries are used, required documentation identified per the funding agreements is collected and retained to fulfil NRCan’s oversight role.
  • Ensure that approvals of proactive disclosures of grants and contributions information are completed in a timely manner.
Impact
  • Dedicate sufficient resources to performance measurement, to assess the effectiveness (impact) and efficiency of the prize challenge.
  • As early as possible in challenge design or implementation, consider ways to effectively contribute to concrete follow-on strategies towards the further advancement of government priorities.

Recommendations and Management Response

Recommendation Management Response
  • The ADM ETS should review and document lessons learned from the Impact Canada – Clean Technology Stream, including those that may emerge in final prize award and any post-program follow-on, and make these available within NRCan, including with corporate services, and across government such that they can inform the design and delivery of future prize challenges.
Management Agrees. The Office of Energy Research and Development (OERD) will further formalize and standardize its processes for documenting lessons learned from Impact Canada Cleantech Challenges. OERD will also explore options and develop a plan for sharing lessons from Impact Canada Cleantech with NRCan senior management and other sectors.

Following the awarding of prizes and program wrap up, OERD will develop a lessons learned summary document, which will be updated as lessons emerge.

OERD will continue to share lessons with the PCO Impact and Innovation Unit to support the development of “Challenge Resource Materials” that are shared and presented to other federal departments and the public (e.g. PCO case studies, PCO Challenge Guide and PCO advice on challenge juries).

OERD will also continue with its practice of giving presentations on Impact Canada within NRCan and to other departments upon request.

Position Responsible: DG, OERD on behalf of ADM, ETS  

Date to achieve:

  • December 30, 2021 to develop plan for documenting and sharing lessons
  • September 30, 2022 to develop lessons learned document and then updated as lessons emerge
  • The ADM ETS should review and finalize the performance measurement strategy for the ICI-CTS. This strategy should include:
    • Logic model (or theory of change), indicators, data collection methods, data sources, and potential counterfactuals.
    • Metrics for the program stream as a whole, as well as for each challenge or initiative to demonstrate the achievement of outcomes. 
    • As applicable, metrics to inform GBA+ considerations.
    • A plan for the timing and means by which reporting on program results and impacts will be communicated to senior management.
Management agrees. OERD will update and finalize the core Impact Canada Cleantech performance measurement strategy, including:

  • Finalize the draft Impact Canada Cleantech logic model, indicators, data collection methods, data sources, and, where appropriate and possible, identify suitable counterfactuals. OERD would like to acknowledge that due the nature of transformative innovation and RD&D, OERD will also draw on principles from innovation literature and developmental evaluations by integrating new outcomes, indicators, and updating assumptions based on emergent findings.
  • Work is underway to update and finalize core metrics for individual challenges and the overall program.
  • Work is underway within OERD to integrate GBA+ metrics into program data collection and monitoring. OERD will continue to collaborate within NRCan, and with PCO and Statistics Canada to collect and monitor GBA+ and equity, diversity and inclusion data where possible.  

Position responsible: DG OERD on behalf of ADM,ETS

Date to achieve:

  • March 31, 2022 for finalization of performance measurement strategy.
  • Ongoing performance measurement and monitoring until 2029 (at minimum)    
  • The ADM ETS should ensure that the risk-based recipient audit planning and methodology is strengthened to include the tracking and monitoring the status and results of recipient audit recommendations.
Management agrees. OERD will strengthen its risk-based recipient audit plan and methodology to ensure that it better documents and tracks the status and results of audits at the Branch (OERD) level, rather than Division level. OERD has already undertaken a review of its risk-based audit processes and updated its methods for planning, selecting and monitoring audits to ensure that all OERD programs implement NRCan’s Guide on Recipient Auditing in a consistent manner (ex. consistent sampling method). OERD piloted its updated risk-based recipient audit planning and selection method in Q2 2021. OERD is currently refining methods, and developing new guidance documents and tools for monitoring OERD’s annual audit plan. Implementation of processes and tools for monitoring audit recommendations and compliance will occur in Q3&Q4 2021-22. This will include optimising the use of NRCan’s AMI database for tracking audit results, and proponent’s audit history.

Position responsible: DG, OERD on behalf of ADM, ETS

Date to achieve: March 31, 2022

Introduction

This report presents the findings, conclusions, and recommendations from the joint audit and evaluation of the Impact Canada Initiative (ICI) – Clean Technology Stream (CTS). The engagement examines the design and delivery of the program from its announcement in Budget 2017 to March 2021, with updates reflecting the impact of COVID-19. The Audit and Evaluation Branch (AEB) of Natural Resources Canada (NRCan) conducted this engagement in accordance with the Treasury Board (TB) Policy on Results (2016) and the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing and the Government of Canada’s Policy on Internal Audit.

This is the first audit or evaluation of programming launched under the Impact Canada Initiative. The results of the joint engagement provide lessons learned that can be used to inform the future use of outcomes-based approaches to innovation or related mechanisms.

Program Information

Announced in Budget 2017, the Impact Canada Initiative (ICI) is a whole-of-government effort, led by the Privy Council Office (PCO), that is designed to help departments accelerate the adoption of outcomes-based approaches to deliver results to Canadians. The ICI’s outcomes-based approach is intended as a new way of opening up the problem-solving process to innovators who can bring fresh perspectives and new ideas. The ICI promotes the flexible use of grants and contributions through a range of innovative funding models, including prize challenges. These approaches can be supplemented with additional funding supports to recipients and/or third parties for project elements such as training, technical assistance, business planning, and evaluation. The Government of Canada has developed a common set of Terms and Conditions (Ts&Cs), housed within PCO, as a horizontal authority for all departments to pursue this outcomes-based programming.

Announced as part of the ICI in Budget 2017, NRCan’s Clean Technology Stream (CTS) is one of the first program streams developed under this new initiative. The objective of the ICI-CTS is to co-develop and launch a series of prize challenges to achieve breakthroughs in clean tech innovation, addressing climate change and environmental impacts while harnessing new economic opportunities. Challenges are designed, marketed and delivered differently from the types of funding programs that governments and their stakeholders are accustomed to running and participating in. They provide incentives (both financial and non-financial) to encourage a broad range of innovators to tackle problems where solutions are not apparent, or current responses are not achieving the desired results. Challenges may be structured to use:

  • Pure prizes where specific technical criteria are clearly defined at the outset and must be met before prizes are awarded;
  • Stage-gated approaches where challenge participants receive incentives at different stages of the challenge to build capacity or narrow the field of applicants before selecting winning projects; and/or
  • Competitive accelerators where prize funding is layered into intensive and time-limited business supports to propel cohorts of early stage innovators towards achieving particular outcomes.

Impact Canada also creates space for Indigenous innovation initiatives. These initiatives borrow elements from challenge-based approaches but rely less on the competitive aspects of the challenge model, and emphasize instead the importance of Indigenous leadership and collaboration with and among communities.1 This latter approach is not explicitly recognized in the program’s foundational documents but rather emerged as a result of NRCan and other federal departments’ engagement with Indigenous peoples during delivery of the ICI.

Under the authorities of the ICI, specific parameters of each clean tech challenge were to be developed, co-designed and refined by NRCan through extensive stakeholder engagement before launch and challenge delivery. Two focus areas were committed to as part of program approvals, i.e., one focused on sustainable aviation fuels and another on supporting Indigenous communities to reduce their reliance on diesel. The program has since launched five challenges and one initiative (see Figure 1). Each of these has its own mission and unique set of goals, performance targets, incentive structures, timelines, collaborators, participants, outreach strategies, and evaluation criteria.

Impact Canada’s umbrella Ts&Cs include an Annex specific to the CTS. Budget 2017 allocated NRCan $75 million (M) over two years for its delivery. This was subsequently extended to four years prior to program launch, from 2017-18 to 2020-21. The program’s funding authorities notionally distributed the funding between grants ($50.8 M) and contributions ($16.9 M), with the remainder earmarked for program administration. The actual funding profile was to be determined once challenge design was complete. Distribution of planned expenditures assumed challenge design would be completed in Summer/Fall 2017, with a full suite of challenges launched or ready to launch by Winter 2018.

Figure 1: Profile of ICI-CTS Focus Areas

Sky's the Limit (STL) Challenge

  • Launched in August 2018, a two-part challenge to accelerate innovation, production and use of sustainable aviation fuel (SAF) in Canada.
  • Part 1: Green Aviation Fuels Innovation. Top 4 consortia-based teams each receive up to $2.15M to support development of a 10L sample of the greenest and most cost-effective SAF. Grand prize of $5M to be announced by Spring 2022.
  • Part 2: Cross-Canada Flight. To complete the first cross-Canada flight using at least 10% SAF made in Canada. The first producer to meet all criteria would win a pure prize of $1M. Competion ran March 2019-January 2021.

Women in Clean Tech (WICT) Challenge

  • Launched in May 2018, this challenge works to improve gender balance in clean technology by enabling women to rapidly advance their clean technology solutions. It is delivered by the MaRS Discovery District via a contributon agreement with NRCan.
  • Six women each receive a stipend of $115K a year for 3 years, plus business and technical support from MaRS and $250K in value for federal labs to test and validate ideas in order to move toward commercialization.
  • Grand prize of $1M (to woman who made the most progress advancing her technology and growing her business) to be announced in Fall 2021.

Charging the Future (CTF) Challenge

  • Launched in July 2019, challenge is designed to increase the pace of Canadian innovation of battery technologies to strengthen the battery value chain in Canada and the world, and accelerate the most promising made-in-Canada innovation of battery technologies from the laboratory towards the marketplace.
  • Top 5 finalists receive up to $770K each to develop prototypes of proposed solutions.
  • Winner of $1M grand prize (project with highest likely impact on Canada's battery ecosystem) to be announced in Spring 2022.

Crush It! (CI!) Challenge

  • Launched in October 2018, challenge is designed to develop a new clean technology solution that transforms how energy is used for crushing and grinding rocks in the mining industry.
  • Total of 12 semi-finalists, with selected small-scale innovators awarded $10K each to help prepare and present their ideas to challenge jury. Top 6 finalists receive up to $860K each to start building and testing their solution.
  • Winner of $5M grand prize to be announced in Fall 2021.

Power Forward (PF) Challenge

  • Launched in October 2018, brings together innovators in Canada and the United Kingdom (UK) to develop innovative smart grid solutions for more integrated, cleaner, and flexible power grids for 2030 and beyond. Delivered as an international partnership, at least 20% of the proposed project should include firms from the partner country.
  • Total SME financial support of $1M/£600K in each country to eligible teams who have the most promising project vision. Top 7 finalist teams receive up to $3.15M/£1.8M each to build their pilot project.
  • Winner of $1M grand prize to be announced by Spring 2022.

Indigenous Off-Diesel Initiative (IODI)

  • To support Clean Energy Champions and their communities with tailored clean energy training, access to expertise and financial resources to develop and start implementing an ambitious plan to reduce diesel use for heat and power.
  • Phase 1: Clean Energy Champions participate in the 20/20 Catalysts program and receive $20K to get started. Phase 2 and 3: Up to $1.3M per community for training, capacity building, and to develop and implement community-scale clean energy plans.
  • In Spring 2023, up to an additional $9M will be awarded to leading communities to support project implementation over three more years.

In September 2020, the Minister of Natural Resources approved a program change to remove the expiry date of the Clean Tech Program Stream Annex in response to COVID-19. Funding was extended to 2021-22. Additional program modifications made in response to the pandemic (Box 1) and are reflected in the profile of focus areas presented in Figure 1.

Box 1: Program Response to COVID-19, by Focus Area

The list below includes the concrete actions that were taken in each focus area to support the delivery in the ICI-CTS in response to COVID-19.

  • Sky’s the Limit (STL): Up to $150K in additional funding offered to each finalist (total $600k) and extension for final prize submission to August 31, 2021. GARDN (a challenge collaborator) was also offered up to $36K in additional funding.
  • Crush It!: Up to $60K in additional funding offered to each finalist (total $360K) and extension for final prize submission to May 31, 2021.
  • Power Forward (PF): Up to $150K in additional funding offered to each finalist (total $450K for Canadian finalists)2 and extension for final prize submission to October 1, 2021.
  • Charging the Future (CTF): Up to $70K in additional funding offered to each finalist (total $350K) and extension for final prize submission to October 29, 2021.
  • Women in Clean Tech (WICT): Up to $470K in additional funding offered, i.e., $345K for finalists and $125K for MaRS (a challenge delivery agent3) and extension for final pitch event where the winner will be selected to September 2021.
  • Indigenous Off-Diesel Initiative (IODI): Up to $25K in additional funding offered to Clean Energy Champions (total $300K for Phase 2 activities) and extension for Phase 3 submission to March 31, 2022. The Indigenous Clean Energy (ICE) Social Enterprise and the Pembina Institute (both collaborator in the initiative) were offered additional funding ($805K and $325K, respectively) and an extension to March 31, 2022.

Governance Structure and Program Accountabilities

The ICI is a whole-of-government mechanism. PCO has established a Centre of Expertise (COE) within its Impact and Innovation Unit to support the design of program streams and individual departmental projects across a range of challenge areas. While projects launched under the ICI-CTS program stream are expected to follow PCO’s Impact Canada Ts&Cs, NRCan has accountability for delivery of the ICI-CTS.

Within NRCan, the ICI-CTS falls under the responsibility of the ADM of the Energy Technology Sector (ETS). The specific parameters of each challenge were developed by ETS’ Office of Energy Research and Development (OERD), in collaboration with other sectors as required. For example, NRCan’s Lands and Minerals Sector (LMS) led the design and delivery of the Crush It! challenge. NRCan’s Deputy Minister was responsible for ultimate approval of challenge design prior to implementation.

While PCO does not manage program funding or oversee project implementation, the PCO COE may still provide advisory support to NRCan, as requested. NRCan is expected to make use of a government-wide interactive website (developed by PCO as a core component of the ICI) to both inform Canadians on the challenges launched and as the portal to process applications.4 NRCan is also required to inform the PCO COE on the design of its challenges and the outcomes achieved through individual projects so that PCO can track overall outcomes achieved under the ICI structure.

Expected Results

The Impact Canada – Clean Technology Stream is part of the Energy Innovation and Clean Technology Program under NRCan’s Core Responsibility for ‘Innovative and Sustainable Natural Resources Development’ in the Departmental Results Framework.

The ICI-CTS’ performance measurement framework is still in development. PCO has developed a logic model and narrative that presents (in a general way) a basic theory of what Impact Canada expects to achieve with its suite of challenges, including a broad objective to accelerate the use of innovative and experimental approaches across the Government of Canada.5 Building off this theory, Figure 2 presents a (draft) logic model developed by NRCan’s OERD specific to the ICI-CTS. This model nests the outcomes targeted by each challenge and initiative (indicated by green boxes in the figure) within the expected results for the program stream as a whole (blue boxes).

Expected results of the ICI-CTS are aligned with other NRCan energy RD&D programs, with a focus on advancing clean technology (clean tech) closer to commercial readiness to improve the environmental and economic performance of Canada’s clean tech and natural resource sectors. The ICI-CTS is also expected to broaden the reach of NRCan’s traditional clean tech programming, by increasing public awareness of clean tech problems or issues, stimulating innovation ecosystems by building non-traditional collaborations, and mobilizing new talent or solvers. Challenges were intended to be inclusive in order to attract and encourage participation from different sectors. To this end, the list of eligible recipients is broad and can include incorporated and unincorporated groups, private sector and not-for-profit organizations, other governments and other levels of government, Indigenous organizations, academia, and individuals, with the range of eligible recipients narrowed as appropriate for individual challenges based on desired outcomes.

Collaboration and partnership-building are also meant to be a core feature to the design and delivery of each challenge. Depending on the focus area, collaborators have been used as delivery agents, to provide challenge participants with a range of supports, to build connections within the innovation ecosystem and/or to draw the public’s attention to a challenge.

While recognizing the importance of the outcomes targeted by each challenge and initiative, particularly to the relevance of the program, this joint engagement is focused on expected results for the ICI-CTS as a whole. This includes an expectation for experimental learning from the design and implementation of the challenges and initiative as an outcome that overarches all expected results for the program.

Figure 2: (Draft) NRCan Logic Model for the ICI-CTS

Text Description

Figure 2: (Draft) NRCan Logic Model for the ICI-CTS

Infographic showing the draft logic model for the ICI-CTS.
The first row features two (2) ultimate outcomes, which are:

  • First Outcome - Improved environmental performance of Canada's natural resource sectors.
  • Second Outcome - Improved economic performance of Canada's clean technology and natural resource sectors.

The second row features six (6) challenge outcomes that support program’s ultimate outcomes. The six challenge outcomes are:

  • Gender parity in the cleantech sector
  • Decarbonised aviation
  • Integrated, cleaner and flexible power grids ready for 2030
  • Significant reduction of energy use at Canadian mines
  • Significant reduction in reliance on diesel in remote communities
  • Canada’s battery value chain is globally competitive

The third row features two (2) intermediate outcomes, which are:

  • New disruptive technologies, alternative business models, and capacity building activities facilitate
  • Project investments and ecosystem supports move emerging technologies closer to commercial readiness

The fourth row features six (6) challenge outcomes supporting program’s intermediate outcomes, which former are:

  • Strengthened female leadership in the cleantech sector
  • Sustainable aviation fuel produced in Canada with lower GHG emissions
  • Next-generation Canada/UK smart grid demonstration projects ready for scale-up
  • New mining technology reduces energy use in Canadian mines by at least 20%
  • Up to 15 Indigenous communities have new clean energy plans and projects underway to transition off diesel
  • Canada's five best battery technology innovations being advanced

The fifth row features four (4) program’s immediate outcomes, which are:

  • Increased collaboration in clean technology challenges and projects
  • Increased investment by stakeholders in clean technology ecosystem
  • Increased awareness and participation in clean technology (i.e., new talent mobilized)
  • Increased public awareness of clean technology issues among Canadians.

The sixth row features five (5) program outcomes, which are:

  • Increased Co-designed challenges and initiatives
  • Communications and outreach strategies
  • Collaborator arrangements
  • Proponent supports
  • Solutions proposed and awarded

The seventh row features as sequence of seven (7) activities, which are from left to right:

  • Challenge Selection
  • Challenge Design
  • Challenge Launch
  • Challenge Delivery
  • Final Award (Prize)
  • Transition Solutions
  • Measure & Share Results

The eighth row from the top of the graphic features the results form the challenge selection and identifies selected challenges, which are from left to right:

  • Women in Cleantech
  • Sky's the Limit
  • Power Forward
  • Crush It!
  • Indigenous Off-Diesel Initiative
  • Charging the Future

The ninth row from the top of the graphic outlines the program’s input, which from left to right are:

  • NRCan Funding ($76.5M from 2017-18 to 2021-22)
  • NRCan Personnel (Approx. 10 FTE per FY)
  • Facilities (e.g., federal labs)
  • PCO Support (e.g., ICI Portal)
  • Terms & Conditions

On the far right of the infographic, a double-sided arrow indicates that experimental learning is happening throughout the program’s steps that appear in the program logic model.

Engagement Objectives and Methods

The AEB included a commitment to conduct this joint engagement in its Integrated Audit and Evaluation Plan 2019-2024. The AEB identified the need for this engagement through its risk-based planning process and in response to a TB commitment to complete an evaluation. The engagement also meets requirements for evaluation of ongoing grant and contribution (Gs&Cs) programs under section 42.1 of the Financial Administration Act (FAA) and the Treasury Board (TB) Policy on Results. The Evaluation Division led this engagement, in collaboration with the Audit Operations Division. Where relevant, the approach for this joint engagement also followed the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing (IIA Standards) and the TB Policy on Internal Audit.

The objectives of the joint engagement are:

  • To assess the extent to which the program’s design and delivery facilitates the effective and efficient achievement of immediate results and complies with relevant authorities.
  • To identify and document lessons learned from the experimental approaches under the Impact Canada – Clean Technology Stream that can be applied to future interventions.

The original scope of the engagement included NRCan’s design and delivery of the Impact Canada – Clean Technology Stream from 2017-18 to 2019-20, including outcomes for the program stream as a whole and the activities and outputs for each of its six focus areas. This was expanded in March 2020 to enable the engagement to consider the impact of COVID-19 on delivery of the program stream. While we consider the role of PCO in facilitating design and delivery by NRCan, the engagement’s scope excludes the performance of PCO and that of international collaborators in the Power Forward Challenge.

Engagement Questions and Sub-Objectives

Table 1 identifies the questions and related sub-objectives to be addressed in this joint engagement. Appendix A summarizes the detailed criteria applied in assessing these sub-objectives.

Table 1: Engagement Questions and Sub-Objectives

Engagement Questions Sub-Objectives
  • Are the Impact Canada – Clean Technology Stream and its related challenges relevant?
  • To assess the extent to which the program is relevant.
  • Are adequate and effective governance structures in place to support the Impact Canada – Clean Technology Stream?
  • To assess whether adequate and effective governance structures and processes have been designed and implemented to support the achievement of the Impact Canada – Clean Technology Stream’s outcomes.
  • To what extent does the experimental design of the Impact Canada – Clean Stream facilitate the achievement of intended outcomes?
  • To assess how and for whom the Impact Canada – Clean Technology Stream’s experimental tools are effective in facilitating the achievement of intended outcomes.
  • Do the delivery and monitoring of the Impact Canada – Clean Technology Stream’s activities comply with relevant authorities?
  • To assess whether adequate and effective processes and controls are in place to support the Impact Canada – Clean Technology Stream’s compliance with relevant departmental guidance and the TB Policy on Transfer Payments.
  • To assess whether key financial and operational controls have been designed and implemented, and are operating effectively.
  • What have been the outcomes (expected and unexpected) that the Impact Canada – Clean Technology Stream has achieved to date?
  • To assess the extent to which the Impact Canada – Clean Technology Stream is making progress towards its expected immediate outcomes.
  • To identify if any unexpected outcomes (positive or negative) have resulted from the Impact Canada – Clean Technology Stream.
  • To what extent has the program been responsive to the internal and external factors that (positively or negatively) influence its ability to achieve intended outcomes and operate with efficiency?
  • To assess the extent to which the Impact Canada – Clean Technology Stream appropriately considers the internal and external factors that could impact its ability to effectively or efficiently achieve its intended outputs and outcomes.     
  • What are the conditions required to accelerate the use of the Impact Canada – Clean Technology Stream’s innovative approaches to future interventions?
  • To identify lessons learned that could support the positive evolution of government systems to enable greater innovation.

 

Engagement Methods

The joint engagement limits the duplication of effort that would be required to conduct separate audit and evaluation projects. The engagement was carried out in a manner that ensures the team’s neutrality and objectivity, as per respective professional requirements and standards, and ensures that observations and conclusions are evidence-based.

The approach and methodology used in this engagement followed the TB Policy on Results, including related Standards on Evaluation, the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing (IIA Standards), and the TB Policy on Internal Audit. These standards and policies required that the project be planned and performed in such a way as to obtain reasonable assurance that engagement objectives are achieved. The engagement included tests considered necessary to provide such assurance.

The joint engagement used five data collection methods (see Table 2), designed to ensure validity of the information collected and allow for the triangulation of evidence.

Table 2: Engagement Methods

Literature Review Key Informant Interviews Document Review File and Database Analysis Survey of Non-Finalists

Given the relative novelty of prize challenges for the Government of Canada, the engagement team completed a review of available literature to identify best practices and lessons learned in the design and delivery of such mechanisms, including a review of their application in benchmark challenge-based programs and innovation accelerators.

The evaluation team conducted interviews with key informants (n=93), including NRCan senior managers, program managers and challenge leads, federal labs, external collaborators, and nearly all finalists from across the six focus areas of the ICI-CTS.6 Questions addressed related to program relevance, performance and lessons learned.

The audit team conducted additional interviews with both NRCan program staff and corporate services to address questions related to program governance and compliance with authorities.

The joint engagement team completed an extensive review of public and program documentation, including financial information, related to the design and implementation of the Impact Canada – Clean Technology Stream to inform the examination of the program’s relevance, design and delivery, and outcomes achievement.

The audit team reviewed a sample of 25 funding agreements across the six focus areas to determine whether funding agreements comply with relevant FAA requirements.

The engagement’s database analysis also included a review of the information collected via PCO’s web portal, NRCan’s funding agreement tracker, and social media analytics to inform analysis of outcomes achievement.

The engagement conducted two surveys of non- and semi-finalists – one for the five challenges and another to reflect the distinct design and language of the IODI. The surveys obtained information about these applicants’ experiences with the program and their perspectives on the achievement of immediate outcomes. The surveys were sent to over 300 applicants; 7 were IODI finalists who could not attend interviews. In total, 85 participants responded to the surveys (response rate of 24.6%).

Engagement Limitations and Considerations

The engagement used multiple lines of evidence in order to mitigate against any limitations associated with individual methods. This enabled the triangulation of evidence across sources of information to identify valid findings and conclusions relative to the engagement’s questions. Nevertheless, the following limitations should be considered when reviewing the findings from this engagement:

Timing of Evaluation. An evaluation of the ICI-CTS was required to be completed at this time to comply with the TB Policy on Results and the Financial Administration Act. However, it is really too soon to report on the impact of this program; measurement of program outcomes will require tracking the trajectory of results over the long-term.

COVID-19. COVID-19 had a significant impact on the design and delivery of both the program and the joint engagement. The conduct (data collection) phase of the joint engagement was delayed by several months to allow time for program collaborators, stakeholders and participants to adjust to the pandemic. The program has also effected important design changes in response to pandemic-related delays and financial pressures (e.g., to the structure of payments and timelines for project completion). Related decisions were not all finalized and/or communicated at time of data collection. Pressures related to COVID-19 also limited the ability of some key informants to participate in the engagement.

Document Review. The document review, file and database analysis were constrained by the information that is collected and maintained by OERD. Documentation required for review was generally accessible but varies considerably in quality and consistency by focus area. Given timing of data collection, mid-project reports to inform our analysis of progress towards outcomes were not yet available for all focus areas. Funding agreements were reviewed in consideration of NRCan’s responsibilities for their delivery, and limited work was performed in assessing areas of responsibility where delivery agents were used.

Analysis of internal reporting mechanisms included a review of reporting documentation beginning as early as challenge design and development to challenge launch, which spanned the period from April 2017 through to December 2018. Documentation related to one challenge (CTF) was not included in this material due to its development and launch in a later period.

Key Informants. Respondents for interviews and the engagement’s surveys were not all familiar with some elements of design and implementation. In some cases, the respondent was not necessarily the same as the applicant for a specific project and so could not comment on early activities and outputs in program delivery (e.g., effectiveness of initial outreach or innovations in the application process). The time elapsed between co-design or application process and the conduct of the engagement also meant some respondents could not recall all details or make any specific recommendations for improvement. Companies receiving services from the Centre for Mining Innovation (CEMI), a collaborator on Crush It!, were not included among respondents for this engagement and we lack their perspectives on CEMI’s appropriateness or effectiveness.

Survey. The overall survey response rate of 24.6% is considered adequate for purposes of this engagement. Response rates by focus area range from 20.4% for the IODI to 50% for STL. However, the total number of applicants and thus survey respondents is not divided evenly among focus areas. The relatively small sample for some specific focus area makes it difficult to report disaggregated data on a challenge-by-challenge basis.

Additional limitations and considerations are identified where relevant throughout this report.

What We Found – Relevance

Summary of Key Findings:

The ICI-CTS is relevant. The program aligns with federal government and NRCan priorities, roles and responsibilities. Impact Canada supports mandate commitments to increase experimentation across the Government of Canada, and is structured to give more flexibility to proponents to propose innovative solutions. The narrow focus of each challenge and initiative sets this apart from other funding programs, with the suite of clean tech challenges acting as vehicles to enhance the communication of priorities in the Government of Canada’s environmental agenda.

The selected challenges and initiative also meet key criteria for the prioritization of focus areas. They target breakthrough (i.e., non-incremental) solutions, have a high potential for benefits to Canadians, are complementary to existing government and non-government efforts without creating duplication, and their design demonstrates an ability to leverage collaborators and resources to accelerate results.

The ICI-CTS aligns with federal government and NRCan priorities, roles and responsibilities.

In 2015, a Ministerial mandate letter directed the President of the Treasury Board to “work with […] colleagues to ensure that they are devoting a fixed percentage of program funds to experimenting with new approaches to existing problems and measuring the impact of their programs”. Budget 2017 announced the Impact Canada Initiative and identified this as a mechanism by which this mandate commitment could be achieved. The ICI is perceived to fill an important gap in the Government of Canada’s programming toolkit by establishing a mechanism to work with and support departments to employ innovative funding models. Interviewees agreed that the program is structured in such as way as to be more flexible and less administratively burdensome for proponents, giving more flexibility to propose innovative solutions.

The Impact Canada – Clean Tech Stream is more specifically aligned with federal government and NRCan priorities for clean tech. In 2015, the Minister of Natural Resources was mandated to invest in clean tech producers in our natural resource sectors so that they can tackle Canada’s most pressing environmental challenges and create more opportunities for Canadian workers. Canada also made a Mission Innovation pledge to double its investments in clean energy and clean tech research and development by 2019-20 and encourage private sector investment in early-stage clean energy innovation companies in Canada. In Budget 2017, the federal government committed more than $2B to numerous initiatives (including the ICI-CTS) aimed to boost the growth of Canada’s clean tech sector. More recent Ministerial mandate letters (2019) and Throne Speeches (2019) continue to prioritize positioning Canada as a global leader in clean tech.

From the perspective of program representatives, the narrow focus of each challenge or initiative sets this apart from other funding programs, with the suite of focus areas acting as vehicles to enhance the communication of priorities in the Government of Canada’s environmental agenda. For example, four of the challenges and the initiative specifically target environmental benefits related to energy efficiency and greenhouse gas (GHG) mitigation. The WICT challenge is targeted more broadly at environmental benefits, but aligned with priorities to increase gender equity and the participation of women in the Canadian economy and clean tech. The IODI is aligned to priorities for both reconciliation and building the resilience of Indigenous communities, including a mandate commitment (2015) to support the transition of Indigenous communities from reliance on diesel-fueled power to clean, renewable and reliable energy by 2030.

The Impact Canada – Clean Technology Stream gave priority to the selection of relevant challenges and initiatives.

During program design, an analysis of best practices determined that running between four and six clean tech challenges was optimal given the program’s funding profile. However, only two focus areas were identified in program approvals – i.e., supporting Indigenous communities to reduce their reliance on diesel and a sustainable aviation fuels challenge to reduce GHG emissions in aviation – with the remainder to be prioritized based on a specific set of criteria. The criteria and our observations on related program alignment are as follows:

  • Breakthrough Solutions. The program was expected to focus on unlocking transformative (i.e., not incremental), high-impact solutions to complex and persistent problems in the clean tech sector. We found that the program has a clear definition of what would constitute a “breakthrough”; its approach brings focused attention to barriers and opportunities specific to each problem area without specifying how the result is to be achieved, leaving space for innovation to occur. There is evidence that the transformations targeted were well researched in the design phase. The objectives of the challenges and initiative are diverse but are all expected to provide innovative solutions to disrupt the market and/or create knock-on or ripple effects that will boost the entire ecosystem in their problem area (Table 3). While the IODI is unlikely to spur growth in new clean technologies, it presents a breakthrough in ensuring that traditional knowledge and consideration of systemic inequalities are at the center of program design and delivery. Recognizing the federal government’s responsibility to respect the right of Indigenous peoples to self-determination, the design of the IODI relies on communities to develop their own solutions.

Table 3: Issues Addressed by ICI-CTS Focus Areas

Focus Area The Issue
Women in Clean Tech

Women are a powerful force in Canada's innovation economy, but are significantly underrepresented in the area of clean tech. It's time to change that.

Sky’s the Limit

Existing technologies and aircraft efficiency improvements aren’t enough for the aviation industry to achieve its ambitious goals of carbon neutral growth. Accelerating the affordability and availability of sustainable aviation fuel is an important part of Canada’s transition to a low-carbon economy.

Power Forward

Between now and 2030, millions of new devices, including electric vehicles, renewables, batteries, and smart devices) will challenge traditional operating models of energy systems, adding complexity and volatility to grids.

Crush It!

Comminution processing consumes a disproportionate amount of the total energy used in mining, is a contributor to the mining sector’s GHG emissions, and represents one of the industry’s largest costs. Over the history of comminution, breakthrough advances have been limited.

Charging the Future

The batteries industry anticipates exponential growth to meet expected demand on electrical vehicles, support greater integration of renewable energy, and provide opportunities to help manage the electricity grid. Although batteries have significantly developed since their invention, there continues to be unlocked potential to harness from such technologies.

Indigenous Off-Diesel Initiative

There are over 200 remote communities in Canada that rely on diesel for both their heat and power, the majority of which are Indigenous communities. While diesel has remained the status quo for many years, rapid advances in renewable and energy efficiency technologies make it possible to more easily integrate cleaner energy systems. The federal government has committed to ensure that Indigenous communities that currently rely on diesel are powered by clean, reliable energy by 2030.

Text Description

Infographic that illustrates, using a basic Venn diagram, the program’s potential benefits to Canadians that are of five types:environmental, policy, societal, economic and technical

  • High Potential for Impact and Benefits to Canadians. The program was expected to prioritize focus areas with a strong potential for reducing environmental impacts and improving economic performance, as well as for increasing the participation of women and Indigenous peoples. We found that all focus areas were designed to target environmental benefits. In some cases, targeted benefits have been quantified (e.g., project proponents for Crush It! are challenged to generate a step-change reduction equal to or better than a 20% reduction in the energy consumed by crushing and grinding when compared to a pre-established baseline). Challenges were also selected in areas where there are significant opportunities for economic benefit and market growth. Technical benefits would be derived from the solutions themselves as well as from potential ripple effects across the entire value chain or that create new value chains, and are expected to amplify the environmental and economic benefits. Early or accelerated commercialization of clean technologies would also provide a competitive advantage in the global market. Unlike the other challenges, the main benefits of the IODI are expected to be at the community level. Similarly, the support to six finalists for WICT on its own is not expected to significantly impact the gender balance in the clean tech sector. However, in both cases a knock-on effect would be achieved if others are inspired to take action. The clean tech sector as a whole should also ultimately benefit from increasing participation of diverse groups (e.g., women, Indigenous peoples) and new solvers in clean tech.
  • High Degree of Complementarity. The program was expected to prioritize focus areas that would have a high degree of complementarity with existing government and non-government efforts in the given problem space. We found that the focus areas selected meet this criterion. Overall, we found that the design & delivery of the ICI-CTS fills a perceived gap in dedicated support for high-risk, high-reward clean tech solutions. The focus areas support priorities that are also addressed through other clean tech programs, but each fills an identified gap without creating duplication. While the identification of follow-on strategies is outside the scope of the program, there is also evidence of viable pathways to scale successes for each focus area (e.g., clearly identified opportunities for handoff to other government programs, venture capital, etc.).
  • Ability to Leverage Collaborators and Resources. The program was expected to prioritize focus areas with the ability to leverage collaborators and resources to help design, deliver or promote participation in the prize or challenge, including by acting as a conduit to important stakeholder groups, providing technical expertise, or by contribution of additional resources (financial or in-kind). There is evidence that the opportunities to leverage relevant collaborators to support and accelerate results were an important consideration incorporated into the design of each focus area. NRCan also expected the ICI-CTS to encourage proactive engagement and collaboration with traditional and non-traditional stakeholders to enhance Canada’s innovation ecosystem. The challenges are thus designed to provide a unique platform for networking and forming collaborative arrangements, highlight promising research, development and demonstration (RD&D) ideas and help the ecosystem overcome barriers.

What We Found - Governance

Summary of Key Findings:

Overall, we found that governance structures, processes and mechanisms for ICI-CTS were defined, established and operating effectively.

Roles, responsibilities and accountabilities. Roles, responsibilities and accountabilities were clear in principle. The governance structure for the Impact Canada Initiative of both NRCan (and the OERD team responsible for its delivery) as well as the PCO’s Center of Expertise (COE), were found to be adequately established, defined, and communicated across authoritative documents. This included clear delineation between the responsibilities of PCO and NRCan and outlined areas of complementary and overarching responsibilities. However, in consideration of this unfamiliar policy context, difficulties were faced in the initial stages of program design and development resulting from differences in perspectives concerning the program’s interpretation of the ICI Ts&Cs and associated authorities endorsed by PCO across key intradepartmental players, including various corporate functions (i.e., NRCan’s CoE on Gs&Cs, Legal and Procurement).

Internal Reporting. Throughout the design and development of the ICI, internal reporting mechanisms were found to be established and functioning as intended at all levels of program management. These reporting mechanisms (both formal and informal) enabled key communication on the program’s progression to senior management at intervals that supported timely reporting across all phases of design and implementation.

Roles, Responsibilities and Accountabilities within the ICI-CTS

The governance structure for the management of the ICI-CTS at NRCan has been defined in authoritative documentation that describes the roles, responsibilities and accountabilities of NRCan, a key federal partner for the Impact Canada Initiative. The ICI-CTS is supported through the organizational structure of ETS’ OERD, responsible for the Program’s delivery. The authoritative document reviewed refers to a well-defined program framework that would leverage OERD’s existing governance structure, given its experience with transfer payment administration, and identifies resources for ensuring the successful delivery of the Program, with defined roles and responsibilities.

The complementary and overarching roles, responsibilities and authorities of PCO’s COE on Impact Canada, as well as its interactions with federal partner departments operating as part of this initiative were also described, with clear delineation across the two governance structures. Documents recognize that PCO’s authority would not extend to direct program management or funding but instead, assist with overseeing and enabling the departments’ approach to innovative programming. NRCan would operate using the Ts&Cs of PCO; however, NRCan would remain the authority for the direct management and implementation of the ICI-CTS. PCO would primarily be responsible for tracking progress on the approach of the ICI from an overall perspective, as well as general outcomes.

At NRCan, key roles and responsibilities for Gs&Cs are shared at the corporate and sectoral levels. The NRCan Guide on Transfer Payments identifies key players for the administration and management of Gs&Cs, including Program Staff, the Center of Expertise on Grants and Contributions (CoE on Gs&Cs), Legal Services, the Finance and Procurement Branch (FPB), Delegated Authorities, and the Transfer Payment Review Committee (TPRC), which has since been replaced by the Finance and Real Property Committee (FRPC). Overall, the roles, responsibilities and accountabilities of key intradepartmental players for the management of Gs&Cs programs were found to be defined and communicated through the departmental intranet website and key guideline and procedural documents and well understood within the Department.

Across the organization, the governance structure and processes in place for the management of Gs&Cs were found to facilitate the involvement of key intradepartmental players in situations where required, or requested. In the case of the ICI-CTS, and as with all new Gs&Cs programs at NRCan, these structures necessitated the involvement of Legal Services and the CoE on Gs&Cs in performing oversight and review activities of the development of Ts&Cs for the ICI-CTS and subsequent funding agreements. We found that early engagement involving NRCan’s key players for Gs&Cs had taken place, which played an integral role throughout the program’s design and development.

Subsequently, the program faced some difficulties during the initial stages of its design and delivery attributed to systemic and departmental barriers. Specifically, it was noted that there were often opposing perspectives in interactions between the program and corporate functions. The ICI-CTS was looking to challenge status quo and experiment with innovative ways of doing business, whereas NRCan’s corporate functions are responsible for enforcing policies, providing guidance, and identifying and containing areas of perceived risk or concern. Many of the challenges faced were found to have resulted from a lack of communication of and familiarization with the policy context and authorities offered through PCO’s Ts&Cs. This led to fundamentally different views in the interpretation of the program’s Ts&Cs, and the intersection of these viewpoints among corporate colleagues and the program led to resistance during the earlier phases of challenge design and development. However, since that time, the differences between various key players have been addressed with an ongoing collaborative approach.

Internal Reporting Mechanisms

Through an examination of various reports provided to senior management on the progress of the Impact Canada Initiative, we found that formal reporting structures were in place at all levels of management. The formats of these documents include briefing notes, presentation decks, speaking points, and corresponding project documentation, provided to the Director General (DG), Assistant Deputy Minister (ADM), Deputy Minister (DM), and the Minister’s Office at NRCan, in addition to several other departmental executive-level senior management committees, where relevant. The frequency of these internal reports to senior management were found to take place at intervals that supported timely and informed decision-making by executive management. Given the high profile nature of the ICI, many of these internal reporting documents were shared in order to provide updates, and to obtain Ministerial endorsement/approval relating to the proposed challenge design, structure and format for funding.

We reviewed reporting documentation beginning as early as challenge design and development to challenge launch, which spanned the period from April 2017 through to December 2018.

We also obtained reporting documentation from the implementation phase of the challenges, in the form of Memorandums to the ADM, which presented key project-related information. These reports provide key information on applications received, assessed, eligibility, evaluation and ultimately, a recommendation for approval. The interval of these reports varied as they depended on the each of the challenges unique structure and timelines.

Through the review of the documentation provided to senior executive management, we found that the information provided contained sufficient detail to inform the future phases of challenges that were, at this time, developed and launched. Overall, the team concluded that information provided to senior management contained sufficient detail to inform the future phases of challenges.

What We Found – Design

Summary of Key Findings:

We found that the design of focus areas under the ICI-CTS was consistent with principles of Impact Canada and best practices in challenge models. This design incorporated numerous experiments within an experiment. Our observations related to specific elements important to the experimental design of the program are as follows:

  • Co-Design. Co-design allowed NRCan to target key issues while scaling the challenge to what would be achievable by solvers, including consideration of specific barriers for the problem area. Stakeholder engagement in co-design prior to launch was extensive and brought visibility to increase interest and collaboration among players in the ecosystem. NRCan was responsive to feedback in both design and delivery phases. Risks related to co-design (e.g., potential conflict of interest) were well managed.
  • Communications and Outreach. There was effort dedicated to the development of front-end communication strategies. Regardless, program staff perceived that this could have been more innovative and better resourced. Overall, communications attracted “enough” of the “right people” to apply (based on number of quality proposals). Proponents were less positive about communications on program progress than for program launch noting that there was less opportunity or emphasis on this than they expected when they decided to apply (in part due to restrictions imposed by COVID-19).
  • Collaborative Arrangements. Collaborators played an important role in facilitating and/or accelerating delivery of focus areas. There is evidence that the collaborators selected were both appropriate and effective. Working with ICE and the all-Indigenous jury on IODI was perceived as a positive move towards new model of collaborative (rather than Crown-led) program delivery.
  • Financial and Non-Financial Supports. Funding flexibilities and proponent supports were designed to facilitate participation from diverse solvers and address barriers to participation, and were recognized to be effective in achieving these objectives. Participation was motivated more so by financial supports, with staged support noted by proponents to be more important than the (uncertain) final prize. Perceptions of the effectiveness of non-financial supports varied significantly across focus areas. For some focus areas, non-financial supports may not have (fully) aligned with the specific cohort of finalists’ perceived needs. Providing access to federal labs presented an opportunity to mobilize new players in the clean tech ecosystem, as these labs have more expertise and equipment than some smaller firms or consortia may have access to on their own.  While there is evidence of viable pathways to scale success for each focus area, this requires dedicated efforts that extend beyond the scope and timelines of the ICI-CTS but which may be accelerated as a result of the program.

Challenges have a set of inherent features meant to attract new talent and new ideas, and accelerate progress towards solving problems that matter to people. As a result, challenges are designed, promoted and delivered differently than the types of funding programs that governments and their stakeholders are accustomed to running and participating in. Overall, we found that the design of focus areas under the ICI-CTS was consistent with principles of Impact Canada and best practices in challenge models. Many focus areas also incorporated numerous experiments within this broader experiment, for example, testing new models of Indigenous participation (IODI) or new ways of working with international partners (Power Forward). We examined how and for whom the program’s experimental tools are effective in facilitating the achievement of intended outputs and outcomes, with a focus on five elements common to all focus areas, i.e.: inclusive innovation (co-design), communications and outreach, collaborative arrangements, flexible funding mechanisms, and additional (non-financial) proponent supports.

Inclusive Innovation (Co-Design)

The first key steps in any challenge are to understand, identify and define the specific problem that the challenge will seek to address and set the design parameters that will support its most effective delivery. Literature supports the benefits of inclusive innovation (co-design) in supporting this process. Co-design processes brings together experienced, diverse groups of internal and external stakeholders to apply their collective knowledge and form synergistic programming. Engaging with stakeholders is critical for identifying and understanding the issue, and selecting and defining a specific problem that will form the basis of the challenge. Stakeholders can also confirm that a challenge model is feasible and appropriate to seek solutions to the problem and that the financial and non-financial supports to be used are designed to meet the needs of proponents and end-users, thus helping to achieve outcomes faster and at lower total cost. Engagement of diverse stakeholders during the co-design process also increases network connections and strengthens problem-solving communities, creating further opportunities for collaboration that can support long-term innovation.

We found that all focus areas undertook a co-design process. NRCan used its own existing networks and those of its collaborators to reach out to a wide range of stakeholders in the relevant ecosystems, including traditional and non-traditional stakeholders. The timeline and specific approach to co-design varied significantly by challenge, with mechanisms used to engage stakeholders that included consultations, webinars, design workshops and surveys. However, each followed an iterative process that allowed the challenge concept to be validated more than once by groups of key stakeholders, with final approval by NRCan senior management.

Interviewees familiar with the co-design process agreed this was among the ICI-CTS’ main strengths. It allowed NRCan to refine the design to target key issues but also scale the challenge based on given design parameters (time and funds that could be allocated by NRCan) to what would be achievable by solvers, including consideration of specific barriers and pathways for the problem area. There is a risk in co-design of being so responsive to feedback that people propose and vote on ideas that are not practical or feasible. NRCan was perceived to have successfully managed stakeholder expectations in this area. While consensus could not always be achieved, there is evidence that NRCan shifted design of challenges as a result of co-design input. For example, the IODI shifted its design from a “challenge” to an “initiative” with feedback from Indigenous stakeholders that competition between communities as part of the off-diesel program was not in line with their values. Extensive engagement also shifted its design from a technology to a capacity building focus. 

Co-design also brought visibility to the challenges, as an early signal to key stakeholder groups that a challenge would be launched. For most challenges, there is evidence that the co-design process also resulted in furthering collaboration among players in the ecosystem. Given the international nature of Power Forward, inclusion of high profile events in co-design was important to not only collect stakeholders’ insights to support challenge design but also to start building partnerships critical to challenge delivery. By extension, given the small size of the nascent innovation ecosystems, there is also a risk of unfairly advantaging stakeholders engaged in co-design who may later also become proponents. There is evidence that NRCan considered and mitigated this risk.

Interviewees did not perceive significant barriers to participation in co-design. For IODI and WICT, there was a clear focus on co-design on inclusivity – i.e., who benefits from the innovation, who participates in the process of innovation, and who manages the outcomes of innovation. This means that co-design did not stop at challenge launch. These focus areas continue to be co-designed with input from finalists to ensure that capacity building elements and other supports meet their specific needs. Community engagement and support is critical to advance projects under the IODI, ensuring that those participating have a say in the design of projects that will impact their community. Both WICT and IODI also use an external jury that reflects the participants in the challenge itself. For IODI, many respondents viewed this co-design process as a positive step towards Indigenous self-determination.

Communications and Outreach Strategies

Literature supports the importance of communications as a core element in challenge design. Recognizing that this would be a key determinant of the overall success of the program, NRCan’s program team allocated about 12% of its initial operating (O&M) budget to communications.

There is evidence that the ICI-CTS program worked closely with NRCan’s Communications and Portfolio Sector (CPS) to develop and apply a front-end communications and outreach strategy for each challenge and initiative. The objectives of these communications were to promote visibility of the focus areas to potential solvers, looking to attract more non-traditional solvers and small-scale innovators than previous programs, and to gain exposure for participants and collaborators. While the specific strategy varied by focus area, there is evidence of a wide range of media and approaches used to achieve communications objectives. Each challenge and initiative worked with collaborators to promote awareness of the program. This external engagement was important to maximize reach given the nascent or disperse nature of targeted ecosystems and the objective to attract non-traditional solvers (who by definition are not part of NRCan’s existing network). Participation in major conferences and events helped garner interest, attracting attention with high-profile participants like the Prime Minister of Canada and Margaret Atwood. Focus areas also made use of PCO’s centralized website, with four of the challenges using this as a key communications platform (WICT and IODI relied more heavily on their collaborators’ websites).

Survey respondents most commonly reported hearing about an Impact Canada challenge via either an innovation, clean tech or industry network and/or from a friend, colleague or employer (see Figure 3). Similarly, the majority of finalists interviewed stressed the importance of NRCan’s collaborators and general word-of-mouth in notifying them of the opportunity to participate. These results varied significantly by challenge. For example, survey respondents for CTF were the only respondent group for whom the Impact Canada website was identified as the most common means to learn about the challenge (37.5%). WICT was the only challenge for which social media was reported to be an important source of information (30%).

Figure 3: How did the ICI-CTS’ applicants and semi-finalists hear about Impact Canada?

Text Description

A chart with horizontal bars that provides (in percentage) the value of each source through which surveyed participants first heard about the challenge or initiative.

How did you or your organization first hear about this challenge or initiative? Select all that apply.

Radio

0.0 %

Innovation, clean technology or industry events

4.1 %

Traditional news media

4.1 %

Other

8.2 %

Government organization

8.2 %

Academic organization

8.2 %

Impact Canada website

8.2 %

Referred by another funding program

10.2 %

NRCan website or communication

12.2 %

Contacted by an existing or potential partner

12.2 %

Social media

14.3 %

Website of a collaborator in program delivery

16.3 %

Friend, colleague or employer

20.4 %

Innovation, clean technology or industry network

20.4 %

Interviewees generally agreed that these communications were effective in attracting “enough” of the “right people” to apply (based on the number of quality proposals). Regardless, given its importance to challenge delivery, program staff perceived that this could have been better resourced (e.g., a dedicated communications resource within the OERD team and/or more support from CPS). In their view, capacity within NRCan’s CPS following the launch of focus areas was a major factor limiting the program’s delivery of communication products and activities. Given the importance of community support to the IODI, some Clean Energy Champions also suggested that culturally appropriate (or translated) speaking points may have aided in explaining this initiative to their community. Otherwise, NRCan’s approach to the IODI was reported to be appropriately adjusted for Indigenous participants with consideration for culturally appropriate communications.

We also examined the design and delivery of communications that would give profile to the solvers and the issues addressed by their solutions as they progress through the challenge, gaining exposure that could help attract potential collaborators or investors. Survey responses suggest that outreach and communications was a motivational factor in some respondents’ decision to participate in the ICI-CTS, though by far not the most important (ranked 10 out of 15 choices). Again, there is variability among focus areas, with the importance of exposure ranked lower as a motivation for participation in the IODI.

There is evidence of some strategies in place for communications to promote solvers. To date, participant-generated content (e.g., presentations at pitch events and showcase videos to be shared on social media) has been the most important feature. In 2020, a social media strategy was also developed aimed at ramping up the promotion of finalists as challenges move towards the selection of winners. However, with exception of WICT (where communications are managed by MaRS), finalists were generally less positive about communications on program progress than for program launch noting that there was less opportunity or emphasis on this than they expected when they first applied to participate. While noting that delivery of communications benefitted from the collaboration of PCO, some interviewees perceived this to have made the communications process more bureaucratic and less in line with innovative communications observed in benchmark challenges. Interviewees also suggested that the PCO platform could have been used more effectively and updated more frequently to better promote the challenges.

From March 2020 to date, there have been limitations on PCO’s communications and for publication on NRCan’s social media due to COVID-19. Conference participation and other planned events have also been reduced and shifted to a virtual context. Of the six focus areas, CTF was the last to be launched in October 2019. COVID-19 was thus perceived to have the greatest impact on communications for this challenge, delaying the announcement of finalists for a couple of months. Finalists noted that this was not optimal as it removed some of the “bang” from the announcement and impeded their ability to advertise this information to attract collaborators and investors at a critical stage.

Collaborative Arrangements

Literature indicates that collaborating with various players in the innovation ecosystem to co-develop and deliver activities in challenge models can spark breakthrough innovation and solve difficult technical challenges faster and at lower total cost. To this end, Impact Canada supports the ability of federal organizations to collaborate with expert intermediaries (and/or an international partner in the case of Power Forward) to build the capacity of innovators and communities, to support challenge delivery, or to deliver challenges (or specific components).

All of the ICI-CTS’ focus areas engaged collaborators, either formally or informally, to co-develop and deliver activities. The number and role of collaborators varied significantly by challenge and initiative, activities supported can generally be broken into five broad categories:

  1. Outreach. All focus areas worked with one or more collaborators to deliver strategies to increase NRCan’s reach in attracting proponents. This was particularly important given the nascent and/or fragmented nature of the related innovation ecosystems. Collaborators were reported to play an important role in raising awareness of the challenges for applicants. Overall, 16.3% of survey respondents identified “the website of a collaborator in program delivery” as a key means by which they first heard about the challenge.
  2. Networking and Ecosystem Development. Many focus areas worked with one or more collaborators to facilitate proponents’ partnership building efforts, including where the development of consortiums was required pre-application. This support was particularly important for Power Forward given the tight timelines (5 months) available to build international partnerships pre-application.
  3. Facilitating Scale-up of Solutions. Many focus areas engaged collaborators to provide services that would assist proponents in scaling-up their ideas by building capacity for commercialization and business development and/or by identifying and helping to remove related barriers. For example, NRCan engaged the Pembina Institute on the IODI to provide policy support intended to address barriers to community-driven clean energy projects both for participating communities and others who may later face similar barriers. Similarly, NRCan engaged the Centre for Mining Innovation (CEMI) on Crush It! to undertake outreach and engagement that would enhance Canada’s mineral processing innovation ecosystem by mobilizing new and existing solver communities, building new linkages across the innovation landscape, and setting the stage for the commercialization and adoption of emerging solutions. To enhance diversity of problem solvers, this included offering commercialization and business development services to support scaling-up the ideas of clean tech innovators in mineral processing to clients that were not participants of Crush It!
  4. Delivery Agent. Two focus areas used collaborators as agents in delivery. The IODI is being delivered with the Indigenous Clean Energy (ICE) Social Enterprise (see Box 2). NRCan engaged MaRS, a pioneering organization in the field of business acceleration for innovators, as the lead delivery agent for WICT. NRCan gave these two collaborators important roles in delivery of the application process, including supporting development of evaluation criteria and identification of jurors, and the development and delivery of capacity building. Interviewees perceived the engagement of these non-government collaborators as delivery agents as allowing challenges to be more nimble than they otherwise may be if delivered by the federal government.
  5. Co-Delivery Agent (Partner). NRCan established a formal international partnership for co-delivery of Power Forward with the UK Department for Business, Energy and Industrial Strategy (BEIS), with responsibility for funding of proponents and provision of awards shared between the two jurisdictions.

There is evidence that NRCan selected collaborators based on the alignment of their mandate, their expertise and established credibility, and/or the reach of their existing networks. Overall, program representatives and finalists perceived that the collaborators engaged to support delivery were both appropriate and effective. In particular, working with ICE and an all-Indigenous jury on the IODI was perceived as a positive move towards new model of collaborative (rather than Crown-led) program delivery.

Box 2: Indigenous Clean Energy Social Enterprise (ICE)

ICE is a pan-Canadian not-for-profit organization that advances Indigenous inclusion in Canada’s energy futures economy through Indigenous leadership, and broad-based collaboration with energy companies, utilities, governments, development firms, clean tech innovators, academic sector, and capital markets. Its support to the IODI includes access to its 20/20 Catalysts Program – a national clean energy capacity building program for Indigenous individuals from communities and organizations across Canada – and the ICE Network – a collaborative space for people working to advance Indigenous clean energy projects and Indigenous participation in the clean energy sector.

Flexible Funding Mechanisms

The Impact Canada Initiative’s Terms and Conditions for Grants and Contributions (Ts&Cs) are built in a flexible manner to support departments in their experimentation with prizes, challenges, micro-funding, and outcomes-based and innovative programming approaches using grants and/or contributions. Under the ICI, challenges take place within the same basic structures and rules as other funding mechanisms, but have distinct features that differentiate them from traditional programs (see Table 4). This may include instances where recipients are not known in advance and no agreement is signed with a recipient in advance of achieving the outcomes set out in the prize or challenge.

Literature indicates that there is no precise science to setting the value of a financial prize, but it should be proportionate to the investment innovators are asked to make in developing a solution as well as the potential market value of the new product or service. The prize does not need to cover the entire cost of the development but it does need to cover at least some of the risks challenge participants assume. The amount and distribution of funding devoted to the challenge model must be large and secure enough to shape priorities and influence concrete action. A stage-gate approach, incorporating shorter-term success milestones can facilitate interim progress and collaboration between solvers to develop better quality prototypes or solutions.

Table 4: Comparing Challenges to Regular Government Funding Mechanisms

  Traditional Funding Program
(Call for Proposals)
Impact Canada
(Challenge Model)
Principal Financial Instruments

Grants or Contributions.

Grants, Contributions, or a blended mix of both.

Basis of Payments

Payments based on estimated costs and activities incurred by recipients. Emphasis on paying for eligible expenditures at lowest possible cost.

Awards are determined based on the value of the prize or challenge and not related to eligible expenditures or costs incurred by recipients. Emphasis on payments linked to outcomes and/or milestones.

Approach to Assessing Potential Funding Recipients

Based on internal assessments and scoring processes by government departments.

Blended methods, including use of external juries of subject matter experts and judging processes (with a high level of autonomy). Assessments based on pre-determined and publicized outcomes criteria. Final prize may be based on a pre-determined metric or comparative assessment (e.g., best in field).  

Source: Adapted from PCO, Impact Canada Challenge Guide (2021)

We found that the design and delivery of flexible funding mechanisms under the ICI-CTS was consistent with these best practices and the principles of flexible funding under Impact Canada. With a uniform emphasis on payments based on outcomes, focus areas used a mix of micro-grants, grants and contribution agreements. These funding decisions were informed by co-design and through consultations with industry experts. For at least two challenges (PF and WICT), there is evidence that the design of funding mechanisms was also developed with consideration and comparison to other government-led funding programs and/or inspiration from benchmark accelerator programs (e.g., Cyclotron Road fellowship program at the Lawrence Berkeley National Lab in California) and signal checking against other prizes. Funding from NRCan was not sufficient to fully advance technologies, nor was it intended to be. This enhanced participants’ commitment to the success of the project as, with exception of IODI, their organization(s) still had to assume some financial risk. While projects needed to be scaled to the supports available, funding for the ICI-CTS was generally perceived to be more generous than that available from other government programs and/or to be well directed to fund a specific gap not met by other programs.

STL’s Cross-Canada Flight Challenge was the only pure prize challenge. Most other focus areas (including STL’s Green Aviation Fuels Innovation Competition) used a stage-gated approach, with each stage building on the previous and rewarding proponents for advancing their solution. Program representatives perceived this to lower the risk related to each funding agreement, as payments were split between milestones. Perhaps more importantly, the design of this staged funding gave explicit consideration to how funding could mitigate barriers to increase participation from diverse problem solvers and help them advance in this problem space, regardless of the final outcomes of each finalist. For example, the funding structure for Crush It! helped level the playing field by awarding selected small-scale innovators $10K each to help prepare and present their ideas to challenge jury. Similarly, NRCan and BEIS offered support to small and medium sized enterprises (SMEs) wishing to participate in Power Forward to enable consortium-building and project development activities, which could include feasibility studies. The WICT challenge gave finalists an annual stipend and the IODI paid Clean Energy Champions a salary to allow them to dedicate their time to their project. For IODI, staged funding was also essential to address a need within communities for up-front investment to reach the initiative’s objectives. Funding for the IODI was also adjusted during implementation in response to feedback from Champions and delivery collaborators, with a decision to issue micro-grants of $25K in Phase 2 to address the issue of funding from Phase 1 contribution agreements flowing more slowly than expected.

There is evidence that this design strategy was effective in attracting solvers; participants across all focus areas perceived financial supports as among the most important motivational factor in their decision to participate.7 This is particularly true for staged funding (including stipends and salaries), with close to 94% of survey respondents identifying this as very important or critical to their participation. By contrast, the final prize award was only identified as an important motivation by 80% of respondents. This is consistent with feedback during interviews, where finalists explained that their decision to participate relied more on how far they believed their solution could advance with the certainty of staged funding than a final prize that they were uncertain to win.

The design and delivery of the financial supports was also positively viewed by all interviewees, including finalists. In general, interviewees appreciated the flexibility that the outcomes-focused funding model gave to the proponents to determine how funds should be allocated specific to their project. This approach was also perceived to lower the reporting burden for proponents (compared to receipts-based reporting), which allowed solvers to spend less time on administration and more time on advancing their solution. However, milestone-based payments were also a new approach for both NRCan and proponents, and respondents in both groups reported some difficulty in determining how to define and report against relevant milestones. This took additional up-front effort to resolve. While the majority of interviewees were positive, there were also mixed views on the extent to which in practice the timing of milestone-based payments more effectively supported participation of SMEs; some final recipients noted that there could be more flexibility in this area to meet the particular cash flow needs of each solver. Some finalists also noted a possible barrier to participation from academia, as the financial practices of most academic institutions do not allow for payments based on milestones. The extent to which this is an issue would depend on the nature of the specific challenge.  

Across all focus areas, most finalists (particularly SMEs and start-ups) noted that projects would likely not have been able to advance or would not have advanced as rapidly without these funding supports. At the same time, we found that the pathways to scale success in all problem areas requires investment which exceeds the amounts that could be provided through the ICI-CTS’ prize awards. This is consistent with the theory of challenge-based models and program expectations, i.e., the funding provided by the ICI-CTS was not expected to be sufficient to bring technology towards full commercialization; it was intended to accelerate the process and help to leverage follow-on investments. Given resources available, the development of follow on strategies for participants to continue to build on their success after the final prize award is outside the scope of the program. To date, only WICT and IODI have secured and communicated a clear follow-on strategy. Even so, while Champions for the IODI were clear that this is not meant to be a competitive challenge, at the time of data collection there was still a lack of clarity around how the final $9M would be allocated among communities. Some interviewees perceived this to be increasing feelings of competitiveness as the IODI moved closer to the final deadline.

Finalists across all focus areas perceived the absence of clear follow-on strategies to be a gap. However, many proponents also perceived the level of funding provided to advance their idea as a “stamp of approval” in their solution by NRCan that they could use to entice further investment or collaboration into their solution, thus accelerating results. The program is also designed to increase intelligence (knowledge and application) of the state of technology and of the innovation ecosystem. Moving forward, this intelligence is available to inform perspectives for other programs and actors and may also accelerate the design or implementation of parallel initiatives in related focus areas.

Proponent Supports

Literature indicates that it is a best practice for challenges to incorporate supports that build capacity within participating organizations and mobilize further investment. A prize structure that offers non-financial support such as support and advice from experts, access to testing facilities, networking opportunities, and exposure to investors can be particularly useful for challenges that require innovators to develop new knowledge and capacities.

Similar to financial supports, we found that the design and delivery of non-financial supports under the ICI-CTS was consistent with these best practices and the broader principles of the Impact Canada Initiative. The design of these supports was tailored to the needs of each focus area, as informed by co-design and through consultations with industry experts. However, perceptions of the effectiveness of non-financial proponent supports varied by focus area (Table 5).

Survey results indicate that, where applicable, non-financial proponent supports were much less important for motivating participation than financial supports. Across all challenges, finalists indicated that the level of support required depended on the existing maturity or experience of the proponent organization and the Technology Readiness Level (TRL) of their solution (Box 3). Participants in WICT and the IODI perceived non-financial supports to be somewhat more important in motivating their participation, particularly access to experts, mentors and capacity building (and in the case of WICT, business incubation support).

Box 3: Technology Readiness Level (TRL)

Technology Readiness Level (TRL) is a measure used to assess the maturity of evolving technology during its development and in some cases during early operations. The lowest level, TRL 1, indicates that information already learned from basic scientific research is taking its first step towards a practical application. A technology at the highest level, TRL 9, has been fully incorporated into a larger system, has been proven to work smoothly, and is considered operational.

Overall, there was also a significant difference in perspective among finalists and semi-finalists; semi-finalists were generally much less satisfied with non-financial supports.8 Satisfaction in supports for networking and outreach was higher than satisfaction for capacity building and technical support. However, no respondents identified difficulties in accessing any of these non-financial proponent supports.

Table 5: Perspectives on the Effectiveness of Non-Financial Proponent Supports

  Proponent Supports Perspectives on Effectiveness
Charging the Future

Limited to communications and outreach.

Plans for promotional content on challenge progress were delayed by COVID-19. While not supporting challenge delivery, validation of final solutions by an expert panel during final project evaluation perceived as very valuable to leverage investment in solutions post-challenge, regardless of whether participants win the final prize.

Crush It!

Networking and presentation of solutions at showcase events.
Collaboration from CEMI to build the innovation ecosystem and support commercialization.

Overall, finalists suggested that NRCan could have better designed or delivered supports to align with their needs as a specific cohort of solvers.

The majority of finalists reported that showcase events underachieved; those who had existing networks in mining did not make new connections and those who did not found networking at these events to be a challenge. COVID-19 negatively impacted events hosted after March 2020.

Finalists also noted that they did not or could not access commercialization support from CEMI. However, CEMI did provide support to semi-finalists and other organizations within the mineral processing innovation ecosystem to improve broader ecosystem outcomes (in accordance with the aim of its contribution agreement). Perspectives of these organizations were not captured by the joint engagement, and could have provided additional insight into the effectiveness of CEMI’s support.

Sky’s the Limit

NRCan supported GARDN to create an online platform to facilitate networking on SAF, including a dedicated space to discuss STL.

Networking opportunities for proponents were perceived to be important to build consortia, particularly given the nascent nature of the SAF ecosystem.

Networking during the co-design phase has also led to better connections among federal players engaged in SAF. While not a direct support to participation, the federal government is now looking to identify its procurement needs for SAF, which could help incent investments and advance the commercialization of solutions post-challenge.

Power Forward

Collaboration from UK BEIS and British High Commission, MaRS, and Trade Commissioner Services (in Canada and UK) to support international consortium-building (e.g., virtual matchmaking, trade missions, market intelligence products).

Networking and presentation of solutions at showcase events.

There is evidence that consortia-building supports benefited a broad range of companies. While all agreed that supports for international engagement were valuable to mitigate barriers to participation, some finalists indicated that they did not make use of these supports.

Survey results indicate a potential gap between successful and unsuccessful applicants. When asked about barriers to participation, unsuccessful applicants indicated that they lacked awareness of networking events and/or perceived that more were required to connect companies with potential collaborators.

Women in Clean Tech

Collaboration from MaRS to provide expert advice, capacity building, and business incubation support.

Networking and presentation of solutions at showcase events.

Access to support from federal labs to test solutions.

Finalists for WICT were very positive about the business incubation and networking supports provided by MaRS.

MaRS consulted with finalists to build a customized series of skills training workshops, including technical and business-specific knowledge. However, some finalists suggested a need for more flexibility to self-select which training to attend based on pre-existing knowledge.

Mechanics of agreements with federal labs were a challenge. Finalists appreciated this support but some wanted more flexibility in use of funds

Indigenous Off-Diesel Initiative

Clean Energy Champions participate in 20/20 Catalysts, a hands-on training program.

Ongoing in-kind support from both NRCan and ICE (e.g., training and networking).

Opportunity to engage with mentors.
Work completed by the Pembina Institute to identify and help resolve policy barriers to energy projects.

All interviewees indicated that capacity building was critical, well designed and well delivered to support participation in the IODI, especially from ICE.

In-kind support for navigating government processes and networking was widely appreciated and generally perceived as effective. Clean Energy Champions indicated that webinars and other information provided to help them advance their projects was well designed and timed to proactively address their needs. At the time of data collection, they had not yet all engaged with their assigned mentors.

While the Pembina Institute’s support was positively perceived by both NRCan representatives and collaborators in the IODI, a few IODI finalists indicated that they were still not clear on this organization’s role.

As indicated in Table 5, one of the innovations being tested by the ICI-CTS was the use of federal labs to support challenge delivery (included as a support for WICT). Providing access to these resources presents an opportunity to mobilize new players in the clean tech ecosystem as federal labs have more expertise and equipment than some smaller firms or consortia may have access to on their own. For WICT, the use of labs as a design feature included implicit assumptions that: (1) finalists supported by the challenge required the support of federal labs; and (2) there was a research centre whose equipment and expertise would be a good match for each finalist. Overall, interviewees agreed that there was value to finalists in accessing federal lab support and that this helped speed up technology advancement. However, the specific mechanism used to secure and fund this support federal lab support was new for NRCan, i.e., Science and Technology Assistance for Cleantech (STAC) model (Box 4). Finalists’ projects are also very diverse, addressing solutions in very different clean tech sectors and stages of business development (TRL). Respondents reported that it was thus difficult to match and negotiate agreements with each distinct lab given Canada’s lack of centralized lab structure, with finalists perceiving that there could have been better or more coordination between program staff and federal labs in the lab matching process. There is also evidence that it took time to resolve issues related to administrative procedures and finalize funding agreements with each lab. As a result, some STAC collaborations started early and ended early in the program; whereas, others started late and will end late in the program (e.g., the finalists were announced in September 2018, but one lab agreement was not signed until February 2020).

Box 4: Science and Technology Assistance for Cleantech (STAC)

Developed as part of NRCan’s Clean Growth Program that launched in 2017, STAC is intended as a means by which the federal government can further support and accelerate outcomes for Canada’s clean tech sector. The STAC model is directed at SMEs, which typically face capacity gaps impeding their ability to advance their technologies and services to market, such as a lack of technical expertise, research infrastructure, and cash flow. STAC helps to address these gaps by allowing recipients (SMEs) to access the substantial and unique science and technology resources that exist at federal research centres. Agreements made using this collaboration model enable NRCan to transfer operational funding for the non-financial contribution of science and technology services directly to the relevant federal research centre, with the use of funds driven by the needs of the recipient and expected to be mutually beneficial to the results of the project.

We also found some evidence of a greater demand for access to federal labs. In absence of this as part of challenge design, a small number of proponents in STL and CTF independently engaged federal labs as partners to support their work on a fee-for-services basis (not supported by money from NRCan). While not an issue created by ICI-CTS, some indicated this as a potential barrier to mobilization of new talent as private industry does not always know how to access services from federal labs, or even what equipment or expertise may be available.

What We Found – Processes and Controls

Summary of Key Findings:

Overall, we found that processes and controls are in place to support compliance with relevant departmental guidance and the TB Policy on Transfer Payments. We also found that key financial operational controls have been designed and implemented and are operating effectively most of the time.

Selection Process. Across all focus areas, documentation was available to support proponent assessment in alignment with the selection and evaluation criteria. For Crush It! an opportunity was noted to improve the completeness of documentation surrounding the decision-making process that takes place through consensus meetings. In addition, there was an opportunity to improve the retention of documentation available for focus areas that used a delivery agent as part of the assessment process (i.e., WICT).

Compliance of Funding Agreements. The development of funding agreements were found to satisfy TB and NRCan policy requirements, and aligned with the PCO Ts&Cs for the ICI. The program consulted with corporate functions (i.e. Legal and NRCan’s CoE on Gs&Cs) to assist with the review of select agreements and obtain the necessary endorsement, where required.

Risk-based Accountability Structure. Impact Canada’s risk-based recipient audit plan was identified as a component of the broader OERD-level plan. Program identification of auditable agreements was found to align with NRCan policies and procedures on the recipient-audit process, with consideration to the inherent risk ratings of the agreements. The OERD risk-based audit plan identifies both active and planned recipient audits. While it uses NRCan’s Agreements Module Interface (AMI) to capture audit related information (i.e. status, findings and recommendations), the program’s process for tracking, managing and resolving issued recommendations is not clear, including how this information is shared and communicated.

Payment Compliance. We assessed whether payments were made in accordance with their respective funding agreements and in compliance with relevant sections of the Financial Administration Act (FAA). All samples selected were noted to be signed by the appropriate individual with a valid financial signing authority and adhered to the required stacking provisions.

Proactive Disclosures. For all 25 sampled agreements examined, proactive disclosures were found to be verified for accuracy prior to publishing, received approval for reporting, and were publically disclosed. However, for 12 of 25 sampled agreements, the publishing of the proactive disclosure information was not performed within its respective reporting period, as required by the TB Guideline on the Reporting of Grants and Contributions Awards.

Conflict of Interest (COI). We found that across all focus areas, considerations were made for COI among selection/evaluation committee members and/or juries. This ranged in form, from individually signed and completed COI attestation forms to the inclusion of COI consideration through Committee or Jury Terms of Reference (TOR) documents.

Selection Process

Pre-determined assessment criteria were generally found to have been finalized and communicated prior to challenge launch across all focus areas. The criteria were found to contain clear and detailed eligibility and assessment information, which was publically communicated to interested parties in advance through online publication, supported by Ministerial announcements. However, we did identify the following opportunities for improvement for focus areas that used a delivery agent as part of the assessment process:

  • For the IODI, the majority of the eligibility and assessment criteria were developed in advance of challenge launch. Selection criteria for the first phase of this Initiative, led by ICE, were found to have been finalized after launch but prior to any selection/evaluation process. The finalized assessment criteria did not deviate significantly from the eligibility criteria previously developed. Further, consistent with co-design principles, information on the nature of the eligibility criteria were shared and discussed with potential applicants through engagement sessions before they were finalized.
  • For WICT, where assessment criteria were developed by MaRS, there was an opportunity to clarify how these criteria were to be applied to applicant’s proposals as part of the assessment process.

For each focus area, we found an evaluation committee and/or Jury was established to review and assess applications. However, a complete and finalized listing of committee membership was not always available in documentation provided. Although a listing of committee members is not a requirement, it is an overall good practice as it helps to ensure information is available, documented, and complete.

We also noted that documentation was generally sufficient to satisfy selected proponent’s assessment against the selection criteria of the individual challenge or initiative; however, there were a few instances where this documentation was missing or incomplete. For Crush It!, an opportunity was noted to improve the availability of documentation surrounding the decision-making process that takes place through consensus meetings. For WICT, there was an opportunity for NRCan to improve its retention of documentation related to the process completed by MaRS as an external delivery agent.

Funding Agreement Compliance

The NRCan Center of Expertise (CoE) on Grants and Contributions (Gs&Cs) is responsible for providing guidance and advice related to Gs&Cs within the following areas: Treasury Board submissions, program, risk, and agreement management, policy and procedures, service standards, financial systems, and recipient audit and evaluation. In fulfilling its mandate, the CoE’s roles and responsibilities extend to the development of tools, guides, checklists and other sources of information to assist program managers in administering their transfer payment program in addition to providing training to all employees on the management of Gs&Cs.

NRCan has developed documentation, tools, templates and processes to assist program managers when launching new programs and drafting contribution agreements. The CoE established the NRCan Guide on Grants and Contributions, a resource that outlines policy instruments, processes, templates and tools, on-line training and expert advice. The CoE supports programs by ensuring staff have the tools, advice and direction in the management of transfer payment programs and the use of public funds. The NRCan Guide on Grants and Contributions contains several resources to help guide program managers in their development of grant and contribution agreements, including tools/templates for generating and amending grant and contribution agreements. The contribution and grant agreement generators, along with their schedules, provide users with a standard NRCan approved template for use when drafting agreements.

A sample of funding agreements were selected and reviewed in order to determine whether they were developed in accordance with TB and NRCan policies, directives and guidelines. The agreements examined included representation across the six focus areas under the ICI-CTS, and included a selection of both grant and contribution agreements. Overall, we found that the majority of selected agreements reviewed satisfied the requirements for their development. The program consulted with corporate functions (i.e. Legal and CoE) to assist with the review of select agreements and obtain the necessary endorsement, where required.

Risk-Based Accountability Structure

Recipient audits serve as component of the continuum of activities that should be used for ongoing monitoring activities by program staff. The CoE on Gs&Cs has developed an NRCan Guide on Recipient Auditing (last updated February 2020) to provide guidance to program managers on their roles and responsibilities for recipient auditing and a Recipient and Project Risk Management Model (RPRMM) in order to assess the risk of projects. According to the RPRMM, recipients having an overall high-risk level may be subject to mandatory recipient audits, with a random selection of recipient audits for recipients that have low-to-medium risk levels.

Overall, we determined that the OERD has developed and implemented a broad, branch-level risk-based recipient audit plan, which incorporates the selection of projects through the ICI-CTS and follows the CoE’s RPRMM. This plan is updated annually through a call-out exercise, whereby project managers consider factors throughout an agreement’s administration, including new or emerging risks, perceived issues, or concerns that may arise, for possible inclusion. The risk-based recipient audit plan was found to provide a snapshot of both active and upcoming audits, and, includes a selection of recipients for audit consideration identified as part of mandatory random probability sampling for moderate-to-low risk recipients.

The engagement team also confirmed that recipients selected for audit are captured within NRCan’s Agreements Module Interface (AMI), a database containing departmental information on Gs&Cs agreements. The AMI system allows users to track recipients selected for audit, the status/completion of the audit, findings, and lessons learned, however, the information captured is minimal in nature. Given that completed audits and their recommendations do not appear to be formally tracked and monitored as part of OERD’s recipient audit plan, or through other developed mechanisms, it is not clear how recommendations are managed and resolved. In the absence of a process to track and monitor audit recommendations, the Program’s ability to consider the results of past audits and apply lessons learned to future projects may be limited. In addition, while the recipient audit plan considers recipients for mandatory random probability sampling, we were not able to determine the methodology and selection process to inform the random selection of these recipients.

Payment Compliance

The engagement team reviewed a sample of 25 funding agreements across the six focus areas to determine whether funding agreements comply with relevant FAA requirements. The findings of this examination are outlined in Table 6.

Table 6: Compliance of Payments with FAA Requirements

  Examination Results Issues noted
Section 32

All 25 sampled agreements or amendments were signed by an individual with the appropriate delegated financial authority.

N/A – No issues noted.

Section 34

All 25 sampled agreements or amendments were signed by an individual with the appropriate delegated financial authority.

For one instance tested, the Section 34 certificate contained a signature that was not valid at the time of signing due to a system error. This certificate was later signed by an individual with the appropriate authority.

N/A – No issues noted.

Stacking Provisions

All 25 sampled agreements or amendments met the stacking provisions per the ICI Ts&Cs and their respective funding agreements.

N/A – No issues noted.

Section 33

All 25 sampled agreements or amendments were signed by an individual with the appropriate delegated financial authority.

  • One instance where Section 33 approval occurred after payment release.
  • Two Section 33 QA Checklists were unavailable for review due to office closures stemming from the COVID-19 pandemic.
  • General inconsistencies were noted between the completion of the Section 33 QA Checklist across paper and electronic forms.

Management letter issued to CMSS with Section 33 related findings and areas for consideration.

General inconsistencies were noted between the completion of the Section 33 Quality Assurance Checklist across paper and electronic forms; however, the NRCAN Finance group is aware of this and efforts are currently underway to enhance electronic forms.

Proactive Disclosure

The Access to Information Act (ATIP) requires the public disclosure of information related to grants and contributions with a value over $25,000 (as well as those Gs&Cs under $25,000 that are amended with a value exceeding $25,000) to be published in electronic form. The Treasury Board of Canada Secretariat (TBS) Guideline on the Reporting of Grants and Contribution Awards supports the proactive disclosure information requirements under the ATIP and provides guidelines in the identification, collection, publication and monitoring of Gs&Cs data.

Although the requirement for the Government of Canada to proactively disclose the awarding of Gs&Cs information is not new, Bill C-58, which received Royal Assent in 2019, included amendments to the ATIP  (Section 87) to require that Gs&Cs information be reported within 30 days after the reporting quarter in which the funding agreement is entered into. For all 25 sampled Gs&Cs agreements examined, proactive disclosures were found to be verified for accuracy prior to publishing, received approval for reporting, and information related to these Gs&Cs agreements were all publically disclosed. However, for 12 of 25 sampled agreements, the proactive disclosure was not performed within its respective reporting period, as required by TBS. The NRCan CoE on Gs&Cs monitors the status and timelines of Gs&Cs information to be disclosed, and has implemented a call-out process reminding Programs of upcoming cut-off dates. In instances where information is not available by the quarterly due date, it is escalated and reporting to senior management and posted in the following quarterly batch.

Conflict of Interest

The NRCan Guide to Transfer Payments contains high-level guidance related to the requirements for the selection of members responsible for the review and evaluation of applicants’ proposals. Required practices include ensuring the evaluation committee is established prior to reviewing applicant proposals, includes three or more persons, and that its members are separate and independent. The onus is on Sectors to determine whether real or perceived conflict of interest (COI) considerations exist.

Through our review of documentation of the selection/evaluation committees and jury members, evidence was available to support program consideration for COI across all focus areas. We found COI considerations were included and documented in various forms, including individual COI attestation forms, or consideration for COI through the selection/evaluation committees’ or juries’ terms of reference. Overall, it was evident that the selection/evaluation committee and juries (where applicable) included consideration for COI among its members.

What We Found – Expected and Unexpected Outcomes

Summary of Key Findings:

It is too soon to indicate whether intermediate and ultimate outcomes will be achieved. Finalists are progressing towards outcomes specific to their project and are achieving progress milestones. Semi-finalists also indicated that they continue to advance their projects at or beyond the pace they expected when they decided to apply. Most non-finalists indicated that they are continuing to advance their solutions; however, the extent to which this can be attributed to Impact Canada is unclear.

The design and delivery of all focus areas considered and worked to increase collaboration and investment, enhance public awareness, mobilize new talent, and address barriers to participation of new and/or smaller players. Calculation of results against most NRCan indicators was not possible given data available. Impact Canada uses innovative approaches that necessitate new methods and strategies for measuring program outcomes and impacts. While a notional performance framework was included with program approvals, NRCan is still working on its own and with PCO to enhance this framework and to develop challenge-specific outcomes and related methodologies. Difficulties in the design and application of performance metrics still need to be resolved. While these gaps limit what the engagement can conclude related to progress on immediate outcomes, available evidence suggests that progress has been made in each area.

We examined the extent to which the ICI-CTS is making progress towards program-level immediate outcomes identified in its draft logic model (Figure 2). These include:

  • Increased collaboration in key technology challenges and projects, as evidenced by the development of collaborative networks and partnerships.
  • Increased investment by stakeholders in the clean tech ecosystem, as evidenced by the ratio of funding leveraged.
  • Increased awareness and participation in clean tech, as evidenced by an increase in non-traditional proponents and collaborators (i.e., mobilization of “new talent”).
  • Increased public awareness of clean tech issues among Canadians, as evidenced by the reach of communications.

As described in earlier sections of this report, there is evidence that the design and delivery of each focus area considered and worked to achieve these immediate outcomes. While the calculation of results against most NRCan indicators is not possible given data available to this engagement, evidence suggests that the ICI-CTS is making progress in each area.

Performance Measurement

Impact Canada uses innovative approaches that necessitate new methods and strategies for measuring program outcomes and impacts. Literature indicates that key differences in the design of challenge-based models also extend to performance measurement. By design, challenges are outcomes based initiatives, wherein key performance indicators are the challenge assessment criteria. As a result, some of the incremental and proxy indicators that are typically used to assess progress of traditional programs are not necessary. If the result is not achieved, the funding is not awarded. 

This does not mean that additional performance metrics are not required. Consistent with its broader objective to accelerate the use of innovative and experimental approaches across the Government of Canada, the ICI seeks to understand how the challenge approach itself adds value as a public sector instrument (as compared to traditional ways of doing business). According to PCO, there is a need to generate better evidence that the perceived risks of running a challenge, and the level of investment that is required to run them, is justified by the achievement of better outcomes. Ongoing monitoring of performance against key metrics is also important to accountability, ensuring that senior management receives comprehensive updates of the program’s performance as it progresses.

In 2017, funding authorities outlined a notional performance framework for the Impact Canada Initiative as a whole, as well as specific outcomes and targets for the Clean Technology Stream. Each focus area was also to have its own unique set of goals and performance targets, with performance information to be collected on an ongoing challenge-by-challenge basis. Given the broad range of possible pathways that challenges and individual projects could take, OERD indicated that work on these methods could only start once it had a clearer indication of the specific innovation pathways and solutions to be pursued.

In 2019, PCO released a first set of guidelines for impact measurement in support of its work under Impact Canada.9 In 2020, PCO released a logic model that presents a basic theory of what Impact Canada expects to achieve with its broad suite of challenges.10 The outcomes in this logic model are consistent with the original framework approved in 2017. These include:

  1. Immediate outcomes related to the challenge process – i.e., enhanced public awareness, mobilization of new talent, and building of collaborative networks and collaborative arrangements;
  2. Intermediate outcomes related to development of sustainable business models – i.e., increased investment, enhanced skill and capacity, and development of innovative products, services, or technologies; and
  3. Ultimate outcomes linked to socioeconomic and environmental benefits for Canadians.

NRCan developed a draft logic model for the ICI-CTS in 2019-20 (Figure 2). In March 2021, OERD indicated that it still working separately and with PCO to enhance the related performance measurement strategy, including additional indicators, methods, data sources and potential counterfactuals, with an emphasis on methods for tracking results post-program (other than self-reports). OERD is also still working towards developing more challenge- and solution- specific outcomes and methodologies that will allow for more precise and sensitive monitoring of results following the announcement of grand prize awards. Currently, OERD is working with others (e.g., PCO, Statistics Canada, and the University of Alberta) to explore methodologies to measure the outcome of programs, including use of quasi-experimental methods that combine Impact Canada program data with administrative / statistical data to assess key outcomes for challenge participants against comparator groups. PCO is also collaborating with Statistics Canada to conduct a horizontal impact assessment of challenges across the federal government, using quantitative and qualitative data. However, program managers noted that overall progress in this area has been delayed due to resource constraints and COVID-19, further complicated by the lack of existing methods for effectively tracking and measuring intermediate-and long-term effects of prize-based challenges.

Within this context, program managers for the ICI-CTS are aware that numerous difficulties in the design and application of performance metrics still need to be resolved. The engagement found that these include but are not limited to:

  • Lack of Program Comparators. A key question for the ICI-CTS is whether the challenge model worked better than known alternatives. However, for most focus areas, there are no bundles of projects in core programming against which to make a relevant comparison. Where comparison may be possible, results may be skewed by the design of OERD’s core programs. For example, prior or concurrent OERD funding calls often intentionally support the same technology or proponent at different stages of innovation, and therefore may have higher results by default. Other programs (e.g., NRCan’s Energy Innovation Program) often use very targeted funding calls that may limit the pool of applicants, thus limiting their relevance as comparators for proposed metrics such as the number of applicants per dollar of NRCan funding. The ICI-CTS’ challenges also do not have the same matched funding or (with exception of consortia-based challenges) collaboration requirements as traditional programs, which could also skew comparisons.
  • Definitional Issues. Performance metrics include some basic terms that still require better definition. For example, who is included in the definition of “new talent”? Does this include just the main applicant or all project partners? In what ways does an organization need to collaborate in order to be considered a “collaborator”? Do inputs from growing consortia count as leveraged investments?
  • Expected Results Expected to Evolve. Metrics related to business model outcomes such as investment and collaborative arrangements are expected to evolve over time and are subject to increase or decrease. Baseline data on related metrics was collected with applications, but progress towards outcomes requires frequent updating throughout delivery and for an extended period post-challenge award. It may take several years to be able to detect any changes in outcomes as compared to counterfactuals. This makes it significantly more challenging to track and benchmark progress.
  • Limitations in Project Reporting.  There are ongoing reporting obligations for semi-finalists and finalists designed by the challenges. We observed that the templates for milestone reports were set up to collect data that would track progress but, for reports that were on file, the nature and quality of the information collected varied significantly – for some focus areas, reports provided few details that could be used to support performance measurement, with some sections left blank.11 Project reports may also be insufficient for collecting data on longer-term outcomes or spin-off results, particularly if these results come from solvers that were not semi-finalists or finalists. The nature of the prize-based challenges is such that OERD cannot mandate follow-on reporting as it can with traditional funded projects. Quasi-experimental tracking methods do not rely on project reports but will likely require access to business or charitable registration numbers that were not consistently collected as part of applications.
  • Gender-Based Analysis Plus (GBA+). There is no indication of gender specific metrics and data collection in place (other than as it relates to challenge eligibility for IODI and WICT). However, program staff indicated that both OERD and PCO are working with Statistics Canada to collect gender, equity, diversity and inclusion data at the firm level.
  • Measuring the Effectiveness of Failure. NRCan’s draft logic model for the ICI-CTS focuses both on the results to be achieved by each focus area and the use of innovations to develop solutions to specific problems. As with traditional programs, the theory of change presented by this logic model is oriented towards the achievement of outcomes (i.e., success). However, in a challenge model, failure to arrive at a solution and award the final prize may also be a perfectly acceptable experimental outcome. How can challenge-based models best capture the effectiveness of this failure?

While NRCan has collected some data as part of applications and project reports, as noted above, the nature and level of detail of this data is inconsistent across focus areas and contains information gaps. Raw data has not been subject to analysis to date. These gaps limit what the engagement can conclude related to progress on immediate outcomes.

Increased Collaboration

Literature suggests that a key benefit of challenge models derives from strengthening the problem-solving community to encourage closer collaboration and long-term innovation. The ability to increase collaborative networks and partnerships in the program’s focus areas was identified as critical given the nascent state of the related innovation ecosystems in Canada, and the ICI-CTS created opportunities to build networks and partnerships among their communities of solvers (and in some cases end users such as commercial airlines) as an integral part of the process.

In the ICI-CTS’ notional performance framework (2017), NRCan targeted year-over-year progressive increase in the number of partnerships developed and an increase in the number of partners per dollar spent than comparable bundles of projects under core programming for each challenge. Calculation of results against these indicators is not possible given data available. However, there is strong evidence to suggest that finalists are developing their own formal and informal partnerships among diverse stakeholders. The extent and scope of these collaborative arrangements varies by challenge and by project.

Many finalists highlighted that they established new relationships across the ecosystem that would not have occurred or would have occurred more slowly in the absence of the ICI-CTS. Factors contributing to their success in this outcome area included networking supports and increased visibility. Finalists in consortia-based challenges (PF and STL) indicated that the requirement to form collaborative arrangements was beneficial but also presents some risks (e.g., ongoing partner commitment). Finalists in WICT and the IODI also noted the importance of support for networking among peers in their cohort.

NRCan also undertook significant efforts to identify, engage and connect players during the co-design and application phases. For example, data indicates that semi-finalists in Power Forward collectively established approximately 186 new connections (prospective partners) as a result of networking events in January 2019 and 70 new connections in June 2019. However, in contrast to results for finalists, less than half of all non-finalists surveyed indicated their collaborative networks or partnerships increased as a result of participation (Figure 4). Of survey respondents who indicated that they had formed new connections (n=28), only 13.2% indicated that NRCan or a collaborator had played an important role in facilitating these connections. These results were more or less consistent across focus areas.

Figure 4: Change in collaboration resulting from ICI-CTS reported by survey respondents

Text Description

Figure 4: Change in collaboration resulting from ICI-CTS reported by survey respondents

Strengthened existing partnership(s)

12.8 %

New partnership(s) within own sector

19.1 %

New partnership(s) with other sectors

10.6 %

Partnerhip(s) have not been affected

38.3 %

Strengthened existing network(s)

8.5 %

Expanded network(s) within sector

14.9 %

Expanded network(s) with other sectors

27.7 %

Collaborative network(s) have not been affected

42.6 %

For the purpose of performance measurement, NRCan defined “partners” as project participants other than NRCan or the applicant organization. However, many challenges also engaged collaborators to continue to identify and build ecosystem connections throughout challenge implementation. This includes NRCan’s support to GARDN to create an online platform to facilitate networking on sustainable aviation fuel, and to CEMI to build new linkages across Canada’s mineral processing innovation ecosystem. The result is better mapping or networking of the innovation ecosystem for these focus areas. There is also evidence that external collaborators and federal players are now better connected. These connections can now be leveraged by NRCan and other organizations to advance future initiatives.

Increased Investment

As noted earlier in this report, funding from NRCan through the ICI-CTS was not sufficient to fully advance technologies, nor was it intended to be. Further investment in clean tech solutions is required to continue to advance solutions, regardless of the final prize. Literature indicates that challenges should incorporate supports that mobilize further investment and generate new opportunities for solvers. The design of the ICI-CTS’ focus areas incorporated opportunities to connect and build relationships with investors (supports varied by challenge).

In the ICI-CTS’ notional performance framework (2017), NRCan expected NRCan program funding for each challenge to be leveraged by participants at a ratio of more than 1:1 (including financial and in-kind investments) and an overall ratio of investments higher than that observed in core programming. However, none of the focus areas included a requirement to obtain matched or leveraged funding as a criterion for application. Rather, consistent with literature on prize-based challenges, NRCan anticipates that participants will accrue investments and in-kind support as projects advance, via networking and collaboration events, increased publicity, etc. While still supportive of Indigenous communities sourcing additional investments, NRCan has also explicitly excluded the IODI from its targets in this area.

Again, calculation of results against these indicators is not possible given data available. However, there is evidence indicating that most finalists’ projects have leveraged such investments, at least to some extent. For example, one finalist reported having raised over $5M from government and private sector sources since joining WICT. Evidence suggests that participants in PF and STL had generally made the least progress in this area, with finalists indicating that it was a challenge to attract investment in these areas at the technology’s current level of maturity (TRL) or given other market and regulatory barriers. However, further leveraging across all focus areas could reasonably be expected as projects continue to advance.

The nature of a challenge is also such that the majority will not win. Funding under the ICI-CTS was available to support a limited number of participants. Program representatives indicated that they forwarded the more promising proposals of some non-finalists to other potential funders, such as via the Clean Growth Hub.12 Among all non-finalists and semi-finalists surveyed, only 7.8% indicated that their proposal was referred to another program or initiative. Nevertheless, among respondents indicating that they had continued to advance their project, 55.2% indicated that they had secured new sources of investment (financial or in-kind) since applying for the challenge.

Most non-finalists and finalists could not directly attribute increased project investment to Impact Canada (e.g., some had previously or concurrently applied to other funding programs) but finalists acknowledged that their success in this area was facilitated by the reputational “stamp of approval” gained by participation in the program. Finalists perceived being selected among finalists as validation of project potential that serves as a positive signal for investors and customers, particularly for prospective investors that were not familiar with the innovative technology being advanced by the project.

Mobilization of New Talent

Literature indicates challenges are strategically generally run in an open and transparent manner to encourage broad participation among diverse actors, in order to produce a wide range of potential solutions. Inclusive innovation invites diverse groups of people outside the existing or established innovation ecosystem to participate in solving public-facing problems, on the assumption that this will also bring new perspectives, skills, ideas, resources, etc. Extra provisions are required to draw out and engage traditionally underrepresented solvers. Challenges feature low barriers to entry to help entice entrants with lower capacity or those who may otherwise be uninterested in the burden of traditional application and reporting processes associated with many government grant and contribution programs.

There is evidence that the design and delivery of all focus areas considered and worked to attract new talent and address barriers to participation of new and/or smaller players, including in some cases a specific focus on underrepresented groups (e.g., women and Indigenous people). NRCan engaged with collaborators to facilitate this outcome, many of whom were also new to collaboration with NRCan or have taken on new or expanded roles in supporting the innovation ecosystem as a result of the ICI-CTS.

Beyond involving a range of stakeholders in program design, innovations in the application process were perceived by program representatives and finalists as facilitating the participation of new talent. Many smaller organizations and underrepresented groups can experience barriers to application writing, such as a lack of experience or human resource capacity given the time burden required to develop detailed proposals. Indigenous communities may face additional barriers for applications such as internet bandwidth (particularly in remote areas), language barriers, and competing priorities during culturally important periods of the year (hunting, fishing, and celebrations). Timing of application may also be important for some female entrepreneurs, where intake periods can compete with priorities such as school holidays. Overall, there was clear design consideration for how application requirements could be streamlined to reduce burden for applicants, with the application process tailored to the specific design and target audience of each focus area. All applicants had access to information (e.g., process maps and applicant guides) and support as they worked on their applications to ensure project readiness. Selection processes for many focus areas also included features such as applicant interviews and pitch events where semi-finalists or a short-list of applicants had the opportunity to directly explain their proposal to the jury or selection panel.

The majority of proponents reported a positive impression of the application process, but satisfaction of finalists and non-finalists differed for some aspects of the process (Table 7).

Table 7: Comparison of Finalists’ and Non-Finalists’ Perspective on the Innovative Application Process

  Finalists’ Perspectives Non-Finalists’ Perspectives
Clarity of Application Requirements

Vast majority satisfied. General appreciation that the outcomes-focus was flexible to accepting more innovative solutions and did not give preference to specific technologies.

Majority satisfied but a small number indicated that what to expect in the application process could have been better communicated

Ease of Application Process

Majority found ease of application easier or less burdensome than previous experiences with federal funding.

Majority found the ease of application to be “about the same” as previous experiences with federal funding. Given the innovative nature of the new technologies being advanced, some wanted more flexibility in the application form to fully explain their idea.

Timing of Application Process

Majority satisfied that they had adequate time to complete initial application but timelines to develop pitch presentation were tight for some finalists.

Majority satisfied that they had adequate time to complete initial application.

Supports for Application Process

Finalists across all focus areas commended the support that was available from NRCan (or MaRS) to complete the application.

Mostly neutral on the utility of supports. Webinars and information sessions reported as most useful for challenges; e-mail or telephone support reported as most useful for the IODI.

Evaluation of Proposals

Majority satisfied. Criteria drove some finalists to define aspects of their solution that had not been previously considered.

Only 54.2% satisfied with the clarity of evaluation criteria. Many wanted more feedback from the evaluation committee or jury.

Both finalists and non-finalists noted that the cost to putting together an application makes it important to ensure that the application process is scaled to reflect (1) the value of each challenge’s financial supports and (2) the likelihood of being selected as a semi-finalist or finalist. While application was generally recognized to be less burdensome than other funding programs, some respondents indicated that the level of effort required to complete an application under the ICI-CTS (including pitch presentation where applicable) may have still posed a barrier for applicants who are unable to commit significant time or funding to the application process. This may be a particular barrier for early stage applicants, i.e., those with less experience or a lower level of base funding. Three focus areas did provide funds to help capacity stretched organizations (SMEs, small-scale innovators and Indigenous communities) prepare their proposal. Respondents suggested that this sort of support could have been extended to more challenges or participants, or be made available earlier in the application process, but this may not always be appropriate given types of solvers or feasible given timelines. Some also suggested that the ICI-CTS could have mitigated barriers by providing more information on what to expect in the application process, e.g., improved application templates (including examples of information required).

Respondents for the IODI (participants and non-participants) did not perceive any significant barriers to the application process, other than in some cases the availability of their own time. Interviewees identified a capacity issue with not having enough qualified people in their communities to focus or advise on completing applications, which means that they often have to rely on external consultants to complete funding applications. They perceived that these difficulties were largely addressed by the simplicity of the application process and support from program staff for the IODI.

In the ICI-CTS’ notional performance framework (2017), NRCan expected that increased awareness and participation in clean tech would be demonstrated by a achieving a higher number of applicants per dollar of NRCan funding and a higher proportion of “unconventional” applicants than for comparable funding calls under core programming in the same technology areas. Unlike increased collaboration and increased investment, the program did not expect to achieve its targets for the mobilization of new talent until 2028. Our engagement found early evidence that new talent has been mobilized by the ICI-CTS but related results are currently difficult to quantify.

NRCan has defined “unconventional applicants” as those who do not typically apply for NRCan clean tech programs. The vast majority of finalists interviewed were able to compare the application against their experience with other federal programming, which indicates that these finalists are not new to federal funding programs. Similarly, only 16.7% of surveys respondents applying on challenges indicated that they had not previously applied for funding from the Government of Canada. However, that they applied does not mean that they received funding. Previous funding experience may have also been for a different solution or in a different sector.

More importantly, we found the ICI-CTS’ definition of new talent may be too narrow. There are many other ways that “new talent” and “non-traditional proponents” can be defined. Interviews and survey data provide some proxy indicators on the extent to which new talent has been mobilized to date in other ways:

  • New Field or Sector. In total, 20.4% of survey respondents indicated that they had applied to participate in a focus area that represented a new field or sector for them or their organization. All of these responses were related to CI! and WICT. By contrast, the majority of respondents for STL and PF indicated that their organization was already working on a solution to the issue addressed by the challenge at the time of application. These results are similar to responses provided by finalists during interviews. However, while some interviewees for STL had previous experience working with biofuels, they clarified that the challenge has pushed them to consider more seriously SAF as part of their business. Similarly, Power Forward had a focus on systems integration, meaning that by definition the solutions would already need to be in progress.
  • New / Smaller Players. A large number of finalist organizations were SMEs or start-up businesses. Statistics Canada’s Survey on Financing and Growth of Small and Medium Enterprises (2017) offers the first-ever snapshot of the financing and growth activities of clean tech small and medium-sized enterprises (SMEs) in Canada.13 Results reveal that the profile of SMEs operating in the clean tech sector were larger and older than all SMEs, were more likely to seek financing, and their financing requests were generally approved. This may also explain why so many applicants and finalists were not new to federal funding programs, and supports the idea that this narrow definition of new talent may not be the best indicator.
  • Underrepresented Groups. Two focus areas were specifically designed to support outcomes for underrepresented groups; the processes and capacity support built into WICT and IODI helped attract and maximize the participation of women and Indigenous communities. Evidence indicates that they have both been successful in mobilizing new talent. Efforts to spotlight these participants’ clean tech and clean energy projects are designed to further inspire participation within these target groups.
    More generally, data from Statistics Canada’s survey indicates that ownership of clean tech SMEs tended to be less diverse than that of all SMEs. In 2017, this survey found that clean tech SMEs tended to be majority owned by men (81%), compared with 64% of all SMEs. Clean tech SMEs were also less likely to be majority owned by visible minorities, Indigenous peoples and people with a disability. Similarly, 18% of clean tech SMEs had primary decision makers who were born outside of Canada, compared with 25% of all SMEs. The flexible and inclusive design of challenges under the ICI-CTS provide a greater opportunity to address gaps from the broader lens of reconciliation, equity, diversity and inclusion (REDI). However, as noted above the program has not systematically collected data that would enable analysis of GBA+ consideration. Related data collected by our engagement is also limited; it provides a profile for survey and interview respondents but not the organization to which these individuals belong.

Enhanced Public Awareness

Enhanced awareness among communities of potential solvers is a critical early success factor for challenges, important to both attracting participants and influencing market demand. Over time, challenges can also have the effect of communicating government priorities and better engaging the general public on the broader policy issue(s) at hand. This can help generate greater public demand or social licence for the challenge host to more effectively deliver on its core mandate in the future.

Since 2017, the program’s social media strategy has focused on the promotion of specific milestones in the challenges (e.g., challenge launch, announcing finalists, challenge deadlines, etc.) and the ad hoc amplification of challenge finalists’ content. Finalists have been invited to proactively share links to online content showcasing relevant activities (e.g., project progress) that NRCan could echo on the PCO website and its own social media platforms (Twitter, Facebook, and LinkedIn). Where applicable, videos included in posts were also submitted separately as deliverables required by the finalists’ contribution agreements (mid-project videos to report on progress). In 2020, NRCan developed a further social media strategy aimed at ramping up the promotion of challenges and finalists as they move towards the selection of winners.

In the ICI-CTS’ notional performance framework (2017), NRCan planned to measure increased public awareness of clean tech issues among Canadians by the year-over-year progressive increase in social media by channel. Program managers indicated that while this indicator may make sense at the PCO level, this is not a realistic expectation for the ICI-CTS. This program’s focus areas will have natural increases and decreases in attention related to major announcements.

There was no data provided to the engagement that would indicates a year-over-year trend in public awareness. However, there is some evidence of data being collected. In 2020, NRCan’s CPS produced a social media analytics report for the program, which indicates the reach of content across focus areas and social media platforms. This analysis shows that posts related the ICI-CTS performed relatively well in terms of both reach and engagement rate, particularly when compared to the average for government content (Table 8). The results specific to each focus area differ, with the highest engagement rates for WICT and STL and the lowest engagement rates for Crush It! (perhaps due to the very technical nature of solutions). However, the results provided in the report do not cover communications beyond NRCan’s accounts, which may have been extensive. While data is not available across all focus areas, data reported by MaRS indicates significant reach for some communications (e.g., 37,000 social media impressions related to posts for PF and 670,000 related to WICT). One finalist for Power Forward also reported 7,000+ views on their recent video post on Facebook.

Table 8: Cumulative Reach of NRCan Social Media Communications, May 2018-April 2021 (All Challenges)

Media   Number of Posts Impressions per Post (Reach) Engagement Rate (%)

Twitter

English

244

1,114,114

1.02

French

244

155,451

0.93

Facebook

English

40

630,229

0.16

French

37

142,393

0.03

LinkedIn

English

24

766

1.06

French

8

42

0.37

Note: Impressions per post indicate the number of times a post is viewed. The engagement rate represents the percent of people reached who found the post relevant enough to like, share, comment, etc. An engagement rate above 0.9% is considered good for organic government social media content.

Source: Impact Canada Social Media Analytics Report (NRCan CPS)

There is also anecdotal evidence of traditional media coverage and interest in NRCan’s challenges and finalists’ projects but a detailed media analysis was not completed by the engagement.

Unexpected Outcomes

Our engagement also sought to identify whether any additional or unexpected outcomes (positive or negative) have resulted from the ICI-CTS. For the purposes of our engagement, this would include any results other than those targeted in the performance framework. Our analysis in this area is somewhat limited as it is challenging to identify unexpected outcomes when the ICI-CTS’ performance measurement framework is still in development, meaning that all expected outcomes have not been clearly identified and documented.

Based on evidence available, the engagement did not identify any negative unexpected results from the ICI-CTS. However, we did note a few positive achievements. For example:

  • Across all focus areas, semi-finalists and finalists indicated that their projects were generally advancing faster than they expected when they decided to apply. While not entirely unexpected, several non-finalists also indicated that the challenge had motivated them to better organize or advance their proposed solution; two of these individuals indicated that they had since received funding support through programs in other jurisdictions.
  • For WICT, the “cohort effect” was highlighted across interviews as a positive unexpected outcome. The cohort model of the WICT challenge has helped the six finalists develop a mutually supportive peer-to-peer network, despite being engaged in a prize competition. This unexpected outcome was perceived to have accelerated the pace at which the finalists’ projects are progressing.
  • Particularly for the IODI and WICT, interviewees identified personal benefits of participation, e.g., that participation has helped boost their confidence, leadership abilities, and provided new skills that they can now expand to opportunities in other areas.
  • Lab representatives indicated having benefited from the opportunity to learn about the current state of research across a range of clean tech projects and where they are best placed to support further development.

Our engagement was not designed to assess progress towards intermediate and ultimate outcomes, as it is too soon to indicate whether these will be achieved. Nevertheless, we found progress at least insofar as finalists are progressing towards outcomes specific to their project; projects are achieving expected project milestones. The key exception is STL’s Cross-Canada Flight Challenge, where the challenge has closed without the pure prize being awarded as no producer met all the criteria before the submission deadline (January 1, 2021). However, this should be viewed as a lesson learned rather than a failure on the part of the program (see Box 5). Across all focus areas, finalists indicated that Impact Canada had contributed to the advancement of their project. The majority indicated that they would not have been able to move their solution forward without this support. While about 20% of finalists interviewed for any given focus area indicated that they would still have advanced their projects in the absence of the ICI-CTS, they nevertheless believed that the pace of advancement would have been slower without this support.

Box 5: Lessons Learned from the STL Cross-Canada Flight Challenge

The STL Cross-Canada Flight Challenge was the only challenge to use a pure prize. While the prize for this challenge was not awarded, NRCan learned important lessons about how to design and deliver challenges using this model.

NRCan was aware that there was a high risk it would not award the final prize. The prize value of $1M was determined in co-design to be substantive enough to incent participation while still limiting the impact of failure on achievement of program outcomes (represents less than 2% of the total budget for ICI-CTS). Impact on outcomes was also limited by maintaining separation between the pure prize and STL’s Green Aviation Fuels Innovation Competition.

The absence of a winner for this component also confirmed that the SAF ecosystem was not ready at that time to support production of “made-in Canada” sustainable aviation fuel (particularly given impact of COVID-19). Success may have been achieved had the challenge left an open end date for submissions, but NRCan’s parameters were bound by the program’s time-limited funding profile.

Marking a key difference from traditional programs, the results logic for the ICI-CTS suggests that progress towards outcomes should also be observed in solvers who are not selected as finalists, albeit to a more limited extent. The nature of a challenge is such that the majority will not win the grand prize, however the challenge process might generate new opportunities for the solvers that would not have otherwise transpired. Non-winners may develop fruitful partnerships, promising prototypes, new business processes, or attract other investments in their activities that all go on to improve socioeconomic or environmental outcomes in their own ways. This is all part of the impact of a well-designed challenge. In response to the engagement’s survey, the majority of semi-finalists and unsuccessful applicants indicated that they are continuing to advance their project (60.5%) or different opportunities in a related area (19.8%). Semi-finalists in this group indicated that they continue to advance at or beyond the pace they expected to achieve when they decided to apply. The extent to which these results can be attributed to Impact Canada is unclear given available evidence.

What We Found – Factors Influencing Program Delivery

Summary of Key Findings:
There are a number of internal and external factors influencing program delivery. Within its sphere of influence, the program has been responsive to mitigate or address barriers to delivery.

We examined the extent to which the ICI-CTS appropriately considers the internal and external factors that could impact its ability to effectively or efficiently achieve its intended outputs and outcomes. We found that the program has identified and developed approaches to respond to internal and external factors that impact (or are expected to impact) its delivery. These factors, together with additional factors identified by our engagement, are outlined in Table 9. Some of these factors, such as increased flexibility in decision-making, had a positive influence on implementation. The use of a program stream (versus individual challenges) enabled NRCan to build internal capacity, including a team of dedicated human resources, and provided more flexibility to adapt along the way. Other factors, such as COVID-19, had a more negative influence. We found that within its sphere of influence, the program has been responsive to mitigate or address barriers to delivery.

Table 9: Factors Influencing Program Delivery

Internal Factors

 

 

 

 

 

 

Novelty of Challenge Model. NRCan (and PCO) learned as it progressed in the implementation of this innovative program. As a result, some key elements took longer than expected to design and deliver, e.g., the negotiation of collaborative arrangements and milestones. Related delays posed difficulties given the tight timelines for each focus area. This was mitigated to the extent possible by early engagement with key internal players (e.g., NRCan’s corporate services), as well as being informed by lessons learned from benchmark challenge and business incubation models. Latter challenges such as CTF were able to advance more quickly based on lessons learned from earlier challenges.

Complex Internal Collaboration and Decision-making. The ICI-CTS required broad involvement of decision makers within NRCan and across central agencies. This led to some internal challenges around interpretation, accountability and communication. For example, as discussed under Governance, program representatives indicated that the unfamiliar policy context resulted in fundamentally different interpretations of the ICI-CTS Ts&Cs among key intradepartmental players. In addition, it was not always clear whether corporate feedback was required for approvals or was advice for consideration. The relationship with PCO under Impact Canada umbrella was helpful to advance use of flexibilities but was also an added layer of administration.

Cultural Barriers. While OERD was advised by PCO to interpret the Ts&Cs as needed, this was difficult in practice as corporate services (e.g., legal, CoE) are inherently more risk averse. This was mitigated by early engagement to ensure that corporate services understood the program’s policy context and the expectations of executive management. This resulted in some important learning by doing that can now be applied to future programs (within NRCan and Government of Canada as a whole).

Flexibility in Design and Delivery. Flexibility in the program’s Ts&Cs allowed the ICI-CTS to work in ways that would not be allowed in more traditional programs. Encouraged to be innovative, OERD also looked for opportunities and took some calculated risks that resulted in valuable mid-course corrections after the challenges had already launched (e.g., by adding a sixth finalist for WICT and adding a micro-grant for the IODI as a bridge between phases). 

Relationship with Collaborators. Engagement of collaborators as the lead for planning and delivery on large objectives helped accelerate results across focus areas. Use of collaborators allowed NRCan to tap into their specialized support, skills and networks but required NRCan to release some control of decision-making. Any related issues were mitigated by good communications.
Internal Resourcing. With plans to leverage existing governance and program capacity, NRCan committed up to 10 FTEs to the ICI-CTS, including a program manager responsible for delivery of the program and one lead per challenge (a best practice). The actual allocation varied by fiscal year, with the most FTEs allocated in 2019-20 (13.9 FTEs). However, challenges take more time to develop and implement than a traditional Gs&Cs program, and the administrative hurdles faced by the ICI-CTS drew heavily on internal resources. In addition, some positions envisioned in initial funding documents (e.g., dedicated communications and evaluation support) were never resourced. These difficulties created a very intensive staff workload that likely contributed to some of the gaps identified by the joint engagement.
Quality of Staff. Across all focus areas, finalists acknowledged NRCan ICI-CTS team as being exceptionally responsive and collaborative, actively looking for solutions rather than reiterating program rules. This was perceived to be critical in facilitating the challenge delivery process, helping to mitigate barriers to participation or progress.

External Factors

 

 

 

COVID-19. Health restrictions, economic turmoil and other uncertainty stemming from COVID-19 created unexpected difficulties for proponents that impacted their progress – e.g., adjustments to remote work, unequal access to labs, issues with cash flow and supply chains, loss of external partners or investment, cancellation or delay of events and promotional communications, etc. NRCan (and MaRS) took steps to understand the issues and identify the best possible mitigations. Box 1 identifies the concrete actions that were taken to support proponents, but the ongoing engagement and support that program representatives provided (and continue to provide) to each proponent to respond to their unique COVID-19 challenges is not easily captured. These decisions and changes to the program were accompanied by hours of ongoing discussions with proponents to understand the ever-changing dynamic of the pandemic. The supports, that needed to be “right fitted”, were realized through professional and compassionate engagement during a difficult time. While this support and resulting flexibilities were welcomed, finalists perceived slow decision-making to create uncertainties with impacts on project outcomes or project costs.

Sector-specific barriers. The nature of the innovation ecosystem influences results for participation and investment by stakeholders or ability to extend results. It is difficult for nascent industries to align with market demands for the product (e.g., SAF and very novel innovations), and there may be other technological, regulatory or market barriers to overcome. For example, the battery industry, particularly related to transportation, can be guarded on innovation to maintain competitive advantage. The mining industry is relatively conservative and so may favour known players. Though NRCan considered these barriers, not all could be resolved by challenge design.

Decisions of External Parties. NRCan can work to influence but cannot control the decisions of external parties. For example, as a joint Canada-UK challenge, Power Forward required that decision-making be shared to develop a challenge that would fit the needs and capacities of both jurisdictions. Alberta Innovates launched a parallel SAF challenge in 2019 that could have further accelerated results for STL, but this was subsequently discontinued. The success of the IODI relies on ongoing support from participating communities.

Motivation of Applicants. Numerous external factors influenced motivation to apply and participate in Impact Canada. Overall, besides access to financial supports, the engagement’s survey results indicate that challenge proponents were the most motivated to participate by the chance to advance their technology, including its commercialization, followed by the alignment of the challenge with personal or organizations interests. Most proponents indicated that they would not have participated without these incentives. For the IODI, all survey respondents indicated that they were the most motivated by potential community benefits. Indirect community benefits – e.g., job creation and health benefits – were reported to be more important than energy security. The opportunity to improve environmental outcomes was also ranked by most respondents as being very important.14 This is consistent with feedback from finalists during interviews. The program responded by designing challenges in outcome areas and with financial and non-financial supports that would be attractive to applicants and mitigate barriers to participation.

What We Found – Lessons Learned

Summary of Key Findings:

Perspectives on challenge model. Interviewees perceived the challenge model as effective and appropriate, with the exception of IODI (where a competitive model was found to be culturally inappropriate). There was no consensus on whether this model should be replicated in the future. Decisions on when to use this model should carefully consider the context and outcomes to be achieved.

Challenge guidance. At the outset, there was limited guidance from the Government of Canada to inform design and implementation. NRCan has identified systemic and departmental barriers to the use of flexible mechanisms. These lessons learned have already fed into guidance released by PCO and should help accelerate future use of related models by NRCan and bring new perspectives to corporate practices.

Building on progress. The issues targeted by each challenge will not be fully addressed by the ICI-CTS, nor are they intended to be. There is no one, “silver bullet” solution to these complex problems. However, the ICI-CTS is meant to help advance and accelerate solutions. Interviewees indicated ongoing programming is necessary to maintain momentum and promote or accelerate adoption of solutions. Specific proposals varied by challenge but examples included further funding, support for market development, policy and regulatory changes.

Replicating success. The challenge model may not be the only design element that NRCan would want to replicate; NRCan has gained a lot of experimental learning by testing additional innovations in each focus area. For example, the co-design process added significant value to target key outcomes, including consideration of specific barriers for the problem area. Programs and supports for new talent are important to increase participation of new solvers (e.g., SMEs) and remove barriers for underrepresented groups. Other best practices are highlighted throughout this report. Future use of these mechanisms need not be limited to challenge models.

What are the conditions required to accelerate the use of the Impact Canada – Clean Technology Stream’s innovative approaches to future interventions? Our engagement sought to identify best practices and lessons learned through the design and delivery of the program’s prize challenges and experimental design features that could support the positive evolution of government systems to enable greater innovation. There are various lessons learned highlighted throughout this report. This section focuses on those lessons that are either overarching or not covered in previous sections.

Perspectives on Challenge Model

Interviewees generally perceived the challenge model as effective and appropriate given the objectives of the specific challenges that were launched under the ICI-CTS, with the exception of the IODI (where design ultimately recognized that a prize competition was inappropriate). Besides the flexibility, visibility and capacity boost provided by financial and non-financial supports, finalists expressed strong appreciation for the accelerated timelines for decision-making that were a feature of challenge delivery. However, there was no consensus on whether this model should be replicated in the future. It is a new tool that provides an alternative to traditional funding programs, but decisions on when it is appropriate to use this model should be carefully considered in light of the context of the outcomes to be achieved. PCO’s Impact Canada Challenge Guide (2021) highlights that a challenge may not be the optimal instrument choice if:

  • The solution to the problem is evident, currently in the market, and/or the best choice for the solution provider is clear;
  • The primary goal is to provide core funding or ongoing support to build capacity to meet program objectives over a longer term;
  • The ultimate objective is to procure a solution exclusively for the Government of Canada’s own use;
  • The subject matter is particularly sensitive for stakeholders and an element of competition would detract from solving the problem; or
  • There are regulatory or legislative barriers to the uptake of solutions.

While the above are common examples, they are not hard rules. For example, Innovation, Science and Economic Development Canada (ISED)’s Innovative Solutions Canada is a competitive, challenge-based research and development program designed to solve internal departmental operational issues and/or to fill a gap in the marketplace, with federal departments as the intended first (but not necessarily only) customers.

It is also important to understand the needs and interests of potential solvers. For example, WICT finalists gave more importance to business incubation support than to the competitive aspects of the challenge model. Respondents across challenges also noted that emerging clean tech is an inherently risky field, and that the use of milestone payments may be best suited to where there is a relatively high degree of certainty on how things will roll out. Otherwise, flexibility is needed to ensure that emerging developments (including technological developments or changes in international standards) can be considered in (re)negotiation of the milestone schedule.

Challenge Guidance

While design of the ICI-CTS’ focus areas captured lessons learned from benchmark challenges at the formative stage, at the outset of the program there was limited guidance from the Government of Canada to draw from to inform design and implementation. As a result, NRCan had to learn by doing, which created some difficulties and delays. However, NRCan now has a number of lessons learned that could help accelerate future use of related models by NRCan (and other government departments) and bring new perspectives to corporate practices. NRCan’s lessons learned have already fed into recent guidance released by PCO as a resource to guide future programming. For example, recognition of Indigenous innovation initiatives in PCO’s Impact Canada Challenge Guide (2021) emerged as a result of NRCan and other federal departments’ engagement with Indigenous peoples during delivery of the ICI.

Specific lessons learned (mostly compiled by NRCan program representatives and supplemented by evaluation) include:

  • Ensuring Early Engagement on Co-Design. Co-creation is a foundational element to the challenge model but two focus areas (support for SAF and to move Indigenous communities off-diesel) were committed to in funding authorities before the interventions were designed. As noted above, engagement with stakeholders in the co-design phase led to the prize challenge model being deemed inappropriate for the IODI. If possible, future interventions should undertake engagement and/or consultation in advance of committing to a focus area.
  • Working with the Fiscal Framework. The Government of Canada’s fiscal framework is not built for nimble programming. As shown in Figure 5, the program’s notional funding allocation by fiscal year was determined in funding authorities at the outset of the program, but the actual funding profile was designed with subsequent stakeholder engagement in the co-design phase. Further amendments were made in response to COVID-19. While there was relatively limited deficit or lapse of funds in any given fiscal year, realignment of resources across fiscal years required extensive use of the Annual Reference Level Update (ARLU) reprofile mechanism, which was reported by program managers to be cumbersome and resource-intensive. If possible, including a carry-forward provision in funding authorities could simplify use of flexibilities in allocating funds across fiscal years.

Figure 5: Evolution of Expenditure Profile of ICI-CTS (Before and After Co-Design)

Text Description

Figure 5: Evolution of Expenditure Profile of ICI-CTS (Before and After Co-Design)

Infographic showing the evolution of program expenditures before and after co-design in a five-year period, from FY 2017-2018 up to FY 2021-2022. The infographic encompasses four sets of data that include for each FY, the Original Budget (Before Co-Design), Revised Budget (After Co-Design), the Actual Expenditures and the Deficit / Lapse.

Budget original (avant co-conception)

Budget révisé (après coconception)

Dépenses réelles

Déficit/fonds non utilisés

2017-18

1,314.0

1,110.0

1,109.9

0.1

2018-19

27,968.8

5,109.1

5,241.1

-132.0

2019-20

28,067.1

17,308.7

17,739.2

29.7

2020-21

15,953.4

25,908.4

21,859.0

3,597.3

2021-22 (YTD)

0.0

23,868.0

2,467.2

-5,047.3

  • Managing Uncertainty. NRCan experienced difficulties in managing uncertainty related to the ICI-CTS’ innovative funding agreements. This includes uncertainty as to whether the pure prize grant for STL would be issued, with the potential for a resulting $1M in lapsed funds at end of program. This also includes the difficulties in managing milestone-based contribution agreements where payments are released once results are achieved. While the program must earmark these funds in its budget as per the notional timelines in the agreement, this creates a risk of lapsed funds if milestones are not achieved or are delayed (the latter resulting in a need to find funds in the next fiscal year). Again, if possible, including a carry-forward provision in funding authorities could simplify use of flexibilities in allocating funds across fiscal years. The accelerated timelines of a challenge model also provide less flexibility to deal with unexpected events (e.g., COVID-19, replacement of participants). To the extent possible, it is important to have contingency plans to deal with the unexpected.
  • Overcoming Cultural Resistance. While encouraged by the principles of the Impact Canada Initiative to implement innovative practices, the program was often met with resistance when proposing to do things differently because it did not align with conventional practices. This frame of thinking is common across the government, and made it difficult for the program to reach a consensus with corporate colleagues. A key lesson learned is that collaboration with corporate services is crucial when implementing a new, innovative program. Corporate services should be engaged early, as part of the program team, to enable them to have the time to really understand the program and its objectives. Senior management involvement is also required help manage expectations for the program’s implementation, while ensuring that senior managers are also aware of the limitations imposed by the broader policy framework. However, innovative new mechanisms - e.g., flexible milestone agreements, use of contribution agreements (vs. procurement) to develop collaborative arrangements, etc. – have now been tested and demonstrated to facilitate solutions and build connections in the innovation ecosystem in ways that advance the objectives of the challenges. Experimental learnings from the ICI-CTS should be used to impart new perspectives, modernize methods, and maximize efficiency in existing corporate processes.
  • Understanding Limits to Co-creation. There are policies and directives in existing frameworks (TB Policy and Directives) that limit the extent to which federal programs can collaborate with external entities, including limitations in how federal labs can collaborate with proponents. For example, the Government of Canada cannot directly benefit or be seen to benefit from the outputs of a contribution agreement or carry out any work that would suggest an employee/employer relationship. This means that NRCan cannot work hand-in-hand with co-creation organizations in the same way as may be done in external challenge models; NRCan cannot receive direct advice from collaborators and cannot direct collaborators in their activities. However, program representatives for the ICI-CTS have found that with adjustments (e.g., by using the term “collaborator” instead of “partner”, attending engagement sessions hosted by collaborators that focus on broader context), the program can achieve results similar to those intended in co-creation. While collaborators should always be carefully selected through research and engagement to determine who is best positioned to support results and benefits to Canadians, small collaborators require funding since they do not have the capacity to cover their resources but may also want ownership. Program representatives perceive the ideal collaborator in this context as one who does not need funds from the program to support the initiative (i.e., they see enough value in it to use their own resources).
  • Staff Skillsets. Collectively, the ICI-CTS program team was perceived to have the right mix of skillsets (e.g., project management, content expertise, etc.). However, implementation has required OERD to rely on internal resources beyond staff dedicated to the ICI-CTS; the level of resources necessary to deliver a new program model should not be underestimated. Consideration should also be given to ensuring that specialized skillsets are available to dedicated to aspects of project management that pose greater complexities than in a traditional Gs&Cs program, such as delivery on innovative communications objectives, and performance measurement and impact assessment.

Building on Progress

For most focus areas under the ICI-CTS, one solution will win the grand prize. However, the important issues targeted by each challenge and initiative (Table 3) cannot be fully addressed by the challenge, nor are they intended to be. There is no one, “silver bullet” solution to these complex problems. The solutions stimulated by these challenges may not be sustainable without further intervention. Interviewees indicated ongoing programming is necessary to maintain momentum, promote and/or accelerate adoption of solutions, both within and outside the Government of Canada. Suggestions varied by challenge but finalists and collaborators did not necessarily think that further funding was the answer – or at least not the only answer. Additional proposals included support for market development, policy and regulatory changes. Within OERD, it is understood that addressing barriers beyond technology is often required. These barriers are identified in OERD by applying its Innovation System Analysis Tool and associated methodology. We found that NRCan has learned a lot about the ecosystems through delivery of the ICI-CTS – how they work, where there are gaps, who the key players are – that can now be used to build on success and make further progress in addressing these areas as government priorities.

Replicating Success

NRCan has also learned a lot from its other design experiments applied in each focus area (the “experiments within an experiment”). The challenge model may not be the only design element that NRCan would want to replicate. For example, the co-design process added significant value to target key outcomes, including consideration of specific barriers for the problem area. Federal research centres may have a larger role to play in supporting RD&D. Programs and supports for new talent are important to increase participation of new solvers (e.g., SMEs) and can remove barriers for underrepresented groups. The testimonials from finalists illustrate the importance of targeted opportunities like WICT and the IODI, specifically for women and Indigenous peoples. Other elements we found to be best practices or real strengths in the ICI-CTS are highlighted throughout this report.

Conclusion

We found that the ICI-CTS is relevant. The program aligns with federal government and NRCan priorities, roles and responsibilities to support innovations in clean technology, supports mandate commitments to increase experimentation, and is structured to give more flexibility to proponents to propose innovative solutions.

Overall, we found that governance structures, processes and mechanisms for ICI-CTS were defined, established and operating effectively, including internal reporting mechanisms. Processes and controls are also in place to support compliance with relevant departmental guidance and the TB Policy on Transfer Payments, and are operating effectively most of the time.

The ICI-CTS provided an opportunity to test innovative ways to address problems for which solutions are not apparent, and generate evidence of which efforts work best to create greater public value in areas of high priority. Many of the ICI-CTS’ focus areas also incorporated numerous experiments within this broader experiment, for example, testing new models of Indigenous participation or new ways of working with international partners.

We found that the design of focus areas under the ICI-CTS was consistent with principles of Impact Canada and best practices in challenge models. Stakeholder engagement in co-design was recognized as a key strength for the program. This allowed NRCan to target key issues while scaling the challenge to what would be achievable by solvers, including consideration of specific barriers for the problem area. The program was responsive to this feedback and, while there was some room for improvement, the resulting funding flexibilities and non-financial proponent supports built into the design of each focus area facilitated participation from diverse solvers. While communications could have been more innovative and better resourced, the program attracted a sufficient number of quality proposals.

We examined the extent to which the ICI-CTS is making progress towards program-level immediate outcomes identified in its draft logic model, including increased collaboration and investment by stakeholders in the clean tech ecosystem, increased awareness and participation in clean tech (i.e., mobilization of “new talent”), and increased public awareness of clean tech issues among Canadians. While available evidence suggests that progress has been made in each area, the ICI-CTS uses innovative approaches that necessitate new methods and strategies for measuring program outcomes and impacts. A key difference in the design of challenge-based models is that if the result is not achieved, the funding is not awarded. We found progress at least insofar as finalists are progressing towards outcomes specific to their project; projects are achieving expected project milestones. Regardless, difficulties in the design and application of performance metrics still need to be resolved. There is a need to generate better evidence that the perceived risks of running a challenge, and the level of investment that is required to run them, is justified by the achievement of better outcomes.

The issues targeted by each focus area will not be fully addressed by the ICI-CTS, nor are they intended to be. There is no one, “silver bullet” solution to these complex problems. While there is evidence of viable pathways to scale success for each focus area, this requires dedicated efforts that extend beyond the scope and timelines of the ICI-CTS. However, the program is intended to help advance and accelerate solutions, both as a direct impact of the solutions proposed in each focus area and by applying the intelligence gained during implementation to inform other programs and actors, which may in turn also accelerate the design or implementation of parallel initiatives in related focus areas.

Overall, we found the challenge model to be a new tool that provides an alternative to traditional funding programs, but decisions on when it is appropriate to use this model should be carefully considered in light of the context of the outcomes to be achieved. We expect NRCan to continue to apply lessons learned from the ICI-CTS in its own design of future prize challenges and other initiatives, whether under the umbrella of Impact Canada or as part of a separate initiative. Future use of design elements that were found to be strengths of the ICI-CTS (e.g., co-design) need not be limited to challenge models.

Appendix A: Engagement Sub-Objectives and Criteria

The objectives of this engagement are:

  • To assess the extent to which the program’s design and delivery facilitates the effective and efficient achievement of immediate results and complies with relevant authorities.
  • To identify and document lessons learned from the experimental approaches under the Impact Canada – Clean Technology Stream that can be applied to future interventions.

The following table identifies the questions addressed by the engagement, as well as the sub-objectives and criteria used to conduct the engagement:

Sub-Objectives Criteria

Question 1: Is the Impact Canada – Clean Technology Stream relevant?

Sub-Objective 1: To assess the extent to which the program is relevant.

1.1 The Impact Canada – Clean Technology Stream aligns with federal government and NRCan priorities, roles and responsibilities.

1.2 The Impact Canada – Clean Technology Stream gave priority to the selection of relevant challenges and initiatives, i.e., each challenge:

  • targets breakthrough solutions that focus on transformative (i.e., not incremental), high-impact improvements that break new pathways in clean technology;
  • demonstrates a high potential for impact and significant benefits to Canadians;
  • demonstrates a high degree of complementarity with existing government and non-government efforts in the given problem space, including the viability of pathways to scale successes (e.g., clearly identified opportunities for handoff to other government programs, venture capital, etc.); and
  • demonstrates the ability to leverage partners and resources to help design, deliver, or promote participation in the prize or challenge, including via contributing resources (e.g., additional prize funds, in-kind resources, etc.), acting as a conduit to important stakeholder groups, or providing technical expertise.

Question 2: Are adequate and effective governance structures in place to support the Impact Canada – Clean Technology Stream?

Sub-Objective 2: To assess whether adequate and effective governance structures and processes have been designed and implemented to support the achievement of the Impact Canada – Clean Technology Stream’s outcomes.

2.1 NRCan roles, responsibilities, and accountabilities are adequately defined and communicated.

2.2 Program progression is monitored and communicated to senior management and used to inform future program phases.

Question 3: To what extent does the experimental design of the Impact Canada – Clean Stream facilitate the achievement of intended outcomes?

Sub-Objective 3: To assess how and for whom the Impact Canada – Clean Technology Stream’s experimental tools are effective in facilitating the achievement of intended outcomes.

3.1 The design and delivery of inclusive innovation (co-design) facilitates the achievement of intended outputs and outcomes.

3.2 The design and delivery of communications and outreach strategies facilitates the achievement of intended outputs and outcomes.

3.3 The design and delivery of collaborator arrangements facilitates the achievement of intended outputs and outcomes.

3.4 The design and delivery of flexible funding mechanisms facilitates the achievement of intended outputs and outcomes.

3.5 The design and delivery of proponent supports (e.g., training, networking, etc.) facilitates the achievement of intended outputs and outcomes.

Question 4: Does the delivery of program activities comply with relevant authorities?

Sub-Objective 4: To assess whether adequate and effective processes and controls are in place to support the Impact Canada – Clean Technology Stream’s compliance with relevant departmental guidance and the TB Policy on Transfer Payments.

4.1 Selected proponents meet the selection criteria of the Program.

4.2 Funding agreements have been drafted in accordance with PCO-approved Program Terms and Conditions and the Policy on Transfer Payments.

4.3 A risk-based accountability structure has been designed and implemented to support monitoring and reporting of funding agreements.

Sub-Objective 5: To assess whether key financial and operational controls have been designed and implemented, and are operating effectively.

5.1 Payments to proponents have been made in accordance with funding mechanisms identified in approved funding agreements and in accordance with the FAA.

5.2 Processes are in place to ensure that appropriate disclosure of agreements is verified for accuracy and approved prior to Web posting.

Question 5: What have been the outcomes (expected and unexpected) that the Impact Canada – Clean Technology Stream has achieved to date?

Sub-Objective 6: To assess whether the Impact Canada – Clean Technology Stream is making progress towards its expected immediate outcomes.

6.1 There is increased collaboration in key technology challenges and projects, as evidenced by the development of collaborative arrangements and collaborative networks.

6.2 New talent is being mobilized to participate in clean technology, as evidenced by an increase in non-traditional proponents and collaborators.

6.3 There is increased investment by stakeholders in clean technology research, development and demonstration (RD&D) projects.

6.4 There is enhanced public awareness of clean technology problems or issues, as evidenced by reach of communications.

6.5 There are adequate performance measures and data collection procedures in place to support the determination of effectiveness, efficiency and economy.

Sub-Objective 7: To identify if any unexpected outcomes (positive or negative) have resulted from the Impact Canada – Clean Technology Stream.

7.1 There is evidence of additional (unexpected) outcomes.

Question 6: To what extent has the program been responsive to the internal and external factors that (positively or negatively) influence its ability to achieve intended outcomes and operate with efficiency?

Sub-Objective 8: To assess the extent to which the Impact Canada – Clean Technology Stream appropriately considers the internal and external factors that could impact its ability to effectively or efficiently achieve its intended outputs and outcomes.               

8.1 The Impact Canada – Clean Technology Stream has identified and developed approaches to respond to internal and external factors that impact (or are expected to impact) its delivery.

Question 7: What are the conditions required to accelerate the use of the Impact Canada – Clean Technology Stream’s innovative approaches to future interventions?

Sub-Objective 9: To identify lessons learned that could support the positive evolution of government systems to enable greater innovation.

9.1 There are best practices and lessons learned through the design and delivery of the Impact Canada – Clean Technology Stream’s prize challenges and experimental design features that could be applied to future interventions.

Appendix B: Joint Engagement Team

Chief Audit and Evaluation Executive

Michel Gould

Evaluation Team

David Ash – Senior Advisor, Evaluation
Stephanie Kalt – Evaluation Manager
Christiane Ngo Manguelle, PhD – Senior Evaluator
Edmund Wolfe – Senior Evaluator
Harpreet Sahota – PARDP
Aishwarya Babu - PARDP

Audit Team

Linda Jones – Senior Director, Audit Operations
Katherine Venus – Audit Manager
Raj Brar – Senior Auditor
Nour Al Youssef - Auditor

Footnotes

1. Note on Terminology: There are important differences in the language used to describe Indigenous innovation initiatives. Within this report, the joint engagement uses the term “focus areas” to describe all of NRCan’s prize challenges and the Indigenous initiative launched under the ICI-CTS. For simplicity, we also uses the term “finalists” to describe proponents selected as final participants receiving funding to advance solutions in these focus areas, including Clean Energy Champions and their communities. However, we recognize that this term is normally not applicable to the Indigenous Off-Diesel Initiative as it relies on a collaborative rather than competitive model.

2. This total does not include additional funding provided by BEIS to their finalists.

3. The use of the term ‘delivery agent’ throughout this report refers to ‘MaRS’ as an intermediary; i.e., an initial recipient, redistributing funds to ‘ultimate recipients.’ Initial recipients are defined as a recipients that are to further distribute a contribution to one or more persons or entities, known as ‘ultimate recipients’. The use of the term ‘agent’ does not infer any sort of ‘agency’ arrangement between NRCan and MaRS.

4. PCO’s Impact Canada Portal: https://impact.canada.ca

5. PCO Logic model and narrative - Impact assessment of challenges under Impact Canada -https://www.canada.ca/en/innovation-hub/services/reports-resources/logic-model-narrative.html

6. One of six finalist teams from Crush It and one of seven finalist teams from Power Forward declined to participate in an interview. Interviews were only held with representatives of eight communities participating in IODI. However, six additional communities opted to complete the engagement’s survey. In all other focus areas, we interviewed one or more finalists from each challenge or initiative.

7. Only participants in the IODI did not rank this as the primary motivation for participation. While they noted that funding was critical to advancing their clean energy plans, most IODI participants indicated that their primary motivation for participation was the potential achievement of social and environmental benefits for their community.

8. The engagement’s survey asked unsuccessful applicants to provide input on the extent to which financial and non-financial supports motivated them to apply to the ICI-CTS but did not ask them to comment on the effectiveness of these mechanisms in supporting challenge delivery. 

9. PCO (2019). Measuring Impact by Design: A Guide to Methods for Impact Measurement. https://www.canada.ca/en/innovation-hub/services/reports-resources/measuring-impact-design.html

10. PCO (2020). Logic Model and Narrative: Impact Assessment of Challenges under Impact Canada. https://www.canada.ca/en/innovation-hub/services/reports-resources/logic-model-narrative.html

11. It was not clear whether blank fields were a reporting omission or an indication of no results to report.

12. The Clean Growth Hub is a free service by the Government of Canada that works with clean technology producers and adopters to help find federal programs and services to advance clean tech projects.

14. Respondents were not asked to provide additional details. It is not clear if this importance was attributed to local environmental impacts (e.g., air quality and mitigation of diesel spills), global impacts (e.g., climate change), or both.

Page details

Report a problem on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: