Evaluation of the Forest Sector Innovation (FSI) Programs

Audit and Evaluation Branch
Natural Resources Canada

FINAL REPORT – December 30, 2019

Table of Contents

• List of Acronyms
• Executive Summary
• Introduction
• Evaluation Objectives and Methods
• Evaluation Limitations and Mitigation Strategies
• Findings: Relevance
  • Performance – Effectiveness
  • Performance – Efficiency
• Conclusions
• Appendix 1: Evaluation Team

List of Acronyms

AEB

Audit and Evaluation Branch

CFS

Canadian Forest Service

CLT

Cross-laminated Timber

CWFC

Canadian Wood Fibre Centre

FRII

Forest Research Institute Initiative

FSI

Forest Sector Innovation

GHG

Greenhouse Gas

IFIT

Investments in Forest Industry Transformation

NBCC

National Building Code of Canada

NRCan

Natural Resources Canada

PCF

Pan-Canadian Framework on Clean Growth and Climate Change

R&D

Research and development

SLAP

Softwood Lumber Action Plan

SME

Small- and Medium-sized Enterprise

TT

Transformative Technologies

Executive Summary

About the Evaluation

This report presents the findings, conclusions, and recommendations from the evaluation of the Canadian Forest Service’s suite of forest sector innovation programs (FSI) within the Forest Sector Competitiveness program. FSI encompasses the Investments in Forest Industry Transformation (IFIT) program; the Forest Research Institute Initiative (FRII); and the Forest Innovation Program (FIP). FIP is comprised of the Transformative Technologies (TT) program, which is its core component, and the Canadian Wood Fibre Centre (CWFC).

Collectively, the programs are designed to invest in innovative research, development and demonstration (RD&D), commercial pilots and knowledge transfer that will support the forest sector’s transformation to ensure its competitiveness. Two of these programs – the TT and the FRII programs are delivered by FPInnovations.

The evaluation covers the period from 2013-14 to January 2019. Actual expenditures for the programs (2013-14 to 2017-18) were approximately $271 million. IFIT and the TT program component account for most of these expenditures.

The objective of the evaluation was to assess the relevance, effectiveness and efficiency of the FSI program.

An evaluation of the Expanding Market Opportunities, covering a similar time period, is being conducted separately. This program, like the forest sector innovation programs, is part of the Forest Sector Competitiveness program.

What the Evaluation Found

Relevance

The evaluation found that the FSI programs continue to be relevant and aligned with federal government and NRCan roles and responsibilities, and priorities. These programs address a gap in funding support for high-risk innovation in the forest sector.

There is an ongoing need for the federal government to fund the FSI programs to address challenges to the sector’s competitiveness. There is also a growing recognition of the importance of FSI with respect to both climate change adaptation and mitigation solutions. The transition to the bioeconomy supports Canada’s environmental performance goals. The bioeconomy also has the potential to generate revenue streams from what may currently be considered forest waste products.

The forest sector’s significance to Canada’s economy supports the federal role in this area. Moreover, other national jurisdictions with significant forest sector economies, have comparable innovation programs to increase competitiveness and to advance the bioeconomy.

The location of the programs at the federal level helps to even the playing field in cases where there is variation in provinces’ capacities to support forest sector innovation, and provides a “big picture” understanding of threats and opportunities affecting the national forest sector. Utilizing a collaborative model, CFS is well-positioned to deliver research programs that are aligned with industry needs.

With respect to the TT program component, which is delivered by FPInnovations, our evaluation found that the CFS role with respect to the TT planning process, needs to be strengthened to ensure an appropriate balance between transformative and incremental activities. Stakeholders generally noted that while a mix of incremental and transformative activities is appropriate, they perceived that the TT program component requires more emphasis on transformative innovation.

Effectiveness

Evaluation evidence suggests that FSI has made progress towards its intended outcomes with respect to developing and advancing innovations, and providing evidence for codes, standards, policy-development, and decision-making pertaining to forest management and sustainable development.

Specific areas of progress include:

• Advanced building systems: The TT and IFIT programs have contributed to the adoption of advanced building systems such as mid-rise building wood-construction by informing the development of supporting codes; improving technologies and higher-value products (eg., cross-laminated timber); piloting technologies and automated production systems; and by producing technical guidance products for industry.
• Bioproducts: The TT program has resulted in significant advancement of novel bioproducts (i.e., cellulose nanocrystals, cellulose filaments, and lignin). Progress has been made in developing various product applications for these materials. These bioproducts have the potential to be used as a raw material for different products and to replace petroleum-based raw materials that are in a wide range of products (e.g., plastics).
• Enhanced Forest Inventory: CWFC has made substantial contributions to provincial and industry uptake of enhanced forest inventory tools and practices. Enhanced Forest Industry technologies provide better information for the assessment of wood and fibre attributes to inform forest planning and management decisions.

While CFS has made inroads into strengthening its engagement, transitioning to a bioeconomy will require efforts to further broaden linkages with a range of stakeholders, particularly small and medium-sized enterprises (SMEs), other sectors, and academia.

Program effectiveness is also affected by the insufficient capacity of the forest sector to use research to innovate. Our evaluation found that there were innovation gaps impeding outcome achievement. These include the need for more advanced bioproduct applications research, market research and support for novel products; and the need for a more secure supply of bioproducts.

Some early economic benefits have been observed at the firm level, therefore suggesting that the programs are paving the way for long-term improved forest sector competitiveness. IFIT, a program that targets the later stages of the innovation continuum is making good progress. Completed IFIT projects are reporting positive economic benefits such as increased return on investment and revenues.

While the forest sector continues to rely heavily on commodity products, there has been some progress with respect to the development and production of higher value wood products.

Efficiency

Evaluation evidence suggests that several aspects of the program design and delivery contribute to the efficient production of deliverables and results. The collaborative model and its broad-spectrum approach were frequently cited as key strengths of the FSI programs. In addition, CFS undertakes periodic assessments of its research directions, workplans, and progress.

FPInnovations, a central hub for research, was cited by external stakeholders as an effective model for collaborative and non-proprietary research and development. However, they viewed its membership structures, which was being modified at the time of this evaluation, as constraining the Transformative Technologies program and increasing the risk that the program is not optimally reaching its target groups.

IFIT delivers large, complex projects and has experienced challenges relating to the periodic call for proposals process which was reported to constrain selection of projects to those that had a higher level of preparedness at the time of the calls. In addition, stakeholders suggested that funding rules restricting the carry over of funds from one fiscal year to another should be made more flexible in light of the realities of large-scale innovation projects, which depend on many external, sometimes unpredictable, factors. Finally, our evaluation found that there is a need to identify and implement novel approaches to encourage more high risk projects.

A review of performance information identified several performance measurement issues. Our evaluation team found that FIP (TT program and CWFC) and FRII reports lacked overall aggregate reporting on program status and outcomes, making it challenging to assess the extent of outcome achievement of FIP. Better metrics and methods are needed to measure outcomes such as technology advancement. In addition, the use and uptake of knowledge products and decision-making tools is not typically reported. IFIT does a reasonably good job of monitoring its key outputs and outcomes, but its program reports lack clarity.

Recommendations

Recommendation	Management Response
1. The Canadian Forest Service should play a stronger role to ensure that the Transformative Technologies program has an optimal balance of transformative and incremental innovation activities to support the transition to the bioeconomy.	Management agrees with this recommendation As part of FPInnovations redesign of their research strategy, the Canadian Forest Service (CFS) will actively participate and influence the planning process of their work plan to ensure there is emphasis on transformative innovation activities in support of the CCFM Forest Bioeconomy Framework. The evolution of the new TT program will target both transformative and incremental innovation activities.(more details provided in Recommendation #2). (ADM, CFS responsibility) Target Date: March, 2020
2. The Canadian Forest Service should work with its partners to further broaden the programs’ reach and increase collaboration with SMEs, academia, and other sectors. The Canadian Forest Service should consider and implement novel approaches for pilot projects to increase collaboration and knowledge transfer between industry, FPInnovations, and academia.	Management agrees with this recommendation The CFS will work with partners to increase its collaboration with SMEs, academia and other sectors. The IFIT program will use the small enterprises stream to target SMEs through more aggressive outreach activities for upcoming calls for proposals. Small enterprises follow a streamlined application and due diligence process as outlined in the IFIT guidance document on how to submit a proposal. The CFS will also work to improve direct funding opportunities, collaboration and knowledge transfer with academia, national organizations, SME’s and other industrial sectors through the launch of competitive pilot project platform collaborations with FPInnovations, industry and academia. In particular, CWFC will widen its reach, by working more closely with FPInnovations on national collaborative projects to increase the development and deployment of scientific tools and products to the forest industry and provinces. (ADM, CFS) Target Date: March, 2020
3. The Canadian Forest Service should work with partners to advance projects that will address pre-commercialization innovation gaps.	Management agrees with this recommendation The CFS will work with partners to address pre-commercialization innovation gaps where no current funding is available. This will be accomplished through market research studies for novel products and bioproduct applications R&D needed to bring biomaterials to market. The IFIT program’s Feasibility Studies project category will be expanded to include different types of assessments needed to bridge gaps in the innovation spectrum. This component will now be called Complementary Studies. Examples of eligible projects include feasibility studies for innovation technologies or trials for market acceptance of new bioproducts. (ADM, CFS) Target Date: March, 2020
4. The Canadian Forest Service should improve its performance measurement and reporting of its programs and projects: Identify and implement valid metrics, targets and methods for measuring key outcomes of innovation programs (e.g. technology advancement, value and use of knowledge products and decision-support tools). Reports should clearly, accurately and consistently report on the status of milestones and progress at the project and program levels.	Management agrees with this recommendation CFS has developed a solid approach by following a methodology for effective performance reporting according to the Performance Information Profile (PIP) for the Fibre Solutions and the Forest Sector Competitiveness programs. In addition, work is underway to strengthen and streamline the performance measurement system and performance metrics, including GBA+ considerations, as part of the renewal efforts of Budget 2019. CFS will assess the feasibility of using a CRM tool which would be planned to be available by summer 2020. (ADM, CFS) Target Date: March, 2020 for new methodology, ongoing for implementation through to March 2023
5. The Canadian Forest Service should work with Treasury Board Secretariat to enhance elements of IFIT’s design and delivery. These changes would facilitate optimal project selection, encourage high risks projects, and improve the management of complex, innovation projects: Consider new, more flexible models of funding intake. Identify strategies to enhance funding flexibility that will better allow carry-over of funds from one fiscal year to the next. Identify novel strategies to encourage high risk projects (e.g., challenge-based contests, a phased funding approach, adopting higher risk tolerance for smaller projects).	Management agrees with this recommendation As part of the Budget 2019 program renewal, IFIT will be exploring ways to increase flexibility for its applicants and recipients:   The program will identify flexible proposal intake options that strike a balance between a single call for proposals process and a continuous intake process and that can be practically applied given the three-year funding period and the average completion time of a IFIT project (18-24 months). CFS will engage with TBS on experimentation measures that may allow for more funding flexibility between fiscal years. The program will review options that align with the Transfer Payment policy. IFIT program objectives include the derisking of innovation and as result, the program inherently encourages proposals for high risk projects. A streamlined risk assessment process for small enterprises is also in place. The program will implement novel strategies, for example, putting in place a formal process to encourage complementary studies or product trials for promising technologies, process, or bioproducts. Furthermore, IFIT will encourage solutions to industry challenges through its new Targeted Measures project category. (ADM, CFS), Target Date: March 2020 to confirm outcome of engagement with Treasury Board Secretariat and program renewal. Target Date: October 2020 to commence implementation of flexibility measures.

Introduction

This report presents the findings, conclusions, and recommendations from the evaluation of the Canadian Forest Service’s (CFS) suite of forest sector innovations programs (FSI), covering the period from 2013-14 to January 2019. The FSI programs encompass the Investments in Forest Industry Transformation (IFIT) program; the Forest Research Institute Initiative (FRII) and the Forest Innovation Program (FIP). FIP is comprised of the Transformative Technologies (TT) program, the key component, and the Canadian Wood Fibre Centre (CWFC).

These programs each target a different section of the innovation continuum, including fundamental research, development, demonstration, and early deployment. The FSI programs are part of the Departmental Results Framework’s Forest Sector Competitiveness Program.

Natural Resources Canada’s (NRCan) Audit and Evaluation Branch (AEB) undertook the evaluation between May 2018 and June 2019. It was conducted in accordance with the 2016 Treasury Board Policy on Results, and responds to requirements under section 42.1 of the Financial Administration Act as well as a Treasury Board commitment to evaluate the programs by March 2019.

Program Information

Since the inception of FIP and FRII in 2007 and IFIT in 2010, these programs have been intended to increase the sector’s competitiveness through innovation for sector transformation. This has been particularly important during challenging economic times for the sector, such as the 2008 recession and during the ongoing Canada-U.S. softwood lumber dispute. In fact, the programs are continuing under the federal Softwood Lumber Action Plan. Collectively, the programs are designed to invest in innovative research, development and demonstration (RD&D), commercial pilots and knowledge transfer that will support the forest sector’s transformation to producing higher-value forest products. Whereas the CWFC and IFIT aspects are delivered by NRCan CFS staff, the TT program component and FRII are delivered by FPInnovations, a not-for-profit research institute that receives funding from federal, provincial and industry stakeholders to undertake collaborative pre-commercial research and development for the forest sector, as well as undertake technology transfer activities.

FPInnovations’ research plan is developed with guidance from the organization’s Board of Directors— of which the Assistant Deputy Minister of the CFS is an observer— as well as in consultation with the organization’s members and six program advisory committees, which were being modified at the time of this evaluation. CFS performs oversight of the activities of FPInnovations’ and its partners under Contribution Agreements for the TT program and FRII.

As part of FIP, the TT program supports FPInnovations to deliver collaborative, pre-commercial, non-proprietary research, development and demonstration (RD&D), commercial pilots and technology transfer for innovative technologies and processes in the forest sector. These investments are intended to lead to a more value-chain approach to Canadian forest products.

With employees located across the country, CWFC works closely with FPInnovations and other stakeholders to undertake wood fibre research and facilitate end-user uptake. CWFC aims to develop targeted and environmentally responsible solutions to challenges faced by Canada’s forest industries.

The IFIT program offers non-repayable contributions to successful applicants to implement innovative, first-in-kind technologies in their facilities. The program provides funding for large-scale demonstration projects, as well as commercial pilots, with a view to helping these technologies get to market. From its inception to 2018-19, IFIT has funded 31 projects. These projects fall into the following categories: Green Energy (6), Biomaterials (8), Solid Wood (10), and Biochemicals (7).

Actual Expenditures by Program

The FSI programs represent approximately $271 million (M) in NRCan expenditures from 2013-14 to 2017-18. IFIT and the TT program have the highest expenditures with $118M and $121M respectively. During this time, IFIT funded 20 projects, and the TT program, CWFC, and FRII funded an average of 95 projects per year. With the exception of FRII, which receives A-base funding, the FSI programs primarily receive C-base funding.

Evaluation Objectives and Methods

The objective of this evaluation is to assess the relevance (i.e., continued need and alignment with federal and NRCan priorities and roles/responsibilities) and performance (i.e., effectiveness and efficiency) of the CFS Forest Sector Innovation programming (FSI) and to consider how the program’s design influences effectiveness.

The evaluation used six key lines of evidence, consisting of qualitative and quantitative data:

Document Review	Literature Review	Database and File Review	Key Informant Interviews	Industry online survey	Case Studies
A review of program documents, as well as strategic and corporate documents for NRCan, the federal government and FPInnovations. This line of evidence informed findings across all evaluation questions.	A review of published literature, with a focus on best practices in forest sector innovation and innovation systems in Nordic countries.	A review of project-specific financial and results data from IFIT’s database. To minimize selection bias, projects (2017-2018 and 2018-2019) were randomly selected, including 30 from the TT program, 11 from FRII, and 9 from CWFC.	Ninety (90) interviews were conducted (11 of which were NRCan representatives and 79 were with external stakeholders -industry, provinces/ territories, and academia).	An online survey of industry stakeholders was conducted to solicit views on the programs’ relevance and performance, facilitating factors and barriers to innovation. The survey had a 25% response rate (n=58); the majority of respondents were from small- and medium-sized enterprises (SMEs) and organizations that did not receive funding from the FSI programs.	The evaluation completed six case studies to understand the impacts and influencing factors related to the following themes/ projects: TMP-Bio project; advanced building systems; cellulose nanocrystals; AE Côte-Nord Bioenergy project; extracting lignin for new forest products; and enhanced forest inventory.

Evaluation Limitations and Mitigation Strategies

The evaluation used multiple lines of evidence in order to mitigate any limitations associated with individual methods. These enabled the triangulation of evidence across sources of information to identify valid findings and conclusions relative to the evaluation questions. Nevertheless, the following limitations and mitigation strategies should be considered when reviewing the findings from this evaluation:

• Numerous studies and analyses of the broader impacts of R&D note that there exists a long lag time between investment in research and realization of tangible productivity gains. To mitigate for this challenge the case studies and document review included a longitudinal view of the FSI programs’ contributions, sometimes going back years before this evaluation period. This longitudinal view illustrates the timelines for achievement, as well as the various factors that influence effectiveness.
• The evaluation included a survey of industry stakeholders. These stakeholders were drawn primarily from a CFS mailing list and also from other relevant industry lists from various sub-sectors (located on the Internet). The response rate of 25% was within acceptable survey standards, but the sample size was small (n=58), making credible comparisons among survey sub-groups (e.g. smaller and larger firms, different sub-sectors) impossible.

The sample was primarily made up of small organizations (less than 100 employees), who produced a variety of forest products. However, because the sample is not representative of the forest industry and its various sub-sectors, the evaluation utilizes the survey results with caution and cites the results when corroborated by other lines of evidence.

Most of the survey respondents were neither members of industry associations or FPInnovations nor participants in CFS forest sector programs. This did help to achieve the evaluation’s intention to reach beyond recipients of program contributions and better assess the relevance of the program and the barriers and opportunities for innovation.

What We Found

Relevance

To what extent is there a continued need for the program?

To what extent is the program aligned with federal government and NRCan priorities, roles and responsibilities?

Findings: Relevance

There is a continued need for the forest sector innovation programs

All relevant lines of evidence confirm that there is an ongoing need for the Investments in Forest Industry Transformation program, the Forest Innovation Program (Transformative Technologies program component and the Canada Wood Fibre Centre), and the Forest Research Institute Initiative (FRII). Collectively these programs aim to increase the forest sector’s competitiveness while supporting the transition to the bioeconomy and a low-carbon economy future.

Bioproduct sectors around the world are forecasted to expand over the next few years. Forests are recognised as a significant contributor to any development of a bioeconomy as a source of renewable biomass. Globally the key drivers are climate change, sustainable development and economic development.

The sector’s reliance on its commodity base has left it vulnerable to changes in market conditions, as evidenced during the 2008 recession and the ongoing Canada – U.S. softwood lumber dispute. In addition, the market for newsprint and printing paper continues to decline. The development of new products and new applications for existing forest products is expected to help the forest sector adjust to changes in market demands. Stakeholders indicate that both the TT program and IFIT play a valuable role in de-risking innovation and in developing higher-value fibre products.

Documents indicate that fibre feedstock is the single largest cost for the forest industry and that the fibre supply is declining (primarily attributed to climate change impacts). The TT program’s work related to maximizing the value of fibre and more efficient harvesting and use of forest materials is particularly relevant to this context. In addition, CWFC’s silviculture work designed to improve growing rates and fibre quality is aligned with the need to optimize the value of wood fibre.

The Forest Sector Innovation programs can play a role in addressing climate change adaptation and mitigation needs

There is a growing recognition, as indicated by peer-reviewed articles and interviews with experts, of the potentially important role of the programs with respect to both climate change adaptation and mitigation. According to our literature review, optimal environmental benefits are most likely to be realized when combined with several strategies such as sustainable forest management practices, efficient forest product utilization, high residue recovery rates, including residue from wood waste and high efficiency utilization of harvested biomass. Moreover, stakeholder feedback also confirms that the bioeconomy is perceived as very important, with the potential to generate revenue streams from what may currently be considered waste products.

There is a strong correlation between innovation and competitiveness

The evaluation team’s review of academic literature concluded that innovation development and adoption is an important part of facilitating the long-term competitiveness of the forest sector. In addition to supporting forest sector transformation, innovation allows the sector to expand into non-traditional markets, increase sector productivity, and compete globally by targeting markets for niche products. Interviewees also highlighted the critical role of the TT program component and FRII’s research on codes and standards in increasing market access and facilitating the commercialization and adoption of technologies.

Continuing federal funding for the TT program and FRII allows FPInnovations to maintain a consistent research budget directed towards collaborative, non-proprietary research

Findings from stakeholder interviews and program documents indicate that the TT program and FRII funding allows FPInnovations to maintain a consistent collaborative research budget, therefore supporting continued innovation activities despite changes in market conditions and industry support. There was a decline in FPInnovations’ revenue from 2015 to 2018. This decline is partially attributed to the drop in funding from FPInnovations’ Strategic Research Alliances, which are funded by industry. These alliances are partnerships between FPInnovations industry members and other stakeholders to undertake more advanced research. Federal funding represents an increasing proportion of FPInnovations’ revenue during this time.

The Forest Sector Innovation Programs are aligned with federal government and NRCan priorities and responsibilities

The FSI programs are aligned with the Government of Canada’s priorities to enhance the competitiveness of the natural resources sectors and environmental performance. The programs demonstrate alignment to Government commitments under the 2016 Pan-Canadian Framework on Clean Growth and Climate Change (PCF). A review of program documents found that the programs align with the transition to a low-carbon economy by identifying opportunities to produce bioproducts and bioenergy, supporting innovation for sustainable forest operations and high-value fibre products, and facilitating the market adoption of environmentally-friendly wood construction and bioproducts.

Through these actions, the programs are also consistent with the pillars of the 2017 Canadian Council of Forest Ministers’ Forest Bioeconomy Framework. The programs also reflects several objectives of the 2016 to 2019 Federal Sustainable Development Strategy: effective action on climate change; modern and resilient infrastructure; greening the environment; clean energy; clean growth; and sustainably managed lands and forests.

The federal government’s role is consistent with forest sector innovation. The Natural Resources Act, which states that the Minister of Natural Resources shall “seek to enhance the responsible development and use of Canada’s natural resources and the competitiveness of Canada's natural resource products”. It is also consistent with the Forestry Act, which states that the Minister “shall provide for the conduct of research relating to the protection, management and utilization of the forest resources of Canada.”

While other federal government departments also support innovation programs, external stakeholders interviewed believe that NRCan has the appropriate expertise and experience to support FSI programming. In addition to its expertise on all parts of the fibre value chain, interviewees noted that CFS’s strong connections with industry stakeholders position it well to deliver research programs aligned with industry needs. However, stakeholders also noted that the transition to the bioeconomy will require more integration and coordination with other sectors. CFS has taken steps to achieve this through their support of CCFM’s Bioeconomy Framework and the development of a federal bioeconomy strategy (in progress during the time of the evaluation) in coordination with other federal partners.

In addition, the location of innovation funding supports at the federal level also helps to level the playing field in cases where there is variation in provinces’ capacities to support forest sector innovation, and provides a “big picture” understanding of threats and opportunities affecting the national forest sector.

The forest sector is a significant contributor to Canada’s economy

The federal role is appropriate because the forest sector is a significant contributor to Canada’s economy.

Other national jurisdictions with sizeable forest sector economies have comparable innovation programs to increase competitiveness and to advance the bioeconomy

Investing in innovation to combat challenges to the forest sector’s competitiveness is not unique to Canada. Other governments with significant forest sector economies, such as Finland, Sweden, and Australia, have created programs to support innovation in the forest sector, increase its competitiveness and advance the bioeconomy.

Stakeholders consider long-term, high-risk research to be an appropriate area for federal government involvement

The FSI programs typically target long-term, high-risk innovation that would not likely be undertaken by other stakeholders. The TT program and CWFC support pre-competitive research, an area, according to stakeholders, where industry is less likely to be involved. IFIT funds first-in-kind commercial scale technologies in the forest sector due to their high-risk; this allows the program to de-risk innovation at this stage. This is considered an appropriate role for government because the first in industry to pilot an innovation is usually unable to capitalize on their advantage, or access funding from investors.

The FSI programs address the lack of funding options available for high-risk innovation in the forest sector. Interview, survey and document evidence indicated that this is a significant barrier to innovation in the sector, and that the private sector is less likely to invest in high-risk innovation. Evidence from our survey, interviews and the literature review indicates that insufficient capital and difficulties in accessing external financing are among the most significant challenges for industry in developing innovations.

CFS leadership is required to ensure an appropriate balance between incremental and transformative research

Stakeholders generally noted that while a mix of incremental and transformational activities is appropriate, they perceived that more emphasis on transformative innovation is needed to support the transition to the bioeconomy. Evaluation evidence indicates that FPInnovations’ design limited CFS’s influence on the TT program component’s research directions. Stakeholders suggested that CFS develop clearer guidance as to what activities are transformative in light of its priority to transition to the bioeconomy.

In our review of a random sample of TT project files (2017-2018), about half of the projects supported incremental innovation. However, of the total budget for the sampled projects, about one third of the budget was allocated to incremental research versus two thirds of the budget to transformative research. While it is not known if this budget allocation is representative of the past five years, it does suggest that a considerable portion of the TT program’s activities is being directed towards incremental innovation.

What We Found

Effectiveness

To what extent do FSI program activities result in expected outcomes?

What factors are influencing effectiveness?

Performance - Effectiveness

FSI programs are progressing towards their intended immediate and intermediate results

Expected Results

Innovation takes time and does not progress along a linear path

Prior to presenting the findings on outcomes, it is useful to understand how innovations typically progress. A literature review on innovation indicates that there is typically a significant time lag from early stage research to commercialization. Moreover, the innovation process is unpredictable. Knowledge created during the various stages of innovation can inform further research at an earlier stage on the continuum.

Innovation is also cumulative - the more the knowledge base grows, the better the capacity develops to address barriers to technology adoption, such as the initial high costs of technologies. Therefore, measures of success for an innovation program have to consider not only how technologies advance, but also how the knowledge generated throughout the innovation process is shared and used.

The stages of the innovation continuum

Available performance information hindered the evaluation’s assessment of overall results, particularly with respect to the FIP

The FIP (TT program and CWFC) performance reports did not provide clear evidence of overall program progress on key deliverables and outcomes. In addition, an assessment of achievement of outcomes against targets was not possible given that many of the TT and IFIT program targets were ambiguous and reporting against existing targets was inconsistent. Therefore, our evaluation was not able to fully assess the magnitude of progress and impact, particularly with respect to the TT program component.

Outcome Achievement – Findings

The forest sector innovation programs are broadening their reach, but efforts are needed to further expand linkages with small and medium-sized businesses, academia and other sectors

According to the innovation literature, effective collaboration is an essential component of innovation and research programs. The right stakeholders need to be involved to plan, to conduct, to share and to use research results. The literature indicates that SMEs are important to developing transformative technologies, and non-traditional industry partners – including from other sectors (e.g. energy, chemical) – are important to expanding markets, broadening research, and creating an innovation advantage.

Our evaluation found that the various FSI programs are maintaining and strengthening partnerships and networks. Many project files reviewed show involvement of provincial, federal, and forest industry stakeholders. However, efforts are needed to engage smaller companies and other sectors to facilitate a transition towards a forest bioeconomy.

The previous evaluation of the FSI programs contained a recommendation to improve collaboration with SMEs and with non-traditional forest sector partners to facilitate the sector’s transformation. CFS has taken several actions to address this recommendation, but efforts are needed to further broaden reach.

For example, in 2017, NRCan and the Canadian Council of Forest Ministers (CCFM) launched a Forest Bioeconomy Framework. The Framework establishes a path that will result in a forest bioeconomy that uses sustainable biomaterials from healthy forests for high value-added manufacturing; this is intended to be achieved through forest sector innovation, collaboration, and investment. CFS is also working towards a broad bioeconomy strategy for 2019, in coordination with other federal departments and agencies.

Interviewees and documents indicate that science continues to be siloed within CFS, and with FPInnovations, and other federal labs, particularly the National Research Council. However, there have been reported planning and coordination improvements between NRCan and FPInnnovations with respect to bioenergy.

In 2014, IFIT added an additional applicant stream for funding based on company size, which provided a simplified process for smaller companies (i.e., companies with fewer than 100 employees and $50M/year in sales) with lower dollar-value projects. Since then, IFIT project data shows that there has been an increase in the number of SME applications to IFIT. However, an examination of IFIT documentation revealed that only four of the 20 projects funded during this evaluation period had partners from non-traditional sectors.

IFIT has funded a larger proportion of SME costs than those of large companies. Of 20 large company projects, IFIT funded only 18% of the total project costs, while industry provided 61%. As regards 11 projects with SME proponents, IFIT funded 40% of the total costs, and industry provided 40%.

While these design modifications have helped the program reach more SMEs, some interviewees suggest that more extensive outreach targeting other sectors and SMEs is required.

FPInnovations has implemented initiatives to improve reach to SMEs, but its membership structure is perceived as limiting the reach of the TT program

Similar to the previous evaluation, interview, survey and document evidence indicates that FPInnovations membership structure limited the reach of its programs, including the TT program. An analysis of FPInnovations’ 2017-18 industry membership conducted for this evaluation shows that the proportion of memberships of other sectors and smaller firms is limited. Respondents to our survey of industry stakeholders, the majority of which were SMEs, most frequently cited the cost of membership as a key barrier to joining FPInnovations. At the time of this evaluation, FPInnovations was undertaking a review of its membership structure.

To improve its accessibility to SMEs, FPInnovations launched an Associates web-based program in 2015 designed to align with the needs of wood manufacturing SMEs across Canada and to provide quick access to FPInnovations experts and information. However, there is no information in the FPInnovation performance reports as to the extent of use of this program.

There is a need to strengthen collaborations and linkages between FPInnovations and academia.

Academia performs research and development activities in a number of disciplines that are relevant to forest sector transformation. Therefore, it is important to optimize this linkage to ensure sufficient knowledge exchange between the key stakeholder groups.

From 2010 to 2015, university representatives were connected to FPInnovations from the eight forest sector R&D networks that were set up by the Natural Sciences and Engineering Research Council of Canada (NSERC). These network grants were awarded to successful applicants for a 5-year period to fund multidisciplinary research projects in research areas that required a network approach.

A few academic stakeholders cited the need for specific funding directed to academic and FPInnovations partnerships and broader networks that include industry. FPInnovations Annual Performance Reports show a decline in 2016-17 in the number of projects involving collaborations with academics. Other suggestions for improving linkages include a sabbatical program for FPInnovations, and work exchanges between FPInnovations and academia.

Regarding CWFC, interview and case study evidence shows good engagement with industry, provinces and academia. Program documents indicate that CWFC regularly collaborates with over 70 stakeholders from across the forest innovation spectrum. Based on CWFC project data, CWFC leveraged $2.5 (includes both funds and in-kind) from other partners for every dollar spent from 2013 to 2016.

CWFC was the leader in the development and establishment of the Assessment of Wood Attributes from Remote Sensing (AWARE) network, an NSERC-led collaborative research and development project for universities, government and industry, which provincial and industry stakeholders describe as a very effective network.

Some external stakeholders noted that CWFC could further broaden its reach to a wider circle of academic and other stakeholders, as well as to provincial representatives at senior management and policy levels. CWFC is currently undertaking various strategies to further strengthen its collaborations. For example, CWFC plans to support the development of a national research initiative (production of resilient forests to respond to climate change issues) to be co-led with FPInnovations and in collaboration with a broad range of stakeholders.

The forest sector innovation programs are advancing knowledge, technologies, and processes

Evidence suggests that the FSI programs have resulted in new or enhanced knowledge, products, processes, and technologies. In addition, the evidence suggests that the programs have resulted in the pilot and commercial scale-up of several technologies, particularly in the areas of biomaterials, advanced building systems, and improved processes and tools aimed at increasing operational efficiencies.

Evidence indicates that IFIT has supported the scale-up of advanced technologies to a large extent. The following figure shows that a substantial portion of IFIT projects (42%) involve world first innovations and that 36% of IFIT projects have demonstrated the manufacture of non-traditional products.

The evaluation found examples of new technologies and processes produced through the TT program funding. For example, the program component supported the development of multi-functional wood panels, which are able to replace wood-frame walls in construction while not requiring additional insulation, reducing construction costs by 6% to 7%, and shortening construction time by approximately half a day. Similarly, substituting kraft lignin for petroleum-based resin at softwood plywood mills is expected to improve resin efficiency and panel product quality while reducing production costs by $500,000 annually at a 15% replacement rate (substitution rate of lignin), or more than $1M at a 25% replacement rate.

However, given insufficient performance information, it was not possible to assess the extent to which new TT technologies progressed. CFS interviewees note that there are plans to implement a quantifiable measurement scale known as technology readiness levels (TRLs) that will measure how much the technology advances towards commercialization. As of 2018-19, TRLs were not included in TT performance reports.

There have been advancements in bioproducts research, but these advancements have been slower than anticipated

Case study, document, and interview evidence shows progress with respect to improvements in production processes and in developing new applications for novel bioproducts, such as cellulose nanocrystals and lignin-based products. These types of bioproducts are essentially wood fibres that are broken down to the smallest of scales through specialized patented processes. These bioproducts have the potential for a wide variety of applications, including in construction products, aerospace materials, food additives, medical and pharmaceutical applications, and paints and coatings.

Case study and interview evidence indicates that while research pertaining to bioproduct applications has progressed, this research has advanced at a slower pace than expected. These delays were attributed to gaps in later stage research directed at developing product applications. Some interviewees also noted the tendency to underestimate the difficulty of moving from research to commercialization, especially for projects involving advanced biomaterials. Some federal, industry and provincial stakeholders perceived that there is an information gap between the developers of niche value-added forest products and intended end-users, and that the sector needs to improve its understanding of the value chain of these products, starting with an understanding of end-user needs.

FPInnovations’ intellectual property (IP) policy was also perceived by industry stakeholders to slow the pace of technology adoption. FPInnovations provides a “black-out period” of three years during which it only provides licenses to its members. In addition, interviewees noted that industry was reticent to become involved in projects if they do not have clear ownership of the IP. FPInnovations has recently revised its IP policy in an effort to improve industry engagement.

FPInnovations research on cellulose nanocrystals (CNCs) is considered by experts and stakeholders to be an outstanding technical achievement. CNCs are a natural, recyclable, non-toxic nanomaterial that is extracted from cellulose using a chemical process. CNCs’ unique properties present opportunities for its use in a variety of industrial sectors including forestry, aerospace, oil and gas, and pharmaceuticals. In the past five years, FPInnovations has adopted a more focused research approach, which interviewees and documents indicate has led to more accelerated CNC product development and new partnerships with industry.

The CNC case also illustrates the amount of time needed to adopt novel materials in commercial products (at least 10 years according to experts). One of the key hurdles that FPInnovations, supported by TT funding, overcame was the production of sufficient CNC to facilitate product applications research. The adoption of CNCs also faces significant regulatory and standards-related challenges. As a result, CFS and FPInnovations have focused more effort over time on standards development, not only for CNCs, but also for other biomaterials.

A challenge to market development is the need to work one-on-one with potential end-users to demonstrate the advantages and practicalities of incorporating CNC into existing products. Some external stakeholders noted that ongoing funding for product applications research is needed in addition to capital costs. This type of research was cited as high-risk, time-consuming and resource intensive.

The forest sector innovation programs undertake a variety of knowledge transfer activities, but there remain issues with industry receptor capacity

Document, interview and case study evidence indicates that CWFC, TT, FRII, and IFIT are conducting a range of knowledge transfer activities. Based on a recommendation in the previous evaluation to increase communication of results to enhance likelihood of replication of IFIT projects, IFIT has increased its dissemination efforts. For example, to encourage adoption of IFIT supported technologies by other firms, IFIT has posted project information sheets and fact sheets on technologies “worth replicating”; the extent to which this posted information is used however is not known.

With respect to FPInnovations, industry and other stakeholder members have exclusive access to research reports and publications, tools, guides and manuals, needs analysis, hardware and software solutions and leading-edge technology. Industry members can access a network of experts, and receive free consultations, technical advice, and technical updates.

In addition, knowledge exchange specialists are embedded in FPInnovations and CWFC activities. Most of these activities are done by FPInnovations, but also involve other players such as the Canadian Woodlands Forum, which has access to a large network of industry stakeholders. CWFC also leads regular cross-country meetings with provincial and industry representatives, which stakeholders reported were critical for knowledge exchange and generating interest and commitment for new research and technologies.

However, project document and interview evidence indicates that there is insufficient industry capacity to use the TT program research results. To address this lack of capacity, FPInnovations reports that it has increased its webinars and interaction with industry members.

However, uptake of research, according to interview and case study evidence, not only depends on dissemination activities, but also requires a market pull approach, aligning innovations with end-user needs. The evaluation found examples of effective combination of technology push and market pull approaches – for example advanced buildings systems, as described in the next section of this report.

Forest sector innovation science has contributed to the development of codes and standards, showing the most significant progress in the area of advanced building systems

Document, interview, and case study evidence indicates that the TT program’s research and expertise, as well as FPInnovations’ participation in various code committees, have informed the development of codes and standards largely with respect to advanced building systems and materials. According to interviewees and case study evidence, these efforts facilitate enhanced market access and acceptance of Canadian wood and innovative wood products.

The TT program and the FRII have also informed codes and standards related to engineered wood and other value- added wood products such as the mid-ply wall system, cross-laminated timber (CLT), and prefabricated panels and modules. Many of these products are suitable for use in mid-rise wood frame buildings and other larger commercial buildings combining wood and other building materials. In addition, the Expanding Market Opportunities Program also provides funding to FPInnovations and the NRC for technical research related to advanced building systems.

Interview and documentary evidence indicate that FRII and the TT program’s research on wood durability, preservatives, joints, fire testing, and earthquake design has contributed to changes in the National Building Code of Canada (NBCC) and related standards. This work is often done in collaboration with provinces, the Canadian Wood Council and the National Research Council (NRC). In 2015, the NBCC informed by this research and expertise, introduced prescriptive provisions to allow for mid-rise wood constructed buildings of up to 6-storeys. Various provinces have adopted key elements of the NBCC in their provincial building codes, increasing the allowable number of storeys that can be built from wood from four to six.

Document, interview, and case study evidence indicates that the development of national and international standards for cellulosic nanomaterials and other biomaterials and biochemicals (e.g. lignin) is progressing; however, interviewees note that development of these standards is still in the early stages. FPInnovations research and expertise informed the first Canadian standard developed for the measurement and characterization of cellulosic nanomaterials (i.e., CSA Z5100). Documents report that this served as the basis for Canada to lead on the International Organization for Standardization’s development of a standard for cellulose materials for the characterization of cellulose nanocrystals.

The forest sector innovation programs have informed policy development and forest planning and management decisions

Interview, document and case study evidence suggests that FSI programs are informing forest sector-related management and policy decisions and practice. Documents also indicate that CFS has contributed to knowledge and science advancement related to the bioeconomy, which is expected to contribute to policies and regulations on topics such as biomass availability in Canada, the development of a greenhouse gas (GHG) accounting framework for biomass in Canada, and Environment and Climate Change Canada’s Clean Fuel Standard.

Evidence indicates that FRII, the TT program, and CWFC’s work is also influencing forestry management decisions, practices, and policy development. According to several provincial and industry interviewees, CWFC research, funding, expertise, and coordinating efforts have contributed to forest planning and management decisions at the provincial levels and, to some extent, industrial levels. Case study evidence indicates that enhanced forest inventory techniques have assisted the provinces and the forest industry to manage the forest resource more efficiently, by pointing to high value stands of trees, and allowing them to be harvested more efficiently. Provincial representatives note that CWFC genomics research and expertise has informed their planning decisions. Provincial and industry interviewees and documents suggest that FRII research has had strong influence on transportation policy and practices. For example, FPInnovations’ research has influenced winter weight provincial policies (New Brunswick, Alberta, and Saskatchewan), which allows forest companies to haul heavier loads for longer periods of time.

Other federal government departments and provincial representatives note that access to CFS and FPInnovations expertise informs their own forest sector research and market development programs.

The forest sector innovation programs have resulted in the adoption of research, technologies, and sustainable forestry management practices, but adoption is not yet widespread

The FSI programs have demonstrated progress towards adoption of novel or value-added innovation products or systems, including:

• Mid-rise wood construction: The number of mid-rise wood-constructed buildings completed has increased to nearly 580 buildings in Canada. This has been facilitated by the NBCC code changes and provincial adoption (Ontario, Quebec, and Alberta) of elements of the national code, allowing an increase in wood frame buildings from four to six storeys.
• Cross-laminated timber: The program facilitated commercial production of cross-laminated timber in Canada through research, pilot plant support, research for the development of related codes, and the creation of guidance manuals for industry use. However, the manufacturing supply of CLT continues to be limited in Canada.
• Enhanced Forest Inventory: The programs have resulted in provincial and industry uptake of enhanced forest inventory tools and practices.

Interview, survey and case study evidence indicates that adoption of incremental and process innovations is occurring. Industry stakeholders and survey respondents frequently reported the implementation of techniques to optimize mill energy use, manufacturing, harvesting, transportation, operational efficiencies, and safety. However, the extent to which these innovations have been implemented is unclear.

Evaluation evidence suggests that IFIT supported technologies have not yet been widely adopted by other companies. Many interviewees and survey respondents indicate that it is generally too early for the achievement of this outcome. The 2015 IFIT-commissioned study on the replicability potential of IFIT projects as well as later analysis identified 15 IFIT projects that have good potential for replication. Our evaluation’s survey of industry stakeholders also shows replication potential of IFIT projects, as more than half of our survey respondents that received IFIT funding indicated that other companies have expressed an interest in replicating the technology. Interviewees noted barriers to replication such as the inability of novel products to compete with existing products in terms of demonstrated performance, quality, or cost advantages (e.g. cost of producing products is comparatively high).

In terms of adoption, CWFC has made reported gains with respect to their work on enhanced forest inventory. As of 2019, more than 30 million hectares of forest cover have LiDAR coverage, representing almost 10% of Canada’s forest cover. More than 50 million hectares is in the planning or proposed stages for LiDAR acquisition. Interest and uptake have been particularly strong in Atlantic Canada, Alberta, Quebec and Ontario. Provinces and industry representatives report that CWFC has strongly influenced uptake by contributing research and expertise; by supporting various test sites across the country; by playing a strong brokering and coordination role; and through its knowledge exchange activities.

EFI test sites supplied economic data on cost savings and benefits. These cost savings were associated with better decisions such as those related to harvesting plans using LiDAR-enhanced forest inventory data. For example, at the Ontario site, $400,000 was saved annually with an estimated payback time (to cover costs of flying LiDAR) estimated to be as short as two years.

Economic benefits have occurred at the firm level, but they are not yet widespread

Some economic impacts of the program have been observed at the firm level, therefore suggesting that the programs are paving the way for long-term improved forest sector competitiveness. However, it is likely too soon to observe the effects of FSI-supported innovation on the forest sector’s competitiveness, particularly for transformative innovations. FPInnovations has licensed 21 of its technologies to various firms, although the extent of sales emanating from these licenses is not available in FPInnovations’ performance reports.

Although FIP and FRII do not report on overall realized cost savings, there are examples of work undertaken through these programs, which indicate potential and realized cost savings. For example, from 2012 to 2018, FPInnovations, CanmetENERGY, and CFS collaborated on studies to optimize energy use in pulp and paper mills, and identified over $55M in prospective energy savings and additional power generation opportunities in 15 pulp and paper mills located in seven provinces. As of 2018, 40% of the identified potential savings were realized by 10 of the 15 mills that implemented the optimization techniques, representing a total of $8.9M per year in energy savings and additional revenues.

IFIT is making progress towards longer-term outcomes with completed projects reporting early economic benefits

IFIT targets later stages of the innovation continuum and, as such, is the only FSI program that tracks information related to the economic benefits experienced by its recipients. As of 2017-18, 20 of 31 implemented IFIT projects were successfully completed, increasing the capacity of 14 facilities to produce 97 new or diversified products and implementing process improvements in six facilities.

While some progress is occurring in the advancement and production of higher-value wood products, the forest sector continues to rely heavily on commodity products

This section examines import and export trends of value-added and commodity wood and pulp and paper products. However, these trends cannot be directly attributable to FSI programs. Moreover, it is too early to realize economic impacts, particularly with respect to transformative research.

The evaluation team’s analysis defined commodity wood products as those that have undergone little transformation to create finished products, whereas value-added products have undergone further manufacturing. Similarly, commodity pulp and paper products include pulp that has not been used to manufacture paper or other products, as well as scrap paper and paperboard. Value-added pulp and paper products have undergone further manufacturing into finished products for consumers, including various types of papers and their uses, as well as hygiene products based on forest fibres.

Status Report: Reported Early Economic Benefits of IFIT projects (2 years post project completion)

Indicator	Anticipated (31 projects)	Actual to date (n=# of projects reported to date)
Average Return on Investment (ROI)	22%	31% (n=12)
Average Return on Capital Equity (ROCE)	29% (n=18)	23% (n=8)
Total new revenue generated	$367M	~$85M (n=12)
Total cost benefits and savings	n/a	~$13.5M (n=8)
Total new jobs created	359	185 (n=14)
Total jobs secured	4,915	2,153 (n=18)
Temporary employment (hrs)	1,056,585	341,444 (n=14)
Total # of new or diversified products	45 (n=29)	97 (n=13)
New patents	n/a	41
Note: The Table shows good progress made on several indicators. Keep in mind that not all projects are completed, nor have all the completed projects had sufficient time to fully realize benefits.

The evaluation team also found that assessment of product diversification, particularly with respect to novel products was challenging. Current systems for product classification and categorization make it difficult to identify production and export trends for specific products such as cross-laminated timber. In addition, new bioproducts use components from a variety of sectors, leading to a lack of coherent trend data on these products.

Since 2009, exports for commodity and value-added wood, pulp and paper products have increased. However, despite the forest sector’s continuing recovery from the 2008 recession, as evidenced by the increase in the total value of its exports from $24.6 billion in 2009 to $37 billion in 2017, the sector still relies heavily on its commodity products as sources of revenue.

The forest sector has exported more value-added wood products in the last decade than it imported. However, when excluding newsprint from consideration (in which exports continue to decline), the sector has had a continuous trade deficit for its value-added pulp and paper products. Moreover, the vast majority of pulp and paper mill revenues continues to be derived from traditional products. This indicates that there is room for growth for the sector to develop novel high-value pulp and paper products to offset losses in demand for traditional products in this area, as well as to expand into novel markets for fibre-based products.

On the wood products side, Canada maintains a trade surplus in several high-value traditional products, such as oriented strand board (OSB), construction products (windows, doors, shingles, shakes, panels, and construction joiners), plywood and veneer sheets, and particle board. The exception to this is fibreboard, in which there is a significant trade deficit.

There are innovation gaps impeding the achievement of longer-term FSI outcomes

Gap between lab-scale R&D and projects supported by IFIT

As noted by several CFS and external interviewees, research on biomaterials requires continuing, more advanced research to explore various applications for these materials. The TT program is not designed to address this gap as it funds non-proprietary, earlier stage, collaborative research. According to a CFS representative, this type of research would help take advantage of commercial-scale facilities that can produce these bioproducts but do not yet have a large enough market in Canada.

Gap in market research and support for novel forest products

Several external stakeholders report the need for market development for novel forest products. Note that there is a specific recommendation pertaining to this issue in the Evaluation of the Expanding Market Opportunities Program. However, interviewees caution that some novel products are not yet ready for market development as additional product application research is required. CFS and some external stakeholders also noted that there is a gap in market development support at the firm level. External stakeholders also noted that there is a need for more market research to better understand the potential of forest-based biomaterials.

Need for consistent and adequate supply of biomaterials and other novel forest products

Several stakeholders note that additional funding beyond IFIT’s support for ‘first in Canada’ projects is needed to build the supply of novel forest products. Take-up of the novel product is constrained because producers and other stakeholders are unwilling to replace existing supplies with a product that only has one supplier.

Industry and provincial interviewees also note there is a need for an enhanced market access program, which would require improved links to customers, who are increasingly outside the pulp and paper industry and less likely to be part of established networks such as industry associations.

There are plausible linkages between environmental performance and the forest sector innovation programs

As with economic impacts, it is too early to realize more widespread environmental outcomes, with respect to improved environmental sustainability. However, the literature, document and interview evidence suggests that there are plausible linkages between this outcome and FSI program activities. Document and interview evidence indicates that knowledge and expertise from FRII, the TT program and CWFC is being used for decisions relating to sustainable development.

While IFIT monitors environmental benefits, IFIT foundational documents indicate that in 2017 no targets or outcomes relating to environmental benefits were established. However, the objective for IFIT, as stated in its Terms and Conditions is to support forest industry transformation that will make the forest industry more commercially and environmentally sustainable.

According to the literature review, research such as that conducted through FSI, has the potential to contribute to both climate adaptation and mitigation strategies. The literature, documents and interview evidence indicate that the Canadian forest bioeconomy is poised to play a key role in the transition to a modern low-carbon economy. Forest-based bioproducts have the potential not only to improve competitiveness, but also to lower GHG emissions, particularly when wood product waste is used to replace petroleum products.

Between 2005 and 2015, NRCan notes that the forest industry has achieved both reductions in energy use through greater efficiencies and reduction in GHG emissions by changing its fuel mix (i.e. bioenergy is accounting for an increasing share of the energy mix). It was also noted that decreased production and the decline of the pulp and paper industry during this time also contributed to this trend. According to program documents, wood-based materials use less energy and emit fewer GHGs over their life cycle in comparison to other more traditional building materials, therefore reducing the overall carbon footprint of wood-constructed buildings.

Evidence suggests that the observed immediate and intermediate results are attributable, to a good extent, to FSI program funding, and to several program design and delivery features

Interview, survey, and case study evidence suggests that in the absence of FSI funding, the outcomes would not likely have occurred, or would not occurred to the same extent.

According to interview and case study evidence, internal factors positively influencing effectiveness include the programs’ collaborative model, CFS leadership and its active role in facilitating partnerships and networks, CFS and FPInnovations’ expertise and long-standing experience.

Case studies show that long-term and consistent funding played a critical role in the development of biomaterials and product applications for these materials (e.g. cellulose nanocrystals, cellulose filaments).

External Factors Influencing Outcomes

Documents and interview evidence pointed to a number of external factors that are impeding outcome achievement. With respect to biofuels, the absence of a Clean Fuel Standard to generate demand for biofuel was noted by interviewees to significantly impede adoption of biofuel technologies. Other factors include a declining fibre supply (which also creates opportunities for research), and market factors such as the uncertainty created by a lack of Canada-U.S. softwood lumber agreement.

Internal Factors	External Factors
CFS and FPInnovations experience and expertise FPInnovations model for pre-commercial R&D Embedded knowledge exchange (FPInnovations, CWFC) and effective dissemination channels CFS brokering role Collaborative model, broad-spectrum approach, IFIT funding model, CWFC regional delivery	Forest Bioeconomy Framework for Canada Climate change actions Growing interest in green buildings Supportive provincial policies and funding
Balance of incremental vs. tranformative R&D? CWFC visibility; lack of clear linkages to FPI (expected to improve – national research initiative is being developed and co-led with FPInnovations) FPInnovations membership and advisory processes impeding reach Evidence of siloed research, but improving – Canmet, FPInnovations and CFS engaged in planning and coordination – bioenergy/ biorefineries	Industry receptor capacity, short term focus and declining R&D investments Support gaps for advanced research, market research for non-traditional forest products Market factors (softwood lumber dispute) Regulatory environment (e.g., absence of Clean Fuel Standard) Climate change impacts, fibre supply issues Proximity to U.S.; potential loss of IP to U.S.

Research findings on the forest sector in Canada and the U.S. describe the culture of the industry as conservative and identify that the industry has not sufficiently invested in research and innovation. Previous NRCan evaluations have pointed to the risk averse nature of the Canadian forest industry, which can serve as an impediment to innovation and transformation of the forest sector.

According to an analysis of the sector, the volatility of commodity prices makes it difficult for forest-product firms to plan their future, biasing them towards production rather than innovation when prices are high, and results in the sector’s reluctance to invest in innovative activities when prices are low. In addition, Canadian forest companies are heavily reliant on suppliers as a source of information for innovation, further limiting their receptor capacity.

Performance - Efficiency

Forest sector innovation programs are generally being implemented as planned

Evaluation evidence suggests that the FSI projects are generally implemented as planned, although some delays were noted. With respect to the TT program and FRII, delays were primarily attributed to partner coordination issues or to unforeseen technical challenges.

CWFC, the TT program and FRII project performance reports did not provide clear information as to the status of deliverables and milestones, making this assessment challenging. Moreover, a file review of randomly selected TT projects did not yield clear results. Of the 30 TT projects, 13 indicated that they were on track although some of these 13 project reports noted challenges or had revised their tasks. Seventeen (17) of the 30 projects reported that they were partially on track. It was not possible to assess the implications of delays or changes in tasks based on the project reports because this type of information was not included. Moreover, the reports did not consistently indicate mitigation strategies to address delivery issues. In some cases when there were changes in partners’ priorities, the project scope was reduced.

Evidence from program documents indicates that for the most part IFIT has been implemented as planned, producing planned outputs, although the program did lapse some funding in 2013-14. IFIT signed 20 new contribution agreements with companies this evaluation period, 8 of which are complete and the remainder are on track. Since inception in 2010, IFIT has had 32 signed contribution agreements with companies. As of October 2018, one was unsuccessful (previous evaluation cycle), and 20 have been completed.

FIP (TT and CWFC) is planned with guidance from various bodies, including the Science Advisory Committee portfolio review committees (pertaining to CWFC), and the Program Advisory Committees, which bring together representatives from federal, provincial, and territorial governments, industry, and academia. As roll-up information on changes to program components was not available for the document review, the evaluation team’s analysis was not able to conclude on the degree to which actual program implementation differed from the original design. The document review did find evidence, however, that the CFS undertakes periodic assessments of its research directions, workplans, and progress reports.

The total actual expenditures for FSI programs, from 2013-14 to 2017-18, was $271.4M compared to $282.3M planned expenditures. Actual spending for the TT program was approximately $8M higher than planned spending from 2013-14 to 2017-18.The most significant lapse in funds ($17.5M) occurred in 2013-14 for IFIT. In that year a project withdrew at the last minute because of an information request made by the proponent’s funding partner. (This project was funded in the next Call for Proposals). IFIT subsequently made changes to its due diligence process to minimize the risk of this reoccurring.

Almost 87% of the actual FSI expenditures were allocated to grants and contributions (G&Cs). The ratio of administration costs (operations and maintenance (O&M) and salaries and employee benefit plans (EBP)) to G&C expenditures suggests efficient administration of funds.

In addition, Budget 2014 permitted IFIT to establish a 4-year funding profile that was better able to reflect variation in costs associated with ramping up, implementing, and ramping down IFIT projects. This approach was better aligned with the realities of funding IFIT projects that are typically large scale, capital intensive, and technologically complex.

From 2010 to the 2017 renewal, the program successfully received an exemption (Section 6.4.4 on the Directive on Transfer Payments) that permitted the department to advance payments to be made in a fiscal year to cover the federal government's share of expected eligible expenditures to be incurred by the recipient through to September of the following fiscal year. This exemption was removed in the 2017 T&Cs on the request of the TBS “to ensure that companies could make the capital investments and progress payments required to viably manage these large-scale, complex projects.”

To improve this situation, a CFS representative suggested that Canada’s contribution to a project be under a holdback line item to commit that money in the fiscal year in which a contribution agreement is signed, regardless of when the eligible expenditures are actually incurred. The held back amount would then be released once claims are submitted by the recipient over the course of the project. This would allow NRCan and the recipients to better manage cash and reduce administrative burden in having to strictly manage fiscal year amounts.

There are a number of positive elements that appear to enhance the efficiency of the forest sector innovation programs, although changes are needed to improve the programs’ reach

Evaluation evidence suggest that the program design is an efficient means of producing its intended outputs and outcomes. The broad array of programs along the innovation continuum and its collaborative model are considered by stakeholders to be key strengths of the program. However, the current program design is perceived to be insufficient for engaging some stakeholder groups such as SMEs. Various suggestions were made such as the modification of FPInnovations advisory and membership structures; dedicated funding for partnerships (e.g. FPInnovations/ academia).

The literature indicates that the success of forestry bioeconomy initiatives is not only dependent on R&D funding but on external institutional supports such as knowledge transfer, capacity building, and horizontal partnerships. Canada, like other jurisdictions implementing bioeconomy strategies is constrained by gaps in supply chains or missing actors in the value chain.

According to the literature and a forestry expert, Norway, Sweden and Finland have experienced success in deploying a tripartite (industry/government/academia) approach to accelerate their respective forest sector innovation agendas. Many of the companies within the Finnish and Swedish forest products sectors have moved up the production value chain, with governments playing a key role in facilitating this transition. Finnish government programs in support of the country’s bioeconomy strategy are characterized by an emphasis on the balance between traditional processes and products and high-value niche products.

Further, the literature review identifies various factors that are associated with successful innovation:

• Encouraging exchanges between industry and other key stakeholders.
• The availability of longer-term dedicated resources was frequently found to be a key facilitating factor for forest sector innovation. Among Nordic countries, funding levels for innovation initiatives are well-established and available over long periods of time (up to ten years). Sweden and the U.S. provide longer-term financial support targeting domestic large-scale demonstration and commercialization of innovations; and
• Engaging end-users in earlier stages in public-private innovation endeavours.

External stakeholders perceived that the sector-based approach to innovation is appropriate, but the transition to the bioeconomy will require continued integration and coordination efforts

The programs’ sector focus was perceived by interviewees to be important given the significance of the forest to the Canadian economy, including rural economies. Without the sector focus, some interviewees noted that there is a risk of underfunding forest sector innovation in favour of other energy and clean technology innovation endeavours. In addition, external stakeholders noted that the sector specific approach was appropriate because the broad range of CFS programs from harvesting to production means that there is specialized expertise in all these areas, allowing for a holistic approach to forest sector innovation.

However, documents and interviewees noted that the transition to the bioeconomy will require more integration and coordination with other sectors. There may be opportunities to ensure coordinated access to the FSI programs while still maintaining a sector specific approach. The Clean Energy for Rural and Remote Communities was cited as an example of a program where the one-window approach works well as different streams are run by different sectors. Some external stakeholders suggested, particularly with respect to the bioeconomy, the need for a convenor to help guide programs across departments. CFS participates as required in the DG Project Review Committee with respect to the Clean Growth Hub. Moreover, at the time of the evaluation, NRCan was developing a bioeconomy strategy in coordination with other federal departments and agencies.

CFS has good strategic planning and priority setting processes

CFS interviewees and documents indicate that CFS regularly conducts stakeholder consultations prior to planning and renewal processes. For example, as part of the CFS portfolio management and planning, CFS conducted stakeholder consultations in 2016 and 2017 to identify issues, challenges, and opportunities facing the forest sector. Consultations were conducted with staff, academia, provinces, industry, industry associations (e.g. Forest Products Association of Canada), research institutes, and other government departments. In addition, engagement with stakeholders regarding the implications of climate change was undertaken. These consultations helped to articulate key priorities and the CFS ten-year research agenda.

According to interview and documentary evidence, CFS has demonstrated continual learning in delivery of its innovation programs. The programs have incorporated lessons learned and best practices into their renewal processes.

FPInnovations’ challenges with respect to their advisory and planning processes increase the risk of misalignment between the Transformative Technologies program and long-term industry needs

With respect to FPInnovations, the annual research plan, which includes proposed TT activities, is endorsed by its National Research Advisory Committee, which has representation from the forest sector innovation system. Program Advisory Committee meetings are held twice a year for each strategy, and industry firms can become directly involved through these committees. FPInnovations submits an annual work plan to CFS which contains information on each proposed project.

However, interviews and documents indicate challenges that have impeded long-term strategic planning particularly with respect to fully engaging FPInnovations industry members in the advisory and planning processes. In addition, FPInnovations is not broadly representative of the forest sector, although interviewees point out it has better representation on the pulp and paper side.

A few industry and provincial stakeholders noted that FPInnovations organizational restructuring during the evaluation period was creating a sense of instability and impeding planning. These Industry and provincial stakeholders perceive that FPInnovations needs a more collaborative model for priority setting. It was suggested that there needs to be better engagement with provincial associations and research institutes to obtain more regional perspectives. Some stakeholders indicated that outreach could be done by other organizations in the lead with the federal government facilitating these discussions, allowing FPInnovations to focus on research.

NRCan and external interviewees noted that FPInnovations priorities need to align with their areas of strengths and that in the recent past they have been “stretching themselves too thin”. It was suggested that FPInnovations partner more with universities, federal government departments, and provincial research institutes and lead in those areas in keeping with their strengths.

These challenges were perceived by stakeholders to have negative implications for the TT program increasing the risks of its misalignment with long-term industry needs.

Documents indicate, that prior to 2016, better market data and more active foresighting activities (i.e. more engagement on the front end) were needed with respect to FPInnovations planning. However, there is evidence to suggest that this situation is improving – as part of the TTP Pathways Project, foresight work teams aim to include other stakeholders such as academia, federal and provincial representatives. The Future Foresight work (on strategic priority areas) is also expected to improve planning and program development efforts commencing in 2018-19. It should also be noted that FPInnovations advisory and planning processes were under revisions during the timeframe of the evaluation and have recently been restructured.

According to program documentation and interview evidence, at the strategic level there appears to be unclear linkages between CWFC and FPInnovations. Interviewees attributed this to lack of clarity as to the role of CWFC with respect to FPInnovations and inadequate communication mechanisms between the two organizations. CWFC is in the process (2018-19) of developing a national research initiative comprised of provincial, academic, non-government organizations and industry. It will be co-led by CWFC and FPInnovations. CFS representatives indicate that this will help to strengthen communications with FPInnovations and enhance role clarity.

The periodic IFIT call for proposals creates challenges for optimal selection of complex projects

According to documents and interviews, the periodic IFIT Call for Proposals process creates challenges as it may constrain the optimal selection of projects. Some stakeholders noted that the periodic Call for Proposals means that some projects are not ‘shovel ready’ during the CFP process. This increases the risk of selecting projects based on ability to meet timelines (projects that are lower risk or have shorter timelines) as opposed to the best projects. Based on the recommendations of the previous evaluation – IFIT did implement a longer duration for the third and fourth Call for Proposals. However, some external stakeholders suggested that this has not sufficiently resolved the issue. Interviewees suggested that an ongoing intake would address this situation and according to a CFS representative would allow CFS more time to conduct outreach.

External and CFS representatives also suggested that novel approaches are needed to attract more high-risk IFIT proposals – “to push the envelope even further.” Suggested strategies include creating more challenge-based contests similar to Impact Canada; utilizing a phased funding approach for very high-risk proposals; and taking greater risks on smaller projects.

With the exception of IFIT, project reporting focuses on technical progress and output information

CWFC, the TT program, and FRII report progress on activities, accomplishments, and deliverables with a focus on technical progress, such as output information on publications and presentations. These reports lacked clarity on project status and progress. Program reports provided information on specific projects (e.g. FIP Annual Reports), but lacked overall aggregate reporting on program status and overall progress with respect to outputs and outcomes.

In general, IFIT monitors its key outputs and results. It tracks appropriate indicators such as leveraging, number of products and patents developed, economic indicators such as Return on Investment and Return on Capital Expenditures, new revenue generated, jobs secured and generated, and environmental indicators, such as energy produced, water saved, and GHG emissions reduced. The evaluation did identify some data quality issues however, such as data entry and calculation errors.

In terms of performance reporting, IFIT reports should consistently and clearly specify whether the results are anticipated or actual. In addition, aggregated results should indicate the number of projects reporting results.

CFS representatives noted capacity issues in maintaining and updating the IFIT databases. They noted that an automated monitoring and reporting system would help to reduce the reporting burden for CFS and funding recipients and reduce the opportunity for data entry errors.

FIP (TT and CWFC) has contributed to and produced numerous guides, tools and information products related to the design and construction of mid- and high-rise buildings, cross-laminated timber, and enhanced forest inventory. FPInnovations has also developed a number of decision support tools for forest operations aimed at supporting decisions and improving efficiencies. However, performance information is lacking on the value, uptake and use of these tools by industry (e.g. Biomass Utilization App, FPSuite). While interviewees provided positive comments on several of these products, their use and uptake are not reported.

The programs require revisions to performance measurement systems to include the integration of considerations for gender-based analysis plus to ensure that the program collects the information it needs to support efforts aimed at increasing equity and diversity in the forest sector.

Conclusions

Relevance

There is a continued need for the broad suite of forest sector innovation programs to help transition the traditional wood, pulp, and paper industry to the bioeconomy. Forests are recognised as a significant contributor to development of a bioeconomy as a source of renewable biomass. The forest sector innovation programs are aligned with sustainable economic development and climate change adaptation and mitigation strategies.

CFS should play a more influential role with respect to the TT program planning processes to ensure that there is sufficient emphasis on transformative innovation to support the transition to the bioeconomy. FPInnovations’ planning processes have limited the role that CFS plays in ensuring that the TT workplan sufficiently reflects an optimal balance of transformative and incremental activities.

Recommendation 1: The Canadian Forest Service should play a stronger role to ensure that the Transformative Technologies program has an optimal balance of transformative and incremental innovation activities to support the transition to the bioeconomy.

Performance

The FSI programs are making good progress towards their intended outcomes with respect to codes and standards development, and developing and improving technologies, knowledge, processes and tools. While the evaluation did not find instances of widespread adoption of new technologies and practices, progress has been made regarding advanced building systems (increase in the number of mid-rise wood frame buildings), the production of higher value wood products, and the uptake of enhanced forest inventory tools.

While CFS has made inroads into broadening its engagement, transitioning to a bioeconomy will require efforts to expand linkages with a range of stakeholders, particularly small and medium-sized enterprises (SMEs), other sectors, and academia.

Several design and delivery elements are positively contributing to the programs’ efficiency and effectiveness. For example, FPInnovations provides a hub for research and the Transformative Technologies program, delivered by FPInnovations, has yielded good results such as the advancement of novel bioproducts. However, its membership structure, which was undergoing revisions at the time of the evaluation, limited the reach of the TT program, particularly with respect to SMEs.

Recommendation 2: The Canadian Forest Service should work with its partners to further broaden the programs’ reach and increase collaboration with SMEs, academia, and other sectors. The Canadian Forest Service should consider and implement novel approaches for pilot projects to increase collaboration and knowledge transfer between industry, FPInnovations, and academia.

Pre-commercialization innovation gaps are impeding adoption of novel forest products. There is a need for more advanced research to test novel bioproduct applications and for more market research for novel forest products.

Recommendation 3: The Canadian Forest Service should work with partners to advance projects that will address pre-commercialization innovation gaps.

Better metrics and methods are needed to measure the effectiveness of FSI programs (TT program and FRII). The TT program, FRII and CWFC programs need to clearly and consistently report on progress of their deliverables and overall results. The evaluation had challenges assessing the extent to which projects and technologies had advanced. Moreover, the programs produce numerous technical guidance documents and decision-making tools, but their use and uptake is not reported.

Recommendation 4: The Canadian Forest Service should improve its performance measurement and reporting of its programs and projects:

1. Identify and implement valid metrics, targets and methods for measuring key outcomes of innovation programs (e.g. technology advancement, value and use of knowledge products and decision-support tools).
2. Reports should clearly, accurately and consistently report on the status of milestones and progress at the project and program levels.

IFIT is an effective program that funds large, complex projects. Over the course of the evaluation period, CFS has implemented strategies to minimize the risk of carrying-over funding from one fiscal year to the next. It has also expanded the time period for the Call for Proposals. However, the evaluation found that further flexibility is required to manage a complex innovation program and to facilitate optimal project selection. Project selection would also be enhanced through the consideration and adoption of novel approaches aimed at attracting more high-risk projects.

Recommendation 5: The Canadian Forest Service should work with Treasury Board to enhance elements of IFIT’s design and delivery. These changes would facilitate optimal project selection, encourage high risks projects, and improve the management of complex innovation projects:

1. Consider new, more flexible models of funding intake.
2. Identify strategies to enhance funding flexibility that will better allow carry-over of funds from one fiscal year to the next.
3. Identify novel strategies to encourage high risk projects

Appendix 1: Evaluation Team

Christian Asselin, Chief Audit and Evaluation Executive (CAEE)
David Ash, Senior Advisor to the CAEE
Zelda Yule, Evaluation Manager
Odilia Maessen, Evaluation Analyst
Doaa Abou Hussein, Evaluation Analyst
Edmund Wolfe, Evaluation Analyst