Evaluation of the Climate Change Adaptation Program

Audit and Evaluation Branch
Natural Resources Canada
Presented to the Performance Measurement, Evaluation and Experimentation Committee (PMEEC)
February 13, 2023

Table of Contents

• List of Tables and Figures
• List of Acronyms
• Executive Summary
• Introduction
  • Program Profile
    • Program Description
    • Program Expected Results
    • Program Governance
    • Program Resources
    • Program Operating Context
  • Evaluation Objectives, Methods, and Limitations.
    • Evaluation Limitations
• What We Found
  • Efficiency – Does the Program model support the achievement of Program objectives?
    • Evolution and Changes to the Program.
    • Program Design and Delivery
    • Unmet Needs, Unintended Outcomes, and Sufficiency of Resources
  • Efficiency – To what extent does the program consider EDI factors?
    • EDI Considerations
  • Effectiveness – To what extent has the program progressed in achieving its intended outcomes?
    • Outputs
    • Immediate Outcomes
    • Intermediate Outcomes
    • Long-term Outcomes
    • Ultimate Outcome
    • Performance Measurement Strategy
• Conclusion
• Appendix A: Evaluation Matrix
• Appendix B: Evaluation Team

List of Tables and Figures

• Table 1: Total Planned Expenditures for CCIAD-AP by Year over the Evaluation Period
• Table 2: Total Actual Expenditures for CCIAD-AP by Year over the Evaluation Period
• Table 3: Total Planned Expenditures for CCGP by Year over the Evaluation Period
• Table 4: Total Actual Expenditures for CCGP by Year over the Evaluation Period
• Table 5: Evaluation Challenges and Mitigation Strategies
• Table 6: Implementation of Recommendations from 2015 Evaluation
• Table 7: Facilitating Factors Influencing CCAP
• Table 8: Challenging Factors Influencing CCAP
• Table 9: Summary of Approaches by Other Countries
• Table 10: Themes Covered by Adaptation Program G&C Projects 2017-2021
• Table 11: Examples of Climate Change Adaptation Measures Implemented by Stakeholders

• Figure 1: CCAP Logic model
• Figure 2: CCAP Organigram
• Figure 3: Four Lines of Evidence
• Figure 4: A Network of Networks
• Figure 5: Lessons Learned from CCGP’s SWOT Analysis Conducted During the 2019 Workshop
• Figure 6: CCGP Spending by Program Areas, 2016-2021
• Figure 7: CCGP Spending by Funding Sources, 2016-2021
• Figure 8: CCIAD-AP Spending by Funding Sources, 2016-2021
• Figure 9: Examples of EDI in practice
• Figure 10: Percent of Stakeholders Reporting the Inclusion of Climate Change Adaptation in Plans and Strategies in NRCan Benchmark Survey, 2012 and 2018

List of Acronyms

AFN	Assembly of First Nations
BRACE	Building Regional Adaptation Capacity and Expertise Program
CCGP	Climate Change Geoscience Program
CCIAD	Climate Change Impacts and Adaptation Division
CCIAD-AP	CCIAD Adaptation Program
CCAP	Climate Change Adaptation Program
CIRNAC	Crown-Indigenous Relations and Northern Affairs Canada
CPA Canada	Chartered Professional Accountants of Canada
DARTT	Director-level Adaptation and Resilience Task Team
DFO	Department of Fisheries and Oceans
DGARC	Director General Adaptation and Resilience Committee
EDI	Equity, Diversity, and Inclusion
ECCC	Environment and Climate Change Canada
EBP	Employee Benefits Plan
FTE	Full time employee
GBA+	Gender-based Analysis Plus
GC	Government of Canada
G&C	Grants and Contributions
GIS	Geographic Information System
GSC	Geological Survey of Canada
HAOB	Hazards, Adaptation and Operation Branch
IPCC	Intergovernmental Panel on Climate Change
LMS	Lands and Minerals Sector
NAS	National Adaptation Strategy
NKA	National Knowledge Assessment
NRCan	Natural Resources Canada
OGDs	Other Government Departments
O&M	Operations and Maintenance
PCF	Pan-Canadian Framework on Climate Change and Clean Growth
PCSP	Polar Continental Shelf Program
PIEVC	Public Infrastructure Engineering Vulnerability Committee
PIN	Permafrost Information Network
Platform / Adaptation Platform	Canada’s Climate Change Adaptation Platform
RAC(s)	Regional Adaptation Collaboratives
SWOT	Strength, Weakness, Opportunity, and Threat
TC	Transport Canada
WG(s)	Working Group(s)

Executive Summary

This report presents the findings, conclusions, and recommendations of the evaluation of the Climate Change Adaptation Program (CCAP) at Natural Resources Canada, which includes the Climate Change Impacts and Adaptation Division’s Adaptation Program (CCIAD-AP) and Climate Change Geoscience Program (CCGP). The evaluation responds to a Treasury Board commitment to evaluate as part of the Horizontal Climate Change Adaptation Initiative led by Environment and Climate Change (ECCC).

NRCan’s Audit and Evaluation Branch undertook the evaluation between April 2020 and December 2021. The evaluation covered the five-year period from 2016-17 to 2020-21. The evaluation focused on program performance and the following three objectives:

1. Assess the program model’s capacity to adapt to the evolving context;
2. Assess whether the program has put in place measures to achieve Equity, Diversity, and Inclusion (EDI) objectives with an emphasis on Indigenous peoples; and
3. Assess the program’s contribution to achieving its intended outcomes with an emphasis on the long-term outcomes.

The evaluation also examined the program’s implementation of recommendations from the previous evaluation, as well as lessons learned, and best practices related to the design and delivery of the program.

What the Evaluation Found

Efficiency and Economy of the Program Model

As climate change impacts are increasing in intensity and frequency, and in light of the federal government’s continued commitments to climate action, CCAP has an important role to play in improving the ability of Canada’s regions, communities and economic sectors to deal with climate change impacts.

The evaluation found that the CCAP model is efficient and economical in supporting the achievement of the program’s objectives. CCAP has evolved to meet the new and changing needs of stakeholders. CCIAD-AP has generally evolved in the design and delivery, such as adding Indigenous organizations as partners to Canada’s Climate Change Adaptation Platform Plenary and enhancing inclusivity and transparency of the National Knowledge Assessment process. Further, the Building Regional Adaptation Capacity and Expertise Program (BRACE) worked with the provinces to identify key priorities for building regional adaptation capacity and expertise and to develop projects consistent with program objectives. However, stakeholders believed that CCIAD-AP has not necessarily evolved as quickly as needed, given the increased severity and frequency of climate change impacts in the last few years, adding that the program could improve its overarching self-assessment process. Similarly, CCGP has evolved in its design and delivery, updating its strategic priorities over time by focusing on Canada’s Arctic region. CCAP implemented all the recommendations from the 2015 evaluation.

The CCAP model generally supports the program objectives as part of the Pan-Canadian Framework on Clean Growth and Climate Change. CCIAD-AP and CCGP each provide unique and important functions while collaborating on issues and activities of common interests. The CCIAD-AP model is effective to facilitate networking opportunities, discussions on emerging issues and priorities, knowledge sharing on cross-cutting issues, and the production of relevant knowledge and tools. The CCGP matrix organization is an effective program structure that enables effective partnerships and collaborations. Findings from the literature review showed that some of the approaches implemented by select countries are similar to the approaches already taken by CCIAD-AP, suggesting that the program is effective in keeping abreast with trends in the field. The evaluation identified best practices, lessons learned, alternative program models, and unintended outcomes that are further explained in the evaluation report.

The evaluation found that both CCIAD-AP and CCGP are efficient in managing their resources. However, the evaluation identified areas for improvement and exploration to further increase the efficiency and economy of the program model. In particular, while the implementation of climate adaptation measures is not part of CCIAD-AP’s existing mandate, stakeholders suggested that CCIAD-AP could act as a catalyst to make meaningful contributions this area. Currently, knowledge about climate change impacts and adaptation is available, but there are still critical gaps and challenges in the practical implementation of climate adaptation measures.

Lastly, the evaluation found that CCAP has considered EDI factors in all its key components. CCIAD-AP and CCGP considered EDI factors in their activities and outputs, including engagement, integration of Indigenous knowledge, and communications. These findings are encouraging but CCAP still has the potential to expand its reach and influence, and CCAP should maintain the momentum to further advance this critical aspect of the programming. In particular, CCIAD-AP should consider opportunities to be more equitable, diverse, and inclusive by further the involvement of Indigenous peoples and other under-represented groups.

Effectiveness in Achieving the Intended Outcome

The evaluation found that the CCAP has produced and disseminated outputs as planned, making increasingly effective use of web-based tools, which is then positively reflected in the achievement of its intended outcomes.

Through CCIAD-AP, target stakeholders have increased awareness of climate change impacts and access to resources to support climate change adaptation actions. Target stakeholders also have an increased capacity to use and apply climate adaptation tools and information in their work. CCIAD-AP has also enabled target stakeholders’ participation in and contribution to the identification of priorities for climate adaptation.

Target stakeholders have also increased awareness and use of CCGP products. CCGP has enabled end-users’ access to knowledge products to support the identification of priorities for climate change preparedness and adaptation. Collaborative research, fieldwork, and partnerships help CCGP scientists to generate scientific knowledge on climate change impacts. CCGP data and products are published on GEOSCAN, Permafrost Information Network, and CanCoast (among other places) to communicate its scientific results to national and international end-users. In 2019, the Geological Survey of Canada (GSC) started to publish annually CCGP’s most significant achievements via the GSC Report on Results and Delivery.

The evaluation found that CCAP is progressing towards its intermediate outcomes. Target stakeholders have identified climate change adaptation measures to address risks and opportunities arising from climate change due to their involvement in CCIAD-AP. These measures span all stages of the climate adaptation continuum to reflect the nature of the stakeholders’ work and level of advancement in integrating climate adaptation. Through CCGP, decision-makers from various levels of government have made science-based decisions related to climate adaptation based on, e.g., an improved understanding of hydro-climatic conditions, long-term glacier monitoring, permafrost-climate-infrastructure interactions, ground-ice conditions, and on coastal changes.

The evaluation found that CCAP is progressing towards its long-term outcomes. Through CCIAD-AP, numerous target stakeholders from different areas have included climate change adaptation in their plans and strategies, including communities and businesses, all levels of government, as well as professional sectors and associations. Although at the beginning phase, target stakeholders have also begun to implement climate adaptation measures due to the influence of CCIAD-AP. More work is required to stimulate action from all target stakeholders. Concrete examples of areas where CCGP is influencing the implementation of climate adaptation measures include information on hydro-climatic conditions near First Nations communities in the Hudson Bay Lowlands; responsible land management and adaptation to possible sea-level changes due to glacier melting in the Canadian Arctic and the Western Cordillera; strategies for major existing and proposed transportation routes, upgrades at the Iqaluit Airport; and the relocation of the Hamlet of Tuktoyaktuk in the Northwest Territories.

Finally, the evaluation found that CCAP is progressing towards its ultimate outcome. The overall trends of progress observed for the intended outcomes suggest that CCIAD-AP and CCGP have cultivated stakeholders’ capacity to act and adapt to climate-related changes. For example, the support provided by CCGP allowed stakeholders to implement climate adaptation measures to improve the resilience of the Hamlet of Tuktoyaktuk and the surrounding area, which had positive impacts on the community’s economy and security. More broadly, CCIAD-AP stakeholders perceived an increase in their capacity to act and adapt to climate-related changes, but more time would be required to be able to concretely assess resilience.

The evaluation identified several factors that have influenced results of the programs and assessment of their impacts. Notable, progress observed is limited to CCAP’s target stakeholders; more work is required in other organizations, communities, and sectors to build Canada’s resilience (see the program’s theory of change). Despite the promising trends of progress observed, there are also some limitations in CCAP’s performance measurement strategy. While CCIAD-AP and CCGP have put in place several performance measures, there are areas for improvement. CCIAD-AP and CCGP have not been consistent with collecting performance information on all their key components. Performance information for the longer-term outcomes remains a challenge for CCAP.

Recommendations and Management Response

Recommendation	Management Response and Action Plan
The ADM, Lands and Minerals Sector (LMS) should further explore avenues to accelerate the implementation of climate change adaptation measures.	Management response: Management agrees. Building on LMS’ co-leadership in the development of the National Adaptation Strategy (NAS), and leadership on the Economy and Workers System under the NAS, the ADM-LMS commits, via the Director General-Hazards, Adaptation & Operations Branch (HAOB) and Director General-Geological Survey of Canada (GSC), to work with partners to address knowledge and skills barriers for climate change adaptation. This will be achieved through targeted research, enhanced engagement and knowledge products, case studies of adaptation actions, and funding projects to develop information and skills for adaptation. Specifically: LMS-GSC will undertake research on the cryosphere, sea-level, and coastal changes with a stronger focus on hazards and risks to produce new knowledge products for communities and other stakeholders. LMS-GSC will coordinate with federal partners, Indigenous groups, Provinces and Territories, and academia to develop research priorities for contribution to ECCC’s Climate Science 2050 Science and Knowledge Plan. LMS-HAOB will undertake and disseminate case studies of adaptation actions that document the lessons learned so that other decision-makers can learn from those experiences. LMS-HAOB will co-fund projects that address knowledge gaps and emerging issues and support adaptation skills development in professionals such as planners and accountants. LMS-HAOB will launch a new program to accelerate coordinated adaptation action in coastal regions. LMS-HAOB will work with other federal departments to develop revised federal governance on adaptation to deliver co-ordinated action on the objectives and targets of the NAS. The revised governance will improve collaboration with Provinces, Territories, Indigenous Peoples, and other partners to deliver more coordinated adaptation programming. Position responsible: DG-HAOB and DG-GSC Timing: LMS-GSC will include hazard and risk-related research activities in its climate change programming by May 31, 2024. LMS-GSC will contribute collaborative research priorities to CS2050 Plan by December 15, 2023. LMS-HAOB will issue a call for proposals for co-funded projects by June 30, 2023. LMS-HAOB will undertake adaptation case studies and disseminate results to the Adaptation Platform by April 1, 2025. LMS-HAOB will launch a Climate Resilient Coastal and Northern Communities program by September 29, 2023. LMS-HAOB, working with other federal departments, will develop revised federal governance by September 29, 2023.
The ADM, LMS should continue to explore and implement approaches to further integrate considerations for Equity, Diversity, Inclusion, and Accessibility in Climate Change Impacts and Adaptation Division-Adaptation Program (CCIAD-AP), with particular emphasis on the participation of diverse Indigenous communities and other under-represented groups that are affected by climate change impacts.	Management response: Management agrees. LMS-HAOB will invite participation of under-represented groups in Canada’s Adaptation Platform plenary and working groups to enhance collaboration on Inclusion, Diversity, Equity and Accessibility in adaptation. LMS-HAOB will update its Climate Change Adaptation Terms and Conditions to better support Grants and Contributions projects led by Indigenous communities. LMS-HAOB will increase requirements for consideration of Inclusion, Diversity, Equity and Accessibility in its co-funded projects. Position responsible: DG-HAOB Timing: Invites sent to under-represented groups to join the Adaptation Platform by October 30, 2023. Amendment of the Climate Change Adaptation Terms and Conditions by June 30, 2023. Increased requirements for Inclusion, Diversity, Equity, and Accessibility in co-funded projects will be added to the call for proposals by June 30, 2023.
The ADM, LMS should review the performance measurement strategy of Climate Change Adaptation Program. In particular: Update the strategy to collect the performance information that best informs progress and accomplishment of its expected results and updated targets (CCIAD-AP and Climate Change Geoscience Program). Specific to CCIAD-AP, ensure that the program is monitoring and measuring outputs and outcomes of all the key delivery mechanisms, including projects subsidized by the program (e.g., the Platform, BRACE, etc.).	Management response: Management agrees. a) DG-GSC will update its strategy for performance measurement including revision to logic model and metrics. DG-HAOB will update its performance measurement strategy to contribute to measurement of the National Adaptation Strategy Objectives and Targets. b) DG-HAOB will take steps to ensure improved project monitoring and reporting of results takes place, for example by adjusting its full-time equivalent staff resources. Position responsible: a) DG-GSC, DG-HAOB b) DG-HAOB Timing: a) CCAP performance measurement strategy is updated by September 29, 2023. Updates to the Performance Measurement Strategy to contribute to the NAS Objectives and Targets will be done in collaboration with ECCC and Other Government Departments by December 29, 2023. b) Improved project monitoring and reporting is in place for CCIAD-AP by September 29, 2023.

Introduction

This report presents the findings, conclusions, and recommendations of the evaluation of Natural Resources Canada (NRCan)’s Climate Change Adaptation Program (CCAP). The Audit and Evaluation Branch (AEB) of NRCan conducted the evaluation between April 2020 and December 2021 in accordance with the Treasury Board Policy on Results (2016). The evaluation examined the activities from CCAP’s adaptation component [i.e., Climate Change Impact and Adaptation Division’s Adaptation Platform (CCIAD-AP)] and geoscience component [i.e., Climate Change Geoscience Program (CCGP)] between 2016-17 and 2020-21. This program was last evaluated in 2010-11 and 2014-15. The evaluation responds to a Treasury Board commitment to evaluate as part of the Horizontal Climate Change Adaptation Initiative led by Environment and Climate Change Canada (ECCC). The evaluation results will be an important source of information for the ECCC horizontal evaluation to be conducted in the near future.

Program Profile

At the outset of the program in 2016, climate change impacts were already felt across Canada, particularly in Canada’s northern and coastal regions. Climate change impacts pose significant risks to Canadian communities, their health and well-being, the economy, and the natural environment. Therefore, there is a need to ensure that infrastructure and communities can adapt to climate change. NRCan has been involved in climate change initiatives for over 20 years. CCAP is a long-standing climate change adaptation program led by NRCan. The program’s predecessors were the Climate Change Geoscience and Adaptation Sub-Program (2005-06 to 2009-10) and Climate Change Adaptation Sub-Program (2010-11 to 2014-15).

Program Description

Climate Change Impacts and Adaptation Division’s Adaptation Program (see Box 1). CCIAD-AP aims to advance climate resilience across Canada by facilitating collaborative action among the public sector, the private sector, non-governmental organizations, academia, and Indigenous organizations to 1) address shared adaptation priorities; 2) address knowledge gaps for natural resource sectors and communities; 3) enable the use and exchange of knowledge and tools; and 4) build capacity to enable decision-makers to act.

CCIAD-AP’s mandate is bounded by the Federal Adaptation Policy Framework^{Footnote 1}, which seeks to help Canadians to better understand the climate change impacts as well as to have the necessary tools and capacity to adapt to climate change effectively. The framework defines the federal government role as 1) generating and sharing knowledge; 2) building adaptive capacity to respond and helping Canadians take action; and 3) integrating adaptation into federal policy and planning.

CCIAD-AP aims to increase decision-makers’ understanding of the implications of climate change on their operations, and to support decision-makers with climate change adaptation by planning for and managing the risks and opportunities resulting from a changing climate. The program advances climate adaptation in Canada via three key delivery mechanisms:

• Since 2012, the Canada’s Climate Change Adaptation Platform (the Platform) aims to support and enhance collaboration, knowledge creation and dissemination to equip the socioeconomic sectors and regions with the tools to address the complex and crosscutting issue of climate adaptation. The Platform brings together federal, provincial, and territorial governments, Indigenous organizations, industry, professional and not-for-profit organizations, and researchers with knowledge, capacity, expertise, and financial resources to collaborate on delivering climate adaptation action and building resilience across Canada. The Platform’s goal is to facilitate collaborative climate adaptation actions among stakeholders via its various components (see Box 2).
• The National Knowledge Assessment (NKA) process supports the assessment and sharing of science-based knowledge and tools produced in a collaborative process that involves subject-matter experts from all orders of government, Indigenous organizations, universities, professional and non-governmental groups, and the private sector. The NKA process aims to 1) enhance the understanding of climate change impacts and adaptation in Canada and provide the evidence base for responding; 2) increase awareness of the relevance of climate change to Canadians and need for timely action; 3) expand engagement in the assessment process by working with stakeholders and engaging broadly with interested Canadians; 4) equip and empower amplifier organizations to share broad findings with their audiences and create targeted products; and 5) document progress made on advancing climate adaptation action in Canada and provide a baseline for future analyses of progress. In addition to conventional science sources, the current process (called “Canada in a Changing Climate: Advancing Our Knowledge for Action”) also considers other ways of knowing, including Indigenous knowledge and practitioners experience. The NKA process also includes a component to fund regional risk assessments with each province.
• The Building Regional Adaptation Capacity and Expertise (BRACE) program works in direct partnership with provinces to build the capacity of organizations, professionals, and communities, and to apply climate change considerations in decision-making and to implement climate change adaptation actions. BRACE takes a customized regional approach to address the unique needs of each province, encouraging thematic and regional collaboration on climate adaptation in areas like natural infrastructure, professional training and natural resource sectors. Projects are using a wide range of approaches for building climate adaptation capacity, such as delivering focused training, preparing locally relevant tools, encouraging regional collaboration and facilitating communities of practice.

Climate Change Geoscience Program: CCGP delivers scientific research and analysis on geoscience issues to support climate change adaptation. This research is organized around four components: 1) current and future conditions in permafrost regions; 2) coastal dynamics; 3) extreme events; and 4) glacier monitoring. These research efforts include an extensive use of remote sensing techniques to monitor and inform future projections of climate change on geological processes.

In 2011, CCGP was reoriented to focus on Northern Canada (i.e., north of 60°). The new focus was strongly aligned with Canada’s Northern Strategy^{Footnote 2}, and to account for the fact that there were large areas with sparse baseline geoscientific information. The project themes were identified following extensive consultation with Northerners and the GSC staff. The program prioritized its focus areas to maximize its impacts on infrastructure and communities for northern resource development. Furthermore, northern territories and coastal areas are more affected by severe climate change impacts and have less capacity for climate change adaptation. CCGP geoscience information is used not only to identify climate change impacts on infrastructure in Canada’s Arctic region, but also to assess the effects of climate change on geohazards and on Earth surface processes in order to enhance community resilience.

• Canada’s North: CCGP’s expertise and systems are used almost exclusively to support climate change adaptation in permafrost and coastal regions, and glacier mass balance monitoring in northern areas.
• Canada’s Coastal Areas: CCGP supports climate change adaptation in coastal areas (including non-northern areas) by conducting research as well as making information available in relation to sea level change and coastal sensitivity mapping. Much of this information is provided via the CanCoast information system.

As part of its four components listed above, the program conducts work to help Indigenous communities in the North better understand climate change impacts, as these communities are concerned with the ongoing changes in their lands. The questions posed by the Indigenous communities help to guide CCGP research, while the program in turn produces information that assists these communities.

Program Expected Results

In general, CCIAD-AP’s theory of change articulates that CCIAD-AP supports collaborative development of knowledge and tools. This knowledge and these tools build awareness of climate issues among target stakeholders, followed by building the capacity of target stakeholders to use and apply these adaptive capacity building resources. These resources are accessed and applied by target stakeholders to identify threats and priorities for climate adaptation, followed by facilitating the identification of climate change adaptation measures by target stakeholders to address risks and opportunities arising from climate change.

In general, CCGP’s theory of change articulates that through CCGP, NRCan conducts monitoring, research, and fieldwork on climate change impacts, often in collaboration with target stakeholders. The program releases geoscience knowledge, products, and practical advice. As decision-makers and end-users become aware of these, they access and use these resources to identify priorities for climate preparedness and adaptation activities, which subsequently facilitates science-based decision-making by stakeholders to reduce climate risks and adapt to climate change.

Together, this leads to climate change adaption being integrated in target stakeholders’ plans and strategies, as well as climate adaption measures being implemented by target stakeholders. In the long-term, resilience to climate change impacts is improved in target stakeholders, and by extension, in Canada’s regions, communities, and economic sectors. It is important to emphasize that the achievement of the ultimate outcome – i.e., resilience to climate change impacts is improved in Canada – will depend on various factors including those that are external to the program’s sphere of control. The program logic model is illustrated in Figure 1.

Figure 1: CCAP Logic model

Program Governance

Within NRCan’s Departmental Results Framework (2022-23), CCAP is under Core Responsibility 1: Natural Resource Science and Risk Mitigation, and the program contributes to Departmental Result #3: “Communities and industries are adapting to climate change”. CCAP is delivered by the Lands and Minerals Sector (LMS), which was formed in 2017 following a restructuring of NRCan’s sectors.

CCIAD-AP and CCGP report to separate Directors General (DGs) within LMS. CCIAD-AP is a division under the Hazards, Adaptation and Operation Branch (HAOB), and CCGP is part of the Geological Survey of Canada (GSC). At the time the evaluation was conducted, CCIAD-AP managers were reporting to the Director, CCIAD, and CCGP manager was reporting to the Director of GSC – Atlantic (see Figure 2).

Figure 2: CCAP Organigram

The activities of both components are carried out independently of each other. Coordination between CCIAD-AP and CCGP largely happens at a strategic policy level through their participation in the Director General Adaptation and Resilience Committee (DGARC) as well as the Director-level Adaptation and Resilience Task Team (DARTT), which are co-chaired by NRCan’s Director CCIAD and DG HAOB along with ECCC. This committee and task team are among the Government of Canada (GC) interdepartmental collaboration and coordination mechanisms used for the Pan-Canadian Framework on Climate Change and Clean Growth (PCF) – Adaptation and Climate Resilience pillar. These mechanisms provide the structure for departments to coordinate and share departmental activities related to climate change to reduce overlap.

Program Resources

Planned and actual expenditures over the 5-year evaluation period for CCIAD-AP are presented in Table 1 and 2. The main source of funding was through temporary funding (C-base). Spending fluctuated during the evaluation period due to activities supported through temporary project-based funding. Overall, CCIAD-AP spent slightly less than the planned budget over the evaluation period.

Table 1: Total Planned Expenditures for CCIAD-AP by Year over the Evaluation Period

Expenditure type	2016-17	2017-18	2018-19	2019-20	2020-21	Total
O&M	$652,929	$1,206,412	$857,120	$776,787	$720,271	$4,213,519
Salaries & EBP	$1,264,482	$1,349,835	$1,778,768	$1,848,598	$2,006,827	$8,248,510
G&C	$600,000	$1,000,000	$5,940,000	$7,380,000	$3,793,220	$18,713,220
Total	$2,517,411	$3,556,247	$8,575,888	$10,005,385	$6,520,318	$31,175,249

Source: CCIAD-AP Financial Data.

Table 2: Total Actual Expenditures for CCIAD-AP by Year over the Evaluation Period

Expenditure type	2016-17	2017-18	2018-19	2019-20	2020-21	Total
O&M	$552,148	$570,089	$844,516	$751,857	$570,000	$3,288,610
Salaries & EBP	$1,117,790	$1,378,832	$1,690,747	$1,848,598	$2,006,827	$8,042,794
G&C	$537,574	$926,129	$5,783,813	$7,380,000	$3,793,220	$18,420,736
Total	$2,207,512	$2,875,050	$8,319,076	$9,980,455	$6,370,047	$29,752,140

Source: CCIAD-AP Financial Data.

Table 3 and 4 present the planned and actual expenditures for CCGP over the 5-year evaluation period. Note that the planned expenditures include: the Operations and Maintenance (O&M) funds allocated from A-base, C-base, and other government departments (OGDs), and the estimated salaries and Employee Benefits Plans (EBP) for the GSC FTEs who contributed to the CCGP program. The actual expenditures include: the O&M funds spent from A-base, C-base and OGD allocations, as well as additional cash and in-kind support secured from partners and collaborators. This includes in-kind logistical support from the Polar Continental Shelf Program which is critical for conducting program fieldwork; salaries and EBP for FTEs from the GSC as well as FTEs from other organizations that CCGP relied upon to help deliver the program.

Table 3: Total Planned Expenditures for CCGP by Year over the Evaluation Period

Expenditure type	2016-17	2017-18	2018-19	2019-20	2020-21	Total
O&M	$696,915	$904,000	$765,300	$668,800	$721,186	$3,756,201
Salaries & EBP	$1,985,000	$2,337,645	$2,621,008	$2,204,686	$2,787,401	$11,935,746
Total	$2,681,915	$3,241,651	$3,386,308	$2,873,486	$3,508,587	$15,691,947

Source: CCGP Financial Data.

Table 4: Total Actual Expenditures for CCGP by Year over the Evaluation Period

Expenditure type	2016-17	2017-18	2018-19	2019-20	2020-21	Total
O&M	$1,525,430	$1,801,250	$1,464,087	$1,302,905	$737,166	$7,298,870
Salaries & EBP	$2,620,000	$2,904,042	$3,108,312	$2,706,306	$3,006,439	$14,345,099
Total	$4,145,430	$4,705,292	$4,572,399	$4,009,211	$3,743,605	$21,643,969

Source: CCGP Financial Data.

Program Operating Context

The increasing frequency and severity of natural disasters driven by climate change in the last 10 years has brought greater attention to climate change issues. Specifically, climate change adaptation is required to address the growing and irreversible climate change impacts affecting communities across Canada like extreme weather, flooding, wildfires, and coastal erosion.

In recent years, the GC has identified the need for transitioning to clean growth and a resilient economy. The GC recognizes that comprehensive climate change adaptation efforts must complement ambitious climate change mitigation measures aiming to reduce greenhouse gases (GHGs) to address climate change (see Box 3). The context for these actions has evolved continuously over the period of evaluation.

In December 2015, Canada and 194 other countries reached the Paris Agreement^{Footnote 3}, which sought to strengthen the efforts to limit the global average temperature rise to well below 2°C and pursue efforts to limit the increase to 1.5°C. It aims to lower GHG emissions and to foster climate resilience in the context of the temperature goal.

Building on the momentum set forth by the Paris Agreement, the Canadian government developed the PCF in 2016, with the provinces and territories and in consultation with Indigenous peoples^{Footnote 4}. The PCF aims to meet Canada’s GHG reduction targets, grow the Canadian economy, and build Canada’s resilience to a changing climate. Under the PCF Adaptation and Climate Resilience pillar, all levels of governments have identified new actions to build resilience to climate change across Canada in five key areas: 1) Translating scientific information and Traditional Knowledge into action; 2) Building climate resilience through infrastructure; 3) Protecting and improving human health and well-being; 4) Supporting particularly vulnerable regions; and 5) Reducing climate-related hazards and risks. Federal actions focus on providing stakeholders with the knowledge and capacity to understand impacts and take action to reduce risks related to climate change.

It should also be noted that the COVID-19 measures had a significant impact on all programming. CCAP and BRACE both had to help projects adjust to limits to in-person interactions as well as changes to project end dates and reporting. CCGP was limited in its ability to send staff into the field.

Evaluation Objectives, Methods, and Limitations

The topic of relevance was not included in the scope of the evaluation because the previous evaluations^{Footnote 5} found that the program was relevant. Further, the program has existed since 1998; climate change issues have become more important and prominent in Canada and globally since then. NRCan’s roles and responsibilities related to climate change adaptation have not changed significantly. As the program remains relevant, evaluation resources focused on the issue of performance (i.e., efficiency and effectiveness). Accordingly, the current evaluation’s objectives were:

• Assess the program model’s capacity to adapt to the evolving context.
• Assess whether the program has put in place measures to achieve EDI objectives with an emphasis on Indigenous peoples.
• Assess the program’s contribution to achieving its intended outcomes with an emphasis on the long-term outcomes.

The evaluation also examined the implementation of recommendations from the previous evaluation, as well as best practices and lessons learned related to the design and delivery of the program.

Evaluation questions and indicators used to address these objectives are as outlined in Appendix A.

While CCIAD-AP and CCGP share the ultimate outcome, they do not share all the immediate, intermediate and long-term outcomes. Therefore, reporting of the evaluation findings was structured to highlight the performance of different components of each program, where relevant.

Data were collected using four lines of evidence (see Figure 3). A document review was conducted to obtain an overall understanding of CCIAD-AP and CCGP (including operations and reported results). A literature review was performed to identify best practices as well as lessons learned from the implementation of similar programs in Australia, the United Kingdom, and the United States, with particular attention to six main themes (e.g., the factors that influence the achievement of medium- and long-term results, and the evolving needs of stakeholders). In-depth interviews were also conducted to gain information on key stakeholders’ views on topics related to the evaluation questions, particularly the achievement of intermediate, long-term and ultimate outcomes. Four case studies from CCGP key program areas were featured to illustrate how its activities uniquely contributed to the intended outcomes. These case studies were not originally part of the evaluation design but became de facto a line of evidence given the way CCGP performance metrics were designed.

Figure 3: Four Lines of Evidence

Evaluation Limitations

Although the evaluation was designed to collect data using multiple lines of evidence to enhance the reliability of results, there remained a few challenges that required mitigation strategies (see Table 5).

Table 5: Evaluation Challenges and Mitigation Strategies

Challenge	Mitigation Strategy
Timing of the evaluation during the COVID-19 pandemic had significant implications for program delivery. Program management and staff had reduced availability to participate in the evaluation. As a result, the evaluation took longer than expected to complete. The scope of the evaluation does not capture the most current fiscal year (i.e., 2021-22).	Consulted directly with the program management before requesting information to assess their capacity to respond to the request. Prioritized the necessary inputs from program managers and worked within their availability. Searched directly for relevant information in internal and external sources, such as the relevant program documentation in NRCan central repository. Completed document and literature review in advance of the key informant interviews so that interviews were more targeted.
Shortcomings in CCIAD-AP’s systematic monitoring of projects and progress did not facilitate a coherent document review.	Focused on early results reported in the CCIAD-AP Annual Reports, and then triangulated the findings with the interview findings.
Due to consultation fatigue and limited availability, the planned evaluation survey was not conducted.	Augmented the sample of key informants to ensure that the sample was representative of CCIAD-AP stakeholder groups and activities. Ensured that interviews were targeted so that they were conducted efficiently.

In general, climate change and disaster risk reduction research has experienced issues related to the definition and measurement of “resilience.” Some issues that have been raised in the literature are the diverse conceptualization and meaning of resilience that vary by field, as well as the challenges with drawing a direct attribution to a factor when there is a change in resilience. There is also the lack of pre- and post- disaster data to objectively determine if the resilience cultivated through the implementation of climate change adaptation measures is effective in mitigating climate change impacts. The difficulty in measuring resilience is thus common in climate adaptation programs as there is no one-size-fits-all approach to climate adaptation. The diverse conceptualizations of resilience provide individuals, communities, and governments with the flexibility to decide which climate adaptation measures are relevant for them. Consequently, measuring resilience as well as attributing it to a specific program is complex and challenging. The evaluation of CCAP’s progress towards the ultimate outcome was affected by these constraints.

Accordingly, the evaluation addressed these limitations to the extent possible by first defining “resilience”. The evaluation used the definition by Intergovernmental Panel on Climate Change (IPCC); this definition is adapted from the United Nations Office for Disaster Risk Reduction’s International Strategy for Disaster Reduction. Accordingly, the evaluation conceptualized “resilience” as a capacity to act and adapt (“anticipate”) in the face of climate risks. During the evaluation period, the evaluation team was not able to assess the other aspects of resilience (“absorption, accommodation, or recovery”) because the implementation of climate change adaptation measures was at the beginning phase and the lack of long-term data on the efficacy of these measures in real-life climate change and disaster events.

Then, the evaluation considered trends in the progress towards the preceding intended outcomes as a proxy measure to assess CCAP’s performance for its ultimate outcome. That is, the evaluation examined to what extent CCIAD-AP and CCGP have cultivated stakeholders’ capacity to act and adapt (or “adaptive capacity”) based on the overall evidence, guided by the programs’ theories of change. The results chain describes that adaptive capacity built by the program’s activities and resources help increase Canada’s resilience through the implementation of climate adaption measures. There are numerous factors that are internal and external to the program that affect successful implementation of climate change adaptation measures and the effectiveness of these measures in improving resilience.

What We Found

Efficiency – Does the Program model support the achievement of Program objectives?

Evolution and Changes to the Program

CCAP has generally evolved to meet new and changing needs of stakeholders.

In December 2020, the GC committed to developing Canada’s first National Adaptation Strategy (i.e., NAS) ^{Footnote 7} in collaboration with all levels of government, Indigenous peoples, and other key partners as part of its Strengthened Climate Plan^{Footnote 8}. Led by ECCC, the strategy is intended to establish a shared vision for climate resilience in Canada by identifying key priorities for increased collaboration, aligning collective and individual actions towards faster, coordinated and systemic climate adaptation, as well as establishing a framework for measuring progress at the national level. By providing a national framework for action, the strategy also intends to help ensure that all parties have clarity on roles and responsibilities as well as to support climate adaptation actions that are cohesive, targeted, and strategic. The NAS was under development during the evaluation period; it was released in November 2022.

In response to the evolving context, CCAP has evolved in several ways since the last evaluation in 2015. As climate change intensifies, the integrated nature of its impacts has become more obvious, justifying a systemic approach to climate change adaptation to accelerate actions to reduce climate risks and increase climate resilience.

Evidence showed that the evolution of the CCAP model is generally facilitated by two types of process: internal self-assessment process, and ideas derived from stakeholder consultations. In particular, the evidence revealed that CCIAD-AP has evolved its design and delivery based on stakeholders’ feedback and the learnings gained over the course of its cycles (e.g., feedback from the Plenary’s members, commissioned external studies, etc.). The Platform adopted in-depth sessions and introduced the “Adaptation Leaders Forum” to encourage a systemic approach to identify opportunities, explore new collaborations, and enhance engagement at a strategic level. The Platform Plenary has evolved in its membership and functions. CCIAD-AP recognizes that Indigenous communities are rights holders of the land, and that they are experiencing severe climate change impacts as well as the importance of including them in its Plenary table as essential partners. The Plenary membership was extended to include representatives from the Assembly of First Nations (AFN), the Inuit Tapiriit Kanatami, and the Métis National Council General Assembly. The Plenary has also been progressive and timely in bringing forward knowledge as well as emerging trends and issues. Evidence showed that there were discussions held during the Plenary to link the role of climate change adaptation in advancing social equity. As the Plenary membership continues to grow, CCIAD-AP management indicated that they were looking at changing the Plenary model to allow for a broader engagement, such as supporting the engagement of under-represented groups.

Evidence revealed that the Platform’s WGs have generally evolved their design and delivery based on their updated State of Play reports on a five-year rolling process. The WGs examine questions on emerging issues and solutions as well as innovative policy options. For example, as natural infrastructure became more understood and accepted as a climate change adaptation measure, there was a shift in funding calls by the Coastal Management WG to support the understanding of barriers to the adoption of these solutions. New WGs are created through the Plenary to focus and report on emerging areas that have been identified, such as the Biodiversity WG and Climate Services WG. However, not all WGs are evolving to the changing context. Interviewees stated that while some WGs were in a position of continuous evolution and learning, others were more static.

Evidence demonstrated that the Platform’s RACs have evolved their design and delivery to meet the needs of stakeholders. RAC organizations consider the way CCIAD-AP supports their adaptation work complies with best practices and suggestions they found in the literature. Program staff are open to hearing stakeholders’ feedback and to adapting the program’s activities when needed. For instance, some RAC interviewees indicated that they were encouraged by the program to utilize creative and novel approaches due to the COVID-19 pandemic (e.g., adjusting their delivery mode from a one-day to a two-day workshop to meet their clientele’s needs).

Evidence demonstrated that the NKA process has evolved its design and delivery by building on lessons learned from previous assessments to enhance the delivery of its process. It has evolved to become more inclusive and transparent with a broader visibility than in the past. Evidence indicated that the current NKA process is more open with the potential to attract more experts to contribute to its development. It has also included other ways of knowing (e.g., Indigenous knowledge and practitioner experience) in addition to conventional science sources. The NKA process also conducts surveys after the release of its report series to identify stakeholder needs, and then evolve its design and delivery accordingly. For instance, the NKA process learned that it was challenging to meaningfully incorporate Indigenous knowledge efficiently and economically in its standard report. Consequently, the NKA process supported the production of an Indigenous stand-alone report, with a lead author from the AFN. The NKA process has also evolved its design and delivery based on research on other countries’ findings.

In 2017, BRACE was introduced as part of CCAP for more emphasis on capacity development. Evidence showed that BRACE has evolved its design and delivery to meet the needs of stakeholder. It has built flexibility in its design by entrusting provinces with the development of projects that meet their needs and by instating self-assessment practices to support continuous improvement. Interviewees stated that BRACE Committee held workshops where BRACE project leads and provincial representatives met to collaborate and exchange knowledge, ideas and information, including the challenges with and lessons learned from their current approaches to capacity building; this helped proponents improve their action going forward. CCIAD-AP also commissioned a study to determine the state of climate change adaptation capacity building in Canada to reassess BRACE priority areas for the next cycle. During the COVID-19 pandemic, BRACE also worked extensively with project proponents and the program’s financial advisor to identify avenues to increase flexibilities for projects.

While evidence generally showed that CCIAD-AP has evolved to meet the needs of stakeholders, interviewees believed that the program has not necessarily evolved as quickly as needed, given the pace of climate change. They also noted that the program is lacking an overarching self-assessment process that would help identify synergies in learning among its key delivery mechanisms. In some cases, the self-assessment process for the Platform’s co-funded projects is not consistently applied. Few exchanges on best practices are observed between the most and less active WGs.

Evidence showed that CCGP has evolved its design and delivery to meet the needs of stakeholders. The program seeks feedback from stakeholders on its research activities. Further, interviewees provided positive feedback on program delivery. For instance, interviewees stated that the program sought feedback from representatives of the Hamlet of Tuktoyaktuk regarding the value of CCGP’s permafrost and coastal erosion research activities and related advice. Stakeholders from Tuktoyaktuk said that the relationship enabled the Hamlet to adapt to climate change in a much better manner than without CCGP’s support. CCGP also conducts meetings and produces reports to plan and monitor progress on key projects and other important deliverables. Further, the program performs ongoing management activities and strategic planning to ensure the program continues to be relevant for stakeholders. For example, the program conducted workshops involving CCGP personnel and other key stakeholders in 2019 and 2022 to share program results, seek feedback on the program future, and establish priorities for the next program cycle. These actions allow CCGP to update its strategic priorities and activities over time, shifting its focus areas to maximize program impact (e.g., northern and coastal areas) while meeting the critical needs of stakeholders in Canada’s Arctic region.

CCAP implemented the recommendations from the previous evaluation, which contributed to the achievement of the program outcomes.

Evidence showed that CCAP implemented the recommendations from the 2015 evaluation, which contributed to the achievement of the program outcomes (see Table 6).

Table 6: Implementation of Recommendations from 2015 Evaluation

Recommendation From 2015 evaluation	Implementation of Recommendation
NRCan should update the information contained in the benchmark study to ensure that CCA [Climate Change Adaptation] activities remain relevant to targeted stakeholder needs and are consistent with the Federal Adaptation Policy Framework.	This recommendation was addressed through the conduct of the NRCan’s 2018 Benchmark Survey. This is a recurring survey to be conducted every four years
The CCA Sub-Program should develop a mechanism that enables current and future stakeholders to access the knowledge and information generated on an ongoing basis.	CCIAD-AP established Contributions Agreements with four regional organizations to extend reach to new audiences as well as leveraged the Platform members’ respective networks to extend the reach of dissemination activities. CCGP developed a communication and dissemination strategy to improve access to the knowledge generated by the program to ensure full alignment of the program with government open data policies at the start of the program cycle in 2016. Additionally, significant investments were made in the Permafrost Information Network (PIN) and CanCoast to further improve access to the knowledge generated by the program, resulting in noticeable improvement to these information systems. CCGP also uses GEOSCAN to post documents and make them accessible to stakeholders across Canada and abroad.
The CCIAD-AP should review and align program costs with program business lines so that the costs of outputs can be more readily measured.	CCIAD-AP put in place and utilized new coding for expenses at the required level of detail. However, the evaluation team notes that the program was still unable to provide the information at the necessary level of detail needed to assess program efficiency.

Facilitating and challenging factors have influenced CCAP performance and assessment of its impact.

Evidence revealed facilitating and challenging factors influencing CCAP’s performance and assessment of its impact (see Table 7 and 8).

Table 7: Facilitating Factors Influencing CCAP

CCAP (both components)
Climate change impacts’ frequency and severity	The increased severity and frequency of climate change impacts in the last few years have increased awareness of the need for climate change adaptation in the population.
Government policy priorities	The GC’s policies have further prioritized climate change issues in recent years.
CCIAD-AP
Strong leadership and staff dedication	CCIAD-AP has effective leadership as well as staff dedication and continuity that have all reinforced relationships and trust with the project proponents; this helps them to work together efficiently.
Engagement of the stakeholders	The active and dynamic participation of the stakeholders in the Platform Plenary and the Platform’s WGs contributes to accelerating the identification, launch and advancement of new initiatives.
CCGP
Collaboration and communication between scientists and community	Collaboration and two-way communications between CCGP scientists and community members (e.g., Hamlet of Tuktoyaktuk and Iqaluit airport) established over several years enable CCGP to add significant value to the local stakeholders.
Partnership with diverse stakeholders	CCGP establishes partnership with other parts of NRCan, other government departments, and other organizations (e.g., provincial hydro utilities and universities), leveraging their expertise and financial support.

Table 8: Challenging Factors Influencing CCAP

CCAP (both components)
Climate diversity in Canada	Canada’s climate is diverse due to its large geographical area. Therefore, the climate change impacts need to be addressed in a broad and variable context (e.g., from coastal erosion and sea level rise to forest fires, etc.).
CCIAD-AP
The COVID-19 pandemic	Switching to virtual meetings disrupted the Platform Plenary’s dynamics as well as CCIAD-AP projects’ activities and momentum (e.g., mining companies were unable to engage with the program because their operations were impacted by the pandemic).
Stakeholder diversity	Climate change adaptation work is complex in areas involving diverse and large number of stakeholders and jurisdictions (e.g., alignment of regional RACs with provincial climate change strategies, changing provincial priorities and commitments for climate change issues, tailoring knowledge and tools to diverse audiences, etc.). This often involves a lot of collaboration and time to identify needs and challenges as well as to develop an approach to address them while ensuring that the program meets its intended outcomes.
CCGP
The COVID-19 pandemic	Almost all the site visits and field trips for the four CCGP program areas had to be halted during the pandemic. This slowed the work and affected the quality of the information collected in some instances such as glacier mass balance monitoring.

Program Design and Delivery

The CCAP design is generally supportive of its objectives as part of the PCF.

Evidence demonstrated that the CCAP model is generally supportive of its program objectives with CCIAD-AP and CCGP each providing unique and important functions required to enable the implementation of climate change adaptation actions and plans as part of the PCF. CCIAD-AP operates largely through funding third-party organizations on a wider range of interests, which affects its ability to influence and measure uptake of its outputs. Both CCIAD-AP and CCGP were also found to collaborate on issues and activities of common interest. For example, CCIAD-AP contributed to translating and disseminating the knowledge produced by CCGP. CCGP scientists participated in CCIAD-AP as members of the WGs, including the Coastal Management WG, and CCGP scientists also delivered presentations to the Platform Plenary.

Evidence revealed that the CCIAD-AP model generally supports its program objectives, and by extension, provides support to the CCAP program objectives as part of the PCF. Interviewees stated that they were satisfied with the program design and delivery. Evidence revealed that the Platform Plenary model is efficient. The Plenary facilitates networking opportunities as well as discussions on emerging issues and priorities. For instance, interviewees stated the Platform Plenary’s “network of networks” (see Figure 4) of individuals and organizations who are both users and producers help ensure that knowledge and tools are targeting stakeholders’ needs and priorities. The Plenary helps increase awareness, enabling knowledge and tools required by stakeholders to take actions. For instance, interviewees emphasized that the Plenary enabled stakeholders to move from having conversations about climate change into taking actions to adapt to climate change impacts. They also stated that the model of distributed ownership contributed to stakeholders’ engagement as exemplified by the Plenary members’ high rate of attendance.

Figure 4: A Network of Networks

AAFC: Agriculture and Agri-food Canada, CIRNAC: Crown-Indigenous Relations and Northern Affairs Canada, DFO: Fisheries and Oceans Canada, ECCC: Environment and Climate Change Canada, HC: Health Canada, ISC: Indigenous Services Canada, INFC: Infrastructure Canada, NRCan: Natural Resources Canada, PCA: Parks Canada Agency, PHAC: Public Health Agency of Canada, PSC: Public Safety Canada, SCC: Standards Council of Canada, TC: Transport Canada.

Source: Natural Resources Canada, Canada’s Climate Change Adaptation Platform, Annual Report, April 2019 – March 2020, page 16.

There were mixed results for the Platform’s WG model. The roundtable discussions within each WG help facilitate knowledge sharing and are beneficial to its members because they allow stakeholders to address cross-cutting issues. However, interviewees questioned if NRCan should have more involvement in the WGs by regularly bringing the various WGs together to encourage cross-collaboration; this approach would increase members’ awareness of the interconnectedness of their work and encourage a systemic approach instead of limiting to specific themes.

Likewise, there were mixed results for the Platform’s RAC model. Interviewees stated that RACs worked well because they contributed to the level of awareness and understanding of climate change vulnerabilities and adaptation options at a regional level; however, the efficiency of RACs differed based on their internal functioning. Regional RACs (i.e., covering multiple provinces) face administrative complexities in managing the funds and in reporting. They also have difficulties in finding a methodology to identify common climate adaptation priorities. Further, interviewees indicated that the regional RACs experienced challenges in establishing a work plan that met both the federal funding criteria and each provincial member’s strategic plan. Therefore, some regional RACs did not seek renewal because the provinces would prefer to work separately.

Evidence revealed that the NKA process model is generally efficient. The NKA process advisory committee extends across Canada, bringing diversified expertise and perspectives in the review of climate change science compiled in the reports. To improve the production of knowledge, the NKA team includes a self-assessment learning system such that lessons learned from previous processes and research from other countries are considered at the launch of a new assessment cycle. For instance, interviewees stated that the synthesis of knowledge from the various sources helped identify gaps and emerging issues in the current state of research. Reports from this process also contribute to raising awareness of stakeholders and the public in general and to building the case for adaptation action.

Evidence showed that the BRACE model is generally efficient. BRACE projects are co-funded with the provinces based on the provincial issues and resources. BRACE enables the production of relevant professional development resources for organizations to support their members and for the integration of changing climate impacts in their work. It also helps develop regional networks. In addition, the transition to eLearning to adjust to the COVID-19 pandemic has significantly modified the scope of the projects both in terms of the work involved in converting the course content for an online delivery and the increased number of online participants to serve. Interviewees noted that there were plans to progressively expand these networks to the national level. Interviewees also indicated that if BRACE works with Indigenous communities in the future, it should ensure that Indigenous organizations are leading climate change adaptation projects that affect their communities. They also mentioned the need to strengthen the coordination of federal departments’ actions targeting Indigenous communities and organizations. While many professionals at the provincial level were interested in eLearning, they were unable to participate because they were not part of the key professions for climate adaptation targeted by BRACE (e.g., engineers, accountants, planners, and natural resource managers).

Evidence revealed that the CCGP model also generally supports the achievement of its program objectives, and by extension, provides support to the CCAP program objectives as part of the PCF. CCGP delivers scientific research and analysis on geoscience issues to support climate change adaptation. Interviewees stated that they were generally satisfied with the program design and delivery. Evidence showed that CCGP matrix organization is an effective program structure. The program facilitates the contribution to the project management and decision-making process of CCGP scientists and other personnel from across the country, independent of their geographical locations.

Evidence showed that CCGP is effective in establishing partnerships to increase the reach and impact of CCGP projects, leveraging the CCGP scientists’ unique expertise while mitigating the relatively small number of personnel. CCGP scientists cooperate with external collaborators wherever possible. Collaborative relationships and agreements were found to be one of the program strengths. The program has a long tradition of collaboration with a variety of partners and stakeholders (e.g., federal government departments and agencies, universities, local stakeholders in Northern Canada). Interviewees explained that the parties to the collaborative relationships typically sought CCGP scientists because of the scientists’ unique expertise. The collaboration can take various forms, such as CCGP scientists working on a specific project while the other parties provide funding, service-in kind or both, or joint scientific research with a university partner. These collaborative relationships have enabled CCGP to achieve an impact that would not be possible solely with its own resources as a significant portion of CCGP resources come from external sources (as illustrated by Figure 6 later in the report). Another type of collaboration is CCGP working closely with federal partners to implement whole-of-government approaches and related strategies.

Best practices, lessons learned, and alternative program model were identified and/or applied for CCAP.

The literature review conducted by the evaluation team revealed approaches implemented by select countries to adapt to climate change impacts (see Table 9). The literature review demonstrated that some of these approaches are similar to the approaches already taken by CCIAD-AP (e.g., use of social media and a platform to enable climate change adaptation), suggesting that the program is effective in keeping abreast with the field. The literature review also suggests some additional approaches that could inspire the thinking to further enhance CCIAD-AP’s performance. However, it is important to note that a limitation of this line of evidence is that some of these approaches cannot be applied by the program as it requires the actions of other entities within the GC.

Table 9: Summary of Approaches by Other Countries

Australia	United Kingdom	United States
Use of Facebook, Twitter, and YouTube to disseminate major outputs as well as partnerships with university communication departments. Creation of a branch of Artificial Intelligence, which can assist with climate change adaptation, in government science body – e.g., Commonwealth Scientific and Industrial Research Organization and major funding organizations. Development of user-friendly one-stop communication tool for main climate vulnerabilities (e.g., coastal dynamics – CoastAdapt).	Adoption of a clear and binding legislative framework (i.e., Climate Change Act 2008). Attribution of a supra-legislative status to the issue of climate change. As an integral part of the Climate Change Act 2008, the Adaptation Reporting Power allows the Secretary of State to request reports on steps critical infrastructure providers are taking to prepare for climate change. The main findings are then published.	Gateway to authoritative science, tools, and resources to help manage risks and respond to changing environmental conditions provided by the United States Global Change Research Program (USGCRP) and its 13 member agencies. Development of platforms by the USGCRP (e.g., Climate Change Adaptation Resource Center, U.S. Resilience Toolkit). Contribution of universities in monitoring the climate adaptation measures put in place.

The evaluation team also observed several notable best practices that were applied by CCAP to enhance its performance. Specific to CCIAD-AP, as a best practice, the program meets the needs of under-represented groups by considering the social and cultural context of the community (e.g., supported Indigenous stand-alone report via the NKA process, invited elders to present at regional workshops via the Platform’s RACs, RAC honoraria). Another best practice is BRACE facilitates cross-country fertilization of ideas and maintain communications between projects through workshops with BRACE committee members and regular meetings. The program also facilitates the advancement of desirable outcomes in climate adaption in communities by leveraging pre-existing relationships and networks that are embedded in a particular community (e.g., BRACE leveraged trusted organizations in the climate adaptation space to work with stakeholders’ communities, climate change coordinators at the local governments relayed the learning from RAC events to other entities in their community).

Evidence also revealed several lessons learned that were identified by CCIAD-AP. A lesson learned is that the size of the program is not commensurate with the magnitude of the issue and need (i.e., the complexity and urgency of climate change impacts) meaning that it has to adjust its programming accordingly. Another lesson learned is the success in the implementation of climate change adaptation measures depends on several factors with some factors beyond the program’s control; this includes how well the needs have been identified and how the information produced by CCIAD-AP fits within the system of governance of the jurisdictions for the implementation of proposed solutions.

For CCGP, a best practice is the program’s sharing of key scientific accomplishments (including examining knowledge gaps) and seeking feedback and ideas (e.g., exploring how best to align current accomplishments with stakeholder priorities) on ways to improve the program by conducting workshops with stakeholders using the Strength, Weakness, Opportunity, and Threat (SWOT) analysis to brainstorm and discuss ideas (see Figure 5). Another best practice is the way the program produces and uses information and communication internally (e.g., semi-annual program management deck) to regularly monitor the achievements of key projects, identify best practices, and apply lessons learned from these projects.

Figure 5: Lessons Learned from CCGP’s SWOT Analysis Conducted During the 2019 Workshop

Stakeholders provided some suggestions for alternative program models that could enhance the performance of CCAP. Specifically, as effective climate change adaptation requires an integrated systems approach, CCIAD-AP could broaden its focus area and membership base to increase impact and engagement. This could include involvement of departments with a regulatory mandate to facilitate the implementation of climate adaptation measures. The program could also increase scientific outreach and knowledge exchange with more stakeholders, including with other countries and jurisdictions that have similar climate vulnerabilities; this could include enhancing the accessibility of the Platform’s online interface and information and knowledge management system to facilitate stakeholders’ access to available resources.

Unmet Needs, Unintended Outcomes, and Sufficiency of Resources.

CCAP’s unmet needs and unintended outcomes were identified.

Evidence showed that there are several unmet needs and unintended outcomes of CCAP. In particular, although stakeholders recognized that the implementation of climate change adaptation measures is not part of CCIAD-AP mandate, they perceived that CCIAD-AP could play an additional role in catalysing the implementation of climate adaptation measures identified through the completion of the Platform co-funded projects. Interviewees explained that knowledge about climate change impacts and adaptation is available, but there are still critical gaps and challenges in the practical implementation of climate adaptation measures. Another unmet need is pertaining to the availability of flexible and long-term funding resources to address emerging needs and issues. Interviewees explained that as climate change continues to advance, new issues will continue to emerge with the continuous need for stakeholders to response. Currently, CCIAD-AP does not have open calls for project ideas and proposals that stakeholders could leverage to address emerging needs and issues. Interviews with program staff showed that they would like to have an open fund that is not running on the same iterative timing as CCIAD-AP to address emerging issues. Another unmet need is the program’s lack of a systemic approach to create synergies in learning among the key delivery mechanisms to address needs and challenges. Missed opportunities that were raised by interviewees included between the NKA process and the Platform’s WGs regarding their experience in developing the NKA process reports and WGs’ State of Play Reports; between the NKA process and the Platform’s knowledge mobilization team regarding their experience in communications; or among RAC member organizations and among the Platform Working Groups.

For CCGP, there is an unmet need for funding that would allow the program to further optimize its research impacts for some projects. For example, interviewees stated that resource levels limited their ability to increase the impact of the program, particularly in the Extreme Events and Glacier Mass Balance Monitoring programs, where the program activities and achievements were modest.

Evidence revealed that an unintended outcome of CCIAD-AP is that the program has a positive unintended impact on the engineering community. Engineers Canada in partnership with CCIAD-AP developed the Public Infrastructure Engineering Vulnerability Committee (PIEVC) protocol in 2008; a protocol intended for use by professional engineers and geoscientists to inform decisions on design, construction, maintenance as well as regulation of safe, reliable, and financially sustainable public infrastructure in Canada. This work that started as a project became a program that has been sustained for 16 years (e.g., 600+ engineers and people involved with infrastructure were trained and 40 infrastructures risk assessments were conducted) and has been internationalized (e.g., Germany, Costa Rica, and Honduras). The evaluation did not find any unintended outcome for CCGP.

CCGP and CCIAD-AP are efficient and economical in leveraging their resources.

Evidence showed that CCGP is efficient in leveraging its resources. Despite fluctuations in funding, the program has been able to make progress towards its intended outcomes while procuring funds from various external sources. Figure 6 presents CCGP’s annual O&M spending by program areas from 2016 to 2021. CCGP’s investments by program areas during the last five years fluctuated as depicted. Over the five-year period, $7.3 million were invested in the program. The two key observations from this figure are: 1) permafrost area received more OGD funds in early years and costal received more later in the program cycle; and 2) all projects had reduced funding in 2020-21 because COVID restricted fieldwork.

Figure 6: CCGP Spending by Program Areas, 2016-2021^{Footnote 9}

Source: Figure created by AEB based on the CCGP Financial Data.

CCGP officials obtained funding from a variety of sources to support the achievement of program objectives, such as OGDs [e.g., Transport Canada (TC), Crown-Indigenous Relations and Northern Affairs Canada (CIRNAC), and other science-based department and agencies] and others (e.g., non-federal government partners such as hydro-electric utilities, universities, northern Canada organizations). The average ratio of CCGP funding (A Base and C Base) to all other sources is 0.7 during the evaluation period. Annually, program funding was subject to change due to its provenance from several stakeholders and partner organizations outside the program (see Figure 7). The diagram shows the importance of external stakeholders in supporting the program. For instance, Polar Continental Shelf Program (PCSP) funding provided CCGP scientists with a significant portion of their resources (e.g., critical logistic support for travel and fieldwork in Northern Canada). Note that there was no PCSP funding in 2020-21 as all fieldwork was cancelled due to COVID.

Figure 7: CCGP Spending by Funding Sources, 2016-2021

Note: Some of the support provided by PCSP and organizations in “Other” was a mix of in-kind support (e.g., time, lab analysis and equipment) and money; the in-kind support was monetized and included in the figure.

Source: Figure created by AEB based on the CCGP Financial Data.

Evidence showed that CCIAD-AP was efficient in the management of its resources. Fluctuations in expenditures were mainly seen in the G&C category, with the addition of temporary project-based funding (see Figure 8). During the evaluation period, CCIAD-AP launched BRACE and a new cycle of G&C projects. There was consequently also a small increase in salaries and EBP to implement this temporary funding. Interviewees emphasized that the financial support provided by the program was useful for stakeholders to obtain supplementary funding from other sources.

Figure 8: CCIAD-AP Spending by Funding Sources, 2016-2021

Source: Figure created by AEB based on the CCIAD-AP Financial Data.

Efficiency – To what extent does the program consider EDI factors?

EDI Considerations

CCAP activities and outputs have considered EDI factors.

The GC has committed to ensuring that subgroups of the population are not adversely affected by its programming due to programs and policies that lack the considerations for intersectionality. To this end, the GC has required the use of Gender-based Analysis Plus (GBA+) when analysing its programming, which is an analytical approach that considers a range of intersecting identity factors (e.g., gender, Indigenous status, language, race). GBA+ intends to help assess and identify objectives around EDI.

The evaluation found that CCAP has implemented initiatives to account for EDI factors as it evolves (see Figure 9). Specifically, CCIAD-AP has become more inclusive by including the populations it had historically not engaged and by ensuring that the Indigenous perspectives on climate change issues are brought into the program in a meaningful manner. CCGP activities have a strong focus on northern and coastal areas that have a relatively high population of Indigenous peoples. Taken together, these initiatives have helped strengthen the connections among the GC’s climate change adaptation objectives and EDI objectives; they have also helped strengthen the GC’s reconciliation with Indigenous peoples and recognition of Indigenous peoples’ right to self-determination. Given the encouraging findings, CCAP should maintain the momentum to further advance this critical aspect of the programming. Indeed, CCIAD-AP management indicated that they were looking at changing the Plenary model to allow for a broader engagement, such as supporting the engagement of under-represented groups. There are also opportunities for the program to consider different social identities in the context of climate change impacts and adaption to be more equitable, diverse, and inclusive^{Footnote 10}.

Figure 9: Examples of EDI in practice

Effectiveness – To what extent has the program progressed in achieving its intended outcomes?

Outputs

CCAP has produced and disseminated its outputs as planned.

Evidence indicated that CCIAD-AP has produced and disseminated outputs as planned in its program of work. Products (e.g., knowledge and tools) in the timeframe evaluated were disseminated using a variety of approaches, such as the series of webinars managed by the Platform Secretariat, the NKA website, the NRCan website and GEOSCAN. Several knowledge products were also disseminated by networks of stakeholders and project partners as well as by organizations [e.g., Ontario RAC, Prairies RAC, and Fraser Basin Council in British Columbia (BC)]. Over the evaluation period, these products provided support to achieving program milestones and intended outcomes.

Evidence showed that CCGP scientists and supporting teams have produced and disseminated outputs as planned. Outputs produced in the timeframe evaluated included the publication of a variety of reports and articles, organization of several community meetings, delivery of informative presentations, provision of advice to deal with climate change issues related to permafrost and coastal erosion, as well as development of information systems and related databases available to stakeholders across Canada and internationally. During the evaluation period, over 750 products provided support to the program milestones and intended outcomes.

Immediate Outcomes

Target stakeholders have access to CCIAD-AP products, increased awareness of climate change impacts, and increased capacity to use CCIAD-AP products.

Evidence revealed that the Platform has facilitated access to knowledge and tools as well as increased awareness of climate change impacts. The Platform provides a focal point for stakeholders to advance and exchange knowledge and tools, enabling stakeholders to participate in the establishment of research and knowledge priorities. For instance, interviewees stated that the Platform brought new ideas and information for discussions, while contributing to knowledge and tools exchange and dissemination. The Platform Plenary allows stakeholders to stay informed of the climate change adaptation field. Interviewees agreed that the discussions in the Plenary meetings helped them to learn about new research, potential climate change impacts on their communities or sectors, as well as their potential contributions to the field. The Plenary also helps build a common awareness of climate adaptation challenges and options (e.g., collaboration opportunities on common interests between Provincial and Territorial Governments, industry associations, municipalities, and Indigenous groups). Access to knowledge and tools to a wider audience is enhanced by networking opportunities (e.g., Plenary’s network of networks; see Figure 3), co-funded projects [e.g., climate-related disclosures guidelines by the Chartered Professional Accountants of Canada (CPA Canada) and the professional practice guidance for engineers by Engineers Canada], and the Adaptation Platform Workspace (see Box 4). In particular, the Workspace allows members to gain access to knowledge and tools as well as to share these products with organizations and people within their networks. An external study^{Footnote 11} commissioned by CCIAD-AP indicated several avenues that the program could explore to enhance access to CCIAD-AP products and increase awareness of climate change impacts. For example, create a strategic plan with participation of CCIAD leadership and Plenary that assures “digital culture success” and sustainable long-term operations of the virtual platform.

Evidence demonstrated that the Platform’s WGs have enhanced stakeholders’ access and awareness through their State of Play reports and work plans. For instance, interviewees said that these reports were useful because they provided an overview of the operating context as well as inform on their climate change adaptation efforts. The WGs also facilitate discussions and actions on cross-cutting issues through their meetings and discussions, which is noted to be beneficial for knowledge sharing by interviewees. The G&C projects are inspired by the WGs work plans. Since the launch of the G&C program in 2017, CCIAD-AP has approved 39 projects with a total value of $8.36 million. These projects help fill the knowledge gaps and priorities identified in the various WGs’ reports and work plans. Table 10 illustrates the topics and the specific themes covered by the approved projects.

Table 10: Themes Covered by Adaptation Program G&C Projects 2017-2021

Topic	Theme
Topic 1: Coastal Management	1.1 Alternative and Innovative Options to Hard Protection Infrastructure Solutions
	1.2 Expanded or Innovative Policy Instruments
Topic 2: Energy	2.1 Economic Decision-Making
	2.2. Adaptation Measures and Options
	2.3 Building Capacity in the Energy Sector
Topic 3: Mining	3.2 Canadian and International Best Practices in Mining
	3.3 Risk Assessments on Existing Orphaned or Abandoned Mines
Topic 4: Economics	4.1 Economic Analysis of Costs and Benefits of Climate Change Adaptation
	4.3 Investment Analysis Disclosure
Topic 5: Infrastructure	5.1 Nature Based Infrastructure Solutions in a Changing Climate
	5.2 Infrastructure Interdependencies – Impacts and Adaptation Opportunities
	5.3 Other Infrastructure Projects
Topic 6: Forestry	6.1 Integrated Assessments in the Forestry Sector
Topic 7: Adaptation Training	7.1 Building Capacity of Professionals

Evidence demonstrated that Platform’s RACs has enhanced stakeholders’ access and awareness. Interviewees considered RACs to be contributing to the improvement of the levels of awareness. They also considered RACs to be contributing to the understanding of vulnerabilities, climate change adaptation options, and integration of climate change information in regional planning and decision-making.

Evidence revealed that the NKA process has facilitated stakeholders’ access and awareness through its reports and interactive website. The NKA process also publishes its reports on climate change in Canada in GEOSCAN and on the NRCan website. In 2019, it implemented a new digital delivery strategy through its website. Instead of the one paper report released once every five years, it became a digital delivery of a series of reports released over a five year-period, providing an interactive way to communicate key findings and concerns to decision-makers. The website and series reports’ release were communicated using NRCan’s social media. The social media promotion and interactive website have helped increase access to the NKA process’ products. At the time of the evaluation conduct, the website was viewed more than 350,000 times (as compared to 30,000 on NRCan website) with the release of the “Canada in a Changing Climate” report on April 2, 2019. The Canada in a Changing Climate – 2019 User Survey showed that most of the polled stakeholders knew about the NKA process report and its related products. Regarding the regional risk assessments, six provinces expressed interest in doing the risk assessments and the evaluation team found that one has completed it and the others were progressing towards their completion.

Evidence demonstrated that BRACE has facilitated access and awareness in its stakeholders. For instance, interviewees stated that the quarterly newsletters, webinars, conference call meetings, and in-person workshops fostered collaboration and knowledge-exchange among BRACE projects. Evidence also demonstrated that BRACE is instrumental in increasing climate change adaptation capacity across Canada. Interviewees stated that the support for provinces, territories, and their constituents via BRACE helped stakeholders to develop information, tools, and training required to build climate adaptation expertise for their members (see Box 5).

However, interviewees raised concerns that the work in this space still needs to be sustained and scaled up. The preliminary results from BRACE projects showed that the basic milestones for building an ecosystem for climate adaptive capacity development were on the right track and could serve as a springboard for a wider deployment of skills sought by stakeholders. Therefore, evidence suggests that there is a need to maintain and perhaps expand BRACE activities.

Target stakeholders have increased awareness and use of CCGP products.

CCGP uses three NRCan information systems (as well as external ones) to make geoscience research findings easily accessible to users and stakeholders in Canada and internationally (see Box 6), increasing target stakeholders’ awareness and use of CCGP data, knowledge and tools. Evidence showed that CCGP invested considerable efforts in PIN and CanCoast for its end-users and stakeholders during the evaluation period. CCGP also uses GEOSCAN, which is operated and managed by the NRCan Information Management Division.

Evidence also revealed that CCGP has enhanced awareness and use of its products through fieldwork, research, meetings, and presentations. Specifically, CCGP scientists often use fieldwork visits to meet with local stakeholders and develop long-term relationships. These encounters often include CCGP plain language documents and press articles. Interviewees explained that through these relationships, CCGP scientists and stakeholders were able to foster mutual learning. These relationships enable CCGP scientists to implement the best approach to meet stakeholder needs, which increases the likelihood of stakeholders using CCGP products. For instance, CCGP organized a science day at Tuktoyaktuk in August 2019 to share and learn about ongoing climate change issues and related research in the Western Arctic; more than 200 individuals participated in the event including other scientists, federal and territorial government partners, universities, and community members. Additionally, annual fieldwork over decades allows CCGP scientists to collect and analyse long time series (e.g., CCGP has monitored glaciers in the Arctic and Cordillera for 60 years) and share significant findings via scientific publications and data.

Evidence demonstrated that CCGP has also increased access and use of its products through departmental and interdepartmental collaborations. The program participates in departmental and interdepartmental committees to work collaboratively with federal programs and organizations, such as ECCC, Fisheries and Oceans Canada (DFO), CIRNAC, TC, as well as Defence Research and Development Canada. There are 20 departments involved in the interdepartmental coordination of the PCF – Adaptation and Climate Resilience Pillar. The collaboration mechanisms used are the ADMs oversight committee, DGARC, and DARTT. Both CCIAD and CCGP participate in these committees, co-chaired by senior management from CCIAD/HAOB. According to interviewees, the discussions held through these mechanisms helped sort out responsibilities as well as decide on projects and performance. These informal structures allow departments to be aware of what other departments are doing, enabling better coordination and communication. These collaborative relationships also facilitate the publications of important reports (e.g., 2019 Canada Changing Climate Report) as such work requires significant inter-departmental committee participation. The program also invites numerous representatives from across NRCan and the federal government to events (e.g., Virtual Workshop of GSC’s CCGP) to raise awareness of the program’s activities and achievements and increase use of its products.

Evidence demonstrated that CCGP has increased access and use of its products through the publications of scientific literature for a broad audience. For instance, CCGP scientists shared their expertise via the publication of the 2019 Canada’s Changing Climate Report, led by ECCC. This report included CCGP scientists’ findings related to permafrost, glacier mass balance monitoring, extreme events, and coastal erosion. Often, the CCGP information included in such reports build on several years of research, for example, the assessments in Canada’s Marine Coasts in a Changing Climate published by NRCan in 2016. Furthermore, in 2019, GSC started to publish annually its most significant achievements in the GSC Report on Results and Delivery (most recently released for 2020-21). Each year, a section of the report is dedicated to the key achievements of the CCGP program areas.

Target stakeholders have participated in and contributed to the identification of climate change preparedness and adaptation activities.

Evidence showed that target stakeholders have participated in and contributed to the identification of priorities for climate change adaptation, enabled by CCIAD-AP. For example, interviewees affirmed that they contributed to setting priorities through their participation in the program’s key delivery mechanisms. The Platform activities help stakeholders to contribute to, as well as identify current and future priorities. Evidence showed that, during the evaluation period, the attendance of the Platform Plenary meetings to identify emerging issues and opportunities for climate change adaptation included 49 to 70 participants representing 29 to 37 member organizations, exceeding the target of 30 participants. Stakeholders are also able to provide input in the development of the Platform’s WGs’ State of Play reports and work plans as well as the NKA process to advance the identification of priorities. For instance, document review and interviews showed that the NKA process team conducted a user survey at the beginning of every cycle to obtain feedback from stakeholders on types and formats of information needed as well as topics of interest. The NKA process team also engages with its advisory committee members, authors, and referrers to get advice, input, and help with the NKA assessment’s structure. These consultation mechanisms help inform the assessment reports and knowledge products that eventually influence climate adaptation activities led by stakeholder organizations. Examples of priorities provided by interviewees included questions linked to EDI, the role of finance and investment in climate adaptation, as well as the integration of Indigenous knowledge.

The priority setting process for the G&C program is communicated at the beginning of each program cycle to make sure that the stakeholders of the Platform Plenary and the WGs understand the complex nature of the funded projects and their need to work with the proponents. Most of the G&C funds are committed early in the program cycle. Therefore, new members, particularly those who join in the middle of the cycle, are unable to participate in the priority setting process. CCIAD-AP offers other opportunities to contribute ideas to the Plenary and WG discussions by asking their members to raise any issues that they are facing.

Evidence showed that end-users have access to knowledge products to support the identification of priorities for climate change preparedness and adaptation, enabled by CCGP. Research, fieldwork, and partnerships help CCGP scientists discover new ways to assess, monitor and report on climate change impacts. When new analyses and results are generated, scientific reports are written and published on GEOSCAN to communicate the discovery, such as a research paper published in 2019 by CCGP scientists in the Extreme Events Program^{Footnote 12}. These publications provide support to end-users to identify priorities related to climate preparedness and adaptation (see Box 7, 8, 10, and 13).

Further, CCGP products including those made available in PIN and CanCoast, support end-users with climate change preparedness and adaptation. They provide specific information on higher risk areas due to permafrost conditions and coastal sea changes. For example, TC, northern communities and their infrastructure planners and service providers, as well as many others, can use PIN to know the conditions of northern soils to help assess climate change impacts on roads, houses, buildings and other infrastructure. Likewise, a variety of stakeholders, including different levels of government, coastal engineers and industry consultants, use CanCoast to determine where flooding and coastal erosion caused by climate change are likely to occur this century.

Intermediate Outcomes

Target stakeholders have identified climate change adaptation measures to address risks and opportunities arising from climate change.

Evidence showed that target stakeholders have identified climate change adaptation measures due to their involvement in CCIAD-AP. The nature of the climate adaptation measures provided as examples by interviewees spans all stages of the climate adaptation continuum to reflect the nature of the stakeholders’ work and level of advancement in integrating climate adaptation. That is, the evaluation did not use a standard list of climate adaptation measures to assess program performance for this indicator because climate adaptation measures are diverse, and the implementation of climate adaptation measures is a result that comes further in the results chain. Taking these into consideration, evidence showed that stakeholders are taking significant steps to identify climate adaptation measures. For example:

• The WGs’ State of Play reports list a variety of available climate change adaptation measures developed with the program’s support or by other stakeholders. For instance, the Infrastructures & Buildings WG’s State of Play Report contained a series of different measures, such as climate data, infrastructure design requirements and methods to implement climate adaptation into infrastructure management planning.
• The regional risk assessments funded through the NKA process help provincial governments with the identification of vulnerabilities, risks, and priorities, enabling some of these provinces to act on the relevant vulnerabilities by exploring climate change adaptation options through the Platform and BRACE projects (e.g., innovative coastal protection measures to address coastal erosion issues).
• Co-funded projects develop guidance (e.g., the Decision Support Tool for Climate Change Adaptation in Small Coastal Communities in Atlantic Canada by RAC Atlantic supported decision-makers in identifying the most feasible climate adaptation options) and provide case studies (e.g., the Climate Change Adaptation for the Mining Sector guide focused a site-specific process to identify climate adaptation options) on how to identify and select climate adaptation options.
• Several professional associations have identified their climate change adaptation measures. For example, the program’s information helped CPA Canada to develop the education and the knowledge necessary to embed climate adaptation in its material and guidance. The program also helped Engineers Canada to integrate climate adaptation considerations into their professional practice guidelines. In addition, climate adaptation measures are identified as part of the engineering projects using the PIEVC protocol.

Decision-makers from all levels of government that are involved in CCGP have made science-based decisions related to climate change adaptation.

Evidence demonstrated that target stakeholders have made science-based decisions on climate change adaptation due to the work produced by CCGP. All CCGP program areas have contributed to the implementation of climate adaptation measures as illustrated by the featured examples (see Box 7, 10, and 13), including in glacier mass monitoring (see Box 8).

Long-term Outcomes

Target stakeholders have included climate change adaptation in their plans and strategies.

Evidence, such as that from NRCan’s 2018 Benchmark Survey, indicates that target stakeholders have included climate change adaptation in their plans and strategies because of the influence of CCIAD-AP. CCIAD-AP’s targets were to have 30% of businesses and 40% of communities surveyed report the inclusion of climate change adaptation in plans and strategies by 2023 and 2026 respectively. As shown in Figure 10, while these targets have not yet been achieved, early results show progress on the right trajectory for success. Interviewees corroborated the observed trends, reporting that their involvement with the program helped them and those in their networks to advance the integration of climate adaptation in their plans and strategies, despite implementation still being at an early stage.

Figure 10: Percent of Stakeholders Reporting the Inclusion of Climate Change Adaptation in Plans and Strategies in NRCan Benchmark Survey, 2012 and 2018

In 2018, a Collaborative Audit of Federal, Provincial and Territorial Climate Change Programs^{Footnote 13} highlighted that climate change action in Canada was still a work in progress for GC as well as the provincial and territorial governments. The latter were indeed found to be at disparate levels of progress and development in terms of their jurisdictions’ climate risk assessments comprehensiveness, climate change adaptation strategies, and action plans specifications. Nevertheless, evidence showed that governments at all levels have introduced climate adaptation in their plans and strategies because of the influence of CCIAD-AP. CCIAD-AP provides departments and agencies from all levels of government with the evidence to raise the profile of and help build a case for climate adaptation. For instance, interviewees noted that federal departments and agencies as well as Provincial and Territorial Governments (see Box 9) used the Plenary and NKA’s processes and reports to advance key policy initiatives and identify potential climate adaptation measures. This includes the National Adaptation Strategy that was being developed during the evaluation. Further, CCIAD-AP has stimulated progress by funding regional risk assessments updates in six provinces under the NKA process. Interviewees from participating provinces and territories found the NKA process funding for regional assessment was useful because the assessment provided them with more tailored information on what their sectors and departments needed to integrate climate adaptation in their strategies and plans.

Evidence showed that the inclusion of climate change adaptation measures in stakeholders’ plans and strategies is occurring in sectors and professional associations. For example:

• The Platform Mining WG engaged the Canadian Minerals and Metals Plan Secretariat on climate change adaptation priorities throughout the development of the Canadian Minerals and Metals Plan^{Footnote 14}. Launched in 2019, the plan included climate adaptation as a key action as well as provided the strategic directions for the mining industry and its stakeholders.
• CCIAD-AP supported a project led by the Canadian Electricity Association to develop an approach to incorporate climate change adaptation considerations into the electricity sector’s infrastructure planning and investment decisions.
• CCIAD-AP contributed to the launch of the January 2019 Climate Adaptation Leaders Forum – Finance and Investment Pilot that resulted in finance and investment community’s commitment to support climate change adaptation strategies in the Canadian investment value chain.
• CCIAD-AP endorsed and informed CPA Canada’s strategic priorities in response to increasing pressures for more climate-related risks disclosure, which helped shape their climate change adaptation project activities (e.g., training and practice guidelines for both private and public sectors organizations).

Target stakeholders have implemented adaptation measures.

The implementation of climate change adaptation measures is beyond CCIAD-AP’s control. The decision to implement climate adaptation measures lies with the responsible authorities rather than the program. The CCIAD-AP work respects the existing decision-making power of the responsible authorities for the implementation of climate adaptation measures. Moreover, the implementation of climate adaptation measures depends on various external factors, such as the availability of funding to implement and maintain climate adaptation measures. The program does not provide any funding support for the implementation of climate adaptation measures.

Despite these limitations of CCIAD-AP, evidence showed that target stakeholders have begun to implement climate change adaptation measures because of the influence of CCIAD-AP. The program has contributed to the dissemination of information about climate adaptation measures, resulting in the implementation of climate adaptation measures in different sectors across Canada (see Table 11). However, interviewees emphasized that the implementation of climate adaptation measures was just at the beginning phase and more work would be required to strengthen the momentum. Indeed, not all target stakeholders have yet implemented the climate adaptation measures in their plans or strategies.

Table 11: Examples of Climate Change Adaptation Measures Implemented by Stakeholders

Type of Climate Change Adaptation Measures	Example
Development of climate change adaptation plans and strategies	Increase in the number of municipalities with a complete climate adaptation plan that includes risks to infrastructure. For example, the Province of BC as well as the cities of Vancouver and Campbell River planned and implemented measures on the ground that would make them better adapted to flooding from sea level rise.
Development of or amendments to existing policies and programming	Changes introduced to DFO’s Small Craft Harbours program to address the impacts of sea level rise when renewing facilities or maintaining existing ones. DFO oversees more than 1500 structures across Canada. Development of the BC Ministry of Transportation and Infrastructure’s Best Practices report, which provides guidance on integrating climate change into highway infrastructure management (including planning, engineering, and operations activities).
Management and regulation of built or natural physical structures	Managed realignment of the Truro-Onslow Dyke infrastructure in the province of Nova Scotia was moved back to allow the reintroduction of tidal flow and restoration of the Tidal Wetland. Repair of Bercy maritime promenade in the Gaspé Peninsula. Implementation of natural infrastructure beach regeneration at the Saint-Omer bridge in Chaleur Bay in Carleton-sur-Mer, Québec. Some municipalities have protected naturalized areas for managing storm water and storm flooding events. For example, the Naturalized Storm Water Management Pond (City of Dieppe, New Brunswick), and the green rainwater infrastructure and rainwater management initiative (City of Vancouver, BC).
Disclosure of climate risks and adaptation measures	In collaboration with CCIAD-AP, CPA Canada created materials for education and professional learning to help incorporate climate adaptation in accountants’ work. Research conducted by CPA Canada in 2016 and 2019 concluded that improved accounting practices related to climate change adaptation measures were used by several organizations. They were also disclosing information on climate risks and climate adaptation measures implemented in their communications with stakeholders and the broader Canadian community.

Evidence showed that target stakeholders have begun to implement climate change adaptation measures because of the influence of CCGP. All CCGP program areas were found to influence the implementation of climate adaptation measures (see Box 7, 8, 10, and 13), particularly in relation to permafrost and coastal adaptation.

Ultimate Outcome

Resilience to climate change impacts is likely improved in Canada regions, communities, and economic sectors.

The overall trends of progress observed in the findings for the immediate outcomes, intermediate outcomes, and long-term outcomes suggest that CCAP is progressing towards its ultimate outcome, with meaningful contributions from CCIAD-AP and CCGP. All lines of evidence showed that CCIAD-AP and CCGP achieved their intended outcomes. There were increased access to and use of resources, increased awareness of climate change impacts and available resources, increased capacity to respond to climate change impacts and risks, as well as increased participation in and contribution to identifying climate change adaptation priorities. Target stakeholders were able to identify climate change adaptation measures as well as make science-based decisions to respond to climate change impacts and risks. Although the implementation of climate adaptation measures was beyond the control and mandate of CCAP, there was evidence of the early implementation of climate adaptation measures and incorporation of climate adaptation in plans and strategies in several instances. Interviewees perceived this to be the case. They stated that the actions that have taken place because of the influence of CCIAD-AP contributed to adaptive capacity building, and thus, resilience. They emphasized that their regions, communities, and sectors were in a better position than they were 10 years ago. However, they acknowledged that more time would be required to be able to concretely assess resilience. Interviewees also highlighted some concerns in the field:

• Challenges in securing sufficient investment have delayed actual implementation of climate change adaptation measures, and thus, it is difficult to assess these measures in the current climate.
• There is an urgency to accelerate adaptive capacity building to build resilience to keep pace with the worsening impacts of climate change.

CCIAD-AP management was found to be aware of the limitations in the field. They recognized the need to accelerate actions on climate adaptation to adequately address the complexity or urgency of the issue. They further indicated that they would be exploring various avenues to accelerate the implementation of climate adaptation solutions, including connecting existing expertise and innovation for a widespread implementation in an equitable and flexible way.

The examples from CCGP program areas featured earlier in the report (see Box 7, 8, 10, and 13) also affirm that CCAP is progressing towards the ultimate outcome through the contributions of CCGP. Another example to further illustrate how CCGP, and by extension CCAP, is progressing towards the ultimate outcome can be observed from CCGP activities in the Northwest Territories.

Performance Measurement Strategy

CCAP has put in place several performance measures but there are areas for improvement to increase efficiency in collecting performance information.

Despite the promising trends of progress observed, there remains some challenges in the measurement of results. Evidence showed that CCIAD-AP has several performance measures in place. For instance, the program has started to conduct a recurring survey every four years since 2018. The program commissioned the NRCan Benchmark Survey, conducted by Earnscliffe Strategy Group^{Footnote 16}. Information collected from this survey was found to be reasonably informative for assessing progress towards the immediate outcomes, such as access to information, expertise, and tools that support climate change adaptation (e.g., percentage of communities reporting having access to the information and tools), awareness about climate change impacts (e.g., percentage of businesses reporting to be aware), and stakeholders’ capacity to apply tools and information (e.g., percentage of communities reporting having capacity to apply the tools and information). During the evaluation period, the 2022 NRCan Benchmark Survey had not yet been completed. The program has also implemented a continuous, systematic tracking method for some of its key delivery mechanisms. For instance, the NKA process has opted for a full digital delivery through a dedicated interactive website, which facilitates obtaining web analytics and social media data related to its products. The NKA process team also uses their website to provide the public the opportunity to see real-time state of progress in the development of the various reports in the series.

Evidence also showed areas for improvement. While a continuous, systematic tracking has been implemented for the NKA process, other key delivery mechanisms do not have a similar system. For instance, the Platform does not have a real-time performance monitoring system for its outputs (e.g., WGs’ and co-funded products), dissemination activities, and access information. Some performance information (e.g., download of the outputs) is only uploaded to the databases at the end of the program cycle, rendering some challenges in tracking the Platform’s performance during the cycle. CCIAD-AP does not have a complete report on the production and dissemination of “arm’s length” products that are not owned by the program (e.g., output production, dissemination, and utilization by stakeholders), and it also does not have the means to obtain such information (e.g., statistics on direct downloads). Furthermore, some methods to collect performance information could be streamlined. For instance, interviewees noted that while BRACE and the Platform’s financial reporting gave project monitoring managers the opportunity to review the progress of their projects with project leads, the financial reporting requirements were quite onerous.

Evidence demonstrated that CCGP also has several performance measures in place. CCGP tracks key performance information as part of its management processes. Annually, CCGP approves its projects and allocates its budget, including human and financial resources. CCGP also reports its key achievement in GSC Annual Reports, which describe CCGP’s significant achievements. The program also produces other reports that include information on progress. For example, three senior CCGP scientists reported on their most significant findings in the 2019 Canada’s Changing Climate Report issued by ECCC. However, there are areas for improvement. The program has not been consistent over the last five years (i.e., the evaluation period) when tracking quantitative statistics and performance information on its achievements, including indicators related to their two main information systems (i.e., PIN and CanCoast). It should be noted that these statistics are managed by other groups at NRCan than CCGP.

Performance information for the longer-term outcomes remains a challenge for CCAP. It is difficult to measure and draw a direct attribution of any change in climate change adaptation and resilience to the program. The program has no control over the implementation of climate adaptation measures. While the program committed to developing case studies to demonstrate the achievement of its longer-term result in the case of CCIAD-AP, there was no evidence of the implementation of this measure during the evaluation.

Conclusion

The objective of this evaluation was to examine CCAP’s capacity to adapt to the evolving context, the extent to which the program has put in place measures to achieve EDI objectives, and the program’s contribution to achieving its intended outcomes with an emphasis on the long-term outcomes. The evaluation also examined the implementation of recommendations from the previous evaluation, as well as best practices and lessons learned related to program design and delivery. Overall, the evaluation found that CCAP has achieved meaningful progress and achievement, although there are few areas for improvement as reflected in the recommendations.

Does the Program model support the achievement of Program objectives? All lines of evidence suggest that the CCAP program model supports the achievement of the program objectives, as part of the PCF. Both CCIAD-AP and CCGP have the capacity to adapt to the evolving context while maximizing the program impacts in an efficient and economical manner. CCIAD-AP and CCGP each provide unique and important functions while collaborating on issues and activities of common interests. The evaluation also identified potential approaches that CCAP could consider to further enhance its performance. Notably, while the role that CCAP plays is likely to remain important in the future as the impacts of climate change continue to grow, several stakeholders emphasized that CCIAD-AP could play an additional role to catalyse the implementation of climate change adaptation measures given that this remains an important gap and challenge.

To what extent does the program consider EDI factors? Climate change adaptation is an important issue that needs to be considered by all organizations, communities, and sectors to effectively build Canada’s resilience. All lines of evidence suggest that CCAP has put in place measures that consider EDI factors. CCIAD-AP and CCGP have included considerations for EDI factors, with a dedicated focus on Indigenous peoples as well as northern and coastal areas. However, CCAP has the potential to expand its reach and influence, including opportunities to create a more equitable, diverse and inclusive approach in climate adaptation by including Indigenous peoples and other underrepresented groups that are impacted by climate change.

To what extent has the program progressed on achieving its intended outcomes? All lines of evidence suggest that CCAP is progressing on the achievement of its intended outcomes. CCIAD-AP and CCGP have conducted various meaningful activities to enable climate change adaptation, although this is still at the early stage. Despite these promising findings, the evaluation found that performance measurement strategy of the program needs to be improved. Although the observed trends suggest progress towards enhancement in Canada’s resilience, more time is required to make a definitive conclusion for the ultimate outcome. As the long-term results of the program can only be observed in several decades, it is especially important for the program to have a robust performance measurement strategy to ensure that it is able to collect key performance information over time.

Appendix A: Evaluation Matrix

Questions	Indicators	CCIAD - AP	CCGP
To what extent has the Program progressed on achieving its intended outcomes?	1.1 Evidence that outputs are produced and disseminated as planned.
	1.2 Evidence of Program contribution to achieving intended outcomes.
Immediate Outcomes
	1.2.1 Target Stakeholders have access to information, expertise, and tools they need to support adaptation action (CCIAD-AP).
	1.2.2 Stakeholders have an increased capacity to use and apply climate change adaptation tools and information in their work [CCIAD-AP(BRACE)].
	1.2.3 Stakeholders are increasingly aware of climate change impacts (CCIAD-AP), and of NRCan geoscience data, knowledge and tools related to climate change. (CCGP).
	1.2.4 Stakeholders increasingly use NRCan geoscience data, knowledge, and tools related to climate change to inform their climate change adaptation activities (CCGP).
	1.2.5 Stakeholders participate in and contribute to the identification of priorities for adaptation activities in Canada (CCIAD-AP).
	1.2.6 End-users have access to knowledge products to support the identification of priorities for preparedness and adaptation activities in Canada (CCGP).
Intermediate Outcomes
	1.2.7 Adaptation measures have been identified by targeted stakeholders to address risks and opportunities arising from climate change (CCIAD-AP).
	1.2.8 Federal, provincial, and territorial decision-makers make science-based decisions related to climate change adaptation (CCGP)
Long-term Outcomes
	1.2.9 Adaptation measures are implemented by targeted stakeholders (CCIAD-AP & CCGP).
	1.2.10 Adaptation to climate change is included in stakeholders’ plans and strategies (CCIAD-AP).
Ultimate Outcome
	1.2.11 Resilience to climate change impacts is improved in Canada regions, communities, and economic sectors.
Does the Program model support the achievement of Program objectives?	2.1 Evidence that program design (e.g., decision-making structures, resource allocation, design and delivery mechanisms, collaborative agreements, and working arrangements) support program objectives as part of the PCF.
	2.2 Evidence that program evolved to meet new or changing needs of stakeholders.
	2.3 Views of stakeholders of alternatives in the design and the delivery of the program.
	2.4 Identification of lessons learned and evidence of best practices from internal as well as external sources applied.
	2.5 Identification of factors influencing program performance and assessment of their impact.
	2.6 Identification of unmet needs and unintended (positive & negative) outcomes, if any, of CCAP.
	2.7 Extent to which the implementation of recommendations from previous evaluation contributed to the achievement of the Program’s outcomes.
To what extent does the Program consider EDI factors?	3.1 Evidence that program activities and outputs consider EDI factors, particularly in engagement, consultation, integration of Indigenous knowledge, and communications strategies.
	3.2 Documented examples of accommodation of EDI factors.

Appendix B: Evaluation Team

Michel Gould, Chief Audit and Evaluation Executive
Stephanie Kalt, Director of Strategic Evaluation
Marc Riopel PhD, Evaluation Manager
Christian Kratchanov, Evaluation Manager
Maria El-Omari, Senior Evaluation Analyst
An Gie Yong PhD, Evaluation Officer
Marc-Antoine Bordeleau, Internship Student

The evaluation team would like to acknowledge those individuals who contributed to this project, particularly program representatives and members of the Evaluation Working Group who provided insights and comments as part of the evaluation.