Evaluation of the Geoscience to Keep Canada Safe Program

Audit and Evaluation Branch
Natural Resources Canada

Presented to the Performance Measurement, Evaluation and Experimentation Committee (PMEEC),
June 3, 2022

Table of Contents

• List of Acronyms
• Executive Summary
  • About the Evaluation
  • Evaluation Findings
  • Recommendations and Management Response
• Introduction
  • Program Profile
  • Evaluation
• What We Found
  • Relevance
  • Question 1: Is the program role appropriate and necessary within the context of changes in the program operating environment?
• Effectiveness – Progress on outcomes
  • Question 2: To what extent has the program contributed to achieving intended outcomes?
  • Immediate Outcomes
  • Intermediate Outcomes
  • Long-term / Ultimate Outcomes
• Program Design and Delivery
  • Question 3: Does the program model support the achievement of program objectives?
• Equity, Diversity and Inclusion (EDI)
  • Question 4: To what extent does the Program consider EDI factors?
• Best Practices and Lessons Learned
  • Question 5: Are there any best practices and lessons learned related to the design and delivery of the program?
• Conclusions
• Appendix A: Evaluation Matrix
• Appendix B: Description of Sendai Priorities

List of Acronyms

ADM, LMS	Assistant Deputy Minister, Lands and Minerals Sector
AEB	Audit and Evaluation Branch
BC	British Columbia
CCGP	Climate Change Geoscience Program
CHIS	Canadian Hazards Information Service
CSSP	Canadian Safety and Security Program
DRR	Disaster Risk Reduction
EDI	Equity, Diversity and Inclusion
EEW	Earthquake Early Warning System
EM	Emergency Management
EMA	Emergency Management Act 2007
EMS	Emergency Management Strategy for Canada
FTE	Full-Time Equivalent
GBA+	Gender-based Analysis Plus
GC	Government of Canada
GEMF	Global Earthquake Model Foundation
GKCS	Geoscience to Keep Canada Safe
GSC	Geological Survey of Canada
HAOB	Hazards and Operations Branch (of LMS)
ICAO	International Civil Aviation Organization
IOC	Intergovernmental Oceanographic Commission
LMS	Lands and Minerals Sector
NBCC	National Building Code of Canada
NRCan	Natural Resources Canada
PS	Public Safety Canada
PSG	Public Safety Geoscience
SF	Sendai Framework for Disaster Risk Reduction
USGS	United States Geological Survey

Executive Summary

About the Evaluation

This report presents the findings, conclusions, and recommendations of an evaluation of the Geoscience to Keep Canada Safe Program (GKCS) at Natural Resources Canada, which includes Lands and Minerals Sector (LMS) activities related to geo-hazards delivered by the Geological Survey of Canada (GSC) and the Hazards, Adaptation and Operations Branch (HAOB). The immediate goal of the Program is to increase available scientific knowledge and provide alerts on geo-hazards and space weather. Decision-makers in governments and critical infrastructure then use this information to inform decisions related to public safety and emergency management so that Canada is resilient to disasters in the long-term.

The evaluation covered the six-year period from 2014-15 to 2019-20 and actual expenditures of approximately $93 million. The evaluation scope included science, research, and monitoring activities carried out by the GSC’s Public Safety Geoscience Program (PSG) and HAOB’s Canadian Hazards Information Service (CHIS) in relation to geo-hazards, specifically those that are abrupt, yet infrequent incidents such as earthquakes, volcanoes, landslides, tsunamis, and geomagnetic storms. To meet a specific Treasury Board commitment, the evaluation scope also included CHIS’ National Earthquake Monitoring Initiative (NEMI) funded between 2014-15 and 2018-19. Data was collected through review of literature and program documents; interviews with 22 internal and external stakeholders, including four external experts; an online survey of stakeholders as targeted by the Program; and three case studies, 1) space weather, 2) Thompson Valley Landslide and 3) NEMI.

The evaluation examined long-standing activities that were evaluated in 2010 and 2014, and for which there was ample evidence of consistent progress on immediate outcomes over time. The 2014 evaluation was unable to make conclusive findings on intermediate outcomes. Since there was no indication that the Program’s ability to achieve progress on immediate outcomes had changed, this evaluation focused on identifying more conclusive findings regarding intermediate and long-term outcomes.

The Program operating context changed significantly since the 2014 evaluation. Canada signed the Sendai Framework for Disaster Risk Reduction (SF) in 2015, and then developed the Emergency Management Strategy for Canada (EMS) in 2017. These instruments re-focused efforts in disaster risk reduction and emergency management towards risk assessment and disaster prevention instead of loss reduction. In doing so, they introduced a whole of society approach¹ to disaster prevention and risk assessment. The evaluation assessed the extent to which the Program role, as government’s subject matter expert for geo-hazards, remained relevant in this new operating context.

Evaluation Findings

Relevance in a changing operating environment

The Program remained highly relevant to government objectives around public safety and emergency management. All sources agreed that the subject matter expertise found within the Program was a valuable asset for Canada in disaster risk reduction and emergency management. However, the evaluation identified a need for knowledge translation and dissemination of science information to an expanded group of stakeholders and users. Meeting this need is important to the whole of society approach to disaster risk reduction adopted in the SF and EMS.

While all sources indicated that the Program had a critical role in accommodating this need by virtue of the expertise it housed, evidence was mixed as to the precise nature of the role the Program should play. This contributed to confusion among external stakeholders over the role of the Program in relation to federal involvement as led by Public Safety Canada. Some interviewees envisioned an increased public facing role in engagement and collaboration with non-technical users. Others saw NRCan as having an expanded leadership role and viewed NRCan as well-positioned to accommodate the role given its successful collaborations with external stakeholders to date. However, these perspectives interpreted the Program role in a way that that goes beyond what the Program was designed to support. In this light, NRCan acts as subject matter expert supplying information to government partners who are then responsible for the diffusion of understanding. In addition, it raised the challenge of capacity within the Program to meet an increased demand for its expertise yet maintain its unique role in research to understand geo-hazards. The evidence suggests that there is a need to review the Program role and contribution to the enabling environment for knowledge translation for disaster risk reduction and public safety in light of the changing expectations generated by the EMS and SF.

Progress on Outcomes

Similar to the findings from the 2014 evaluation, the evaluation found that the Program again made significant progress on immediate outcomes. An increased body of authoritative scientific knowledge and information was directly attributable to the Program. Models and tools to assess risk were available, but were considered somewhat inaccessible to a layperson user. Alerts and notifications consistently met standards for timeliness.

Progress was also evident at the intermediate level of outcomes. Nine in 10 survey respondents agreed that decisions about disaster risk reduction and emergency planning were made using scientific evidence and knowledge. Survey respondents also held the view that NRCan information played a significant role in their own increased understanding of geo-hazards, as well as that of those responsible for taking decisions. Furthermore, many stakeholders and partners said there would be an impact on their organization if NRCan geo-hazards information was unavailable or not produced.

Success was closely connected to collaborative relationships and capacity. Where these existed, uptake of information and use of tools and models in risk assessment was clearly evident. For example, CHIS worked with the Nuclear Waste Management Organization (NWMO) to evaluate potential sites for a deep geological repository under a long-standing agreement. The PSG participated in a strong collaboration funded by the Canadian Safety and Security Program (CSSP) to develop a rapid risk assessment for seismic hazard that induced local plans for mitigation. Through interviews and the survey, stakeholders reported that they felt better equipped to respond to hazard events.

The evaluation also found evidence that moderate progress was being made on the longer-term and ultimate level outcomes. Survey findings showed that respondents expressed a growing level of confidence that they were better prepared for hazard events, and attributed their confidence to NRCan. While there was significant collaboration and the Program benefited from joint efforts in identifying information and filling information gaps, plans to reduce gaps often lacked further development and resources to be implemented. While some of this activity fell outside of the direct mandate of NRCan, stakeholders expressed the view that NRCan (and the Program) was implicated due to the need for multidisciplinary tables to deal with the gaps in knowledge transfer and the critical fact that expertise in Canada largely resided within NRCan. On the whole, interviewees and survey respondents expressed cautious optimism that Canada was more resilient. However, experts and some other interviewees noted that measuring resilience was challenging. They noted that while Canada has experienced disasters, the country had yet to be tested by a catastrophic incident with the extensive damage to critical systems and trauma a catastrophe entails.²

Design and Delivery

The Program was well-designed to support outcomes related to research and monitoring at the immediate outcomes level. Evidence from all sources clearly demonstrated that projects were delivered on time and on budget, and that activities reflected stakeholder needs. However, the Program was less well-designed to support the whole of society approach to disaster risk reduction.

NRCan lacked the mechanisms to collaborate across hazards internally. Both internal and external interviewees noted that information was not easily shared among other NRCan programming with similar goals in public safety and resilience. Several Program personnel provided examples of this, including no uptake on suggestions to co-locate equipment and an unclear process to access data and imagery held in other sectors. Others noted that NRCan groups came together effectively to coordinate NRCan’s involvement in the action plan to implement the federal role in the EMS and suggested that a more permanent table for sharing information in disaster risk reduction and emergency management be established within NRCan.

External stakeholders were confused by the hazard-specific groups originating within NRCan. From their perspective, these groups drew on their limited capacity to conduct risk assessments and expressed concern that multiple engagements would exceed their ability to participate. Government partners in disaster risk reduction and emergency management noted the risk of having hazard-specific approaches to risk assessment that conflicted with the multi-hazard approach adopted. They suggested that NRCan needed a coordinated approach to avoid confusion and ensure common approaches to reduce burden at the local level where little capacity to conduct hazard identification already existed.

Equity, Diversity and Inclusion (EDI)

The Program made strides in integrating EDI considerations into its activities. The evaluation found several examples of the Program’s accommodation of related factors, including: integrating Indigenous knowledge to accurately date historic tsunami events; engaging Inuit translators for community presentations; and participation in Science Day in Tuktoyuktuk. However all sources recognized that more needed to be done to ensure that people with diverse socio-demographic characteristics such as race, gender, cultural identity, location or ethnicity, did not experience negative consequences or barriers due to Program activities. Suggestions were mostly related to consideration of Indigenous populations, including engaging with communities to share research findings and knowledge affecting local communities, and enhancing opportunities for employment in geo-hazards research among Indigenous people.

Best Practices and Lessons Learned

The evaluation identified several practices within the Program that facilitated effective operations. For example, PSG’s mid-year and year-end reports for research projects assisted in tracking project performance and identifying challenges in time for effective mitigation. CHIS adopted a practice of dashboard reporting within the NEMI that contained pertinent information on progress in a readily accessible format. Canada’s membership in the Global Earthquake Model Foundation involved multiple partners. This approach to membership stretched the resources of multiple stakeholders and established grounds for closer collaboration within the country.

The evaluation identified several lessons learned. Firstly, collaboration and relationship-building take time and requires dedicated capacity. Taking this into account during the planning phase of activities improved uptake of information. Additionally, poorly defined roles in overlapping areas of NRCan limits intra-departmental interaction and creates confusion among stakeholders. This reduces opportunities for collaboration, creates confusion regarding reporting responsibilities where objectives overlap, and can lead to an inefficient allocation of NRCan resources.

Recommendations and Management Response

The evaluation found that the GKCS program was well managed and achieved its immediate outcomes, and was able to fulfill very well its primary role as the supplier of scientific and technical information on geohazards to the Government of Canada. However, the implementation of the Sendai Framework and the Emergency Management Strategy for Canada has impacted the context of the program and has implications for achieving its intermediate outcomes. The following recommendations thus reflect the evaluation’s conclusion that the Program should examine its role in view of the changes in the operating environment.

Recommendations	Management Response
1. Recognizing that Public Safety Canada leads the whole-of-society approach for emergency management and disaster risk reduction, the ADM, LMS should continue to work with federal partners to clarify the roles and responsibilities of the GKCS Program in relation to federal involvement in disaster risk reduction, particularly in respect of knowledge translation in a whole of society approach to DRR and EM. The goal would be to clarify roles and expectations in relation to DRR so that the Program can align capacity and manage demands for the DRR expertise housed within the Program.	Management agrees. ADM-LMS commits, via DG-HAOB and DG-GSC, to raise the issue of LMS’s role and expectations in relation to disaster risk reduction with PS. The GKCS Program will continue to deliver scientific, operational and policy advice to PS through ongoing bi-lateral DG (LMS and PS) discussions. Further, LMS commits to participating in a number of interdepartmental DG tables to assist PS to advance whole-of-government disaster risk reduction where there is overlap with LMS’ mandate. This includes the participation of the DG LMS/HAOB at the DG Emergency Management Committee, the DG National Risk Profile Committee, the DG Interdepartmental Table on Earthquake Risk, and, as required, through direct participation in the federal/provincial Senior Officials Responsible for Emergency Management (SOREM) table. DG HAOB will meet regularly with DG GSC and together will ensure that their accountable directors (GSC/Pacific and HAOB/CHIS) raise relevant components of the GKCS Program at these tables.  In particular, LMS’ will ensure its science and policy advice is included in a) the biennial Cabinet Report and b) the biennial Public Report on the National Risk Profile. LMS will continue to support PS and the newly-created Minister of Emergency Preparedness at the Privy Council’s Office as they develop priorities. Position responsible:  DG, HAOB and DG, GSC Timing:  Public Report on NRP, March 31, 2023; Cabinet Report, June 2021 (complete). Additional updates and briefings through PS-led DG level committees.
2. The ADM, LMS should initiate a forward-looking discussion on disaster risk reduction and risk assessment activities across NRCan sectors in order to improve internal coordination of overlapping activities. At minimum, the ADM-LMS should support the GKCS Program to explore the linkages between climate change and hazards research in an effort to a) review LMS’s efforts in knowledge translation to decision-makers under the Emergency Management Strategy funding (Budget 2019) and b) for GSC’s Public Safety Geoscience program, in particular, to advance research linkages between climate change and hazards research for hazards expected to be altered by a changing climate.	Management agrees. ADM-LMS will continue to assess potential for future work within LMS between disaster risk reduction and climate change adaptation, particularly in light of the National Adaptation Strategy in expected in late 2022. a) To clarify LMS approaches in knowledge translation, ADM-LMS will bring LMS’s work on disaster risk reduction, hazard and risk assessment activities to the ADM Policy Committee. This will include presentation of LMS’ efforts to design results for practitioners, which makes the program’s knowledge and science advice on disaster risk reduction accessible to decision-makers. b) LMS does not have the lead role in NRCan for disaster risk reduction. However, within GKCS, in particular the GSC’s Public Safety Geoscience Program, further research work centred on the potential for a changed climate to accelerate geohazard events such as landslides (including those that contribute to flooding), slope stability (marine and terrestrial) and coastal hazards will be examined and inform program delivery further the program’s intermediate outcomes. Position responsible:  a) DG HAOB and DG GSC; b) Director GSC Pacific Timing:  a) Presentation on LMS disaster risk reduction to ADM Policy Committee by January 31, 2023.  b) PSG mid-cycle program adjustments by May 1, 2023.

Introduction

This report presents the findings, conclusions, and recommendations from the evaluation of the Geoscience to Keep Canada Safe Program (GKCS). The Audit and Evaluation Branch (AEB) of Natural Resources Canada (NRCan) conducted the evaluation between August 2020 and June 2021 in accordance with the Treasury Board Policy on Results (2016). The evaluation examined activities of the program between 2014-15 and 2019-20. These activities were last evaluated in 2014-15³.

Program Profile

Program Description

Geo-hazards research and monitoring are long-standing activities of NRCan and carried out to support public safety and security. Through GKCS, monitoring, research and planning related to natural geo-hazards (earthquakes, landslides, tsunamis, volcanoes, and space weather) and human-induced hazards (i.e., radiological and nuclear incidents) are carried out. By providing information to assist decision-makers to better understand and act on hazard risks, the Program helps federal and other levels of government (including international governmental organizations), the private sector, and professional organizations to prevent, mitigate, prepare for, respond to and recover from natural disasters (see figure 1).

Figure 1 : Program Theory Geoscience to Keep Canada Safe (GKCS)

The GKCS program activities are delivered by the Hazards and Operations Branch (HAOB) and the Geological Survey of Canada (GSC) of the Lands and Minerals Sector (LMS) at NRCan. Under the HAOB, the Canadian Hazards Information Service (CHIS) is responsible for the monitoring and alerting systems related to seismicity and space weather. The Public Safety Geoscience Program (PSG) of the GSC carries out the research and fundamental science into geo-hazards as well as risk assessment. These two groups provide information about hazards so that Canadians have information to enhance safety in the event of a natural disaster, model the impacts of hazards, and develop tools to support decision-making and the development of mitigation strategies and action plans. Within the Program, PSG and CHIS have separate functions.

The PSG is primarily a research component, examining geo-hazards and their impacts and conducting risk assessments. The PSG adds to the knowledge base needed about geo-hazards to support evidence-based decision-making, and targets critical gaps in understanding geo-hazard occurrences and their impacts. The PSG program develops tools intended to support emergency management, development, planning, and regulatory decisions to help increase resilience and decrease risk to keep Canadians safe from earthquakes, terrestrial and submarine landslides, volcanoes, tsunamis, and space weather.

The CHIS is primarily an operational entity providing alerts and notifications in order to provide advance warning of hazard occurrences and supporting stakeholder actions for preparedness and response. It also maintains the observational and monitoring equipment and systems that supply data, and conducts research activities in support of operational enhancement. CHIS operations with respect to space weather and seismicity fall within the scope of the evaluation.

Program Operating Context

While GKCS Program activities themselves have remained largely unchanged over time, the scope and priorities have shifted in line with changing approaches to emergency planning and disaster risk reduction. The GKCS Program operates within the larger context of emergency management (EM) and disaster risk reduction (DRR). Approaches in this field have evolved rapidly since 2005, primarily driven by increased frequency and losses from hazard events. Even though floods and wildfires are more frequent and account for most of the losses, sustained public attention on hazards has impacted the level of urgency in preparing for all hazards, including those that are less frequent, but carry a greater possibility of high losses. For example, estimated losses from a potential earthquake in Vancouver rose from $12 billion in 1995 to almost $75 billion in 2013.⁴ In a 2016 report, the Conference Board of Canada estimated losses of $127.5 billion resulting from a catastrophic earthquake on Canada’s west coast with economic impact cascading throughout the country and across industries over a three-year period.⁵

As a result of the sustained attention on emergency management and disaster risk reduction, public policy and programming is rapidly evolving. Internationally, the United Nations Sendai Framework for Disaster Risk Reduction (SF) was signed in 2015. The SF introduced the whole of society approach to disaster risk reduction with a focus on preventing disaster. In signing SF, Canada agreed to implement the Sendai guidance. Moreover, Canada has worked to align its own emergency management and disaster risk reduction strategies to the priorities identified in the SF. The four SF priorities are described further in Appendix B.

In Canada, the field of emergency management and disaster risk reduction is complex. Responsibilities are shared among many stakeholders, with immediate response primarily being a responsibility of local jurisdictions. In addition, Canada’s geography is vast. Risk levels vary across the country by region and type of hazard. Capacity also differs among the provincial, territorial, and municipal jurisdictions. As a result, a “one size fits all” approach is ineffective for the Government of Canada (GC) in carrying out its role. Partnership is therefore a key principle underlying the federal-provincial-territorial agreement Emergency Management Framework for Canada (2017) which established the principles for federal-provincial-territorial cooperation.

The Federal efforts in these areas are led by Public Safety and Emergency Canada (PS) as per the Emergency Management Act 2007 (EMA). The EMA defines the role of the GC and names specific responsibilities for government departments. Under the EMA, as the lead federal ministry, PS led a series of consultations and engagements in 2017 that culminated in an Emergency Management Strategy for Canada: Toward a Resilient 2030 (EMS). In the EMS, Canada articulated a whole of society, all-hazards disaster risk reduction approach for Canada through five priority activities that reflect the SF. In Budget 2019, Canada announced resources to implement the federal portion of the EMS. The GKCS Program received funding to implement an early warning system (within CHIS) and carry out work to contribute to the National Geo-hazards Risk Assessment Profile (within PSG).

Program Resources

Actual expenditures over the 6-year evaluation period were estimated at $93 million as shown in Table 1. The main source of funding was on-going (A-base) and provided through the Main Estimates. Spending fluctuated during the evaluation period due to activities supported through temporary project-based funding. These resources were provided by other government departments and non-GC partners, or through temporary funds (C-based) provided through Treasury Board.

Table 1: Program expenditures by component during evaluation period (2014 to 2020)

Component	2014-15 ($)	2015-16 ($)	2016-17 ($)	2017-18 ($)	2018-19 ($)	2019-20 ($)	Total ($)
PSG	7,221,451	8,058,940	7,709,163	8,137,554	8,389,363	8,420,000	47,936,471
CHIS	4,850,516	5,785,781	13,605,131	7,161,531	7,025,222	7,115,369	45,543,550
Total	12,071,967	13,844,721	21,314,294	15,299,085	15,414,585	15,535,369	93,480,021

Approximately 118 Full-Time Equivalents (FTEs) supported the Program. PSG activities were carried out by approximately 58 FTEs based in GSC facilities across the country (Dartmouth, Quebec City, Calgary, Sidney, and Vancouver). CHIS activities were carried out by approximately 60 FTEs based in Ottawa, Yellowknife, and Sidney, British Columbia (BC).

Evaluation

Objectives and Methods

Based on the considerations used to plan the Evaluation of GKCS, the following four evaluation objectives were identified:

1. Assess program contribution to i) achieving intermediate and longer-term outcomes and ii) meeting stakeholder needs;
2. Assist the Program with establishing its baseline performance data to enable future performance measurement;
3. Identify and document best practices and lessons learned to inform implementation of the 2019-24 Program activities; and
4. Identify potential Equity, Diversity and Inclusion (EDI) considerations and impacts using Gender-Based Analysis Plus (GBA+).

To meet these objectives, the evaluation examined issues of relevance, performance, EDI, and best practices and lessons learned according to the following five questions. The evaluation matrix in Appendix A presents the questions and their related indicators.

1. Is the program role appropriate and necessary within the context of changes in the program operating environment? (Relevance)
2. To what extent has the program progressed on achieving its intended outcomes? (Performance)
3. Does the program model support the achievement of program objectives? (Performance)
4. To what extent does the program consider factors around EDI factors? (GBA+ analysis)
5. Are there any best practices and lessons learned related to the design and delivery of the program? (Best practices and lessons learned)

Data was collected according to five lines of evidence as shown in Figure 2.

Figure 2: Five Lines of Evidence

The evaluation focused on the activities of geo-hazards research and monitoring of abrupt but infrequent natural hazards such as earthquakes, tsunamis, landslides, volcanoes and geomagnetic storms (space weather). Hazards resulting from climate change, wildfires and floods, or nuclear incidents are outside the focus of the evaluation. The GKCS Program Logic Model in Figure 3 reflects the activities and expected results that fall within the scope of this evaluation.

Figure 3: GKCS Logic Model (based on evaluation scope)

Immediate outcomes are largely within the control of the Program. Through the activities carried out by the PSG and CHIS, the outputs produced directly support the achievement of the immediate results. This relationship has been well-established in previous evaluations. The program performed well on achieving its immediate outcomes in previous evaluations, and other departmental reporting indicated that condition remains the same. Since this evaluation found similar results based on evidence that is well-documented elsewhere, the report presents a condensed version of evidence supporting the findings on immediate outcomes.

Intermediate outcomes fall somewhat outside the Program’s direct control, but within its sphere of influence, most obviously through direct involvement with stakeholders. In the case of GKCS, the intermediate outcomes reflect the level of uptake and use of GKCS information and alerts. The first intermediate outcome is that responses and mitigation strategies will be based on the scientific evidence. This will be assessed through evidence of examples where decisions were made based on the science, trends in stakeholder views regarding their understanding of hazards and their impacts (increasing / improving over time) and stakeholder views and documented evidence that stakeholders rely on CHIS alerts and notifications in their decision-making. A second intermediate outcome expects that emergency and disaster management stakeholders are equipped to plan for and respond to hazard events. Essentially, this means that the tools and information produced by the program to support decision-making meet stakeholders’ needs, enable effective decision-making, are adaptable to local conditions, and that mechanisms to share information and collaborate are effective and were assessed according to those criteria.

Long-term outcomes fall further along the Program’s sphere of influence, and are achieved through the intersection of a wide variety of factors. In the case of GKCS, the long-term outcome considered was the extent to which Canada’s preparedness, response and recovery from natural resources is enhanced. This reflects the work undertaken at various jurisdictions in disaster risk reduction, and measures the Program’s contribution to progress. The ultimate outcome is that Canada is resilient to natural hazards. Again, the Program was assessed based on identifiable progress where the Program reasonably contributed. For both of these impact levels, the Program’s contribution was hard to precisely define, and was gleaned from stakeholder views for the most part.

A note on terminology

The logic model outcomes reflect the terminology used in the fields of disaster risk reduction and emergency management. For the purpose of clarity, Table 2 provides definitions for the technical terms used in this report.

Table 2: Definitions used in this document

Terms	Definitions
All-hazard / Multi-Hazards Risk Approach	An approach that recognizes that the actions required to mitigate the effects of emergencies are essentially the same, irrespective of the nature of the event, thereby permitting an optimization of scarce planning, response and support resources. The intention of all-hazards generic emergency planning is to employ generic methodologies, modified as necessary by particular circumstances. All-hazards incorporates natural and man-made hazards threats including traditional emergency management events such as flooding and industrial accidents; as well as national security events such as acts of terrorism; and cyber events.
Catastrophe	A catastrophic incident is one of such extreme and remarkable severity or magnitude that a nation’s collective capability to manage all response requirements would be overwhelmed, thereby posing potential threats to national security, national economic security, and/or the public health and safety of the Nation. (Source: Federal Emergency Management Agency)
Critical infrastructure	Refers to processes, systems, facilities, technologies, networks, assets and services essential to the health, safety, security or economic well-being of Canadians and the effective functioning of government. Critical infrastructure can be stand-alone or interconnected and interdependent within and across provinces, territories and national borders.
Disaster	A serious disruption of the functioning of a community or a society causing widespread human, material, economic or environmental losses which exceed the ability of the affected community or society to cope using its own resources. A disaster is a function of the risk process. It results from the combination of hazards, conditions of vulnerability and insufficient capacity or measures to reduce the potential negative consequences of risk.
Disaster Risk Reduction	Systematic efforts to analyze and reduce the causal factors of disasters. Reducing exposure to hazards, lessening vulnerability of people and property, wise management of land and the environment, and improving preparedness and early warning for adverse events are examples of DRR.
Emergency management	The prevention and mitigation of, preparedness for, response to and recovery from emergencies.
Hazard	A potentially damaging physical event, phenomenon or human activity that may cause the loss of life or injury, property damage, social and economic disruption or environmental degradation.
Mitigation	Sustained actions taken to eliminate or reduce risks and impacts posed by hazards well before an emergency or disaster occurs; mitigation activities may be included as part of prevention. Measures may be structural (e.g. flood dikes) or non-structural (e.g. land use zoning and building codes).
Prevention	Actions taken to avoid the occurrence of negative consequences associated with a given threat; prevention activities may be included as part of mitigation.
Preparedness	A phase of emergency management consisting in making decisions and taking measures before an emergency, in order to be ready to effectively respond and recover.
Response	A phase of emergency management implemented immediately before, during or after an emergency, and consisting in activities aimed at limiting or preventing damage to life, property or the environment.
Resilience	The capacity of a system, community or society potentially exposed to hazards to adapt, by resisting or changing in order to reach and maintain an acceptable level of functioning and structure.
Vulnerability	The conditions determined by physical, social, economic and environmental factors or processes, which increase the susceptibility of a community to the impact of hazards. It is a measure of how well prepared and equipped a community is to minimize the impact of or cope with hazards.
Whole of Society approach	The whole of society approach is one that seeks to leverage the existing knowledge, experience and capabilities within emergency management partners in order to strengthen the resilience of all.
Source: Compiled from the Federal Policy on Emergency Management and the UNESDOC Glossary of Basic Terminology on Disaster Risk Reduction, except as otherwise noted.

Evaluation limitations

The evaluation team acknowledged the unique circumstances caused by the COVID-19 pandemic. While outside the immediate scope of the evaluation, the team documented instances of Program adaptation to COVID-19 to support NRCan efforts to learn from its experience operating during a pandemic.

The evaluation used a mixed-methods approach to mitigate limitations associated with individual methods. The triangulation of evidence across multiple methods of data collection is an evaluation technique that helps to reduce the impact of (unconscious) bias in order to identify reliable findings and draw valid conclusions. However, the following limitations should be considered when reviewing the evaluation findings.

• This evaluation was conducted completely during a time of active public emergency protocols due to the COVID-19 pandemic. As such, public safety was at an unusually high level of visibility. It is possible that the high level of alert impacted the perspective of key informants and survey respondents. To mitigate, the evaluation team triangulated interview and survey evidence with sources documenting stakeholder discussions around the topic of DRR and EM (e.g. conference proceedings, witnesses to House of Commons Standing Committees) prior to the emergency protocols being emplaced.
• Floods and wildfires fall outside the scope of GKCS program activities. As such, the evaluation findings are not intended to cover those types of hazards. However, the evaluation team recognized that floods and wildfires are prominent natural hazards and come first to mind for many people, including stakeholders in the Program. There was a risk that survey respondents and interviewees would provide information more pertinent to floods and wildfires than the hazards within scope. To mitigate that risk, informants were reminded specifically to exclude consideration of floods and wildfires in forming their comments in order to remain within the scope of the evaluation during interviews. The survey questions were also worded to remind respondents of the scope.
• The online survey was emailed to approximately 200 program staff and stakeholders identified through the Program contact lists, with a response rate of 39%, split approximately evenly between Federal Government (36%), Academics (37%), and Other external groups (22%, included industry and provincial, territorial and municipal governments). While the response rate was high for an online survey, the results reflect only the perspective of those who have had direct interaction with the Program and should not be construed to reflect the Canadian population at large. Moreover, the small numbers of responses to some of the questions did not allow for meaningful comparison across groups.

What We Found

Relevance

Question 1: Is the program role appropriate and necessary within the context of changes in the program operating environment?

Indicators:

• Extent to which the activities of the Program align and respond to i) the needs of the intended beneficiaries and stakeholders; and ii) the guidance provided in the Sendai Framework and Canada’s EM Strategy.
• Extent to which the role played by NRCan is complementary to other federal organizations and viewed as appropriate by stakeholders.
• Evidence that the role of NRCan is clear and well understood within NRCan, and among external stakeholders.

Program role is necessary and consistent with the Emergency Management Act (EMA)

All lines of evidence pointed to the continued need for GKCS programming. The document review and the interviews showed that the Program served as the subject matter expert on earthquakes, volcanoes, landslides, tsunamis, and space weather, and had a unique role in monitoring and providing alerts and data on seismicity and space weather consistent with the EMA. Interviewees consistently acknowledged that without the GKCS role in conducting research and providing information, there would be a piecemeal approach to emergency management and disaster risk reduction across the country. The impact would be reduced opportunities for standardization and consistency in approaches that facilitated the sharing and application of knowledge. In addition, NRCan provided a larger-scale context, particularly across jurisdictional boundaries necessary to fully understand local geo-hazards and assess related risks. Survey respondents largely indicated that if NRCan stopped producing geo-hazard information products, their organizations would be significantly impacted. Notably, 63% of respondents indicated that their ability to assess risk would be impacted “to a great extent” if the NRCan information on geohazards was unavailable.

NRCan plays a support role as the subject matter expert to government on geohazards, and fulfills specific commitments under the EMA and other international agreements. Under the EMA, PS is the lead federal department with respect to emergency management and disaster risk reduction, and responsible for coordinating policy response and communications with respect to hazards. Additionally, the Strategic Emergency Management Plan outlines specific responsibilities for NRCan, namely the provision of scientific, technical, and/or policy leadership for geological hazards, space weather, and seismic alerts. In practice, stakeholders such as PS and provincial governments turned to NRCan as the subject matter expert for geo-hazards, and NRCan supported these stakeholders by providing the requisite scientific information and alerts to ensure that they were adequately informed about hazards and hazard risk.

The Program’s monitoring capacity enabled Canada to meet international commitments. For example, as outlined on CHIS’ official website, “CHIS discharges NRCan responsibilities for ongoing monitoring under the Comprehensive Nuclear Test Ban Treaty (CTBT).” The 13 stations that Canada shared with the International Monitoring System of the CTBT were included in the 2014-19 NEMI upgrade project. Canada also contributed CHIS seismic data to the National Tsunami Warning Centre in Alaska as a member of the Intergovernmental Oceanographic Commission of UNESCO. Space weather information provided by Canada through GKCS contributed to the International Aviation’s Organization (IAO) ability to act on its protocols.

The Program activities aligned with the guidance provided in the Sendai Framework for Disaster Risk Reduction and Canada’s Emergency Management Strategy

During the evaluation period, the external Program operating environment changed with the signing of the SF in 2015. This document focused on the importance of reducing the impact of disasters through the proactive mitigation and prevention of disasters themselves, as opposed to simply responding post-event. There were four priorities under the SF:

1. Understanding disaster risk;
2. Strengthening disaster risk governance;
3. Investing in disaster risk reduction for resilience; and
4. Enhancing disaster preparedness for effective response and to “Build Back Better” in recovery, rehabilitation, and reconstruction.

In 2017, the Government of Canada, provinces and territories adopted the EMS. The EMS identified five priority areas of activity, and which was based on the four priorities of the SF. It outlined Canada’s whole-of-society approach to disaster risk reduction and emergency management. Document review and interview evidence revealed that the Program’s activities supported the priorities of the SF and EMS. The Program helped stakeholders understand disaster risk by providing scientific information and alerts through PSG and CHIS. The Program has invested resources in disaster risk reduction for resilience by contributing to the development of the National Building Code of Canada (NBCC) and investing in an early earthquake warning system. The Program contributed to disaster preparedness for effective response and to “Build Back Better” in recovery, rehabilitation, and reconstruction by considering GBA+ factors in the vulnerability datasets and models it produces. Table 3 provides examples of Program alignment to the Sendai Priorities and corresponding EMS priority areas.

Table 3: Program Activities align to Sendai Priorities and Corresponding EMS Priority Areas

Sendai Priorities	Corresponding EMS Priority Areas	Program Activities
Understanding disaster risk	Improve understanding of disaster risk in all areas of society	All projects of the Public Safety Geoscience Program and Canadian Hazards Information Service
Strengthening disaster risk governance	Enhance whole-of-society collaboration and governance to strengthen resilience	PSG negotiated Canada’s Global Earthquake Model Foundation (GEMF) membership with partners. A steering committee was struck, and workshops on GEMF’s OpenQuake software were conducted. Risk Project staff contributed to organizing a key panel discussion at Fifth Regional Platform for Disaster Risk Reduction in the Americas in Montréal. Program was reporting performance information reflecting the Sendai Priorities, and was involved in the development of appropriate indicators for Canada’s reporting.
Investing in disaster risk reduction for resilience	Enhance disaster response capacity and coordination and foster the development of new capabilities.	Contributions to the National Building Code of Canada. Investment in early earthquake warning system.
Enhancing disaster preparedness for effective response and to “Build Back Better” in recovery, rehabilitation and reconstruction	Increase focus on whole-of-society disaster prevention and mitigation activities. Strengthen recovery efforts by building back better to minimize the impacts of future disasters.	Investment in early earthquake warning system. Consideration in GBA+ factors of vulnerability.

The Program largely met stakeholders’ needs, but new needs are emerging that impact Program role

Overall, stakeholders perceived the GKCS Program as largely meeting their needs. All lines of evidence indicated that research and information products aligned to topics relevant to stakeholders in emergency management and disaster risk reduction. Needs perceived to be unmet were more likely to relate to perceived deficiencies in information and existing products. Addressing many of the unmet needs such as availability of local level geo-hazard data however, were outside the sole responsibility of the Program, and would require coordination and collaboration among various jurisdictional entities to address. In interviews and the survey, stakeholders suggested that removing silos across hazards would help to resolve these challenges.

The EMS, which outlined Canada’s approach to disaster risk reduction and emergency management, expressed the shift in the Sendai Framework as follows:

Traditionally, EM has been primarily concerned with preparedness and response activities, but the current risk environment requires a shift in focus toward proactive prevention/mitigation efforts and forward-looking recovery measures. Greater investment in mitigation can help prevent disasters or significantly reduce the social, economic, cultural heritage and environmental costs when events do occur. Forward looking recovery measures allow communities not only to recover from disasters, but also to build back better in order to reduce vulnerability to eventual hazards.

This evolution in EM is consistent with the international concept of Disaster Risk Reduction (DRR), which is defined by the United Nations (UN) as “systematic efforts to analyze and reduce the causal factors of disasters. Reducing exposure to hazards, lessening vulnerability of people and property, wise management of land and the environment, and improving preparedness and early warning for adverse events are examples of DRR”. It is important to consider that the impacts of disasters are not uniform across society, and that different variables can intersect and contribute to the level of risk facing vulnerable populations (e.g., gender, age, disability, socioeconomic conditions).⁶

The SF focuses on reducing the impact of disasters through mitigation / prevention of the disaster itself instead of loss reduction from an event. The manner of implementation is through sophisticated risk assessments, whole-of-society involvement, and a multi-hazard approach⁷. Among the survey respondents who said they were familiar with the Sendai Framework for Disaster Risk Reduction (n=20), five said they foresaw their needs for geohazards information changing over the next 10 years as a direct result of the SF. When asked to describe the changes they foresaw, these respondents mentioned the following: geohazards information that was accessible to non-experts; more multi-hazard datasets; and new informational products or enhanced existing products to support decision-making in reducing risk.

According to the document and literature review, the approach to disaster risk reduction found in the SF fully acknowledged that disasters were a product, not only of the hazard event itself, but also the manner in which the natural environment intersected with the social, cultural and political environment. This risk-based approach involved a high level of multidisciplinary and multi-jurisdictional collaboration in order to be successful⁸. Due to the change in approach, there was an expectation that disaster risk management systems would rely on science more heavily.⁹

Gap in knowledge transfer identified

As noted earlier, the EMS signalled a significant shift in focus towards geo-hazards risk assessment and whole-of-society exercise. Inherent in the approach was a need for information more readily consumed by a lay user. Thus, the pool of relevant stakeholders had increased beyond the scientific and technical users targeted by the Program. Most of those interviewed identified a gap in between the accessibility of the available information to support understanding by non-technical users who integrated the scientific information for effective mitigation and response planning. Meeting those needs created new demands for subject matter expertise in knowledge translation. This raises implications for the Program as the primary subject matter expert for geo-hazards within the Government of Canada. However, evidence was mixed as to what role the Program should play in meeting the new demand.

All of the interviewees valued the expertise provided through the Program, and 86% of survey respondents cited expert knowledge as a key source of their information on geo-hazards. Experts particularly saw the expertise as a significant asset for Canada. However, concerns were raised about the capacity of the Program to meet the increased need.

Whole-of-society approach to disaster risk reduction and emergency management drives expanded role for Program on the science value chain and unmet needs in knowledge translation

NRCan scientists were often called on to provide guidance to local stakeholders on an ad-hoc basis to support the uptake and use of the Program’s scientific information products and tools. This strained the capacity of the Program. On the one hand, the engagement was necessary in a whole-of-society approach. On the other, the Program’s ad-hoc engagement activities had the potential to detract from NRCan scientists’ capacity to conduct fundamental research and scientific activities on geohazards, which no other group was doing. External interviewees indicated that they understood the capacity issues affecting the Program, but that the Program nevertheless should play a larger role in bridging the information gap between NRCan and local communities, and with non-technical users. Internal interviewees noted that the Program lacked the capacity to undertake the long-term relationship building required for consistent engagement at the local level where adverse events largely occurred. While engagement activities were underway within specific projects, they were supported by resources from other established programs.

Interviews revealed that experts as a group identified the importance of expanding the Program’s role as an integral part of the natural hazards science value chain. They expressed that in addition to its current work in assessing hazard (identify, understand, assess, and exposure), the Program could also increase its efforts in contributing to vulnerability and risk assessments.

Climate change has implications for the geohazards risk assessments and needs integration to support a strong science foundation

All lines of evidence pointed to the need to integrate climate change information when identifying hazards and assessing risk levels. Melting permafrost, coastal erosion, and precipitation had potential impact on hazard locations, particularly in relation to landslides and tsunamis. In addition, static risk assessments might not have accounted for changes in the natural environment. For the Program, this meant that identifying potential hazard locations and assessing risk should be an ongoing exercise to provide a solid science foundation to support public safety and resilience.

The need for better integration between disaster risk reduction and climate change research was discussed at the Disaster Risk Reduction Roundtable¹⁰ in 2017. During its strategic review exercise, PSG identified closer synergies with the Climate Change Geoscience program (CCGP) as an opportunity, given their shared mandate under the shared goal of ‘Keeping Canada Safe’. In planning the 2019-24 cycle, PSG added a review of changing climate conditions on landslides as a key science question to address in the next iteration of their Landslides and Marine Geo-hazards project. Interviewees and survey respondents also noted that climate change impacts required better integration into the science foundation for geo-hazard identification and risk assessment.

Effectiveness – Progress on outcomes

Question 2: To what extent has the program contributed to achieving intended outcomes?

Indicators:

• Evidence that outputs are produced and disseminated as planned
• Evidence of Program contribution to achieving intended outcomes

Program outputs were produced as planned

Over the course of the evaluation period, research conducted through the GKCS Program added to the scientific knowledge available to stakeholders. To support the achievement of results, the program produced a wide variety of outputs that were tracked by project and documented for reporting purposes. These knowledge products were readily available through the scientific literature in peer-reviewed journals, or on open source electronic spaces like GEOSCAN. A review of the program documents shows that outputs were generally delivered as planned. Figure 4 presents a conservative estimate of the number of program information outputs produced between 2014 and 2020. The evaluation also sought information on how stakeholders used and became aware of the program outputs through a survey of stakeholders.

Figure 4: GKCS Program Outputs

Stakeholder use and awareness of outputs

Through the survey, respondents were asked to select the type(s) of information they used from a list of categories. The types of information most used by respondents were scientific geohazards information (67%), decision-making support tools and product (39%), space weather forecasts (29%), and Seismic alerts (27%).

The evaluation was also interested in how stakeholders became aware of the outputs used. Based on the survey conducted, among the stakeholders who used NRCan’s information products (n=65), most (85%) became aware of these products via professional contacts. Other main sources of awareness were the NRCan website (52%), published academic and technical journals (52%), and academic conference or symposium (48%).

Immediate Outcomes

Immediate Outcome: Authoritative science foundation on geo-hazards is available to a large extent

Producing an authoritative body of scientific knowledge is an immediate first step in achieving subsequent outcomes of the Program. The scientific research program of GKCS was designed to fill gaps in understanding earthquakes, landslides and marine geo-hazards, volcanoes, tsunamis, and space weather. The goal was to identify and characterize geo-hazards and related risks to provide a sound basis for risk assessment and decision-making to support uptake of the information in later stage outcomes. Products included a variety of maps that identify geo-hazards, like the Canada’s first volcanic ash map published in 2015 (Figure 5).

Figure 5: Volcanic Ash Map

Among those who reported using scientific geo-hazards information in the survey (67%), the most used sources of Program geoscience information were expert knowledge (86%), followed by scientific reports open files (78%), research datasets (76%), maps (74%) and journal articles (66%). Stakeholder feedback on Program information was generally positive. Survey results showed that overall, 97% of respondent who used NRCan information products said they were satisfied with the information, including 72% who said they were "very satisfied".

When survey respondents were asked how information provided by NRCan could be improved, 40% pointed to improving information sharing and access (i.e., sharing data in a timely manner or making data available for analysis or use), 34% suggested expanding or improving the information products offered, and 26% mentioned improving the website/interface. Notably, 14% of respondents simply stated they were satisfied with the information provided through NRCan and/or that NRCan does a good job in providing information. Interviewees made similar suggestions. Some interviewees pointed out that gaps remained in understanding geo-hazards in Canada (e.g., northern Canada, volcanoes), it was relatively early in the risk assessment work, and there was a need for improved access to updated data and online datasets. In addition, they noted that the need for more scalable data so that they could better assess geo-hazards at the local level.

All lines of evidence indicated that stakeholders considered Program information authoritative. Two-thirds of survey respondents indicated that information products were “authoritative” or “very authoritative.” Respondents particularly noted the authoritativeness of the expert knowledge and the scientific journal articles published with GKCS research findings. Interview findings were similar. Some external stakeholders noted during interviews that part of the value of collaborating with NRCan was direct access to the expertise of the scientists. Most of the external interviewees, including all of the experts, mentioned that the expertise housed under the Program was a vital asset for Canada in emergency management and disaster risk reduction.

Documentary and interview evidence indicated that external stakeholders sought out GKCS Program scientists for help in developing and/or validating their own approaches to disaster risk reduction. For example, a member of the Risk Assessment project team was asked to review the new social vulnerability component of the Global Earthquake Model. Space weather scientists from the Program were involved in developing a manual to guide implementation of new protocols for the International Civil Aviation Organization (ICAO). Manitoba Hydro and CHIS are involved in long-standing collaboration with respect to impacts of geomagnetic storms on hydro transmission lines.

In northern Canada, there is lack of geoscience data in general, and Indigenous and Northern Affairs Canada (INAC) provided funding to help increase available data and understanding of marine geohazards. Interview and documentary evidence suggested that a low density of seismic monitoring stations in northern Canada limited available data about seismic activity for that region. Moreover, melting permafrost and coastal erosion were giving rise to concerns about landslides and tsunamis, where previously the risk was considered much lower. Some interviewees pointed out that understanding of marine geo-hazards and volcanoes was low, although project work for the 2019-24 iteration of the PSG was specifically designed to address some of those areas. For example, the work plan would focus on identifying volcano hazards in southwest BC as a precursor to a risk assessment of population and infrastructure.

In its planning, the Program also acknowledged the need to address climate change implications on risk, and made plans to research the interactions between climate change and geo-hazard risk. The Program planned on reviewing the impacts of changing climate conditions such as heavy precipitation and drought on landslides, and conducting research to develop guidelines on factors that increase susceptibility to landslides.

Immediate Outcome: Decision-making support tools, models, and approaches are available to stakeholders in emergency management and disaster risk reduction to a large extent

Building on work from the previous iteration of the Program (2009-2014), a variety of decision-making support tools, models, and approaches became available to stakeholders in emergency management and disaster risk reduction over the period evaluated (2014-2020). Examples included the following:

• A building-level assessment of earthquake risk and risk reduction potential completed by the Program for the City of Vancouver supported initial stages of a seismic retrofit program. BC planned to use outputs of this work to inform provincial seismic retrofit policy.
• In 2016-17, the activity Prompt Evaluation of Seismic Risk (ER2 tool) was completed as part of the Earthquakes project. It was funded by the Canadian Safety and Security Program (CSSP). Rapid Risk Evaluator (ER2) is a web-based application that puts risk assessment tools directly in the hands of non-technical end users.
• A Risk-based Land-use Guide was published by the Program. Public Safety Canada requested that the Program adapt it for use in all of Canada.
• A Total Electron Count (TEC) map service was launched in 2016-17. TEC maps help users to correct for Global Positioning System position error due to electromagnetism. NRCan produces maps containing near real-time information at 15 minute intervals and daily maps at 2 hour intervals.

Consistent with the Program plan, tools, models and approaches were disseminated to targeted stakeholders. While the dissemination effectively reached targeted stakeholders, expert opinion from the interviews coalesced around the need to expand targeted stakeholder groups in order to advance the uptake of the available decision-making support information and adopt a whole of society approach.

The Program had started to integrate the whole of society approach to its dissemination activities as suggested by the SF. Through collaborative networks in British Columbia, project scientists engaged with local jurisdictions (e.g., Understanding Risk British Columbia). In the landslides and marine geohazards project work, the Program engaged west coast Indigenous communities in relation to tsunamis. Presentations were made in Tuktoyaktuk and local communities on Baffin Island as part of the same project. Through the risk assessment project, the Program was heavily involved in organizing an online forum through Understanding Risk British Columbia in 2020.

The Program has also adopted other practices that aligned to the SF and whole of society approach to disaster risk reduction in developing its risk reduction tools and models to support decision-making.

• In 2015, the GSC commissioned a review of risk assessment tools being supported by NRCan to assess their ability to provide quantitative risk assessment information. The assessment was a first step in ensuring alignment with the SF and the National-Scale Geo-hazard Risk project. One of the key criteria in assessing the tools for risk assessment was the ability to integrate a wide range of data to identify vulnerabilities.
• In 2017, Canada became a member of the Global Earthquake Model Foundation (GEMF). Involvement in GEMF enabled the expertise of Canada to be shared with the world. Through GEMF, Canada gained direct access to the information produced from around the world as earthquakes happen. This enabled Canada to integrate information on best practices in earthquake risk assessment and disaster risk reduction and emergency management as they became available.

In each case identified, the dissemination was supported by collaboration, and often included resources from other sources. When active dissemination occurred beyond the traditional scientific and technical fields where existing relationships and connections are strong, plans for such activity was built into the agreement with the project funder. For example, CSSP program funds allocated resources for sharing the results of the marine geo-hazards research.

Interview evidence suggested that dissemination of the tools and other decision-making support information was hampered by a lack of planning around dissemination. A few interviewees connected to the landslides project noted that they participated in the development of guidelines for landslides. However, the guidelines “went nowhere” due to an insufficiency of funds to prepare the documents for publication. Some internal and a couple of external interviewees identified missed opportunities for dissemination of information and tools by being unaware of other NRCan-sponsored research and engagement activities in related fields.

Immediate Outcome: Alerts and notifications are available in real-time to a large extent

Timeliness of information was a critical feature of alerts, forecasts, and notifications. All lines of evidence corroborated the view that seismic alerts and space weather forecasts were received in a timely manner. Under the Departmental Results Framework, a key indicator for NRCan was the percentage of times that notification of seismic activity was made within prescribed time parameters. Annual reports published to the NRCan website indicated that prescribed targets were met for each year. As planned, updates performed to the monitoring equipment resulted in fewer latent delays due to equipment lags in transmission times in the NEMI project that received funding for equipment upgrades. Of those who reported using seismic alerts and space weather forecasts in the survey, all (100%) of them rated timeliness of these products as at least “somewhat timely”.

Alert and notification systems were highly coordinated among those who provided alerts and notifications within Canada and with international partners, as revealed in the following examples:

• Canada’s membership in the Intergovernmental Oceanographic Commission (IOC) of UNESCO enabled its tsunami warning capability. Through agreement under the IOC, data collected by CHIS was transmitted to the National Tsunami Warning Centre operated by the United States which then issued tsunami warnings for Canadian waters.
• A high level of cooperation between the United States Geological Survey (USGS) and the Program enabled cross-border earthquake response activities and data sharing.

Intermediate Outcomes

Intermediate Outcome: 2.4. Responses and mitigation strategies to hazard events integrate scientific knowledge and alert information to a moderate extent

Generally, the evidence demonstrated that science and alert information produced by the Program was integrated into geo-hazard response and mitigation strategies as the result of direct collaboration. Uptake and application of the scientific knowledge and alert information was limited to larger municipalities, infrastructure installations, and provincial and government agencies. It is worth noting that all of them had a high level of interaction with the Program. The interaction was either through whole of government approaches such as the review of critical infrastructure in the event of an emergency, or through research collaborations established through academia or funded through external grants.

Evidence from the survey and interviews indicated that the information supported effective decision-making to a lesser extent. A slight majority (56%) reported that the information supported effective decision-making to a “moderate” or “great extent.” On the other hand, almost half reported that NRCan information contributed to effective decision-making to a “small extent” (25%) or “not at all” (19%). Likewise, interview evidence was mixed. Stakeholders who had direct collaborative engagement with the Program rated it highly based on their experience. However, others noted that impact on decision-making was limited. While overall they rated the available tools positively, they were more likely to rate support for effective decision-making as moderate during the interviews.

The document review and interviews recorded numerous instances where decisions were made, action taken, and policies and regulations developed based on information provided by the Program. Among them were the following:

• Building Code of Canada: CHIS produced the seismic model, and PSG conducted research on seismic impacts. This information was built into the National Building Code of Canada (NBCC). This work had been ongoing since 1953 and every five years the Building Code of Canada integrated new scientific understanding on seismic risk. For example, the 2015 NBCC incorporated the national seismic risk model and provided information for further updating in 2020.
• The Program’s space weather scientists contributed to a manual for implementing new protocols established by the International Civil Aviation Organization (ICAO). Canada was also a part of the collaborative engaged by the ICAO to provide space weather data. As noted by one survey respondent: “Technical leadership on providing practical real-time information for the power grid has been foundational for much of the support that we can provide today. We also applaud the recently upgraded data services from the magnetometer network which provides some of the most timely space weather information in the world and which we use to support our space weather mission in the U.S.”
• Space weather scientists collaborated with Public Safety and other government departments to assess space weather risk on critical infrastructure. The work resulted in “A Guideline for Enhancing Canada’s Critical Infrastructure Resilience to a Catastrophic Space Weather Event.”
• Collaborative work with the District of North Vancouver resulted in a risk-based land use planning and earthquake emergency preparedness action plan. 
  • A Risk Map Atlas was produced that provided outputs of the pilot study to understand earthquake risk and loss reduction in a mid-sized Canadian urban community.
  • The study used the modelling software Hazus-MH Canada to estimate the damages caused by earthquakes to buildings, roads, pipelines, and people.
  • The initial analysis was then repeated to show the potential changes in outcome if buildings in the community were retrofitted to modern building codes.
  • This comparison analysis helped to motivate the community to develop an earthquake ready action plan, and could be applied to other stakeholders and locations.¹¹
• Guidelines and protocols for landslide monitoring methods were developed in the Thompson River Valley research activity that was a collaborative effort on the part of the Program, academia, the railways, and Transport Canada. On completion, the recommendations were integrated into railway operations. Also, NRCan scientists were applying the methodology developed to determine landslide risk levels in other locations such as the Assiniboine and Qu’Appelle River valleys.

Nine in 10 survey respondents agreed that decisions about disaster risk reduction and emergency planning were made using scientific evidence and knowledge, including 52% who strongly agreed. Survey respondents also held the view that NRCan information played a significant role in their own increased understanding of geo-hazards, as well as that of those responsible for taking decisions. Many stakeholders and partners, moreover, said there would be an impact on their organization if NRCan geo-hazards information was unavailable or not produced.

As shown in Figure 6, survey respondents overwhelmingly acknowledged the contribution of NRCan information to their improved understanding of geo-hazards over the course of their careers¹². When asked the extent to which NRCan information contributed to their improved understanding, 98% indicated that it played at least some part, with 86% saying to a great (61%) or moderate (25%) extent, regardless of the length of their career.

Figure 6: Extent to which NRCan information contributed to respondents’ understanding of geo-hazards

In addition, survey respondents identified a variety of types of decision-makers, including (in order of frequency), operations staff (those with technical knowledge), emergency planners, responders and crisis centres, all levels of government, senior government officials, those in the private sector, and politicians as being responsible for emergency management and disaster risk reduction. Seventy-seven percent (77%) of survey respondents believed that, from their perspective, these decision-makers also gained an increased understanding of geo-hazards (Figure 7). The vast majority of those (83%) said that the information provided by NRCan contributed to this increased understanding at least “to a moderate extent”, including 45% who said “to a great extent”.

Figure 7: Contribution of NRCan information to increased understanding among decision-makers

Stakeholders relied on the alerts and notifications provided through the Program. All lines of evidence presented well-documented examples that CHIS seismic monitoring data and alerts, as well as the associated analysis, that were integrated into the plans of infrastructure operators, building and construction operations. Examples included the following:

• The ministère des Transports du Québec used seismic alerts provided through CHIS data to launch inspections of its roadworks and infrastructure. Relevant interview information confirmed that over a specified threshold, the seismic level reported triggered automatic inspections of the infrastructure and roads within its jurisdiction.
• In assessing potential sites for a deep geological storage facility, the Nuclear Waster Management Organization (NWMO) had an ongoing collaboration with CHIS. Through this work, specific sites were tested for seismicity to identify their suitability. Interview evidence indicated that the organization relied heavily on the collaborative work arrangement in assessing various sites.
• On request, CHIS conducted seismic assessments of various sites for external stakeholders. Program documents contained numerous site assessments, mostly for use in the building industry.

Stakeholders also included other sources of alerts and notifications in their planning. In relation to seismic alerts, some stakeholders with responsibility for immediate response to hazards indicated that they referred to information from the United States Geological Survey (USGS) for primary alert because it was provided faster. The more detailed information received from NRCan sources was used to “ground truth” the information to determine the most appropriate response at the local level. However, inconsistencies in risk levels as assessed by separate agencies led to planners adopting the lower risk levels.

Intermediate Outcome: 2.5. Emergency and disaster management stakeholders are equipped to plan for and respond to hazard events to a moderate extent

The extent to which emergency and disaster management stakeholders were equipped to plan for, and respond to, hazard events was closely associated with location and hazard type, as well as involvement in collaborative networks focused on geo-hazards and disaster risk reduction. In Canada, responsibility for disaster events and emergency management lays primarily at the local level. For stakeholders to be equipped to plan for, and respond to hazard events, they need information to identify local risk levels and options to mitigate that risk before an event, as well as information for situational awareness during a hazard event. The Program provided key information to support such activity.

There was some indication that decision-making tools developed by the Program were being adapted to regional and local needs. In Tuktoyaktuk, a tool used to assess earthquake risk was being adapted to assess landslide risk. Methods and information developed through the Thompson River Valley landslide research activity were being applied to other locations in the Assiniboine and Qu’Appelle River valleys. The ER2 assessment tool was being adapted for local use across the country. During interviews however, stakeholders consistently pointed out that that decision-making support tools were at various stages of development depending on hazard type, and planning was uneven across the country. For example, a relatively large number of tools was available for earthquake risk assessment but far fewer for tsunami risk. Planning and response measures were more advanced in British Columbia where the provincial government was highly engaged in supporting geo-hazards emergency planning and disaster mitigation efforts.

As reported through the interviews and survey, stakeholders held the view that the information and tools provided through the Program largely met their needs. According to the survey data (see Figure 8), the majority of respondents (77%) said that their needs were “mostly” or “entirely” met. Survey respondents particularly identified that the Hazard maps, space weather forecasts, and earthquake catalogues as key tools.

Figure 8: Extent to which geo-hazards information provided through NRCan met needs

Interviewees for the most part distinguished between response and planning for geo-hazard events. Among them, almost half noted that emergency management response was more advanced than prevention and risk reduction planning. Some of those suggested that professional cultures played a role. They noted that emergency management was a more established field that focused on operational response and need for real-time information during an event. Disaster risk reduction on the other hand, was a relatively new area of expertise and was focused on planning for possible events. As a result, it was generally challenging to motivate efforts and investment to reduce risk and expand existing relationships. In their view, it would take time for disaster risk reduction to become a predominant thought process in the emergency management field. Others noted that disaster risk reduction went beyond emergency management response. Local approaches therefore needed to better integrate planners and local and regional municipality managers in preparing for disasters by improving land use and zoning in hazard-prone areas.

Central to the whole of society approach underlying the Sendai Framework was a high level of information sharing and collaboration. Evidence was weak however that existing mechanisms to share information and collaborate such as Canada’s Disaster Risk Reduction Platform were sufficiently effective in supporting stakeholders to be adequately equipped to plan for, and respond to hazard events.

Survey evidence in open text questioning showed that among the unmet needs identified was a need for greater collaboration and cooperation across fields of expertise, and information such as guidelines to support integration of geoscience data into usable risk assessments. One respondent stated the need as a “bridge between experts at NRCan and users of information to understand how data is used and the downstream impact to help in decisions around balancing of risk with cost.”

In the interviews, most stakeholders pointed to collaboration with the Program as a key component in being equipped to plan for and respond to hazards. However, they also noted that there were few opportunities outside of the well-established scientific and technical fora in geo-hazards and emergency management for non-expert learning and networking – particularly for those new to the field of emergency management and disaster risk reduction. In addition, a couple of external stakeholders noted that Canada still largely operated in silos of expertise when it came to geo-hazards and disaster risk reduction and was less advanced in national collaboration relative to other countries they had worked in.

Overall, very few interviewees were familiar with the SF and the EMS. However, interviewees familiar with the SF and the EMS reported that the Program was applying the guidance of the SF, and that guidance from the SF and EMS helped them to focus their priorities and activities. Documented examples noted earlier suggested that in spite of challenges in capacity, the Program was implementing the whole of society approach in its activities. The Program was also working to improve its warning capacity. For example, upgrades to the seismic monitoring system between 2014 and 2019 reduced latent delays in alert notifications, and the 2019 federal budget provided funding for CHIS to implement an earthquake early warning (EEW) system. 

Long-term / Ultimate Outcomes

Long-term Outcome: Canada’s preparedness, response and recovery from natural disasters is enhanced to a moderate extent

Stakeholders expressed a moderate level of confidence that they were better prepared to act during hazard events. The survey results in Figure 9 show that 81% of stakeholders were at least moderately confident that they (their organizations) were better prepared to act during hazard events than at the beginning of their career, and 71% attributed their confidence level to NRCan to a moderate or great extent. Interview evidence found a similar level of confidence, but also noted that there was room for further improvement. They cited the need for better integration and support for sharing information and lessons learned in order to be fully prepared; resources to support a collaborative risk reduction model overall; and implicated NRCan in addressing these issues because of its expertise. Experts interviewed largely shared this view, and believed that the Program was doing what needed to be done.

Figure 9: Confidence in preparedness to act with attribution to NRCan information

Evidence of planning at the strategic level to reduce gaps and resource them overall was limited. The Disaster Risk Reduction Roundtable was positioned as a key venue for identifying information gaps at the strategic level. For example, through this venue, discussion at the 2017 DRR Roundtable on integration of climate change helped to articulate the overlap between disaster risk reduction and climate change adaptation and mitigation. However, there was little evidence that the Roundtable served to develop work plans for resourcing or acting on the identified gaps. Interviewees suggested that the challenge of multiple priorities reduced its effectiveness in developing action plans and identifying synergies among stakeholders focused on geo-hazards. External interviewees also noted that the existing platforms to exchange information and develop action plans for disaster risk reduction were inadequate. They suggested that NRCan was well equipped to fill the gap due to its subject matter expertise, and its experience with the Canada’s Climate Change Adaptation Platform.

Stakeholders believe that the GKCS Program contributed to Canada’s increased resilience to geo-hazards

Among stakeholders, there was a general belief that Canada was progressively more resilient to natural hazards, particularly over the last decade. As shown in Figure 10, survey respondents considered NRCan information and knowledge as a factor impacting their assessment of Canada’s resilience. Most of the survey respondents (71%) believed that Canada was more resilient to natural hazards than it was at the start of their career. Of those, 29% saw NRCan information and knowledge as a factor in their assessment to a great extent, while 55% saw it as a factor to a moderate extent.

Figure 10: Assessment of resilience and attribution to NRCan information and knowledge

Interviewees also believed that resilience in Canada had improved over the last decade. Some mentioned that improved communications and information-sharing across jurisdictions enabled communities to rebound after experiencing recent hazardous events such as wildfires and floods, and it reasoned this experience would be applicable to other events. Other interviewees placed caveats on the level of resilience attained. For example, the experts interviewed cautioned that while there were good indications of progress as in low loss of life statistics, Canada had not been tested with a true catastrophe. Therefore, the progress on resilience could not be proven.

In addition, even though the Insurance Bureau of Canada estimated that eight dollars in losses was avoided for every dollar invested in disaster risk reduction, many of those interviewed cautioned that the measurement of resilience in monetary terms did not factor in the human costs. The impact of trauma in a community due to a disaster for example, was cited as one area that was challenging to measure. Appropriate indicators to measure resilience have been identified by the DRR community as a need for measuring performance under the Sendai Framework.

Program Design and Delivery

Question 3: Does the program model support the achievement of program objectives?

Indicators

• Evidence that program design (e.g., decision-making structures, resource allocation, design and delivery mechanisms, collaborative agreements and working arrangements) supports program objectives and reflects guidance under the Sendai Framework and EM Strategy
• Factors influencing program performance and assessment of their impact

The Program design supported the immediate Program outcomes to a large extent

The Program was well-designed to support the immediate Program outcomes. Projects undertaken within the evaluation period were well-executed. For example, the case study evidence demonstrated that in all three cases – Thompson Valley landslide, space weather, and NEMI – projects were successfully scoped and completed as planned with budgets aligned to actual spending.

All sources indicated that the research model adopted by the Program was highly effective. It was characterized by collaboration and supported by a high level of scientific and technical expertise. Through formal collaborations, the research in PSG attracted funding from other federal government departments and external organizations. For example, in 2018-19, each dollar spent on project activities from NRCan departmental resources (A-base excluding salary) within PSG leveraged about seven dollars in outside funding. The ability to attract funds confirmed the alignment to stakeholder need and strength of the research. However, it also raised the risk that science and research activities fundamental to evidence-based decision-making could become reliant on temporary funds. Some interviewees noted that this limited long-term research commitments and also reduced opportunities to develop the long-term relationships critical to successful uptake of information at the local and regional levels.

Collaborations were supported by formal agreements such as memoranda of understanding, interdepartmental agreements and letters of intent. These arrangements supported research and monitoring by allowing for data sharing, infrastructure support and joint investigation. In one case where the relationship was multi-faceted, NRCan created a single umbrella agreement that enabled multiple projects to be initiated. Other agreements for longer-term research were extended based on the mutual benefits of the agreement. For example, in 2015 an existing 10-year Letter of Agreement between the Department of Fisheries and Oceans Canada, NRCan, and UNAVCO Inc. was extended to 2025. The agreement enabled the installation and monitoring of seismic equipment covering the Pacific north-west.

External interviewees involved with collaborative activities said that they were able to achieve more in a shorter time period, and attributed project success to the involvement of Program staff as previously noted. A couple also mentioned that the use of formal agreements provided clarity on responsibilities and roles. For example, an agreement with the ministère des Transports du Québec clarified the role of the Program and the Province of Québec with respect to landslides and interaction with municipalities. Canada’s interdisciplinary membership in GEMF was cited in several interviews as a key collaboration that enabled the Program share expertise within and outside Canada. One of the experts interviewed specifically mentioned that the unique membership approach adopted by Canada in GEMF helped ensure a multidisciplinary sharing of information within the country, and was a model that other countries might want to consider. One of the interviewees for the Thompson River Valley landslide project noted that permitted access to existing right of ways along the railway corridor was a factor in advancing the research. It would have been more difficult without the involvement of the railways themselves.

While collaboration was a cornerstone for conducting research and monitoring activities with external organizations, there was room for improvement in how the components in the Program interacted to deliver the Program. The previous evaluation report (2014) found that CHIS and PSG worked more effectively with external stakeholders than with each other, and noted that opportunities for collaboration between the two were being missed.¹³ The evaluation concluded that this finding was still largely true. Some of this dissonance could be attributed to the operational versus research orientation of each component, which was further exacerbated by internal reorganization in 2017. However, the evaluation observed that collaboration on issues of public safety and disaster risk reduction within NRCan was low overall. Evidence from all lines of evidence demonstrated that NRCan lacked a well-defined internal strategy for coordinating overlapping activities related to public safety and disaster risk reduction.

Design was somewhat limited in supporting uptake and use of information for decision-making

Interviewees noted overlap with climate change adaptation, floodplain mapping, wildfire risk management and other risk assessment work to enhance public safety. For some of the internal and external interviewees, the role and interaction among these NRCan groups was unclear and ad-hoc. They noted several examples where improved clarity could enhance performance and optimize capacity, including: (1) coordinating access to experts and engaging with local and regional jurisdictions; (2) establishing and reporting on the NRCan contribution to the SF indicators; and (3) sharing information within NRCan to support project work.

Internal interviewees noted there were no formal internal mechanisms to coordinate work between these separate teams. While early planning information from the PSG indicated an awareness of the benefit of collaborating with the CCGP, there was little evidence of collaborative planning or activity between the two. During the development of the federal plan to implement the EMS, NRCan coordinated its contribution. Several internal and other government department interviewees noted the effectiveness of that process and suggested a similar table be established more permanently. Developing indicators and performance measures to report on the SF was a work in progress and was challenging due to the lack of internal cohesiveness in reporting across the Department.

Internal interviewees reported that this fragmentation was such that the Program found out about the work of its departmental colleagues from external stakeholders. In addition, several of the internal staff noted challenges in accessing data found in other parts of the Department that would be helpful in conducting work on landslides, tsunamis, earthquakes and volcanoes. As a result, Program staff believed they were missing opportunities to achieve economies of scale and to streamline communication and engagement with external stakeholders that would support the production and uptake of information.

Similar observations were made by external stakeholders. Two of the other government departmental stakeholders and several of the other external stakeholders interviewed, including two from the group of experts, expressed the view that the uncoordinated approach taken by NRCan created the risk of disjointed risk assessment systems in floods, wildfires, earthquakes and other hazards. From their perspective, it would be difficult to integrate hazard-specific approaches in a multi-hazard approach consistent with the vision of the SF at later stages in development. Since programming was at the early stages, there was an opportunity to integrate right from the start. Some of the internal and external interviewees noted that local stakeholders’ capacity was limited for engaging with multiple groups from NRCan and created confusion, particularly for what was viewed as essentially the same purpose – public safety and disaster risk reduction.

Overall, challenges within the projects were effectively mitigated where they were controlled directly by the Program. However, on-going challenges occurred with respect to computer technology and equipment and data communications which were outside the control of the Program. In the NEMI case study, timely support was considered a constant challenge in procurement of computer technology and technical equipment. Program documents also identified lengthy periods for reviewing proposed equipment purchases and delayed response in making repairs to equipment as challenges for mission critical operations within CHIS.

In the strategic interviews, as well as other internal interviews, IT equipment challenges were identified as ongoing throughout the evaluation period. Some internal interviewees reported developing “work-arounds” to facilitate data sharing with external partners. Some of the external interviewees who engaged in sharing data commented on the challenges with cumbersome methods of transferring and accessing data for collaborative work.

Based on the interview evidence, the unique expertise housed at NRCan was a key element of success in the initial phases of risk assessment and hazard identification at the local level. However, as noted previously, a particular challenge for the Program was in conducting the outreach and engagement necessary to support uptake of information for decision-making. Allocating science capacity to develop the long-term relationships and engagement necessary to support the uptake could run counter to the fundamental scientific objectives of the Program. In addition, it could impede career advancement for those scientists.

The involvement of the responsible jurisdictions was also a key factor in success. For example, advances made in British Columbia were facilitated by the direct support of the Government of BC. Work in the Ottawa-Montréal seismic corridor was facilitated by the involvement of the Cities of Gatineau, Ottawa, and Montréal.

Stakeholders suggested some ways to increase alignment to the SF and the EMS and thereby improve the program design in supporting the achievement of its intermediate outcomes. The key area being a platform for discussion of disaster risk reduction and increased coordination within NRCan on activities with similar purpose. Factors impacting program performance were largely linked to the whole of society approach engendered by the SF and the EMS, role and the lack of formal structures for coordinating activities related to Public Safety internally within LMS, and within NRCan overall. Outside of these challenges, program performance was also impacted by hard to replace expertise, and data and technological challenges.

Equity, Diversity and Inclusion (EDI)

Question 4: To what extent does the Program consider EDI factors?

Indicators:

• Evidence that program activities and outputs consider EDI factors, particularly in engagement, consultation, integration of Indigenous knowledge, and communications strategies
• Documented examples of accommodation of EDI factors

The Government of Canada has committed to ensuring that subgroups of the population are not adversely affected by its programming due to an intersection of demographic characteristics. It does so through gender-based analysis plus (GBA+), which is an analytical approach to help assess and identify objectives around equity, diversity, and inclusion. Under the Directive on Results, evaluation teams were directed to consider GBA+ where relevant to consider EDI factors. This was consistent with Priority #1 of the Sendai Framework, which states that Disaster risk management needs to be based on an understanding of disaster risk in all its dimensions of vulnerability, capacity, exposure of persons and assets, hazard characteristics and the environment.

There was evidence that EDI factors were considered by the Program in its design and delivery. Program planning documents for the 2014-2019 iteration of the PSG showed the Program increasing its efforts to integrate Indigenous knowledge into its research activities. Interview evidence confirmed that such efforts positively impacted the research. For example, a significant historic west coast tsunami was more accurately described through comparison with the oral history of the local Indigenous peoples. In another case, Indigenous knowledge influenced the location of sampling sites for fieldwork on landslides in Hudson Bay. According to the interview evidence, this had the further benefit of relationship-building between Indigenous communities and NRCan and generated interest in further collaborative research on geo-hazards. One international expert interviewed for the evaluation recognized the engagement and collaborative work on the west coast of Canada in relation to earthquakes and landslides as best practice.

In its dissemination strategy, the Program communicated research results directly to Indigenous and northern communities. The document review identified at least 23 presentations on research findings and geohazards were made to Indigenous associations / communities and partners between 2018 and 2020. Program documents also showed that sharing research results with local communities and stakeholders was built into some of the research project activities. In northern Canada, presentations were made in Inuktitut to enhance reception within communities. These arrangements were largely funded or facilitated through other programs or external partners’ support.

Evidence also pointed to the fact that more needed to be done. Only 54% of the survey respondents agreed that hazard risk assessment considered vulnerable populations. A majority of interviewees pointed to the need for local data as a key ingredient in risk assessment that consider impacts on vulnerable people. According to the interview and documentary evidence however, the low integration of geolocation data with hazard risk and socio-economic data limited risk assessment for vulnerable populations. Program documents consistently cited the challenge of interoperability of modeling and risk assessment tools as a challenge in supporting risk assessment at the local level. External interviewees stated that the need for local data impacted the ability of the local planners to effectively target plans to reduce risk for vulnerable populations. A couple of the stakeholders from other government departments noted the dissemination of alerts as a concern. Cultural considerations were a key issue in how different populations responded to alerts. For these interviewees, reviewing alert systems and content were an important element for improving alert response among sub-populations.

Interviewees made various suggestions for improving the consideration of EDI within the Program. Suggestions mostly related to consideration of Indigenous populations. Figure 11 illustrates examples of how the Program considered GBA+ factors. Suggestions made during interviews included the following:

• Recognize Indigenous knowledge and credit it appropriately, as well as work done at the community level.
• Continue to engage with communities to expand opportunities to share learning and skills. Science Day in Tuktoyuktuk and delivering research results directly to communities were examples of this.
• Enhancing opportunities for employment in geo-hazards research activities for Indigenous populations.

Figure 11: Considering EDI - Examples

Regarding the management of resources, program interviewees indicated that they strove for gender balance in leadership roles by rotating leadership opportunities in research projects to provide managerial experience more broadly. External interviewees generally expressed the view that the field of emergency management was male dominated, but noticed that the demographics of the scientists was changing and more diversity emerging. However, they also noted that the skill level was very high and hard to find. Therefore, the requisite skills was the most important consideration. Generally, they expressed optimism that new generations of scientists were more consultative and open to engaging with stakeholder groups, which helped to mitigate (but not fully overcome) the lack of diversity.

Best Practices and Lessons Learned

Question 5: Are there any best practices and lessons learned related to the design and delivery of the program?

Indicators:

• Views of stakeholders of alternatives in the design and the delivery of the program
• Identification of lessons learned and evidence of best practices applied

Several lessons learned related to the design and delivery of the Program.

Lesson Learned: Collaboration and relationship-building take time

Document review and interviews revealed several lessons learned related to the design and delivery of the Program. The first lesson learned was that collaboration and relationship building is time-intensive, and required dedicated capacity. Stakeholders indicated that although Program scientists put in a lot of effort engaging with local communities and other organizations, the Program’s work in this regard was often only possible because of capacity tied to their in-progress research activities. Accordingly, it often ended when the research activities were completed. This situation undermined the Program’s ability to foster strong, long-term relationships with stakeholders that facilitated increased understanding of risk, which was a key component of the Sendai Framework for Disaster Risk Reduction.

Lesson Learned: Role definition within NRCan

The second lesson learned was that various teams within NRCan working on hazard areas related to public safety and disaster risk reduction lacked internal mechanisms for collaborating and had roles that were poorly defined in relation to each other. External stakeholders indicated that this was confusing as they were sometimes unsure which team within NRCan they should be contacting with respect to information about specific research areas. Internal stakeholders indicated that this reduced opportunities for intradepartmental collaboration and created confusion regarding reporting responsibilities, and increased the risk of an inefficient allocation of Program resources.

Program’s application of best practices

Best Practice: Collaborative research approach

Document review and interviews revealed several best practices applied by the Program. One best practice is that the Program applied a collaborative research approach. For example, the Program collaborated with the Government of British Columbia on identifying earthquake risk and had a key role in many of its Understanding Risk BC workshops. The Program was also a member of the GEMF, and worked collaboratively with other stakeholders on several projects including: Space Weather; CSSP; and DRR Pathways projects. The Program’s willingness to collaborate with stakeholders increased participants’ uptake of scientific information, as NRCan scientists worked closely with them to ensure that they were able to access, understand, and make use of the Program’s outputs.

Throughout all of its collaborations, the Program defined its role with internal and external stakeholders through written agreements, ensuring that all parties had a clear understanding of the scope of their respective contributions. The Program’s collaborative approach also leveraged program resources in a hyper-efficient manner, which has helped PSG leverage its funds to generate $7 of value for every $1 allocated to the Program.

Best Practice: Mid-year and year-end reports

Another best practice was Mid-year and year-end reports produced within the GSC. This helped the Program identify any potential issues as early as possible and enabled changes as needed to unsure the optimal use of program resources. In development of these reports, the Program also documented the challenges and accomplishments of the Program in real-time, which allowed for the efficient tracking of performance metrics and the proactive advancement of improvement strategies.

Best Practice: CHIS NEMI project documentation and reporting

The NEMI project was notable for its best practice in reporting and project documentation. The use of the dashboard for senior management that reported progress was regularly updated. A complete set of project team meetings was available which documented ongoing challenges and progress. The project also developed a summary presentation of the accomplishments within the project. Field trips were blogged that provided a unique look at the work undertaken within the Project.

Based on the evidence provided throughout the report, key factors that facilitated and challenged the Program can be summarized as presented in Figures 12 and 13.

Figure 12: Facilitators of Program Performance

Figure 13: Challenges Impacting Program Performance

Conclusions

The objective of this evaluation was to assess the relevance, effectiveness, and design of the GKCS program. The evaluation was designed to consider the significant changes in the Program’s operating environment with respect to the Sendai Framework and the Emergency Management Strategy for Canada, and assess whether there were any impacts on the Program. In terms of effectiveness, it focused on the intermediate and longer-term outcomes that previous evaluations could not assess. With respect to program design, the evaluation examined whether the design supported the achievement of results, and documented lessons learned and best practices applied in the Program.

Overall, the evaluation found that the Program achieved meaningful progress towards expected results. This was particularly evident at the immediate outcomes level and where the Program had well established relationships with stakeholders. The expertise located within the program is considered an asset for Canada in disaster risk reduction, and stakeholders credited the program with improving their understanding of geo-hazards and associated risks. The program’s research agenda was relevant and efficiently implemented. In this regard, direct collaboration was an especially effective tool for developing and implementing research around questions relevant to targeted stakeholders. These strengths added to the reputation of the Program as an effective and valued player in the fields of emergency management and disaster risk reduction.

The GKCS program performed very well in the role for which it was originally designed and intended – as the subject matter expert on geo-hazards under the Emergency Management Act providing scientific and technical information to primarily technical and scientific users. However, the implementation of the whole of society approach through the signing of the Sendai Framework and the implementation of the Emergency Strategy for Canada shifted the operating environment for the Program. While the Program took measures to adapt to the changing context, limitations in the original conceptualization manifested over the evaluation period. More specifically, these limitations are associated with Program role, Program reach and Program design.

This shift in the Program operating context drives new expectations and needs around knowledge translation that extend beyond the technical and scientific stakeholder groups originally targeted by the Program. Each has implications for the Program role, achievement of results and program design that warrants consideration by Program management to ensure the Program is well-positioned to address the changes.

The whole-of-society approach rests on a foundation that involves a wider and more diverse range of stakeholders. However, the current Program targets highly technical and scientific stakeholders. The need for technical and scientific information will remain, but a growing demand by general and non-technical users involved in DRR and EM planning will increase demands for less technical information products and direct access to the Program expertise. An increased reliance on science is expected in advancing EM and DRR strategies overall. However, the Program role vis a vis meeting those emerging demands remains unarticulated. In addition, a key finding is that the implementation of the Sendai Framework and the Emergency Management Strategy for Canada impacted the role of the program and had implications for the Program design. Chief among these is the role of the program relative to other federal players in the emergency management and disaster risk reduction. With its focus on involving a wider range of stakeholders, and the increased reliance on the unique program expertise looking forward, the question of how to manage the demand on program capacity with the primary role of science and research become more urgent.

Collaboration is a cornerstone of a multi-hazard, whole of society approach to DRR. Internal to NRCan, various groups involved in DRR, either directly or indirectly, have no mechanisms to effectively collaborate. While this is more of a challenge in relation to stakeholder engagement and strategic policy levels than at the operational level, it creates a risk that approaches to DRR developed by NRCan groups with external stakeholders remain siloed. Fundamentally then, the objectives of the EMS and the Sendai Framework would not be fully realized. Recognizing and responding to these areas requires further attention and will position the GKCS program and NRCan to carry out its role more effectively and contribute more ably to the achievement of results.

Appendix A: Evaluation Matrix

Issues/Questions	Indicators
1. Is the program role appropriate and necessary within the context of changes in the program operating environment?	1.1 Extent to which the activities of the Program align and respond to i) the needs of the intended beneficiaries and stakeholders; and ii) the guidance provided in the Sendai Framework and Canada’s EM Strategy
	1.2 Extent to which the role played by NRCan is complementary to other federal organizations and viewed as appropriate by stakeholders
	1.3 Evidence that the NRCan role is clear and well understood within NRCan, and among external stakeholders
2. To what extent has the program progressed on achieving its intended outcomes?	2.1 Evidence that outputs are produced and disseminated as planned
	2.2 Evidence of Program contribution to achieving intended outcomes
Immediate Outcomes
2.1 Authoritative science foundation on geo-hazards and related risk assessments is available to stakeholders in emergency planning and disaster risk reduction. (PSG primarily)	Stakeholder level of satisfaction with accessibility and usability of program information
	Evidence that stakeholders view NRCan information as authoritative (e.g., requests for information/expertise, references to program outputs)
2.2 Decision-making support tools, models, and approaches are available to stakeholders in emergency planning and disaster risk reduction (CHIS and PSG)	Evidence of direct dissemination to targeted stakeholders
	Evidence that best practices identified in the Sendai Framework are being applied in the development of decision-making support tools and risk assessments
2.3 Information to support decision-making during a hazard event is provided to Canadians and other stakeholders in real-time (CHIS primarily)	Evidence that notifications and alerts are provided to stakeholders within prescribed time parameters
	Evidence that alert and notification systems are coordinated among those who provide alerts and notifications
Intermediate Outcomes
2.4 Responses and mitigation strategies to hazard events integrate scientific knowledge and alert information	Evidence of i) action taken, ii) decisions made, iii) policies and regulations developed based on scientific (NRCan) information
	Trends in stakeholder views regarding their understanding of hazards and their impacts (increasing / improving over time)
	Stakeholder views and documented evidence that stakeholders rely on CHIS alerts and notifications in their decision-making
2.5 Emergency and disaster management stakeholders are equipped to plan for, and respond to, hazard events	Evidence that decision-making support tools are adapted to regional and local needs
	Stakeholder views that that the information and tools provided i) meet their needs and ii) enable effective decision-making
	Evidence that mechanisms to share information and collaborate are in place, are operating, and are considered effective by stakeholders (both in advance of, during, and after hazard events)
Long-term and Ultimate Outcomes
2.6 Canada’s preparedness, response, and recovery from natural disasters is enhanced (Long term outcome)	Evidence that information gaps are collaboratively identified and plans to reduce gaps are developed and resourced
	Stakeholder level of confidence that they are better prepared to act during hazard events
	Expert opinion that Canada’s preparedness for natural disasters has improved since 2005
2.7 Canada is resilient to natural hazards (Ultimate outcome)	Expert opinion to the trend in Canada’s resilience since 2005 (Hyogo Framework)
	Perceived extent to which the Programs helps Canada to be more resilient to natural hazards
	Evidence of incremental improvement on Canada’s resilience to natural hazards (losses avoided, relative recovery time, etc.)
3. Does the program model support the achievement of program objectives?	3.1 Evidence that program design (e.g., decision-making structures, resource allocation, design and delivery mechanisms, collaborative agreements and working arrangements) support program objectives and reflect guidance under Sendai Framework and EM Strategy
	3.2 Identification of factors influencing program performance and assessment of their impact
4. To what extent does the program consider EDI factors?	4.1 Evidence that program activities and outputs consider EDI factors, particularly in engagement, consultation, integration of Indigenous knowledge, and communications strategies
	4.2 Documented examples of accommodation of EDI factors
5. Are there any best practices and lessons learned related to the design and delivery of the program?	5.1 Views of stakeholders of alternatives in the design and the delivery of the program
	5.2 Identification of lessons learned and evidence of best practices applied

Appendix B: Description of Sendai Priorities

Sendai Priority	Description	Program Role / Example
Understanding disaster risk	Policies and practices for disaster risk management should be based on an understanding of disaster risk in all its dimensions of vulnerability, capacity, exposure of persons and assets, hazard characteristics and the environment. Such knowledge can be leveraged for the purpose of pre-disaster risk assessment, for prevention and mitigation and for the development and implementation of appropriate preparedness and effective response to disasters.	Central function of the Public Safety Geoscience Program and Canadian Hazards Information Service
Strengthening disaster risk governance to manage disaster risk	Disaster risk governance at the national, regional and global levels is of great importance for an effective and efficient management of disaster risk. Clear vision, plans, competence, guidance and coordination within and across sectors, as well as participation of relevant stakeholders, are needed. Strengthening disaster risk governance for prevention, mitigation, preparedness, response, recovery and rehabilitation is therefore necessary and fosters collaboration and partnership across mechanisms and institutions for the implementation of instruments relevant to disaster risk reduction and sustainable development.	PSG negotiated Canada’s Global Earthquake Model Foundation (GEMF) membership with partners. A steering committee was struck, and workshops on GEMF’s OpenQuake software were conducted. Risk Project staff contributed to organizing a key panel discussion at Fifth Regional Platform for Disaster Risk Reduction in the Americas in Montreal. Program is reporting performance information reflecting the Sendai Priorities, and is involved in the development of appropriate indicators for measuring Canada’s alignment to the SF.
Investing in disaster risk reduction for resilience:	Public and private investment in disaster risk prevention and reduction through structural and non-structural measures are essential to enhance the economic, social, health and cultural resilience of persons, communities, countries and their assets, as well as the environment. These can be drivers of innovation, growth and job creation. Such measures are cost-effective and instrumental to save lives, prevent and reduce losses and ensure effective recovery and rehabilitation.	Earthquake monitoring infrastructure upgrade Earthquake Early warning system Monitoring of water movement at Ripley landslide.
Enhancing disaster preparedness for effective response and to “Build Back Better” in recovery, rehabilitation and reconstruction:	The steady growth of disaster risk, including the increase of people and assets exposure, combined with the lessons learned from past disasters, indicates the need to further strengthen disaster preparedness for response, take action in anticipation of events, integrate disaster risk reduction in response preparedness and ensure that capacities are in place for effective response and recovery at all levels. Empowering women and persons with disabilities to publicly lead and promote gender equitable and universally accessible response, recovery, rehabilitation and reconstruction approaches is key. Disasters have demonstrated that the recovery, rehabilitation and reconstruction phase, which needs to be prepared ahead of a disaster, is a critical opportunity to “Build Back Better”, including through integrating disaster risk reduction into development measures, making nations and communities resilient to disasters.	Involvement in “La Grande Secousse” in Québec Involvement in Can /(BC)/US drill Comparison of risk level before and after suggested retrofit using ER2 risk assessment tool Investment in early earthquake warning Consideration of GBA+ factors of vulnerability Contributions to the National Building Code of Canada

Footnotes

Figure Footnotes