Evaluation Report: Green Mining Initiative

Table of Contents

• Executive Summary
  • Introduction
  • Evaluation Issues and Methodology
  • Relevance
  • Performance
  • Recommendations and Management Response
• 1.0 Introduction and Background
  • 1.1 Overview
  • 1.2 Background
    • 1.2.1 Overview of the Green Mining Initiative
    • 1.2.2 The PAA
    • 1.2.3 GMI Logic Model
    • 1.2.4 Resources at the Program and Component Level
    • 1.2.5 Program Structure and Governance
• 2.0 Evaluation Approach and Methodology
  • 2.1 Evaluation Scope and Objectives
  • 2.2 Evaluation Methodologies and Limitations
  • 2.2.1 Methodologies
  • 2.2.2 Limitations and Mitigation Strategies
• 3.0 Evaluation Findings
  • 3.1 Relevance
    • 3.1.1 Is there a continuing need for the GMI?
    • 3.1.2 Are the GMI program and activities consistent with government priorities and NRCan strategic objectives?
    • 3.1.3 Is there a legitimate, appropriate and necessary role for the federal government in the GMI program?
  • 3.2 Performance - Effectiveness
    • 3.2.1 To what extent have intended outcomes been achieved as a result of the GMI?
    • 3.2.2 Have there been any unintended (positive or negative) outcomes?
    • 3.2.3 What are the chief internal and external factors influencing achievement of the GMI program objectives?
  • 3.3 Performance - Efficiency
    • 3.3.1 Is the GMI program the most economic and efficient means of achieving outputs and progress toward outcomes?
    • 3.3.2 Is performance information being collected and used on an ongoing basis?
• 4.0 Conclusions and Recommendations
  • 4.1 Conclusions
  • 4.2 Recommendations
• Annex A: Natural Resources Canada Program Alignment Architecture 2013-2014
• Annex B: Evaluation Matrix

Acknowledgements:

The Evaluation Project Team would like to thank NRCan’s Minerals and Metals Sector and in particular CanmetMINING for its collaboration during the evaluation. We would also like to acknowledge the contribution of all those individuals who provided insights and comments crucial to this evaluation.

The Evaluation Project Team was under the direction of Jennifer Hollington, Head of Evaluation, as well as Gavin Lemieux, Director of Evaluation. The evaluation was managed by Jamie Riddell, with the support of Jarret Boon and Michelle Brazil. Evaluation services were provided by TDV Global Inc.

Executive Summary

Introduction

This report presents the results of the evaluation of the Green Mining Initiative (GMI), Sub-Program 2.2.2 in Natural Resources Canada's (NRCan) Program Alignment Architecture,^{Footnote 1} which was undertaken from August 2013 to September 2014.

The mining industry faces ongoing challenges to reduce its environmental footprint. In response, NRCan’s Minerals and Metals Sector (MMS) established the GMI to accelerate the research, development and deployment of green mining technologies and practices; enhance innovation and productivity in mining; improve environmental performance; and improve the competitiveness of the Canadian mining industry thereby positioning Canada's mining sector as the global leader in green mining technologies and practices.

From 2009-10 to 2012-13, all projects carried out under the Sub-Program fell under four main pillars: 1) Footprint reduction; 2) Innovation in mine waste management; 3) Mine closure and rehabilitation; and 4) Ecosystem risk management. During 2012-13, work conducted under the Green Mining Initiative at CanmetMINING was reorganized into five priorities: 1) Clean Water; 2) Best Practices in Mining Environment Management; 3) Energy Efficiency in Mining and Milling; 4) Northern Mineral Development; and 5) Mining Extraction Innovation.

The objective of the evaluation was to assess the relevance and performance (efficiency, economy, effectiveness) of the activities within the Green Mining Initiative (GMI), according to the Treasury Board’s Policy on Evaluation (2009). The evaluation covered the period 2009-10 to 2012-13 and included all activities of the GMI. The program expenditures over the four-year period were budgeted at $37.5 million.

Evaluation Issues and Methodology

The evaluation was completed as per the requirement of the Treasury Board Policy on Evaluation (2009) that all direct program spending be assessed every five years. The evaluation objective was to assess the relevance and performance (effectiveness, efficiency and economy) of the GMI and to make recommendations for consideration by NRCan program and senior management.

The study consisted of document, file and literature review, industry survey, 27 in-depth interviews with key stakeholders as well as eight case studies.

Relevance

Need for the Program

Program documentation and interview findings illustrate that there is a well-defined need and role for the federal government in accelerating the research, development and deployment of green mining technologies and practices, as well as in reducing the environmental impact or footprint associated with mining. GMI is well aligned to government and NRCan objectives and priorities. NRCan has the mandate and expertise to deliver on the objectives of the GMI.

Industry expressed a need for research-centric green mining support from the Government of Canada (GC). GMI’s pillar/priority activity areas are well aligned with industry needs, as validated by the annual Energy and Mines Ministers’ Conference (EMMC) and the Green Mining Initiative Advisory Committee (GMIAC). GMIAC meets quarterly to discuss priorities and the research agenda and to obtain updates. GMI is pivotal in addressing a range of economic, environmental, social, regulatory and science needs associated with green mining and, in turn, is instrumental in achieving a competitive advantage for the country.

Alignment with Government Priorities and NRCan Strategic Objectives

GMI is consistent with the 2010, 2011 and 2013 Speeches from the Throne, Canada’s Economic Action Plan as provided in Budget 2010 and 2011, and is linked to the Northern Strategy. It supports federal government priorities including strengthening Canada’s research advantage, fostering commercialization and business innovation, protecting Canada’s natural environment, supporting Aboriginal and Northern communities, and strengthening Canada’s international engagement.

GMI is aligned to NRCan’s Strategic Outcome 2 (Natural Resource Sectors and Consumers are Environmentally Responsible), Program 2.2: Technological Innovation; and Sub-Program 2.2.2: Green Mining.

Alignment with Federal Roles and Responsibilities

There is a legitimate and appropriate role for the federal government’s support of such green mining activities. The mandate of the GMI accrues from the role of the Minister of Natural Resources as defined under the Department of Natural Resources Act, S.C. 1994.

The government has a legitimate role in leading an initiative that enables mines to be greener and in leading projects that can have tangible environmental and economic benefits for Canadians. The GMI is the only federal initiative in place that looks at the entire mining lifecycle through four comprehensive pillars: Footprint Reduction; Mine Waste Management; Mine Closure and Rehabilitation; and Ecosystem Risk Management. These comprehensive pillars and five related priorities focus on reducing the amount of waste produced in mining and minimizing its impact on the environment at all stages of the mine lifecycle.

Performance

Achievement of Expected Outcomes

The evaluation found that the program had effectively produced a wide range of project outputs and had been successful in achieving some of its immediate outcomes. In particular, the vast number of projects had contributed to an expanded knowledge base of innovative mining technologies. Examples include Green Mines Green Energy, Mine Environment Neutral Drainage (MEND), bio-waste for remediation of mine tailings, ventilation technologies and alternative vehicles for use underground.

At the intermediate outcomes level, the adoption of green mining innovation technologies leading to commercialization is difficult to assess as this is dependent on economic realities, time needed to adopt technologies (e.g., purchasing, retrofitting) and reluctance by industry to be first in implementing new technologies (i.e. to disproportionately assume the risk for the industry). While green mining practices are viewed as a desirable outcome, the advantage in adopting innovative technologies lies in being “first to be second” when the technology is proven as there is then a smoother regulatory approval process, fewer costs, and lower associated risks.

The achievement of the ultimate outcomes of “an innovative, environmentally and socially conscious industry” and a “positive public perception and image of the industry” has yet to be measured and would likely need a baseline assessment to better understand the scope and reach of the GMI.

Demonstration of Economy and Efficiency

GMI successfully leverages NRCan funds at the project level thereby increasing the resources available to complete project activities. Interview and case study evidence indicates that the practice is central to the program structure, and continues at ratios estimated to be 6:1 overall, and even as high as 7:1 in the case of the Green Mines Green Energy project. However, while successful leveraging occurs at the project level, the absence of clearly articulated research priorities may have resulted in missed opportunities for further leveraging.

The program could take advantage of opportunities and efficiencies with other federal entities (e.g., Environment Canada (EC), Industry Canada (IC), Foreign Affairs, Trade and Development Canada (DFATD)) and regional development agencies (e.g. Atlantic Canada Opportunities Agency (ACOA), Western Economic Diversification Canada (WD)) that might have a vested interest in the success of the GMI or the achievement of joint objectives (e.g., trade, innovation).

While significant effort has been devoted to collecting broader CanmetMINING information, within the key tool for performance data (i.e., the “dashboard”), there does not appear to be a standard for data collection, performance information or metrics across GMI projects. Lack of consistency in project data made it challenging to identify performance at the GMI program level. This resulted in a limited assessment of GMI during the evaluation. An improved performance measurement and reporting strategy would allow CanmetMINING to identify GMI specific-efficiency and economy outcomes in the future.

Recommendations and Management Response

Recommendations	Management Response	Responsible Official/Sector (Target Date)
1. Opportunities exist to engage other federal entities as the research conducted under the GMI extends across several federal initiatives. To take full advantage of this, it is recommended that GMI conduct a review of federal programs to identify potential efficiencies and areas for collaboration.	Accepted. A formal environmental scan of GMI will be done by CanmetMINING that will include other government departments such as the National Research Council, specialized funds like Sustainable Development Technology Canada, academia and Canada Mining Innovation Council.	ADM Minerals and Metals Sector (December 2015)
2. Given the evidence highlighting the need  to attract new partners through clearly articulated program priorities, it is recommended that GMI establish an overarching internal green mining research agenda that will guide long-term priority setting.	Accepted. CanmetMINING will develop a broad research agenda in consultation with stakeholders to guide priority settings and project decisions for internal projects. We are also developing a Green Mining narrative to help frame and communicate the initiative.	ADM Minerals and Metals Sector (December 2015)
3. Given that no GMI-specific performance measurement and reporting strategy exists, it is recommended that a performance measurement and reporting strategy for the GMI be developed by the Program to systematically track progress on outputs and outcomes and ensure consistent performance data are collected across projects. This PMRS should include a logic model and associated narrative.	Accepted. CanmetMINING will put in place a performance measurement and reporting strategy by integrating a logic model for its GMI operations. It will encompass indicators of outputs and outcomes.	ADM Minerals and Metals Sector (December 2015)

1.0 Introduction and Background

1.1 Overview

The following is an evaluation of Natural Resources Canada's (NRCan) Sub-program 2.2.2, Green Mining. The evaluation covers the period 2009-10 to 2012-13 and $37.5 million in NRCan expenditures over the four-year period.

The evaluation’s objective was to assess the relevance and performance (effectiveness, efficiency and economy) of the GMI and to make recommendations for consideration by NRCan program and senior management.

1.2 Background

1.2.1 Overview of the Green Mining Initiative

The Green Mining Initiative (GMI) was announced in 2009^{Footnote 2}, and endorsed at the Energy and Mines Ministers’ Conference by federal, provincial and territorial mines ministers. The GMI brought together green mining Science and Technology (S&T) and Research and Development (R&D) under the Green Mining Sub-Program, work which had already been undertaken with various stakeholders in the mining sector by the Mines and Minerals Sector (MMS).

Building on the efforts of the National Orphaned and Abandoned Mines Initiative and Green Mines Green Energy project (GMGE), the Green Mining Initiative was designed to develop collaborative approaches and share relevant scientific knowledge in key research areas across the mining sector. The Director General (DG) of CanmetMINING was responsible for the Green Mining Initiative program and ensuring that the objectives were met.

The “GMI targets the development of innovative energy-efficient technologies required for mining to leave behind only clean water, rehabilitated landscapes and healthy ecosystems. It aims to improve the mining sector's environmental performance, to promote innovation in mining and to position Canada's mining sector as the global leader in green mining technologies and practices.”^{Footnote 3}

In recent years, NRCan commissioned two studies, Delphi Group (2010)^{Footnote 4} and Meyers Norris Penny (2011)^{Footnote 5}, to determine the state of green mining technology in the Canadian mining sector and to identify barriers to the uptake of green mining innovations. While both reports noted that Government regulations were not barriers^{Footnote 6} unto themselves to the adoption of green mining technologies and practices, some derivative issues could in essence become barriers. As stated in the Delphi Group Report^{Footnote 7} “the process of assessing and approving environmental performance is resulting in challenges for regulators, communities and mining companies, including:

• Lack of information/evidence on the performance of a novel or new mining technology increases perceptions of risk;
• Capacity to substantiate/verify the performance of new technologies for regulators and communities; and
• The ability to define the stringency of environmental standards to address local eco-system concerns.”

The Meyers Norris Penny (2011)^{Footnote 8} report further noted that there was significant risk aversion to implementing new technologies in the mining sector due to associated costs and the competitive nature of the companies. The GMI was designed to address the issues raised in this report by undertaking pre-competitive research and development and pursuing pilot and demonstration projects to help reduce the risks associated with new technologies. In addition, GMI works to address key competitive challenges in the mining industry by helping to maintain and expand Canada’s mineral resource advantage and leading role in the global arena.

Canada’s key competitors in the global mining sector are investing in green mining technologies as resource-rich countries place a greater emphasis on environmental performance to allow investment in mining and project development. Reducing the environmental impacts of mining and related activities continues to be one of the most significant and important challenges for the sector. The industry’s public image is closely tied to its environmental performance, and the industry recognizes it must continue to improve to ensure it remains responsible, competitive and productive, in Canada and internationally.

At the Energy and Mines Ministers’ Conference in September 2009, the GMI was endorsed by federal, provincial, and territorial mine ministers.^{Footnote 9} Furthermore, the Canada Mining Innovation Council (a consortium of industry, academic and government leaders) endorsed the GMI and included it as part of its national Mining Research and Innovation Strategy.

The objectives for the first five years of the Initiative were to:

• Improve environmental performance within the mining sector;
• Promote innovation within the mining sector through new technologies; and
• Realize market opportunities in green mining technologies.

From 2009-10 to 2012-13, all projects carried out under the Sub-Program fell under four main pillars, namely:

• Footprint reduction;
• Innovation in mine waste management;
• Mine closure and rehabilitation; and
• Ecosystem risk management.

Within the 2012-13 timeframe, work conducted under the Green Mining Initiative at CanmetMINING was reorganized into five priorities:

• Clean Water;
• Best Practices in Mining Environment Management;
• Energy Efficiency in Mining and Milling;
• Northern Mineral Development; and
• Mining Extraction Innovation.

More specifically, GMI is intended to:

• Target the development of innovative energy-efficient technologies required for mining so that only clean water, rehabilitated landscapes and healthy ecosystems remain. It aimed to improve the mining sector's environmental performance, to promote innovation in mining and to position Canada's mining sector as the global leader in green mining technologies and practices;
• Realize market opportunities in green mining technologies;
• Undertake pre-competitive research and development and pursue pilot and demonstration projects to help reduce the risks associated with new technologies; and
• Raise industry confidence to implement new green technologies to reduce the environmental impacts of mining and related activities.

1.2.2 The PAA

In NRCan’s Program Alignment Architecture (PAA), the Green Mining Initiative contributes to NRCan’s Strategic Outcome (SO) 2: Natural Resource Sectors and Consumers are Environmentally Responsible, Program 2.2: Technological Innovation; and Sub-Program 2.2.2: Green Mining.

1.2.3 GMI Logic Model

The logic model^{Footnote 10} depicted in Figure 1 demonstrates the causal relationships between the GMI Program’s activities and the intended immediate, intermediate, and final outcomes. This logic model was developed by the evaluation team and the program during the initial phase of the evaluation.

Figure 1: Green Mining Initiative Logic Model

Text version

Figure 1:  Logic Model

Figure 1 presents a logic model of the Green Mining Initiative (GMI). The logic model is organized in a series of activities, outputs, and outcomes that are linked in a chain of results. The two Activities have directly associated Outputs that are targeted to those identified in the Reach column. On the figure, Reach refers to the target audiences of the GMI Outputs and is depicted between the Outputs and Immediate Outcomes.

Completing the Activities and producing the Outputs is expected to result in the achievement of the Immediate Outcomes. Achieving the Immediate Outcomes is expected to contribute to the achievement of Intermediate Outcomes, which in turn will contribute to the achievement of the Ultimate Outcomes.

Canada’s Green Mining Initiative – Logic Model

Activities	1.  Funding of S&T activities in the areas of: Northern Mineral Development Mine waste management in the north Ecosystem protections and remediation of northern contaminated sites Mine design for the North Mining Extraction Innovation:  improving underground mining processes Energy Efficiency in Mining and Milling: technologies for improving mining efficiency and energy usage reducing energy consumption in mining operations. Clean Water: assessment of environmental risks of mine-related contaminants technologies for sustainable use of water in the mining industry. Critical and Strategic Minerals Processing: Bioleaching Mining Environmental Management: mine waste management and reclamation regulatory input and technology transfer.	2.  Funding of Green Mining Stewardship Activities in the areas of: Advocacy Regulatory Affairs Stakeholder Engagement
Outputs	New scientific knowledge New technologies New processes Pilots Demonstrations Best practices Validated approaches Scientific publications	Regulatory Input Stakeholder Engagement Mechanisms Communications Knowledge exchange and transfer Information products.
Reach	Aboriginal Affairs and Northern Development Canada Environment Canada Other Government Departments Mining Corporations Provinces and Territories International Research Organizations (e.g. CMIC, CEMI)
Immediate Outcomes	Expanded knowledge base of innovative mining technologies and practices that eliminate or reduce environmental impacts and risks Expanded understanding of regulatory, stakeholder and environmental concerns and requirements Expanded understanding of Canada’s Green Mining Initiative
Intermediate Outcomes	Adoption of green mining innovation technologies leading to commercialization Expanded understanding of regulatory, stakeholder and environmental concerns and requirements Greater understanding by stakeholders of Canada’s commitment to environmentally sound and economically viable mining practices
Ultimate Outcomes	An Innovative, environmentally and socially conscious industry Positive public perceptions and images of industry

The current logic model acknowledges two broad ranges of activities underpinning the GMI namely, science and technology (S&T) activities as well as activities related to green mining stewardship. The first set of activities comprises S&T activities in the following six key areas :

• Northern Mineral Development: focuses on mine waste management, ecosystem protection and remediation of contaminated sites and mine design for the north. This segment of activities is focused on northern sites and is a pivotal Government of Canada priority.
• Mining Extraction Innovation: activities and projects focus on improving and modernizing underground mining processes at the core of mining operations.
• Energy Efficiency in Mining and Milling: activities and projects focus on novel technologies for improving mining efficiency and energy usage as well as reducing energy consumption in mining operations. Energy usage is a key cost driver in mining operations.
• Clean Water: activities and projects focus on assessing and addressing environmental risks of mine-related contaminants as well as fostering the development of technologies for sustainable use of water in the mining industry.
• Critical and Strategic Minerals Processing: activities and projects focus on bioleaching, a process of using bacteria instead of chemicals to dissolve metals. When compared with chemical methods of separation, bioleaching is slower but cleaner.
• Mining Environmental Management: activities and projects focus on topics such as mine waste management and reclamation as well as regulatory input and technology transfer.

The outputs generated from this activity set range from the development of new scientific knowledge, new technologies, new processes, pilots, demonstrations, best practices, validated approaches right through to scientific publications. These outputs are aligned with the needs identified by partners and stakeholders that are also derived from the Mines Ministers’ Conference and Green Mining Committee as well as departmental priorities.

The second activity set, green mining stewardship activities, aim at mobilizing stakeholder engagement, gleaning a better understanding of regulatory affairs, as well as providing a platform for advocacy for green mining technologies, research and development. The outputs achieved cover those ranging from regulatory input, to the creation of stakeholder engagement mechanisms, communications and knowledge exchange/transfer and Information products.

The delivery of these activities and the sustained delivery of outputs are intended to culminate in the achievement of the following immediate outcomes:

• Expanded knowledge base of innovative mining technologies and practices that eliminate or reduce environmental impacts and risks;
• Expanded understanding of regulatory, stakeholder and environmental concerns and requirements; and
• Expanded understanding of Canada’s Green Mining Initiative.

From an intermediate outcomes perspective, the program aims to foster the industry’s adoption of green mining innovation technologies as well as enhance stakeholders’ understanding of Canada’s commitment to environmentally sound and economically viable mining practices. These aspects are important to achieving the ultimate goal of creating/reinforcing an innovative, environmentally and socially responsible industry as well as achieving better public perception of the mining industry.

1.2.4 Resources at the Program and Component Level

Annually, GMI financial resources include approximately $8.5M^{Footnote 11} in NRCan A-Base funding, of which 75% is dedicated to salaries, and an additional $2-3M^{Footnote 12} in cost recovery from projects. Over the four-year evaluation period, GMI resources totalled $37.5 million. Resources were also leveraged from partners (industry, academia, NGOs) on a project-by-project basis through in-kind and cash contributions (e.g., $47.2M^{Footnote 13} in leveraged project funding for 2011-2012).

1.2.5 Program Structure and Governance

GMI activities are carried out as part of the Canada Centre for Mineral and Energy Technology (CANMET) by approximately 70 FTEs based in the CanmetMINING facilities in Ottawa, Sudbury, and Val-d’Or. Project leads are responsible for the day-to-day management of their specific activities. Four Program Managers in CanmetMINING are responsible for selecting projects depending upon which Pillar/Priorities these projects fall. The Project Managers administer these projects and report to the Director General (DG) of CanmetMINING. The DG directs the overall Green Mining activity, is responsible for strategic planning, and ensures that the objectives are met.

In addition, the GMI also receives oversight and guidance from two committees:

• Green Mining Initiative Advisory Committee (GMIAC): Co-chaired by the Canada Mining Innovation Council^{Footnote 14} (CMIC) and NRCan, this committee provides overall guidance for the Green Mining Initiative in order to ensure that it maintains its relevance to key stakeholders such as the provinces and territories, industry and academia; and
• Federal Provincial Territorial Committee on Green Mining: This committee is tasked to identify further science and technology needs which can contribute to an effective and efficient regulatory system; and to identify opportunities to accelerate the development and implementation of green mining technologies.

The following figure illustrates the organizational chart for GMI:

Figure 2: Green Mining Organizational Chart^{Footnote 15}

Source: GMI Evaluation Assessment, 2013

Text version

Figure 2: Green Mining Organizational Chart

Figure 2 is an organizational chart showing a three-level management structure for GMI. 

The top level is the Director General who has five direct reports. The five direct reports are as follows:

1. Science Policy Analyst – no direct reports
2. Executive Assistant – no direct reports
3. Regional Director, Green Mining Innovation – Extraction
4. Director, Green Mining Innovation – Processing
5. Director, Business and Operations

The following four positions report to the Regional Director, Green Mining Innovation – Extraction:

1. Program and Site Manager – Ground Control (Bells Corner)
2. Program Manager – Alternative Energy Technologies (Sudbury)
3. Program and Site Manager – Mining Extraction Technologies (Val d’Or)
4. Program and Site Manager – Radioactive Waste Stabilization (Limebank)

The following three positions report to the Director, Green Mining Innovation – Processing:

1. Program Manager, Processing and Mine Waste Management
2. Program Manager, Mine Closure and Ecosystem Risk Management
3. Program Manager, Analytical Services and CCRMP

The following four positions report to the Director, Business and Operations:

1. Manager, Business Affairs and Communication
2. Manager, Finance and Administration, Health, Safety and Security
3. Manager, Quality Systems and Engineering and Technical Services
4. Manager, Information Systems

Source: GMI Evaluation Assessment, 2013

2.0 Evaluation Approach and Methodology

2.1 Evaluation Scope and Objectives

The Treasury Board Policy on Evaluation (2009) requires that Government of Canada (GC) evaluations examine a program’s relevance and performance (effectiveness, efficiency, and economy). In 2013-14, the NRCan Strategic Evaluation Division (SED) undertook an evaluation of GMI. The evaluation covered the activities and $37.5 million in NRCan expenditures over the four-year period 2009-10 to 2012-13, and considered the following five evaluation issues:

Relevance

1. Is there a continued need for the Program? (Assessment of the extent to which the Program continues to address a demonstrable need and is responsive to the needs of Canadians).
2. Alignment with Government of Canada (GC) priorities. (Assessment of the linkages between program objectives and federal government priorities and departmental strategic outcomes).
3. Alignment with federal roles and responsibilities. (Assessment of the role and responsibilities for the federal government in delivering the GMI).

Performance: Effectiveness

4. Achievement of Expected Outcomes. (Assessment of progress towards immediate, intermediate and ultimate outcomes).

Performance: Efficiency

5. Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes.

This report provides the findings for each of these defined evaluation issues and questions. In Section 3, findings are organized and discussed by evaluation question and associated corresponding sub-questions. Conclusions with recommendations for the GMI are presented at the end of the report in Section 4.

2.2 Evaluation Methodologies and Limitations

2.2.1 Methodologies

For this evaluation, data collection consisted of gathering information from the following sources:

• document/file/literature review;
• interviews;
• online survey of GMI stakeholders (industry, academia, provinces/territories); and
• case studies (8).

Summary technical reports were developed for each line of evidence. Further, a logic model was developed by the evaluation team in conjunction with the Program.

Document Review

Over 786 files in 16 folders were identified in the GMI SharePoint directory^{Footnote 16} during the Evaluation Assessment^{Footnote 17} and the development of the Evaluation Methodology report. For the purposes of the Document Review, additional documents were identified by the program and through an extensive web-search. Together, these documents provided a comprehensive record of the Initiative including background information, terms of reference, financial information, internal and external reports and presentations, program updates and other relevant documents. Table 1 summarizes the nature of the most salient^{Footnote 18} types of documents per type:

Table 1 – Documents for Review

Type of Document	Number
100 – Governance and Administration Data Includes program history and evolution, program structure, reporting structure, program accountability, organizational chart, performance management framework (indicators – data collection, etc.), logic model, policy documents and legislation, other related issues	33
200 – Financial and Resource Data (Including Materiality) Includes average annual expenditures, expenditure breakdown per activity type (pillars or priorities), average staffing levels (FTEs), industry participation – cost sharing (financial or in-kind) per pillar or by priority), general funding information, other related issues	6
300 – Program Data Includes GMI projects per pillar, GMI projects per priority focus, general information on projects	7
400 – Stewardship Data Includes advocacy, regulatory affairs, stakeholder engagement	12
500 – Program Output Data (Key Performance Indicators (KPI) Includes new scientific knowledge, new technologies, new processes, pilots, scientific publications, validated approaches, demonstrations, best practices, dashboard	3
600 – Stewardship Output Data Includes regulatory input, stakeholder engagement mechanisms, communications, knowledge exchange and transfer, information products	15
700 – Outcome Data Includes expanded knowledge base of innovative mining technologies and practices that eliminate or reduce environmental impacts and risks, expanded understanding of regulatory, stakeholder and environmental concerns and requirements, expanded understanding of Canada’s Green Mining Initiative	2
800 – Performance Measurement Data (Efficiency and Economy) Includes business cases, action plans, critical path	9
900 – Other Documents Includes published articles on green mining, annual reports and financial reports, literature, supplemental information of interest to the evaluation (e.g., evaluation assessment of the GMI, methodology report/detailed work plan for the evaluation of the GMI)	8
TOTAL	95

Interviews

With the help of program staff and project/program records, a total of 41 interviewees were identified across four categories: NRCan Senior Management, NRCan Program Researcher, Other (Federal) Government Departments and Provinces/Territories, and Industry representatives from the mining industry. Twenty-seven interviews were completed for the evaluation. Table 2 outlines the categories of interview groupings that were used for analysis:

Table 2 - Interviewees

Grouping	Description	Number of Interviewees
NRCan Senior Management	Includes NRCan Senior and Program Managers	4
NRCan Program Researchers	Includes Program Managers and Research Scientists	12Footnote 19
Other Government Departments and Provinces/Territories	Includes OGDs (Environment Canada, Sustainable Development Technology Canada, and the PCO); and Provinces(Alberta and Québec)	6
Industry	Includes representatives of the mining industry and CMIC	5

Online Survey of GMI Stakeholders

The purpose of the stakeholder survey was to reach GMI stakeholders to assess the relevance of the Program to their needs and priorities, as well as its effectiveness in addressing those needs. The 20 survey questions were developed by the evaluation team with input from the Evaluation Advisory Committee (EAC). The survey consisted of a rating scale and/or Likert questions with some open questions to collect individual respondent’s comments.

Based on consultations with program staff, it was determined that the best target population for the survey was the membership of the Canadian Mining Innovation Council. Due to confidentiality concerns, CMIC was required to administer the distribution of the survey. The bilingual survey was launched with the support of the Director General (DG), CanmetMINING on Monday, March 17, 2014 and closed on Thursday, March 27, 2014. According to the CMIC’s Executive Director & CEO, there were 568^{Footnote 20} recipients with a 24.5% open rate (i.e., 138 people opened the email). Of this open rate, 18 people out of 568 recipients completed the questionnaire for a completion rate of 3%. Survey results are considered indicative only given the low response rate.

Case Studies

Case studies were chosen by the evaluation team, in conjunction with the Program based on the types of partner organizations, project delivery dates, funding and resourcing levels and degree of adoption. To identify case studies, projects were assessed against the criteria in order to determine those most suitable to inform the evaluation. Eight were selected as appropriate. The case studies were conducted using interviews and document reviews. Findings were identified per case study and then combined for analysis to create one technical report to inform the evaluation.

Based on the criteria, the selected case study topics were:

• Case Study #1: Alternative Binder
• Case Study #2: Ventilation-on-Demand
• Case Study #3: Transformation & Dissolution Protocol
• Case Study #4: Green Mines Green Energy
• Case Study #5: Hybrid Scoop
• Case Study #6: Inter-Governmental Collaboration
• Case Study #7: MEND and NOAMI
• Case Study #8: ECOEII Hybrid Filtration System

The case studies were conducted using interviews and document review as lines of evidence.

Logic Model

At the beginning of the evaluation, the GMI lacked an agreed upon logic model. The evaluation team, with the support of Program stakeholders, developed a draft logic model for the Initiative to identify program outputs, and immediate, intermediate and ultimate outcomes for use in the evaluation.

2.2.2 Limitations and Mitigation Strategies

The evaluation methodology provided the evidence required to conclude on all issues and questions. However, there were some limitations associated with the evaluation as outlined in Table 3.

Table 3: Evaluation Limitations and Mitigation Strategies

Limitations	Mitigation Strategies
The industry survey response rate was much lower than anticipated, affecting the reliability of the survey results. Accordingly, survey evidence is indicative only. It is possible that the low response rate may be attributable to a few factors, namely: lack of awareness of potential respondents of the Green Mining Initiative sub-program (i.e., they may not have recognized the name of the sub-program but may have recognized it under the broader CanmetMINING); the requirement to seek additional information from another source (“description of the Sub-Program is available online at: www.nrcan.gc.ca/minerals-metals/technology/4473”), may have given the impression to respondents of additional steps in the survey process and potentially adding the perception of complexity and time to complete the survey; the more detailed program description was contained in the introduction to the survey and while long, certain extracts may have been beneficial in the cover letter to attract the interest of potential respondents. the absence of a defined list of projects or scope of Green Mining Initiative sub-program; and other competing demands for time.	A reminder was sent to industry survey recipients midway through the survey period. The number of interviews from the industry stakeholder group was increased in order to mitigate this limitation; however, no additional industry stakeholders were available to provide input.
At the program level there was limited financial/HR data (e.g., financial breakdown beyond planned vs. actual spending, complete business case financials, HR levels and apportionment to the various pillars/priorities, etc.).	The Program was asked to address high level gaps. In other instances, where it was believed that the level of effort and time required to manipulate financial systemsFootnote 21 to achieve the stated aim would have been too onerous, a recommendation was made to the Program to refine its tracking and reporting in the future.
A performance management framework was not in place during the evaluation period. The program lacked a logic model, performance indicators and data collection strategy which impacted the ability of the evaluators to assess the true performance of the GMI.	The Evaluation Team worked with the program to develop a logic model and related indicators for the evaluation.
It is important to note that while the evaluation questions focus on the GMI as a program (pillars/priorities), the vast majority of responses/perspectives were provided at the project level.	Interviewees were asked to provide program level responses but mainly stayed within the project realm.

3.0 Evaluation Findings

3.1 Relevance

3.1.1 Is there a continuing need for the GMI?

Analysis:

What needs are program activities addressing?

• The primary beneficiary of the GMI is the mining and exploration industry. The mining industry sector^{Footnote 22} includes industry associations and research organizations (e.g., Mining Association of Canada, Canadian Mining Innovation Council (CMIC)^{Footnote 23}, Prospectors and Developers Association Canada (PDAC)). Their needs pertain to:
  • Support for mining-related innovation;
  • Identification of gaps in R&D and technology validation and adoption;
  • Assistance in addressing regulatory inefficiencies impeding the timely adoption of mining-related innovation;
  • Identification of current and emerging research trends and key issues that need to be addressed in “green mining” research in Canada;
  • Focusing research efforts on the development of technologies and procedures to reduce the environmental costs of mining activities in Canada, and the economic costs of environmental good performance; and
  • Helping to improve the image of the industry with respect to environmental performance and earning a social licence to operate (SLO).
• A significant portion of industry’s needs have/are being filled through the research projects that have been/are being initiated by GMI and the dissemination of results.
• Provincial and Territorial Governments have been interested in increased federal, provincial and territorial regulatory harmonization. The GMI has aligned to recent provincial mining strategies (i.e., in BC, Quebec and Ontario) and continues to endeavour to foster regulatory harmonization.
• Other Government Departments (OGDs) have identified a need to improve the regulatory processes in order to address challenges impeding industry innovation (e.g., lack of awareness/understanding of new technologies; delays and uncertainty in costly, time consuming regulatory processes). NRCan (MMS) is a responsible authority for the majority of environmental assessments of mining projects involving Tailings Impoundment Area (TIA) proposals. These assessments are carried out with OGDs which include Department of Fisheries and Oceans Canada (DFO), Transport Canada (TC) and the Aboriginal Affairs and Northern Development (AANDC) in the case of Nunavut projects. MMS’ scientific expertise in the area of mine effluent management participates in Environment Canada (EC) initiatives aimed at improving the administration of Mines and Metals Effluent Regulations (MMER). MMS works closely with EC and DFO to identify further opportunities to improve the efficiency and effectiveness of the TIA scheduling process through regulatory amendments and/or legislative changes.
• The industry’s short-term research needs for solutions to specific industrial problems often conflict with the long-term nature of university research. The GMI has a dual role as the developer and provider of knowledge as well as having expertise in innovative technologies. Applied research has historically bridged this dichotomy.
• Aboriginal communities have identified needs concerning economic development and sound stewardship of resources. Mining companies have become Canada’s largest private sector Aboriginal employer. GMI has carried out projects (e.g., ventilation on demand, tailing rehabilitation, etc.) that benefit local communities, worker health and safety, as well as many projects that enhance the economic viability of the industry.
• The General Public is interested in reducing the environmental impact of mining activities:
  
  [One] of the greatest areas of public interest in the mining industry is the environmental impact of mining activities. Concerns about acidic drainage, heavy metals contamination, releases of tailings or other wastes into natural waters, and the general issue of environmental footprint have led to a widespread negative public image of mining.^{Footnote 24}

Are the needs continuing?

• Needs are identified through input from the CMIC and an advisory committee. This practice remains in place and continues to be the main vetting mechanism for ongoing needs. As a demonstration of continued need, the GMI successfully leveraged significant resources (financial and in-kind) from industry and other stakeholders.
• Interviewees unanimously stated that there was a continuing need for the GMI. Some stated that the need would increase based on population expectations and more strenuous regulations. The GMI was instrumental in addressing a range of economic, environmental, social, regulatory and science needs associated with green mining which in turn were instrumental in contributing to a competitive mining advantage for the country.
• The survey results indicated a gap between the areas of need that were the most important to the survey respondents, and the level to which GMI was meeting these needs. While it was acknowledged that most of the respondents did not know whether the GMI and its activities were addressing their needs in the priority areas identified, it was suggested that the areas of need reflect a desire for weighted support.
• All case study projects linked clearly to a pillar or stated NRCan priorities. Some linked to more than one. Upon review it was observed that all case study projects were conducted in areas of identified need. Furthermore, the eight case study project interviewees indicated an on-going need for further work and support.

Has the GMI adapted to meet new or changing needs of stakeholders?

• According to interviewees, from its inception in 2009, the GMI worked toward responding to stakeholder needs as they arose and changed over time (e.g., ranging from addressing regulatory barriers in 2011 to a renewed focus on emerging issues). The literature did not contain any Public Opinion Research or satisfaction level research to corroborate the extent to which it had adapted to meet the new challenges.
• Almost all of the stakeholders indicated that the GMI had adapted to meet the new or changing needs of stakeholders. From a delivery perspective, projects were adjusted to reflect specific stakeholder needs and availability, on a case-by-case basis.

3.1.2 Are the GMI program and activities consistent with government priorities and NRCan strategic objectives?

Analysis:

To what extent is the GMI linked to federal government priorities?

• The GMI is consistent with the 2010, 2011 and 2013 Speeches from the Throne, Canada’s Economic Action Plan as provided in Budget 2010 and 2011, and is linked to the Northern Strategy. GMI directly supports the Federal Sustainable Development Strategy (FSDS) which outlines goals, targets and implementation strategies under four themes. Under the theme of “Addressing Climate Change and Air-Quality,” GMI supports the goals of reducing greenhouse gas emissions and mitigating the impacts of climate change as well as minimizing threats to air quality so that the air Canadians breathe is clean and supports healthy ecosystems.
• NRCan contributes to Green House Gas (GHG) mitigation and air quality through Ecosystem Risk Management.
• GMI is consistent with NRCan objectives. For instance, NRCan outlined the following government priorities, strategies and frameworks to which the GMI contributes:^{Footnote 25}
  • Supporting the Federal Government’s Economic Action Plan through responsible and sustainable resource development. It also fostered the support of communities through employment and healthier Aboriginal and Northern communities/environments in alignment with the Northern Strategy);
  • Strengthening Canada’s research advantage through the enhanced competitiveness of Canadian industries; and support for R&D and innovation;
  • Fostering commercialization and business innovation through improving demonstration and adoption of new technologies;
  • Protecting Canada’s natural environment; and
  • Strengthening Canada’s international engagement in areas such as helping address issues related to Corporate Social Responsibility (CSR) as well as the Arctic.

To what strategic departmental objectives and priorities are GMI activities, outputs, and outcomes linked?

• GMI is aligned to NRCan’s Strategic Outcome 2 (Natural Resource Sectors and Consumers are Environmentally Responsible), Program 2.2: Technological Innovation; and Sub-Program 2.2.2: Green Mining.

Table 4 - GMI and NRCan's Program Alignment Architecture

Level	Description
SO 2	Natural Resource Sectors and Consumers are Environmentally Responsible
Program 2.2 Technological Innovation	Solutions to the environmental challenges faced by the natural resource sectors require sustained efforts in research, development and demonstration because the current level of science and technology is inadequate to address these concerns. However, the natural resource sectors neither have all the necessary knowledge nor make the necessary investments in innovation due to the potential poor return on investment. The objective of this program is to encourage academia, industry and the public sector to research, develop and demonstrate innovative solutions to environmental challenges encountered in the natural resource sectors. This objective is achieved through the generation and dissemination of scientific knowledge, and the development and demonstration of new technologies.
2.2 Result:	Stakeholders invest in research, development and demonstration to address environmental challenges
Sub-Program 2.2.2 GMI	Mining and processing have impacts on land, water and air. Technology development and commercialization entail significant financial, market, and technical risk, because uptake is dependent upon regulatory requirements, business investment priorities and availability of funding. This program reduces business risks by developing and demonstrating innovative mining technologies and practices that eliminate or reduce environmental impacts and risks. Business needs, technology gaps and priorities are identified with input from the Canada Mining Innovation Council and an advisory committee. Applied research aims to a) reduce land disturbance; water, energy and hazardous chemical use; waste volumes; and releases to the environment and b) accelerate site restoration. By responding to business needs and seeking out synergies, the program focuses development efforts on technologies with the most potential to reduce environmental impacts and risks and to expand domestic and international business opportunities for mining companies, technology developers and consultants. Findings also a) contribute to the scientific, technological and socioeconomic basis for updating federal, provincial and territorial mining and environmental regulations and policies and b) inform policy priorities in other countries that drive demand for technologies and services.
2.2.2 Results:	Technology developers increase demonstration of environmental technologies; Industry partners increase financial and in-kind contributions; and academic, government and other non-industry partners increase financial and in-kind contributions

Source: NRCan Performance Measurement Framework 2012-13

3.1.3 Is there a legitimate, appropriate and necessary role for the federal government in the GMI program?

Analysis:

Should the Government of Canada be involved in Green Mining, S&T and R&D?

• The mandate of the GMI accrues from the role of the Minister of Natural Resources as defined under the Department of Natural Resources Act, S.C. 1994.
• The mining sector is a core component of the Canadian economy. All lines of evidence^{Footnote 26} indicate a role for the Government of Canada (GC) in Green Mining S&T and R&D. The government has a legitimate role in leading an initiative that enables mines to be greener and in leading projects that can have tangible environmental and economic benefits for Canadians.
• While indicative only, the vast majority of respondents (15 respondents) indicated the GC should be supporting the GMI. While three individuals indicated that they did not know, there were not any “No” responses.
• The GMI is the only federal initiative that considers the entire mining lifecycle. It does so through four comprehensive pillars: Footprint Reduction, Mine Waste Management, Mine Closure and Rehabilitation, and Ecosystem Risk Management. These pillars focus on reducing the amount of waste produced in mining operations and minimizing its impact on the environment at all stages of the mine lifecycle. GMI also promotes innovation (development and deployment) in the mining sector.

Are there other parties who could deliver aspects of the GMI that are now delivered by NRCan?

• Almost all of the interviewees agreed that there might be other parties such as universities, CMIC, etc., that could deliver aspects of the GMI; however, none had the mandate, impartiality, capability or capacity to deliver the entire program.
• A majority of respondents (9 respondents) indicated that they did not know if there were other parties that could deliver the GMI. There were 5 respondents who indicated “yes” and 2 respondents who stated “no.”

3.2 Performance - Effectiveness

3.2.1 To what extent have intended outcomes been achieved as a result of the GMI?

Level and Outcome	Comment	Assessment
Immediate Outcome: Expanded knowledge base of innovative mining technologies and practices that eliminate or reduce environmental impacts and risks	Achieved to a certain extent with stakeholders that have either participated in the project(s), attended workshops, etc. The reach of the expanded knowledge was not systematically captured by the program and as such is difficult to assess.	Partially Demonstrated
Immediate Outcome: Expanded understanding of regulatory, stakeholder and environmental concerns and requirements	A better understanding of the regulatory environment and barriers has been fostered by GMIFootnote 27. Some understanding of environmental concerns has been achieved from the Mines Minister Conference and liaison with Industry.	Partially Demonstrated
Immediate Outcome: Expanded understanding of Canada’s Green Mining Initiative	Communication of results has in certain instances been good, especially at the project level. However, communication of the “program” outcomes requires improvement. Results have been communicated to industry or subsets of industry.	Partially Demonstrated
Intermediate Outcome: Adoption of green mining innovation technologies leading to commercialization	Some progress has been demonstrated toward the achievement of this outcome. Some green mining innovation technologies have led to commercialization, however their scope and reach could not be quantified.	Opportunity for Improvement
Intermediate Outcome: Greater understanding by stakeholders of Canada’s commitment to environmentally sound and economically viable mining practices	From a stakeholder perspective, GMI has increased stakeholders’ understanding of environmentally sound and economically viable mining practices.	Opportunity for Improvement
Ultimate Outcome: An innovative, environmentally and socially conscious industry	It is too early to measure results.	Too early to measure
Ultimate Outcome: Positive public perceptions and image of the industry	It is too early to measure results.	Too early to measured

Analysis:

Immediate Outcome: To what extent has GMI led to an expanded knowledge base of innovative mining technologies and practices that eliminate or reduce environmental impacts and risks?

All completed projects have been congruent with pillars/priorities, more specifically:

• GMI has carried out a number of projects that led to an expanded knowledge base of innovative mining technologies and practices. This knowledge has been shared through a number of venues, workshops, publications and face-to-face interaction with stakeholders.
• There are a number of GMI projects that have led to an expanded knowledge base of innovative mining technologies and practices that eliminate or reduce environmental impacts and risks including:
  • Green Mining Green Energy; MEND; bio-waste for remediation of mine tailings; ventilation technologies; alternative vehicles for underground; etc.
• Three survey respondents indicated that GMI somewhat/mostly had led to an expanded knowledge base of innovative mining technologies and practices that eliminated or reduced environmental impacts and risks.
• The case studies revealed that GMI should or had led to an expanded knowledge base of innovative mining technologies and practices that eliminated or reduced the environmental impacts and risks. Stakeholders and workshop attendees indicated that they benefited from the information produced by GMI.
• While performance information is being collected at the project level, it is not linked to a larger, GMI-level repository that can allow the assessment of the extent to which GMI has led to an expanded knowledge base.

Immediate Outcome 2: To what extent has the GMI fostered an expanded understanding of regulatory, stakeholder and environmental issues, barriers and requirements

• GMI has carried out activities, Delphi Group 2010 and Meyers Norris Penny 2011), to determine the state of green mining technology in the Canadian mining sector. These studies identified barriers to the uptake of green mining innovations that enhanced the understanding of regulatory issues that need to be addressed.
• The GMI Working Group of the Energy Mines Ministers Conference has contributed to providing an understanding of stakeholder issues and requirements.
• The GMI has engaged stakeholders in workshops to help address barriers to the adoption of green mining technologies and practices in the Canadian mining sector.
• Five survey respondents indicated that GMI somewhat/mostly led to an expanded understanding of regulatory, stakeholder and environmental concerns and requirements.

Immediate Outcome 3: To what extent has the GMI been effective in expanding awareness of green mining?

Communication of results has in certain instances been good, especially at the project level. However, communication of the program outcomes has been lacking. Results have been communicated or targeted to industry or subsets of industry.

• At the project level, GMI has expanded the awareness of green mining among a targeted set of stakeholders (e.g., federal, provincial and territorial Energy and Mines Ministers, leaders from industry, academia, provinces and territories, NGOs, and some federal departments such as Environment Canada (EC) and Fisheries and Oceans Canada (DFO)) through presentations, workshops, etc.
• From a program perspective, initial information materials about the pillars contributed to expanding awareness of green mining. However, it was unclear whether similar materials pertaining to priorities were disseminated.
• Although indicative only, five survey respondents indicated that GMI was somewhat/mostly effective in expanding awareness of green mining.

Intermediate and Ultimate Outcomes

At its inception, the GMI did not establish a performance management framework that would allow for the design of a comprehensive logic model and performance indicators and a related data collection strategy. While a logic model was created for the purposes of evaluation, the data to support an assessment of intermediate and ultimate outcomes were challenging to identify.

• The GMI has been managed as an initiative that has been more focused on the short-term project-based outcomes, rather than longer-term program outcomes. While it is acknowledged that most, if not all, activities link to pillars and priorities, linking outputs to intermediate or ultimate outcomes presents a greater challenge.
• Intermediate Outcome: Adoption of green mining innovation technologies leading to commercialization: Some green mining innovation technologies have led to adoption and commercialization. However, it should be noted that while NRCan has an Intellectual Property Office with some patents and licenses, GMI does not appear to be actively engaged in commercialization and appears to be more focused on publication and research.
• Intermediate Outcome: Greater understanding by stakeholders of Canada’s commitment to environmentally sound and economically viable mining practices: The GMI itself is a demonstration of commitment to environmentally sound and economically viable mining practices. Among its direct partners and contributors, it thus demonstrates a greater understanding of these issues. Beyond this group within the mining sector as a whole, it is difficult to comment on the extent that GMI has specifically contributed to this increased understanding. Attribution for this level of outcome will always be difficult to obtain, though the continued involvement of GMI partners in the Initiative does show that some progress has been made.
• Ultimate Outcome: An innovative, environmentally and socially conscious industry: Some interviewees mentioned that while these objectives were desirable, economic imperatives and implementation concerns (e.g., regulatory, risk of failure, costs, etc.) affected the implementation of novel solutions. The program would need to collect data that would validate the achievement of this ultimate outcome. It was deemed too early to measure results.
• Ultimate Outcome: Positive public perceptions and image of the industry: The evaluation of this ultimate outcome would require consultation with the public as well as non-government organizations representing the environment (directly and/or indirectly) over a long-time scale to set baselines and find attributable progress as a result of GMI activities. It was deemed too early to measure results based on the timeframe required for industry to validate and endorse new technologies/practices.

3.2.2 Have there been any unintended (positive or negative) outcomes?

Analysis:

No positive or negative unintended outcomes were identified by the evaluation through the lines of evidence.

3.2.3 What are the chief internal and external factors influencing achievement of the GMI program objectives?

Analysis:

A number of internal and external barriers^{Footnote 28} to implementing GMI program objectives were observed and noted from the lines of evidence. These barriers focused on the following three themes:

1. Regulatory inefficiencies

Regulatory issues: As noted in the documents as well as corroborated by interviews, duplication and inconsistencies lead to confusion and uncertainty in issues such as lack of federal-provincial-territorial harmonization; lack of clear, consistent interpretation guidelines; unclear data requirements; some plans/agreements required prior to completed assessments; and regulatory standards reflective only of traditional, polluting technology;

Lack of knowledge among regulatory stakeholders: Interviewees noted that a significant number of regulators lacked awareness, knowledge, and understanding of the latest and evolving technologies and were reluctant to interpret regulations to factor in this evolution;

Interviewees noted that regulators were reluctant to accept alternative validation practices for new technologies/practices leading to cumbersome validation requirements, as well as expensive, time consuming pilots ; and

First to be second: Interviewees noted that companies were interested to be “first to be second” in innovation, as this tended to be proven technology; smoother regulatory approval process; fewer costs, delays, and risks.

2. Lack of Communications and Knowledge Transfer

According to the literature review^{Footnote 29}, there have been some challenges surrounding communications and knowledge transfer. It should be noted that GMI has placed considerable emphasis on conducting knowledge exchange and communications with project stakeholders and has also participated in a number of exchange fora to convey and transfer knowledge. While some interviewees noted that GMI had been successful in sharing, others sought more visibility on the scope and extent of discoveries at a program level, which in turn could assist in identifying relevant knowledge. Two areas of improvement highlighted focused on green mining itself as well as the availability of government support.

Green mining: Lack of awareness, communication, knowledge transfer, and understanding of green mining among stakeholders (including financial institutions/markets), impeding innovation efforts and technology adoption. In particular, interviewees noted a lack of awareness and knowledge regarding new developments (e.g., industry members not aware of new technologies if they were not directly involved in a project).

Awareness of Government support: Some interviewees observed that it was likely that a number of mining stakeholders might not be aware of the GMI and Canada’s progress in green mining leadership and global competitiveness and productivity. There could be value in integrating efforts of other government departments.

3. Financial Barriers

The last category of barriers noted in literature and commented on by some interviewees pertained to the financial risks and limited financial resources to trigger adoption of nascent technologies.

High financial risks: Mining is a capital and energy-intensive industry operating in a high risk environment, which can lead to slow changes (e.g., need to maintain compliance, financial performance, reputation and relationships with Government and communities; more interest in adopting new technologies if proven to be substantially better). As such, industry is cautious to embark on absorbing innovation costs that will take a significant amount of time to recover through increased productivity or reduced operating costs. The payback period is a crucial consideration that affects the decision to adopt or not adopt proposed technologies.

Limited financial resources: Limited financial resources in a current climate of uncertainty and depressed ore prices mean that companies must prioritize their investments, thus hindering the adoption of “unproven” technologies.

Lack of effective financial incentives: The literature review highlighted that fiscal and other incentives were viewed as being too low to entice investments and steer Canada towards green mining leadership and continued global competitiveness. One interviewee noted that despite the economic significance of mining across Canada, including Northern communities, there is limited Government funding available to the industry compared to other sectors.

As such, there are real and perceived barriers to the adoption of green mining technologies. The program is a vehicle that can alleviate these barriers by forging ahead with state of the art green mining projects intended to increased efficiency and reduce operating costs/footprint.

3.3 Performance - Efficiency

3.3.1 Is the GMI program the most economic and efficient means of achieving outputs and progress toward outcomes?

Analysis:

What is the difference between planned and actual spending? What is the explanation for this difference?

There was little variance between planned and actual spending between 2010-11 and 2012-13 which suggested that good financial management was in place during those years. In 2009-10, the significant difference between planned and actual spending was attributable to changes in the PAA structure of CanmetMINING which created some inconsistencies as to what was included under GMI.

Year	Planned	Actual
2009-10	$4,858,000	$7,262,720
2010-11	$8,290.000	$8,293,694
2011-12	$11,699.000	$11,420,276
2012-13	$10,948.000	$10,574,160

What steps has the GMI taken in order to minimize the use of resources in the achievement of results?

The GMI has emphasized the practice of leveraging funds and has received in-kind resources to achieve project results. While it was noted that Green Mines Green Energy (GMGE) project leveraging was 7 to 1, most of the NRCan interviewees were familiar with the practice of leveraging funds and often quoted an average of 6 to 1 leveraging. The document review found evidence of leveraged and in-kind funding. For instance, in 2011 it was reported that research funding totalled $9.2 M. Of that, there were $6 M in leveraged funds (65%), with a MMSL contribution of $3.2 M. GMI projects are able to attract outside resources thus enabling more work to be completed for the same amount of NRCan resources.

What, if anything, about the GMI needs to be adjusted?

• The document review highlighted the following areas of adjustment::
  • The need for more financial resources and strategic partnerships;
  • The need to enhance/establish partnerships with regulators;
  • The need to communicate more effectively with all stakeholders, particularly regulators who may not be familiar with the mining industry;
  • The need to consider options for networking and maintaining contact with the industry given the current budgetary constraints; and
  • The need to address the issue of replacing departed Highly Qualified People (HQP) and/or options for delivering the program.
• Interviewees suggested:
  • Fewer priorities;
  • Better communication of the good work internally and externally;
  • Refining the plan that the GMI is measured against and identifying annual KPIs, milestones, etc.; and
  • Refining the prioritization of effort/funding based on the relative importance of the pillars.
• When asked if there was anything about the GMI program they would like to change, six respondents stated “yes.” Some of the respondents provided ideas for change including using the CMIC “as a delivery vehicle with increased industry participation.” It was also suggested that the GMI should try to “achieve a higher profile” and “seek more industry involvement.”

Among the adjustments identified above, a consistent theme was the need to communicate more effectively with all stakeholders, particularly regulators (and also citizens), who might not be familiar with the mining industry. Also, there was a need to refine the plan that the GMI was measured against (e.g., identify annual KPIs, milestones etc.).

To what extent have the GMI outputs been produced and at what cost?

The GMI is producing project outputs; however, dedicated information on the exact quantity for all years of the evaluation period was not available at the program level. Interviewees and NRCan Program Researchers were in agreement that GMI outputs had been produced at the individual project level. As for the cost at which the outputs were produced, these were in accordance with project budgets allocated. It is important to note that while the evaluation question focuses on the GMI as a program (pillars/priorities), the vast majority of answers/perspectives were provided at the project level.

3.3.2 Is performance information being collected and used on an ongoing basis?

Analysis:

To what degree is GMI performance information used for decision-making by the GMI program?

The general view among NRCan Senior Management interviewees is that GMI performance information is being collected and used on an ongoing basis for decision-making. One NRCan Senior Management interviewee noted, “GMI has been recognized for having good performance measurement information and data. This information is needed to make multi-million dollar decisions.”

It was unanimously agreed among NRCan Senior Management interviewees that the Dashboard^{Footnote 30} was an extremely valuable tool which enabled them to monitor and track projects daily. Based on this information, “the program is always adjusting. It is web-based and enables ongoing departmental reporting.” The CanmetMINING Business Office staff indicated that the Dashboard was developed internally and had been up and running for almost a year and a-half. They described it as “work-in-progress” and noted that it did not track in-kind contributions.

While there were tools in place to collect performance information, namely the Dashboard, a performance management framework was not established from the commencement of the GMI. While it was observed that performance and budgets were closely monitored at the project level, the collection and oversight of program level performance information were problematic. It was observed that the information reported on the Dashboard was not GMI-specific and it would have been time consuming to attempt to extract program level performance information. While a logic model was derived during the course of this evaluation, performance data and a related data collection strategy were not implemented at the beginning of the GMI, making it very difficult for the program to “reconstruct” the GMI performance story.

Performance information was collected and used at the project level as noted. Individual projects were presented at mid-year and end-of-year assessments, and outputs were tracked at the project level. Files reviewed for the case studies indicated that projects were managed individually, and project files assembled based on the initiative and leadership of the respective project managers. Some project files contained a fair amount of performance information, whereas others contained less.

Overall, from a program standpoint, the absence of a clear GMI program logic model and related performance indicators mean that the program is managed at the project level. The Dashboard appears to be a step in the right direction.

4.0 Conclusions and Recommendations

4.1 Conclusions

Relevance

• There is a need and role for the federal government in Green Mining R&D and S&T, and the focus of the work carried out under GMI is aligned with GC priorities and mandate.

Effectiveness

• There are indications that the GMI is producing industry-relevant outputs. While a significant level of activities and projects are carried out, it is difficult to determine the extent to which program effectiveness has been achieved as results/outputs tend to be linked to projects, rather than pillars or priorities.
• There is a lack of performance data to measure the achievement of outcomes at the program level.
• While there is some evidence of industry adoption, the GMI does not track or report on uptake. It is recognized that industry adoption may take a significant amount of time and that companies are not compelled to report uptake or attribute uptake to the GMI.
• Some green mining innovation technologies have led to commercialization. However, it should be noted that while NRCan has an Intellectual Property Office, with some patents and licenses, it does not appear that GMI is active in commercialization and appears to be more focused on publication and research.
• It was expected that the extent to which outcomes had been achieved would have been tracked by “the dashboard.” However, it appeared that while “the dashboard” can track CanmetMINING information, the systems could not single out GMI-specific information.

Efficiency and Economy

• While significant effort has been devoted to collecting broader CanmetMINING information, there does not appear to be a standard for data collection, performance information or metrics across projects. Information was not collated in terms of percentage of completion or progress reports that outlined the status of outcomes. The link between project deliverables and project outcomes in relation to the GMI program was not established when the project was submitted. Reporting of performance should be at the program level as well as at the individual project level.
• From a project documentation standpoint, files could be organized in terms of project intent (objective), proposed resourcing level, performance measurement and link to performance management framework, project milestones and level of effort (schedule), proposed project outcomes, results (expected vs. achieved), lessons learned and deliverables. This approach would greatly assist in program management and evaluation.

4.2 Recommendations

Recommendation 1: Opportunities exist to engage other federal entities as the research conducted under the GMI extends across several federal initiatives. To take full advantage of this, it is recommended that GMI conduct a review of federal programs to identify potential efficiencies and areas for collaboration.

Given the breadth and range of research topics and partners within GMI projects, there is an opportunity to work with other federal entities [Environment Canada, (EC), Industry Canada (IC), Department of Foreign Affairs Trade and Development (DFATD) etc.] to achieve larger joint objectives (e.g., trade, innovation). Collaboration for mutual benefit also would increase the potential to attract more resources to the Program. Such a review should aim to ensure that the outcomes identified for GMI are not being fulfilled by others and vice versa and/or that support is better leveraged /coordinated. There are benefits of performing a horizontal review, namely:

• identifying shared objectives and leveraging funding and resources more effectively among federal entities;
• Enhancing program delivery effectiveness of respective partners; and
• Facilitating the measurement of results for multiple entities by delivering coordinated support to a concrete project.

Recommendation 2: Given the evidence highlighting the need to attract new partners through clearly articulated program priorities, it is recommended that GMI establish an overarching internal green mining research agenda that will guide long-term priority setting.

One of GMI’s strengths is the systematic tracking of outputs at the project level. The evaluation found however that there were issues with tracking at the higher program/initiative level. While CanmetMINING has been successful in managing and tracking below and above this level, without information specific to GMI it becomes difficult to tell the performance story of the Initiative, to highlight successes, to attract new partners and to develop unified program level objectives. The Initiative should consider establishing a research agenda, akin to Canadian Safety and Security Program (CSSP) Call for Proposals,^{Footnote 31} for innovative science and technology (S&T) projects. A comprehensive Call for Proposal program allows an entity to:

• Identify capability deficiencies, gaps and priorities with the coordinated assistance from stakeholders;
• Set the strategic course of projects and activities that best address capability gaps;
• Identify program priorities to guide investment decision making;
• Align program investments to strategic objectives;
• Enhance tracking and monitoring of progress against identified gaps and expected outcomes; and
• Facilitate the identification of leveraging opportunities among federal partners.

Recommendation 3: Given that no GMI-specific performance measurement and reporting strategy exists, it is recommended that a performance measurement and reporting strategy for the GMI be developed by the Program to systematically track progress on outputs and outcomes and ensure consistent performance data are collected across projects. This PMRS should include a logic model and associated narrative.

A performance measurement strategy should be established to systematically track progress on outputs and outcomes. Technical projects (including deliverables and outcomes) should be appropriately linked to the pillars and priorities and consistent performance data collected for these. For specific GMI projects, each project file should include a clear statement as to the linkage and importance of the proposed work to pillars/priorities, and should highlight the differentiating input/contribution that NRCan provides to the project.

Annex A: Natural Resources Canada Program Alignment Architecture 2013-2014

As depicted in the NRCAN 2013-2014 Strategic Outcomes and Program Alignment Architecture/Program Activity Architecture (PAA), for the Mining and Minerals Sector (MMS), GMI is currently situated as follows:

Text version

2013-2014 Strategic Outcomes and Program Alignment Architecture/Program Activity Architecture

Strategic Outcome 1: Canada’s Natural Resource Sectors are Globally Competitive

1.1 Market Access and Diversification
1.1.1 Mineral and Metal Markets Access and Development
1.1.2 Forest Products Market Access and Development
1.1.3 Energy Market Regulation and Information

1.2 Innovation for New Products and Processes
1.2.1 Mining Innovation
1.2.2 Forest Sector Innovation
1.2.3 Geomatics Innovation

1.3 Investment in Natural Resource Sectors
1.3.1 Mineral Investment
1.3.2 Forest-based Community Partnerships
1.3.3 Targeted Geoscience Initiative 4 (TGI4)
1.3.4 Geo-mapping for Energy and Minerals
1.3.5 New Energy Supply
1.3.6 Major Projects Management Office Initiative

1.4 Statutory Programs – Atlantic Offshore

Strategic Outcome 2: Natural Resource Sectors and Consumers are Environmentally Responsible

2.1 Energy-efficient Practices and Lower-carbon Energy Sources
2.1.1 Renewable Energy Deployment
2.1.2 Support for Clean Energy Decision-making
2.1.3 Alternative Transportation Fuels
2.1.4 Energy Efficiency

2.2 Technology Innovation
2.2.1 Materials for Energy
2.2.2 Green Mining
2.2.3 Clean Energy Science and Technology

2.3 Responsible Natural Resource Management
2.3.1 Forest Ecosystems Science and Application
2.3.2 Groundwater Geoscience
2.3.3 Environmental Studies and Assessments
2.3.4 Radioactive waste management
2.3.5 Earth Observations for Responsible Development of Natural Resources

Strategic Outcome 3: Canadians have Information to Manage their Lands and Natural Resources, and are Protected from Related Risks

3.1 Protection for Canadians and Natural Resources
3.1.1 Explosives Safety and Security
3.1.2 Materials and Certification for Safety and Security
3.1.3 Forest Disturbances Science and Application
3.1.4 Climate Change Adaptation
3.1.5 Geohazards and Public Safety

3.2 Landmass Information
3.2.1 Essential Geographic Information
3.2.2 Canada’s Legal Boundaries
3.2.3 Polar Continental Shelf Logistics Support

The expected results from the Performance Measurement Framework (PMF) are provided in Table 5.

Table 5: Performance Measurement Strategy – Expected Results for GMI^{Footnote 32}

PAA Level	Order	Program Name	Expected Results
Strategic Outcome	2	Natural Resource Sectors and Consumers are Environmentally Responsible
Program	2.2	Technological Innovation	Stakeholders invest in S&T to address environmental challenges
Sub-Program	2.2.2	Green Mining	Technology developers increase demonstration of environmental technologies Industry partners increase financial and in-kind contributions Academic, government and other non-industry partners increase financial and in-kind contributions

Annex B: Evaluation Matrix

Evaluation Question	Evaluation Sub-Question	Indicator	Lines of Evidence	Data Sources
Relevance
Evaluation Issue 1: Continued Need for Program
R1. Is there a continuing need for the GMI?	R1 (a). What needs are program activities addressing?	Identification of stakeholder needs addressed by GMI program.	Document review	Policy, program documents, project files (e.g., expression of interests, Council of Ministers of Mines and Energy Papers, committee meeting minutes etc.)
			Interviews	Interviews NRCan Senior and Program Managers, Program Researchers, OGDs, Industry, Academia, Other Jurisdictions and Interest Groups.
			Survey	Industry respondents
	R1 (b). Are the needs continuing?	Consistency (comparison) of stakeholder needs with GMI (pillar/ priority) activities, outputs and outcomes. Expressed continued support/value (leveraged funds etc.) for the program by stakeholders.	Document review	Policy, program documents, project files, requests/ suggestions from industry, requests for adjustment
			Interviews	Interviews NRCan Senior and Program Managers, Program Researchers, OGDs, Industry, Academia, Other Jurisdictions and Interest Groups.
			Survey	Industry respondents
	R1 (c). Has the GMI adapted to meet new or changing needs of stakeholders?	Identification of new stakeholder needs/foci that have arisen between 2009 and 2013.	Document review	Policy, program documents, project files, requests/ suggestions from industry, requests for adjustment
			Interviews	Interviews NRCan Senior and Program Managers, Program Researchers, OGDs, Industry,
Evaluation Issue 2: Alignment with Government Priorities
R2. Is the GMI program and activities consistent with government priorities and NRCan strategic objectives?	R2 (a). To what extent is the GMI linked to federal government priorities?	Consistency of activities, outputs and outcomes with federal priorities.	Document review	Federal level documents (Speech from the Throne, GoC direction etc.)
			Interviews	Interviews with OGDs and Industry
	R2 (b). To what strategic departmental objectives and priorities are the GMI activities, outputs, outcomes linked?	Consistency of GMI activities outputs and outcomes with NRCan and MMS Strategic Outcomes and priorities.	Document review	PAA, RPP, PMF as well as MM’s Strategic Outcomes and Priorities
			Interviews	Interviews NRCan Senior and Program Managers, Program Researchers,
Evaluation Issue 3: Alignment with Federal Roles and Responsibilities
R3. Is there a legitimate, appropriate and necessary role for the federal government in the GMI program?	R3 (a). Should the Government of Canada be involved in Green Mining, S&T and R&D?	Consistency of GMI activities, outputs and outcomes with federal legislation/role. Consistency of GMI activities, outputs and outcomes in relation to other international government programs.	Document Review	NRCan Mandate, RPP, PAA
			Interviews	Interviews NRCan Senior and Program Managers, Program Researchers, OGDs,
			Case studies	Interviews, project files, literature on similar international programs from key countries
	R3 (b). Are there other parties who could deliver aspects of the GMI that are now delivered by NRCan?	Identification of the extent to which other parties have the mandate, capability/capacity to deliver on the GMI pillars/priorities.	Document review	OGD mandates and descriptors
			Interviews	Interviews NRCan Senior and Program Managers, Program Researchers, OGDs, Industry, Academia, Other Jurisdictions and Interest Groups.
			Survey	Industry Survey
			Case studies	Project files and interviews
Performance – Effectiveness
Evaluation Issue 4: Achievement of Expected Outcomes
PS1. To what extent have intended outcomes been achieved as a result of the GMI?	PS1 (a). To what extent has GMI led to an expanded knowledge base of innovative mining technologies and practices that eliminate or reduce environmental impacts and risks?	Number and scope of GMI generated activities per pillar/priority focused on eliminating or reducing environmental impacts and risks (new scientific knowledge, new technologies, best practices, new processes, pilots, reference materials, validated approaches, partnerships and collaborations). Number, scope and reach of sharing mechanisms implemented to share/ develop innovations, best practices etc.	Document review	Project files, outcomes reports, reviews, management reports
			Interview	Interviews NRCan Senior and Program Managers, Program Researchers
			Survey	Industry Survey
			Case studies	Project files and interviews
	PS1 (b). To what extent has the GMI fostered an expanded understanding of regulatory, stakeholder and environmental issues, barriers and requirements	Number and scope of stakeholder engagement mechanisms established. Number and scope of regulatory reviews and consultation activities to identify and address green mining issues and barriers performed.	Document review	Consultant report, terms of reference, reports
			Interviews	OGDs, Industry, Academia, Other Jurisdictions and Interest Groups
			Survey	Industry survey
	PS1 (c). To what extent has the GMI been effective in expanding awareness of green mining?	Number and scope of information products generated. Number of stakeholder and interest group liaison and green mining information sharing mechanisms established.	Document review	Communication material, web postings, listings, program files
			Interviews	Interviews NRCan Senior and Program Managers, Program Researchers, OGDs, Industry, Academia, Other Jurisdictions and Interest Groups.
			Survey	Industry Survey
PS2. Have there been unintended (positive or negative) outcomes?	PS2 (a). Identify the unintended outcomes.	Identification of the unintended outcomes, if any, of the GMI program.	Document review	Program files, evaluations, minutes and correspondence
			Interviews	Interviews NRCan Senior and Program Managers, Program Researchers, OGDs, Industry
PS3. What are the factors (both internal and external) that have facilitated or hindered the achievement of expected results?	PS3 (a). To what extent factors within or beyond the GMI program outputs contributed/hindered outcomes.	Identification of enabling factors and barriers to achievement of GMI outcomes.	Document review	Program files, committee minutes, consultant reports
			Interviews	Interviews NRCan Senior and Program Managers, Program Researchers, OGDs, Industry, Academia, Other Jurisdictions and Interest Groups.
			Survey	Industry Survey
Performance – Efficiency
Evaluation Issue 5: Efficiency and Economy
PE1. Is the GMI program the most economic and efficient means of achieving outputs and progress toward outcomes?	PE1 (a). What is the difference between planned and actual spending? What is the explanation for this difference,	Comparison of planned and actual spending.	Document review	Budget files, RPP, planned vs. actual
			Interviews	Interviews NRCan Program Researchers,
	PE1 (b). What steps has the GMI taken in order to minimize the use of resources in the achievement of results?	Identification of efficiency, cost saving measures and cost recovery measures. Identification of leveraged and in-kind funding.	Document review	Program cost sharing and leveraging files, committee minutes, consultant reports
			Interviews	Interviews NRCan Senior and Program Managers, Program Researchers
	PE1 (c). What, if anything, about the GMI needs to be adjusted?	Identification of unmet needs, undesired outcomes, and design/delivery needs	Document review	Stakeholder correspondence, meeting minutes
			Interviews	Interviews NRCan Senior and Program Managers and Industry
	PE1 (d). To what extent have the GMI outputs been produced and at what cost?	Comparison between program-anticipated outputs and actual outputs. Approximate cost of production of outputs, as available.	Document review	Planned vs. actual budget information
			Interviews	Interviews NRCan Program Researchers
PE2. Is performance information being collected and used on an ongoing basis?	PE2 (a). To what degree is GMI performance information used for decision-making by the GMI program?	Identification of how GMI performance information is used for decision-making.	Document review	Dashboard, PMF, evidence of indicator-based decisions
			Interviews	Interviews NRCan Senior and Program Managers