Evaluation of the Mining Innovation Sub-program

Evaluation Division
Audit and Evaluation Branch
Natural Resources Canada

July 2018

Table of Contents

• Acronyms and Abbreviations
• Executive Summary
• Acknowledgements
• 1.0 Introduction and Background
  • 1.1 Overview
  • 1.2 Background
  • 1.3 Sub-program Description
  • 1.4 Sub-program Resources
• 2.0 Evaluation Approach and Methods
  • 2.1 Evaluation Scope and Objectives
  • 2.2 Evaluation Methods
  • 2.3 Evaluation Limitations and Mitigation Strategies
• 3.0 Evaluation Findings: Relevance
  • 3.1 Ongoing and Continued Need for the Sub-program – Demonstrated
  • 3.2 Alignment with Federal Government Priorities and NRCan Strategic Objectives – Demonstrated
  • 3.3 Legitimacy and Appropriateness of Federal Role – Demonstrated
• 4.0 Evaluation Findings: Performance
  • 4.1 Efficiency and Economy – Partially Demonstrated
  • 4.2 Effectiveness – Partially Demonstrated
  • 4.3 Factors Impacting the Achievement of Outcomes
• 5.0 Conclusions and Recommendations
  • 5.1 Conclusions
  • 5.2 Recommendations
• Annex A: Mining Innovation Sub-program Logic Model
• Bibliography

Acronyms and Abbreviations

CCRMP	Canadian Certified Reference Materials Project
CEMI	Centre of Excellence in Mining Innovation
CMIC	Canada Mining Innovation Council
CREEN	Canadian Rare Earth Elements Network
CRM	Certified Reference Materials
EMMC	Energy and Mines Ministers’ Conference
GMI	Green Mining Initiative
GMIAC	Green Mining Initiative Advisory Committee
GMI-IGWG	Green Mining Initiative Inter-governmental Working Group
IBP	Integrated Business Plan
IEC	International Electrotechnical Commission
IP	Intellectual Property
ISO	International Organization for Standardization
LMS	Land and Minerals Sector
MAC	Mining Association of Canada
NRCan	Natural Resources Canada
PAA	Program Alignment Architecture
PTP-MAL	Proficiency Testing Program for Mineral for Analysis Laboratories
R&D	Research and Development
RDI	Research, Development and Innovation
REE	Rare Earth Elements
S&T	Science and Technology
SCC	Standards Council of Canada
VNR	Vote-netted revenue

Executive Summary

Introduction

This report presents the findings of the evaluation of Natural Resources Canada’s (NRCan’s) Mining Innovation sub-program. This sub-program is managed by CanmetMINING, a federal government science and technology (S&T) laboratory under the oversight of NRCan’s Lands and Minerals Sector (LMS). The evaluation covers approximately $26.7 million in NRCan expenditures over the five-year period of 2010-11 to 2014-15.

The purpose of the evaluation is to assess the relevance and performance (i.e., effectiveness, efficiency and economy) of the Mining Innovation sub-program using a mix of qualitative and quantitative methods, including reviews of program files, documents, literature, interviews and case studies.

Sub-program Overview

The objective of the Mining Innovation sub-program is to contribute to the innovation and global competitiveness of the Canadian mining industry by improving its productivity through the adoption of innovative techniques and technology. In particular, the sub-program aims to support innovation in the mining industry by:

• Researching and developing new mining-related tools, technology, methods, and protocols;
• Disseminating knowledge and research to stakeholders (including regulations);
• Conducting pilot tests related to new mining-related processes;
• Collaborating with provinces and territories, standard bodies (e.g., on regulations), and other stakeholders;
• Promoting the need to be innovative in mining processes to industry stakeholders; and
• Providing specialized services to clients (e.g., testing, analyzing samples, validating systems, and certifications).

Activities carried out by the sub-program are delivered in four project areas: (1) Mining Extraction Innovation; (2) Mineral and Metallurgical Processing; (3) Canadian Certified Reference Materials Project (CCRMP); and (4) Proficiency Testing Program for Mineral Analysis Laboratories (PTP-MAL).

Evaluation Findings: Relevance

The evaluation identified an ongoing need for: support and collaborative approaches for R&D; delivery of specialized services (i.e., conformity assessment services, laboratory proficiency testing); and coordination of mining innovation efforts at the national-level. The Mining Innovation sub-program aligns with federal government priorities and NRCan’s strategic outcomes, as well as with federal roles and responsibilities, including NRCan’s mandate. The large number of stakeholders and organizations involved in mining innovation has led to some confusion among stakeholders as to the role of the sub-program in the area of innovation in mining.

Evaluation Findings: Performance (Efficiency and Economy)

The Mining Innovation sub-program employs practices that facilitate the efficient and economic production of outputs and contribution to intended outcomes. A key strength is its collaboration with multiple stakeholders, which helps reduce the risk of duplication of effort, facilitates the sharing of expertise, and helps the sub-program identify the needs of the mining industry related to innovation. At the time of the evaluation, CanmetMINING also maintained multiple accreditations, which increase its reliability and enable it to deliver specialized services (i.e., CCRMP and PTP-MAL). These accreditations also ensure a level of consistency in project management.

A number of stakeholders suggested the sub-program could build on its foundation of collaboration and expertise to strengthen its leadership role by, developing a national strategy on mining innovation to better focus innovation R&D in the mining sector. In addition, while the use of VNR affords the flexibility to respond to client demand for services, there is an opportunity for the sub-program to work towards stabilizing its funding to better position itself to achieve sub-program outcomes. In particular, sub-program planning is sometimes done at the project-level based on client needs or requests rather than at a more strategic-level that would address broader policy objectives.

Similarly, performance measurement was generally done at the level of individual projects and according to the ISO 9001 project guide methodology, rather than for the sub-program as a whole. This presents a potential challenge to the sub-program to ensure it is delivered effectively, efficiently, and economically.

Evaluation Findings: Performance (Effectiveness)

The sub-program is making progress towards achieving its four intended immediate outcomes: (1) reduced risks associated with adopting new techniques and technologies; (2) increased access to relevant markets for industry stakeholders; (3) increased awareness and understanding of innovative technologies by industry stakeholders; and (4) increased understanding of industry’s needs, priorities, concerns, and requirements.

It was also possible to align project-level information with the two intended intermediate outcomes: (1) adoption of mining innovation (techniques, technologies, etc.) by industry stakeholders; and (2) improved productivity (including health and safety components) of the industry. Internal and external stakeholders, however, raised the concern that it may be beyond the reach of the sub-program to influence industry productivity.

In addition, the evaluation identified some factors that could positively or negatively impact the ability of the sub-program to achieve its intended outcomes, including: changing commodity prices, which could impact the ability or willingness of stakeholders to invest in new or innovative techniques or related R&D; practices of other countries (e.g., tendency towards protectionism in China and India) or investments in other sectors (e.g., investment in innovation in the Canadian Mining industry is lower than other sectors such as forestry, automotive, and manufacturing); financial (e.g., return on investment), and non-financial (e.g., organizational culture change) risks that industry may be unwilling to assume; potential constraints associated with non-disclosure/intellectual property (IP) agreements; and level of collaboration among stakeholders, which is important to facilitate outcome achievement.

Recommendations and Management Response: Based on the findings and conclusions of the evaluation, three recommendations are made to the Assistant Deputy Minister of LMS. A management response and an action plan to all three recommendations have been formulated.

Recommendation 1: LMS should identify and communicate to stakeholders the roles of all parties relevant to mining innovation, and work with appropriate partners to help identify priorities for R&D related to innovation in the Canadian mining sector.
Management Response and Action Plan	Responsible Manager, Division, Branch	Target Date
Management concurs with this recommendation. LMS: Developed with industry stakeholders a national strategy to collaborate on mining innovation  (completed June 2017); Sought endorsement of the national strategy from Energy Mines Ministers Conference/EMMC (endorsement received August 2017) Will promote, communicate and implement the accepted national strategy in collaboration with stakeholders (October 2018) Will monitor through the Assistance of Mining Innovation Site the adoption by stakeholders of the new national strategy (on-going as of October 2018) In collaboration with the Canada Mining Innovation Council, will develop technology roadmaps on key technological challenges of the Mining industry (March 2020)	DG CanmetMINING, LMS	March 2020

Recommendation 2: LMS should revisit the sub-program’s design and delivery with specific consideration of the appropriate balance between strategic priorities and client-specific needs. This should include exploring whether other financial mechanisms/funding models would better support program delivery.
Management Response and Action Plan	Responsible Manager, Division, Branch	Target Date
Management concurs with this recommendation. LMS: Developed a Five Year Research Plan addressing the fore coming Strategic priorities, including the Green Mining activities. The research plan has been shared with stakeholders through GMIAC presentations February 2016 and was  posted on the website (December 2016) Will implement the new NRCan Departmental Results Framework to address new ways of delivering the program and to ensure reporting on the performance measurement (April 2018) Will explore financial mechanisms in consultation with NRCan Financial Management Branch and policy experts to seek assistance and guidance on the program’s way forward (March 2019)	DG CanmetMINING, LMS	March 2019

Recommendation 3:  LMS should improve performance measurement and reporting to ensure adequate information is available to assess efficiency and effectiveness. For Mining Innovation activities, this means developing and implementing a performance measurement strategy, including: a logic model; theory of change (i.e., identification of risks, contexts, and assumptions that explain how the program is intended to be delivered and achieve its intended outcomes); as well as performance indicators, data sources, methods, and data collection and reporting frequency for all elements of the logic model (i.e., activities, outputs, immediate, intermediate, and ultimate outcomes).
Management Response and Action Plan	Responsible Manager, Division, Branch	Target Date
Management concurs with this recommendation. LMS: Taking into account the Performance Information Profile for the activities under the Green Mining Innovation, will put in place the required Performance Management Framework with key outputs and outcomes through a logic model. The framework will be aligned with the Five Year Research plan (July 2018) Will ensure that bi-annual performance science presentations to management are performed to discuss plans, variances, share success stories and  mitigation strategies (on-going)	DG CanmetMINING, LMS	September 2018

Acknowledgements

The Evaluation Team would like to thank those who contributed to the evaluation of the Mining Innovation sub-program, particularly members of the Evaluation Advisory Committee, Lands and Minerals Sector (LMS), including CanmetMINING staff, as well as others who provided insights and comments crucial to this evaluation. The evaluation project was managed by Olive Kamanyana in collaboration with Mary Kay Lamarche, Barthelemy Pierrélus and Amélie Veillette. David Ash, William Blois and Gavin Lemieux provided direction. Christian Asselin, Mark Pearson and Glenn Hargrove, provided senior management oversight. TDV Global Inc. also contributed to this evaluation.

1.0 Introduction and Background

1.1 Overview

This report presents the findings of the evaluation of Natural Resources Canada’s (NRCan’s) Mining Innovation sub-program. This sub-program is managed by CanmetMINING, a federal government science and technology (S&T) laboratory under the oversight of NRCan’s Lands and Minerals Sector (LMS). The evaluation covers approximately $26.7 million in NRCan expenditures over the five-year period of 2010-11 to 2014-15.

The purpose of the evaluation is to assess the relevance and performance (i.e., effectiveness, efficiency and economy) of the Mining Innovation sub-program, as required by the Treasury Board of Canada’s Policy on Evaluation (2009) which was rescinded and replaced with the 2016 Policy on Results. Evaluation planning and data collection occurred between April 2015 and April 2016.

1.2 Background

Canada’s geology is rich in economically viable minerals and metals deposits, which makes Canada one of the largest mining nations in the world (Mining Association of Canada. Mining facts). According to these mining facts, more than 60 minerals and metals are produced across the country, and 1,289 companies are involved in the extraction of minerals, with forecasted investments of $160 billion in the Canadian mining industry over the next 10 years. In terms of share of global production, Canada is first in potash; second in uranium, nickel and niobium; third in cobalt, aluminum and platinum group metals; fourth in salt, sulphur and tungsten; and fifth in diamonds, graphite and gold.

Canadian mining sector competitiveness is built on a strong investment climate and solid regulatory structures for mining activities. Because mining activity in Canada is primarily under the jurisdiction of provinces and territories, the Canadian mining industry is “governed by dozens of federal, provincial and territorial acts and regulations...” (Mining Association of Canada. Regulatory effectiveness). In addition to the provincial and territorial acts and regulations, there are a number of federal acts and regulations that must be adhered to, including the: Canadian Environmental Protection Act; Fisheries Act; Navigable Waters Protection Act/Navigation Protection Act; Species at Risk Act; Migratory Birds Convention Act; and Transportation of Dangerous Goods Act.

1.3 Sub-program Description

As noted above, CanmetMINING manages the Mining Innovation sub-program. CanmetMINING is a federal government research laboratory that provides research and scientific advice to the mining and minerals industries in collaboration with provincial and territorial governments and other federal government departments involved in promoting or regulating these industries. It has facilities in Ottawa, Val-d’Or, and Sudbury, where it conducts research and development (R&D) on a variety of “processes and technologies involved in extracting ore from the ground and transforming it into a concentrate, mineral product or metal”. It also has a satellite office in Smithers.

The objective of the Mining Innovation sub-program is to contribute to the innovation and global competitiveness of the Canadian mining industry by improving its productivity through the adoption of innovative techniques and technology. In particular, the sub-program aims to support innovation in the mining industry by:

• Researching and developing new mining-related tools, technology, methods, and protocols;
• Disseminating knowledge and research to stakeholders (including regulations);
• Conducting pilot tests related to new mining-related processes;
• Collaborating with provinces and territories, standard bodies (e.g., on regulations), and other stakeholders;
• Promoting the need to be innovative in mining processes to industry stakeholders; and
• Providing specialized services to clients (e.g., testing, analyzing samples, validating systems, and certifications).

Per the logic model (presented in Annex A), these activities are intended to produce the following outputs:

• Validated approaches, patents, and licences;
• New scientific knowledge, new technologies, and new processes;
• Scientific publications, demonstration projects, and information products;
• Pilot projects; and
• Regulatory scientific input, stakeholder engagement mechanisms (e.g., consultation), and communication products.

These activities and outputs, in turn, are expected to result in the following immediate, intermediate, and ultimate outcomes:

• Immediate Outcomes:
• Reduced risks associated with adopting new techniques and technologies;
• Increased access to relevant markets for industry stakeholders;
• Increased awareness and understanding of innovative technologies by industry stakeholders; and
• Increased understanding of industry’s needs, priorities, concerns, and requirements.
• Intermediate Outcomes:
• Adoption of mining innovation (techniques, technologies, etc.) by industry stakeholders; and
• Improved productivity (including health and safety components) of the mining industry.
• Ultimate Outcome:
• An innovative and globally-competitive Canadian mining industry.

To deliver on these outcomes, the Mining Innovation sub-program is delivered through four components:

1. Mining Extraction Innovation, which focuses on reducing the risk of adopting new extraction technologies and processes, as well as developing new products, particularly related to deep mining.
2. Mineral and Metallurgical Processing, which facilitates research on critical metals, rare earth elements (REE) and chromite processing. Although Canada has exploitable deposits of critical metal, REE and chromite, it has no current production.
3. Canadian Certified Reference Materials Project (CCRMP), which prepares and certifies reference materials for analytical laboratories in the mineral, metallurgical, earth science and environmental industries.
4. Proficiency Testing Program for Mineral Analysis Laboratories (PTP-MAL), which defines the metals of interest and concentration range of each, and also establishes the criteria and target values for assessing laboratory proficiency.

According to its 2013/14 to 2016/17 Integrated Business Plan (IBP), CanmetMINING’s approach to innovation is built on multi-stakeholder R&D projects to leverage resources and expertise. To that end, Innovation, Science and Economic Development Canada recognises that CanmetMINING often conducts its research “in partnership with industry, provincial governments, other federal departments, universities, and international agencies”.

1.4 Sub-program Resources

Mining Innovation sub-program activities, through CanmetMINING, are undertaken using in-house and contracted research. Contracted research could involve sharing costs or tasks between CanmetMINING and a partner organization (e.g., industry, provinces, other federal departments, universities, etc.), or cost-recovery whereby CanmetMINING completes the work (e.g., CCRMP and PTP-MAL) and recovers the cost for this from the client. Table 1 identifies expenditures for the sub-program from 2010-11 through 2014-15.

Table 1: Mining Innovation Sub-program Actual Expenditures* by Resource Type, 2010-11 to 2014-15

Expenditure Type	2010-11	2011-12	2012-13	2013-14	2014-15	Total
Mining Extraction Innovation
O&M	189,100	433,200	414,000	437,200	447,400	1,920,900
Salaries	1,119,613	1,329,800	1,059,700	1,062,000	686,121	5,257,234
EBP	223,922.60	265,960	211,940	212,400	137,224	1,051,447
G&C	-	-	-	-	-	-
Sub-total	1,532,636	2,028,960	1,685,640	1,711,600	1,270,745	8,229,581
Mineral and Metallurgical Processing
O&M	254,500	46,300	14,200	-	500	315,500
Salaries	1,120,888	1,278,800	1,018,765	1,109,800	744,200	5,272,453
EBP	224,178	25,760	203,753	221,960	148,840	1,054,491
G&C	-	-	-	-	-	-
Sub-total	1,599,566	1,580,860	1,236,718	1,331,760	893,540	6,642,444
CCRMP
O&M	22,840	47,300	39,900	38,400	31,600	180,040
Salaries	96,076	103,400	85,700	104,700	72,000	461,876
EBP	19,215	20,680	17,140	20,940	14,400	92,375
G&C	-	-	-	-	-	-
Sub-total	138,131	171,380	142,740	164,040	118,000	734,291
PTP-MAL
O&M	-	10,400	6,800	8,000	500	25,700
Salaries	33,300	25,900	21,400	28,000	19,200	127,800
EBP	6,660	5,180	4,280	5,600	3,840	22,560
G&C	-	-	-	-	-	-
Sub-total	39,960	41,480	32,480	41,600	23,540	179,060
Overhead**
O&M	1,113,609	817,330	846,577	2,218,525	768,656	5,764,697
Salaries	832,659	988,749	775,600	973,627	722,800	4,293,435
EBP	166,532	197,750	155,120	194,725	144,560	858,687
G&C	10,270	-	-	-	-	10,270
Sub-total	2,123,070	2,003,829	1,777,297	3,386,877	1,636,016	10,927,089
Sub-program Total
O&M	1,580,049	1,354,530	1,321,477	2,702,125	1,248,656	8,206,837
Salaries	3,202,536	3,726,649	2,961,165	3,278,127	2,244,321	15,412,798
EBP	640,507.60	745,330	592,233	655,625	448,864	3,082,559.60
G&C	10,270	-	-	-	-	10,270
Total	5,433,362.60	5,826,509	4,874,875	6,635,877	3,941,841	26,712,465

Notes:
* Expenditures represent total expenditures using A-base (ongoing) funds and vote-netted revenues (VNR). Ongoing A-base funds accounted for 75% of expenditures and VNR for 25%.
** Overhead includes the non-science activities that are not directly assigned to a specific component of the sub-program, such as finance, administration, Director General’s Office Health and Safety.

As noted above, the sub-program also delivers its work by sharing costs or tasks with a partner organization or by recovering costs for work completed for a client. Table 2 presents the total vote-netted revenue (VNR) generated by the sub-program from 2010-11 through 2014-15.

Table 2: Mining Innovation Sub-program VNR Generated by Sub-program Component, 2010-11 to 2014-15

Sub-program Component	2010-11	2011-12	2012-13	2013-14	2014-15	Total
Mining Extraction Innovation	688,823	755,679	1,074,373	1,153,063	561,732	4,233,671
Mineral and Metallurgical Processing	76,428	16,213	27,2370	11,211	2,706	133,795
CCRMP	995,868	1,324,415	1,458,195	1,048,879	1,119,231	5,946,588
PTP-MAL	130,436	193,477	221,636	215,905	212,757	974,211
Total	1,891,555	2,289,784	2,781,441	2,429,058	1,896,426	11,288,264

2.0 Evaluation Approach and Methods

2.1 Evaluation Scope and Objectives

The evaluation of the Mining Innovation sub-program assesses the sub-program’s relevance in terms of addressing a demonstrable need, aligning with federal government roles and responsibilities, and aligning with federal priorities and departmental strategic objectives. It also assesses the sub-program’s performance in terms of its effectiveness, efficiency and economy.

The scope of the evaluation was calibrated to reflect the fact that the Green Mining Initiative (GMI), a closely related NRCan sub-program, had been evaluated in 2015. The GMI targets the development of innovative energy efficient technologies to improve the mining sector’s environmental performance, to promote innovation in mining and to position Canada’s mining sector as the global leader in green mining technologies and practices. Both sub-programs share a management structure, including NRCan management, committees, and industry partners. As such, the evaluation examined the efficiency of this structure. Furthermore, the scope does not include an assessment of the ultimate outcome (an innovative and globally-competitive Canadian mining industry) given the long-term nature of the outcome and limited expectation of observing measurable progress that could be attributable to the sub-program.

This evaluation considered the following eight evaluation questions:

Relevance:

1. Is there an ongoing and continued need for this sub-program?
2. To what extent do the priorities of the sub-program align with federal government priorities and NRCan strategic objectives?
3. Is there a legitimate and appropriate role for the federal government to play with respect to mining innovation?

Performance (effectiveness):

4. To what extent have the intended outcomes of the sub-program been achieved?
5. To what extent have there been any unintended impacts (positive or negative)?
6. What internal and external factors impact the achievement of the intended outcomes?

Performance (efficiency and economy):

7. Is the sub-program the most efficient means of achieving the intended outputs and outcomes?
8. Is the sub-program the most economic means of achieving the intended outcomes?

The findings for each evaluation question are presented in Sections 3 and 4. For each question, a rating and summary of the supporting evidence are provided. The rating statements outlined in Table 3 assisted in ensuring consistency in reporting and facilitating the interpretation of qualitative data.

Table 3: Definition of Rating Statements

Statement	Definition
Demonstrated	Intended outcomes or goals were achieved or met.
Partially Demonstrated	Considerable progress was made to meet intended goals, and achievement is expected based on current plan.
Partially Demonstrated – Action Required	Some progress has been made to meet the intended outcomes or goals. Management attention is needed to fully achieve desired objective or result.
Not Demonstrated	Limited or no progress has been made to meet the intended outcomes or goals as stated, or the outcome is no longer applicable (due to changes in the external environment).

2.2 Evaluation Methods

This evaluation used multiple lines of evidence to assess the evaluation issues, including:

• File review: Of a total of approximately 185 project files, 56 were reviewed across the sub-program (The 56 projects reviewed included: 33 from Mining Extraction Innovation; 9 from Mineral and Metallurgical Processing; 13 from CCRMP; and 1 from PTP-MAL ). Files were selected using two criteria based on the findings of a preliminary file review of approximately 10 project files: (1) representation from each project area; and (2) availability of complete files consistent with the quality assurance requirements of CanmetMINING’s internal filing system. To meet the quality assurance requirements, each project was expected to contain the following: completed and signed mandatory document (Mandoc) form; project proposal; project prospectus; project plan and costing; project agreements/Memoranda of Understanding (MOUs); invoices (if external); project amendments; progress reports; annual reports; and post mortem reports.
• Document review: Approximately 50 documents were reviewed, including: key program documentation; NRCan strategic documents (e.g., Reports on Plans and Priorities, business plans), and Government of Canada strategic documents (e.g., Budgets, Speeches from the Throne). The document review provided general information on the relevance and performance of the sub-program.
• International and scientific literature review: This was undertaken to position NRCan’s Mining Innovation sub-program relative to achievements of others in similar areas, in particular to respond to evaluation questions related to relevance and efficiency and economy. The focus of this review was on the United States, Australia, Europe, South Africa, China, international private laboratories and research organizations, and domestic stakeholders (provinces and territories, research consortiums, academia, mining associations and networks, etc.). While literature was found on websites of those governments and organizations, additional literature was provided by program representatives and stakeholders during or following interview sessions given their involvement in the area of mining innovation.
• Stakeholder interviews: Stakeholder interviews were used to gather in-depth insights from individuals who are most expert in delivery of the program. A list of stakeholders was identified with the help of the program staff and project files review. A total of 24 interviews were conducted with 15 internal stakeholders (i.e., senior management and staff) and nine external stakeholders (i.e., two other government departments, three provinces and territories, and four representatives from the mining industry).
• Case studies: In collaboration with program staff, the evaluation team selected nine projects for detailed examination to provide illustrative examples of various aspects of sub-program delivery and effectiveness. These were selected to reflect the different areas of research, types of partner organizations, project delivery dates, as well as funding and resource levels, and suitability to inform the evaluation. The case studies were conducted using 15 interviews (nine with project proponents and four with NRCan project managers) and document review. Projects selected for case studies were:
• Mining Extraction Innovation (6):
• Analysis of Ground Support Materials and Properties for Xstrata Ni-Rim;
• Evaluation of the behaviour of Swellex;
• Sécurité des machines d’extraction de mine souterraines;
• Développement d’un système de surveillance en continu des câbles de hissage miniers;
• Développement de cables synthétiques; and
• Underground Coal Mining Safety Research Collaboration.
• Mineral Processing (2):
• Collaborative research on recovery of Neodymium, Yttrium, Dysprosium, Praseodymium and Niobium from the Montviel ore; and
• Behaviour of chromium during chromite ore processing applicable to exploitation of ring of fire chromite deposits.
• PTP-MAL (1):
• Proficiency Testing Program.

2.3 Evaluation Limitations and Mitigation Strategies

As outlined in Table 4, there were some limitations associated with the data collection faced in this evaluation. Table 4 also identifies the mitigation strategies used to address limitations.

Table 4: Evaluation Limitations and Mitigation Strategies

Limitation	Mitigation Strategy
The program is fully compliant with ISO 9001:2008, for which audits are conducted regularly. This standard requires organizations to have in place a quality management system for all processes, including a quality assurance process for all projects requiring them to contain standard materials (e.g., completed and signed mandatory document; project proposal, prospectus, plan, and costing; agreements/ memoranda of understanding; etc.). Nevertheless, for the purposes of the assessment of relevance and performance (i.e., effectiveness, efficiency and economy), there were sometimes inconsistent reporting and documentation practices across projects (e.g., a mix of hard and soft copy files, lack of reporting against outputs). As a result, not all files could be reviewed.	A sample of files was selected from each of the project areas where files were deemed to be complete based on the quality assurance requirements of the CanmetMINING internal filing system. This helped ensure the reliability of the evaluation’s assessment of, and findings related to, efficiency in the production of outputs and outcomes.
At the outset of the evaluation, there was no logic model in place for the Mining Innovation sub-program. As a result, performance information was not specifically tracked according to intended outputs and outcomes at the sub-program level. Rather performance information was generally gathered at the level of individual projects resulting it challenges linking project-specific activities to sub-program outputs and outcomes. As a result, it was not possible for the evaluation to assess fully the effectiveness of the sub-program.	The evaluation team facilitated a logic model workshop with the sub-program to develop a logic model with sub-program level activities, outputs, and outcomes. This became the basis for the assessment of effectiveness. The evaluation also relied upon multiple lines of evidence from sources other than program data (e.g., case studies, stakeholder interviews) to identify the extent to which outputs were produced and were contributing to the achievement of outcomes.
Because financial information was generally captured at the project level and no logic model existed, the information was not available at a level that would facilitate a direct assessment of efficiency and economy of the sub-program (e.g., cost per outputs, international comparison, etc.) as part of the evaluation.	The evaluation assessed indirect measures of efficiency and economy, including overall resource utilization, different aspects of program design and delivery, to look for characteristics indicative of an efficient and economic means of achieving intended outputs and outcomes and identify potential areas for improvement.
NRCan’s Mining Innovation and Green Mining sub-programs share systems and committees, and often have common stakeholders. As a result, it was sometimes difficult to differentiate findings between the two sub-programs during the data collection and analysis phases of the evaluation.	To the extent possible, the evaluation mitigated against this by using multiple lines of evidence to try to identify findings specific to the Mining Innovation sub-program; however, it was not always possible to differentiate between the two even when triangulating data from the multiple lines of evidence. Nevertheless, given the close linkages between the two sub-programs, findings were deemed relevant to the broader mining innovation activities delivered by NRCan Data collection instruments were also designed to focus specifically on outcomes related to the Mining Innovation sub-program.

3.0 Evaluation Findings: Relevance

3.1 Ongoing and Continued Need for the Sub-program – Demonstrated

Summary: The evaluation identified a variety of ongoing needs related to increasing innovation in the mining industry, including: support and collaborative approaches for R&D; specialized services such as conformity assessment services and laboratory proficiency testing for accreditation; and national coordination related to mining innovation.

Support and Collaborative Approaches for R&D

As of 2014, global REE production and the associated S&T expertise were located almost exclusively in China, which also dominated the production and export of products containing REE. However, because resources in China are diminishing, with less than 30 years worth of estimated supply remaining, China has been implementing restrictions on its REE exports (since 2005), which has resulted in significant price increases for REE on multiple occasions. The Standing Committee on Natural Resources learned in 2014 “that Canada has significant rare earth resource potential, and that developing REE is essential to the Canadian economy and global competitiveness (Standing Committee on Natural Resources, June 2014). To take advantage of this, however, public and private expertise is necessary to pool the limited REE knowledge in Canada.

To that end, consultations led by CanmetMINING in 2012, with both Canadian and international stakeholders, revealed a need for a coordinated approach to deliver and advance REE-specific R&D in Canada. Workshops led by the Mining Innovation sub-program identified gaps related to REE and chromite, which became the impetus for CanmetMINING establishing the Canadian Rare Earth Elements Network (CREEN) “ to collaborate, coordinate and optimize REE research in Canada and link to international efforts”.(CanmetMINING. Integrated business plan, 2013).  CREEN is an industry-led network consisting of industry, academia, consultants, and government, with a vision is for Canada to establish REE production and economic growth.

In 2015, NRCan included a REE and chromite initiative as part of its Mining Innovation sub-program focused specifically on addressing industry R&D gaps in this area, which was funded for approximately $25 million over five years, starting in 2015-16.  In addition, NRCan allocated approximately $2 million to the REE and chromite initiative from existing reference levels. This focus on REE is consistent with evidence from the documentation that identifies a need for collaborative R&D to help overcome barriers faced by this emerging sector, including from 2014, when the Standing Committee on Natural Resources noted that in order to diversify the global supply of REE, Canada needs to be involved (Standing Committee on Natural Resources, June 2014) All interviewees (internal and external) identified the importance of maintaining the federal government contribution to innovation by supporting a sustainable and secure supply of REE because the Canadian mining industry is facing challenges and international competition.

Senior management from the sub-program also indicated that industry is looking for low cost options for ore extraction and technologies that will increase productivity and lower operating costs (e.g., technologies that will increase energy efficiency, as energy is a significant cost in mining). The document review corroborated this need, identifying a specific need for R&D to develop measures to overcome challenges faced by the mining industry in relation to: discovering new deposits; operating deeper mines; increasing energy costs; and navigating complex regulatory requirements. In 2015, the Mining Association of Canada (MAC) identified trends showing three consecutive years (ending in 2015) in which Canada’s share of international exploration investment had fallen, suggesting competition for global mineral investment and financing challenges faced by ‘junior’ companies.

Conformity Assessment Services and Laboratory Proficiency Testing

Evidence from the evaluation suggests that NRCan laboratories are currently the only laboratories in Canada to offer many of the conformity assessment services sought by industry. Literature suggests this is different than the way these services are delivered in other countries, such as New Zealand, Australia, and Switzerland where these services are delivered by the private sector, or South Africa, where delivery of these services is shared between public and private organizations – without this service being offered elsewhere. Even though the relative benefits of private versus public delivery of conformity assessment services was not explored as part of this evaluation, it was found that in Canada, this services are not being offered elsewhere and so there is a need for NRCan laboratories to continue providing it. External interviewees, in particular, believed these services target innovative aspects of the Canadian mining industry. They also recognized the value of the impartial conformity assessment services and laboratory proficiency assessment provided by CanmetMINING as well as their potential contribution to enabling innovation in the mining industry.

Related to the conformity assessment services, the sub-program, through the PTP-MAL project area works with the Task Group Laboratories Minerals Analysis Laboratories Working Group of the Standards Council of Canada (SCC) to define “the metals of interest and the concentration range for each, as well as establishing the criteria and target values for assessing laboratory proficiency” (Natural Resources Canada website, Proficiency Testing Program ). In particular, on the NRCan website, it is stated that “PTP-MAL is a means by which a mineral analysis laboratory can assess its performance for one or more analytical methods independently of internal quality control”. It is also stated that if a laboratory wants to be accredited, or wants to maintain its accreditation, by the SCC to the International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) standard of ISO/IEC 17025 (i.e., General requirements for the competence of calibration and testing laboratories), participation in the PTP-MAL accreditation program is required.

National Coordination Related to Mining Innovation

Some external stakeholders identified that there seems to be a lack of a clear definition of CanmetMINING’s role in coordinating mining innovation relative to the role of others. Although CanmetMINING interviewees indicated that the many players involved in mining innovation are complementing each other, all external interviewees noted that the absence of an overarching collaborative national strategy on mining innovation seems to affect the selection of relevant mining projects that are recognized by all stakeholders. Many external interviewees suggested CanmetMINING could strengthen its leadership role and support the federal government in establishing an innovation strategy at the national level, where the role of each party would be identified.

3.2 Alignment with Federal Government Priorities and NRCan Strategic Objectives – Demonstrated

Summary: The Mining Innovation sub-program aligns with federal government priorities as identified in federal Budgets, as well as the Government of Canada’s S&T strategy. The sub-program also aligns with NRCan’s strategic outcome, Canada’s Natural Resources Sectors are Globally Competitive, and the Department’s S&T strategy.

According to the documents reviewed, the Mining Innovation sub-program is consistent with federal government priorities, as indicated in federal Budgets from 2014 to 2016. In the 2015 Budget, for instance, REE and chromite specific properties were highlighted as critical inputs to the defence, aerospace, automotive, energy and consumer electronic industries. Specifically, the Unlocking Rare Earth Elements and Chromite initiative announced in this Budget allocated $23 million over five years, starting in 2015-16, to address the technical challenges of separating and processing REE for use in advanced manufacturing applications and products.

Mining Innovation is a sub-program in NRCan’s 2014-15 Program Alignment Architecture (PAA), under Program 1.2, Innovation for New Products and Processes, which supports Strategic Outcome 1, Canada’s Natural Resources Sectors are Globally Competitive. To support this strategic outcome, the following related expected results were identified for the program and sub-program:

• Innovation for New Products and Processes Program: Natural resource sectors increase production of new products and processes; and
• Mining Innovation sub-program: Technology developers increase demonstration of innovative mining and processing technologies (Natural Resources Canada’s 2014-15 Report on Plans and Priorities).

The evaluation also found that the sub-program aligns with NRCan’s Science and Technology Strategy: Creating a Sustainable Canadian Resource Advantage Through Science and Technology (2010). As noted in this strategy, NRCan’s S&T “serves the public interest through the generation of reliable, credible knowledge that advances science and fosters innovation, informs public decision making and delivers tangible benefits to Canadians”. This departmental S&T strategy supports the broader Government of Canada S&T strategy: Mobilizing Science and Technology to Canada’s Advantage (2007).

3.3 Legitimacy and Appropriateness of Federal Role – Demonstrated

Summary: The Mining Innovation sub-program aligns with federal roles and responsibilities, in particular with the mandate of NRCan as specified in the Department of Natural Resources Act (1994). For the most part, NRCan involvement is considered by internal and external stakeholders to be appropriate, and important to improving scientific and technological capacity of the Canadian mining industry, including assisting the industry as a whole to address needs pertaining to innovation in the sector. In particular, CanmetMINING’s provision of specific services – conformity assessment services and laboratory proficiency assessments – were identified as providing confidence to the public through reliable and certified analytical information. However, the evaluation identified that there are many stakeholders and organizations involved in mining innovation, which leads to some confusion among internal and external stakeholders of the distinct role of the Mining Innovation sub-program in the area of innovation in mining.

Documents review and NRCan staff interviewees indicated that the Mining Innovation sub-program aligns with NRCan’s mandate, as defined under section 6 of the Department of Natural Resources Act (1994, c.41). For example, CCRMP activities help the department fulfill its mandate as outlined in section 6(d), that the Minister shall “participate in the development and application of codes and standards for technical surveys and natural resource products and for the management and use of natural resources”.

Senior management interviewees noted that there is a federal role to play in enabling innovation and competitiveness for companies operating in Canada so Canadians can benefit from increased productivity and employment in mining. In this regard, CanmetMINING, including through the Mining Innovation sub-program, has historically played a coordination role of R&D activities in the minerals and metals sector and provided specialized services to the mining industry.

As part of the evaluation, all interviewees acknowledged this historical role. They also recognized the potential for the expertise within CanmetMINING to improve scientific and technological capacity of the mining industry in order to assist the industry as a whole to address needs pertaining to innovation in the sector. However, questions arose during the evaluation as to the precise role NRCan plays in developing innovative mining practices given the number of players involved.

As noted above, the Mining Innovation sub-program delivers specialized services such as conformity assessment services and laboratory proficiency assessments. The literature review identified that in other countries government does not always provide these services (i.e., in some cases, services were delivered by the private sector; in others it was a joint responsibility between public and private sectors). Although the evaluation did not explore the relative benefits of the private versus public delivery model for these services, without these services being delivered elsewhere there is a need for NRCan laboratories to continue providing these services. A memorandum of understanding between the SCC and CanmetMINING assigning the role for these services to CanmetMINING, provides an objective assessment “to ensure mineral analysis testing laboratories meet minimum quality and reliability standards to ensure a demonstrated uniform level of proficiency among these mineral analysis testing laboratories” (Standards Council of Canada, 2017). According to NRCan staff, a recent service review was conducted to understand the benefit of services to the mining sector and who should be providing them. As a result, some analytical services were transferred to private industry, but not the conformity assessment services or laboratory proficiency assessments. Examples of analytical services transferred to the private sector are: Total Respirable Dust (TRD), Respirable Combustible Dust (RCD) and Respirable Quartz Dust, using X-ray Diffractometry and direct-on-filter (XRD). Responsibility was transferred from CanmetMINING to the following laboratories: Concord Analytical; Institut de recherché Robert-Sauvé en santé et en sécurité du travail (IRSST); Vale; and SGS. Interviewees did note that the CCRMP and PTP-MAL services provide confidence to the public through reliable and certified analytical information. This was identified as important following several incidents in which the reliability of assays made available to investors was questioned.

4.0 Evaluation Findings: Performance

As noted above (section 2.3, Table 4, p. 9), a logic model was developed at the outset of the evaluation and became the basis for the assessment of performance. Because no logic model was in place prior to this, performance information was not specifically tracked according to the intended outputs and outcomes of the sub-program.

4.1 Efficiency and Economy – Partially Demonstrated

Summary: From 2010-11 to 2014-15, the Mining Innovation sub-program expended approximately 88% of its available budget. Of this, VNR made up approximately 25% of actual expenditures with the remaining 75% coming from ongoing A-base resources. All sub-program components generate VNR through the implementation of service fees and it is used to fund the sub-program as a whole, regardless of where it is generated. In delivering the sub-program, NRCan works collaboratively with multiple stakeholders from industry, provinces and territories, other federal departments, academia and international agencies. This collaboration helps reduce the risk of duplication of effort and facilitates the leveraging of resources and expertise. It also helps the sub-program identify the needs of the mining industry related to innovation. However, the evaluation found a potential opportunity for the sub-program to strengthen its leadership in the area of mining innovation. In particular, CanmetMINING could build on its strong collaboration and existing expertise to work with stakeholders to better focus innovation R&D in the mining sector (e.g., develop a national strategy on mining innovation). This could help increase the competitiveness of the industry while also ensuring priorities reflect common objectives. In addition, CanmetMINING maintains multiple accreditations that increase its reliability and enable it to deliver the CCRMP and PTP-MAL components of the sub-program. These standards ensure a level of consistency in different areas of program management. However, the evaluation found that performance measurement was generally done at the level of individual projects rather than for the sub-program as a whole. This presents a potential challenge to the sub-program to ensure it is delivered effectively, efficiently and economically.

Resource Utilization

Funding for the Mining Innovation sub-program is a mix of ongoing A-base resources (62%) and VNR (38%). It was also noted that VNR is generated and expended by both the Mining Innovation and Green Mining sub-programs. While Mining Innovation accounted for 59% of VNR expenditures, the remaining VNR would have been expended under Green Mining.

Financial data and interviews identified that all sub-program components expend and generate VNR through a fee for services delivered to mining industry clients. This is consistent with the Policy on Special Revenue Spending Authorities (2000) (replaced on April 1, 2017 with the Directive on Charging and Special Financial Authorities.) , which notes that VNR “is provided to fund fluctuating demands from user groups, which while consistent with program objectives, would otherwise jeopardize the relatively stable ‘core’ budget of the department”.

From 2010-11 to 2014-15, the Mining Innovation sub-program expended approximately 88% of available funds (A-base and VNR); however, the sub-program appears to have only expended approximately 59% of generated VNR over this time period. ‘Overhead’, which includes all activities not directly assigned to a specific component of the sub-program (e.g., finance, administration, etc.), accounts for the majority of sub-program expenditures (41%), with the remaining expenditures being allocated across the four components. When considering only VNR, the majority is generated by the CCRMP (51%) and Mining Extraction Innovation (40%) components, as illustrated in Table 5 (below), while the majority of VNR expended is allocated to ‘overhead’ (41%), followed by Mining Extraction Innovation (31%) and Mineral and Metallurgical Processing (25%).

Table 5: Breakdown of Mining Innovation Sub-program Expenditures by Source of Fund, by Sub-program Component (2010-11 to 2014-15)

Moreover, the evaluation found that sub-program project planning is sometimes done based on client-specific needs under each program component, rather than at a more strategic level that specifically identifies what activities or projects are needed to achieve the intended outcomes of the sub-program. While the use of VNR affords the flexibility to respond to the demand for services, planning at the component-level based on client needs or requests creates the potential risk that the sub-program will prioritize projects based on client demand rather than addressing broader policy objectives. CanmetMINING recognizes the potential risk of depending on external funding sources, particularly during economic downturns in the Canadian mining industry, as was the case during the time period covered by this evaluation, and is working towards “…stabilizing its revenues, which is essential to maintain its capacity to undertake innovative, well-aligned strategic programs”(CanmetMINING. Integrated business plan, 2013).

Partnerships and Collaboration

According to the Canada Mining Innovation Council (CMIC), in 2014 “there [were] over 100 organizations with an emphasis on assisting the mining industry including 12 separate research, development and innovation (RDI) organizations scattered across Canada, not including the centres of expertise in universities and colleges”. In its annual report, CMIC noted that “the level of collaboration between the centres is not sufficient to drive transformative change in the industry”. However, the evaluation found that of the organizations identified through the literature and stakeholder interviews, these organizations were sharing expertise – including with CanmetMINING – and were working with academia to serve the interest of mining companies through the exchange of knowledge, technology, and networks. Examples of organizations that were part of the networks include: Centre for Excellence in Mining Innovation (CEMI); Deep Mining Research Consortium (DMRC); International Minerals Innovation Institute (IMII); Mining Innovation, Rehabilitation and Applied Research Corporation (MIRARCO); CMIC; Canadian Institute of Mining, Metallurgy and Petroleum (CIM); and Expertise de calibre international en traitement de minerais (COREM).

A report from the 2015 Energy and Mines Ministers’ Conference (EMMC) highlights that “collaboration is a key aspect of innovation leadership in Canada’s energy and minerals sectors” because of “the high complexity of resource production and use…to ensure ongoing innovation, despite the currently challenged economic conditions”. Similarly, EMMC identified public and private sector partnerships as appropriate and important since they help “focus limited capacity and resources on areas where the challenges are the greatest”.

Much of the work of the Mining Innovation sub-program is conducted collaboratively with multiple stakeholders from industry, provinces and territories, other federal departments, academia and international agencies. More specifically, the sub-program “…collaborates with industry and academia to address pressing needs…[and] partners with private industry to develop and deploy innovative green mining processes and technologies”( CanmetMINING, Integrated business plan, 2013) This is consistent with the 2008 Pan-Canadian Mining Research and Innovation Strategy, which recognizes the importance of strategic collaboration, as well as activities undertaken by CanmetMINING related to the Mining Innovation sub-program (e.g., collaborating with provinces and territories, standard bodies, and other stakeholders).

As stated in the 2013-17 CanmetMINING Integrated business plan, in conducting its work, the sub-program is “in a constant dialogue with key innovation-driven organisations such as the Canada Mining Innovation Council (CMIC) to discuss collaborative opportunities and take advantage of synergies”. It is stated that often, this collaboration and partnership is done through committees and working groups. In addition to CREEN (discussed above) key examples of these committees include:

• Green Mining Initiative Advisory Committee (GMIAC), which is co-chaired by CanmetMINING and an industry member, and consists of 20 members from industry, academia, non-governmental organizations, other federal government departments, and provinces and territories. Through this committee, CanmetMINING obtains expert advice on its work, which ensures it aligns with, and is relevant to, the needs of stakeholders.
• Green Mining Initiative Intergovernmental Working Group (GMI-IGWG), which is a partnership between CanmetMINING (including this sub-program) and provinces and territories. Through this partnership, the working group helps “address barriers to the adoption of green mining technologies in Canada.

Through these partnerships and collaborations the sub-program also leverages resources (financial or in-kind), where possible, to share costs and expertise. For example, according to the Standing Committee on Natural Resources (June 2014), in the area of REE R&D, the sub-program has leveraged “five dollars from industry, in terms of in-kind contribution, for every dollar invested by the federal government”. For the sub-program as a whole, NRCan’s Departmental Performance Reports identified leveraging of approximately $2.3 million in 2012-13 ($1,900,000 from industry, and $437,000 from non-industry) and $2.0 million in 2013-14 ($1,557,000 from industry, and $482,000 from non-industry), respectively. The evaluation also found clear evidence of leveraged resources (both in-kind and financial) on a project-by-project basis.

Collaboration and partnerships through committees and working groups enables the Mining Innovation sub-program to undertake consultations with stakeholders to identify needs and priorities related to innovation. These consultations in turn help reduce duplication by ensuring alignment of priorities and facilitating the sharing or leveraging of resources and expertise when possible.

National Governance

All interviewees (internal and external) identified that NRCan has a key role to play in strengthening leadership of the federal government by putting forward updated, integrated, and coordinated mining innovation needs assessments that link R&D activities to the broader mining innovation policy agenda and related industry needs. These interviewees felt this would help determine the needs of stakeholders, including CanmetMINING and the Mining Innovation sub-program, and reduce the risk of duplication or potential competition while coordinating investment in, and attention to, common goals that might help increase competitiveness of the industry.

This is consistent with challenges the Mining Association of Canada suggests need to be addressed in the Canadian mining industry to continue to advance innovation (Facts & figures of the Canadian mining industry, 2014, pp. 50-51). These challenges include:

• A fragmented minerals sector innovation continuum (i.e., from R&D to demonstration and deployment) resulting from a lack of national scale coordination and industry R&D and innovation funding; and
• A mining sector that lacks the ability to network effectively and focus Canada’s R&D and innovation community while also integrating industry service providers into the innovation continuum.

External interviewees noted that the federal government could build on the expertise and strong tradition of collaboration already in place in universities, provinces and territories, and other research centres for the transfer of innovations to the mining industry and the development of a collaborative national strategy on mining innovation, where the role of each of the various stakeholders would be outlined. Related to this, many external interviewees expressed that, at the time of the evaluation, CanmetMINING did not play a strong coordination role in mining innovation and suggested CanmetMINING could play a leadership role by establishing such a national strategy on mining innovation.

To that end, CanmetMINING has played a leadership role by undertaking a number of initiatives, such as creating the Canadian Mining Innovation Council to provide a forum “…to enable better communication, collaboration and action on strategic priorities across Canada and with global partners” (Canadian Mining Innovation Council (2008), as well as undertaking multiple national-level studies on barriers to innovation in mining and state of commercialization in mining.

Standardization

The literature review revealed that, at the time of the evaluation, CanmetMINING, which administers this sub-program, was the only federal organization accredited to ISO 9001:2008 and ISO/IEC 17043 (Currently, it maintains accreditation to ISO/IEC 17043 and 17025). ISO 9001:2008 is a standard for the general management of quality of any process or area of management including: project planning; financial management; human resource management; administrative services; and verification of business operations.  ISO/IEC 17043 is a standard specific to proficiency testing programs, such as PTP-MAL. In order for the sub-program to continue to accredit clients to the standard of ISO 17025, CanmetMINING must maintain this accreditation.

Performance Monitoring

No evidence was found of a performance measurement strategy that would enable the monitoring of intended outcomes at the sub-program level. Rather, the evaluation identified that performance measurement was generally done at the level of the individual projects, with sometimes inconsistent reporting and documentation practices across projects (e.g., document filing process, mix of hard and soft copy files, version control issues, lack of reporting against outputs). For instance, information related to the production of outputs was generally only available at the level of the individual mining companies. As a result, it was difficult to identify a mechanism through which the sub-program could systematically monitor financial and non-financial performance on an ongoing basis in order to correct course where necessary and ensure the effective, efficient and economic delivery of the sub-program.

4.2 Effectiveness – Partially Demonstrated

Summary: The evaluation determined that the sub-program has made progress towards achieving its four intended immediate outcomes. It was also possible to align project-level information with the first intermediate outcome (adoption of mining innovation [techniques, technologies, etc.] by industry stakeholders) showing that clients are adopting innovations tested or developed in collaboration with the sub-program, but the evaluation did not find evidence of these being adopted by industry at large. In addition, the evaluation found evidence that the sub-program is contributing to some extent to the second intermediate outcome (improved productivity [including health and safety components] of the industry). Nevertheless, some concern was raised by internal and external stakeholder interviewees that this outcome may be beyond the reach of the sub-program. No unintended impacts were identified.

Internal stakeholders suggested that intermediate and ultimate outcomes identified in the logic model are ambitious and may not accurately reflect impacts the sub-program could reasonably be expected to achieve. Likewise, external interviewees felt these outcomes might be beyond the reach of the sub-program given its focus on client-driven needs and transactions. As noted in section 4.1, performance measurement was often done at the level of the individual projects rather than for the program as a whole, thus limiting the program’s capacity for ongoing monitoring at the sub-program level.

Nevertheless, as part of the evaluation, it was possible to align project-level performance information with the four immediate outcomes and two intermediate outcomes (adoption of mining innovation [techniques and technologies, etc.] by industry stakeholders; and improved productivity [including health and safety components] of the mining industry) identified in the sub-program logic model.

Immediate Outcomes 1 and 3: Reduced risks associated with adopting new techniques and technologies; and increased awareness and understanding of innovative technologies by industry stakeholders – Partially Demonstrated

Findings from case studies and the file review show that some key outputs were produced that contributed to reduced risks associated with industry adopting new techniques and technologies. The evaluation found that in many cases, this risk was reduced by increasing awareness and understanding among industry stakeholders of innovative technologies and new techniques. In particular, CanmetMINING laboratories conducted research and provided the necessary knowledge and tools to increase stakeholder understanding of new techniques and technologies that would then help reduce the risks associated with their adoption by industry. For example:

• Projects associated with dynamic and static ground support (rock bolt) R&D made test results from new and innovative technologies publicly available to increase awareness and understanding of these to help reduce the risk of implementing new ground support technologies underground;
• Development of a testing protocol for friction bolts, which had not been done successfully in a laboratory under dynamic conditions anywhere in the world, enabled the evaluation of the behaviour of these bolts under dynamic conditions;
• Testing of the synthetic cable enabled stakeholders to understand its behaviour and to take steps to reduce the risks associated with its implementation; and
• Development of process flowsheets and reference materials for REE and chromite R&D projects increased the development of techniques and technologies to access these mineral deposits, which are described by stakeholders as a key element of pre-feasibility for the eventual development of efficient processing techniques for REE.

CanmetMINING staff also reported on the sub-program’s efforts to understand barriers to industry’s adoption of technologies. For example, at the time of the evaluation, CanmetMINING was conducting a study on the state of commercialization of green technologies to identify barriers to adoption.

The sub-program includes collaboration and partnership with multiple stakeholders through committees and working groups (e.g., CREEN, GMIAC, GMI-IGWG, etc.). Through these forums, as well as publications, the sub-program is able to share findings and information with stakeholders to increase their awareness and understanding of new techniques.

Immediate outcome 2: Increased access to relevant markets for industry stakeholders – Partially Demonstrated

The accuracy of laboratory measurements related to both CCRMP and PTP-MAL are important to the mining industry because it is a cornerstone of commodity valuation. CRMs, for instance, can be used to validate the mineral analysis used to quantify the mineral concentrate and provide confidence to potential buyers that the stated mineral values are accurate. As a result, the use of CRMs affects decisions concerning the economic viability of exploration and mining, the value of the commodity, and actions to safeguard the environment. Similarly, by improving the reliability of measurements performed in the laboratory and serving as a control to verify the accuracy and precision of instrumentation/analytical methods, the conformity assessment services delivered through PTP-MAL increase access of the Canadian mining industry to relevant markets by giving confidence to investors and other markets in the quality of the work and analytical data produced by the Canadian mining industry.

To perform chemical and statistical analysis, the sub-program relies on donated samples of raw materials from Canadian companies in order to produce these reference levels. From 2010 to 2015, the sub-program performed chemical and statistical analysis and issued certificates for more than 22,200 bottles of certified reference materials (CRMs) that are sold within Canada (33% of sales) and to a range of clients internationally (67% of sales), which led to worldwide annual industry sales of approximately $1.4 million. Over this same time period, over 80 analytical laboratories worldwide participated in each PTP-MAL cycle.

As noted previously, if a laboratory wants to be accredited to ISO/IEC 17025, which is strongly recommended for being listed on the stock exchange, it must participate in the PTP-MAL program. With this perspective, stakeholder interviewees noted that services provided through the CCRMP and PTP-MAL respond to the need of mining companies and investors related to exploring and/or opening new mines with more confidence to inform decisions related to investments, which might include taking the risk to access new markets. In particular, accuracy and confidence in the chemical analysis used to determine the quantity and quality of potential mineral deposits is important to inform decision makers and investors prior to risking the millions of dollars required to develop new mines.  According to the Standing Committee on Natural Resources (June 2014), these services are also critical to support CanmetMINING research projects, such as those related to the development of new mineral separation processes, including the REE project, which is “essential to the Canadian economy and global competitiveness”. The search for a more efficient and cost-competitive process to produce REE from Canadian deposits required the production of new reference materials and a process to validate laboratory proficiency when dealing with the complex mineralogy in these types of new deposits (Mines to markets Natural Resources Canada rare earth elements technical workshop: Summary report, NRCan 2015).

Immediate outcome 4: Increased understanding of industry’s needs, priorities, and requirements – Partially Demonstrated

The evaluation identified that the sub-program has undertaken activities to increase its understanding of industry’s needs, priorities, and requirements. This includes:

• Participation in various international, provincial and territorial, and industry conferences;
• Establishment of the CREEN to share information and best practices related to research and development in mining;
• Initiating and participating in the Deep Mining Research Consortium;
• Acting as an external observer on the CEMI’s Board of Directors;
• Hosting annual technical advisory committees related to CCRMP; and
• Chairing, co-chairing, and/or participating in a variety of additional committees related to mining innovation on which there is also representation from the Canadian mining industry, analytical laboratories, other levels of government, and other government departments.

These fora provide the sub-program with the opportunity to consult and communicate with its stakeholders to facilitate an understanding of the needs, priorities, and requirements of various mining industry stakeholders. While the mechanisms are in place to do this, the evaluation was unable to confirm the extent to which this outcome has been achieved.

Intermediate outcome 1: Adoption of mining innovation (techniques, technologies, etc.) by industry stakeholders – Partially Demonstrated

The evaluation identified a few examples of clients implementing innovations that were tested or developed in collaboration with the sub-program; however, it did not find evidence that the mining industry at large was adopting innovations (techniques, technologies, etc.). Examples of client-specific innovations implemented by clients include: ContiscanTM real-time rope monitoring for mine hoisting, which was licensed in 2012 and became commercially available in 2013; fibre reinforced shotcrete, a new type of ground support, which is one of the main support types use at Nickel Rim South Mine; and a minerals processing flowsheet, which has been used by at least one company to prepare its techno-economic assessment of a developing mine. CanmetMINING staff indicated that they continue to work with clients after the R&D knowledge or technologies have been made available in an effort to ensure their use and adoption by clients.

However, CanmetMINING staff identified potential challenges to the adoption of mining innovations by industry stakeholders, namely that new technologies often require regulatory changes that can take years, and Canadian mining companies rarely want to be the first to adopt a new technology if it has not been tested elsewhere. The evaluation could not identify the extent to which these challenges were impeding adoption by industry.

Some internal and external interviewees felt that this outcome might be ambitious and may be beyond the reach of the program; however, sub-program documentation identifies improving adoption of innovation by the mining industry as part of its objective.

Intermediate outcome 2: Improved productivity (including health and safety components) of the mining industry – Partially Demonstrated

According to an impact study commissioned by NRCan in 2015, the sub-program is contributing to some extent to improved productivity (including health and safety components) of the mining industry. The purpose of this Green Mining Initiative Impact Study was to quantify and monetize various socioeconomic impacts, including economic productivity and miner health and safety (including changes to codes and regulations), resulting from NRCan supported technologies and projects. The study considered 13 projects, including seven from the Mining Innovation sub-program. As shown in Table 6, six of these seven projects were deemed to have contributed to impacts of decreased workplace injuries and fatalities and/or increased productivity/time savings.

Table6: Mining Innovation Projects Contributing to Improved Productivity and Health and Safety

Project	Impact
	Decreased workplace injuries and fatalities	Increased productivity/time savings
Certified Reference Materials	X	X
Extraction and Stabilization of Radioactive Wastes*	-	-
Testing of Tendons and Other Ground Support Elements	X	X
Contiscan	X	X
Québec Hoisting Plants Safety and Optimization	X	X
Optimization of Cyanide for Gold/Silver Recovery	-	X
Microseismic Monitoring of Oil Sands	X	X
Notes: * Although the extraction and stabilization of radioactive wastes project did not contribute to either of these impacts, it did contribute to other impacts assessed as part of the impact study, including: decreased mining and business costs/increased revenues; decreased water contamination/acid mine drainage; decreased land degradation impacts; and decreased wildlife/fisheries impacts. Source: HDR Corporation for NRCan. (2015). Green mining initiative impact study – final report, p. 7.

More specifically, this 2015 study indicated that some of these projects are contributing to this outcome in the following ways:

• The use of Contiscan removes the need to manually inspect hoist cables, “…thereby reducing the need to shutdown the mine on a weekly basis, which increases the productivity of the mine”. It is also expected to increase cost savings related to increased productivity from being able to hoist greater volume, reducing the number of cables required, and also improve system safety.
• As a result of tests done as part the testing of tendons and other ground support elements project, there is “an expected reduction in rock bursts…[which] means fewer injuries…”.
• As a result of the microseismic monitoring of oil sands project, companies are able to “identify and fix shallow well failures in an effective and timely manner…[which] would result in reduced well downtime and improved production…”.

Not withstanding the findings of this impact study, internal and external interviewees expressed concern that this outcome may be beyond the reach of the sub-program given its focus on client-driven needs and transactions through individual projects, rather than addressing broader policy objectives to contribute to this outcome.

4.3 Factors Impacting the Achievement of Outcomes

Summary: Some factors were identified that could positively or negatively impact the ability of the sub-program to achieve its intended outcomes. These include: commodity prices; practices of other countries or investments in other sectors; financial, and non-financial risks; potential constraints associated with non-disclosure/intellectual property (IP) agreements; and collaboration among stakeholders.

According to the document and literature reviews, the ability of the sub-program to achieve its intended outcomes could be impacted by external factors such as commodity prices, directions being taken by other countries, or investments in other sectors. In 2013-14 for instance, CanmetMINING had identified that “…falling [commodity] prices in Canada…resulted in a 7.9% decline in value of mineral production between 2011 and 2012.” (CanmetMINING, Integrated business plan, 2013) In an unstable climate such as this, mining industry stakeholders may not be able or willing to invest in new or innovative techniques or related R&D, thus making it difficult for the sub-program to achieve its intended outcomes.

Similarly, if the tendency of other countries is towards protectionism, there may be fewer markets for industry stakeholders to access, which could decrease their competitiveness. CMIC also identified that investment in innovation in the Canadian mining industry “is low and declining compared to other sectors such as forestry, automotive and manufacturing, and is approximately nine (9) times less than that of another competing country, Australia” ( CMIC Annual Report 2014). This could present a potential challenge to the sub-program’s ability to influence its intended ultimate outcome of an innovative and globally-competitive Canadian mining industry.

In addition, CanmetMINING staff identified more specific factors that might impede the adoption of mining innovation by industry stakeholders, such as financial (e.g., return on investment), and non-financial (e.g., organizational culture change) risks that industry may be unwilling to assume.

Evidence from the file review, case studies, and interviews, also suggests that IP, licensing, and confidentiality clauses that may be part of project agreements can impact the adoption of innovation by target audiences. It was noted that some projects are protected by non-disclosure agreements, thus limiting the ability to share knowledge from these projects with the broader mining industry. That said, non-disclosure/IP agreements are negotiated on a case-by-case basis, so the sub-program is generally able to mitigate the impact, including by trying to ensure projects focus on innovations that could benefit the industry as a whole and working with multiple stakeholders to do so.

CanmetMINING management also noted that collaboration with some stakeholders (e.g., CMIC) is important to the Canadian mining industry at the national level, as it facilitates the sub-program’s ability to increase stakeholder awareness and understanding of innovative techniques, which could reduce risks associated with the adoption of these techniques. It also enables the sub-program to increase its understanding of industry’s needs, priorities, and requirements to ensure alignment of priorities.

5.0 Conclusions and Recommendations

5.1 Conclusions

The Canadian mining industry faces ongoing challenges to remain competitive. As such, there is a continued need for support and collaborative approaches for R&D as well as conformity assessment services as these are enablers to innovation and economic competitiveness. In this regard, the Mining Innovation sub-program strives to assist the Canadian mining industry in mitigating challenges through support for R&D and the provision of conformity assessment services. To deliver on this, the sub-program relies on collaboration with the mining industry, private sector, other federal government departments, other levels of government, research consortia, and academia. However, the evaluation identified a need for improved national coordination related to mining innovation given the number of stakeholders involved.

The evaluation found the Mining Innovation sub-program to be consistent with the priorities of the Government of Canada, as well as NRCan’s strategic objectives. The sub-program activities and objectives also align with federal roles and responsibilities, and specifically, the NRCan mandate. Nevertheless, it was determined that the sub-program’s distinct role in R&D related to mining innovation may not be clear to internal and external stakeholders given the number of players involved in developing innovative mining practices.

In terms of performance, the evaluation found that the sub-program is making some progress towards achieving its four immediate outcomes and both of its intermediate outcomes.

Although the evaluation was unable to directly measure the efficiency and economy of the sub-program, indirect measures suggest that the sub-program uses some practices to ensure efficiency and economy. These include working collaboratively and in partnership with multiple stakeholders (e.g., industry, provinces and territories, other federal departments, academia and international agencies), leveraging resources and expertise where possible, and maintaining standardization of processes (i.e., accreditation). However, the evaluation identified two gaps that could hinder the ability of the sub-program to be as efficient and economic as possible, namely the absence of: a performance measurement strategy that would enable the monitoring of intended outcomes at the sub-program level; and a collaborative national strategy on mining innovation to better focus innovation R&D in the mining sector.

5.2 Recommendations

Based on the findings and conclusions of the evaluation, the following recommendations are made to the Assistant Deputy Minister of LMS:

• Recommendation 1: LMS should identify and communicate to stakeholders the roles of all parties relevant to mining innovation, and work with appropriate partners to help identify priorities for R&D related to innovation in the Canadian mining sector.
  
  
• Recommendation 2: LMS should revisit the sub-program’s design and delivery with specific consideration of the appropriate balance between strategic priorities and client-specific needs. This should include exploring whether other financial mechanisms/funding models would better support program delivery.
  
  
• Recommendation 3: LMS should improve performance measurement and reporting to ensure adequate information is available to assess efficiency and effectiveness.

For Mining Innovation activities, this means developing and implementing a performance measurement strategy, including:  a logic model;  theory of change (i.e., identification of risks, contexts, and assumptions that explain how the program is intended to be delivered and achieve its intended outcomes); as well as performance indicators, data sources, methods, and data collection and reporting frequency for all elements of the logic model (i.e., activities, outputs, immediate, intermediate, and ultimate outcomes).

Annex A: Mining Innovation Sub-program Logic Model

Bibliography

1. Canada Mining Innovation Council (CMIC). (2014). CMIC Annual Report 2014.  Retrieved from: http://cmic-ccim.org/about/annual-report/
2. Canadian Mining Innovation Council. (September 2008). The pan-Canadian mining research and innovation strategy: Strengthening the competiveness of a responsible Canadian mining industry through excellence in research, innovation and commercialization. “A Report to Federal, Provincial and Territorial Mines Ministers from the Canadian Mining Innovation Council”.
3. Canadian Mining Innovation Council. (2008). The Canadian Mining Innovation Council and the Pan-Canadian Mining Research and Innovation Strategy, presented to CANMET-MMSL Technical Advisory Committee.
4. Canadian Mining Innovation Council (CMIC). CMIC Annual Report 2014. Retrieved from: http://cmic-ccim.org/wp-content/uploads/2014/04/CMIC_AnnualReport_2013.pdf
5. CanmetMINING. Integrated business plan, 2013/14 to 2016/17.
6. CREEN. (2013). The Canadian Rare Earth Elements Network…helping feed the global supply chain. Presented at COM13 Rare Earth Symposium. Retrieved from: http://www.cim.org/~/media/Files/PDF/Rare-Earth/CREEN%20Slides%20for%20web%20site.ashx?la=en.
7. Department of Justice. Department of Natural Resources Act (S.C. 1994, c.41). Retrieved from: http://laws-lois.justice.gc.ca/eng/acts/N-20.8/
8. Energy and Mines Ministers’ Conference. (July 2015). Innovating for a Strong Canadian Energy and Mining Sector. Retrieved from: http://www.nrcan.gc.ca/publications/11102
9. Government of Canada (2015). Strong leadership: A balanced-budget, low-tax plan for jobs, growth and security.
10. HDR Corporation for NRCan. (2015). Green mining initiative impact study – final report.
11. Innovation, Science and Economic Development Canada. “CanmetMINING” from Company profile – Canadian company capabilities. Retrieved from: http://www.ic.gc.ca/app/ccc/srch/nvgt.do?lang=eng&prtl=1&estblmntNo=234567037879&profile=cmpltPrfl&profileId=1921&app=sold
12. Mining Association of Canada. Mining facts. Retrieved from: http://mining.ca/resources/mining-facts
13. Mining Association of Canada. Regulatory effectiveness. Retrieved from: http://mining.ca/ourfocus/regulatory-effectiveness
14. NRCan. (March 2015). Evaluation report: Green mining initiative. Retrieved from: http://www.nrcan.gc.ca/evaluation/reports/2015/17190
15. Mining Association of Canada. (2015). Facts and Figures 2015. Retrieved from: http://mining.ca/documents/facts-and-figures-2015
16. Mining Association of Canada. (2014). Facts & figures of the Canadian mining industry, 2014. Retrieved from: http://mining.ca/documents/facts-and-figures-2014.
17. Natural Resources Canada. Proficiency Testing Program. Retrieved from: http://www.nrcan.gc.ca/mining-materials/certified-reference-materials/proficiency-testing-program/7829
18. Natural Resources Canada. 2014-15 Report on Plans and Priorities. Retrieved from: http://www.nrcan.gc.ca/plans-performance-reports/rpp/2014-15/15441
19. Natural Resources Canada. (2010). Science and technology strategy: Creating a sustainable Canadian resource advantage through science and technology.
20. Natural Resources Canada. (2015). Mines to markets Natural Resources Canada rare earth elements technical workshop: Summary report – July 2015.
21. Standards Council of Canada. (2017). Accreditation services program: SCC requirements and guidelines for the accreditation of mineral analysis testing laboratories.
22. Standing Committee on Natural Resources. (June 2014). The rare earth elements industry in Canada – summary of evidence. Retrieved from: http://www.ourcommons.ca/DocumentViewer/en/41-2/RNNR/related-document/6669744
23. Treasury Board. (2000). Policy on special revenue spending authorities [rescinded 2017-04-01], section 2. Retrieved from: https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12248&section=html