Evaluation of Trans Mountain Expansion Phase IV Coordination

Audit and Evaluation Branch
Natural Resources Canada

Presented to the Performance Measurement, Evaluation and Experimentation Committee (PMEEC)
October 24, 2022

Indigenous Land Acknowledgement

The Audit and Evaluation Branch (AEB) of Natural Resources Canada acknowledges that our offices, located in Ottawa, are on the traditional and unceded Territory of the Algonquin Anishinaabe People. We recognize that those reading this report may live and work in a different traditional Indigenous territory. Though our offices are in one place, we are committed to drawing on a range of knowledges and experiences to support evidence-informed decision-making that has broad potential benefits and can hopefully contribute to collective action to build a stronger society.

Within the province of British Columbia, the Trans Mountain pipeline crosses numerous traditional territories and 15 First Nation Reserves. Within the province of Alberta, the Trans Mountain pipeline route spans across multiple First Nations Traditional Territories of Treaty 6, Treaty 7, Treaty 8, and the Métis Nation of Alberta.

We recognize that the Indigenous Nations within the colonial borders of Canada have lived on this land, Turtle Island, and their respective territories for time immemorial. We recognize that their languages, cultures, vibrant histories, and continuous stewardship over the land enriches our communities and generations past, present, and future.

Table of Contents

• List of Acronyms
• Executive Summary
• Recommendations and Management Response and Action Plan
• Introduction
• Evaluation Objectives and Methods
• Evaluation Limitations and Mitigation Strategies
• What We Found: Planned vs Actual Delivery
• What We Found: Effectiveness of the Strategic Policy and Coordination Team
• What We Found: Effectiveness of the Phase IV Partnership Office
• What We Found: Facilitating and Impeding Factors
• What We Found: Ongoing Need
• What We Found: Lessons Learned
• Conclusion
• Appendix 1: TMX Initiatives and Lead Departments/Agencies
• Appendix 2: Evaluation’s Indigenous Engagement
• Appendix 3: NRCan-P4PO Tools
• Appendix 4: Summary of Lessons Learned in Past Evaluations
• Appendix 5: Evaluation Team

List of Acronyms

Executive Summary

About the Evaluation

This report presents the findings, conclusions, and recommendations from the evaluation of the coordination structures supporting the Government of Canada’s post-decision implementation (Phase IV) of the Trans Mountain Expansion (TMX) project. NRCan's AEB undertook the evaluation between June 2021 and March 2022, with participation from partner departments and agencies engaged in the coordination of TMX Phase IV implementation (i.e., Canadian Coast Guard, Environment and Climate Change Canada, Fisheries and Oceans Canada, and Transport Canada).

The evaluation examined the design and effectiveness of these coordination structures from June 2019 to December 2021, including the activities of partner departments in facilitating the intended results. The objectives of this evaluation are:

• To assess the extent to which TMX Phase IV coordination structures have been implemented as planned and can demonstrate progress towards intended results.
• Given results achieved to date and status of implementation, to assess the ongoing relevance of NRCan’s TMX Phase IV coordination.
• To identify and document lessons learned from early implementation that can be used to improve the performance of these functions and/or be applied to future interventions.

The evaluation responds to a Treasury Board (TB) commitment to complete an evaluation by March 2022 and aligns with the broader approach to evaluation of TMX Phase IV coordination approved by the Heads of Evaluation of both NRCan and partner departments in 2019.

What the Evaluation Found

Planned vs Actual Delivery

The evaluation found that the Government of Canada has established suitable governance mechanisms to coordinate the Crown’s implementation and delivery of the TMX-Phase IV activities and that these governance structures were operational in 2019. NRCan’s TMX coordination structures - i.e., the Strategic and Policy Coordination team (NRCan-SPC) and Phase IV Partnership Office (NRCan-P4PO) were also both fully operational in 2019. Overall, while the roles fulfilled by NRCan have been relatively well defined and consistent, the organizational structure and titles associated with NRCan’s TMX coordination structures have changed frequently over time to account for evolving roles.

These TMX coordination structures were structured and operating as planned, with one key exception – NRCan-P4PO has not fully fulfilled its intended role as a “single window” for Indigenous engagement. For various reasons, partner departments have developed or maintained their own relationships and are interacting directly with Indigenous groups. Notwithstanding this approach, NRCan-P4PO is still perceived by most respondents to play an important role in management of unique issues that are more sensitive or complex and in cross-departmental coordination.

Each department is accountable for the TMX Phase IV initiative(s) it leads, and reports on this through their own governance structures as well as interdepartmentally. The workload related to TMX coordination is not perceived as sustainable by partner departments, which report having underestimated the level of effort that would be required for coordination and reporting.

Effectiveness of the Strategic Policy and Coordination Team

Overall, we found that NRCan-SPC has created effective tools and processes to support federal coordination. While NRCan-SPC coordination has enabled efficiencies in some areas, improvements are still required for existing mechanisms to ease the flow of information. Our observations relating to NRCan-SPC’s specific activities are as follows:

• Progress Reporting to Governance Bodies. Overall, progress reporting to governance bodies is effective and key governance bodies are well supported. However, the Director General-level committee could be more effective.
• Strategic Oversight and Direction. Strategic direction and oversight provided were perceived to be effective by senior federal respondents but ineffective by those at the working level. The efficiency of tools and supporting processes could be improved. Nevertheless, while consensus on decisions was not always possible, NRCan-SPC is perceived to be open and transparent in working to resolve cross-cutting issues in the face of divergent views.
• Consistent Messaging. NRCan-SPC was perceived to be effective in influencing consistent messaging for the public, Indigenous peoples, and partner departments, although some key information products (e.g., updates to TMX website) were significantly delayed. There is also a perceived lack of clear delineation between the activities and responsibilities related to “communication” versus “engagement”.
• Regulatory Coordination. NRCan-SPC was generally perceived as being effective in fulfilling its intended role for regulatory coordination. Respondents from TMC noted that they were not entirely satisfied with the speed of regulatory approvals but acknowledged that NRCan helped facilitate and expedite this process (along with key decision makers in DFO and ECCC).

Effectiveness of the Phase IV Partnership Office

Federal key informants generally perceived NRCan-P4PO to be an innovative approach to engagement on major projects. However, there may be a gap between Ministerial expectations on this point, i.e., need to move quickly and work differently, and actual practice at the working level. Regardless, federal respondents largely agreed that NRCan-P4PO has been effective in its coordination role, helping partner departments navigate complexity, addressing more sensitive issues impacting a small number of Indigenous groups, and providing a coordinated voice on whole-of-government issues. 

NRCan-P4PO has put in place various trackers and tools as required to support coordination of the whole-of-government engagement process. While these tools are effective, the evaluation found limitations in some tracking systems to support effective performance measurement. These systems were not robust enough to provide accurate assessment of the depth of NRCan’s engagement.

The engagement strategy developed by NRCan-P4PO was finalized relatively late in Phase IV implementation. While this did not delay initial outreach to Indigenous groups, key communications products (e.g., P4PO newsletter and TMX Indigenous Portal) were either delayed or underused.

Evidence suggests that NRCan-P4PO regularly engages with a large number of Indigenous groups and has met its performance targets for initial outreach and engagement. However, NRCan-P4PO metrics do not provide information on the meaningfulness of engagement. While they represent a limited sample of Indigenous groups, the evaluation’s Indigenous respondents generally rated the effectiveness of P4PO’s engagement as “poor” to “fair” and provided diverse perspectives on the meaningfulness of its engagement. The evaluation found evidence of a few successful attempts to triage, streamline and tailor the Crown’s engagement to reflect Indigenous groups’ concerns, interests, and project-related impacts. However, the federal approach to engagement on TMX has not been effective in reducing engagement fatigue. When compared to other GOC initiatives, NRCan-P4PO did not stand out as a model of coordination on engagement.

It is not clear whether required improvements would be the responsibility of NRCan-P4PO or partner departments. Even if successfully implemented for TMX, there would still be a limitation in this project-specific single window approach in that it would not coordinate across other (non-TMX) federal initiatives.

Facilitating and Impeding Factors

We found several external and internal factors that have or could influence the effective and efficient delivery of federal coordination related to TMX Phase IV implementation. Impeding factors with the greatest impact on the effectiveness of NRCan-SPC were perceived to be GOC capacity and the federal decision-making and accountability regime for TMX (i.e., that NRCan was given the mandate and accountability for coordination but does not have the authority to direct the actions of partner departments). By contrast, key factors impeding the effectiveness of NRCan-P4PO relate more so to the federal government’s and Indigenous groups’ capacity for (coordinated) engagement.

The quality of support from NRCan-TMX teams and ongoing support from senior management have facilitated effectiveness of TMX coordination. While there are some factors over which the NRCan-TMX coordination structures have little to no influence (e.g., unforeseen natural disasters), the evaluation found several instances where NRCan has identified and addressed impediments to effective coordination.  For example, NRCan-P4PO has recognized and made attempts to respond to a need for additional capacity for Indigenous groups. Regardless, Indigenous respondents to the evaluation perceived room to improve skillsets for engagement (e.g., tailored communications) and reported challenges with building and maintaining relationships, particularly given high levels of turnover in federal staff.

Ongoing Need for TMX Phase IV Coordination

NRCan and partner departments generally agree on the continued need for TMX Phase IV coordination and advocated a need to preserve the corporate knowledge built during the implementation of TMX Phase IV. This need will endure at least as long as the accommodation measures are still being implemented and/or construction is ongoing. While NRCan may currently be best placed to lead on coordination across departments, where the coordination function is located is not perceived to be important provided it continues to ensure that the Crown’s commitments made during Phase III consultations and Phase IV are being met.

There is stronger support expressed towards maintaining NRCan-SPC than NRCan-P4PO. For senior managers, the key question relates not to continued need for NRCan-SPC but rather the resource levels required to sustain oversight and coordination. By contrast, many respondents indicated that the role of NRCan-P4PO could be reduced now that accommodations are in progress and partner departments have built their own direct relationships with Indigenous groups.

Lessons Learned

As highlighted above, the evaluation identified the areas for improvement that could support ongoing delivery of TMX Phase IV. It also identified lessons related to engagement post-decision on major projects, as well as a need to manage the precedent set by TMX and further advance reconciliation with Indigenous peoples. These lessons learned do not lead to easy solutions but are also not unique to TMX. Related lessons learned from current and past program reviews should be reviewed collectively and shared across the federal government to inform future initiatives.

Recommendations and Management Response and Action Plan

Recommendation	Management Response and Action Plan
Recommendation 1. NRCan’s ADM Nòkwewashk should work with partner departments to review and streamline the level of effort required to support sustainable coordination. This should include a review of internal tools and processes (e.g., frequency and mix of coordination products) to reduce burden and provide a more holistic view of expected input and related timelines.	Management response: Nòkwewashk concurs with this recommendation, and has already taken steps to streamline products, activities, and processes to reduce the workload burden on TMX Partner Departments. Actions include: In Spring 2022, Nòkwewashk undertook a collaborative review of all coordination and communication products that require interdepartmental input; as well as all interdepartmental governance meetings. Representatives from the five TMX Partner Departments were asked to rank each of the products and meetings based on their overall effectiveness at supporting coordinated implementation. On the basis of this collaborative review, products have either been maintained, reduced in scope or frequency, or discontinued. Similarly, this review led to the consolidation of several recurring working level meetings in to one interdepartmental working group. The result of this review was a simplified but comprehensive governance structure; and a reduced list of effective interdepartmental products that will be carried forward. Nòkwewashk will repeat this exercise every fiscal year in collaboration with TMX Partner Departments, with an aim to only maintain coordination and communication products and governance meetings that are the most effective at supporting coordinated implementation. Position responsible: Manager, TMX Implementation Timing: Review of products to take place annually, at the end of each fiscal year to program sunset. Final review expected by April 1, 2025.
Recommendation 2. To increase their utility, NRCan’s ADM Nòkwewashk should work with partner departments to refine and strengthen the federal government’s approach to external communications on TMX Phase IV implementation. This would include clearer delineation of roles and responsibilities and process improvements that result in more timely and tailored communications, particularly for Indigenous groups.	Management response: Nòkwewashk concurs with this recommendation, and will work to strengthen and refine communications on TMX Implementation, within the constraints of its significantly reduced capacity. Nòkwewashk will continue to provide coordination support for future communications regarding the TMX initiatives as a whole. TMX Partner Departments are responsible for proactive, two-way communications with Indigenous groups regarding the individual initiatives under their purview. Actions include: Nòkwewashk will work with TMX Partner Departments and Natural Resources Canada’s Communications and Portfolio Sector to refine the federal government’s approach to communications, including further streamlining approvals processes for communications products under its purview (i.e., the TMX Website and the Indigenous Portal) to ensure more timely and efficient updates. The revised communications approach will include: Well delineated roles, responsibilities, and accountabilities; Mechanisms for resolving disputes regarding content; Clear, firm timelines for input and approvals; and Focus on developing accessible, plain language content. A full update of the TMX Website and Indigenous Portal is planned to take place before March 31, 2023. Position responsible: Manager, TMX Implementation Timing: April 1, 2023
Recommendation 3. NRCan’s ADM Nòkwewashk should revise its approach to Indigenous engagement in ways that build trust and cultivate meaningful relationships with Indigenous groups (e.g., frequent check-in calls from engagement officers). This engagement should be more proactive and include measures to mitigate the negative impact of staff turnover on maintaining relationships.	Management response: Nòkwewashk concurs with this recommendation, and building trust and meaningful, and long-term relationships with Indigenous groups is at the core of its mandate. Nòkwewashk is in the process of establishing long-term, place-based mechanisms for developing and strengthening these relationships that are not tied to individual projects or programs. Nòkwewashk will continue to conduct proactive engagement related to TMX through existing mechanisms such as the TMX Indigenous Advisory and Monitoring Committee. However, NRCan has limited resources to carry out its role in coordinating post-decision implementation of TMX and its related initiatives, and will no longer maintain its Phase IV Partnerships Office. Nòkwewashk’s role in carrying out proactive engagement specifically related to the TMX Project and associated initiatives through the Phase IV Partnerships Office has been wound down in favour of broader, place-based, engagement focused on long-term relationship-building. Nòkwewashk will continue to play an important role in being supportive and responsive to Indigenous groups that raise complex or cross-cutting issues related to the TMX Project. Actions include: In Budget 2022, Nòkwewashk received funding to expand the Indigenous Partnerships Office (IPO). The expansion of IPO is focused on establishing ‘regional hubs’ of staff across Canada focused on taking a place-based approach to building and sustaining relationships with local Indigenous communities across Canada. Nòkwewashk also received funding through Budget 2022 to continue to support the TMX Indigenous Advisory and Monitoring Committee to facilitate a collaborative, inclusive, and transparent approach to monitoring and oversight of TMX. Nòkwewashk will continue to be responsive and supportive to Indigenous groups regarding complex or cross-cutting issues related to the TMX Project. Position responsible: ADM Nòkwewashk, NRCan Timing: Actions will be ongoing. Review of progress to date to be completed by April 1, 2025.
Recommendation 4. NRCan’s ADM Nòkwewashk should develop performance metrics for TMX Phase IV implementation that are more meaningful to Indigenous groups.	Management response: Nòkwewashk concurs with this recommendation, and has adjusted its performance indicators to reflect its role in supporting ongoing Indigenous engagement related to TMX implementation to the extent possible given the dedicated resources available. Actions include: As part of its performance measurement strategy, Nòkwewashk has adjusted its performance indicators to measure: Supportiveness and responsiveness to Indigenous groups on TMX-related matters; Ability to support fulfillment of custom commitments to Indigenous groups made during the Phase III consultations; and Level of satisfaction from Indigenous groups with the effectiveness of support provided by Nòkwewashk during implementation. Nòkwewashk will work with Indigenous groups directly to ensure that performance metrics and indicators are meaningful to Indigenous groups for any new or renewed program proposal put forward. Position responsible: ADM Nòkwewashk, NRCan Timing: Performance indicators must be fulfilled by program sunset (March 2025). Final reporting expected by June 30, 2025.
Recommendation 5. NRCan’s ADM Nòkwewashk should work with partner departments to improve the definition and communication of roles and responsibilities for coordination of TMX Phase IV implementation, including working collaboratively to develop dispute resolution mechanisms.	Management response: Nòkwewashk concurs with this recommendation, and will work with TMX Partner Departments to ensure there is a mutual understanding of roles and responsibilities. Actions include: In March 2022, Nòkwewashk shared updated Terms of Reference for each of its governance committees, revised in collaboration with TMX Partner Departments. NRCan will review these Terms of Reference on an annual basis, with the next review scheduled for March 2023. The updated terms of reference should: Establish clear and predictable objectives for interdepartmental work; Ensure topics and issues are able to be brought forward and resolved through the governance structure; Establish clear expectations for all parties around timing for input, reviews, and approvals; and Be accompanied by a clear and predictable forward agenda. Position responsible: Director General, Regulatory Coordination, Consultation, and Indigenous Economic Inclusion, Nòkwewashk Timing: Annual update of Governance TORs to program sunset (next scheduled for March 2023). Final review expected by April 1, 2025.

Introduction

This document presents the findings, conclusions, and recommendations from an evaluation of the coordination structures supporting the Government of Canada’s post-decision implementation (Phase IV) of the Trans Mountain Expansion (TMX) project. The evaluation examined the design and effectiveness of these coordination structures from June 2019 to December 2021.

The evaluation responds to a Treasury Board (TB) commitment to complete an evaluation by March 2022. NRCan's Audit and Evaluation Branch (AEB) undertook the evaluation between June 2021 and March 2022, with participation from partner departments and agencies engaged in the coordination of TMX Phase IV implementation. These include Canadian Coast Guard (CCG), Environment and Climate Change Canada (ECCC), Fisheries and Oceans Canada (DFO), and Transport Canada (TC). The evaluation also engaged with Indigenous groups involved in TMX to obtain their perspectives on relevance and effectiveness.

Program Information

The Trans Mountain Expansion project (the Project) will twin the existing Trans Mountain pipeline built in 1953, increasing the pipeline’s capacity from 300,000 to 890,000 barrels per day. It will follow existing rights of way for 89% of its 1,147-kilometre length. This project also expands the Westridge Marine Terminal in Burnaby, British Columbia. Construction of the twinned pipeline started in November 2019 and was expected to be completed mid- to late-2022 (now 2023). Box 1 presents a timeline of key events related to the Project that led to TMX Phase IV.

Box 1: Timeline of Key Events from TMX Phase III Consultations to TMX Phase IV Implementation

In June 2019, the TMX Project decision was accompanied by a GOC commitment to implement eight accommodation measures to address potential impacts to Indigenous rights and interests identified during Phase III consultations, as well as respond to the Canada Energy Regulator’s 16 recommendations to the GOC to mitigate or avoid potential effects associated with the Project and related marine shipping (e.g., marine safety, oil spill prevention and response, studies of terrestrial and cumulative effects) and to build the capacity of and long-term relationships with Indigenous peoples. There are 129 Indigenous groups eligible to access some or all these accommodation measures, with specific eligibility based on the 2019 Crown Consultation List for TMX.^{Footnote 1} Additional bespoke commitments (unique to one or more Indigenous groups) were also made during Phase III consultations that needed to be implemented. These initiatives are complex and multi-faceted, with integrated roles and responsibilities linked to existing programs and activities of more than seven federal departments and agencies. However, TMX Phase IV implementation is not a horizontal initiative – i.e., individual departments remain responsible and accountable for the delivery of the TMX initiatives for which they are a lead (see Appendix 1).

Given this complexity, key objectives for federal coordination of TMX Phase IV implementation were outlined in funding approvals to include:

1. Ensuring the GOC is mobilized and has the resources necessary to manage post-decision implementation throughout the regulatory phase of the TMX Project, given the proponent's construction schedule.
2. Engaging in meaningful post-decision phase Indigenous engagement and continued involvement of Indigenous peoples throughout post-decision activities.
3. Ensuring that the suite of accommodation measures and responses to NEB [CER] recommendations are implemented in a coordinated manner, and that the ongoing activities are being briefed on and reported to appropriate bodies.

Program Governance

The Minister of Natural Resources is the designated lead on the coordination of TMX Phase IV Implementation. Implementation of related initiatives is governed by the interdepartmental Deputy Minister Oversight TMX Committee (DMOC), as well as the associated Assistant Deputy Ministers TMX Oversight Committee (ADMOC) and Director General+ Coordinating Committee (DG+CC). While the DMOC and ADMOC provide forums for strategic oversight and coordination across whole-of-government, most decisions are now made at the level of DG+CC.

The GOC has funded two TMX coordination structures to facilitate coordination on TMX Phase IV implementation, both led by NRCan:

1. Strategic Policy and Coordination Team - NRCan’s Strategic Policy and Coordination Team (NRCan-SPC) is an internal-facing coordination structure. Its purpose is to help partner departments navigate the complexity surrounding the TMX Phase IV implementation and horizontal coordination. NRCan-SPC has two key functions: (1) policy and coordination (providing policy support on cross-cutting issues, tracking progress, ensuring coordination amongst partner departments, coordinating public communications, finding efficiencies in delivery, and supporting senior management oversight) and (2) supporting and tracking regulatory authorizations, permitting and construction.
2. Phase IV Partnership Office - NRCan’s Phase IV Partnership Office (NRCan-P4PO) is an external-facing coordination structure. Its purpose is to play an active role in coordinating and triaging the Crown's outreach efforts to mitigate engagement fatigue within Indigenous groups. NRCan-P4PO is responsible for engaging in dialogue related to the accommodation measures and custom commitments made during Phase III and Phase IV, as well as for tracking progress on their implementation by partner departments.

Since 2019, these two TMX coordination structures have reported to the Assistant Deputy Minister (ADM) of NRCan’s Nòkwewashk sector.^{Footnote 2} To the end of fiscal year 2021-22, this reporting was via the Director General (DG) of TMX Policy. Following a sector re-organization, these two functions are now part of two separate branches within Nòkwewashk, reporting to the DG of the Regulatory Coordination, Consultation and Economic Inclusion Branch. Each partner department has also established its own TMX Secretariat to support departmental coordination, supported by this overarching governance model.

While most TMX Phase IV initiatives are led by partner departments, NRCan is also responsible, in whole or in part, for two accommodation measures and two CER recommendations that it implements separately from the coordination function. The performance of these initiatives is not included in the scope of this evaluation.

Program Resources

Funding allocated in original funding approvals to government capacity and administration for the coordination of TMX is shown in Table 1. Total incremental funding allocated undertake this work is $31.6M over three to five years. This does not include funding allocations to specific accommodation measures (see Appendix 1).

Table 1: Funding Allocated to Government Capacity and Administration for TMX Phase IV

Department	Incremental Funding Allocated	Years of Funding
NRCan Footnote a	$17.5 M	2019-20 to 2021-22
DFO Footnote b	$8.7 M	2019-20 to 2023-24
CCG	$5.1 M	2019-20 to 2021-22
TC	$0.3 M	2019-20 to 2021-22
ECCC Footnote c	$0	n/a
Total	$31.6 M

Expected Results

Coordination of TMX Phase IV implementation is linked to the Resource Partnerships Sector Program under NRCan’s Core Responsibility for ‘Globally Competitive Natural Resource Sectors’ in its Departmental Results Framework. This Core Responsibility aims to advance and promote market access, inclusiveness, and competitiveness for Canada’s natural resource sectors, in support of jobs and economic growth. This includes supporting participation of Indigenous groups, organizations, or governments in Canada’s natural resource economy. From 2019 to 2021, working to construct and complete the twinning of the Trans Mountain Pipeline was also among the mandate letter commitments of the Minister of Natural Resources, anchored in commitments to protect and create jobs, create economic opportunities for Indigenous communities and use revenues to finance Canada’s clean energy transition.

Expected results for the coordination of TMX Phase IV implementation were outlined in the initiative’s funding approvals. These include:

1. Fulfill the Government of Canada's commitments to the public and Indigenous Peoples for the implementation of TMX-related initiatives.
2. Ensure whole-of-government coordination between departments on the delivery of initiatives.
3. Mobilize existing governance structures (created during Phase III) to optimize existing networks and improve implementation efficiency.
4. Support Indigenous groups to facilitate their access to opportunities related to TMX initiatives.
5. Monitor and facilitate reporting on progress.

There was no logic model guiding the implementation of the initiative given its administrative focus and expected short duration. To support our analysis, the evaluation team developed a logic model for the coordination of TMX Phase IV implementation based on the above noted objectives and expected results (see Figure 1).

Figure 1: Logic Model for the TMX-Phase IV Coordination

Text Description

Infographic showing the Logic Model for the TMX Phase IV Coordination.

The logic model shows the relationships between the inputs, activities, outputs, reach / participants, immediate outcomes, intermediate outcomes, and the overarching TMX ultimate outcome for the three components of the TMX Phase IV Coordination: P4PO, SPC’s policy and coordination function, as well as SPC’s regulatory authorizations, permitting and construction function.

For each of the three components, solid arrows point from “inputs” to “activities” to “outputs” to “reach / participants”, indicating direct influences of the TMX-Phase IV Coordination. Then, dashed arrows point from “reach / participants” to “immediate outcomes” to “intermediate outcomes” to the overarching “TMX ultimate outcome”, indicating indirect influences of the TMX Phase IV Coordination.

Inputs for TMX Phase IV Coordination includes funding over 27 months (2019-2022), which includes NRCan ($22.1M), DFO ($8.7M), CCG ($5.10M) and TC ($0.3M in O & M; $1.5M in G& Cs), and TMX secretariats of OGDs.

For P4PO:

• Activities include P4PO operates as a ‘‘single window’’ for Indigenous engagement.
• Outputs include communication / engagement (e.g., engagement strategy, outreach letters/e-mails, workshops, Indigenous portal, line-wide gathering, etc.) and tracking / coordination (e.g., Phase III commitments, AMs, Communication Log, etc.).
• Reach / participants include Indigenous group, Partner Departments, OGDs, and SPC.
• Immediate outcomes include Indigenous groups’ questions / concerns are timely answered and effectively managed through a coordinated approach, and coordinated and timely delivery of the initiatives and resolution of work issues with the Indigenous communities.
• Intermediate outcome includes GOC’s effective engagement with Indigenous groups on post-decision activities.

For SPC’s policy and coordination function:

• Activities include implements tools and processes for the coordination of TMX-Phase IV activities.
• Outputs include guidance / communications (e.g., TMX weekly updates, etc.), coordination (e.g., engagement toolbox, Phase IV calendar, GCDOCs repository etc.), and oversight [e.g., dashboards (progress on recommendations, AMs), funding tracker, etc.].
• Reach / participants include ADMs/DMs, Partner Departments, NRCan Minister, and P4PO.
• Immediate outcomes include guidance is provided to effectively tackle the complexities inherent to the project’s delivery, effective weekly reports to ministers, effective working relationships with P4PO that ensures consistent and timely messaging, and integrated / holistic overview of execution of the project’s complementary initiatives.
• Intermediate outcome includes effective coordination of the Projects’ implementation among partner departments.

For SPC’s regulatory authorizations, permitting and construction function:

• Activities include coordinates and provides support to the regulatory authorization and permitting process.
• Outputs include regular updates on permitting, regulatory and construction progress; liaison with permitting/regulatory teams (TMC, P/Ts, OGDs, etc.); update on litigation as required; and collaborative meetings with relevant stakeholders dealing with permitting, regulatory authorization, and construction matters.
• Reach / participants include P4PO, Partner Departments, TMC, Provinces, OGDs, and Indigenous groups.
• Immediate outcomes include effective monitoring of construction, permits and regulatory authorizations; coordinated tracking, management, and resolution of permitting, regulatory and operational matters; effective support of to proponent; and federal and provincial relationships are advanced.
• Intermediate outcomes include effective oversight and provision of support to ensure Project is completed the right way.

The overarching TMX ultimate outcome is Indigenous groups are satisfied with and participate in the TMX post-decision engagement activities, which are implemented as planned.

Infographic also shows three boxes that explain the assumptions, internal factors, and external factors that affect the TMX-Phase IV Coordination. Box 1 shows the “assumptions” that includes:

• Institutional arrangements are conducive to cooperation and collaborative co-delivery.
• Shared goals and views among departments about authorities, roles, and responsibilities.
• TMX Teams within departments are fully staffed with the right mix of skillsets (e.g., cultural competencies for Indigenous engagement, program knowledge, creative mindset, etc.).
• Seamless integration of IT solutions across the departments.
• Degree of homogeneity of Indigenous groups (e.g., human capacity, groups’ infrastructure, adherence to Project, etc.).

Box 2 shows “external factors”:

• The outbreak of COVID-19 pandemic.
• Natural disasters in BC and AB (wildfires, flooding).
• High profile Project with high political interest.

Box 3 shows “internal factors”:

• Coordination Function.
• NRCan teams’ support.
• Pace of implementation.
• Varying levels of management capacity across involved departments.
• Senior management ongoing support.
• Project with no central authority but with horizontal results implications.
• Partner departments’ bilateral engagement with Indigenous groups.

Evaluation Objectives and Methods

Led by NRCan’s Evaluation Division, the evaluation functions of implicated partner departments collaborated to develop an evaluation approach for TMX Phase IV Implementation. The resulting approach was brought to the TMX ADM and DM Oversight Committees for concurrence in 2019. As per that approach, this evaluation was led by NRCan’s Evaluation Division with input of partner departments coordinated through an interdepartmental Evaluation Steering Committee.

Evaluation Objectives and Scope

The objectives of this evaluation are:

1. To assess the extent to which NRCan’s TMX coordination structures have been implemented as planned and can demonstrate progress towards intended results.
2. Given results achieved to date and status of implementation, to assess the ongoing relevance of NRCan’s P4PO and SPC teams.
3. To identify and document lessons learned from early implementation that can be used to improve the performance of these functions and/or be applied to future interventions.

The evaluation focused on NRCan's design and delivery of coordination for TMX Phase IV implementation from June 2019 to December 2021. The evaluation considered the activities of partner departments in facilitating the intended results of these coordinating structures to the extent that it impacted performance but excludes functions of these departments for Phase IV which extend beyond coordination (e.g., DFO’s enhanced regulatory review process for TMX).

The performance of specific TMX initiatives, including those led by NRCan, was excluded from the scope of the evaluation. Some will be subject to separate evaluations according to the overall TMX evaluation approach and the internal evaluation plans of lead departments. The scope of this evaluation also excludes activities that took place during earlier phases of TMX (e.g., Phase III consultations).

Given its stage of implementation, it was not possible to fully assess the intermediate or ultimate outcomes of this coordination on TMX Phase IV implementation. The evaluation focused on progress towards immediate outcomes and the extent to which risks identified during program design had been effectively mitigated.

A Gender-Based Analysis Plus (GBA+) completed by the program recognized the importance of a distinctions-based, tailored approach to Indigenous engagement and that major resource projects as currently designed perpetuate an imbalanced relationship between the Crown and Indigenous peoples, as well as being a source of gender-specific issues (e.g., gender violence and socioeconomic challenges). Gender-specific issues are not part of the scope of this evaluation; however, the extent to which this broader context may inform lessons learned in the design and implementation of TMX Phase IV or other major resource projects were considered as part of the evaluation analysis.

Evaluation Questions and Considerations

The evaluation addressed these objectives by responding to the following evaluation questions:

1. To what extent are mechanisms to coordinate the Government of Canada’ implementation and delivery of post-decision activities established and operating as planned?
2. Are NRCan’s P4PO and SPC teams achieving expected results? For whom? Why or why not?
   1. How effective is NRCan-P4PO in facilitating post-decision phase Indigenous engagement and continued involvement of Indigenous peoples throughout post-decision activities?
   2. How effective is NRCan-SPC in facilitating the coordinated and efficient implementation and delivery of post-decision activities?
   3. What has facilitated or impeded the effectiveness and efficiency of these coordination structures for the Government of Canada? For Indigenous groups? Other partners and stakeholders?
3. Is there an ongoing need for NRCan’s P4PO and/or SPC teams?
   1. If yes, what would be their respective roles and who should be responsible?
   2. Have there been areas of activity or duplication that should be streamlined?
   3. What would be the alternative(s)?
4. What lessons have been learned from early implementation of these coordinating structures that can be used to improve the performance of these functions and/or applied to future interventions?

The evaluation was designed with consideration of the program’s information needs, risk and complexity, including GBA+ considerations. Given the objectives of TMX Phase IV, the evaluation design recognized a need to capture Indigenous groups’ perspectives on relevance and effectiveness. This met some limitations during data collection (see evaluation limitations).

The coordination of TMX Phase IV implementation has not been previously evaluated. However, there are a number of past reviews related to management of major projects, horizontal federal coordination and Indigenous engagement that were useful to inform lessons learned for this evaluation.

Evaluation Methods

The approach and methodology used for this evaluation followed the TB Policy on Results (2016) and related Standards on Evaluation. The evaluation used four lines of evidence (Figure 2).

Figure 2: The Evaluation’s Four Lines of Evidence

Evaluation Limitations and Mitigation Strategies

The evaluation used a mixed-method approach with multiple lines of evidence to mitigate against any limitations associated with individual methods. This enabled the triangulation of evidence across sources of information to identify valid findings and draw evidence-based conclusions. However, the following key limitations should be considered when reviewing the evaluation findings. Additional limitations are referenced where relevant throughout this report.

Key Informant Interviews and GOC Survey. While a small number of interviewees declined to participate (n=4), interviews were sufficiently comprehensive to be considered representative of the perceptions of senior managers in NRCan and partner departments. Similarly, there was an adequate response rate for the survey of federal engagement leads and departmental coordinators. However, staff turnover at all levels within NRCan and across partner departments since 2019 means that not all federal key informants were fully familiar with the original intended design of coordination structures for TMX Phase IV.

Document Review. Sufficient documentation was available to inform the evaluation but some information that would have been useful to inform analysis has not been retained. For example, NRCan has not kept a record of all changes in its TMX organizational structure since 2019 or the transitory input of partner departments used to update key tools and processes. Internal reviews (e.g., stock-taking exercise) were in progress but results were not yet available at the time of analysis to inform the evaluation.

Indigenous Engagement. Indigenous engagement stood alone in evaluation design as a separate method to ensure that it received the proper focus. The evaluation team engaged a consultant with expertise in Indigenous engagement from the Kelly-Sears Consulting Group^{Footnote 3} to advise on the engagement approach and lead the conduct of interviews with Indigenous key informants. The evaluation aimed to collect input from a minimum of 30 Indigenous groups (23.3% of the 129 on the Crown Consultation List for TMX). Using a random and targeted sample, the evaluation reached out to 64 Indigenous groups (49.6% of the 129). Despite best efforts (see Appendix 2), the evaluation only received input from representatives in 12 Indigenous groups (9.3% of the 129 or 40% of the evaluation’s target). Particularly given the known lack of homogeneity within this group of respondents, findings from Indigenous engagement can not be generalized across all 129 Indigenous groups. However, there was generally a high level of agreement in perspectives shared across responses.

What We Found: Planned vs Actual Delivery

Timely and effective establishment of mechanisms to coordinate the Government of Canada’s implementation and delivery of post-decision activities, including for engagement with Indigenous groups, was a key activity identified in funding approvals for TMX Phase IV implementation. The evaluation examined the extent to which these mechanisms were established and operating as planned.

The Government of Canada has established governance mechanisms to coordinate its implementation and delivery of the TMX Project.

Governance structures are a key component of the coordinated federal approach. For TMX Phase IV, the GOC expected to mobilize existing governance structures (created during Phase III) to optimize existing networks and improve implementation efficiency. This includes DMOC, which was established as part of Phase III Consultations for the Project, and the associated ADMOC. Analysis of program documents confirms that the GOC maintained these two governance committees and established a third, DG and Director- level committee (DG+CC) to further coordinate its implementation and delivery of post-decision activities.

The GOC expected these governance committees to meet frequently over at least the first three years of implementation to discuss issues related to implementation and delivery. As shown in Figure 3, the actual number of meetings exceeded performance targets in the first year but has since been reduced.^{Footnote 4} NRCan-TMX program staff indicated that while intensive governance was required early on, the need for decision-making from senior management reduced as implementation proceeded. Federal key informants (including senior members of the DMOC and ADMOC) from all departments perceived the number of meetings to be sufficient. The DG+CC also continues to meet regularly.

Figure 3: TMX Oversight Committees Meetings (Planned vs Actuals)

Text Description

Infographic showing TMX Oversight Committees meetings (planned vs actuals), consisting of a bar chart.

Expected versus actual number of meetings by fiscal year for DMOC

Fiscal years	Expected vs. Actual Number of Meetings
	Expected	Actual
2019-20	2	7
2020-21	8	2
2021-22 (Year to Date)	8	1

Expected versus actual number of meetings by fiscal year for ADMOC

Fiscal years	Expected vs. Actual Number of Meetings
	Expected	Actual
2019-20	8	10
2020-21	12	3
2021-22 (Year to Date)	12	1

Source: Program Documents

The NRCan-TMX coordination structures were established and operational in 2019.

Documents show that Phase IV Partnerships Office was established and became operational in 2019, with the first of numerous communications from NRCan-P4PO to Indigenous groups in September 2019. While narrowly missing its staffing target (with 14 positions staffed by 13 FTEs), this performance exceeds NRCan’s target to have NRCan-P4PO established and fully staffed (with 15 FTEs) by March 2020. Evidence also indicates that over this period NRCan-P4PO developed and implemented many of the tools and processes required to support its work (e.g., working groups, including one specific to engagement). By necessity, NRCan-P4PO’s communication approach changed significantly with the onset of the COVID-19 pandemic but interactions with Indigenous groups continued.

An NRCan team responsible for whole-of-government coordination on TMX Phase IV implementation, now referred to as the Strategic Policy and Coordination Team, was also established and has been operational since 2019. Some of NRCan-SPC’s key tools (e.g., TMX Phase IV Calendar) were developed and populated as early as June 2019, with working groups and other key tools and processes implemented in fall 2019. A Regulatory Permitting and Authorization (RAP) function was also created and later subsumed under NRCan-SPC. While NRCan-SPC’s mandate was consistent with the envisioned plan, the team has leaner staff than initially planned (i.e., 9 vs 15 FTEs).

NRCan senior managers perceived the NRCan-TMX coordination structures to be adequately resourced. However, key informants from both the federal government and the TMC noted a high level of turnover in staff across coordination structures over the period of implementation. There are indications that staff turnover has had an impact on planned operations, most importantly in the context of building and maintaining relationships in an interdepartmental project that involves extensive engagement with Indigenous peoples.

The structures and roles of NRCan’s TMX coordination structures align with planned design but have changed frequently over time to account for evolving roles.

The roles and responsibilities of NRCan’s TMX coordination structures are well documented, with key responsibilities for both NRCan-SPC and NRCan-P4PO clearly outlined by January 2020. The responsibilities outlined generally align with and add to roles as described in funding approvals.

Key informants in NRCan and all partner departments indicated a common understanding of the role of NRCan’s TMX coordination structures. They described NRCan as having a crucial role central to whole-of-government coordination. As it is not the lead department on implementing most accommodation measures or CER recommendations, NRCan’s role is not to dictate how initiatives should be implemented but instead provide oversight, strategic direction, and policy support on cross-cutting issues and to ensure coordination between departments and agencies as they navigate the complexity surrounding Phase IV implementation. This includes working to ensure consistent messaging and avoiding unnecessary burden for Indigenous groups. Funding approvals outline the specific role of NRCan-P4PO as shown in Figure 4.

Figure 4: Role of NRCan Phase IV Partnership Office (July 2021)

Text Description

Infographic showing the role of NRCan Phase IV Partnership Office (July 2021).

The left-hand side shows “Engagement”. Engagement includes: on-the-ground engagement with communities; facilitate transition from Phase III consultations to Phase IV engagement; single window concierge service; and support ongoing engagement regarding accommodation measures.

The right-hand side shows “Whole-of-Government Coordination”. The Whole-of-Government Coordination includes: coordinate Indigenous engagement from a whole-of-government perspective; coordinate whole-of-government resolution of outstanding Project-related concerns; and connect with partner departments to respond to Indigenous groups’ priorities and concerns.

While the roles fulfilled have been relatively well defined and consistent, the organizational structure and titles associated with NRCan’s TMX coordination structures have changed frequently.^{Footnote 5} Federal key informants generally agreed that implementation of TMX Phase IV coordination structures has been iterative, explaining that these have appropriately evolved as the program has matured over time.

All departments agreed that the NRCan-TMX coordination structures were operating as planned, with one key exception. One of the key objectives of TMX Phase IV Implementation is to engage meaningfully with Indigenous groups, which funding approvals indicate was to be facilitated by a “single window” approach to engagement led by NRCan-P4PO. This single window would manage the breadth and spectrum of initiatives that involve Indigenous consultations; connect with federal partner organizations to respond to questions from Indigenous groups; and provide timely and informed advice. While respondents from all departments indicated that a “single window” was well intended and/or a good concept, evidence indicates that NRCan-P4PO has not fully fulfilled this role. For various reasons, partner departments still mostly engage with Indigenous groups directly rather than through NRCan-P4PO (see Box 2).

Box 2: Reasons for Partner Departments’ Direct Indigenous Engagement

Key informants from TMC also indicated some duplication of Indigenous engagement activities and support provided to Indigenous groups between TMC and P4PO. Prior to Phase IV, TMC also already had their own agreements and long-standing relationships with Indigenous groups to support the latter in navigating the Project. Building off existing relationships, some Indigenous partners would thus first turn to the TMC Indigenous engagement team with specific questions on accommodation measures. Given TMC is not responsible for these accommodations, it would then organize phone calls or meetings between Indigenous groups and NRCan-P4PO. Nevertheless, the relationships between TMC and NRCan-P4PO were perceived to be mutually supportive, e.g., with instances where TMC has proactively invited NRCan-P4PO to participate in its workshops with Indigenous groups (and vice versa).

Regardless, NRCan-P4PO is still perceived by most respondents to play an important coordination, if not engagement function. Several respondents (including in TMC) also noted the important functions that NRCan-P4PO plays as a “concierge service” (facilitating difficult conversations or resolution of more challenging/cross-cutting issues) or “relationship broker” (assisting Indigenous groups in navigating accommodation measures and related governance), noting that it could otherwise be chaotic for Indigenous groups to navigate the accommodation measures.

A few departments’ key informants also indicated that at the outset there was insufficient focus on CER recommendations. While now being corrected, this issue has resulted in a perceived lack of understanding of how some of these recommendations contribute to related accommodation measures.

The workload related to TMX coordination is not perceived as sustainable by partner departments.

In addition to interdepartmental reporting, each department also reports on the TMX Phase IV initiative it leads through its own TMX Secretariat and governance structures. NRCan perceives interactions with these structures to be effective and efficient, as this enables strong lines of communication between NRCan and partner departments. However, key informants in all partner departments report having underestimated the level of effort that would be required for coordination and reporting. While management of major projects is familiar within the mandate of these OGDs, NRCan’s coordination on TMX Phase IV represents an extra layer of administration that they are not used to on major projects. The related workload is perceived to be taxing on internal resources.

Figure 5: Departmental Coordinators and Engagement Leads Perspectives on NRCan-TMX Coordination Structures

Text Description

Infographic showing Departmental Coordinators and Engagement Leads perspectives on NRCan-TMX Coordination Structures.

• 0% agreed with “current approach is totally sustainable”.
• 44% agreed with “current approach is somewhat sustainable”.
• 28% agreed with “current approach is somewhat unsustainable”.
• 11% agreed with “current approach is totally unsustainable”.

Source: Evaluation Survey

Working- and senior-level federal key informants noted that coordination on TMX creates constant demands to deliver on tight timelines, which puts pressure on the limited capacity in partner departments. Survey participants were asked about the sustainability of the current approach to government coordination given the capacity (resources and workload) of NRCan and partner departments. While no respondents indicated that the current approach was fully sustainable (Figure 5), the perceived impact of this demand varies among the departments (Table 2). Respondents from NRCan, TC and ECCC were the most positive about sustainability. While feedback from DFO was also less positive, only in CCG did the majority (60%) perceive the approach to be unsustainable. Comments indicate that this is attributed, in part, to tight turnaround time in reporting demands. The high level of turnover and need to bring new staff up to speed, including within NRCan-SPC, may contribute to these perceptions.

Table 2: Federal Informants Perspectives on Project Coordination Impacts

Department / Agency	Perspectives on Sufficiency of Planned Resources and Impacts on Project Coordination
NRCan	NRCan senior management perceived NRCan to be adequately resourced for coordination, with 9 FTEs for NRCan-SPC and 13 FTEs for NRCan-P4PO (March 2020). The NRCan-TMX unit has frequently assessed the efficiency of its internal coordination structures, with ongoing changes to NRCan’s internal structure for TMX coordination (including a major restructuring of the sector as a whole).
ECCC	ECCC received resources for project coordination via a transfer of Vote 1 from DFO. According to ECCC informants, a portion of this funding is used to run a small Secretariat (4 FTEs) that they perceive to be efficient. One ECCC senior manager indicated that more resources would be preferable (1-2 FTEs).
TC	TC senior management perceived that the funding it was allocated was relatively lean. TC has drawn on other resources and sources of funding to support the department’s role and alleviate year over year financial pressures, including TMX-IAMC funding as well departmental surplus. One interviewee suggested that one FTE does the job of three individuals. While interviewees mentioned that more resources are required, they did not specify the amount.
DFO	DFO sought Capacity and Administration funding to cover multiple areas of work to support Phase IV implementation, including but not limited to interdepartmental coordination. The bulk of this funding was directed to support a dedicated TMX regulatory review unit as well as leading the implementation of Recommendations 5&6 within DFO, with additional funds to support other departmental programs anticipated to have an increased workload as a result of TMX (e.g., of total funds, $1.07M to DFO Legal Services Unit and $1.82M to DFO Science, Oceans Management and Fisheries and Harbour Management to facilitate the implementation of TMX). Within DFO, coordination responsibilities fall under a dedicated TMX Regulatory Review team in Pacific Region (2 FTEs) along with the Major Projects Division in the National Capital Region (2 FTEs). While this is consistent with intended design, given the volume of work to support interdepartmental coordination, DFO drew some resources allocated to accommodations to further support coordination. DFO also reorganized to place coordination directly under the Associate Regional Director General, Pacific Region to ensure focused senior management attention on the file.
CCG	CCG interviewees estimated that it had created 3-4 FTEs to coordinate administration with NRCan. Again, given the volume of work to support interdepartmental coordination, funding allocated for coordination was deemed insufficient and CCG had to take resources from the implementation of other initiatives to support this coordination.

Overall, partner departments reported not being sufficiently resourced for coordination. As shown in Figure 6, evidence indicates that NRCan, DFO and CCG underspent funding allocated for coordination and administration (and for DFO, to support regulatory review). Overall, NRCan spent 69.3% of its planned budget for coordination of TMX Phase IV, while DFO spent 62.2% and CCG spent 70%. ECCC’s total expenditures were more or less as planned (99.5% of planned budget), overspending by 3% in just one of the three fiscal years. By contrast, TC’s expenditures represented 123.4% of its planned budget. Where lapses did occur, available financial data indicate that these were largely in terms of expenditures on operations and maintenance (O&M). NRCan and partner departments explained that lapses were caused by factors such as COVID-19, with funds not being directed to planned activities such as travel and the hiring of contractors, as well as staff turnover. NRCan also underspent the $1.5M in G&Cs it received to support Indigenous groups’ participation in engagement and consultation activities (see Box 6).

Figure 6: Total Planned versus Actual Expenditures on TMX Phase IV Coordination (All Categories)

Text Description

Infographic showing total planned versus actual expenditures on TMX Phase IV Coordination (all categories), consisting of a bar chart.

Planned versus actual expenditures by fiscal year for NRCan

Fiscal years	Expected vs. Actual Expenditures ($)
	Planned	Actual
2019-20	$4,971.3 million	$3,665.6 million
2020-21	$6,326.9 million	$4,678.6 million
2021-22	$6,226.9 million	$3,795.2 million

Planned versus actual expenditures by fiscal year for DFO

Fiscal years	Expected vs. Actual Expenditures ($)
	Planned	Actual
2019-20	$1,208.3 million	$738.9 thousand
2020-21	$2,044.5 million	$1,175.9 million
2021-22	$1,484.8 million	$1,030.0 million

Notes: DFO developed financial coding for each TMX funding stream. There are potential coding errors in this data; data included here presents DFO’s best efforts to isolate coordination and regulatory review funding to the extent possible.

Planned versus actual expenditures by fiscal year for CCG

Fiscal years	Expected vs. Actual Expenditures ($)
	Planned	Actual
2019-20	$927.4 thousand	$314.5 thousand
2020-21	$1,228.5 million	$1,127.7 million
2021-22	$1,228.5 million	$911.9 thousand

Planned versus actual expenditures by fiscal year for TC

Fiscal years	Expected vs. Actual Expenditures ($)
	Planned	Actual
2019-20	$681.9 thousand	$1,029.5 million
2020-21	$647.1 thousand	$561.8 thousand
2021-22	$445.9 thousand	$599.5 thousand

Planned versus actual expenditures by fiscal year for ECCC

Fiscal years	Expected vs. Actual Expenditures ($)
	Planned	Actual
2019-20	$276.4 thousand	$268.6 thousand
2020-21	$596.9 thousand	$615.2 thousand
2021-22	$485.6 thousand	$468.0 thousand

Notes:

* DFO developed financial coding for each TMX funding stream. There are potential coding errors in this data; data included here presents DFO’s best efforts to isolate coordination and regulatory review funding to the extent possible.

Source: Financial data provided by program staff. Note data does not include funding allocations to specific accommodation measures or implementation of CER recommendations.

What We Found: Effectiveness of the Strategic Policy and Coordination Team

The evaluation examined the extent to which NRCan-SPC and the mechanisms it has put in place were effective in facilitating a coordinated and efficient implementation and delivery of post-decision activities, including integrated, whole-of-government oversight and reporting.

NRCan-SPC has created effective tools and processes to support federal coordination and Indigenous engagement.

Given their importance to maintaining and finding efficiencies in whole-of-government collaboration in the delivery of TMX initiatives, NRCan set a target for NRCan-SPC to establish minimum of four reporting tools and coordination mechanisms by March 2020.The evaluation found that NRCan-SPC exceeded this performance target with more than 10 tools and processes to support federal coordination developed and implemented since 2019. These tools included a Whole of Government Working Group, weekly updates, initiative leave behinds and work plans/accommodation dashboards, which were all in place by March 2020. Each tool and process has its own, stand-alone purpose.

NRCan also set an initial target that the five responsible departments (including NRCan) would provide regular input into each tool/process (at a minimum, quarterly). The department later acknowledged that quarterly updates are not feasible for all products and proposed that the frequency of input be revised to at minimum, annually. These updates involve transitory information. Although input records were not available for most tools and processes in NRCan’s repository, program staff and federal key informants indicated that partners have been active in providing updates specific to their departments. Therefore, while not conclusive, available evidence suggests that the performance target was met.

Federal key informants from most departments generally perceived that these tools and processes provide a systematic, consistent, and comparable approach to obtain basic information (e.g., status of implementation and funding). However, these perspectives differed somewhat between senior and working level staff. Results of the survey of engagement leads and departmental coordinators are shown in Figure 7. Survey respondents generally agreed that most of NRCan-SPC’s tools or processes were effective in facilitating coordination across departments and agencies. Those used to provide internal updates and discuss cross-cutting issues – e.g., the accommodation dashboards, and the funding tracker – were all perceived to be somewhat or very effective by most respondents (≥50%). The only tool or process perceived to be ineffective was the TMX website. In general, feedback from departmental coordinators was more positive compared to engagement leads. Respondents from TC were also more positive relative to all other partner departments.

Figure 7: Engagement Leads and Departmental Coordinators’ Perspective on the Effectiveness of NRCan-SPC Tools and Processes

Text Description

Infographic showing Engagement Leads and Departmental Coordinators’ perspective on the effectiveness of NRCan-SPC tools and processes, consisting of a bar chart. Categories for extent of effectiveness are “ineffective”, “neutral”, and “effective”.

Percent of survey respondents on the extent of effectiveness by tools and processes

Tools and processes	Survey Respondents on The Extent of Effectiveness (%)
	Ineffective	Neutral	Effective
TMX DG+CC	18%	18%	64%
TMX Website	56%	22%	32%
Weekly updates	10%	30%	60%
Leave behinds	36%	27%	36%
AM dashboards & work plans	30%	20%	50%
Funding tracker	11%	11%	67%

Source: Evaluation Survey

By contrast, senior level federal key informants perceived that the DG+CC could be more effective. While this committee was perceived to be an essential forum for discussing and resolving important issues, some respondents indicated that a lack of strong participation from partner departments in providing input may be limiting its effectiveness. Further, these informants suggested that insufficient discussion at the working level and the late distribution of material made it difficult to engage in a meaningful way. NRCan has started to address this issue through informal meetings with counterparts in partner departments to identify where better support and clearer direction is required.

For other tools and processes, negative perceptions among both senior and working level staff were explained to relate less to the quality of products and more to the effectiveness of the process used to create these products. Engagement leads and departmental coordinators explained that the processes are often perceived to be rushed, inconsistent in their reporting, and underestimate the amount of time required for response. Survey and interview findings revealed challenges in obtaining input and approval, IT limitations to share the tools/processes across departments, and duplication of efforts due to the additions of working groups and committees. Respondents suggested that more predictability or advance notice of requests and expected service standards could help improve the process. Federal key informants from a few departments also suggested that more effectively leveraging tools and processes from existing or previous initiatives (e.g., Oceans Protection Plan) could have increased efficiency.

Overall, federal respondents perceived NRCan-SPC as effective in achieving its intended purpose.

Funding approvals for TMX Phase IV implementation indicate that NRCan-SPC had planned to complete an annual survey of engagement leads and departmental coordinators to inform its performance metrics related to effectiveness of coordination. However, at the start of this evaluation (May 2021), NRCan-SPC indicated that such a survey had yet to be completed. The evaluation team thus worked with program representatives on the design of the evaluation’s survey to ensure that it would also meet these information needs.

The evaluation’s survey results show that most engagement leads and departmental coordinators perceive that NRCan-SPC was effective in fulfilling its intended purpose (Figure 8).^{Footnote 6} Engagement leads were less positive than departmental coordinators; those from DFO and ECCC gave more negative feedback compared with those in other partner departments.

Figure 8: Engagement Leads and Departmental Coordinators’ Perspective on the Extent NRCan-SPC Effectively Fulfilled Its Mandate

Text Description

Infographic showing Engagement Leads and Departmental Coordinators’ Perspective on the extent NRCan-SPC effectively fulfilled its mandate, consisting of a donut chart.

Percent of survey respondents by extent of effectiveness

Extent of Effectiveness	Survey Respondents (%)
Effective	50%
Neutral	25%
Ineffective	25%

Source: Evaluation Survey

The evaluation’s findings related to the effectiveness of NRCan-SPC can be further broken down against each of its key areas of activity:

• Progress Reporting to Governance Bodies. This activity was perceived to be the most effective by respondents at all levels. Federal key informants perceived that key governance bodies, specifically ADMOC and DMOC, were well supported and have a good flow of information. Progress reports, products, and issues updates provided to the governance bodies were perceived to be well-organized and consistent. Federal key informants at the most senior levels (i.e., DM and ADMs) felt that NRCan was proactive in addressing their information needs.
• Strategic Oversight and Direction. Strategic direction and oversight provided were perceived to be effective by senior federal respondents but ineffective by most (67%) of those at the working level. Most respondents indicated that efficiency of tools and supporting processes could be improved. Nevertheless, federal key informants agreed that NRCan-SPC has established strong lines of communication with partner departments and is effectively working to develop consistency on cross-cutting issues in the face of divergent views. While consensus on decisions was not always possible (see Box 3), they stressed the importance of open discussion and transparency in decision-making.

Box 3: An Example of Challenges in Resolving Cross-cutting Policy Issues

• Consistent Messaging. As shown in Box 3, there is evidence that NRCan has worked with OGDs to develop consistent messaging in policy areas where departments have divergent views. Many federal respondents indicated that there would not be a consistent position on cross-cutting issues without NRCan playing a coordination and oversight role. Document review and interviews also revealed that NRCan-SPC worked with NRCan-P4PO to provide common speaking points on program availability and awareness for Indigenous engagement. While federal interviewees perceived NRCan to be effective in influencing consistent messaging, 50% of survey respondents perceived this activity to be ineffective. Interviewees from all departments recounted areas for improvement:
  • There is a perceived lack of clear delineation between the activities related to “communication” versus “engagement”, and who is responsible for each. Communications need to be part of engagement, with related coordination relying on the presence (or absence) of a robust engagement plan.
  • Key external communication products (e.g., TMX Indigenous Portal, updates to TMX website) were significantly delayed; these products were viewed as the least effective in supporting expected results. A similar finding applies to communication products delivered by NRCan-P4PO. Federal respondents underscored the many factors that impeded timely delivery of public communication products, including a lack of dedicated budget or expertise for communications within NRCan-TMX and the requirement for horizontal input and approvals from OGDs. Evidence suggests that some missteps in communications from NRCan have created discord and misunderstanding. These were reported to have resulted in additional work for OGDs and have increased the extent of scrutiny from partner departments. While NRCan-TMX representatives reported using related lessons learned for continuous improvement, this has still impeded timeliness of public communications.
  • Some federal key informants also perceived gaps in information sharing between NRCan’s TMX-Indigenous Advisory and Monitoring Committee (IAMC)^{Footnote 7} and TMX coordination structures.
• Regulatory Coordination. The role of NRCan-SPC’s regulatory permitting and authorization function is outlined in Box 4. While NRCan did not set any performance target specific to this area, the evaluation found that its role and function were clear and that it was generally perceived as being effective. Respondents from TMC noted that they were not entirely satisfied with the speed of regulatory approvals but acknowledged that NRCan helped facilitate and expedite this process (along with key decision makers in DFO and ECCC). While partner departments effort to deliver permits and authorizations within legislated timelines and aligned to the TMC’s construction schedule, proponents and regulators have a responsibility to first ensure that all requirements are met with respect to regulatory decisions. There is evidence of frequent meetings with the external stakeholders and partner departments to help understand operational issues, construction progress, and TMC’s priorities. Both TMC and federal key informants perceived that NRCan-SPC was effective in helping to steer the process and solving issues by convening partner departments and external stakeholders (including provincial regulators) and by building awareness of timelines among key parties.  

 Box 4: NRCan-SPC’s Regulatory Permitting and Authorization Function

While horizontal coordination has enabled efficiencies in some areas, improvements are still required for existing mechanisms to ease the information flow.

Funding approvals for TMX Phase IV implementation indicate a key expectation for NRCan-SPC to seek efficiencies in the delivery of Project-related initiatives. Results indicate that half of the surveyed engagement leads, and departmental coordinators considered NRCan-SPC to be ineffective in finding efficiencies in the delivery of post-decision activities; only 20% considered it to be effective. Engagement leads were again somewhat more negative than departmental coordinators. The response did not vary significantly by department. These perspectives are likely an extension of the previously noted issues with government capacity available to address workload pressures.

Federal interviewees were somewhat more positive, noting that horizontal coordination helps to identify areas to collaborate, facilitate work, and break down silos across departments. However, federal key informants from few departments underscored some areas for improvement. While internal communications from NRCan-SPC have been solid, it is unclear if there are good channels for feedback from OGDs to NRCan. There are conflicting views about the effectiveness of DG+CC working groups and committees to funnel products to key decision-makers (i.e., from Directors to DGs for approval). Some noted that it takes a lot of time and resources to prepare for related meetings, but that discussions are still slow (e.g., significant time spent debating about approach to renewal). Key informants also indicated that in some cases there may be a disconnect in the lines of communication between the working and senior levels (decisions are not briefed up or briefed down) and between HQ & regions. There is a perceived need for a better mechanism to keep the process moving. Currently, everything needs to be negotiated, which can slow decision making if there is insufficient DG oversight or attention.

What We Found: Effectiveness of the Phase IV Partnership Office

The evaluation examined the extent to which NRCan-P4PO and the mechanisms it has put in place were effective in facilitating a coordinated and efficient implementation and delivery of post-decision activities related to Indigenous engagement. The evaluation also examined the extent to which the Phase IV Partnership Office has facilitated Indigenous people’s participation in post-project decision activities.

Overall, federal respondents agreed that NRCan-P4PO has been effective in coordination (though not as a single window).

Funding approvals for TMX Phase IV implementation outlined five key outcomes for NRCan-P4PO:

• maintain and strengthen effective relationships with eligible Indigenous groups (through a single window), by triaging and streamlining the Crown’s outreach efforts to prevent capacity fatigue within Indigenous groups;
• coordinate the whole-of-government engagement plan with groups on accommodation measures, permits and authorizations;
• implement a consultation process that supports distinction-based approach by tailoring the consultation based on each Indigenous groups’ concerns, interests, and project-related impacts;
• engage in an ongoing dialogue with Indigenous groups related to implementation of accommodations measures, responses to NEB recommendations, and ongoing regulatory and permitting processes; and 
• track and follow up on any commitments made during Phase III and Phase IV. 

Effective implementation of these outcomes was paramount to ensure that the GOC upheld its commitments to be responsive to Indigenous groups with regards to concerns relating to the Project’s implementation. The delivery of these commitments also provided support toward the GOC reconciliation with Indigenous peoples. Federal key informants generally perceived NRCan-P4PO to be an innovative approach to engagement on major projects. Usually, the GOC does not continue to engage across the entire project life cycle (engagement ends post-decision once consultations are complete). At least conceptually, NRCan-P4PO helps ensure ongoing GOC accountability and discussion with Indigenous groups. A few departments indicated that a project like TMX needs innovation and creative thinking. However, they also perceived a (large) gap between Ministerial expectations on this point, i.e., the need to move quickly and work differently, and actual practice at the working level.

Regardless, federal respondents largely agreed that NRCan-P4PO has been effective in its coordination role. Half of survey respondents (50%) indicated that NRCan-P4PO was effective in coordinating the GOC in responding to Indigenous groups’ outstanding project-related concerns and priorities; only 14% perceived NRCan-P4PO to be somewhat ineffective. Among federal interviewees, NRCan was perceived to have effectively played a sort of concierge role, i.e., to help partner departments navigate complexity and address more sensitive issues impacting a small number of Indigenous groups. OGDs will message the TMX inbox with known or emerging issues, and NRCan-P4PO will assist by opening the door for difficult discussions. There was some indication that this service is appreciated, particularly among those who reported less internal expertise for Indigenous engagement. However, NRCan staff and OGDs agreed that, as departments remain responsible for their own programs, they also needed to be able to resolve their own issues. Regardless, while not without its challenges (see Box 3), interviewees indicated that NRCan-P4PO provides a very useful and valuable role as one voice representing the GOC where whole-of-government issues are concerned, advancing a neutral perspective to help federal departments agree on a path forward.

P4PO effectively supports coordination of messages within one-on-one engagements by providing information on other programs and common speaking points. NRCan-P4PO also shares information gleaned from its own engagement and that of other departments to provide updates on emerging issues and the current status of Indigenous groups (e.g., COVID-19 or wildfire closures, key contacts, etc.). However, to do so effectively assumes that NRCan-P4PO is aware of what was discussed (i.e., are included in one-on-one engagements and/or are briefed on key issues by the lead department) and they know where and when OGDs are going to engage. This is not always the case.

NRCan-P4PO is tracking progress on engagement with Indigenous groups on an ongoing basis, though with some limitations.

Document review revealed that NRCan-P4PO has put in place various trackers and tools as required to support the whole-of-government coordination (Appendix 3). These tracking tools serve two broad functions: (1) tools aimed at documenting/tracking progress made about the engagement with Indigenous groups, including to support performance measurement; and (2) tools supporting the coordination of Indigenous engagement among federal government departments that lead accommodation measures.

The majority of surveyed respondents reported that the specific tools and processes developed by NRCan-P4PO to coordinate engagement are effective (Figure 9). Senior federal officials also noted the effectiveness of these tools. For example, broader coordination of federal players through NRCan-P4PO’s engagement working group helped identify linkages between the TMX and Roberts Bank Terminal 2 project. While its use has waned somewhat over the past few months as OGDs lead more direct engagements, an interdepartmental engagement calendar developed by NRCan-P4PO has also been a key tool that has helped NRCan-P4PO coordinate meetings and be more efficient. Engagement leads indicated that they are kept aware of activities led by other departments and that NRCan-P4PO was effective at reducing or avoiding competing meetings with Indigenous groups.

Figure 9: Departmental Coordinators and Engagement Leads Perspectives on Effectiveness of NRCan-P4PO Tools and Processes

Text Description

Infographic showing Engagement Leads and Departmental Coordinators’ Perspective on effectiveness of NRCan-P4PO tools and processes, consisting of a bar chart. Categories for extent of effectiveness are “ineffective”, “neutral”, and “effective”.

Percent of survey respondents on the extent of effectiveness by tools and processes

	Survey Respondents on the Extent of Effectiveness (%)
Tools and processes	Ineffective	Neutral	Effective
Engagement Working group	20%	20%	60%
Engagement tracker(s)	17%	17%	67%
TMX Calendar	15%	23%	62%

Source: Evaluation Survey

However, federal interviewees expressed mixed views about the NRCan-P4PO tracking systems. Those with positive views noted that NRCan-P4PO knows which Indigenous groups should have access to each accommodation measure and appropriately tracks the extent to which they have engaged. Other interviewees expressed doubts vis-à-vis the adequacy of the performance measurement framework that NRCan-P4PO has put in place (see Table 3), noting that metrics based on its ability to reach out and inform or remind Indigenous groups of the supports available are reasonable measures of service delivery but do not provide information on the effectiveness of that outreach or the meaningfulness of engagement.

The evaluation process also found limitations in some of the P4PO’s engagement and coordination tracking systems to support effective performance measurement. These included outdated or (in a few cases) rescinded information, replication of information among some of the trackers, and misalignment with OGDs. Moreover, the outreach activities (e.g., number of e-mails sent out to groups, the number of groups that reached out the GOC, or the number of meetings held with a specific community) were not captured in a consistent manner and information was disseminated in various repositories, making it impossible for the evaluation team to have a comprehensive picture of the depth of the NRCan-P4PO engagement activities throughout the time covered by the scope of the evaluation. Some of these issues stem from staff turnover, including challenges in knowledge transfer on how to effectively use the tools, and cross-departmental IT limitations. Given these limitations, observations made in the sections that follow about the attainment by NRCan-P4PO of its engagement targets are mostly based on evidence provided through various program performance reports (i.e., the evaluation was not able to replicate this data).

NRCan-P4PO developed an engagement strategy, but it was finalized relatively late in Phase IV implementation.

NRCan-P4PO developed an engagement strategy to guide federal departments through the engagement process, and to ensure that engagement was timely, coordinated, and meaningful for Indigenous groups. While this document provides valuable advice, it was first implemented in October 2020 (revised in 2021), many months into the engagement process. Some departments indicated that the engagement strategy should have been in place at the outset. A few key informants also questioned whether this guide was sufficiently robust. For example, they noted that work on partnerships with Indigenous groups is complex and perceived that NRCan-P4PO could have done more to understand and work within the existing framework of partner departments.

While many interviewees noted a need for proactive communication, a few perceived that P4PO’s approach has resulted in “just in time” delivery of engagement communications and a struggle to identify the most relevant information to communicate. NRCan-P4PO began actively reaching out to groups with information on Phase IV well before the engagement strategy was finalized. However, key communications products were either delayed or underused. Perspectives related to specific tools or processes shared by one or more respondents were to the effect that:

• The P4PO newsletter, intended as a simple tool for information sharing within Indigenous groups, was planned as a quarterly update but had only been published twice by Fall 2021.
• The TMX Indigenous Portal was developed to be a single window for key resources – e.g., application guides, information sheets, speaking points for Indigenous groups, etc. The concept of an Indigenous portal was not included in initial program design but received approval from NRCan’s DM and NRCan Communications in January 2021. However, the portal only went live in August 2021. This launch came too late into implementation to fully benefit its target audience.
• NRCan-P4PO sent letters in January 2021 to Indigenous groups who should have access to accommodations measures but who had not yet engaged via workshops and outreach. Identification of these groups was made possible by P4PO’s engagement tracking. Letters sent were then supplemented by direct follow-up by individuals in line departments (as appropriate, based on community eligibility). This process was perceived to be effective: following the letter, there was some uptick in the number of Indigenous groups accessing accommodations.

The release of these products was slowed by approval from partner departments on language. Moving forward without engaging OGDs would have been difficult – without their input, products may not have included the right information or may have been missing content.

During Phase III consultations, Indigenous peoples indicated a need to have the GOC speaking from a single voice to prevent capacity fatigue within Indigenous groups. Fulfilling this commitment was considered key for the GOC to maintain and strengthen effective relationships with eligible Indigenous groups. While all key informants agree that NRCan-P4PO has not fully fulfilled its intended role as a “single window”, the evaluation found evidence of a few successful attempts to triage and streamline the Crown’s engagement. This includes instances of the following:

1. General information or one-off communication that benefited the vast majority (or totality) of the 129 Indigenous groups. NRCan-P4PO helped streamline engagement with Indigenous partners, by coordinating, with the support of OGDs, broad communication (e.g., outreach letter, fall 2019 workshops, Indigenous Portal). In some cases, NRCan-P4PO staff also managed one-off communication with Indigenous groups on behalf of all implicated departments (e.g., COVID-19 engagement letter).
2. Interdepartmental coordination where NRCan-P4PO staff led interdepartmental meetings, along with specific Indigenous groups, to facilitate access by the latter to accommodation measures. Through these meetings, NRCan-P4PO sought the coordination of federal departments’ actions, which was particularly important when groups needed to access multiple accommodation measures at the same time.
3. Cross-cutting issues brought to the table by NRCan-P4PO where there was a need for collective awareness and/or action. Many of these cross-cutting issues touched upon eligibility to accommodation measures and required that NRCan-P4PO staff coordinate multiple departments’ involvement. NRCan-P4PO coordinated communication concerning the federal departments’ response in these sensitive cases.
4. Targeted communication shared from NRCan-P4PO to a subset of Indigenous groups on behalf of NRCan and partner departments (e.g., letters sent to the groups that had yet to engage).

Despite these efforts, the majority of federal respondents disagreed that the federal approach to engagement on TMX has been effective in reducing engagement fatigue within Indigenous groups. Some noted that, at the outset, NRCan-P4PO’s coordinated engagement was useful to inform Indigenous groups on accommodation measures. This was perceived as more efficient (for both the GOC and the Indigenous groups) than one-off meetings. However, departments continue to engage bilaterally with Indigenous groups. While no federal respondents perceived this bilateral engagement as inappropriate, a few departments’ representatives identified the following disadvantages to the more disaggregated approach:

• Sometimes it comes in opposition with the whole-of-government approach, i.e., lack of coordination may cause Indigenous groups to need to choose which department(s) to prioritize for engagement, at the expense of accommodation measures led by other departments.
• Line departments do not always inform the federal family before they engage in discussions. This makes it challenging to coordinate and weakens the federal government’s ability to reduce burden or engagement fatigue for Indigenous groups.
• NRCan-P4PO is not always present at meetings led by OGDs, which limits its ability to support horizontal messaging.
• There has been confusing and inconsistent messaging about eligibility for accommodation measures for certain Indigenous groups. This has presented some challenges interdepartmentally as well as for NRCan-P4PO external communication with Indigenous groups.

At the same time, even if successfully implemented for TMX, many departments indicated NRCan-P4PO’s ability to reduce engagement fatigue would still be limited in that it would not coordinate across other (non-TMX) federal initiatives.

Evidence suggests that NRCan-P4PO regularly engages with a large number of Indigenous groups and has met its performance targets for engagement.

Maintaining an ongoing dialogue with Indigenous groups was important for the GOC to ensure an effective, coordinated implementation of TMX Phase IV. NRCan set up a performance framework intended to demonstrate that the GOC is continuously engaging with Indigenous peoples. The evaluation team examined program documents and performance reports to assess the extent to which these targets were met. A summary of results is shown in Table 3.

In September 2019, NRCan-P4PO sent an opening outreach e-mail to the 129 Indigenous groups deemed eligible to participate in post-decision activities. The letter introduced the NRCan Phase IV Partnership Office’s role to support Indigenous groups and that of the federal departments leading accommodation measures. NRCan respondents perceive that they made “best efforts” to engage coming out of Phase III. They wanted to make sure that Indigenous groups were well informed. However, a few departments’ respondents agreed that this initial outreach could have been better coordinated with a portal where all the information was in one place.

Table 3: Progress Against NRCan-P4PO Targeted Results

Outcome	Target Footnote a	Result
NRCan-P4PO is actively reaching out to groups to inform them of 1) its role and 2) opportunities to participate and engage in post-decision activities.	100% of Indigenous groups initially contacted by NRCan-P4PO at least once by December 2019.	Achieved.
The Government of Canada can demonstrate that it is continuously engaging with Indigenous groups and is aware of their concerns and/or interests related to the Project.	NRCan-P4PO engages with 60% of Indigenous groups on an ongoing basis and is tracking ongoing progress of that engagement. Footnote b	Reported to be achieved. Footnote e
GOC engagement with Indigenous groups on post-decision activities is substantive. Footnote c	100% of Indigenous groups are offered multiple documented opportunities to engage in post-decision opportunities.	Reported to be achieved. Footnote e
GOC implements commitments made during Phase III consultations.	100% of one-off commitments, made by the Crown during Phase III consultations are implemented. Footnote d	In progress.

NRCan also targeted NRCan-P4PO to provide the totality of the eligible Indigenous groups with multiple opportunities to engage in post-decision activities between 2019 and March 2022. For the purpose of its performance target, NRCan-P4PO defined “engagement” to include any form of one-way or two-way communication with Indigenous groups. Using this broad definition, the examination of program documents suggests that a substantial number of Indigenous groups were given multiple opportunities to engage, following the initial outreach by NRCan-P4PO (see Figure 10). As of September 2021, documents indicate that NRCan-P4PO is engaging on an ongoing basis with up to 122 of the 129 Indigenous groups (95%). There is clear evidence of at least 7 Indigenous groups not being regularly engaged for various reasons (e.g., at the request of the community).

Figure 10: Examples of NRCan-P4PO Engagement Efforts

Text Description

Infographic showing examples of NRCan-P4PO engagement efforts, consisting of four circles.

The first circle shows September 2019: Initial outreach letter to 129 Indigenous groups

The second circle shows October to November 2019: Five workshops with 67 terrestrial groups, and one workshop with 12 marine groups.

The third circle shows by July 2020: 27 bilateral meetings with 16 communities.

The fourth circle shows March 2021: COVID-19 engagement letter to 129 groups.

Source: Program Documents

This engagement involved one-way communication from NRCan-P4PO to Indigenous groups (e.g., letters and invitations) and two-way communication (e.g., meetings and workshops) that allowed groups to participate and provide input into the implementation of the accommodation measures. Dozens of bilateral meetings with staff from NRCan-P4PO and other federal partner departments also provided opportunities for Indigenous groups to express their views, needs or concerns about proposed accommodations measures. However, the evaluation process could not firmly conclude on the depth of Indigenous groups’ engagement, nor determine the extent NRCan-P4PO has incorporated Indigenous peoples’ concerns or needs following interactions with federal departments.

NRCan-P4PO staff is also tracking progress against the one-off (bespoke) commitments made by the Crown during Phase III consultations. Program tracking tools suggested that eight of the 22 commitments (36%) were in progress as of August 2021, while 14 out of 22 (64%) were completed in the same period. Some Indigenous groups were reported to be very proactive and invested in the process to see these completed. NRCan-P4PO is now in the process of reviewing and tracking progress and developing a work plan as to how to reach completeness. However, in some cases, commitments were vague or may be considered ongoing; it will be more challenging to know when these are “complete”.

Notwithstanding these efforts, survey results suggest that only about 41.7% of working level engagement or initiative leads and departmental coordinators indicated that NRCan-P4PO was somewhat effective (16.7%) or very effective (25%) in supporting Indigenous groups in ongoing engagement regarding TMX accommodation measures.^{Footnote 8} About 20% of engagement or initiative leads reported that NRCan-P4PO was somewhat ineffective. In general, respondents indicated that one-off communications (e.g., initial reach out to groups) have been effective whereas products intended as either ongoing or periodic updates (e.g., TMX Indigenous portal) were ineffective in facilitating the engagement of Indigenous groups. Open-ended responses indicate a perception that NRCan-P4PO is “bogged down” by process (e.g., approval delays) and suggested improvements are required in the timeliness and consistency of communications from NRCan to Indigenous groups.

Indigenous groups appear to be unclear on the role of NRCan-P4PO and divided about the extent to which they have been engaged.

The evaluation team gathered Indigenous groups’ perspectives to assess the extent to which they have been effectively engaged by NRCan-P4PO (see Box 5). Logically, to speak to the effectiveness of P4PO, Indigenous groups must first be aware of its existence and be at least somewhat familiar with its mandate. To this end, the evaluation sought to gauge Indigenous respondents’ level of knowledge concerning NRCan-P4PO and its mandate.

Box 5: A Framework for Evaluating Indigenous Engagement

Source: Kelly Sears Consulting

The evaluation’s random sample included at least 16 out of 49 Indigenous groups for which the contact information for key representatives provided by NRCan-P4PO was outdated. Maintaining contact information can be a challenge given turnover of key staff within Indigenous groups, but a lack of up-to-date contact information suggests deficiencies in the effectiveness of P4PO’s outreach.

Of those who did participate in the evaluation:

• Despite its lead role in initial outreach to Indigenous groups on Phase IV, no Indigenous respondent provided a resounding response that they had learned about TMX accommodation measures from P4PO. Some indicated that they may have heard about opportunities to participate from other NRCan sources (e.g., IPO-W), from the TMX-IAMC, or other external sources (e.g., directly from TMC or via other GOC initiatives).
• Respondents for many groups (50% or above) reported some level of awareness of P4PO’s engagement tools (e.g., meetings, emails, Indigenous Portal). However, only a few (less than 50%) were confident that this engagement was indeed from NRCan-P4PO (vs. a partner department). A few also stated that while they knew that NRCan-P4PO had engaged with their group or community, they had not heard from NRCan-P4PO for an extended period.

While a limited sample, input from these respondents also confirmed the evaluation’s finding that NRCan-P4PO has not delivered on its mandate to become the single window on TMX for Indigenous groups. Overall, regardless of their ability to recall engagement with P4PO, the majority expressed confusion regarding who was involved in the TMX Phase IV (i.e., departments, agencies, and individuals). Many indicated they did not know where to look or who to contact if they had a question or concern about the TMX accommodations measures and were unaware that NRCan-P4PO could help facilitate these connections. None of these respondents reported having reached out to NRCan-P4PO to facilitate the making contact with a federal department leading a particular accommodation measure. Instead, the Indigenous informants stated that, if necessary, they would simply contact the lead department directly. Partner departments perceived direct “grassroots” engagement on specific accommodation measures to be building positive relationships between OGDs and Indigenous groups.

Overall, Indigenous groups rated the effectiveness of P4PO’s engagement as “poor” to “fair”.

Overall, the evaluation found that respondents from Indigenous groups that were not familiar with the P4PO’s mandate provided negative feedback on its effectiveness simply because they were not aware of NRCan-P4PO. Otherwise, several respondents explained that they gave low ratings as they perceived this team to provide too much, generic information – i.e., insufficiently tailored for their respective community’s needs. While communications mechanisms, particularly e-mails and letters, were perceived as effective, the frequency of these communications and amount of information were perceived to be overwhelming. These respondents indicated they are participating in multiple federal and provincial programming initiatives and are challenged to cope with the enormous volume of email traffic, and as a result often lacked the capacity (e.g., time and resources) to review non-targeted communications. Similarly, while workshops were perceived as effective, the ability to build relationships because of these events was negatively affected by frequent federal staff turnover and the large number of attendees. The TMX Indigenous Portal was perceived to be a useful communication mechanism but introduced too late.

For the GOC, applying a distinction-based approach to Indigenous consultations on TMX was important to ensure the full participation and ongoing involvement of Indigenous peoples in post-decision activities. To this end, funding approvals indicated that NRCan-P4PO would tailor its process based on each Indigenous group’s concerns, interests, and project-related impacts (as identified in Phase III Consultations). The evaluation team examined the tools, activities and processes the Phase IV Partnership Office has put in place to affect this direction.

The NRCan-TMX Engagement Strategy (2020, 2021) introduced six types of groups based on considerations affecting access and participation in TMX initiatives and provides guidance on how to engage on accommodation measures for each of the six groups. The evaluation found evidence of some of these activities in practice. The engagement activities that NRCan-P4PO developed in collaboration with partner departments and (to lesser extent) with Indigenous groups were shaped to respond to groups’ specific impacts, with the 129 Indigenous groups divided into in three broad groups: Terrestrial, Marine, and combined Marine / Terrestrial. When required, NRCan-P4PO convened and led multidepartment meetings to ensure that specific groups dealing with different accommodation measures could easily access different federal organization through the same venue. By contrast, there is also evidence that NRCan-P4PO engaged in bilateral meetings when preferred by Indigenous groups. The NRCan-P4PO co-led several workshops with terrestrial groups while setting up bilateral meetings with marine groups who have expressed the desire to engage through meetings rather than workshops.

Despite these efforts, 50% of federal survey participants who reported knowledge of engagement on TMX neither agreed nor disagreed that the P4PO’s engagement approach has been sufficiently tailored to the needs and capacity of each Indigenous community. Only 30% somewhat agreed that it had been sufficiently tailored. Respondents from NRCan-P4PO indicated that a tailored approach was not required in all cases. This statement was somewhat echoed by some key informants from partner departments who indicated that smaller Indigenous groups may be more overwhelmed by federal initiatives and be harder to reach or require more support. While this suggests that an issue that needs to be addressed, it is not clear that respondents would consider this to be the responsibility of NRCan-P4PO. Those partner departments that directly engaged with Indigenous groups stated that they are already tailoring their approach to engagement to the best of their ability. See the next section of this report for more information on Indigenous capacity and capacity funding.

Indigenous groups provided diverse perspectives on the meaningfulness of P4PO’s engagement.

The majority of surveyed engagement leads and departmental coordinators disagree that there has been sufficient time for meaningful dialogue with Indigenous groups, with a few reporting that they had heard concerns from Indigenous groups about the frequency and depth of engagement or groups’ capacity to engage.

Respondents from Indigenous groups that were aware of NRCan-P4PO’s mandate expressed diverse opinions on the meaningfulness of engagement. Most Indigenous respondents reported that relationships emerging from Phase III were not strengthened and/or maintained, indicating a need for the relationship to move from one of one-way communication to full (two-way) participation. A few indicated that NRCan-P4PO enabled meaningful engagement, perceiving NRCan-P4PO to have made an honest attempt to provide many opportunities to participate. These respondents perceived that NRCan-P4PO did a “good” to “very good” job of informing groups about TMX accommodations measures, with one indicating that any inquiries were expediently and professionally dealt with by P4PO. Others reported that NRCan-P4PO did not provide sufficient opportunities to their groups to participate in the discussions about TMX accommodation measures. Even those who gave good ratings did so with a caveat that the experience was still generally perceived to be overwhelming.

When compared to other GOC initiatives, NRCan-P4PO did not stand out as a model of coordination.

A review of the relevance and effectiveness of specific accommodation measures was outside the scope of this evaluation. However, the evaluation did examine the extent to which coordination affected by NRCan-P4PO achieved the intended result of increasing the ease of working with the GOC across TMX accommodation measures. As such, informants were asked to provide their perspective about the clarity and accessibility of the application process related to the TMX accommodation measures.

Evaluation findings suggest that there is still room for improvement. Overall, most Indigenous respondents (75% and above) perceived the TMX application process to be clear and accessible, but with some caveats that the process often required assistance from experts engaged by their groups. Respondents from a few groups perceived that TMX Phase IV was less well-coordinated compared to other GOC funding programs because there were too many players involved in the Project. A few groups saw no difference, stating that it was just as well (or not well) coordinated as other GOC initiatives.

What We Found: Facilitating and Impeding Factors

The evaluation examined whether there were factors that facilitated or impeded the effectiveness and efficiency of TMX Phase IV coordination.

Many contextual factors have or could influence effective and efficient delivery of Phase IV coordination.

The evaluation identified numerous contextual factors impacting the effectiveness of TMX Phase IV coordination (see Table 4). The ability to respond, particularly to external factors, will depend on the program’s sphere of influence with some factors inherent to the Project’s nature and scope (e.g., number of Indigenous groups engaged). Other factors relate to the program design and implementation, over which NRCan and partner departments have more control or influence.

Table 4: Contextual Factors That Have Influenced TMX Phase IV Coordination

TMX Project Profile	TMX is a high-profile project with high political interest or sensitivity. The project also has important implications for other GOC priorities, such as the climate change agenda. Failure to implement the Crowns' commitments to Indigenous peoples related to TMX could also negatively influence the advancement of reconciliation. This creates high expectations for performance (i.e., to work quickly and effectively), particularly within NRCan-TMX. Phase IV implementation of TMX is unprecedented in scale and scope and related action must proceed with awareness that it may set a precedent for other major projects.
Horizontal results implications	While not a horizontal program, there are multiple federal departments involved in TMX Phase IV implementation including but not limited to partner departments. There are horizontal policy and operational implications to this situation: success on TMX requires that all departments succeed, not just one.
Concurrent complexity, and varying levels of capacity to manage this complexity	It is hard to fully anticipate needs and risks for any major project. Given that Indigenous groups are not homogenous, it is typically challenging to design GOC programs with results (targets) that are responsive to or reflect the priorities of all those implicated. TMX is particularly complex, with a broad geography that engages numerous Indigenous groups, multiple departments, and many other partners and stakeholders (e.g., industry, provinces, coastal communities, etc.). Indigenous groups also face many competing priorities, including overlap of TMX Phase IV implementation with other federal programs (e.g., Oceans Protection Plan) and major projects (e.g., Roberts Bank Terminal 2). The result is numerous bilateral and multi-level conversations with Indigenous groups where the federal government may compete with itself to get their attention, which taxes capacity and adds to engagement fatigue.
Ownership of TMX	Canada’s current ownership of the pipeline also necessitates coordination with Finance Canada who, through the Canada Development Investment Corporation, ensures accountability for this public investment.
Timeline and pace of implementation	Hundreds of millions of dollars in programming were provided within the five-year funding window for Phase IV implementation. However, while some initiatives are expected to be renewed, existing funding approvals are time limited (sunset between March 2022 to March 2025, depending on the initiative). There is a shared sense of urgency to deliver but not all departments or their initiatives are moving at the same pace. There is also a need to coordinate engagements but not all partner departments or Indigenous groups are ready to engage at the same time. At the time of the evaluation, the deadline for some of this funding was rapidly approaching without clear direction or a funding decision on renewal. Some Indigenous respondents indicated that this was impacting their confidence that the Government will continue to do the work and, by extension, was also perceived by some federal engagement leads surveyed to be limiting uptake of accommodation measures.
Unforeseen events and competing priorities	There were many unforeseen events that have negatively impacted project delivery, including a major health crisis (COVID-19) and natural disasters (e.g., wildfire, flood). Other events, such as the residential school gravesite findings, have also greatly affected Indigenous groups and the attention and resources they have available to engage on accommodation measures. Federal respondents disagreed on the extent to which delays in implementation could be attributed to unforeseen events but agreed that these events have amplified existing pressures and impacted Indigenous groups’ capacity to respond to these competing priorities.

Most federal interviewees agreed that coordination on TMX has been effective in consideration of this broader context. Respondents (particularly in partner departments) indicated that NRCan-TMX may have lacked strong awareness of the full spectrum of GOC priorities and initiatives at the outset of implementation; this was reported to have improved with time and NRCan-TMX is now an important observer / participant in related discussions. Additional departments (e.g., Impact Assessment Agency of Canada [IAAC], Crown Indigenous Relations and Northern Affairs Canada [CIRNAC], etc.) now attend NRCan-SPC and NRCan-P4PO meetings to help improve the flow of information and coordination of engagement.

The TMX Phase IV coordination function and NRCan team have facilitated the delivery on TMX Phase IV.

The following factors were reported to facilitate the effectiveness and efficiency of Phase IV coordination:

• Coordination Function. Federal respondents generally agreed that the existence of a coordination function itself was a factor critical to success, e.g., allowing departments to effectively gauge progress. Coordination also provides a forum for transparent decision-making, where the rationale behind decisions can be openly discussed and issues can be resolved (adding value regardless of whether OGDs agreed on the final decision). These discussions are perceived to benefit from the inclusion of other federal partners (e.g., IAAC, CIRNAC, central agencies).
• Quality of the NRCan Team. The quality, commitment, and support offered by the NRCan team is a factor reported to enable the effectiveness of TMX Phase IV. NRCan’s team was perceived to be highly functioning, with clear points of contact, strong corporate knowledge, and dedication to the work. Within NRCan and across partner departments, most respondents reported having a positive working relationship with NRCan-SPC and NRCan-P4PO.
• Ongoing Senior Management Support, including participation in interdepartmental governance structures, was perceived to be a critical facilitating factor (albeit one some indicated was not always present).

Engagement leads recognized the quality of the support they received from NRCan-P4PO at the ground level. Despite this, most respondents from Indigenous groups (75% or more) perceived that NRCan-P4PO lacked the skillset required to engage effectively (with the caveat that less than 50% of respondents were confident that their engagement was indeed from NRCan-P4PO and not a partner department). Overall, they reported challenges with building and maintaining a relationship exacerbated by frequent federal staff turnover. Interactions with NRCan-P4PO were perceived to be sporadic and/or negatively impacted by a lack of timely follow-up on feedback provided by the community. Some groups reported that it was challenging to keep track of the federal staff and contacts, including but not limited to NRCan-P4PO. As previously noted, groups also perceived that communications were not sufficiently tailored or plain language. Conversely, one respondent (representing three communities) was very positive about the P4PO’s engagement skills; this respondent was a consultant with pre-existing expertise in administering federal grants and contributions on behalf of communities.

Respondents from a few groups (less than 50%) stated that their pre-existing relationship with NRCan related to the TMX project somewhat negatively affected their assessment of P4PO. These groups noted that there were ongoing, unresolved issues with NRCan that needed to be addressed and that their priorities were not being well reflected in the available accommodation measures.

Several factors impede the effectiveness of TMX Phase IV coordination, many with particular implications for Indigenous engagement.

Table 5 provides a summary of the factors found to impede the effectiveness and efficiency of Phase IV coordination, some of which have been discussed previously. A few of these factors are internal, with a negative impact on NRCan and partner departments. These include workload pressures and the level of resources allocated for coordination. Many of these factors also have a direct, negative impact on Indigenous groups already experiencing engagement fatigue.

Table 5: Factors Impeding the Effectiveness of the TMX Phase IV Coordination

Among the factors identified in Table 5, those with the greatest impact on the effectiveness of NRCan-SPC were perceived to be GOC capacity and the federal decision-making and accountability regime. There is general agreement that decision making in coordination of Phase IV implementation has been slower than expected, with delays created in part by extended timelines for review and approvals. Requests for information from NRCan-SPC to partner departments are often communicated as being urgent, but the time delay to effect decisions or actions in response to this information does not always clearly demonstrate that such urgency was required. This creates push back against the NRCan-TMX teams, which negatively affects both performance and job satisfaction (which can further compound pressure by leading to staff turnover).

By contrast, key factors impeding the effectiveness of NRCan-P4PO relate more so to the federal government’s and Indigenous groups’ capacity for (coordinated) engagement. Findings from the evaluation’s Indigenous engagement agree in this regard with early analyses of engagement obstacles completed by P4PO. Respondents from most groups (50% or more) stated that their capacity limited meaningful engagement and prevented them from taking advantage of all the accommodation opportunities available, indicating that there were important foundational capacity issues that needed to be addressed including:

• Human capacity – difficulty in finding, training, and retaining qualified employees to manage the large workload associated with administration of accommodation measures, particularly given the time-limited nature of funding.
• Physical capacity – a need for basic infrastructure that would enable effective participation such as office space to house staff, equipment, good internet connections, etc.
• Information capacity – difficulties faced by Indigenous groups in gathering the information they need to inform their decision-making, including about the Project itself (e.g., construction and project updates from TMC) and challenges in accessing existing data within their traditional territory.

Capacity funding to support participation was included in the design of some accommodation measures. Others, such as TC’s Enhanced Maritime Situational Awareness (EMSA) Initiative, lacked funding for proposal development as well as capacity funding for eligible groups to adopt and implement the EMSA accommodation measure. This makes it difficult for groups to actively participate. Federal interviewees also noted that capacity funding was unavailable to support participation in the implementation of CER recommendations, which they perceived to be a gap.

In addition to capacity funding linked to specific accommodation measures, NRCan received $1.5M in G&Cs to support Indigenous groups’ participation in engagement and consultation activities with the Crown. The availability of this “transition funding” was not communicated to Indigenous groups on a proactive basis but was rather to be used on an as-needed basis to bridge gaps identified by NRCan-P4PO. As of March 2022, only about 30% of this funding had been spent. NRCan-P4PO has recognized and made attempts to respond to a need for additional capacity for Indigenous groups, but there is room for further improvement (see Box 6).

Box 6: Effectiveness of NRCan-P4PO Capacity Support

While some Indigenous respondents reported having received capacity funding to engage, others indicated they were still waiting to receive approved funds from the responsible department(s) or had used their own resources to participate. Regardless, most noted that the current approach to providing for immediate capacity needs does not address foundational capacity issues or allow for sustainability in Indigenous groups’ capacity to engage with the GOC in the future (on TMX and beyond). One community stated that the short-term capacity funding provided might inadvertently harm the groups as it presents a substantial risk of “overextension” (i.e., inability to maintain investments in staff, infrastructure, and equipment after the funding ends). Some federal respondents also questioned whether capacity funding was sufficient or the right type of support in all cases given the needs to be addressed, e.g., there may be a more effective way for federal departments to ensure that Indigenous groups have the scientific and technical capacity required to access the accommodation measures.

There is evidence that NRCan-TMX has effectively responded to some factors within the program’s sphere of influence that were likely to impede effectiveness and efficiency.

The evaluation examined the extent to which the NRCan-TMX coordination structures were effective in addressing identified impediments to effectiveness and efficiency. The evaluation found several instances where NRCan-TMX has identified and addressed these impediments. To some extent, this is part of NRCan’s role in federal coordination. For example, NRCan-TMX was perceived to be very responsive to the needs of senior management. The staff is proactive in identifying emerging issues and providing information required to respond.

Interviewees from all departments also pointed to examples of where NRCan-TMX and/or partner departments recognized hindering factors and took responsive action. Some of the course-correction actions that were underscored during the interviews with federal government informants are as follow:

• There was a lack of clarity around one-off commitments made during Phase III consultations. NRCan reviewed and confirmed responsibility for bespoke commitments and communicated this to OGDs.
• NRCan-TMX and OGDs have adjusted resource allocations from the initially planned allocation to respond to the actual workflow, including the number and level of staff.
• Roles and responsibilities have been periodically reviewed and adjusted to ensure that coordination activities remain aligned. This does not mean that further improvements are not possible. For example, some federal interviewees noted a separation between NRCan-P4PO’s marine and terrestrial engagement teams that served well at the outset (as accommodation measures were very different for each) but can now maybe be streamlined.
• The DG+CC was formed to help with the flow of information between the working and senior levels. Having perceived an issue with the effectiveness of these meetings, NRCan-TMX reported that it has started having informal meetings with counterparts in OGDs to get a sense of clients’ needs to provide better support. Program managers indicated that they may need to adjust their assumptions on the value of activities and be clearer on what activities are required and at what level of responsibility.

The delivery of the TMX-Phase IV and the current evaluation process were adversely affected by several unforeseen events (e.g., COVID-19 and natural disasters) that were not considered in funding approvals. There is evidence that NRCan’s coordination structures tracked the events’ impacts on each community to inform federal engagement (e.g., changes in key contacts) and adjust their own approach to demonstrate sensitivity to pressures introduced. Data also suggest that NRCan coordination structures and OGDs were able to effectively shift in a virtual work in response to COVID, but this was not necessarily true of all Indigenous partners. Particularly with the increased likelihood of climate events impacting natural resources projects, there is a need to further include contingency plans in GOC program designs. Reprofiling exercises and assessments of departmental responses that occurred in response to COVID-19 could provide valuable baseline information to consider when developing contingency plans for such unforeseen events.

What We Found: Ongoing Need

The evaluation examined whether there is an ongoing need for NRCan’s P4PO and/or SPC teams given progress on achieved to date by the proponent on construction schedule and departments on implementation of Phase IV initiatives.

NRCan and partner departments generally agree on the continued need for the TMX Phase IV coordination structures.

Most key informants indicated there was an ongoing need for the NRCan-TMX coordination structures to support work required to complete implementation of accommodation measures. Federal respondents agreed that where responsibility for coordination is ultimately located is not important, provided it continues to ensure the Crown’s commitments made during Phase III consultations and Phase IV are being met. While each member of the federal family has its own specific mandate/role, most respondents indicated that NRCan may currently be best placed to lead on coordination across departments.^{Footnote 9}

The need for federal coordination on TMX could endure for an extended period. Most departments perceived an ongoing need for whole-of-government coordination at least as long as the accommodation measures are still being implemented and/or construction is ongoing. However, most also extended the need for a coordination function for the transition from construction to operation (next 5-10 years). The need for a dedicated regulatory-focused function of NRCan-SPC was perceived to diminish once the project is completed, the pipeline operational and the emphasis on coordination shifts from policy and regulation to operations and maintenance. However, a few departments perceived that there will likely still be a requirement to revisit TMX’s regulatory and policy environments at intervals.

An unexpected theme that emerged from the interviews with the federal key informants relates to the corporate knowledge that was gathered through the implementation of the Phase IV coordination activities. NRCan-SPC and NRCan-P4PO were perceived by some to have developed corporate knowledge (e.g., repositories of Indigenous contact information, key issues faced by Indigenous groups and their resolution, etc.) that needs to be preserved. This could serve as a baseline for any new initiative, provided that this information was centralized and updated.

There is stronger support towards maintaining NRCan-SPC than NRCan-P4PO.

NRCan-SPC was perceived to facilitate oversight and accountability on the TMX project as a whole, ensure commitments are collectively being met and risks are mitigated, and coordinate discussion and dispute resolution on cross-cutting issues. As shown in Figure 11, there was strong support for this ongoing role among surveyed departmental coordinators, with almost 68% indicating an ongoing need. The strongest support came from ECCC. Engagement leads and respondents from other departments were either unsure or equally split between those supporting and rejecting an ongoing need for coordination. Some suggested that departments that are adequately resourced could provide similar coordination via their own departmental TMX Secretariats.

Figure 11: Perspectives on Ongoing Need for NRCan-TMX Coordination Structures

Text Description

Infographic showing perspectives on ongoing need for NRCan-TMX Coordination Structures, consisting of four circles.

The first two circles show the percentage of respondents who indicated there is an ongoing need for SPC role. 33% of Engagement Leads and 67% Departmental Coordinators indicated there is an ongoing need for SPC role.

The last two circles show the percentage of respondents who indicated there is an ongoing need for P4PO role. 30% of Engagement Leads and 63% of Departmental Coordinators indicated there is an ongoing need for P4PO role.

Source: Evaluation Survey

For senior managers, the key question relates not to continued need for NRCan-SPC but rather the resource levels required to sustain oversight and coordination. While they indicated processes could be streamlined, few interviewees perceived that a significant decrease in coordination would be appropriate at this stage of project implementation. A few respondents (particularly those responsible for marine initiatives) underscored the important contribution of these initiatives to broader GOC objectives (e.g., marine safety). Ensuring that these objectives are delivered in a coordinated manner is perceived to contribute to reconciliation.  

There is weaker support for the continued need of the functions fulfilled by NRCan-P4PO. While nearly all respondents (both within and external to NRCan) indicated that they perceived an ongoing need for all or part of its role in federal coordination, they were divided on whether there was an ongoing need for its in Indigenous engagement now that OGDs have built their own direct relationships with Indigenous groups. Many respondents indicated that NRCan-P4PO’s direct role in engagement could be reduced. There were also differences in perspective among departments, with less support for this ongoing engagement role from CCG and almost no support from DFO.

Indigenous groups perceived an ongoing need for (stronger) federal coordination on engagement.

Many groups (50% or above) agreed that a “single window” was a good concept and indicated a need for this sort of coordination, particularly given the size and resources of the GOC as compared to Indigenous groups. They perceived that this would simplify the process for groups. Respondents for several groups indicated that in a (real or perceived) absence of this role, they needed to hire consultants to help them coordinate participation. However, only one Indigenous respondent saw the current role of NRCan-P4PO as redundant to that currently played by departments leading the TMX accommodation measures. A few respondents suggested that NRCan-P4PO could play a larger role in improving engagement approaches across partner departments. However, respondents also underscored the need for a “solid team” with low staff turnover so that meaningful relationships and trust could be cultivated; they perceived NRCan-P4PO to be insufficiently resourced to be able to do so. A few respondents stated that if NRCan-P4PO continued to function as it is, they did not see the value of it.

While recognising that the current model was not ideal, neither federal nor Indigenous respondents identified alternatives to the present engagement approach.

What We Found: Lessons Learned

Throughout this report, the evaluation has identified the areas for improvement that could support ongoing delivery of TMX Phase IV and benefit decision-making on future natural resources projects. These include:

• Horizontal Governance. NRCan’s responsibility for coordination does not match its authority. It is important that there be leadership, direction, and support to create a united vision/approach across departments. There is thus a need to improve the definition and internal communication of roles and responsibilities to ensure effective coordination and implementation of federal TMX initiatives. To minimize conflicts, this would include clearer delineation of where NRCan has the responsibility to give direction and where it has the lead in the dispute resolution process, as well as clearly defining how all OGDs are expected to contribute to this process (including timely approval of public communication products). Through a review of relevant evaluations and lessons learned documents (Appendix 4), the evaluation identified a best practice in effective coordination to provide the lead department with a more robust mandate to reduce overlap and duplication with other departments. It is also key to provide the lead department with authority to facilitate the achievement of the intended outcomes without interfering with partner departments' legislative roles and responsibilities.
• NRCan Roles and Responsibilities. Evidence suggests there may be a need to improve linkages between NRCan-TMX (P4PO) and IPO-W, responsible for the TMX-IAMC. NRCan respondents indicated that improvements in the Nòkwewashk sector’s internal organization are ongoing.
• Federal Workload. The level of effort / resources required within partner departments to support coordination needs to be more sustainable. This could be achieved by either increasing resources and/or streamlining activities. Suggestions for how this could be achieved included:
  • Reduce NRCan’s weekly report to senior management to bi-weekly or quarterly.
  • Provide an advance view of workload expectations across departments that support coordination of Phase IV implementation.
  • Improve timing for distribution of materials to working level committees.
  • Review internal tools and processes (e.g., frequency and mix of coordination products) to reduce burden, avoid duplication, and provide a more holistic view of (timing for) expected input.
  • OGDs can also review their internal structure to ensure that funding requests are aligned with expected resource needs.

NRCan-TMX has been responsive to this feedback. As of May 2022, reports to senior management have switched to bi-weekly updates. As of July 2022, an internal review and streamlining along with a forward calendar were also now in the final stages of implementation.

• Communications. There is a need for more timely, targeted communication tools and content for both internal and external products. GOC respondents further indicated that external communications should be proactive and guided by a defined (interdepartmental) strategy, with clearer delineation of roles and responsibilities.
• IT Challenges. NRCan and partner departments need to collaborate to ensure that input to tools is accurate and updated. This would be facilitated by better access to IT solutions shared across departments, and the direction and skills to use these effectively.
• Indigenous Engagement. All lines of evidence agree that GOC needs to improve the way it coordinates engagement with Indigenous groups. While many suggestions for improvement are beyond the current scope of TMX Phase IV implementation (see lessons learned, below), interviews identified a few improvements that could be actioned immediately. This includes suggestions to create and promote more opportunities to build Indigenous groups’ access to the resources and skills required to liaise effectively with the GOC and maximize the use and benefits of accommodation measures. It also includes suggestions for relationship building which would necessitate ensuring the continuity of key contacts (or an approach to mitigate the impact of staff turnover).
• Performance Metrics. GOC respondents were of the view that NRCan’s current performance metrics for TMX Phase IV coordination focused on administrative outcomes (e.g., number of meetings or outreaches) that are not meaningful for Indigenous groups.

The evaluation highlighted key lessons learned for future major projects.

Key informants identified several lessons they have learned from the early implementation of the TMX Phase IV that could be applied to current and future government initiatives. These include:

• Engagement post-decision. While a substantive level of effort may not always be required, TMX Phase IV has demonstrated that there is a lot of work that can be done post-decision on major projects. This would be guided by the degree to which accommodations are offered or the extent to which the government has committed to implement other programming (such as recommendations) in support of a project decision. Lessons learned from the coordination of TMX implementation could be used to help guide project management on how to resource future post-decision activities, and critical tools or processes to put in place (e.g., public communication strategy). There may also be lessons learned specific to engagement, including on how to engage with non-s.35 rights holding Indigenous groups. Specific to NRCan-P4PO, there is some concern within NRCan and partner departments that there will be a loss of corporate knowledge / expertise that was built if not somehow captured.
• TMX precedent. Most federal respondents perceived the need for horizontal coordination on major projects to be increasing. Major natural resources projects are a key issue for reconciliation. Indigenous groups have increasing expectations for effective and efficient delivery (and accountability to demonstrate that this has occurred). TMX has created a very large process, with a strong oversight mechanism. However, several GOC respondents perceived that extending this model to other major projects would be cost-prohibitive. They see a risk that TMX may set a precedent that is not sustainable.
• Reconciliation. NRCan is the lead department for TMX but it is not its role to be a central portal for Indigenous reconciliation writ large, in the broader GOC agenda. However, there is still a need for coordination to see that these issues are addressed. There is thus an ongoing need for the GOC to be more efficient in its engagements with Indigenous groups; it is paramount to work collaboratively with and leverage the expertise of other federal organizations and with provincial/territorial governments. This will require additional innovation and different approaches that enable strong coordination without slowing down implementation.
  The relationship with Indigenous groups also needs to be consistent, positive, and meaningful (attitude matters). Indigenous groups are still building their capacity to engage effectively and there is thus also a need to provide groups with enough resources to support self-empowerment and full participation into TMX-related opportunities. This includes building on existing strengths and facilitating the development of necessary skill sets and relationships to effectively address existing and emerging social, economic, cultural, and political realities such that Indigenous peoples can realize a sustainable future from their lands and resources based on their goals and priorities.

• Good internal and external communication is critical. Open, transparent, and respectful communication is important to build consensus on cross-cutting issues among departments. It is important that departmental coordinators are linked into the programs, e.g., by attending meetings hosted by those implementing the accommodation measures. Horizontal awareness of the issues discussed is important to inform the work of coordinators and to help programs avoid issues. In developing external communication products, coordinators need to make sure they understand the material or rely on the experts to provide wording.

These lessons learned do not lead to easy solutions. Previous program reviews, including those related to major projects management and impact assessment and regulatory processes, indicate that that both engagement of Indigenous peoples in major projects and coordination of multiple departments are challenging and that the issues encountered by TMX Phase IV are not unique (see examples in Appendix 4). While the perceived need for horizontal coordination on major projects is increasing, to date they have faced persistent issues related to clarity of roles and responsibilities, meaningfulness of performance metrics, challenges with staffing and underestimation of time required for project delivery, and inconsistent engagement and communications approaches leading to engagement fatigue. Overarching recommendations generally agree with suggestions made above to address areas for improvement on TMX. Lessons learned from this evaluation or other reviews and stock taking exercises completed for TMX Phase IV implementation should be reviewed collectively and shared across the federal government to inform future initiatives.

Conclusion

Overall, the evaluation found that the roles of NRCan’s TMX coordination structures align with planned design and that they have been relatively well defined and consistent, despite changes made to account for evolving roles. There is a continued need for the TMX Phase IV coordination and to preserve the corporate knowledge built during the implementation of TMX Phase IV. This need will endure at least as long as the accommodation measures are still being implemented and/or construction is ongoing. While NRCan may currently be best placed to lead on coordination across departments, where the coordination function is located is not perceived to be as important as ensuring that the Crown’s commitments made during Phase III consultations and Phase IV are being met.

NRCan-SPC has created effective tools and processes to support federal coordination. Key governance bodies are well supported. While consensus on decisions was not always possible, NRCan-SPC is perceived to be open and transparent in working to resolve cross-cutting issues in the face of divergent views. The team’s regulatory function is generally perceived as being effective in fulfilling its intended role. However, there are issues related to efficiency. Partner departments perceive workload related to TMX coordination, including processes used to create and update coordination products, as unsustainable. They report having underestimated the level of effort that would be required for coordination and reporting.

Federal respondents largely agreed that NRCan-P4PO has to some extent been effective in its federal coordination role, but not in engagement. While evidence also suggests that NRCan-P4PO has met its performance targets for initial outreach and engagement, these metrics do not provide information on the meaningfulness of engagement. NRCan-P4PO has not fully fulfilled its planned role as a “single window” for Indigenous engagement. As a result, while partner departments have worked to develop or maintain relationships with implicated Indigenous groups, the federal approach to engagement on TMX has not been effective in reducing engagement fatigue. When compared to other GOC initiatives, Indigenous respondents indicated that TMX Phase IV implementation did not stand out as a model of coordination on engagement.

There is evidence that NRCan-TMX has been responsive to some factors within the program’s sphere of influence that were likely to impede effectiveness and efficiency. Nevertheless, the evaluation has identified areas for improvement (e.g., clearer delineation of roles and responsibilities, increased sustainability of federal workload, improved coordination for Indigenous engagement). These are highlighted where relevant throughout this report.

The evaluation also identified lessons related to engagement post-decision on major projects, as well as a need to manage the precedent set by TMX and further advance reconciliation with Indigenous peoples. These lessons learned do not lead to easy solutions but are also not unique to TMX. Related lessons learned from current and past program reviews should be reviewed collectively and shared across the federal government to inform future initiatives.

Appendix 1: TMX Initiatives and Lead Departments/Agencies

Accommodation Measures:

• Salish Sea Initiative (DFO & CCG)
• Aquatic Habitat Restoration Fund (DFO)
• Co-Developing Community Response (CCG & TC)
• Quiet Vessel Initiative (TC)
• Terrestrial Studies Initiative (NRCan)
• Terrestrial Cumulative Effects Initiative (ECCC, NRCan, & DFO)
• Marine Safety Equipment and Training Program (TC)
• Enhanced Maritime Situational Awareness (TC)

More information about each accommodation measure can be found on the TMX Indigenous Portal.

CER Recommendations:

• Recommendation 1: Advancing Knowledge in Support of Managing Cumulative Effects in the Salish Sea (DFO, ECCC, and TC)
• Recommendation 2: Annual Reporting on Progress to Address the Health of the Salish Sea (DFO, ECCC, and TC)
• Recommendation 3: Salish Sea Marine Bird Monitoring and Conservation Program (ECCC)
• Recommendation 4: Marine Conservation Area Feasibility Assessment (Parks Canada)
• Recommendation 5 & 6: Offset Project-related Underwater Noise (DFO, TC, & CCG)
• Recommendation 7: Update Federal Marine Oil Spill Response Requirements (CCG, TC, DFO, & ECCC)
• Recommendation 8: Framework for Mandatory Enhanced Tug Escorts in the Salish Sea (TC)
• Recommendation 9: Canada/U.S. Transboundary Vessel Risk Assessment (TC)
• Recommendation 10: Greenhouse Gas Reduction Measures for Marine Shipping (DFO, ECCC, & TC)
• Recommendation 11: Greater Engagement with the Indigenous Advisory Monitoring Committee on Marine Safety (NRCan, TC, & CCG)
• Recommendation 12: Engagement and Awareness to Prevent Marine Collisions (TC)
• Recommendation 13: Accelerate Enhanced Maritime Situational Awareness Initiative and the Automatic Identification System (TC)
• Recommendation 14: Develop and Deploy New Oil Recovery Technologies (NRCan)
• Recommendation 15: Review of Federal Oil Spill Compensation Regimes (TC)
• Recommendation 16: Develop a Formal Complaint Resolution Program with the Vancouver Fraser Port Authority (VFPA) (TC & VFPA)

More information about each recommendation can be found in this backgrounder.

Appendix 2: Evaluation’s Indigenous Engagement

Recognizing the importance of Indigenous perspectives to inform the relevance and performance of TMX Phase IV implementation, the evaluation considered Indigenous engagement as a separate line of evidence.

The evaluation expected challenges and potential limitations in data collection related to Indigenous perspectives. It planned to remain flexible and use a combination of activities to offer respondents different avenues to provide their input and share their views. To maximize participation, it used NRCan’s existing engagement opportunities to promote the evaluation. This included:

• A presentation about the evaluation to the TMX- IAMC in August 2021;^{Footnote 10}
• An announcement about the evaluation included in the P4PO’s August 2021 newsletter to Indigenous groups, with information on how to contact the evaluation team; and
• An announcement about the evaluation posted on the P4PO’s new TMX Indigenous Portal, including a link to a short online survey and information on how Indigenous groups could otherwise participate. The survey was open for response from October 26 to November 30, 2021.

There was no additional data collected through any of these initiatives. The option to conduct focus groups or sharing circles was also discussed but dismissed given the major disruption to Indigenous groups in British Columbia caused by forest fire and then flooding events in late 2021. Data collected related to Indigenous perspectives was thus limited to interviews, with a couple written responses received by e-mail.

The evaluation set a target of obtaining perspectives from 30 of the 129 Indigenous groups (23.3%). A subset of 50 groups was randomly selected from the 129 involved in the TMX accommodation measures. This list was reviewed to ensure it was a reasonable representation of implicated groups (e.g., region, depth of consultation, etc.) and to exclude any groups that had requested not to be contacted by NRCan-P4PO. As a result, the final number in this initial sample was 49.

These Indigenous groups were invited to participate in interviews via two rounds of outreach by the consultants hired by the evaluation team: 1) from October to December 2021 (28 Indigenous groups were contacted), and 2) from January to February 2022 (21 Indigenous groups were contacted).^{Footnote 11} Respondents were also given the option to provide written responses to the interview guide and, in second outreach, a short email survey. The evaluation team worked with NRCan-P4PO to ensure that contact information was up-to-date, and that the evaluation reached the right individuals or their delegates to speak on behalf of the community. The first two rounds of outreach resulted in a total of 10 interviews.

In a final attempt to increase response rates, the evaluation team conducted a third outreach in March 2022. The third outreach involved a targeted sample of 15 Indigenous groups with whom NRCan-P4PO indicated that it had the most interaction and/or it deemed most likely to participate in the evaluation. These respondents were again offered several means to provide feedback: virtual one-on-one interview with the consultants, written response to a short e-mail survey, or a together as a virtual workshop that would be led by the consultants. While these groups were initially contacted by NRCan-P4PO to request their participation, all follow-ups were conducted by the evaluation team. It was made clear at all phases of engagement that responses collected for the evaluation were to be sent directly to the evaluation team and were not to be shared directly with P4PO. This final outreach resulted in two additional interviews.

In total, the evaluation received input from representatives in 12 Indigenous groups (9.3% of the 129 or 40% of the evaluation’s target). In addition to possible engagement fatigue, participation was likely impacted by the timing of evaluation. Contextual factors (e.g., ongoing COVID-19, natural disasters, residential school discoveries, the 2021 federal election, numerous band elections) coincided with the data collection phase of the evaluation.

All participating respondents were from British Columbia; none were from Alberta. However, the mix of actual respondents was otherwise relatively well distributed across the nature of accommodation measures (terrestrial or marine) and past level of engagement with NRCan-P4PO.

Note: Categorization of Indigenous groups in this table is as defined by NRCan-P4PO in the data provided to the evaluation team. The evaluation team did not seek to validate this categorization.

Appendix 3: NRCan-P4PO Tools

P4PO Engagement Tracking Tools

Text Description

The first section shows P4PO Engagement Tracking Tools:

• Phase III Commitments – Ongoing. Update status concerning the bespoke commitments made by the Crown during the reinitiated Phase III consultations.
• TMX AMs Engagement Tracker – Archived. Information on approved capacity agreements/project funding, and engagement status for P4PO and OGDs.
• Marine & Terrestrial Meeting Tracker – weekly update Number of meetings with specific Indigenous communities types.

P4PO Coordination Tools

Text Description

The second section shows P4PO Coordination Tools:

• Phase IV Calendar – Weekly Update. Shared Outlook calendar to capture all Indigenous engagement meetings during Phase IV. Disseminated weekly to the EWG.
• COVID-19 Tracker – Ongoing Update. Provides status of office closures due to the pandemic per Indigenous community, and any additional notes to inform engagement.
• GCDOCs Communication Folder – Ongoing update. Serves to track ongoing communication with Indigenous communities.
• Google Doc Communication Log – Ongoing update. Serves to track ongoing communication with Indigenous communities.

Appendix 4: Summary of Lessons Learned in Past Evaluations

Text Description

Infographic showing summary of lessons learned in past evaluations.

In 2021 Horizontal Evaluation of World Class Tanker Safety System (WCTSS).

• The WCTSS intended results were not well understood during the program design and made it difficult to assess effectiveness.
• The implementation of the WCTSS suffered due to challenges with staffing, the underestimation of the time required to draft key policy / funding documents, and the volume of distinct projects.
• Collaboration within the federal government for WCTSS initiatives could have been strengthened through a shared lexicon and an enhanced strategic perspective.
• Partnership with Indigenous communities could have been improved through a more coordinated and strategic approach to ensure meaningful engagement.
• Some of the factors that fostered an effective partnership include having a common understanding of (1) shared objectives and alignment with individual organizational objectives, (2) decision-making processes, and (3) the role of each partnership.
• Engagement approaches and communications were inconsistent, which in some instances led to engagement fatigue and confusion.
• The WCTSS lacked a clear governance structure for engagement. Consultations with Indigenous communities were variably led by the national headquarters, regional offices, and different departments.  This contributed to engagement fatigue.

In 2020 Evaluation of West Coast Energy Infrastructure Initiative (WCEI):

• Mixed views were expressed about the effectiveness of the WCEI governance structure, the single-window approach and the decision-making processes regarding funding.
• WCEI are facilitated by the horizontal model where information and proposals were shared between federal departments and the government of British Columbia. However, many respondents were unclear as to the exact role of IPO-West.
• There is value in a single-window approach for Indigenous engagement on West Coast energy projects and NRCan has developed a body of knowledge on how Indigenous communities can be engaged in this context, and that body of knowledge should be documented.
• The whole-of-government approach is maximized when all departments involved are clear about the new roles and responsibilities of each department in relation to their mandate, as well as potential risk of internal and external duplications.
• Different mandates and criteria amongst departments may create barriers to timely decisions.

In 2019 Applying Lessons from the Work of IPO-West:

• Work in multilateral fora is critical and complements, but does not replace, bilateral relationships.
• For projects that touch many communities, or projects in areas with shared or overlapping territories/jurisdictions, the aggregation of diverse communities and partners enables parties to address common priorities and system-level issues.
• Having a separate convenor helps preserve the independence of participating regulators and the positions of proponents, while supporting them in piloting new ways of working with Indigenous peoples.
• Access to resources is required to facilitate joint planning, consensus-based decision-making, management of sensitive issues and inclusive participation of successful partnership-based engagement.

In 2018 Evaluation of Indigenous Partnership West (IPO-West):

• There was initially some confusion as to the role of IPO-West and that of other departments.
• Some stakeholders expressed concerns about the timeliness of funding decisions. Decision-making approach involving multiple departments may be inefficient and not inclusive of partner departments.
• Multiple meetings/committees can be time-consuming, and may create fatigue, overlap and duplication. Back-and-forth between partner departments can be time- and resource- intensive.

Appendix 5: Evaluation Team

Michel Gould, Chief Audit and Evaluation Executive
Stephanie Kalt, Director of Strategic Evaluation
Christiane Ngo Manguelle PhD, Senior Evaluation Officer
An Gie Yong PhD, Evaluation Officer
Florence Guillot, Policy Analyst (Policy Analyst Recruitment and Development Program)
Kelly Sears Consulting Group

The evaluation team would like to acknowledge those individuals who contributed to the conduct of this project, particularly program representatives and members of the interdepartmental Evaluation Steering Committee who provided insights and comments as part of the evaluation.