Audit of the Management of Aging IT Systems

Presented to the Departmental Audit Committee (DAC)
April 10, 2019

Table of Contents

• Executive Summary
  • Introduction
  • Strengths
  • Areas for Improvement
  • Internal Audit Conclusion and Opinion
  • Statement of Conformance
  • Management Response and Action Plan
• Introduction
  • Audit Purpose and Objectives
  • Audit Considerations
  • Scope
  • Approach and Methodology
  • Criteria
• Findings and Recommendations
  • Governance Process
  • Aging IT Priorities
  • Performance Results
• APPENDIX A – Audit Criteria
• APPENDIX B – Aging IT Assessment

Executive summary

Introduction

The Government of Canada’s (GoC’s) 2017-2021 Strategic Plan for Information Management and Information Technology sets out the information management (IM) and information technology (IT) direction for the Government of Canada and identifies government-wide priorities and key activities for all federal departments. This document is a key input to departmental Information Management and Technology (IMT) planning processes.

Federal departments operate their own IM-IT, where each department’s focus is on fulfilling its own individual mandate.  The 2017-2021 GoC Strategic Plan for IM/IT states that this siloed approach continues to lead to numerous issues across government. The GoC Strategic Plan for IM/IT aims to address these challenges by using a whole-of-government or enterprise approach and responding to a number of key drivers, including IM-IT sustainability and aging IT. In this context, aging IT refers to federal government and, more specifically, departmental challenges that affect the sustainability of IT systems over the long-term, such as the availability of software and hardware support, as well as the resources with the skills and knowledge to service these systems.

Natural Resources Canada (NRCan) is a science-based organization that uses its expertise to deliver its mandate to enhance the responsible development and use of Canada’s natural resources and the competitiveness of Canada’s natural resources products.  To deliver on its mandate, NRCan is heavily reliant on various IT systems and processes, which are primarily managed using a decentralized approach across its Sectors with regional offices and laboratories across the country. The Department spends approximately $50 million annually on IT across all Sectors.

The Department’s Chief Information Officer and Security Branch (CIOSB), within the Corporate Management and Services Sector (CMSS), has developed a Draft 2017-2021 IMT Strategy. The Strategy states that the decentralized approach to managing IMT has led to a very complex environment, that is very large, fragmented, vulnerable, costly, and no longer affordable or sustainable.  It goes on to provide a more detailed outline of the current IT state in five areas and a plan on how CIOSB would transform these areas, including moving towards a horizontal view that considers the whole-of-department or enterprise approach needs in IMT decision-making and addresses the underlying problems, including the significant issue of aging IT.              Although the IMT Strategy has not been approved, it has been presented and discussed at the IT governance committee meetings and has resulted in the IMT Transformation Initiative.  Therefore, the content of this strategy is considered relevant.

In addition, NRCan relies on Shared Services Canada (SSC) to deliver and manage infrastructure solutions, which NRCan has identified as being at end of life and no longer sustainable.  CIOSB considers SSC’s support to be paramount in the successful delivery of services and solutions to address the above challenges and in its NRCan 2018-2021 IT Plan states that CIOSB will look to working more closely with SSC, making them a key partner in NRCan’s IMT transformation, including addressing aging IT challenges.

The objective of the audit was to assess the adequacy and effectiveness of NRCan’s management of its aging IT systems.

Given the significance of strategically managing aging IT, this audit was included in the 2018-2021 Risk-Based Audit Plan, approved by the Deputy Minister on April 12, 2018.

Strengths

The Department has established a governance structure with roles and responsibilities that are defined as well as an appropriate level of representation from all Sectors.The governance structure allows for aging IT systems issues and risks to be discussed and addressed.

The 2018-2021 NRCan IT Plan, which incorporates aging IT systems initiatives within its IT priorities, included direction from the Departmental Results Framework and the GoC Strategic Plan for IM/IT. In addition, aging IT investment decisions, once identified, were approved and prioritized through the annual IT Investment Planning process. Furthermore, the IT Investment Planning process does allow Sectors to identify their priorities based on their needs, and there is some evidence that aging IT requirements and projects were integrated in the planning process.

Through the Application Portfolio Management (APM) tool and CIOSB’s initiatives to enhance the quality of information within, CIOSB has most of the required data analytics to be able to measure and report on aging IT for departmental applications.

Areas for improvement

Although a governance structure exists, it is not being used to its full potential to effectively support the management of aging IT systems. There are opportunities to further reinforce the role of the IMTC to incorporate more regular discussions of aging IT systems issues, to clarify roles and responsibilities for aging IT systems, and to establish performance measures to monitor and report on the Department’s progress in addressing aging IT risks.

Aging IT requirements identified in the IT Plan may not be a complete list of actual needs, due to the lack of formal NRCan aging IT strategic direction and reporting to identify such needs in a standardized way. As well, limited to no information regarding Sectors’ application portfolio health indicators, which would identify aging IT systems concerns, has been provided to inform decision-making when identifying NRCan’s priorities.

NRCan has limited aging IT systems performance measures and performance targets, impeding its ability to effectively address aging IT systems issues.Furthermore, there is no formal process to assess the aging IT environment that would assist senior NRCan management in managing its the portfolio of applications.Lastly, although there has been some ad-hoc reporting of the health of the application portfolio using the Application Portfolio Health Overview dashboard, it is not regularly monitored and reported.

Internal Audit conclusion and opinion

In my opinion, while the Department has made some progress in establishing governance and planning processes to manage its aging IT systems, there are several opportunities to improve the adequacy and effectiveness of these processes. By implementing a proactive approach to managing aging IT systems, I believe associated risks and issues will be mitigated more effectively.

Statement of conformance

In my professional judgement as Chief Audit and Evaluation Executive, the audit conforms with the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing and the Government of Canada’s Policy on Internal Audit, as supported by the results of the Quality Assurance and Improvement Program.

Christian Asselin, CPA, CA, CMA, CFE
Chief Audit and Evaluation Executive
April 10, 2019

Acknowledgements

The audit team would like to thank those individuals who contributed to this project and, particularly employees who provided insights and comments as part of this audit.

Introduction

The Government of Canada’s (GoC’s) 2017-2021 Strategic Plan for Information Management and Information Technology sets out the information management (IM) and information technology (IT) direction for the Government of Canada and identifies government-wide priorities and key activities for all federal departments. This document is a key input to departmental Information Management and Technology (IMT) planning processes. Furthermore, the GoC IT Strategic Plan states that departments should prioritize their investments and initiatives and demonstrate alignment with the enterprise direction that the GoC is moving towards.

Federal departments operate their own IM-IT, where each department’s focus is on fulfilling its own individual mandate. The 2017-2021 GoC Strategic Plan for IM/IT states that this siloed approach continues to lead to numerous issues across government, including a duplication of platforms, incompatibility of systems and data models, and inconsistent service delivery among others. The GoC Strategic Plan for IM/IT aims to address these challenges by using a whole-of-government or enterprise approach and responding to a number of key drivers, including IM-IT sustainability and aging IT. In this context, aging IT refers to federal government and, more specifically, departmental challenges that affect the sustainability of IT systems over the long-term, such as the availability of software and hardware support, as well as the resources with the skills and knowledge to service these systems. 

Natural Resources Canada (NRCan) is a science-based organization that uses its expertise to deliver its mandate to enhance the responsible development and use of Canada’s natural resources and the competitiveness of Canada’s natural resources products. NRCan also plays a substantial role in the management of hazards, such as earthquakes, forest fires and floods. As a result, it is heavily reliant on various IT systems and processes, which are primarily managed using a decentralized approach across its Sectors with regional offices and laboratories across the country. The Department spends a total of approximately $50 million annually on IT for all Sectors.

The Department’s Chief Information Officer and Security Branch (CIOSB), within the Corporate Management and Services Sector (CMSS), has developed a Draft 2017-2021 Draft IMT Strategy. The Strategy Draft IMT Strategy states that the decentralized approach of managing IMT has led to a very complex environment, that is very large, fragmented, vulnerable, costly, and no longer affordable or sustainable. It goes on to provide a more detailed outline of the current IT state in five areas: infrastructure; applications; information; resources; and governance. The Strategy describes how CIOSB plans to transform these areas, including moving away from a long-standing vertical or Sector view of how departmental IMT investments should be governed towards a horizontal view that considers the whole-of-department or enterprise approach needs in IMT decision-making and addresses the underlying problems, including the significant issue of aging IT systems. Although the IMT Strategy has not been approved, it has been presented and discussed at IT governance committee meetings and has resulted in the IMT Transformation Initiative.  Therefore, the content of this strategy is  considered relevant.

In addition, NRCan relies on Shared Services Canada (SSC) to deliver and manage infrastructure solutions, which it has identified as being at end of life and no longer sustainable. More, specifically, in its 2017-2022 Draft IMT Strategy, CIOSB states that there is limited sustainability planning for IT infrastructure platforms and that it faces the effects of equipment rust-out, network performance issues, storage constraints, or a combination of all these factors on a regular basis.  CIOSB considers SSC’s support to be paramount in the successful delivery of services and solutions to address the above challenges, and the NRCan 2018-2021 IT Plan states that CIOSB will look to working more closely with SSC, making them a key partner in NRCan’s IMT transformation, including addressing aging IT challenges.

This audit was included in the 2018-2021 Risk-Based Audit Plan, approved by the Deputy Minister on April 12, 2018.

Audit Purpose and Objectives

The objective of the audit was to assess the adequacy and effectiveness of Natural Resources Canada’s management of its aging IT systems. Specifically, the audit assessed whether:

• The existing governance structure has roles and responsibilities that are clearly defined, and provides oversight and support with regards to aging IT and the “enterprise approach” model adopted by NRCan;
• The NRCan Draft IMT Strategy and IT Plan identify the Department’s aging IT priorities, align with NRCan business needs, and include the resource requirements to achieve successful implementation; and
• NRCan has valid performance results and an internal reporting and review mechanism relating to aging IT systems to provide decision makers with accurate, timely, and evidence-based information when reporting on progress and achievements.

Audit Considerations

A risk-based approach was used in establishing the objectives, scope, and approach for this audit engagement. The following areas were identified as having significance to the effective management of the Department’s aging IT systems, and were therefore assessed as increased areas of risk for this audit:

• Oversight provided by current governance structures for aging IT decisions;
• Definition, delineation, and communication of roles and responsibilities for aging IT; and
• Development of an aging IT plan to provide overall strategic departmental direction and priorities for aging IT systems, including capacity considerations and a funding model.

Scope

The scope of the audit included a review of the departmental Draft IMT Strategy and IT Plan; operational plans; governance structures and mechanisms; key modernization projects that address end of life issues; and key performance indicators that are used to report on progress.

The audit team focused its examination on the Corporate Management and Services Sector and included a limited review of the processes and assessments made within each key Sector regarding the identification and management of aging IT risks and priorities and how this information was used in arriving at the departmental-wide aging IT priorities. This included the Canadian Forest Service (CFS), Innovation and Energy Technology Sector (IETS), Lands and Minerals Sector (LMS), Energy Sector (ES), and Strategic Policy and Results Sector (SPRS) (Canada Centre for Mapping and Earth Observation).

The audit focused on relevant activities and documents substantially from January 2018 to December 2018.

The audit also included the liaison and communications between SSC and NRCan related to the determination and management of aging IT priorities; however, the audit did not include a direct review of SSC operations or how they manage the aging IT priorities of NRCan.

The audit team considered the 2016 Audit of Information Technology Governance as part of the work it undertook in the governance component of this audit on aging IT.

Approach and Methodology

The approach and methodology followed the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing and the Government of Canada’s Policy on Internal Audit. These standards require that the audit be planned and performed in such a way as to obtain reasonable assurance that audit objectives are achieved. The audit included tests considered necessary to provide such assurance. Internal auditors performed the audit with independence and objectivity as defined by the International Standards for the Professional Practice of Internal Auditing. The audit approach included the following key tasks:

• Interviews with key personnel with respect to the Department’s aging IT initiatives;
• Review of key documents including Natural Resources Canada’s 2017-2021 Draft IMT Strategy, 2018-2021 IT Plan, aging IT-related committee structures, meeting minutes, and relevant policies and directives; and
• A detailed examination of documentation relating to planning, monitoring performance, and reporting on aging IT activities.

The conduct phase of this audit was substantially completed in December 2018.

Criteria

Please refer to Appendix A for the detailed audit criteria. The criteria guided the audit fieldwork and formed the basis for the overall audit conclusion.

Findings and Recommendations

Governance Process

Summary Finding

NRCan has established and implemented governance processes to support the management of its IMT activities including aging IT systems. There are defined roles and responsibilities for the Chief Information Officer for Aging IT systems, however, they are not as clear for the governance committees and the Sectors. In addition, roles and responsibilities related to aging IT are not easily accessible and are dispersed over a number of NRCan directives, standards, and presentations. Furthermore, Sectors receive limited information and are, therefore, unclear on their roles and responsibilities on aging IT issues and risks.

Supporting Observations

A governance structure that has roles and responsibilities that are clearly defined and provides oversight and support with regards to aging IT systems allows for synergies to be leveraged and the management of corporate aging IT risks to be addressed effectively. The audit sought to determine whether the current NRCan governance structure and oversight bodies for IT allow for the effective management of corporate aging IT risks. In addition, the audit examined whether the roles and responsibilities related to aging IT of the Chief Information Officer Services Branch (CIOSB), and the Sectors are clearly defined, communicated, and understood, to enable them to provide adequate oversight.

Departmental IM/IT Governance Committees

NRCan currently has two main governance committees to manage IT systems: Business Transformation Committee (BTC) at the ADM level, and Information Management and Technology Committee (IMTC), at the DG level with all Sectors represented. A third committee is currently in the process of being implemented, the Architecture Review Board (ARB), was formerly Architecture Review Committee (ARC). This Board will report to the IMTC and will be responsible for addressing enterprise IM and IT architecture, systems, and solutions. The terms of reference of this Board have been designed to provide more authority on enterprise architecture in comparison to the former ARC.

The mandate of the BTC is to advance departmental outcomes, and one of the areas of focus is to enhance the utilization of IM/IT to improve the overall performance of NRCan. It provides guidance to IMTC to ensure that transformation activities are aligned to NRCan strategic priorities. It also provides oversight on the effectiveness and sustainability of NRCan’s numerous government-wide and internal initiatives, including departmental IM/IT projects.

The mandate of IMTC is to provide oversight and direction of IMT services to ensure effective and efficient delivery in support of the Department’s mandate and in achieving its strategic outcomes. It also acts as a decision-making body for operational initiatives, products, or issues, as well as provides a forum for cross-Sectoral information sharing. The Committee’s main responsibilities include reviewing and recommending for approval to the CIO and Assistant Deputy Minister (ADM) CMSS, IMT strategies, multi-year prioritized investment plans, annual expenditure portfolios, and performance measurement criteria related to IMT.

Through a review of the BTC and IMTC minutes, the audit team found limited discussions on aging IT issues, and those discussions were specific to ongoing projects rather than regular reporting of corporate aging IT risks and health indicators. For example, there were discussions around the Aging Linux Platform Renewal project, the Network optimization project, and Application Portfolio Health. IMTC minutes demonstrated that there are ad hoc updates and discussions on aging IT issues such as IT infrastructure, IT assets beyond their useful life, and adequacy of current funding in meeting the needs to address aging IT issues.

Oversight of Aging IT

IT roles and responsibilities exist on two levels. A portion of the network/infrastructure is the responsibility of Shared Services Canada (SSC) and the science network infrastructure and management and maintenance of applications is under the responsibility of NRCan. IT governance committees’ roles and responsibilities are defined in detailed terms of reference; however, the audit team found no mention of aging IT systems in those roles and responsibilities. Sector Director General (DG)-level representatives have the responsibility to represent Sector interests at IMTC and, as part of the IT Investment Planning process, each Sector is required to provide CIOSB with a list of ongoing / upcoming IT-enabled projects, rating each project against the NRCan Prioritization Framework. It was also noted through interviews with Sectors that roles and responsibilities with regards to aging IT systems are not clearly defined between CIOSB and the Sectors. There have been discussions on roles and responsibilities, but nothing formal has been communicated.

In addition, as part of the IMT Transformation Initiative, a Responsible, Accountable, Consulted, Informed (RACI) chart has been developed to define who is responsible, accountable, consulted, or informed across several IMT decision domains. The audit found that aging IT was not clearly identified in any of the decision domains. Also, under this initiative, service delivery agreements between CIOSB and the Sectors are currently being developed that will further define their roles and responsibilities. However, there is no evidence that roles and responsibilities with regards to aging IT systems will be defined.

Risk and Impact

The absence of clear roles and responsibilities regarding the management of aging IT systems can lead to important aging IT issues and risks not being clearly defined and communicated, and this does not enable decision makers to provide proper oversight. This could also lead to aging IT systems or applications failing and negatively impacting NRCan’s operations.

Given its importance, aging IT systems issues and risks need to be monitored and reported on a regular basis. In addition, the lack of a clear mandate to address aging IT systems for the IT governance committees can lead to aging IT projects not being brought forward, which may impair senior management’s ability to make informed decisions to address aging IT systems and identify synergies to manage aging IT systems risks and issues effectively.

Recommendations

Recommendation 1: It is recommended that the Chairs of the IMTC ensure regular discussion of aging IT systems issues and risks; implement a process to assess progress in addressing aging IT systems issues and risks; and report assessment results to the Business Transformation Committee.

Recommendation 2: It is recommended that the ADM CMSS, in collaboration with Sector ADMs, clearly document roles and responsibilities related to aging IT systems, including delineation between CIOSB, the Sectors, and SSC. This should be communicated to all stakeholders.

Management Response and Action Plan

Management agrees with Recommendation 1.

As part of the IMT Transformation initiative, Sector Delivery Agreements (SDAs) will be developed for each Sector and approved by Sector ADMs and the CIO (Co-Chair of IMTC). These SDAs will draw from Application Portfolio Management (APM) data indicating the aging status of systems. SDAs will also capture an action plan to upgrade or decommission systems, and results will be reviewed on a quarterly basis. IT aging issues, risks and overall status will be reported back to IMTC and BTC.

Timing: June 30, 2019

Management agrees with Recommendation 2.

As part of the IMT Transformation initiative, roles and responsibilities between CIOSB, Sectors and SSC are being further defined and documented, including those related to aging IT.  In addition, through NRCan’s renewed IMT Governance, the IT Operations Committee is being stood up, that will report to IMTC, and focus on horizontal IT-related policies, planning and processes.  This includes a focus on IMT service delivery throughout the Department, and the relationship between CIOSB and the Sectors. The CIO will take the lead in ensuring aging IT performance targets are set by each Sector, who are accountable for these targets. Performance against these targets will be measured on a quarterly basis.

Timing: March 31, 2020

Aging IT Priorities

Summary Finding

The 2018-2021 NRCan IT Plan, which incorporates aging IT initiatives within its IT priorities, included direction from the Departmental Results Framework and the GoC Strategic Plan for IM/IT. In addition, aging IT investment decisions were approved, prioritized, and coordinated through the annual IT Investment Planning process. However, aging IT projects identified in the IT Plan do not represent a complete list of actual needs, due to the lack of formal NRCan aging IT strategic direction and reporting to identify such needs in a standardized way. As well, limited information regarding Sectors’ application portfolio health indicators, which would identify aging IT systems concerns, has been provided to inform decision-making when identifying NRCan’s priorities.

Supporting Observations

A clear identification of the Department’s aging IT priorities, that they align with NRCan’s business needs and that include the resources requirements ensures the successful implementation of initiatives related to aging IT. The audit team sought to determine if aging IT requirements had been established and agreed upon through consultations from all relevant Sectors within NRCan. The audit also sought to confirm if elements relating to aging IT had been developed in alignment with NRCan’s corporate planning documents. Finally, the audit team expected that a process had been established to ensure aging IT investment decisions were appropriately approved, prioritized, and coordinated, and that they were properly resources and implemented.

Aging IT Requirements in Annual IT Planning

Although NRCan does not have an Aging IT Plan, it does include aging IT in both its IT Plan and Draft IMT Strategy. Aging IT requirements are identified through the Application Portfolio Management (APM) system, which is updated regularly with information from all Sectors through an annual APM callout, managed by the APM champion in CIOSB. The APM callout focuses on examining applications with low business value, confirming applications flagged for retirement, business and technical value ratings, and mission criticality among others. This is used to update the aging IT assessment rating (immediate attention required, attention required, minimal attention required, no attention required, not assessed) and the T.I.M.E. rating (Tolerate, Invest, Migrate, Eliminate).

The annual IT Investment Planning process, which is used to develop the NRCan IT Plan, does allow Sectors to identify their priorities based on their needs. The audit team found some evidence that aging IT requirements and projects were integrated in the planning process. For example, the prioritization framework includes an element relating to reducing business risk, which includes cyber and aging IT risks. As well, projects that have aging IT as a key driver are identified as such. The audit team, through interviews with Sectors, noted that the opportunity exists to identify aging IT requirements and projects as part of the IT Investment Planning process. However, due to the lack of formal direction for aging IT to guide Sectors, limited to no aging IT projects are identified by Sectors outside of CIOSB. Formal direction for aging IT could be in the form of Key Performance Indicators that Sectors would be required to meet (e.g. Application Portfolio Health Indicator % for applications), or clear direction or targets to decommission in a timely manner all “Eliminate” T.I.M.E. rated applications. Further, the overall message conveyed by Sectors to the audit team was that the business of Sectors is to manage their programs and scientific activities, and not to manage aging IT.

To ensure that IT is aligned with NRCan business needs and with the GoC direction, NRCan uses the annual IT Investment Planning process as the basis to identify priority IT projects, which are identified by the Sectors. The NRCan 2018-2021 IT Plan arrived at its strategic priorities by focusing on risk reduction (aging IT and cyber among others), and identifying NRCan and GoC strategic directions and how CIOSB is aligned with those directions. The NRCan 2018-2021 IT Plan details five key areas that IT will focus on. The audit team found that although the priorities do not directly reference aging IT, three of the priorities involve addressing aging IT issues as an important component. These priorities are increasing NRCan’s cyber resiliency, implementing enterprise solutions, and renewing IT infrastructure.

The new Departmental Results Framework promotes horizontality across natural resources policies and programs and sets out departmental program-driven expected results. The NRCan 2018-2021 IT Plan outlines how the IT priorities, including aging IT, align with those departmental results. For example, for the departmental result to have tools to safeguard Canadians from natural hazards and explosives, CIOSB has identified the renewal of the aging IT infrastructure, the seismic system renewal, and migration of certain mission critical systems to SSC end state data centres.

Aging IT investment decisions

To develop its IT Plan, which includes setting priorities of IT-enabled projects, NRCan conducts its annual IT Investment Planning process. This process attempts to enable an enterprise IT perspective across NRCan through a common structure and approach to IT planning, aligned with Sector programs and priorities. As part of this process, each Sector is required to provide CIOSB with a list of ongoing/upcoming IT-enabled projects, rating each project against the NRCan Prioritization Framework. The prioritization criteria provide a consistent selection approach when projects are rolled up into the NRCan IT Plan, and they ensure alignment with GoC IMT Strategy and NRCan business priorities.

For the 2018-2021 IT Plan, each Sector developed its list of priority projects, and an IMTC Prioritization Workshop was conducted. The Workshop reiterated the NRCan Prioritization Criteria and each Sector DG-level representative presented their Sector priorities. This exercise resulted in the development of the NRCan IT Plan Master Project List. The Master Project List identifies if the project is funded, what are the planned expenditures for a 3-year period, and how it aligns with NRCan and GoC priorities. A sub-set list of NRCan infrastructure project priorities is also prepared. This list is sent to SSC, as these projects need the agreement and support of SSC to be completed. This list is derived from the IT Investment Planning process projects and is sent to the Deputy Minister for formal approval before going to SSC.

The audit team found that the NRCan IT Plan 2018-2021, the NRCan Draft IMT Strategy / IMT Transformation 2017-2022, and the IT Investment Planning process 2017-18 and 2018-19 have all been presented and endorsed by senior management through the IMT and BTC Committees.

However, for the portfolio of applications, the audit team did not find evidence that the aging IT assessment and T.I.M.E. rating available through the APM system were used in the IT Investment Planning process to provide adequate information to decision makers. Without the portrait of application health checks of systems for which they are responsible, Sectors were not able to make fully informed decisions due to a lack of visibility of the aging IT landscape. Additionally, CIOSB does not have visibility of the full cost of ownership of all systems, which would supplement aging IT management decision-making. As a best practice, organizations should be in a position where information regarding the “run” and “maintenance” costs of their applications are centrally known to allow for appropriate risk-based management of the application portfolio.

The audit team also reviewed key aging IT projects to determine if they had been completed using a planned and proactive approach. The audit team found projects are being completed in a reactive approach, as described below.

The first project reviewed was the Microsoft Windows Server 2003 Decommissioning Project (Win2k3). The original end of life date for these servers was July 2010 and maintenance support ended in July 2015. SSC subsequently had to put in place a custom support agreement for an additional year, since the decommissioning was not initiated until the 2016-17 fiscal year. Although the project was late to start, the migration of Win2K3 was ultimately successful, mostly in part due to the traction that the project had since it was led by Treasury Board Secretariat (TBS)/SSC priorities) across the Government of Canada. However, as evidenced by the project’s timeline in comparison to the date the support ended, the audit team noted that this project was reactionary to a government-wide initiative, rather than being identified as an aging IT concern to be addressed.

The second project reviewed was the replacement of the Linux Servers that started in 2018. A critical system failure of a Linux server in mid 2015 in the CFS led to a review of the identification of end of life issues for the Linux servers at NRCan. However, nothing else was done by NRCan until SSC performed a scan of NRCan’s network, as part of the 2017 Cyber Security Action Plan (CSAP).They found many Linux Servers, but could not determine the exact number because of NRCan’s complex network. This led to the Aging Linux Replacement Project in 2018. The CFS agreed to take the lead on this project and are currently piloting an approach that could then be rolled out to other Sectors. NRCan currently has a dashboard that shows that 56% of known NRCan Linux servers are past end-of-life and 23% are unknown.

As at December 2018, NRCan had 645 active applications on its networks. The complete APM system contains over 1,000 applications (both in production and decommissioned). Over the past few years there have been several initiatives and reviews to determine the status of the application portfolio. In 2014-15, NRCan used an external consultant to conduct an analysis and benchmark the health of its application portfolio using the T.I.M.E. framework. This was used to identify a high-level Application Rationalization Strategy focused on better business value, lowering risks and costs across the application portfolio, which had been assessed at 553 applications at the time of this report. The report provided a detailed analysis of the applications with a 2 to 5-year application portfolio roadmap to consolidate custom and redundant applications. At the time, if implemented, the review would have resulted in a reduction of 262 applications. Through interviews, the audit team learned that the report was not followed-up on and that NRCan has not fully implemented the approach proposed.

The Win2k3 and the CSAP projects did result in the decommissioning of many applications. The Win2k3 project led to the decommissioning of approximately 15 aging applications since they would have required extensive work and cost to migrate them onto the Win2k8 environment. The CSAP project also resulted in approximately 25 aging applications being decommissioned due to IT security vulnerability concerns. The audit also noted that the 2018 IT Plan had identified 160 applications that had been decommissioned in 2017 with another 13 planned for early 2018. As at December 2018, these 13 applications were still in production. Although the APM system is more complete than back in 2015, and there are better indicators in place to assess the aging IT issues of the current application portfolio, the audit team did not observe any assessment of the application readiness to migrate to end-state data centers, including cloud services.

Risk and Impact

A lack of information on aging IT issues and risks in the Draft IMT Strategy and IT Plan may lead to investment decisions that do not effectively address systems and IT assets that are at end of life and are critical to NRCan operations.

The current reactive approach by NRCan to addressing aging IT risks and issues may lead to inappropriate planning of resources that does not ensure overall application portfolio health.

Recommendation

Recommendation 3: It is recommended that the ADM CMSS, in collaboration with Sector ADMs, define and agree on a strategy to address aging IT systems issues proactively, which would include performance indicators illustrating progress against the NRCan IT Plan’s strategic goals and targets.

Management Response and Action Plan

Management agrees with Recommendation 3.

Sector Delivery Agreements (SDAs) will be developed for each Sector and approved by Sector ADMs and the CIO, which will contribute to a more proactive approach to managing aging IT. Results will be tracked on a quarterly basis.

Furthermore, NRCan will be setting up an IMT Portfolio Architecture with one of the portfolios focussed on Reducing Business Risk, which is one of the key themes of the 2019/20 IT Plan. A lead for this pillar will be identified and report into IMTC.

Timing: March 31, 2020

Performance Results

Summary Finding

NRCan has limited aging IT systems performance measures and no performance targets, impeding its ability to effectively address aging IT systems issues. Furthermore, there is no formal process to assess the aging IT environment that would assist NRCan senior management in managing its portfolio of applications. Lastly, although there has been some ad-hoc reporting of the health of the application portfolio using the Application Portfolio Health Overview dashboard, it is not regularly monitored and reported.

Supporting Observations

Valid performance results and an internal reporting and review mechanism relating to aging IT would provide decision makers with accurate, timely and evidence-based information when reviewing progress and achievements. The audit expected to find aging IT performance measures and targets had been defined and approved and that processes were in place to ensure that key performance indicators were valid, accurate, and revised when needed. In addition, the audit team expected that a performance monitoring process was in place to evaluate aging IT performance.

Performance measures and targets

The audit team confirmed with senior management through interviews that performance measures and targets have not been formally defined to measure aging IT. The only exception is the aging IT assessment and other ratings that are performed by TBS, based on data provided by NRCan in the APM system.

In 2015, an external consultant reviewed the NRCan application portfolio and recommended that NRCan establish APM performance metrics for efficiency and effectiveness and a communication process to keep decision makers apprised of performance. These performance metrics would mainly comprise of aging IT indicators. More specifically, the report recommended that existing and new T.I.M.E. data be leveraged to measure portfolio changes over time, that they be measured on a regular basis to develop trend information and to establish performance expectations that result from the performance metrics. At the time of the audit, these recommendations had not been implemented.

Performance management process and reporting

The audit found that there is no performance management process, other than the aging IT and T.I.M.E. indicators that are available in the APM system. The audit team did not see any evidence of how these indicators are being used to make decisions regarding the health of the application portfolio.

The audit team found limited, ad hoc aging IT reports/dashboards. As part of an NRCan presentation to SSC, in November 2018, summarizing the results of the Linux Server scans, indicators showed that of the over 1,000 Linux devices that had been identified, the percentage of end-of-life servers was 56%, and a total of 23% were still unassessed. As at the end of the examination phase of this audit, although the presentation had been vetted by CIOSB, it had not yet been presented to IMTC or BTC. As well, NRCan APM Metrics Quick Facts, which are one-page summary charts, were developed to provide an overview of the Application Portfolio Health Indicator (APHI) status in July 2018. These one-page documents were informally presented to CIOSB management for information purposes, but they have not been directly shared with the IT governance committees or the Sectors and have not been prepared on a regular basis.

Although the audit team found some ad hoc APHI reporting, NRCan does not actively monitor and report on aging IT. Using the information in the APM system, the audit team has developed key charts to provide senior management with an overview of NRCan’s application portfolio health, as at December 2018. This supports the feasibility of developing performance indicators and actively monitoring them with the data that is available. Refer to Appendix B for examples of NRCan’s application portfolio health charts.

The audit team was informed on many occasions during its interviews with all Sectors that there were significant aging and reliability issues with the current IT infrastructure at NRCan. As well, the 2017 Draft IMT Strategy and 2018 IT Plan have identified the IT Infrastructure as a significant aging IT risk. In addition, NRCan does not currently have complete visibility into the health of the IT infrastructure. It would be a good practice for NRCan to obtain performance reports regarding infrastructure from SSC.

Risk and Impact

Without valid performance results and an internal reporting and review mechanism, decision makers are unable to make informed decisions on selecting the most appropriate projects to address aging IT issues and risks.

Although NRCan has identified IT Infrastructure issues and risks, it does not have performance information to be able to monitor and assess the IT infrastructure environment and be able to anticipate potential risks of failure, which could significantly impact its business operations.

Without ongoing monitoring of aging IT indicators, decision makers are unable to assess progress against set targets and goals and take corrective action when needed.

Recommendations

Recommendation 4: It is recommended that the ADM CMSS, in collaboration with Sector ADMs:

1. Develop a formal process to assess the aging IT environment to identify performance indicators that will assist management in monitoring their portfolio of applications; and
2. Develop, implement, and regularly communicate an Application Portfolio Overview report (e.g. Number of systems owned by Sector; Business criticality/value of systems; Aging IT assessment of systems including risks; Key observations / recommendations; Progress towards targets).

Recommendation 5: It is recommended that the ADM CMSS engage with SSC to strengthen visibility into the health of the infrastructure that supports NRCan to understand risk exposure and be able to inform decision-making.

Management Response and Action Plan

Management agrees with Recommendation 4.

1. The Application Portfolio Management (APM) program is being strengthened by providing the Sectors the ability to view and update their application information within NRCan’s APM database, which is intended to increase the accuracy and timeliness of application-related data, including aging IT.

APM data will be reflected in the Sector Delivery Agreements (SDAs), which will include aging IT performance indicators, and approved by Sector ADMs and the CIO. Results will be tracked on a quarterly basis. Performance results will be presented at IMTC and BTC.

Timing: March 31^st, 2020

2. Sector Delivery Agreements (SDAs) will include the following:
   • Number of systems owned by a Sector;
   • Business criticality/value of systems; and
   • Aging IT assessment of systems including risks.

Key observations and recommendations will come out of the ADM/CIO meetings and will be tracked as part of the quarterly SDA review process.

Timing: September 30^th, 2019

Management agrees with Recommendation 5.

NRCan will engage SSC to determine if NRCan can obtain a regular report from SSC on the aging status of the IT infrastructure that supports NRCan (i.e. servers, etc.). This reporting will be provided to IMTC and BTC.

Timing: March 31^st, 2020

APPENDIX A – Audit Criteria

The criteria were developed primarily from ISACA’s Control Objectives for Information and related Technology (COBIT) framework as well as relevant Treasury Board and NRCan policies, procedures, and directives. The criteria guided the fieldwork and formed the basis for the overall audit conclusion.

The following audit criteria were used to conduct the audit:

Audit Sub-Objectives	Audit Criteria
Audit Sub-Objective 1: The existing governance structure has roles and responsibilities that are clearly defined, and provides oversight and support with regards to aging IT and the “enterprise approach” model adopted by Natural Resources Canada.	1.1 An effective governance structure and oversight bodies for IT have been implemented that allow synergies to be leveraged in managing corporate aging IT risks and to address issues effectively.
	1.2 Roles and responsibilities for CIOSB, NRCan Sectors, and SSC regarding aging IT are clearly defined, communicated, and enable the decision makers to provide oversight.
Audit Sub-Objective 2: The IMT Strategy and IT Plan identify the Department’s aging IT priorities, that they align with NRCan business needs and include the resource requirements to achieve successful implementation.	2.1 The aging IT requirements in the IMT Strategy and IT Plan have been established and agreed upon through consultations from relevant Sectors within NRCan.
	2.2 Elements relating to aging IT in the IMT Strategy and IT Plan have been developed in alignment with the NRCan’s corporate planning documents.
	2.3 A process has been established to ensure aging IT investment decisions are appropriately approved, prioritized, and coordinated, and that they are properly resourced and implemented.
Audit Sub-Objective 3: NRCan has valid performance results and an internal reporting and review mechanism relating to aging IT to provide decision makers with accurate, timely, and evidence-based information when reporting on progress and achievements.	3.1 Appropriate aging IT performance measures and targets have been defined and approved.
	3.2 Systems and processes for aging IT are in place to ensure that key performance indicators are valid, accurate, and revised when needed.
	3.3 A performance monitoring process is in place to evaluate aging IT performance.

APPENDIX B – Aging IT Assessment

The audit team developed the following examples of NRCan’s application portfolio health charts using NRCan’s Application Portfolio Management system, as at December 2018. The population of the first two charts is 645 active applications.