Audit of the National Non-Destructive Testing Certification Body AU1805

Approved by the Deputy Minister
on July 12, 2018

Table of Contents

• Executive Summary
  • Introduction
  • Strengths
  • Areas for Improvement
  • Internal Audit Conclusion and Opinion
  • Statement of Conformance
  • Acknowledgements
• Introduction
  • Audit Purpose and Objectives
  • Audit Considerations
  • Scope
  • Approach and Methodology
  • Criteria
• Findings and Recommendations
  • Management processes and stakeholder needs
  • Online delivery of cedo examinations
• APPENDIX A - Audit Criteria

Executive summary

Introduction

Non-Destructive Testing (NDT) is a set of techniques used to inspect metals, materials, components and structures of critical infrastructure without permanently altering the inspected items. Several sectors, including Canada's natural resources, transportation, manufacturing, energy and mining rely on different non-destructive testing inspection methods to ensure the integrity of safety-critical components in equipment and buildings such as cranes, heavy equipment, nuclear reactors, pipelines and other applications. The National Non-Destructive Testing Certification Body (NDTCB) is part of the Natural Resources Canada (NRCan) CanmetMATERIALS Technology Laboratory. Initially part of the Minerals and Metals sector of NRCan, CanmetMATERIALS joined the Innovation and Energy Technology Sector in January 2017, to encompass the full suite of energy-related technology innovation. The NDTCB manages the certification of individuals performing non-destructive testing within Canada, and is a critical organization relied upon by multiple industries and user groups to manage the risks to human health and critical infrastructure. The NDTCB provides three core certification services, and there are currently over 9,000 active certified individuals.

The NDTCB is responsible for the granting, maintaining, recertification, suspension, and withdrawal of NDT certifications. These responsibilities include the development of exams, the intake of applications, the processing of fees, the scoring of exams, the issuance of certifications, as well as the monitoring of ongoing compliance to a professional Code of Conduct, and are administered from the NDTCB central office located in Hamilton, Ontario. In order to obtain their certification, candidates must complete their training, obtain their qualifying work experience, and pass their written and practical examinations. The training courses are delivered by 59 third party Training Organizations across Canada in accordance with NDTCB requirements, and the written and practical examinations are administered by 13 third party Authorized Examination Centres across Canada. The written exams are administered manually in paper format, with the exception of the CEDO examination, which has been administered electronically since November 2017. The NDTCB intends to migrate additional examinations electronically online in the future. 

The NDTCB is supported by various committees providing expertise to support the development of program policies, examination and training content, and ensure that the program is compatible with national and international standards. The NDTCB acquired the International Organization for Standardization (ISO) 9001:2015 certification in June 2016 from Public Services and Procurement Canada – Canadian General Standards Board (PSPC-CGSB), and received an accreditation from the International Accreditation Service (IAS) for Bodies Operating Certification of Persons in compliance with ISO 17024:2012 in March 2017. With regards to funding, the NDTCB is based on a cost recovery model with the majority of the operating spending being funded through user fees. An updated fee structure was implemented in July 2017, after remaining unchanged for 15 years. The NDTCB's estimated revenue for 2018-19 is $2.3 million, and the estimated full cost for that fiscal year is $2.8 million.

This audit was included in the 2017-20 Risk-Based Audit Plan, approved by the Deputy Minister on March 30, 2017. The objective of the audit was to assess the overall effectiveness and efficiency of the key management processes supporting the NDTCB.

Strengths

Overall, the NDTCB has established clear procedures and effective processes to manage the administration of NDTCB operations, including effective financial planning, monitoring, reporting and ongoing improvement of practices. Procedures, guidance, and tools are provided to Authorized Examination Centres and Accepted/Recognized Training Organizations to meet NDTCB requirements, and the NDTCB actively seeks input from stakeholders to identify and respond to their needs.

Areas for improvement

Opportunities exist to increase efficiency and to strengthen security of financial information through the use of an IT solution where candidates could input their data directly in the system. Opportunities also exist to strengthen the IT controls related to the recent online migration of the CEDO written examination, in accordance with generally accepted IT standards.

Internal Audit conclusion and opinion

In my opinion, the NDTCB has established and implemented effective key management processes to support its operations and stakeholders needs. Opportunities exist to increase efficiency and security of information by implementing an online portal, and to strengthen IT controls related to the recent online migration of the CEDO examinations. As Chief Audit Executive I believe the increased use of technology by the department to deliver its programs will provide many benefits. This audit reinforces the importance of close collaboration between Program Sectors and the Chief Information Officer Branch to ensure successful implementation of significant IT transformation projects.

Statement of conformance

In my professional judgement as Chief Audit Executive, the audit conforms with the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing and the Government of Canada's Policy on Internal Audit, as supported by the results of the Quality Assurance and Improvement Program.

Christian Asselin, CPA, CA, CMA, CFE
Chief Audit and Evaluation Executive
June 28, 2018

Acknowledgements

The audit team would like to thank those individuals who contributed to this project and, particularly employees who provided insights and comments as part of this audit.

Introduction

Non-Destructive Testing (NDT) is a set of techniques used to inspect metals, materials, components and structures of critical infrastructure without permanently altering the inspected items. Several sectors, including Canada's natural resources, transportation, manufacturing, energy and mining rely on different non-destructive testing inspection methods to ensure the integrity of safety-critical components in equipment and buildings such as cranes, heavy equipment, nuclear reactors, pipelines and other applications.

The National Non-Destructive Testing Certification Body (NDTCB) is part of the Natural Resources Canada (NRCan) CanmetMATERIALS Technology Laboratory. Initially part of the Minerals and Metals sector of NRCan, CanmetMATERIALS joined the Innovation and Energy Technology Sector in January 2017, to encompass the full suite of energy-related technology innovation. The NDTCB manages the certification of individuals performing non-destructive testing within Canada, and is a critical organization relied upon by multiple industries and user groups to manage the risks to human health and critical infrastructure. The NDTCB provides three core certification services: the Canadian General Standards Board certification for non-destructive testing; the X-Ray Fluorescence Analyzer Operator certification program in collaboration with Health Canada; and the written examination for the Canadian Nuclear Safety Commission's Certification of Exposure Device Operator (CEDO). There are currently over 9,000 active certified individuals, with over 1,000 new certifications issued per year. The NDTCB aims to maintain a minimum of 4,800 Canadians holding one or more valid non-destructive testing certifications issued by NRCan.

The NDTCB is responsible for the granting, maintaining, recertification, suspension, and withdrawal of NDT certifications. These responsibilities include the development of exams, the intake of applications, the processing of fees, the scoring of exams, the issuance of certifications, as well as the monitoring of ongoing compliance to a professional Code of Conduct, and are administered from the NDTCB central office located in Hamilton, Ontario. A single individual can be certified in a number of testing methods and levels, and thus can hold or renew more than one certification. Each certification is valid for a period of up to five years, and can be renewed for an additional five years. Recertification through examination is required before the end of the second period of validity of a certification (approximately every ten years). In order to obtain their certification, candidates must complete their training, obtain their qualifying work experience, pass their written and practical examinations, satisfy vision requirements, and sign a professional Code of Conduct. The training courses are delivered by 59 third party Training Organizations (across Canada) in accordance with NDTCB requirements, and the written and practical examinations are administered by 13 third party Authorized Examination Centres across Canada. In 2016, approximately 3,200 written and practical exams were administered for the five non-destructive testing methods in the Canadian General Standards Board certification, with an average success rate of 81.5%. The written exams are administered manually in paper format, with the exception of the CEDO examination, which has been administered electronically since November 2017. The NDTCB intends to migrate additional examinations electronically online in the future.

The NDTCB is supported by various committees. The CanmetMATERIALS (CMAT) Advisory Committee on NDT Personnel Certification reviews CMAT administration of the NDT Personnel Certification Program and recommends management methods to improve the service of the program to the NDT community. The committee also monitors NDT industry trends and development, and ensures communication between the NRCan certification program and the Canadian General Standards Board committees on NDT. The Scheme and Technical Committees provide recommendations on the implementation and technical content of the certification program and consist of subject matter experts. The committees represent the interests of all parties concerned with the certification scheme. There are several technical committees, which play a role in relation to the development of the technical aspects of the certification process for the various NDT methods and support the setting and maintaining of the technical content of examination and certification.

The NDTCB developed and implemented a quality management system (QMS) using GCDOCS that serves as the basis for the management of its activities. The QMS is structured in a way that considers risks and opportunities for the organization, and documents workflows, processes, and procedures to help maintain operational quality. The NDTCB acquired the International Organization for Standardization (ISO) 9001:2015 certification from Public Services and Procurement Canada – Canadian General Standards Board (PSPC-CGSB) in June 2016 for their QMS and received an accreditation from the International Accreditation Service (IAS) for Bodies Operating Certification of Persons in compliance with ISO 17024:2012 in March 2017.

The NDTCB is based on a cost recovery model with the majority of the operating spending being funded through user fees, which are charged for application to the certification (including renewal and recertification) and for the written and practical examinations. An updated fee structure was implemented in July 2017, which streamlined and simplified core service fees. The fees increased 20-25% from the prior structure, which had remained unchanged for 15 years. The NDTCB 's estimated revenue for 2018-19 is $2.3 million, and the estimated full cost for fiscal 2018-19 is $2.8 million.

This audit was included in the 2017-20 Risk-Based Audit Plan, approved by the Deputy Minister on March 30, 2017.

Audit Purpose and Objectives

The objective of the audit was to assess the overall effectiveness and efficiency of the key management processes supporting the NDTCB.

Specifically, the audit assessed whether:

• The NDTCB has established effective and efficient processes to manage the administration of NDTCB operations;
• The NDTCB has established effective processes and tools to address stakeholder needs; and
• The NDTCB has employed information technology tools and systems to support efficient NDTCB operations.

Audit Considerations

A risk-based approach was used in establishing the objectives, scope, and approach for this audit engagement. A summary of the key underlying potential risks that could impact the effective management of the NDTCB include:

The NDTCB operations are based on a cost recovery model, with revenues funded through user fees. This area is of significant importance to the achievement of the program's objectives and was therefore assessed as an increased area of risk for this audit. 

The NDTCB provides direct services to individuals, and manages a large volume of information of a sensitive nature (e.g. personal information of users).The use of automated, electronic processes, and the provision of online delivery methods is necessary for efficient, accurate and secure processing of this information. This area is of significant importance to the achievement of the program's objectives and was therefore assessed as an increased area of risk for this audit.

The NDTCB partners with several third party stakeholders for a large portion of its service delivery (e.g. third party Recognized Training Organizations, Authorized Examination Centres). This area is of significant importance to the achievement of the program's objectives and was therefore assessed as an increased area of risk for this audit.

Scope

The scope of this project focussed primarily on certification activities from June 2016 to March 2018. However, preceding periods were considered for financial planning and forecasting.

This audit did not assess the physical security of the CanmetMATERIALS Technology Laboratory, as this facility was included in the 2016 Audit of Physical Security.​​​​​​

Approach and Methodology

The approach and methodology followed the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing and the Government of Canada's Policy on Internal Audit. These standards require that the audit be planned and performed in such a way as to obtain reasonable assurance that audit objectives are achieved. The audit included tests considered necessary to provide such assurance. Internal auditors performed the audit with independence and objectivity as defined by the International Standards for the Professional Practice of Internal Auditing.

The audit approach included the following key tasks:

• Interviews with key personnel and site visits;
• Review of the NDTCB documents, business processes, and communication materials; and
• Testing a sample of certification transactions and activities.

The conduct phase of this audit was substantially completed in March 2018.

Criteria

Please refer to Appendix A for the detailed audit criteria. The criteria guided the audit fieldwork and formed the basis for the overall audit conclusion.

Findings and Recommendations

Management processes and stakeholder needs

Summary Finding

Overall, the NDTCB has implemented effective processes to manage the administration of its operations, including effective financial forecasting, monitoring of activities and continuous improvement of practices in place for the management of its operations. In addition, effective processes and tools are in place to address stakeholders needs. Efficiency and security of financial information could be improved through the use of an online portal where candidates enter their information directly into the system, including automated candidate fee payments.

Supporting Observations

The NDTCB provides three core certification services to individuals performing non-destructive testing in Canada. In order to provide these certification services, the NDTCB interacts with various stakeholders, including individuals being certified, training organizations, authorized examinations centers, as well as different members of industry. The audit sought to determine whether the NDTCB has established effective and efficient processes to manage the administration of its operations, and whether effective processes and tools are in place to address stakeholder needs.

Management processes

Financial Planning

The funding of the NDTCB is based on a cost recovery model. Cost recovery is the practice of establishing and collecting user fees for services, including regulatory activities. In order to know the amounts to charge to candidates, the NDTCB must forecast its future expected costs, as well as its fees, which are charged for application to the certification (including renewal and recertification) and for the written and practical examinations. The audit sought to determine whether effective financial planning processes are in place for forecasting costs and revenues to support and sustain the NDTCB operations.  

The audit found that the NDTCB has implemented effective financial planning processes for forecasting costs and revenues, in order to support and sustain the NDTCB operations. Specifically, the NDTCB has created and maintained financial forecasting data on costs and revenues that take into consideration trend analysis of the program as well as external factors. Cost calculations are consistent with the general guidelines outlined by Treasury Board. It is also worth noting that the NDTCB is subject to the reporting sections of the Service Fees Act, which came into effect on April 1st, 2018, and will be required to report to Parliament and the public on the revenue from the fees collected as part of its operations.

In 2017, the NDTCB updated its fee structure, which had remained unchanged for the past 15 years. The new fee structure, which represents a 20-25% increase of fees compared to the previous one, simplifies core service fees and provides cost recovery patterns in order to attempt to maintain financial balance stability for the next 10 years. In 2016-17, the NDTCB recovered 87% of its full cost through revenue generated from user fees. Similarly, the NDTBC planned to recover on average 86% of its full cost through revenue generated from user fees in the next three years. Management advised that support for the remaining program operation costs come from CanmetMATERIALS A-Base funding, and that continual adjustments are made to the NDTCB expenses to avoid significant revenue shortages.   

Operating Procedures for the management of the certification process

The audit sought to determine whether the NDTCB has operating procedures to manage the certification process effectively and efficiently. The audit found that the NDTCB acquired the International Organization for Standardization (ISO) 9001:2015 certification from Public Services and Procurement Canada – Canadian General Standards Board (PSPC-CGSB) in June 2016 for their quality management system (QMS) and that they received an accreditation from the International Accreditation Service (IAS) for Bodies Operating Certification of Persons in compliance with ISO 17024:2012 in March 2017. The QMS outlines the main processes that are used to issue certifications, including document management; development, monitoring, and updating certifications; certification workflows; management of code of conduct violations; examination grading; and audit procedures.

The audit obtained and reviewed the operating procedures related to the application, certification, and renewal processes. The audit found that procedures are clear and comprehensive, and that they are effectively implemented. Specifically, the audit reviewed a sample of certification files to determine whether applications were processed according to requirements outlined in operating procedures, and whether the applications met the requirements needed for the candidates to be certified, renewed or recertified. The audit noted that 13 out of the 13 files reviewed were processed according to the requirements outlined in operating procedures, and met the requirements needed for the candidate to be certified, renewed or recertified. Moreover, the applications included sufficient documentation in all cases.

The audit noted; however, that the administration of these processes is heavily reliant on a manual process and data entry, and that IT/IM solutions could be further utilized to support efficient business operations. Specifically, the audit noted that applications are currently submitted manually (mail, fax, email, or in person). The paper files are then reviewed by NDTCB administrative employees and the information is manually inputted into the database. Efficiency could be enhanced with the use of an online portal where candidates enter their information directly into the system. The NDTCB informed the audit team that they are currently exploring the possibility of implementing an updated database with an enhanced client relationship management system for the management of candidate files and associated workflows.  

Given that the NDTCB collects personal and financial information on the individuals applying to become certified, the audit also sought to determine whether effective processes are in place to securely manage candidates' information. The audit found that processes and procedures have been established for managing candidate information, and that controls are in place to ensure the security of the physical paper files and the electronic information. Documents containing candidate personal and financial information are handled and stored in a controlled access work area and record room. In addition, the manual procedures are regularly reviewed and updated to ensure security of information. However, while the NDTCB has put some controls in place to conceal the credit card information of candidates, these controls were not deemed fully effective by the audit team. With the combination of personal and financial information accessible on site, there is a potential risk of loss or misuse of candidate information.  

As part of the operating procedures for effectively managing the certification process, and as required by the ISO 9001 certification, the NDTCB has implemented processes for monitoring and continuous improvement. The audit reviewed the monitoring and continuous improvement processes and found that the NDTCB conducts its own audits, and uses external audits as well as management reviews to continuously improve its operations. The audit also found that management is taking appropriate corrective actions to address recommendations identified in a timely manner. These mechanisms are also used to report to senior management in a timely manner, and contributes to informed decision-making.

The NDTCB also practices continuous improvement through the use of non-conformance report (NCR) and opportunity for improvement (OFI) reporting. Both reports can be initiated throughout the course of business, when NDTBC staff observe non-conformance with QMS processes or opportunities for improvement. They can also be initiated from audits conducted by the NDTCB, management reviews, or candidate feedback. NCR reports outline the nature of a non-conformity identified, as well as corrective actions required to address the non-conformity, parties responsible to take action, and the results of the corrective action, whereas OFI reports identify initiatives that would improve the overall operation of the certification process. The audit team selected a sample of three NC and three OFI reports to determine whether corrective actions were taken in a timely manner. The audit found that corrective actions were taken in a timely manner for all 6 out of the 6 reports reviewed.  

Stakeholder needs

The audit sought to determine whether the NDTCB provides effective procedures, guidance and tools to authorized examination centres and recognized training organizations (RTOs), and whether the NDTCB actively seeks input from stakeholders to identify and respond to their needs. The NDTCB's stakeholders include industries that use non-destructive testing, the network of training organizations and examination centres, and individuals being certified.

The audit found that the NDTCB actively seeks input from its stakeholders to identify and respond to their needs. Specifically, the NDTCB committees have industry representatives, and committees' members meet regularly to discuss processes and updates related to the NDTCB. In addition, the NDTCB has a mechanism in place to obtain candidates feedback through its complaints and appeals process, and tracks the actions taken by the NDTCB to address stakeholders concerns. Evidence also showed that consideration was given to the feedback obtained from individuals being certified when the program reviewed its business processes.

The audit found that the NDTCB provides sufficient procedures, guidance, and tools to its recognized training organizations and authorized examination centres through a comprehensive set of documents that clearly communicate the requirements that they must follow. The NDTCB also ensures that the Authorized Examination Centres (AECs) and RTOs meet their requirements by having them undergo an initial assessment when applying for authorization, and every five years after that. The audit reviewed a sample of three centres (AECs and RTOs) to determine whether assessments were conducted according to documented methodology and procedures, whether adequate follow up were conducted to ensure the timely implementation of recommendations. The audit found that the assessments were in conformance with the methodology and procedures, and that adequate follow-up was performed by the NDTCB in all three of the cases reviewed. It is worth noting that the NDTCB is currently transitioning to a new framework for the management of RTOs. The facilities were notified of the transition in October 2017 and were invited to confirm their intention to continue as NDTCB AECs and RTOs. Those who do not confirm their intention will have their status cancelled in October 2018.

Risk and impact

When processes are heavily reliant on manual activities, there are inherent risks of human error and inefficiencies, resulting in potential additional costs. Moreover, with the combination of personal and financial information accessible in one place, there is a risk of potential loss or misuse of candidate information.  

Recommendations

R1  It is recommended that the ADM-IETS and ADM-CMSS assess the feasibility of using an online portal where candidates input their information directly into the system, including automated fee payments. If the assessment indicates such a system is feasible, it should be implemented otherwise the ADM-IETS and ADM-CMSS should ensure that an alternate process is implemented to increase the efficiency and security of the management of candidate information.

Management response and action plan

R1: Management agrees.

The NDTCB has been pursuing and will continue to undertake the initiative to develop and implement an updated IM-IT system with the recommended components (client info database, CRM, online portal integration and automated fee payments.)

Positions responsible:  NDTCB Director and CMSS-CIO

Timing:

• Complete the review and gathering of business requirements jointly with CIO; by September 2018, identify the solution set and commit implementation plans; by Q1 2019, launch functional system.
• ADM-IETS and ADM-CMSS will periodically discuss and review the project status and establish the implementation priority.

Online delivery of cedo examinations

Summary Finding

The NDTCB has recognized the need to move from paper-based testing to computer based testing for certifying individuals performing non-destructive testing within Canada. In order to do so, the Certification of Exposure Device Operator (CEDO) written examination has been administered electronically since November 2017, through a third party service provider using the cloud computing Software as a Service (SaaS) delivery model. The audit found opportunities to strengthen the IT controls related to the online delivery of the CEDO, in accordance with generally accepted IT standards. Specifically, the audit noted that the protocol between NRCan and the Canadian Nuclear Safety Commission (CNSC) was not updated to reflect the current online testing environment and did not include all the essential elements as defined by Treasury Board (TB) Guideline on Service Agreements. In addition, the existing contract between NRCan and the third party supplier does not fully articulate detailed expectations of all parties. The audit also found that the Government of Canada and NRCan IT controls requirements were not fully met when the system went into production.

Supporting Observations

Until November 2017, all written examinations were paper-based and graded manually by an NDTCB examiner in Hamilton. As a result, it took multiple weeks for users to receive their examination results. With an increasing number of applicants, the NDTCB recognized the need to have a more efficient method to expedite the booking, grading, and processing of examinations. In November 2016, the NDTCB contracted a third party service provider to implement computer based testing for the CEDO written examination, which the NDTCB manages on behalf of the CNSC. The first computer based testing examinations were administered in November 2017, and it is the intention of NDTCB to use the system to conduct all of its written examinations in the near future. The audit expected the implementation of the online delivery of examinations to be completed in accordance with generally accepted IT standards. The audit found that some IT controls were not in place prior to the implementation of the online delivery of the CEDO examination.

The audit reviewed the protocol between NRCan and the Canadian Nuclear Safety Commission related to the CEDO. The audit found that the protocol was outdated, and that it did not reflect the current online testing environment. The audit also noted that the protocol was missing some essential elements, as defined in the TB Guideline on Service Agreements, to prevent ambiguity or complications arising in case of disputes or performance issues. 

With regards to the outsourcing of the online delivery of the CEDO examination, the audit found that there is a valid and complete contract in place between NRCan and the third party supplier. The NDTCB uses the private cloud computing Software as a Service (SaaS) delivery model for the administration of the online CEDO examination. SaaS is a software licensing and delivery model in which a software is licensed on a subscription basis and is centrally hosted. This approach is consistent with the Government of Canada IT Strategic Plan of 2017-21, which encourages departments to explore public and private ''Everything as a Service (XaaS)'' cloud computing delivery model prior to developing in-house solutions. The audit noted; however, that the contract in place with the third party supplier mainly contains high level requirements and does not articulate detailed expectations of all parties or provide a mechanism for issue resolution to act as a scorecard against which to examine performance and results. Without a clear mechanism in place to address these issues, there is a risk that the NDTCB business needs may not be met.

In addition, the audit noted that the NDTCB did not receive independent assurance that the third party supplier complies with all Government of Canada and NRCan IT security policies, and that key Government of Canada and NRCan IT controls requirements were not fully met before the system went in production. For instance, the TB Security Requirement Check List (SRCL), which is a key IT controls document, was not updated after the contract was awarded to a third party supplier even though it is mandatory to use a new SRCL for each new requirement or requisition. The NRCan Statement of Sensitivity was also deemed as being incomplete by the audit team. Without adequate and effective IT controls in place for the CEDO system to ensure the confidentiality, integrity, and availability of data, it could be compromised.

Risk and impact

The data included in the CEDO system is classified as "Protected A" and without adequate and effective IT controls, the confidentiality, integrity, and availability of data in the system could be compromised.  It could also result in individuals being wrongly certified.

Recommendations

R2  It is recommended that the ADM-IETS ensure that protocol for the Certification of the Exposure Device Operators is updated to reflect the current business environment, in compliance with the TB Guideline for Service Agreements: Essential Elements.

R3  It is recommended that the ADM-IETS, in consultation with CIOSB, ensure that adequate IT controls and privacy requirements are in place for the CEDO system, prior to migrating additional examinations electronically online, including: 

• Ensuring that IT controls and privacy requirements related to the CEDO system are assessed, and fall within the acceptable range for the department, in accordance with GC and NRCan guidelines; and
• Ensuring that there are mechanisms in place to address issue resolution and performance expectations for the CEDO system.

Management response and action plan

R2: Management agrees.

The protocol between CNSC and NRCan will be updated to reflect the most up-to-date business processes.

Position responsible:  NDTCB Director

Timing:

Protocol updated by March 2019.

R3: Management agrees.

The IT-system, process and contractual controls for the CBT system will be reviewed and updated.

Position responsible:  NDTCB Director, in collaboration with CIOSB

Timing:

Complete review and update of controls in place by December 2018.

APPENDIX A - Audit Criteria  

The audit objectives and criteria were developed based on the key controls set out in the Treasury Board of Canada's Core Management Controls. The criteria guided the audit fieldwork and formed the basis for the overall audit conclusion.

The objective of the audit was to assess the overall effectiveness and efficiency of the key management processes supporting the NDTCB.

The following audit criteria were used to conduct the audit:

Audit Sub-Objectives	Audit Criteria
Audit Sub-Objective 1: To determine whether the NDTCB has established effective and efficient processes to manage the administration of the NDTCB operations.	1.1 Effective financial planning processes are in place for forecasting costs and revenues to support and sustain the NDTCB operations.
	1.2 Effective business processes are in place to manage candidates' information securely.
	1.3 Operating procedures are in place to manage the certification process effectively and efficiently.
Audit Sub-Objective 2: To determine whether the NDTCB has established effective processes and tools to address stakeholder needs.	2.1 Effective procedures, guidance, and tools are provided to Authorized Examination Centres and Recognized Training Organizations to meet NDTCB requirements.
	2.2 The NDTCB actively seeks input from stakeholders to identify and respond to their needs.
Audit Sub-Objective 3: To determine whether the NDTCB has employed information technology tools and systems to support efficient NDTCB operations.	3.1 Implementation of the online delivery of examinations was completed in accordance with generally accepted IT standards.
	3.2 The NDTCB employs IT/IM solutions to support efficient business operations.