-- Part One --
July 21, 1997
TABLE OF CONTENTS
1. Executive Summary
4. Site-Specific and Corporate Reviews
Inadequate Performance Monitoring
Inadequate Procedural Compliance
Inadequate Quality Assurance
Inadequate Work Protection
Root Causes Not Identified
Security Program Needs
Incomplete or Flawed Processes
Bruce A Emergency Coolant Injection
Bruce B Service Water Systems
Darlington Compressed Air Systems
Pickering Electrical Distribution System
Engineering Design and Design Change
In-Service Environmental Qualification
Fire Protection Functional Inspection
Management Focus on the AssetOrganizational Effectiveness
Resource Utilization & Cost of Services
Analyses and Risk Assessment
Material Degradation Monitoring
Performace Area Ratings
Long standing management, process and equipment problems in Ontario Hydro Nuclear (OHN) plants are well known but have not been aggressively resolved. As a result, the overall performance of OHN is well below the level of performance typically achieved by the best nuclear utilities. Immediate attention is needed to improve performance so that the value of OHN's assets does not depreciate beyond recovery. There is still a tendency to look backward at past performance and take comfort rather than address the significant challenges of the future. In spite of the consequences of carrying on with the current course of action, a sense of urgency to move quickly towards substantial and sustained improvement in performance is not evident throughout the organization.
These concerns as well as other strategic issues led Ontario Hydro's Chief Executive Officer, Dr. O. Allan Kupcis to hire Mr. G. Carl Andognini as Executive Vice President and Chief Nuclear Officer (CNO) in December, 1996. Dr. Kupcis immediately directed Mr. Andognini to secure the type of "brutally honest" assessment of OHN conducted by the American nuclear industry. In January, 1997, Mr. Andognini chartered the Nuclear Performance Advisory Group (NPAG) to perform an Independent, Integrated Performance Assessment (IIPA) of OHN. The IIPA was conducted over a three month period. This report to management presents the findings of the IIPA assessment.
A portion of NPAG's charter is to recommend to the CNO if and when they determine that conditions in the plant(s) degrade to a point where minimum nuclear safety standards are not being met. The CNO will respond quickly to any situation that seriously reduces or challenges the safe operating envelope, up to and including partial or full shutdown of the operating unit(s).
In this regard, while the current course of action has significant financial consequence, the IIPA team confirmed that all of the plants were being operated in a manner that meets defined regulations and accepted standards related to nuclear safety. During the course of the assessment, a few issues were identified by the team that could undermine defense in depth barriers. These issues were raised to senior plant management and resolved to the satisfaction of the team.
In general, the IIPA team ranked all of the operating stations (Pickering , Bruce and Darlington) "Minimally Acceptable". Immediate management attention however, is needed to improve performance or even to maintain current performance. The rank "minimally acceptable" is consistent with the lower ranks that the Institute of Nuclear Power Operators (INPO) would issue and still permit the plants to operate if in America. NPAG believes it is also consistent with a United States Nuclear Regulatory Commission (NRC) Systematic Assessment of Licensee Performance (SALP) rating that would likely result in the plants being placed on the "NRC watch list".
Initiatives such as the OHN Nuclear Recovery Plan and the Pickering Quality of Work Program have identified and/or addressed some key problem issues. In selected areas, progress is being made. For example, the improvement in the visual appearance of several stations and increased reporting of lower level events in the corrective action system are two examples. In other areas, the ability to take corrective actions is inhibited by an insufficiently detailed understanding of the standards and practices required to achieve excellence in nuclear operations.
Unless fundamental problems, most notably a lack of authoritative and accountable managerial leadership, are addressed and corrected, there is limited potential for success at OHN. Moreover, many problems are so deeply entrenched within all aspects of OHN (organizational structures, practices, policies and systems) that individual managers are unable or unwilling to take corrective action.
Clearly, OHN did not make a smooth transition from it's original and highly successful design and construction phase to the second stage, focused on operating and maintaining its nineteen operating nuclear units. A full transition to the second stage requires a new approach to the culture, structure, and management of OHN, as well as a rethinking of employee skill mixes and the regulatory process. To excel, OHN must transform itself into a world class organization with a primary focus on operating and maintaining its existing assets. These standards of excellence were the basis for the IIPA assessment convened by the CNO.
OHN staff at every level are reluctant to ask difficult questions of themselves and others. Failure to establish a questioning attitude is a primary cause of the reduction in the "defense-in-depth" concept. There is no real independent evaluation of proposed operations by people not directly involved in formulating the planned actions, (e.g., is this the safest way to accomplish an operation? Are the operators challenged unnecessarily by the proposed change? Will all required structures, systems and components remain capable of performing their intended functions for their day-to-day mission and all credible accident scenarios?).
In summary, there is much work to be done, at all levels in the organization, to reach the established standards of excellence expected in the nuclear industry. However, OHN is in a unique situation where success can be achieved if the proper actions are taken in a timely manner. The management and cultural problems identified by the IIPA, while extensive, are not substantially different than the effort required to turn around troubled plants in the United States. The members of NPAG have extensive experience directly applicable to such turnaround efforts and their involvement throughout the recovery will enable OHN to achieve success in the most effective, efficient and sustainable manner.
OHN recognizes and acknowledges it's indebtedness to the nuclear professionals who expended more than 35,000 person hours to assess and recommend the overall corrective actions contained within this report and it's supporting reports. The detailed corrective actions are contained in separate documents and plans.
The Independent and Integrated Performance Assessment (IIPA) assessed the following:
The IIPA also included six Safety System Functional Inspections (SSFI's) of specific plant systems and/or programs, modeled on the "vertical slice" methodology developed by the U.S. Nuclear Regulatory Commission (NRC).
The IIPA, including the SSFI's, was conducted by a multi-disciplined team of over 75 experienced personnel from both the Canadian and American nuclear industries. This ensured the teams were able to make the independent, "no-holds-barred" assessment required to develop credible findings and conclusions.
While immediate action is required in many areas, the deficiencies identified by the IIPA team have not yet undermined the minimum safety envelope at the sites to an unacceptable level as determined by OHN or its' regulators.
The IIPA process, by definition, focuses exclusively on deficiencies and not good practices. However, there are many noteworthy and successful programs at the OHN corporate level and specific sites that meet or exceed industry standards. For example, the training program at Darlington for Chemical Technicians, although only a small fraction of the overall training for the station is exceptional and should become the OHN standard.
It also examined performance areas not covered in the IPAP methodology, including organizational effectiveness & regulatory affairs.
Additionally, the Performance Area Attributes defined in Attachment B of Inspection Procedure No. 93808 were augmented with attributes extracted from the following sources;
The IIPA assessed ten functional performance areas:
Independent assessments somewhat similar to the level of scrutiny of an IIPA have led to dramatic performance improvements at stations such as Arizona Public Service's Palo Verde Nuclear Generating Station and the Carolina Power and Light's Brunswick Nuclear Station. Each of these stations was rated at the lowest threshold of acceptable performance when the assessments were initiated. In each case, subsequent recovery plans, undertaken at significant cost, re-established each station as a leader in nuclear safety, and as well as a competitive performer in the industry.
Development of IIPA results and recommendations will be the key precursor to the development of an integrated, consistent OHN Business Plan. This work is consistent with the overall goals of executive management and the Ontario Hydro Board of Directors:
Improving nuclear safety and plant reliability; sustaining and enhancing the value of the nuclear asset; increasing credibility with both the Atomic Energy Control Board (AECB) and the people of Ontario; achieving upper quartile World Association of Nuclear Operations (WANO) ratings in the year 2000.
The IIPA of 1997 provides the baseline for future assessments of OHN people, plant and processes. It gives executive management the ability to gauge the effectiveness of future business plan initiatives while monitoring progress. The use of outside experts ensured that OHN performance was assessed by individuals without regulatory or utility restraints and whose conclusions would be "brutally honest". It is anticipated that a systematic process such as this will be developed for use within OHN in the future.
Underlying the generic and site-specific issues are fundamental causes that are in whole or in part related to:
The IIPA team addressed all of these issues with either generic or site-specific recommendations. As a whole, these recommendations form an essential part of the information needed to develop a strategically integrated and comprehensive Business Plan.
Urgent action is required in many areas to address the IIPA identified deficiencies. These deficiencies represent departures from the "defense-in-depth" concept that forms the cornerstone of the nuclear industry. They result in unacceptable erosions of the margin of safety afforded the public and employees. However, the design of the CANDU plant is robust, and in NPAG's judgment the remaining safety margins are sufficient to protect station workers, the general public and the environment at each OHN site.
|Total OHN||Corporate Support||Bruce||Darlington||Pickering|
Full explanation of meaning of ratings is presented below.
The performance rating code definitions are:
Excellent - Performance exceeds industry standards (world class) and generally produces exceptional results. Current level of management attention is sufficient to sustain performance.
Satisfactory - Performance meets most industry standards and generally produces the desired results. Some additional management attention is needed to improve or sustain performance.
Below Standard - Performance is below industry standards and generally produces the desired results. Increased management attention is needed to improve performance.
Minimally Acceptable - Performance is substantially below industry standards, but produces minimally acceptable results. Immediate management attention is required to improve performance.
Unacceptable - Performance is not acceptable and nuclear safety is compromised. Immediate management action is required to return to an acceptable level of nuclear safety or to justify continued operation until performance is improved.
Unrated - Insufficient data available or collected to permit assigning a rating.
Industry Standard generally means a comparison to the key parameters used by INPO/WANO in their performance monitoring program to measure the proficiency of the nuclear industry. It is a combination of the nuclear safety, production and environmental aspects of nuclear power generation without regard to cost. In this document, however, "Industry Standard" is defined herein to include not only the nuclear safety aspects of the INPO/WANO indicators, but also cost competitiveness. It has been demonstrated that the generating units with the best overall INPO performance index are also the ones with the best (upper quartile) cost-of-generation results (non-fuel OM&A).
[ . . . return to Table of Contents ]Managerial Leadership
FINDINGS AND RECOMMENDATIONS
Over several decades, Ontario Hydro has not maintained a consistent, long-term vision of how its nuclear assets should be maintained and operated. OHN, in turn, has not provided an authoritative and accountable managerial leadership system for use by all OHN managers.
No sense of urgency exists to drive immediate change. Many managers are content with the rate at which the organization is changing. Some even actively resist change. The IIPA, however, found that there is ample opportunity to hasten the pace for increasing the standards of performance and changing the culture, the processes, the organizational structure and the procedural hierarchy. Every employee has to recognize the need to move quickly ahead with required changes.
There are significant numbers of managers at all levels of the nuclear organization who lack the basic management and leadership skills to be successful. They lack a fundamental understanding of the need for and value of a consistent, integrated managerial system.
Ontario Hydro has many highly motivated and experienced employees in managerial roles. However, their ability to provide adequate managerial leadership has been undermined by:
- Organizational structures with inconsistently defined functions and poorly defined or missing functions and roles;
- inadequately specified accountabilities and authorities not only for vertical organizational relationships, but also lateral working relationships;
- lack of consistent and proven requisite (required) managerial leadership practices;
- inadequate training and development programs for managers at all levels;
- unclear or frequently changing direction on the goals and standards for OHN and specific sites;
- lack of a well-managed performance indicator system;
- lessons learned and best practices are neither shared nor required to be shared between sites, work units and central support services;
- frequently changing application of the collective agreement.
Inadequate Definition of Employee Accountabilities
Significant numbers of managers at all levels have not communicated to subordinates the need for understanding the vision and goals of OHN and the manner by which they are to be translated into defined tasks with clear individual and managerial accountabilities for performance. For example, it is easy to find employees at all levels who are assigned tasks with no completion dates, with accountability but no or limited authority, and no awareness of how their work supports OHN or site-specific goals.
It is also relatively easy to find employees in what are assumed to be first line managerial roles, such as Shift Maintenance Supervisors (SMS), who are convinced that they are not accountable for the results of their subordinates and in fact have little or no authority in their role. Worse still, some of the managers of the SMSs don't even know what their shift maintenance subordinates are doing.
Poorly Defined Lateral Working Relationships
There were several instances of missing or poorly defined functions. For example, the IIPA team expected that Nuclear Technology Services (NTS) would be responsible for developing recommended engineering standards for OHN-wide application. However, no such function had been assigned to NTS, and NTS management did not exhibit the leadership needed to propose that it be given such an assignment. As a result, prior to action by the CNO in early 1997, there was no expectation that all sites would operate to the same standards. NTS did not meet the performance expectations of the IIPA; it did however adequately fulfill the functions assigned by previous OHN management.
The practice of offering critical services, such as training and engineering, on a "fee for service" basis has created an attitude that the groups providing the service are not a part of the team. This lack of teamwork hinders performance of the service organizations and causes unclear lines of authority and accountability between the operating stations and corporate support services.
Inadequate Managerial Practices
Some managers at all levels have not established the effective lines of communication required to obtain and clearly understand direction, gather important information, and listen and respond to concerns from employees. As a result, there is a gap between management's perception of performance and actual performance. In fact, many managers were surprised by the negative findings of the IIPA team.
Managers do not routinely attach to assigned tasks clear expectations in terms of quality standards, timeliness or resource limitations. There is little expectation of excellent performance, either by management or employees, and little incentive to deliver excellent performance (financial incentives for example). Failure to perform adequately or poor performance is routinely accepted by management, and many managers have similarly low expectations for themselves.
Station management does present performance plans, daily work plans and outage plans. However, it is either unwilling or unable to do anything about minimally acceptable performance or non-performance. This fact is underlined by the number of missed deadlines and commitments, and increasing backlogs in all areas. For example a high percentage of preventative maintenance activities are not completed each month. Several managers at various levels indicated that there are no consequences when key objectives are not met.
Failure to Support Lower Level Management
Senior station and senior corporate services management does not fully support lower line management. Both good ideas and problems tend to be suppressed due to senior management's lack of support. As a result, "bad news" and effective solutions are not flowing up through the organization. Problems are often "solved" at a level too low to assure sustained performance improvement.
A characteristic of most superior performers in the nuclear power generation field is the enthusiastic acceptance of an intrusive internal program of self-assessment throughout the entire organization. This includes oversight from respected nuclear quality groups comprised of experienced personnel. These oversight groups consider themselves, and are considered by the staff, to be an integral part of the team. They conduct their evaluations on behalf of the Site Vice Presidents and the CNO utilizing standard policies and programs developed at the corporate level with monitoring by corporate to assure adherence and effectiveness across all sites. The IIPA team saw neither effective self-assessment nor oversight accountabilities and authorities at OHN.
- Create organizational structures and alignment of functions that are clearly defined and provide for all required functions placed at the requisite organizational levels;
- assign accountabilities and authorities both vertically and laterally as required;
- hold managers individually accountable for ensuring that they and all employees under their direction are aware of and committed to achieving safe and reliable operations through quality performance of expected behaviors and practices;
- train all managers in basic managerial leadership practices that ensure effective two-way feedback, coaching, promote communications between managers and their subordinates re: task assignments, deadlines, quality, resources, employee concerns;
- establish OHN corporate and site performance indicators that support the right behaviors for safe, reliable operations;
- establish cross-functional and cross-site advisory roles for collecting and sharing lessons learned and best practices;
- establish EVP directed self-assessment groups that will do in-depth honest assessments across all sites on a continual basis on behalf of the EVP;
- create and fill new role of First Line Manager as first level of full managerial accountability and authority, accountable for ensuring timely, quality completion of all work assigned to non-management individual contributors on days and shifts;
- establish on-site and off-site safety review capability reporting to the Site Vice President.
[ . . . return to Table of Contents ]
Culture and Standards
Ontario Hydro has stated for years that "Our people are our most important resource." In spite of this statement, OHN is not looking after its people very well. There are pronounced cultural problems that need to be addressed. The malaise is deep and wide and continues to worsen. It will not be quickly or easily corrected.
This culture is the result of long-standing managerial tolerance for cumbersome and ill- defined work processes that make it virtually impossible to get work done at the shop floor level and in the engineering office. In addition, this leadership has permitted the development of "organization walls" that discourage collateral working relationships and teamwork. Some of the behaviors that the team observed that supports this assessment are:
It's acceptable to cut corners. It is not acceptable to make waves. Those who have made waves have been fired or sidelined. The messenger with bad news will be told to fix the problem. Attention to detail is not important as getting the job done. Not meeting commitments is the norm. One person can't make a difference.
Non-performance is accepted -- or even expected -- because senior management has neither set nor enforced standards for employees nor have they assigned specific accountabilities and authorities to managers and individual contributors. As a result, few employees display all of the behaviors required to ensure that station operations remain within safety limits and that every task is completed down to the last detail.
This culture has a negative impact on productivity. More important, however, is the lack of commitment to the establishment and maintenance of a strong safety culture. The organization as a whole and employees as individuals must learn to consider safety an overriding priority, and safety issues must receive immediate and effective attention. The specific problems itemized below are examples of much larger generic issues for OHN:
Employees lack a questioning attitude; deficiencies with safety systems are tolerated at all levels of the organization; procedures are violated and management is tolerant; justifying that "that is OK"; managers, staff and suppliers are not accountable for timeliness or meeting quality and safety standards. Staff are in effect rewarded for poor performance; training in safety and job related accountabilities and authorities, procedures and tasks is insufficient or ineffective; training and development for managers is insufficient or ineffective for most, non-existent for others; there is insufficient training or preparation for managers in dealing with union-related matters .
Employees generally identify with their business unit, station, union affiliation or department, not Ontario Hydro or OHN, and generally do not work well with other groups. For example, training groups and line management do not exhibit a strong sense of working as colleagues with a common goal (See People and Performance). They are not always focused on the same goals and, at times, have conflicting goals that ultimately have a negative impact on OHN. The necessary cross-functional working relationships have not been established, monitored, and supported by senior managers.
A lack of conservative decision-making is prevalent in OHN stations. Management is neither setting high standards for itself nor demanding the best from other departments, and personnel have not incorporated an adequate safety culture into their normal activities.
The IIPA observed a number of challenges to operations, indicating a lack of consideration of overall safety implications. For example, one challenge involved an alternate cooldown process that was modified and adopted without formal engineering review and appropriate analysis of the configuration.
Decisions are dominated by a production mentality and managers feel excessive pressure to continue planned evolutions. The production-dominated culture and its ultimate impact on long-term station performance was most apparent in management decisions regarding the start of the ten-year inspection at the Darlington vacuum building. This requires a full station outage. Although only about 60 percent of the scheduled outage jobs had received final planning, the decision was made to move forward with the outage on the original schedule, causing the remaining work plans to be issued during the outage when station resources were stretched to the limit.
Existing standards are, for the most part, not consistent with best industry practice, and are not consistently enforced across the entire organization. Lack of compliance is, in fact, commonplace because there is limited monitoring and assessment to identify and correct non-standard performance. The absence of common performance standards has led to a proliferation of varying and, at times, conflicting standards. Even within a single site, there are differences. For example, Bruce A and Bruce B have different managerial, operations and support structures, making it more difficult to implement positive changes across sites.
- Require all managers to communicate immediately and support with their actions a firm commitment to a fully integrated OHN with common goals achieved using common structures, working relationships and managerial leadership practices;
- upgrade operating and technical standards to meet the level established by top- performing nuclear operations and apply standards consistently across OHN;
- establish training and development programs that fully inform all employees what will be required of them in their roles;
- require managers at all levels to require their subordinates to think through the processes and procedures they are required to use and to bring issues, questions, ideas and recommendations of safety concerns, conflicting standards, and improvements to the manager for consideration and decision;
- monitor adherence to OHN policies by all employees and consistently expect and require full compliance;
- provide training for managers on the collective agreements and union relations;
- provide experienced coaches coming from a stronger "conduct of" environment to accelerate the transition to a more self-critical culture.
[ . . . return to Table of Contents ]
People and Performance
Poor work planning and scheduling has led to under-utilization of many workers within OHN. For example, maintenance crews are often unable to start work because materials, equipment or support is not available. In addition to draining productivity, this also undermines employee confidence in the managers who planned the work in the first place. There are also skill mismatches where the wrong people are assigned the wrong jobs, such as engineers preparing work packages and operations and maintenance procedures.
Serious shortages of key management, supervisory and some technical skills exist within OHN, particularly in plant and systems engineering, radiation protection, licensed operations personnel, and training. Too many gaps have been filled with temporary appointments, and there are inadequate plans in place to replace many key employees with specialized skills who are approaching retirement in the next five years. In addition, there is a lack of personnel with the special skills and experience needed for effective self-assessment and oversight.
Assignments often occur without proper regard to necessary requisite skills, particularly at the supervisory and management levels. For example, many employees at Bruce have been placed in front line supervisory positions with no understanding of their accountabilities and authorities. Worse still, after holding these positions for several years, many supervisors have still not received even the most basic training or coaching by their managers on how to manage, set priorities, and handle employee communication. In short, the mix of skills and assignments given are not in alignment with the work to be done. The solution is not to hire more employees, but to provide better training for those we already have.
Generally OHN employees, especially managers (who are charged with stating what work will be done, when, and how) are sufficiently acquainted with the differing standards and "conduct of" requirements in the nuclear industry. However, their attitude of "our technology is so different and superior that we really haven't anything to learn from stations who use other technology" exhibits the kind of technical and managerial attitude that has contributed to the current state of station performance. This heavy reliance on past technical acclaim has set the tone of the current culture of a lack of urgency and inattention to disciplined "operations and maintenance" behaviors and attitudes at the stations.
Individual employee performance is being held back by the lack of managers who understand their accountability and authority for assigning specific work, tasks, expected results (See Managerial Leadership) and by serious and deepening deficiencies in training and development.
Most important, senior management has failed to give training the attention it requires to ensure that all employees are properly trained and qualified to perform their duties. Relations between station management and training support management are poor. They do not exhibit the trust and openness required to achieve excellence through early identification and correcting of problems. Training support is not able to meet station needs, while station management has not been sufficiently involved in the design and delivery of training programs. The training division is understaffed to ensure expected performance.
The most pressing problem in training is a severe shortage of resources. For example, the Western Nuclear Training Department (WNTD) lacks sufficient resources to support Bruce Nuclear Division (BND) operations, and achieve key training objectives in the Nuclear Recovery Plan (NRP) and Peer Improvement Plans. For example, a recent study by the Nuclear Energy Institute indicates that top performing nuclear stations conduct 240 hours of continuing training for control room staff; OHN does 80-120 hours.
- Hold managers accountable for the outputs of their subordinates--what work is assigned, timely completion, maintenance of quality standards, etc.--and for coaching and training to correct deficiencies;
- establish updated job descriptions including vertical and lateral working relationships and expected results for every position in the OHN organization;
- ensure additions to the organizational structure meet station needs, industry standards, and requisite OHN policies organization principles;
- establish new roles within required OHN standardization across sites;
- standardize skill and knowledge sets across all sites;
- create training programs that focus on individual effectiveness improvement, not training delivery, requiring Station Management to be involved in the design and delivery;
- create required OHN training policies and enforce them;
- upgrade content and delivery of training program to industry standards, developing a comprehensive initial training plan for each major job family;
- require all managers be held accountable for ensuring that each subordinate is made available for required training;
- enforce with managers the need to ensure that subordinates possess the required qualifications prior to beginning work;
- establish formal on-the-job training/verification processes for all crafts;
- ensure key personnel, such as station operators, are not overburdened by administrative duties;
- continually monitor deviations from work plans, determine root cause of deviations, and correct with new policies, processes, standards, procedures, training, coaching, and disciplining as required;
- develop innovative solutions, including the use of external sources, to clear the backlogs in training, engineering and other areas; increase productivity by:
- forecasting manpower needs more accurately;
- developing more accurate job estimates;
- "walking down" more jobs prior to assignment; and
- keeping employees fully deployed at all times;
- monitor human performance problems;
- assess impact on required resources due to attrition and demographic analysis and develop replacement staffing strategy as part of OHN business planning.
[ . . . return to Table of Contents ]
Processes and Procedures
As the OHN workforce ages, the loss of more experienced personnel -- those who constructed and commissioned the plants, is accelerating because of retirements, restructuring, etc. The loss of "corporate memory" can be devastating without comprehensive and accurate procedures and mature processes. OHN senior management have not set comprehensive standards for the plants. As a result, many processes and procedures are inadequate or are too outdated to be compatible with the current plant configurations. Often, major inconsistencies exist in the quality, extent, and use of processes and procedures between the various OHN sites.
At times, personnel cannot comply with the established processes or procedures -- problems that have resulted in serious operating and safety issues. Many procedures can not be performed as written, and it typically takes months to get procedures revised and approved. Workers expressed the view that verbatim compliance with many procedures, as they are currently written, is impossible. For example, procedures are not always available for expected plant or system evolutions and Operating Memos are being used in many places instead of procedures or procedure revisions. Due to the extent of these problems, "bad practices" such as allowing shift superintendents to approve on-the-spot hand-written procedures have permeated the existing procedural control processes.
Procedural compliance policies are not consistent with good industry practice. For example, "follow as closely as possible" and "prepare hand-written procedures if necessary" are typical instructions provided for the procedure compliance process. The industry standard is to require strict (verbatim) compliance to procedures as written and approved and not to have the built-in workarounds. Workers become confused by inconsistent enforcement of procedural compliance requirements. Previously identified procedural problems are often not corrected and workers are still directed to continue work without making necessary changes.
Several vital processes are not sufficiently mature to determine how well they will work. For example, processes such as the new 13-week work control schedule are very recent and require time to grow and show improvements in performance. In the interim, lessons learned from their implementation must be shared between the various OHN sites and inconsistencies eliminated.
Procedures and established processes are the primary mechanism for maintaining the "corporate memory" of the workforce. Both require substantial improvement throughout OHN to reduce the probability and consequences of events and to maintain plant safety and reliability.
Inadequate Performance Monitoring
Key elements of an effective performance monitoring process exist within OHN. But they are not integrated to provide a full and accurate picture of plant performance. In general, key issues are not consistently identified, problem analysis is superficial, corrective actions are weak and not fully evaluated, receipt inspection does not exist in all areas, and in-process inspections are generally not required.
Managers have not been held accountable for establishing the performance measures required to identify problems, implement corrective actions, and assign accountability for results. For example, maintenance management at Bruce remains focused on day-to-day issues, not corrective actions that would lead to long-term improvements Moreover, maintenance management has been unable to correct increasing backlogs of maintenance work. As a result, the physical plant material condition is deteriorating and putting in jeopardy the value of OHN's primary assets -- the operating stations.
Worker inefficiencies have not been evaluated and corrected. For example, many plant personnel change into radiation worker "Browns" when it is highly unlikely they will be assigned to work in a contaminated area. This inefficient practice is very costly to OHN and has very little, if any, payback in terms of worker safety.
Inadequate Procedural Compliance
Procedural compliance is not a standard expectation -- a major reason for the excessive human error rate within Operations. Current procedures require assigning only specifically qualified personnel to certain job tasks. In practice, decisions regarding assignment of work are often based on who is available, rather than on existing individual qualification documentation, which is often out-of-date or incomplete.
Operations processes and procedures require improvement to maintain plant safety and reliability. Examples of present deficiencies include:
Procedures are not always available for expected plant or system evolutions. Examples are, integrated procedures for start-up and shutdown, heat sink change-over, and safe shutdown following loss of Class I power; operating memos are being used in place of procedure revisions because of the excessive time required to process revisions. Lack of timely revision discourages operators from identifying deficiencies. The backlog of pending revisions numbers about 350 for some plants with no plans for immediate, significant reduction; "Pickering Forms" are used as substitutes for operating procedures but do not receive the same level of review and approval as a procedure does; procedure adherence policy was observed not to be followed when human factor considerations prevented step-by-step adherence or when equipment deficiencies arose; shift superintendents are permitted to approve procedures prepared on-the-spot when no procedure is available. This approval does not require an assessment of potential effects on the safety envelope; periodic review to ensure long-term integrity of procedures is not being conducted.
Operations Procedures Groups have been established at two sites. However, they are not presently staffed to assume responsibility for managing the entire procedure maintenance process. A clear procedure format and content requirement has not been established by OHN for these groups to use. Based on future staffing projections, the group will not be fully functional for years.
Inadequate Quality Assurance
Quality Assurance is not assigned accountability or authority to monitor correction of deficiencies on behalf of the Site VP and to report when corrective actions are not within the expected time, quality, or quantity-- a common requirement in high-performing nuclear operations.
Inadequate Work Protection
Errors made in work protection documents and work permits are placing workers at risk and inhibiting operators' ability to control plant status. The existing code is unduly complex and is applied inconsistently. The current Work Protection Code is a major contributor to periods of worker idleness since these documents are required to be processed through the operations group where only a few operators are available to handle the large number of maintainers needing approval.
Root Causes Not Identified
Corrective action processes focus on superficial issues, not root causes, leading to excessive levels of repeat issues and rework. Identification of root cause and development of corrective actions and processes is not evident as a management value in OHN.
Security Program Needs
Although the OHN nuclear security program meets the minimum requirements of the Atomic Energy Control Act, it is questionable whether the assets of OHN are adequately protected against threats of sabotage, theft, or intruders. For example, the program doesn't specifically allow security to inspect and search vehicles entering OHN facilities or to search personal belongings being brought on site by employees and visitors for contraband or other disallowed items. Additionally, there is no formal program that ensures personnel reporting to work are free of alcohol, and prohibited substances, or don't exhibit aberrant behavior.
Incomplete or Flawed Processes
Some programs, such as the Hazardous Materials Management Program, the Emergency Preparedness Program, and the Station Calibration Program, are not working well and need attention. For example, the following processes within the Emergency Preparedness Program need attention: control of fire loading, training for radiological emergencies, use of operating memos for procedure revisions, backlog reduction, personnel accounting methods, and response to fire protection system design and equipment deficiencies.
- Standardize processes and procedures throughout OHN;
- initiate a procedure upgrade program that meets current industry standards.
- establish and enforce a procedural compliance policy;
- establish a standard definition and goals for controlling "operator workarounds" and monitor and control to within the goals;
- establish a central operations support group to coordinate standardization improvements at all stations;
- adopt a set of easily understood and implemented standard Work Protection Code procedures for the nuclear stations;
- establish processes and procedures to identify root causes of events and track corrective actions;
- develop a set of standards for "conduct of" type functions in each major OHN group;
- restrict use of "Browns" to contamination areas or work areas with potential for contamination; see recommendations in "Managerial Leadership" section for QA initiatives;
- establish a "fitness for duty" program for personnel having unescorted access to OHN facilities;
- implement a personnel and vehicular search program to assure that prohibited items do not enter the plants.
[ . . . return to Table of Contents ]
Plant (Hardware) and Design
The design of the CANDU plant is robust, and plant hardware (including equipment and materials), for the most part, is adequately reliable. However, several design and material condition issues incrementally erode the margin of safety afforded by the design. Although these issues are serious, they have not undermined the safety margins and safe operating envelope to an unacceptable level. Adequate safety margins remain to protect station workers, the general public, and the environment.
Design basis documentation is not accurately maintained. Vendor manuals are not current, and a large backlog of plant changes have yet to be incorporated into other design documentation. Updating design and vendor manuals is not a priority thereby necessitating the use of operating manuals as well as other available operating documents as alternatives (e.g., setpoints). It is not intended these documents be used for design information.
Compounding the issue of maintaining the design bases is the fundamental lack of appreciation for the impact of unauthorized changes on safe plant operations. In fact, the very definition of what constitutes a plant change from a design basis perspective is not fully understood by all elements of the organization. In some cases, changes were made without full consideration of the design intent and may have affected the integrity of special safety systems. For example, materials and parts substitutions have occurred without following the approved design change processes.
Station activities are not effectively managed to ensure that plant operation and configuration conform to design bases and remain within the bounds of analyzed conditions. Many surveillance and preventive maintenance activities have not been fully developed or are not being satisfactorily implemented to ensure long term availability and reliability. System and equipment performance is not optimized because of conflicting priority and resource management problems (e.g., living with condenser tube leaks for extended periods of time).
As a result, evidence of deteriorating materials and equipment are not difficult to find at each site. The IIPA found that each station had substandard equipment. Important equipment was out of service. Insufficient or improper maintenance resulted in leaks from mechanical joints, packing, seals, and equipment. Operators, engineers and station management accept low standards for equipment performance and condition. None of these groups insists on prompt repairs being made to prevent further deterioration.
In many cases, the implementation of preventive and predictive maintenance initiatives have been inadequate and insufficient to compensate for the normal aging of plant equipment. In general, OHN plants have been increasing the amount of preventative maintenance performed. However, the backlog of corrective maintenance is still increasing. In some extreme cases, preventative maintenance was not performed on station equipment because maintenance resources were completely consumed by more urgent tasks.
The deterioration of plant equipment as a result of ineffective and slow preventive maintenance is of particular concern. This issue has the potential of forcing unit shutdown or severely impacting the company asset and must be resolved as soon as possible.
The lack of a formal, documented process for performing technical assessments of equipment availability under degraded conditions, makes it difficult to declare with certainty when all of the requirements for operability of systems, structures, components and all attended appurtenances are met. Design and equipment issues with potential nuclear safety significance are not dealt with by the engineering organization in a rigorous and controlled manner. This results in answers that may not be correct or timely with respect to the immediate and long-term safety significance of the situation. A production imperative exists with the engineering staff at all stations. When the IIPA raised issues that had potential nuclear safety implications, the engineering staff responded by justifying the situation as being acceptable. In many cases, this justification was based on unsupported or undocumented engineering judgment or on an incomplete assessment of the intended design function of the equipment. The nuclear safety focus and questioning attitude of the engineering staff is lacking.
Design Basis Documentation & Change Control
The Design Basis Documentation is not maintained, understood, or appreciated by the System or Design Engineers. It is recognized that safety reports are deficient in some areas (e.g., not formally assessing the shutdown condition) but action has only recently been taken to address this in a limited way. Design Manuals are out of date and have not been revised to reflect plant modifications. Updates are generally not a priority, and there is a large backlog of plant changes that have not been incorporated into the design documentation. In addition, the Design Responsible Engineer's input is not always sought when making permanent changes to the plant (e.g., in specifying substitute parts). These problems are best demonstrated in the difficulties encountered in resolving the excessive and at times improperly controlled use of jumpers (temporary modifications not receiving full engineering review).
Systems Engineering and Programs
Managers do not fully understand the function of system engineers and the role they play in assuring that plant design remains current and that critical equipment meets the highest standards of reliability. As a result, standards and expectations for systems engineering are not clearly defined and existing resources are not effectively managed. Managers do not understand the role of system engineering in maintaining the link between the design basis and the operating documentation, which provides the tools required to operate inside the defined safe operating envelope; therefore, they do not insist that this work be performed. As a result, engineering programs, when they exist, do not meet industry standards and monitor only degradation. They do not address long-term degradation prevention and performance improvement.
Safety System Functional Inspections Results
Six Safety System Functional Inspections (SSFI's) identified several specific plant hardware and design deficiencies. These deficiencies represented additional examples of performance problems identified within this area. Each deficiency must be evaluated for potential generic applicability and impact across OHN stations. It is recommended that management consider performing additional SSFI type inspections to determine the extent of the types of deficiencies discovered during these six inspections.
Plant Status and Configuration Control
Indicators at all stations point to a lack of understanding and responsibility by the operations staff of the necessity to operate within the analyzed plant configuration. The use of jumpers, often approved at the shift superintendent level without engineering review, is extensive. Large backlogs exist with no concern about potential cumulative effects.
Operational flowsheets used to operate the plant and establish work protection code boundaries are outdated and maintained by operators with different station-to-station administrative controls designed to ensure the integrity of the flowsheet. At some plants, all valves on the flowsheets are shown in the open position, rather than the expected design configuration for the system in an operating state. The engineering drawings show the expected configuration.
Procedures generally do not establish valve lineups including manual valves and vent and drain valves but reflect only a change in valve positions from one state to another. Valving errors have occurred, resulting in spills. The position-assured-valve procedures were found to be less than effective. Shift Superintendents can approve on-the-spot procedures. Operations Managers can approve Operating Memos establishing unanalyzed configurations without a safety review. Special Safety Systems have been inappropriately removed from service by the Shift Superintendent.
- Establish a process to accurately verify and status of system health;
- establish design review and design verification expectations and address design deficiencies;
- update design manuals on a prioritized basis;
- establish and enforce rigorous configuration management and configuration control processes;
- implement a rigorous change control procedure;
- identify and repair degraded equipment;
- establish reliability centered maintenance approach to conduct of maintenance;
- ensure maintenance has sufficient resources to meet plant operational needs;
- perform additional SSFI's as necessary to determine extent of deficiencies identified in SSFI's;
- establish formal process to perform operability determinations and provide justifications for continued operation of structures, systems and components when technically degraded.
[ . . . go to Part Two ]
[ . . . return to Table of Contents ]
GLOSSARY OF ACRONYMS
AC Alternating Current AECB Atomic Energy Control Board AIM Abnormal Incident Manual ALARA As Low As Reasonably Achievable ANO Authorized Nuclear Operator AOV Air-Operated-Valve CANDU CANadian Deuterium Uranium (reactor) CDF Core Damage Frequency CNO Chief Nuclear Officer CPA Chestnut Park Accord D2O heavy water DBA Design Basis Accident DCN DNGS Darlington Nuclear Generating Station ECI Emergency Coolant Injection ECN EDS Electrical Distribution System EDSFI Electrical Distribution System Functional Inspection EP EQ Environmental Qualification EQA Environmental Qualification Assessment EQL Environmental Qualification List ERT Emergency Response Team EVP Executive Vice-President EWS Emergency Water System FRF H2O ordinary (light) water HPD Health Physics Department/TD> IA Instrument Air IIPA Independent Integrated Performance Assessment IPAP INPO Institute of Nuclear Power Operators MG MSLB N2 nitrogen (gas) NRC U.S. Nuclear Regulatory Commission NRP Nuclear Recovery Plan NTS Nuclear Technology Services NPAG Nuclear Performance Advisory Group OH Ontario Hydro OHN Ontario Hydro Nuclear OM&A Operating, Maintenance and A------ (costs) OP&Ps Operating Policies and Principles PNGS Pickering Nuclear Generating Station PRA Probabilistic Risk Assessment PVC Polyvinyl Chloride (plastic) PWU Power Workers' Union QA Quality Assurance RANA Regulatory Affairs/Nuclear Assurance RDF RP Radiation Protection RSS Responsible System Supervisor SALP Systematic Assessment of Licensee Performance SER Significant Event Report SES Site Electrical Systems SDS Shutdown System SMS Shift Maintenance Supervisor SR Safety Report SSFI Safety System Functional Inspection USNRC U.S. Nuclear Regulatory Commission VP Vice President WANO World Association of Nuclear Operations WNTD Western Nuclear Training Department
[ Accident Possibilities at Gentilly-2 ]
Since March 27th 1996, there have been over
100,000 outside visitors to the CCNR web site, plus
(counter reset July 2nd 1998 at midnight)