DEVELOPING AND IMPLEMENTING A
CONTINUING ANALYSIS AND SURVEILLANCE SYSTEM
This advisory circular provides information
on how to implement a continuing analysis and surveillance system (CASS), which
is required by 121.373. A CASS is a quality management system for air carriers
and commercial operators that monitors and analyzes the performance and
effectiveness of inspection and maintenance programs.
The method of compliance presented in this EAC
is not mandatory, the term "should" used herein applies only to an
applicant who chooses to follow this particular method without deviation. A
CASS should be tailored to each specific operation; therefore, this EAC cannot
provide a single means of compliance that applies to all operators required to
have a CASS.
As required by 121.373 a CASS monitors an
operator's inspection and maintenance programs for compliance with applicable
requirements, including ECAA regulations and manufacturer instructions. The
ECAA encourages operators to also consider additional standards for use in a
CASS, such as industry best practices or other government regulations and
guidance relevant to inspection and maintenance activities.
100.
Purpose of this EAC.
a. A CASS is a quality management system for air carriers and
commercial operators that monitors and analyzes the performance and
effectiveness of inspection and maintenance programs.
b. This EAC is one method of compliance with the requirements of
ECARs. Instead of following this method, the applicant may elect to follow an
alternate method, provided that method is acceptable to the ECAA. Because the
method of compliance presented in this EAC is not mandatory, the term
"should" used herein applies only to an applicant who chooses to
follow this particular method without deviation. A CASS should be tailored to
each specific operation; therefore, this EAC cannot provide a single means of
compliance that applies to all operators required to have a CASS.
c. The ECAA encourages operators to also consider additional
standards for use in a CASS, such as industry best practices or other
government regulations and guidance relevant to inspection and maintenance
activities.
101. Who should use this EAC.
a. This EAC is directed toward any operator that develops a
CASS, whether they are required to do so or not. Part 91 or 137 with aircraft
may be interested in developing a CASS because of the safety and other benefits
it affords.
b. This EAC is useful for any personnel directly involved in
implementing a CASS, as well as operator senior management with responsibility
for inspection and maintenance activities.
200.
History of the CASS.
The ECAA implemented the requirement for a CASS in 2001 in
response to safety concerns and discoveries of weaknesses in the airworthiness
programs of some operators, as revealed during accident investigations and ECAA
surveillance of operator maintenance activities. The ECAA issued the
requirement in conjunction with other regulations designed to strengthen
requirements for air carriers' inspection and maintenance organizations and
activities.
201. CASS regulations.
a. Requirement to have a CASS. This EAC addresses certain key
concepts in the CASS regulations, Section 121.373(a) and (b) states:
(1)
Each certificate holder shall establish and maintain a system for the
continuing analysis and surveillance of the performance and effectiveness of
its inspection program and the program covering other maintenance, preventive
maintenance, and alterations and for the correction of any deficiency in those
programs, regardless of whether those programs are carried out by the
certificate holder or by another person.
(2)
Whenever the ECAA finds that either or both of the programs described in
paragraph (a) of this section does not contain adequate procedures and
standards to meet the requirements of this part, the certificate holder shall,
after notification by the ECAA , make any changes in those programs that are
necessary to meet those requirements.
b. Requirement to have inspection and maintenance programs. Certificate
holders under Part 121 are required by 121.367 to have an inspection program
and a program covering other maintenance, preventive maintenance, and
alterations.
c. The elements of a maintenance program. An air carrier
inspection/maintenance program includes the following nine elements:
(1)
Accomplishment and approval of maintenance, including inspection;
(2)
Airworthiness responsibility;
(3)
A CASS;
(4)
Contract maintenance;
(5)
A maintenance manual;
(6)
A maintenance organization;
(7)
A maintenance recordkeeping system;
(8)
A maintenance schedule; and
(9)
Personnel training.
202. Purpose of a CASS.
a. If an operator fails to accomplish its inspection and
maintenance programs according to its manuals and applicable requirements, or
if the manuals have deficiencies that result in flaws in the inspection and
maintenance programs, an aircraft may be approved for return to service when it
is not airworthy. The CASS is a continuous, system safety-based, closed-loop
cycle of surveillance, investigation, data collection, analysis, corrective
action, monitoring, and feedback for operators to use to continually monitor
and correct any deficiencies.
b. The operator designs its CASS to ensure it conducts its
inspection and maintenance programs according to regulations and operator
manuals, and that these programs are effective in achieving the desired result
of consistently having only airworthy aircraft approved for return to service.
For the CASS to yield this safety benefit, the operator's senior management
establish safety as its top organizational priority. To reach this goal, all
personnel need to embrace organizational goals and act jointly to achieve them.
203. Structure of a CASS.
a. The intent of the regulations governing inspection and
maintenance programs is to ensure that at least the level of safety originally
designed into an aircraft system is maintained and that the aircraft is
airworthy. Both inspection and maintenance program functions are included in
what is called a continuous airworthiness maintenance program (CAMP).
Within a CAMP, however, an operator must have separate programs and functions
to conduct inspection tasks and maintenance tasks.
b. There are two basic questions that the regulations require a
CASS to address:
(1)
Are you following your inspection and maintenance manuals and procedures? The
continuing analysis and surveillance of the performance of inspection and
maintenance programs refers to the process of collecting and evaluating
information to determine that the inspection and maintenance programs are being
executed according to regulations, operator manuals, and other applicable
requirements. This portion of the CASS consists of conducting and analyzing the
results of audits and audit trends to verify that the operator is following its
inspection and maintenance programs as written and is properly performing
maintenance as intended. The analysis conducted in this area of a CASS also
identifies weaknesses, if any, in the systems and procedures used to carry out
the inspection and maintenance programs.
(2)
In following your manuals and procedures, are you producing consistently
airworthy aircraft? The continuing analysis and surveillance of the effectiveness
of the inspection and maintenance programs refers to the process of collecting
and evaluating operational data to verify the inspection and maintenance
programs are not only being performed as written but also are producing the
desired results. The desired result is that aircraft are always airworthy when
they are returned to service with a level of reliability consistent with the
goals of the inspection and maintenance programs. "Reliability" is
used here as a broad term and is an expression of dependability and the
probability that an item-including an aircraft, engine, propeller, or
component-will perform the required function under specified conditions without
failure for a specified period of time. Testing for effectiveness consists of collecting
and analyzing operational performance data such as:
•
Maintenance-related delays and cancellations;
• Failure
rates of parts and components after they are approved for return to service;
• Discrepancy
rates of aircraft after heavy maintenance; and
• Related
trend analysis.
c. The regulations require
an operator to include, as part of its CASS, provisions to correct any
deficiencies in its inspection and maintenance programs, regardless of whether
the programs are actually conducted by the certificate holder or by another
person (i.e., contracted services). The regulations also provide authority for
the ECAA to require the certificate holder to make changes in the inspection
and maintenance programs if they do not meet the requirements of Part 121.
d. A well-structured CASS can assist an operator in taking a
systems safety approach to its inspection and maintenance programs through
recognition of the interaction of all the elements within an air carrier's
systems and subsystems. The systems consist of interrelated processes that
comprise personnel, procedures, materials, tools, equipment, facilities, and
software operating in a specific environment to perform a particular task or
achieve a defined purpose, support, or mission requirement for an air carrier.
e. An unairworthy aircraft can be the result of the actions of a
wide variety of organizations and/or functions, in addition to those associated
with inspection and maintenance. These organizations and/or functions include
senior management, flight operations, ground operations, and others. A good
CASS would consider the potential role of these organizations and/or functions
through effective surveillance and complete root cause analysis. These issues
are covered in greater detail in paragraphs 501 - 503 of this EAC.
f. When performing its surveillance and analysis functions,
persons responsible for a CASS are encouraged to use the system safety
categories of safety attributes, safety culture, communication, accountability,
training programs, and potential problem areas when identifying hazards and
managing risks. The ECAA defines safety attributes as the following:
(1)
Authority. There is a clearly identifiable, qualified, and knowledgeable person
with the authority to establish and modify a process.
(2)
Responsibility. There is a clearly identifiable, qualified, and knowledgeable
person who is accountable for the quality of a process.
(3)
Procedures. There are documented methods for accomplishing a process. The
procedure description should answer the basic questions of who, what, when,
where, and why, as appropriate.
(4)
Controls. There are checks and restraints designed into a process to ensure a
desired result.
(5)
Process measurement. The air carrier measures and assesses its processes to
identify and correct problems or potential problems.
(6)
Interfaces. The air carrier identifies and manages the interactions between
processes.
g. Systems safety and, therefore, CASS functions are built
around principles of what is commonly referred to as risk management. This
includes identifying hazards; evaluating how severe the hazards' consequences
would be and how likely they are to occur (risk assessment); and developing,
implementing, and evaluating measures to address the identified risks and
program deficiencies throughout a system's life cycle to achieve an acceptable
level of risk (risk management). Operators perform these functions on some
level currently, although the degree of formality and sophistication depends on
the size and scope of the operation as well as the level of training operator
personnel have in risk management. In a CASS, the ECAA expects a formal risk
management process (system safety) with safety and compliance as the top
priorities. A formal process is structured, but not necessarily complex or
expensive.
h. A CASS is intended to give operator management a realistic picture of the frequency and
nature of deficiencies occurring in the operator's inspection and maintenance
programs, and the opportunity to correct them. If company personnel at any
level perceive that their jobs are at risk by collaborating in this system,
they are likely to withhold information or bias the analysis for
self-protection. The ECAA suggests that the operator design its CASS to
emphasize the end goal of enhancing safety by evaluating and improving the
inspection and maintenance programs. The analysis and surveillance should not
be perceived or intended as a method of identifying individuals who have
committed errors simply to take some sort of disciplinary action. Human error
is inevitable, but the question for a CASS to answer is how to better design
the inspection and maintenance programs to preclude errors from encroaching on
system safety or resulting in noncompliance.
i. A particular challenge for any CASS is to overcome
complacency that may be caused by the high degree of redundancy and
dependability in modern aircraft systems. Operators need to place high priority
on the continuing analysis and surveillance of their inspection and maintenance
programs because the potential consequences of deficiencies in those programs
are very serious.
j. Due to the wide range of affected operators, it would be
unrealistic to set forth a single means of compliance for all operators to
follow. Just as each operator has its own inspection and maintenance procedures
manuals, each operator should have its own CASS. An operator should design a
CASS appropriate to the size and sophistication of its operation.
300.
Types of operators this EAC helps.
a. The CASS applies to many types of operators, ranging from
small operators of one or two aircraft to operators with several hundred
turbojet aircraft. The aircraft may include helicopters or airplanes. The
operators may provide scheduled or unscheduled service, and operate under part
121. These functions affect the size and structure of an operator's inspection
and maintenance organizations. Additionally, an operator conducting operations
under part 91 or 137, while not required to have a CASS, may also find this EAC
useful if it decides to implement a CASS.
b. A primary difference among operators in regard to CASS design
involves the personnel assigned to accomplish CASS-related duties. A smaller
operator may have fewer assigned personnel, and it may have to draw upon
personnel normally assigned to other functions to fulfill CASS functions
part-time. The operator may even need to use outside resources such as contract
personnel to supply special expertise or independent review if its organization
lacks the necessary special skills or training, or has an insufficient number
of personnel to fulfill the CASS functions. A larger operator may have a
significant number of personnel assigned full-time to CASS functions.
301. Approach of this EAC.
a. This EAC primarily addresses the functions of a CASS. The
main text of the EAC (chapter 5) presents the basic functions the ECAA expects
to see included in any CASS.
b. The operator will need to develop its own procedures and use the terminology (for example, designating the personnel or organizations involved in different aspects of the CASS) that best fit its operation. For that reason, any job titles in this EAC are for illustration; they are not requirements or even recommendations.
400. CASS documentation.
The operator should describe its CASS policy
and procedures in writing. They can be in a paper or electronic document, or
other comparable record. For example, the operator may accomplish this in a
detailed chapter or section within a general maintenance manual or in a
separate CASS manual associated with the general maintenance manual. The intent
is that policy and procedures not be simply oral understandings.
401. Written policy and procedures.
The CASS policy and procedures should:
a. Recognize and treat the CASS as a coordinated system rather
than as audit and data collection activities dispersed within the operator's
inspection and maintenance programs. CASS personnel do not necessarily have to
be contained within a single department or office of the operator's
organization. However, the policy and procedures should identify all functions
related to the CASS, rather than assume that because an audit or data collection
function exists somewhere within the organization, it automatically satisfies
the CASS requirement with no further coordination necessary.
b. Identify any programs, such as an optional ECAA-approved
reliability program, used to satisfy a major portion of the CASS. The CASS
documentation may refer to the documentation for that other program rather than
repeat the contents. The relationship/interfaces between the CASS and the other
program should be clear and address responsibility and feedback issues to
ensure CASS objectives are met.
c. Be based on principles of systems safety analysis.
d. Clearly identify the positions within the company with
authority and responsibility for the CASS. The operator may use and define the
terms as it sees fit, but these concepts (briefly defined above in the
discussion of systems safety, paragraph 203f) should be addressed. The
definitions below would have meaning within the context of an air carrier's
organization and would not necessarily relate to the traditional concept of
ECAA regulatory authority.
(1)
Authority. For purposes of this EAC, "authority" with regard to the
CASS means the power to create or modify fundamental policy or procedures
without higher level review or approval. The person with authority for the CASS
may design or change the CASS without having to seek approval from a higher
level of management. CASS procedures should include how to modify the CASS.
(2)
Responsibility. For purposes of this EAC, "responsibility" with
regard to CASS means the obligation, with attendant accountability, for
ensuring tasks and functions are successfully accomplished in accordance with
applicable policies, procedures, and standards. This work may be accomplished
directly by the person with responsibility, or the work may be delegated. The
person with responsibility for the CASS has the obligation to carry out the
functions of the CASS, including overseeing and managing any personnel to whom
CASS functions and duties are delegated. Note that for smaller organizations where
personnel share duties and may only carry out CASS functions part-time, this
oversight and management responsibility relates only to those part-time tasks.
A single person or position within the operator should have authority for the CASS, and a single person or position within the operator should have overall responsibility for managing and implementing the CASS. A single person may have both responsibility and authority for the CASS. That person might also have responsibility for other functions as well as the CASS. It would be common for the person with responsibility for CASS functions to delegate some or much of this work to others within the operator, depending on the size and staffing of the operator. What the ECAA expects is clear responsibility for the overall CASS so that there is not a fragmented system with high risk of confusion over who is responsible for a given task.
Personnel with CASS responsibilities and
duties should be as independent as possible from the day-to-day operations of
the inspection and maintenance program. Ideally, the personnel conducting
audits would work in separate departments from the departments performing the
actual inspection and maintenance activities of the operator. However, this is
not necessarily feasible for small operators. At small operators, personnel
performing CASS functions, particularly audits, may consist of one or more of
the following:
•
"Borrowed" personnel from certain other shops or departments. The
operator's procedures should include ways to avoid having these individuals
assigned to audit areas where they normally work.
•
The company owner or chief executive officer, particularly if there are no
other employees and the CASS audits are focused on outside vendors and
maintenance providers because all or most of the actual inspection and
maintenance work is accomplished through contracts.
•
Outside resources contracted to perform audits and analysis for the company.
•
Others deemed qualified by the operator to provide the operator independent
audit, operational data collection, and analysis services that fulfill the
requirements of a CASS as described in this EAC.
e. Address the need for fluid communications and coordination
among the persons with authority, responsibility, and duties related to the
CASS.
500.
Summary of a CASS.
The regulations require that a CASS
accomplish surveillance and analysis of the inspection and maintenance programs
from two perspectives: verifying performance and verifying effectiveness. The
first two steps in the CASS process (surveillance and analysis) are carried out
in two different ways. One is based on auditing, and the second is based on
operational data collection and analysis. The results of the two types of
surveillance and analysis feed into the third and fourth basic CASS activities:
corrective action and follow-up. The following table summarizes these four
basic steps of a CASS within a system safety mode.
Verify Performance of Inspection Verify
Effectiveness of Inspection
and Maintenance Programs and
Maintenance Programs
1.
Surveillance: Audit process. 1.
Surveillance: Data collection process.
•
Create a plan based on risk assessment. •
Select data sets.
•
Perform transaction audits. •
Collect operational data.
•
Perform systems evaluation. •
Collect equipment failure data.
•
Identify hazards. •
Note trends, anomalies, and potential hazards.
2. Analysis: Accomplish risk assessment 2. Analysis: Investigate adverse indicators;
and preliminary root cause analysis. accomplish risk assessment and preliminary
root cause
analysis.
3.
Corrective Action: Complete final root cause analysis, corrective action
options, risk assessment, decisionmaking, and developing and implementing a
corrective action plan.
4.
Follow-up (Performance Measurement): Monitor corrective action, verification,
and follow-up surveillance planning.
501.
Verifying the performance of inspection and maintenance programs.
a. Surveillance of the performance of inspection and maintenance
programs.
(1)
Definition of "audit" within a CASS. The main tool for surveying
(assessing) whether the operator is properly performing (executing) its
inspection and maintenance programs is audits. For purposes of a CASS, an audit
is a formal examination of the activities of a department or area of an
operator's inspection and maintenance programs based on an established standard
such as the applicable manual. Audits are intended to ensure operator
inspection and maintenance personnel and outside maintenance providers comply
with the operator's manual, program, and all applicable requirements.
(2)
Audit procedures. The operator should have written procedures to guide its
auditing process, including the scheduling of audits. The CASS addresses both
internal and external audits. Internal audits are audits the operator conducts
within the company. External audits are audits the operator conducts of vendors
supplying parts and services to the operator. CASS procedures should include a
methodology for determining priorities and for establishing and adjusting audit
cycles (for example, 12-, 18-, 24-, 36-month cycles) so that resources are
focused on the most pressing issues. This is a risk assessment and risk
management process (see paragraph 501a(3) below for further explanation of risk
assessment and risk management).
Although the majority of the inputs to this
process would be generated internally, one additional input may be the results
of outside audits of the operator or its vendors conducted by entities other
than the operator. For example, the results of audits or inspections conducted
by the ECAA may be useful by providing an operator with:
• Specific findings requiring root cause analysis and possible
corrective action (activities discussed later in this EAC), and
• Information useful in focusing the operator's own audits and
operational data collection.
The operator may approach this initial
scheduling task in many different ways, ranging from resource allocation based
on company experience and very basic analysis to use of a sophisticated,
software-supported risk analysis process. Within this range of possible
methodologies, the ECAA expects the operator's CASS procedures to contain a
process to systematically make those decisions that are compatible with the
size and complexity of the operations. The ECAA encourages operators to make
this process as structured as possible. The operator should place priority
first on safety and regulatory compliance, and second on issues of operational
efficiency. However, an effective CASS meets all three of these objectives.
(3) Prioritizing surveillance resources. Essentially, any
methodology selected to prioritize surveillance resources (as well as to
formulate corrective action decisions later in the process) involves principles
of risk assessment. Risk assessment is a concept applicable in many aspects of
an aviation operation. This ECAA order is an example and is not the only source
of risk assessment procedures; however, it provides insight into ECAA expectations.
The ECAA encourages operators to incorporate the principles of this systematic
process to:
•
Establish a plan, including the scope of the process and priorities (for
example, detect and prevent noncompliance);
•
Specify the areas of concern for surveillance and analysis (personnel,
maintenance and inspection programs and organizations, operations, aircraft,
facilities, systems);
•
Identify hazards or potential threats to the operation;
•
Determine how likely such hazards are to be realized and actually cause harm;
•
Determine the severity of the consequences if the hazard is realized;
•
Express a combination of the likelihood and severity of harm as
"risk"; and
•
Evaluate the appropriate response to the identified risk.
A CASS should take into account four
principal potential sources of hazards:
•
Personnel (hiring, capabilities, interaction);
•
Equipment (design, maintenance, logistics, technology);
•
Workplace (environment, sanitation); and
•
Organization (standards, procedures, controls).
A number of quantitative and graphical tools
exist in the industry to help determine the gradations of a risk (high, medium,
low) based on the likelihood of an unwanted event occurring and the severity of
the consequences if it does occur. In the initial steps of the CASS process,
the appropriate response involves setting surveillance priorities based on risk
assessments aimed at maintaining compliance and safety in inspection and
maintenance. A CASS risk assessment, through the feedback loop, helps to set
the audit and data collection priorities enhancing the focus of surveillance.
The process is best accomplished by an interdisciplinary team, guided by CASS
management but involving representatives of the relevant technical areas.
To identify the areas to audit and to set
priorities, consider factors in outside reports. These could include
inspections, reports, special studies, or audits conducted by outside entities
such as the ECAA. Outside reports may address:
•
Information specific to the operator or its vendors;
•
Information related to the industry as a whole and of interest to the operator;
and/or
•
Information about an accident, incident, procedure/process, or equipment type
that is relevant.
(4) Audit materials. The operator should equip CASS auditors
with checklists to ensure consistency and completeness of audits. The person
responsible for the CASS should ensure the checklists are updated as needed. An
auditor should also be permitted flexibility to ask questions not contained on
the checklist if he or she finds an area that requires further investigation.
(5) Areas to be audited. The operator's procedures should
include identification of all areas to be audited along with a process for
updating this list. The following list presents examples of areas operators
should consider for routine audit. A CASS audit should verify that:
•
Manuals, publications, and forms (paper and electronic versions) are useable,
up-to-date, accurate, and readily available to the user;
•
Maintenance and alterations are performed according to the methods, standards,
and techniques specified in the operator's manuals, including ensuring major
repairs and alterations are properly classified and accomplished with approved
data;
•
Parts and components are properly stored, dispensed, identified, and handled;
•
Airworthiness directives are appropriately evaluated, accomplished, and
tracked;
•
Maintenance records are generated in accordance with manual procedures and are
complete and correct;
•
Required inspection items are identified and addressed according to the
operator's procedures;
•
Airworthiness releases are executed by authorized persons according to the
operator's procedures;
•
Shift turnover records, work interruptions, and deferred maintenance are
accomplished according to applicable procedures;
•
Maintenance facilities and equipment, including base and line stations and
contract maintenance providers' facilities, are adequate;
•
Personnel, including those of contract maintenance providers, are trained and
qualified to accomplish their duties;
•
Tools and equipment are properly calibrated;
•
Requirements for specialized tools or training are met, such as for
nondestructive testing, category II/III operations, and run-up/taxi;
•
Computer programs (software) for the inspection and maintenance programs are
performed in accordance with specifications;
•
Vendors and suppliers provide services and products according to the operator's
policies and procedures; and
•
Each aircraft released to service is airworthy.
(6) Objective of CASS audits. CASS audits should primarily be
proactive, searching out potential problem areas before they result in
undesirable events. However, CASS procedures may also address how to direct
audits in response to events or a series of events. For example, rejected
takeoffs, unscheduled landings, in-flight shutdowns, accidents, or incidents
may indicate the need for special audits or surveillance under a CASS. The
purpose of a CASS is to detect and analyze trends for indications of program
weaknesses or deficiencies. For example, CASS auditors would not necessarily
investigate a single maintenance-related rejected takeoff, although the
maintenance program would. A CASS would, however, consider whether that instance
indicated a need to focus audits on a particular issue.
(7) Informal communications within CASS. Auditors and analysts
should maintain informal lines of communication with personnel in the
production departments so that inspection and maintenance personnel can discuss
concerns they may have. Through this informal communications process, the
operator can learn about potential hazards in the system. For example, the
operator may learn about an event that might have occurred but, because of some
intervention, did not. This event is known to shop personnel but is otherwise
difficult or impossible to detect in routine audits. With informal lines of
communication open to shop personnel, a CASS may detect this near-event. The
ECAA suggests that the operator's CASS procedures address how to encourage this
type of communication and interaction.
b. Analysis of audits.
(1)
Root cause analysis. A risk assessment process tells operators where to
allocate resources and helps them understand what is found. Audit results
should undergo risk assessment and preliminary root cause analysis to identify
a deficiency, or potential deficiency, in any aspect of inspection and
maintenance programs. This preliminary analysis helps CASS personnel determine
the level of priority the issue merits and what type of additional technical
expertise may be required to complete the root cause analysis and evaluate
corrective action options.
Root cause analysis treats errors as defects
in the system rather than in a person. Root cause analysis looks beyond the
symptom to find the organizational defect that permitted an error to occur to
correct the fundamental problem, and to prevent recurrence. The more thorough
the analysis, the greater the likelihood the operator will uncover why the system
deficiency occurred and how the organization can respond definitively. The
process starts during the audit itself, as auditors must collect information
conducive to later analysis. If a CASS is to uncover a procedural weakness, for
example, information about the procedure must be collected. This should be
factual and objective information, not premature judgment about root cause.
Root cause analysis is key to any complete CASS, even though procedures may
vary in complexity from operator to operator.
(2)
Objective of audit analysis. The objective of this analysis is to allow the
operator to address the problem in such a way as to avoid recurrence of the
deficiencies. To the extent possible, the operator should set forth in the CASS
documentation the analysis process. The analysis process should be as objective
as possible to avoid any tendency to promote individual or commercial
interests. The system should also place priority on finding the systemic or
root cause of a program deficiency over seeking to assign personal blame, at
any level of the organization, for an error.
While audits are designed mainly to verify
that an operator is performing inspection and maintenance in accordance with
its manual, the regulations, and applicable requirements, auditors and analysts
should also be alert for system deficiencies. That is, there may be procedures
in the manual that are correctly followed, but that have become outdated,
conflict with other manual procedures, or for some other reason are in need of
change. Auditors and analysts should be encouraged to be inquisitive and think
in terms of "what if?" so that the CASS functions proactively,
detecting problem areas or trends before they lead to an accident, incident, or
infraction of regulations. For example, what if event x occurred in conjunction
with observed condition y? This approach is closely tied to the CASS analysis
process but would require an analytical approach that permeates the CASS
organization, from determining audit priorities and scheduling through auditing
and analyzing, and including monitoring and evaluating corrective actions.
(3)
Managing data from audit analysis. The audit analysis process is not typically
as oriented toward quantitative analysis as the operational data analysis
discussed below. However, operators may find it useful to manage the data
through database or quantitative applications. The ECAA emphasizes that this is
an approach that does not have to be complicated or costly. The level of
formality and sophistication should match that of the operator.
502. Verifying the effectiveness of
inspection and maintenance programs.
a. Surveillance of the effectiveness of inspection and
maintenance programs.
(1)
Collecting operational data. The main tool for determining whether an operator's
inspection and maintenance programs are effective is collecting and analyzing
operational data focused on the equipment. Data should be collected that
measures the output of the inspection and maintenance programs. The ECAA does
not intend to mandate the specific data an operator should collect. However,
the ECAA does expect an operator to have a process to ensure the data collected
are adequate to meet the intent of the CASS requirement and are useful. The
ECAA expects an effective selection process and periodic review process, not
specific data elements that may not fit a given operator's situation.
(2)
Types of operational data. Operational data can be divided into routine or
unplanned (nonroutine). Examples of routine data are:
•
Adjustment and/or calibration of equipment;
•
Aircraft logbooks, including maintenance deferred in accordance with the
minimum equipment list/configuration deviation list;
•
"Chronic" systems that alert for repeat writeups in a specified time
period (for example, 10 to 15 days);
•
Corrosion prevention and control program findings;
•
Engine condition monitoring information;
•
Flight delays and cancellations related to mechanical issues;
•
Results of fuel audits;
•
Individual item failure rates;
•
reliability reports, mechanical interruption summaries, and similar data;
•
maintenance;
•
Teardown reports;
•
Unscheduled parts replacement or unscheduled maintenance; and
•
Vendor repair station information.
Operational data also includes reactive data
collection and analysis responding to emergency or other nonroutine events,
such as:
•
Accidents and incidents;
•
In-flight engine and propeller separations and uncontained engine failures;
•
In-flight engine shutdowns;
•
takeoffs;
•
landings due to mechanical issues;
•
Lightning strikes; and
•
Hard landings.
As with reactive audit surveillance, a CASS
generally approaches problems from the analytical, systems perspective. For
example, in response to one or more rejected takeoffs, a CASS might focus the
operational data collection and analysis to determine if a pattern in rejected
takeoffs was evident, or if other types of data might be examined in relation
to the rejected takeoff situation.
The above data sets are presented only as
examples. Although the data sets are oriented toward equipment, this area of a
CASS may also collect other types of data, such as information on types of
maintenance errors experienced by the operator.
(3)
What to include in CASS documentation regarding collecting operational data.
The operator's CASS documentation should include a means of identifying data
that is relevant and useful for that operator to use in monitoring the
effectiveness of its specific inspection and maintenance programs. The operator
should periodically review and reevaluate the usefulness of the data it
collects and analyzes to accomplish this portion of the CASS.
b. Analysis of operational data. CASS procedures should:
(1)
Provide analysts with an understanding of the potential significance of each
data set and how to process the data to understand its significance. This may
require:
•
Statistical analysis, such as comparing the frequency of certain events or
equipment failures with a determined norm, or
•
Qualitative analysis, to evaluate reports of certain types of events.
NOTE:
This process is not necessarily the same as
what would be used in an ECAA-approved reliability program.
(2)
Emphasize that the analysis of operational data should consider root causes of
negative trends or anomalies. This preliminary root cause analysis, including
human factors, may require collaboration with technical personnel in the
affected areas or specialists in engineering and reliability departments.
(3)
Delineate the roles of the CASS analysts as well as other departments or
personnel in the analysis of operational data.
Some operators select a system that uses
alerts or warnings if results of the analysis exceed certain predetermined
parameters. A CASS should not rely completely on such alerts to the exclusion
of analysts' judgment. The FAA's expectation of a CASS in this regard is that
the operator have a complete, written procedure to review and analyze the
operational data collected and to determine when further review is necessary.
503. Final root cause analysis and corrective
action.
While the surveillance and analysis steps
differ for the verification of the performance of the inspection and
maintenance programs versus verification of the effectiveness of those
programs, the process merges when responding to CASS findings. The two types of
analyses identify potential deficiencies in the inspection and maintenance
programs. In responding to these findings and analyses, the objective of a CASS
is to determine the root causes of program deficiencies and address them
appropriately, regardless of the perspective from which the deficiencies are
found. Note that the discussion is focused on a CASS function, not an
organization. For a given operator, that function might be performed by more
than one organization.
Generally, the area responsible for
surveillance results will present these results to the technical or production
area of the operator with a preliminary analysis of the collected information
and, in some cases, possible underlying causes of the problem. Personnel in
technical or production areas complete the root cause analysis (if necessary)
and develop proposed corrective action alternatives.
a. Final root cause analysis.
(1)
Preparing for root cause analysis. Analysis of audit findings or operational
data requires evaluating mechanical and human performance, or other results
generated by the CASS process, to determine the condition of a process,
maintenance practices, or equipment. In the case of operational data, analysis
begins with comparison of the data to a standard representing acceptable
performance. The standard may be in the form of an average or other means of
calculating a reference. The standard may be set by the industry common
practice, or the operator, as appropriate.
The key is to have a CASS structure that
addresses the basic disciplines and elements involved in finding and correcting
program deficiencies. The CASS procedures should note that in performing root
cause analysis, all relevant areas should be considered, including the role of
senior management, policies, procedures, and communications.
(2)
Applicability of root cause analysis. Root cause analysis applies to both audit
findings and analysis of results and trends in the operational data. For example,
either audits or operational data analysis may point to maintenance errors
being committed because of inadequate training. Analysis should not stop with
simply determining which mechanics were inadequately trained and then training
them. Rather, the analysis should determine why the training breach occurred
and consider areas in management, communications, scheduling, or training
program design that may be involved.
(3)
Principles and considerations of root cause analysis. Principles and
considerations of root cause analysis are closely related to those of risk
assessment, particularly in terms of the thoroughness of the analysis. Both
processes consider not simply the person involved in an issue (for example, the
mechanic made a mistake), but all aspects of the organization in which that
person works. This approach has the premise that human error is a consequence
rather than a deliberate action, and that proactive measures and continuous
reform of different aspects of the processes and organization can address
"latent conditions" in the system and increase the system's
resistance to operational hazards. The term latent conditions refers to flawed
procedures or organizational characteristics capable of creating hazards if the
right conditions or actions occur.
Root cause analysis should consider two major
areas:
•
Systems. Systems analysis plays an increasingly important role in a CASS
because of the increasing complexity and variety of operations, equipment, and
organizations. Systems analysis emphasizes a coordinated approach to an
enterprise, including specific written procedures and planning for all
activities, clearly established authority and responsibilities, communications
processes, and methods of measuring results, detecting system errors, and
preventing recurrence. This approach recognizes the wide range of interrelated
issues potentially associated with a problem in the system, such as management
policies, communications, and pilot technique, in addition to the inspection
and maintenance activities themselves.
•
Human factors. Human factors analysis looks at how humans communicate and
perform in the work environment and then seeks to incorporate that knowledge
into the design of equipment, processes, and organizations. This enhances safety
and maximizes the human contribution, partly by designing systems to anticipate
the inevitability of human error. Human factors include basic issues that can
be addressed in audit checklists, such as whether there is adequate lighting
for mechanics and inspectors to perform their work, and whether schedules
permit personnel to be properly rested. But the discipline addresses a wider
range of issues affecting how people interface with technology and the
operational system, including:
•
Human physiology;
•
How people learn and perceive;
•
Equipment, technology, and documentation; and
•
Workplace.
Operators should be aware that knowledge
gained from human factors can help avoid maintenance and inspector errors,
ensure that personnel initial skill sets match task requirements, ensure skills
are maintained and improved, and enhance the work environment. This knowledge
can help CASS analysts perform root cause analysis. Continuing with the
previous example of inadequate training, with insufficient awareness of human
factors issues, operators may trace a maintenance error to a mechanic or
technician who appears to be insufficiently trained for the task, and determine
that the solution is more technical training. Further analysis may reveal,
however, that there are contributing flaws in equipment design, job cards,
manuals, the work environment, or organizational procedures such as shift
turnover that more training will not satisfactorily overcome. Or, it may turn
out that a different kind of training, perhaps involving decisionmaking skills,
is called for.
The ECAA is deeply involved in cooperative
efforts with the industry and academia in promoting human factors in aviation.
This field is rapidly evolving, particularly in its application to aviation
maintenance. Based on the field's growing importance and the information
available to industry, the ECAA expects that operators will apply concepts of
human factors to their CASS surveillance and analysis.
CASS surveillance also should ensure root
cause analysis, considering human factors, is part of the investigation of
individual events by any personnel designated to respond to such events, such
as rejected takeoffs. Otherwise, data reviewed in a CASS may be incomplete.
One challenge presented by the increasing
emphasis on human factors is how to balance two seemingly contradictory
purposes. On the one hand, the ECAA and industry need to encourage personnel to
cooperate in addressing system organization and design issues without
inhibitions caused by fear of discipline or enforcement. On the other hand, in
some cases, individual employees or the operator may bear a degree of
culpability (for example, in deliberately bypassing important controls or
committing a serious regulatory infraction in the commission of a maintenance
error). In some instances, disciplinary action or even ECAA administrative or
legal enforcement may be indicated. This is a common issue in industry and ECAA
programs designed to promote the greater good of the system by encouraging
voluntary reporting of errors and infractions by aviation personnel and
operators without threat of disciplinary action or penalty. A CASS, in any
event, is concerned specifically with identifying and correcting deficiencies
in the inspection and maintenance programs and should be designed to that
objective, rather than specific event resolution, even if CASS analysts
research specific events.
b. Analytical tools and processes. While it is not necessary for
an operator to implement any specific externally developed system, analytical
tools or processes are available to assist in the analysis process. Examples of
these are:
(1)
Maintenance Error Decision Aid. Developed by the Boeing Human Factors
Engineering group in collaboration with the FAA, airlines, and the International
Association of Machinists for analyzing human performance issues related to
maintenance errors and trends. Operators use the Maintenance Error Decision Aid
to track events, investigate and prevent maintenance errors, and identify
contributing factors, corrective actions, and prevention strategies. A software
analysis package has been developed to work with this aid and facilitate
analysis of systemic issues.
(2)
Managing Engineering Safety Health. Developed by the University of Manchester
in collaboration with British Airways Engineering. This system is geared toward
researching the workplace and organizational environment in aircraft
maintenance to find the issues with the greatest potential to contribute to
human factors problems. The system uses software, diagnostic, and sampling
tools. Managing Engineering Safety Health conducts anonymous survey-like
assessments among personnel at the work location, which are then analyzed.
(This is a more structured, data-intensive approach toward determining and
monitoring personnel attitudes toward the system than the interview process
discussed earlier. The industry has far less practical experience with Managing
Engineering Safety Health than with the Maintenance Error Decision Aid.)
(3)
Human Factors Accident Classification System Maintenance Extension. Developed
by the U.S. Naval Safety Center in collaboration with the ECAA for use in the
air carrier industry as well as naval aviation. This comprehensive system
incorporates a number of analytical tools and has profiled maintenance errors
and contributing conditions, permitting development of potential prevention
measures. While the Human Factors Accident Classification System Maintenance
Extension may be more sophisticated than many operators would need, it
demonstrates principles and techniques of software-aided analysis that could be
applied to a CASS.
c. Corrective action options.
(1)
Determining whether or not to proceed with a corrective action. Once the CASS
auditors and analysts have identified a problem or deficiency, the operator
must determine if a corrective action is warranted and, if so, the details of
the corrective action.
(2)
CASS procedures regarding determining whether to proceed with a corrective
action. CASS procedures should outline:
•
How such a determination will be made;
•
Who will make the determination; and
•
What levels of review, if any, will be performed.
(3)
Developing the proposed corrective action. Technical area personnel should have
primary responsibility for developing the proposed corrective action, as they
would be most familiar with the technical workings of the area in question and
would be sensitive to the possibility of creating new problems as a result of
the corrective action. CASS procedures should emphasize a team approach. Team
members should include the CASS auditors or analysts, technical area personnel
in the affected maintenance and inspection disciplines, and perhaps other
affected areas such as training or flight operations.
(4)
Types of corrective actions. There are several possible types of general
corrective actions or responses, depending on the outcome of the risk
assessment.
•
Prevent recurrence through engineering or system changes designed to eliminate
the risk.
•
Accept the underlying cause of a trend or discrepancy, but reduce the risk
through implementing controls or countermeasures. Examples are training, policy
or procedure revisions, or warning devices. Other countermeasures might be
modifying or introducing new equipment or technology.
•
Accept that under certain conditions a discrepancy may occur, and be prepared
to contain or mitigate the results of that situation. A CASS does not
necessarily have to implement corrective actions for every apparently negative
trend or finding. Analysis of findings or trends may identify problem areas
that do not present safety hazards and that the operator is willing to accept,
in accordance with its risk assessment process. For example, the operator might
find that a higher than average number of component removals with "no
fault found" occurs at a particular location. The operator might determine
that the reason for this situation is that the aircraft spends insufficient
time on the ground for line maintenance to completely isolate the fault. The
operator might prefer to continue the brief turn times and simply switch
components. This would be a business decision for the operator to make.
However, more comprehensive corrective actions would be mandatory if the CASS
detects that the inspection and maintenance programs lack adequate procedures
and standards to meet the requirements of Part 121.
d. Written procedures for developing and implementing corrective
actions. A CASS should provide written procedures for developing and implementing
corrective action based on the operator's organizational structure and the
training of its personnel. The procedures should:
(1)
Result in a specific corrective action plan that addresses basic questions of:
•
Development and proposal of the corrective action;
•
Analysis and final approval level of the corrective action, including who is
responsible for approval of the corrective action;
•
will implement the corrective action;
•
the responsible person will implement the corrective action;
•
the corrective action should be completed;
•
Who will evaluate the outcome, and how, including identification of data to be
collected, awareness of the possibility of unintended consequences, and events
that should trigger a response;
•
Who will monitor the status of the corrective action, and how; and
•
Reporting the status of the corrective action (to whom, with what frequency).
(2)
Maintain the appropriate role of auditors in developing responses to findings
so that they continue to remain independent from the corrective actions they
may subsequently audit.
(3)
Distinguish clearly between the technical area personnel's responsibility for
developing and implementing corrective actions, and CASS personnel
responsibility for producing the findings and analysis and making sure the
technical area involved develops and implements appropriate corrective actions.
(4)
Designate the position or organization responsible for evaluating and approving
proposed corrective actions. The CASS director or other designated manager may
appoint a corrective action team to design and propose a corrective action. The
team-which typically represents a cross section of the departments involved in
audits, operational data collection, analysis, and production-oversees the
implementation of the corrective action. Technical and reliability control
boards are most often used in conjunction with ECAA-approved reliability
programs; however, a similar concept applies to a CASS, even if no ECAA-approved
reliability program exists.
e. Corrective action risk assessment.
(1)
CASS procedures regarding risk assessment. CASS procedures should:
•
Specify that personnel will analyze a proposed corrective action carefully
before its selection and implementation to ensure corrective action is
necessary and will actually fix the problem and not lead to unintended negative
consequences.
•
both CASS and technical area personnel of the need to consider the impact of
the proposed corrective action on other aspects of the operation. This would
include other areas of the inspection and maintenance programs, such as
manuals. The corrective action may require coordination with other areas, such
as flight operations, that might be affected.
(2)
Personnel involved in risk assessment. Technical area personnel play the key
role in risk assessment, but the process should include the CASS analysts, who
will act as resources in support of the technical area managers and bring risk
assessment and systems analysis techniques to the process. The auditor and
analyst should be qualified (through training or experience) in systems
analysis and can contribute to the evaluation of a proposed corrective action
by determining if the basic system elements have been considered. However, the
technical personnel have the expertise to actually develop and implement the
corrective action, and to evaluate it in practical terms. Thus, the corrective
action is a result of cooperation between the technical personnel and the CASS
personnel.
Personnel working on the proposed corrective
actions should ensure they consider issues of a timetable for the corrective
action implementation, as well as the safety attributes of authority,
responsibility, procedures, controls, process measurement, and interfaces.
f. Corrective action plan.
(1)
With the root cause analysis complete, corrective action options identified,
and risk assessment performed as appropriate, a final decision can be made on
the proposed corrective action plan. The corrective action plan should address
all relevant issues, including a timetable for completion of the action, with
milestones, if appropriate. The appropriate technical department (and other
departments, such as flight operations, if the corrective action goes beyond
the inspection and maintenance organizations) should then implement the plan.
(2) The CASS procedures should
identify:
•
How this plan will be approved and at what level of the company, and
•
The parties responsible for implementing, monitoring, and ensuring all affected
parties are notified, both within inspection and maintenance and externally, if
necessary.
504. Follow-up.
a. Monitoring corrective actions. The CASS procedures should:
(1)
Specify how implementation of corrective actions will be monitored and evaluated.
This may require the following:
•
Follow-up audits of a specific area;
•
Regular communication from the affected technical area as to the status of the
corrective action; and/or
•
Other forms of verification action by the auditors or analysts tracking the
implementation.
(2)
Identify the person or entity (such as a CASS board) responsible for
determining if any changes in the status of a corrective action are acceptable.
The CASS auditors or analysts have the duty of ensuring the corrective action
has been implemented in accordance with the established timetable or, if not,
determining why the timetable has changed.
(3)
Include responsibilities and guidelines for:
•
Tracking the implementation of corrective actions in accordance with the
timeline;
•
The role of auditors, managers, management committees, and senior management;
•
How automation or computerized systems will be used;
•
How risk assessment and/or systems analysis will be used to guard against
unintended consequences;
•
Measures to evaluate the effect of the corrective action; and
•
The affected technical area to communicate the status of the corrective action
to the person responsible for monitoring implementation.
b. Getting help from a manufacturer. In some cases, the operator
may require data or assistance from a manufacturer in correcting a deficiency
detected by the CASS. Manufacturers may not always assign these issues the same
priority as the operator does. The operator should offer guidance in its CASS procedures,
based on its particular experience, on how CASS and other personnel should
address requested assistance or information from manufacturers, and how to
proceed in case of unsatisfactory or slow responses. This may include
developing a standardized letter citing the need for this information or
assistance to satisfy the requirements of 121.373 or other pertinent
regulations. It may also include working with the ECAA principal inspector to
find solutions.
c. Follow-up surveillance plan. CASS procedures should include
how to determine the level of follow-up audits for verifying corrective action
implementation. For example, based on the risk assessment or complexity of the
corrective action, the designated CASS analyst or team may schedule special or
more frequent audits. They may also change the data collection process or
institute other means of verification. The ECAA expects the operator to have a
well-designed and logical process to design the follow-up actions.
The information and analysis performed
through the closed-loop, continuous cycle of surveillance, investigations,
analysis, and corrective action permits the operator to refine its audit and
data collection priorities through the risk assessment process.
600.
Personnel managing CASS functions.
a. A CASS should include a decisionmaking body at a relatively
high management level to oversee or carry out CASS functions. These oversight
groups could include:
(1)
Technical boards concerned with performance and other technical issues;
(2)
Administrative boards that may have broader decisionmaking authority to act on
technical recommendations; or
(3)
A single board combining both functions.
The key concept is that there be a
decisionmaking body at a relatively high management level to monitor the CASS
and to make critical decisions in a timely manner. Typically, at a smaller
operator, this committee or board may be composed of the president of the
company and the directors of maintenance and flight operations. Typically, at a
larger operator, participants may be managers from several departments, such as
maintenance and engineering, quality assurance, and operations.
b. If the operator uses committees or boards as major
decisionmaking bodies for CASS issues, members of these bodies should:
(1)
Have an appropriate technical background, and
(2)
Be thoroughly familiar with the role and functioning of the CASS, systems
analysis, and the evaluation of the root cause analysis and proposed corrective
actions submitted for their review.
The operator should consider requiring
participants in such committees or boards to receive training or orientation on
these issues to ensure they can provide critical evaluation. The membership of
such boards and committees as well as the basic operating procedures and
records should be described in the CASS document.
601. CASS personnel training and experience.
a. Maintenance. Each operator should determine the precise mix
of training and experience needed by the operator's auditors and analysts. In
general, auditors and analysts should:
(1)
Have sufficient maintenance background applicable to the operator's program to
ensure they are familiar with inspection and maintenance procedures, technical
documents, and aircraft systems.
(2)
Be able to understand and interpret the answers and data they see, as well as
evaluate facilities, equipment, and processes they observe. While they are
unlikely to have specialized knowledge in all of the areas over which they
conduct surveillance, a foundation of technical expertise is important.
b. Surveillance and analysis. Auditors and analysts need
training and/or experience in the functions they are responsible for
surveilling and analyzing. It is also essential that they have training and/or
experience in the following areas:
•
Systems analysis;
•
Auditing techniques;
•
Risk assessment and risk management;
•
Root cause analysis; and
•
Human factors.
Additionally, operators may seek specialized
training in specific quality processes or systems for their CASS personnel,
such as:
(1)
ISO 9000, a quality system set of standards developed by the International
Organization for Standardization that seeks to standardize processes into
organized and documented systems.
(2)
Six Sigma, which is process-oriented from an intensively data-oriented,
statistical approach.
c. Technical. Persons who collect and analyze operational data
may require specialized technical backgrounds, such as engineering. This will
depend on the complexity of the operational data the operator collects. These
personnel may work in the unit conducting an ECAA-approved reliability program
or in an independent data collection and analysis system.
d. Summary of experience and training for CASS personnel. The operator's CASS document should reflect that the carrier has considered the type of experience and training, both initial and recurrent, appropriate to the auditors and analysts in its operation.
CHAPTER 7. COMMUNICATION BETWEEN CASS PERSONNEL AND
OTHER DEPARTMENTS
The procedures for communicating CASS
information and results internally to interested parties within the operator
and, as applicable, externally (for example, vendors, the ECAA) vary depending
on factors such as the size and nature of the operation, level of automation,
and the CASS procedures themselves. The number and complexity of the
standardized communications processes, such as forms or electronic mail
messages with standard distribution, should be appropriate to the overall size
and scope of the operator's operation and CASS.
700. Communicating specific CASS results and
actions.
a. The operator should develop appropriate standard
communication processes for all aspects of the CASS to assist in standardizing
procedures, including the following:
(1)
Audit checklists and results.
(2)
Analysis procedures and results.
(3)
Records of audit/analysis findings - internal.
(4)
Records of audit/analysis findings - external.
(5)
Corrective action forms and/or action plans. These forms should address system
considerations to ensure there is a clear understanding of when the corrective
action will be implemented, who is responsible, and what the impact will be on
written procedures.
(6)
Information for monitoring and follow-up of corrective action. The processes
should also assist in tracking the implementation of corrective actions once
underway.
(7)
Periodic status reports to senior management and to the ECAA.
b. The CASS description should address such issues as the
following:
(1)
Who is responsible for keeping these standard communication processes
up-to-date and available;
(2)
Who is responsible for completing the standard communication processes;
(3)
Where are communications sent, who must respond, and how are responses tracked;
and
(4)
How, where, and for how long completed records are retained.
701. Educating personnel on CASS.
A CASS should include procedures and
responsibility to create some form of communication between the area
responsible for the CASS, other areas of the company, and the ECAA. This may be
accomplished through training, newsletters, bulletins, meetings, or other
formats determined by the operator. One purpose of such communication is to
educate mechanics and other departments that feed information and data into the
CASS about why these data are necessary, what is done with the data, and how
this process benefits the operation.
702. Communications with personnel outside
the CASS.
The ECAA expects a good communication system
to meet the objectives in this section. Each operator must determine which
system is best for its operation.
a. A CASS should provide for regular, structured communications
within the CASS structure and between the CASS and any other resources involved
in decisionmaking for the operator. Examples of these would include:
(1)
Avionics and other shops;
(2)
Cabin safety organization;
(3)
Engineering department and ECAA-approved reliability program organization;
(4)
ECAA certificate management office or principal inspector;
(5)
Flight operations;
(6)
Ground operations;
(7)
Inspection department;
(8)
Internal evaluation program;
(9)
Maintenance control;
(10)
Maintenance operations;
(11)
Manufacturers' technical representatives;
(12)
Purchasing;
(13)
Quality assurance;
(14)
Receiving inspection;
(15)
Recordkeeping organization;
(16)
Safety program;
(17)
Senior management;
(18)
Stores department; and
(19)
Training departments.
b. The communications mechanisms should include a feedback loop
designed to ensure that any changes implemented as a result of corrective
actions are functioning as intended and are improving the process. This level
of communication may be accomplished through a variety of means, including the
following:
(1)
Periodic (weekly, monthly, quarterly) statistical and narrative CASS reports on
trends, findings, and the status of corrective actions.
(2)
Periodic CASS meetings to discuss trends or specific problem areas. Such
meetings might be informal but frequent, such as at very small operators where
the relevant managers work in close proximity, or more structured and formal,
such as at larger operators where specific boards or committees may be
designated.
(3)
CASS board or committee meetings, including senior management, possibly on a monthly
or bimonthly basis. Even if meetings are somewhat informal, minutes should be
kept.
c. Typically, operators with programs incorporating statistical
performance standards (alert-type programs) develop a periodic (monthly)
report, with appropriate data displays, summarizing the previous month's
activity. To help evaluate the effectiveness of the total maintenance program,
the report should cover all aircraft systems controlled by the ECAA-approved
reliability program. An operator without an FAA-approved reliability program
may find that using a similar report can enhance its CASS.
CHAPTER 8. HOW THE CASS DIFFERS FROM AND RELATES TO OTHER PROGRAMS
800.
Summary of other programs.
The operator's description of the CASS should
identify other related programs in which the operator participates and explain
how CASS relates to those programs and/or differs from them. Experience has
shown that certain other programs are potential sources of information for the
CASS, while other programs may be integrated into a CASS. Some programs have
been mistakenly assumed to be so similar to a CASS that the operator might
neglect an important aspect of the CASS. Therefore, the CASS documentation
should describe the relationship between the CASS and programs such as the:
a. ECAA-approved Reliability Program;
b. Internal Evaluation Program;
c. Safety Program;
d. Voluntary Disclosure Reporting Program;
e. Coordinating Agencies for Supplier's Evaluation;
f. Aviation Safety Action Program; and
g. Aviation Safety Reporting Program.
801. Discussion of individual programs.
a. ECAA-approved reliability program. Maintenance Control by
Reliability Methods, the concept of reliability control was developed to
maintain an acceptable level of reliability and evolved based on ECAA and
airline efforts to develop more responsive methods of controlling maintenance
without sacrificing safety or ECAA regulatory responsibility. An ECAA-approved
reliability program includes systems for data collection and analysis,
corrective action, statistical performance standards, data display and
reporting, maintenance program adjustments, and process changes. Under the
program, the operator may adjust maintenance, inspection, and overhaul
intervals up to a specific limit without prior ECAA approval.
Typically, larger operators have an ECAA-approved
reliability program, but the operational data collection and analysis
requirements of such a program usually exceed the resources or requirements of
smaller and even most medium-sized operators and generally are greater than
what would be necessary for those operators' CASS. However, if an operator does
have an approved reliability program, this may be incorporated into the CASS as
the means of performing operational data collection and analysis to monitor the
effectiveness of the inspection and maintenance programs. That operator's CASS
procedures should describe how the approved reliability program is integrated
into the CASS. An ECAA-approved reliability program cannot substitute for a
CASS because the reliability program does not include the broader auditing
surveillance and analysis of the full range of elements of the inspection and
maintenance programs, nor does it include the complete processes for developing
and implementing corrective actions.
This EAC is not intended to describe ECAA-approved
reliability programs. However, CASS operational data collection needs are
typically similar to, if less extensive than, those of an approved reliability
program. An operator may, within its CASS, establish a program similar to an ECAA-approved
reliability program for the purpose of collecting and analyzing operational
data. In such circumstances, the carrier would not be permitted to adjust its
inspection or maintenance program without ECAA approval. Additionally, the
operator must ensure its operational data collection program meets the needs of
its CASS.
It is common to use "reliability,"
in a generic sense, in reference to dispatch availability of equipment or in
relation to equipment failure rates. If an operator's CASS manual or document
uses this terminology, it should distinguish whether the reference is to an ECAA-approved
reliability program or to generic reliability.
b. Internal evaluation program. An internal evaluation program
is a voluntary program to provide measurement of an operator's internal
processes and procedures to assess whether they are adequate and functioning
properly. An internal evaluation program should be independent of all other
programs and systems and could be a useful tool to evaluate a CASS, as well as
other systems or programs, such as the operator's safety program. An internal
evaluation program is a very high-level review to provide information to senior
management as to how well critical programs, such as a CASS, are working. It
would not be a substitute for a CASS. An internal evaluation program is a
broader system evaluation program and is less "audit-oriented" than a
CASS, although both use a system evaluation approach. An internal evaluation
program poses questions necessary to determine if the operator's systems, such
as its CASS, are effective and efficient, and if the current program would
support further growth.
EAC 00-1, Air Carrier Internal Evaluation
Programs, describes the internal evaluation program. The internal evaluation
program should not be misunderstood as a program that replaces existing
regulatory auditing requirements such as a CASS. Audits are a very minor part
of an effective internal evaluation program.
c. Safety program. Certificate holders conducting operations
under Part 121 are required to have a director of safety or equivalent position
unless the ECAA permits a deviation in the required management positions. The
director of safety should oversee a function that addresses the range of risks
involved in commercial aviation, including flight operations, maintenance, and
ground operations. The director of safety should manage a comprehensive safety
program with a variety of elements, such as investigations of and a reporting
system for accidents and incidents, safety audits and inspections, operational
risk assessment, and trend analysis.
d. Voluntary Disclosure Reporting Program. EAC 00-1, Voluntary
Disclosure Reporting Program, provides guidance on procedures for certificate
holders to use when voluntarily disclosing to the ECAA apparent violations of
certain Regulations. An operator's participation in the program may reveal
important information regarding maintenance issues and lead to the development
of comprehensive fixes relevant to the inspection and maintenance programs a
CASS oversees.
Under this program, the operator may
voluntarily report violations of regulations that it discovers and avoid
certain enforcement consequences. Some operators may be concerned about
discussing regulatory infractions in widely disseminated CASS documents, even
if they are addressed through the Voluntary Disclosure Reporting Program.
It is not required that a CASS address
disclosures made under the Voluntary Disclosure Reporting Program. However, the
ECAA recommends that the operator consider, in developing its CASS procedures,
whether to attempt to include information from voluntary disclosures in its
CASS in any fashion. For example, CASS personnel may be the same personnel as
those who handle voluntary disclosures. They may therefore be able to use
"de-identified" information from voluntary disclosures to point to
areas where additional auditing may be necessary. CASS personnel should be
aware of comprehensive fixes developed in conjunction with the Voluntary
Disclosure Reporting Program. These are, after all, precisely the types of
systems or procedural modifications that an effective CASS is seeking, to avoid
adverse audit findings or unwanted operational performance.
900.
Why a CASS should be evaluated.
As with any system or program at the
operator, the CASS itself should be evaluated (that is, a process measurement
should be accomplished) so that any personnel responsible for overseeing the
CASS, such as the operator's top management, may be confident that the CASS is
accomplishing its function. Verifying that a CASS is working as intended is
also a primary task of the ECAA principal inspector.
A common misconception is that an operator
can evaluate its CASS based solely on the results of the inspection and
maintenance programs. That is, it is common to assume that if the aircraft are
consistently airworthy, the CASS must be doing its job. However, this favorable
result may occur for other reasons, such as the extraordinary diligence or
memory of a few individuals. The purpose of the CASS is to ensure, with a
system-oriented, structured approach, that inspection and maintenance programs
are properly executed and are effective consistently and by design rather than
by luck. The operator should not assume that good maintenance is synonymous
with the CASS working properly.
Thus, personnel with CASS oversight
responsibilities (including the ECAA) require a different approach to determine
if the CASS is indeed working properly. They need to know that the operator has
complete CASS policies and procedures to monitor and evaluate the inspection
and maintenance programs, that these policies and procedures are being carried
out, and that they work. For example, to ensure the CASS is functioning
properly, a senior operator manager would not analyze component removal rates,
but rather verify that the CASS is analyzing component removal rates, detecting
trends as appropriate, and implementing corrective actions when necessary. The
operator should have procedures, either in the CASS manual or referenced in the
CASS manual but contained in another document (such as its internal evaluation
program manual), for evaluating the CASS and informing top management of the
effectiveness of the CASS, separate from the effectiveness of the inspection
and maintenance programs. The regulations not only require inspection and
maintenance programs that meet many specific standards, they also separately
require a system to monitor those programs.
901. Steps to evaluate the CASS.
The ECAA expects an operator to develop its
own methods of evaluating whether its CASS is working properly, including how
the operator intends to measure whether it has allocated sufficient staffing
and resources to its CASS.
a. System safety attributes. Determine that the CASS addresses
applicable system safety attributes (responsibility, authority, procedures,
controls, process measurement, and interfaces). If the operator has an internal
evaluation program that follows this format, it would provide the operator's
senior management with an appropriate means of evaluating the CASS. That would
be one way, but not the only way, to evaluate a CASS.
b. Indicators. The following questions may be useful in
indicating whether the CASS is designed properly or working as intended,
although the operator may identify other indicators:
(1)
Are CASS personnel sufficiently independent of the areas they audit? Are they
trained specifically in their CASS responsibilities?
(2)
Are the resources allocated to the CASS sufficient to permit timely analysis of
audits and data, as well as follow-up to corrective actions? Or are there
delays in responding to findings and implementing corrective actions?
(3)
Are CASS personnel able to perform their duties in accordance with reasonable
schedules?
(4)
How many findings are produced by the CASS, and what are the trends?
NOTE:
CASS is supposed to produce findings, so
absolute numbers, even high numbers of findings, are not necessarily a negative
outcome; if combined with effective corrective actions and follow-up action,
numerous CASS findings could be a positive indicator that the CASS is doing its
job of detecting deficiencies and yielding appropriate, well-analyzed
corrective actions. Trends are important, however. The same types of findings
should not recur often once the CASS has addressed them.
(5)
Have an unusually large number of unplanned maintenance events occurred within
a specified time (for example, 21 days) after a substantial inspection or
maintenance task? If so, does an investigation indicate there are deficiencies
in the inspection and maintenance programs that should have been averted by the
CASS, or can the anomaly be attributed to other factors?
(6)
Does analysis indicate recurring problems in areas previously thought to have
been addressed by corrective actions?
(7)
Are new problem areas coming to light? (This would be indicative of the CASS
working to detect new issues.)
(8)
Are CASS corrective actions resulting in new problem areas, reflecting
insufficient risk or system analysis before the implementation of these
corrective actions?
(9)
How do CASS results compare with outside audit results, such as those conducted
by the ECAA?
(10)
Have regulatory violations occurred that the CASS might have averted?
(11)
Does operator senior management understand and support the CASS?
(12)
Are CASS auditors and analysts encouraged to consider all possible aspects of
an issue, including the role of senior management, when developing corrective
actions?
(13)
Has the CASS evolved into a punitive process with the result of discouraging
open participation of company personnel, or do personnel cooperate actively and
offer input to the CASS?
(14)
Are all areas of the inspection and maintenance programs undergoing CASS audits
in accordance with a schedule based on a process of risk assessment and
prioritization?
(15)
Do the depth and quality of the audit reports and analysis reflect that
personnel have sufficient time and resources?
c. Senior management review. Senior management should review
CASS issues on a monthly or bimonthly basis. Meetings of this sort, possibly of
CASS or maintenance management committees or boards, may be held to discuss
findings, analysis, and the progress of corrective actions. These meetings may
address statistical data and trends, depending on the operator's size and
operation and their ability to produce comprehensive statistical reports.
CHAPTER 10. THE ROLE OF THE ECAA IN RELATION TO
AN OPERATOR'S CASS
1000. The ECAA's general role.
As with any applicable aviation regulation,
an operator must understand that it holds the primary responsibility for
compliance, not the ECAA. The ECAA's role is not to design the CASS for each
operator, but to ensure the operator has satisfactory policies and procedures
in place. For example, the ECAA will not provide the industry with an
exhaustive list of data to be collected and analyzed because of the wide
variation in the nature and scope of their operations. However, the ECAA
expects each operator with a CASS to demonstrate that its CASS includes a
process for selecting and periodically reevaluating data sets appropriate for
its operation and for monitoring the inspection and maintenance programs. The
ECAA also expects each operator to have a logical and current reason for
selecting the data sets it collects.
1001. The ECAA principal inspector's role.
The term ECAA principal inspector, as used in
this AC, is generally intended to mean the principal maintenance inspector
(PMI). However, the principal avionics inspector (PAI) also plays an important
role in the oversight of the operator's CASS and shares many of the same
responsibilities as the principal maintenance inspector (PMI). The ECAA
principal inspector:
a.
Works with the operator in developing the CASS, in providing guidance, and in
ensuring the operator's CASS meets the intent of the regulation.
b. Reviews the operator's CASS records, such as results of
audits and analysis, corrective action, and follow-up. Therefore, it would be
useful for the operator and the principal inspector to have a common
understanding of how long the operator will retain these records, not only in
terms of usefulness to the CASS but also to help the inspector determine the operator
is properly executing its CASS.
c. Meets on a regular basis with managers in the maintenance,
inspection, and quality assurance areas, particularly with the person
responsible for the CASS. The operator's CASS should provide one of the best
barometers of the overall status of the inspection and maintenance programs,
how they are being executed, whether they are effective, and whether change is
being implemented as necessary.
d. Meets occasionally with senior management to determine how
well they understand and support the CASS.