INTERNAL
EVALUATION PROGRAM
This Egyptian Advisory Circular (EAC) provides
information and guidance material that must be used for the development of an
Internal Evaluation Program (IEP).
All certificate holders operating under Egyptian Civil Aviation
Regulations (ECARs) Part 101, 121, 137, 141, 142, 145 and 147 shall implement
this program within their organization.
The procedures and practices outlined in this EAC
are applicable to maintenance departments, flight operations departments,
training programs, and all other related aspects of a certificate holder’s
organization.
The internal evaluation program is a mandatory
program. As a matter of policy,
the ECAA has always encouraged certificate holders to identify, correct, and
disclose instances of noncompliance.
Therefore, the development and implementation of an IEP will benefit
both the certificate holder and the flying public.
EAC00.2.5 General
Definitions of key terms and a description of the
basic elements of an IEP are incorporated in this EAC and are consistent with
recognized quality auditing principles.
When appropriate, these terms have been tailored to conform to aviation
standards and practices.
The standards described herein are intended to help
certificate holders develop their own internal evaluation program.
In prescribing standards, rules, and regulations,
and in issuing certificates under the Egyptian Civil Aviation Regulations, the
ECAA considers the duty resting upon certificate holders to perform their
services with the highest possible level of safety. The validity of the air operator certificate or other
certificate depends upon the ability to conduct a safe operation in accordance
with prescribed rules, regulations, and standards.
Through surveillance and oversight, the ECAA
verifies that certificate holders are upholding their responsibilities. The IEP is intended to facilitate the
ECAA Inspector’s advisory and cooperative capacity by providing a procedure for
identifying and resolving safety related issues.
The following key terms and phrases are defined to
ensure a standard interpretation and understanding of an IEP.
Note: An Airworthiness Directive (AD) compliance
system, applicable to certificate holders that own, operate, or maintain
aircraft/equipment, has been used as an example to further clarify each
definition.
(a)
Evidence: Evidence, as it relates to this EAC is
documented statement of fact, prepared by a certificate holder, that may be
quantitative or qualitative and is based on verifiable observations,
measurements, or tests. Evidence
must generally be in the form of written documentation or reports that support
the analysis and review.
This data is
necessary to substantiate findings/concerns and enables management or
evaluators to determine the root cause(s) of reported findings. Objective evidence generally is
determined from a review of appropriate documents or manuals, equipment
examination, observed activities, and data obtained during interviews.
(b) Controls: The key
procedures, responsibilities, and decision-making positions within an
organization, department, division, or functional area.
A certificate holder who owns, operates, or
maintains aircraft/equipment is tasked with the determination of AD
applicability. This responsibility
is considered a control of the AD compliance system and its design is critical
to the overall effectiveness of the system.
As part of an
internal evaluation, the controls of the evaluated area must be verified and
tested. In some instances,
personnel performing the internal evaluation may have to first determine the
features of a control. For
example, evaluators must first understand how the certificate holder verifies
AD applicability before proceeding with an evaluation of AD completion and
associated records. In particular,
the evaluation would focus on the effectiveness of procedures that would
minimize the risk of simple human error or oversight.
(c) Finding: A conclusion that
confirms noncompliance with a specific standard.
Following a review of status records
applicable to powerplant AD completion, company evaluators concluded that two
applicable ADs lacked appropriate information substantiating compliance. Evidence to support this conclusion included
copies of the actual ADs and referenced service bulletins.
Note: This finding demonstrates noncompliance
with an ECAR.
A Certificate holder has a procedure that
requires AD applicability determination be reviewed and signed by quality
assurance, engineering, and the Vice President of Maintenance. An internal evaluation of the AD system
discovers that for five newly applicable ADs, there is neither a record of the
review nor a signature by the Vice President of Maintenance.
Note: This finding demonstrates noncompliance
with a standard, i.e. an approved company procedure, rather than a specific
ECAR.
(d) Concern: A conclusion,
supported by objective evidence, which is not a finding. However, it is a
condition that may become a finding.
Through the use of its IEP, a certificate
holder found that it was not scheduling aircraft for AD completion until the AD
was within 10 aircraft cycles of expiration. While this procedure had not resulted in any findings, a
review of scheduling logs indicated that several aircraft had been flown to
within one cycle before performing the required AD work. Additionally, maintenance planners
often had to "frantically" reshuffle aircraft schedules to ensure
timely AD completion. The
evaluators believed these circumstances had the potential to become a finding
in the future. Their analysis was
documented as a concern and forwarded to management.
EAC00.2.9 Inspection:
The act of observing a particular event or
action to ensure that correct procedures, established standards, and
appropriate requirements are followed.
Note: The term
inspection is defined in this EAC within the context of quality auditing
principles. It does not address or
define ECAA inspections.
(a)
Audit: A methodical, planned review used to determine how effectively business
is being conducted. It compares
present procedures with established company policies and procedures. The various elements that comprise an
effective audit are:
(1) Personnel interviews;
(2) Review of appropriate documents;
(3) Operational observations;
(4) Selection of various samples;
(5) Activity inspections; and
(6) Documentation of results.
An audit builds on the principles of
inspection. The results of an
inspection assist in the audit objective of determining whether business is
being conducted in accordance with established policies and procedures. During an audit, qualified personnel
look for the existence of a systemic problem but do not attempt to estimate its
extent. The results (findings or
concerns) of an audit must be documented and presented to management for
corrective action decisions.
(b)
Evaluation: An independent review of company policies, procedures, and
systems. An evaluation must be a
comprehensive and continual process that considers the:
(1) Results of audits;
(2) Overall effectiveness of the management organization in achieving
stated objectives; and
(3) Ability of management to respond to new technologies, market
strategies, and social or environmental conditions.
The evaluation
process builds on the concepts of audit and inspection. An evaluation is an anticipatory
process and is designed to identify and correct potential findings before they
occur. Written conclusions and
recommendations made as a result of an evaluation must be submitted to Senior
Management for appropriate action.
Senior Management:
Senior
Management, as it relates to this EAC, is the most senior company individual
that is responsible for the daily operation of the company. This person may be title as Chairman,
Chief Executive Officer, President or equivalent, who has the authority to
resolve issues and take appropriate action. The ECAA believes that Senior Management must be aware of
the plans, results (findings and concerns), and follow-up actions associated
with an IEP. Senior Management may appoint a Senior Management representative
and he must be given the responsibility of ensuring that the IEP is properly
developed, implemented, and maintained.
This management representative must be above the level that directly
supervises work accomplishment/procedural development and must have direct
contact with Senior Management.
This representative does not relieve Senior Management of their
responsibilities under this program.
The internal evaluation program is based on the
premise that certificate holders are primarily responsible for continuously
monitoring and ensuring that their operations are safe and in compliance with
the ECARs. The ECAA encourages
certificate holders to establish and conduct internal evaluations that embrace
the following processes:
(1) A Continual
Process incorporating the techniques of inspections, audits, and
evaluations to assess the adequacy of managerial controls in key programs and
systems. When deficiencies are
identified, corrective action plans and follow-up evaluations are developed and
implemented;
(2)A Review Process, extending beyond regulatory compliance, to
determine the causes of deficiencies and detect needed enhancements to company
operating practices before deficiencies occur; and
(3)
An Independent Process that has straight-line reporting
responsibility to senior management within the organization structure.
The internal evaluation concept stresses
self-audit responsibilities of individual employees. This concept also stresses the evaluation responsibility of
senior management to ensure that company policies and procedures provide for
enhanced safety and compliance within the organization.
Certificate holders interested in developing
an internal evaluation program, are encouraged to include the following
essential elements in their program:
(a)
Independent/defined responsibilities;
(b) Management
reviews;
(c) A process that
is continual with prescribed schedules;
(d) Corrective
action plans; and
(e) Records.
Note: Certificate
holders developing an IEP must prepare a plan defining the program’s procedures
and functional responsibilities. The format for a typical program plan is
provided in the Appendix of this EAC.
EAC00.2.15 Independent/defined responsibility
An IEP must identify the person and/or group
within the organization who has the responsibility and authority to:
(a) Perform evaluations, audits, and inspections;
(b) Identify, record, and provide evidence of findings or concerns;
(c) Initiate, recommend, or provide solutions to findings/concerns
through designated reporting channels;
(d) Verify the implementation of solutions within a specified time; and
(e) Communicate and coordinate activities with ECAA personnel on a
regular basis.
A
certificate holder must identify resources and personnel dedicated to the IEP
and describe their organizational independence within the company. Individuals conducting Internal
Evaluations must not be responsible for managing work in the areas being
evaluated or the tasks being reviewed.
Note: This concept may
have to be modified for small operations.
The
operational size of many certificate holders may justify the costs associated
with
having
full-time, dedicated resources and personnel in a separate IEP department or
group.
For
very small operations, an appropriate IEP might consist of developing
checklists and a schedule (monthly, quarterly, semiannual, or annual) for
accomplishing evaluations. Even in
such cases, the review must include a written statement acknowledging the
completion of the checklist items and the signature of Senior Management.
Note: Occasional independent oversight
of checklist item development and accomplishment must be considered.
Certificate holders, using outside resources or independent consultants in
support of an IEP, must ensure that these resources are coordinated through a
chain of command that reflects independence and contact with Senior Management.
Senior Management must review internal
evaluation results to verify that satisfactory
corrective actions have been
implemented.
The review of internal evaluation
information by Senior Management must be
documented in reports or other appropriate
records generated by the IEP. The
certificate holder must decide the
frequency, format, and structure for informing Senior
Management
of internal evaluation plans, results, and follow-up actions. The
program must include a diagram that depicts
the independence of personnel who
perform
or supervise internal evaluation functions, including some form of straight
line
reporting authority to Senior Management.
This reporting structure be documented
and
included as part of the program plan.
In order to effectively anticipate potential problem areas and initiate
corrections before
actual findings occur, an IEP must be a
continual program, not merely spot check
inspections of operating practices. Stand alone, spot check inspections,
will do little
more than identify symptoms of potential
problems.
A continual process monitors and verifies
whether findings are isolated instances or
symptoms of policy, procedural, or
managerial problems. Scheduled,
follow-up,
and special evaluations must be performed
whenever negative trends are identified.
The continual process is a structured
activity and it is essential to include a schedule of activities. This planned schedule will serve to
verify that the internal evaluation process is:
(a) Complete and
detailed;
(b) Directed;
(c) Credible; and
(d) Recognized by
Senior Management.
The schedule of activities must include a
periodic review cycle for specific areas covered by the certificate holder’s
IEP. However, the scheduling
process must also be dynamic and allow for special evaluations when trends are
identified. In addition, follow-up
evaluations must be performed, as necessary, to verify that corrective action
commitments were met and that they were effective in eliminating any reported
findings or concerns. Planned,
Special, and Follow-Up evaluations all comprise an effective
internal evaluation schedule.
(1) Planned evaluation: Establishes a schedule of events that
will be performed during a prescribed calendar period and divides the complete
schedule into phases. This
evaluation is scheduled to provide ample flexibility for resources to initiate
special or follow-up evaluations.
(2) Special evaluation: Conducted whenever Senior Management identifies
special concerns or priorities.
May also be initiated based on a review of industry trends, ECAA
concerns, or identified internal trends.
(3) Follow-Up Evaluation: Ensures
corrective actions were completed and the reported finding or concern has been
eliminated. May also be
accomplished in response to ECAA surveillance findings.
EAC00.2.23 Corrective Action Plans
An IEP must include procedures ensuring that
corrective action plans are developed in response to findings or concerns. Additional procedures must be developed
to ensure that these action plans were implemented and effectively completed in
a timely fashion.
As an option, Internal Evaluation personnel
may participate in the development of corrective action plans. However, organizational responsibility
and accountability for the development of these plans must reside with the
technical departments cited in the finding or concern. A proper corrective action plan must
include the following elements:
(a) Identification of the finding or concern;
(b) Analysis of objective evidence to determine the systemic or root
cause(s) of the finding or concern;
(c) Identification of corrective steps that will prevent a recurrence of
the finding or concern;
(d) Implementation schedule, including a time frame for completion of
identified corrective steps; and
(e) Individuals or departments responsible for implementing the
corrective steps.
EAC00.2.24 Individuals Responsibilities
The individuals responsible for managing an
IEP must facilitate the corrective action process by performing the following:
(a)Ensuring corrective action plans are developed in response to
findings or concerns and verifying that they include the elements outlined
above;
(b)Monitoring implementation/completion of corrective action plans and
providing senior management with an independent assessment; and
(c)Initiating scheduled and/or unannounced follow-up evaluations to
ensure the effectiveness of the corrective action plans.
The certificate holder must maintain records
documenting IEP performance and results.
Records are considered the principle form of evidence. Documented evidence is essential in
analyzing and determining the root causes of findings or concerns in order to
identify potential areas of noncompliance. It is important to maintain accurate, complete, and reliable
records that document the activities and results of an internal
evaluation. IEP files must include
the following data:
(a) Scheduled evaluation reports;
(b) Special evaluation reports, including the trends or other reasons
associated with scheduling a special evaluation;
(c) Follow-up evaluation reports;
(d) Responses to findings or concerns; and
(e) Corrective action plans.
IEP procedures and responsibilities must be
documented in a formal program plan.
Following are suggestions for the plan’s preparation and structure.
(a)Preparing a program plan
(1) The program plan must
describe the duties, responsibilities, procedures, and organization of the IEP.
(2) Copies of the plan must be distributed to appropriate company
personnel to enhance their awareness and familiarity with IEP procedures. In addition, revisions must be made, as
necessary, to ensure that the program plan continues to reflect the certificate
holder’s current internal evaluation procedures and organization.
(b)Structuring a program plan
A sample
outline of a typical program plan using the elements in this EAC is
provided in the
Appendix. It must be viewed as a
checklist of items
that warrant
consideration during development of an IEP. The number of items addressed and the manner in which they
are documented will ultimately depend on the complexity of each operation.
The certificate holder shall submit their program
to the ECAA for review and acceptance.
A prepared program will provide the ECAA with an opportunity to review
the proposed duties, responsibilities, procedures, and organization of the
certificate holder’s IEP. Additionally, the ECAA will provide support to any
certificate holder requesting assistance with program development and
implementation.
Development of internal evaluation programs must
ensure that company policies and procedures are responsive to growth/change and
that certificate holders continually comply with appropriate safety
requirements.
Terms that will be used consistently in the IEP
must be defined. For example, a
certificate holder must define how results are categorized, i.e., a finding or
concern. These categories and
other applicable terms must be defined and documented for clear understanding
and interpretation by company personnel.
Definitions must be similar to those specified in this EAC.
The certificate holder must specify which
individuals are responsible for performing the following tasks:
(a) Internal
evaluation supervision;
(b) Performance of
evaluations, audits, and inspections;
(c) Identification and
documentation of findings or concerns;
(d) Collection of
objective evidence necessary to substantiate findings or concerns;
(e) Initiating and/or
providing solutions to findings or concerns through designated reporting
channels;
(f) Monitoring the
development and implementation of corrective action plans;
(g) Maintaining and
updating Internal Evaluation files;
(h) Verification of
solution implementation; and
(i) Communication and
coordination of IEP activities with ECAA personnel on a regular basis.
Personnel performing the above tasks must
not be responsible for work accomplishment or management in the areas being
evaluated. The supervisor of the
Internal Evaluation function must either be Senior Management or their
representative with straight-line reporting authority to Senior Management.
When full-time dedicated resources and personnel are not practical, procedures
must indicate that persons having direct responsibility for the areas to be
evaluated are not to be involved in the selection or supervision of the
Internal Evaluation team. In
addition, identified personnel must be exempt from their other duties and
completely dedicated to the IEP while they participate as an evaluator.
An organization chart must be prepared that clearly
illustrates the position of the IEP within the certificate holder’s management
structure. The chart must reflect the program’s independence within the
corporate structure and straight-line reporting to Senior Management.
Reporting procedures must include requirements for
Senior Management’s review of internal evaluation information. They must be routinely apprised of
schedules, plans, results, and follow-up corrective actions. Additionally, this section of the plan
must specify the frequency, format, and structure for reporting information to
Senior Management and schedules to the ECAA.
A certificate holder must specify all areas within
the scope of the IEP review. The
ECAA believes that the most effective IEP will encompass a complete review of
the certificate holder’s entire operation. However, a certificate holder has the option of limiting an
internal evaluation to selected systems or functional areas.
The scheduling
process must be comprised of the following elements:
(a)
Scheduling evaluations over a predetermined calendar period;
(b)
Performing special evaluations when trends are identified or priorities are set
by Senior Management; and
(c)
Completion of follow-up evaluations that verify the effectiveness of corrective
actions.
Procedures and specific individuals
responsible for planning, developing, and coordinating the internal evaluation
schedule must also be defined.
The IEP must have a defined record-keeping
process. Procedures must specify
how records are filed and maintained.
Standard forms or formats for filing reports also must be specified. IEP records must be comprised of the
following:
(a) Scheduled evaluation reports;
(b) Special evaluation reports;
(c) Follow-up evaluation reports;
(d) Responses to findings or concerns;
(e) Corrective action plans; and
(f) Reports concerning completed corrective
actions.
The certificate holder must provide evaluators with
training in recognized quality auditing and evaluation
principles/techniques. This
training could be one or a combination of the following:
(a) In-house courses;
(b) College courses;
(c) Home study materials; or
(d) Industry sponsored seminars and
workshops.