EAC0014

Normal operation safety survey

NOSS

EAC 0014

Revision Page

Item

Rev. No.

Rev. Date

Date of Updating

Signature

EAC 0014

List of Effective Pages

Item	Title	Pages	Dated
EAC 0014

EAC 0014

TABLE of CONTENTS

ITEM	TITLE
1	NOSS – What is it?
2.	Purpose:
3.	Applicability
4.	NOSS Operating Characteristics
5.	WHEN TO CONDUCT A NOSS
6.	HOW TO IMPLEMENT A NOSS
7.	HOW TO USE NOSS DATA
APPENDIX A	THREAT AND ERROR MANAGEMENT
APPENDIX B	TRAINING AND STANDARDIZING NOSS OBSERVERS

1. NOSS – What is it?

• Safety Management Tool based on threat -error managementTEM

• Captures operational context data during everyday, routine operations

• Provides a TEM profile of the organisation

2. Purpose:

A NOSS application will provide a data collection and analysis tool, supplying the safety management system with reliable and valid information on the occurrence of threats, errors and undesired states and their management in normal everyday operations

NOSS differs from other tools of this type in that it is not merely another error capture tool, rather it aims to complement these by capturing information on the internal and external threats to the safety and integrity of the system, including how these are managed by the system processes/procedures and practices; this can include good as well as poor performance

A NOSS application might ultimately provide a status report that can be used as a diagnostic tool, measuring system performance against established standards and thereby providing guidance in the continuous effort to maintain the desired level of safety in the ATM system.

3. Applicability

Monitoring of air traffic services (ATS) operations. The process involves collecting safety-related data from normal operations to identify safety threats that might not otherwise come to the attention of safety managers.NOSS would be based on a human factors tool known as the threat and error management (TEM) model.

NOSS would entail over-the-shoulder observations during normal shifts and would not be allowed in any training situations.

4. NOSS Operating Characteristics

a) Over-the-shoulder observations during normal shifts (i.e. not in OJT situations)

NOSS observations are limited to regularly scheduled operations. Shift checks, initial shift indoctrination or other ojt training are off-limits due to the extra level of stress put on pilots during this type of situation. Having another observer on duty adds to an already high stress level, thus providing an unrealistic picture of performance. In order for the data to be representative of normal operations, NOSS observations must be collected on regular and routine operational shifts.

b) Joint management/controller association sponsorship

In order for NOSS to succeed as a viable safety project, there needs to be support not only from the management side but also from the controllers. The joint sponsorship provides a “check and balance” for the project to ensure that change, as necessary, will be made as a result of NOSS data. When considering whether to conduct a NOSS, the first question to be asked by ATC unit management is whether the controller group representatives endorse the project. If the answer is“No”, the project must not be initiated until endorsement is obtained.

c) Voluntary participation

· Maintaining the integrity of NOSS within an ATS units and the aviation industry as whole is extremely important for long-term success. To accomplish this goal, all NOSS observations are collected with voluntary controller's participation. Before conducting NOSS observations, observers must first ask the ATC operational shift for permission to be observed. If the crew declines, the observer takes another operational shift with no questions asked. If an ATS unit conducting a NOSS has an unreasonably high number of declines, this should serve as an indicator that there are critical “trust” issues to be resolved.

d) De-identified, confidential, and non-disciplinary data collection

· NOSS observers are required not to record names, dates, or any other information that can identify an operational shift or individual. The purpose of NOSS is to collect safety data, not to punish controllers. ATC Units cannot allow themselves to squander a unique opportunity to gain insight into their operations by having controllers fearful that a NOSS observation could be used against them for disciplinary reasons. If a NOSS observation is ever used for disciplinary reasons, the credibility of the entire safety program may be irreparably compromised.

e) Targeted observation instrument

· The NOSS observation form is predicated on the TEM framework. At ATC operational shift’s own initiative (and risk), other conceptual frameworks can be used for NOSS data collection. Whatever framework is used, it must generate meaningful data on a variety of topics, including what the controllers did well, what they did poorly, and how they managed each task of ATC operation.

· A narrative written by the observer should have sufficient detail to allow others to understand the ATC operational task and all its events. The observers need to describe the environmental conditions and events surrounding the controller’ behavior so that the controllers’ performance can be understood in full context.

f) Trained and standardised observers

· Primarily, controllers conduct NOSS. Observation teams will typically include line controllers, instructor controllers, safety controllers, management controllers s that are respected and trusted within the ATS UNITS to ensure line acceptance of NOSS. After observers are selected, they are trained and calibrated in the NOSS methodology, including the use of the NOSS observation form. Observers’ training in the concepts and methodology of NOSS will ensure that the observations will be conducted in the most standardized manner. (See Appendix B.)

g) Trusted data collection sites

· In order to maintain confidentiality, ATS units must have a trusted data repository. The goal is that no individual observations will be misplaced or improperly disseminated through the operational unit.

h) Data cleaning process

· Data-driven programs like NOSS require quality data management procedures and consistency checks. For NOSS, these checks are done at data-verification roundtables. A roundtable consists of three or four department who review all the raw data for possible inaccuracies.

· The end product is a database that is validated for consistency and accuracy according to the ATS unit’s standards and manuals, before any statistical analysis is performed.Targets for safety enhancement

i) Targets for safety enhancement

· The final product of a NOSS is the data-derived targets for enhancement based on emergent patterns in the data. It is then up to the ATS service provider to develop an action plan based on these targets, using experts from within the airline to analyze the targets and implement appropriate change strategies.

j) 10. Feedback results to the controllers

· In order to ensure long-term success of NOSS, ATS service provider must communicate the results back to the line controllers. Line Controllers will want to see not only the results of the audit, but also management’s plan for improvement.

5. WHEN TO CONDUCT A NOSS

There are several factors to consider when scheduling a NOSS.Given all the personnel involved, a NOSS should be scheduled to fit with other operational priorities. For example,

Is there a particular time in the year when more observers will be available?
Is there a better time for the Scheduling department to roster these people?
Also, is there a particular time that is more interesting from a safety or operational perspective? Some examples: bad-weather season, peak traffic season, after the introduction of an operational change such as new aircraft, altered routes, or a merger.

A NOSS must not be implemented immediately after a major incident or accident. The
ATC unites will be in a heightened state of awareness at this time, and ATCo will be overly sensitive to observation; hence, the chances of getting normal data will be diminished.
A minimum, service provider should wait at least one year after a major safety event before scheduling a NOSS.
Once a service provider has done a NOSS, a critical question is when to do the next one.
NOSS data provide a baseline against which to measure improvements. A realistic time frame to review NOSS results, develop and implement action plans, and monitor results is three years. Hence, to measure the effectiveness of organizational changes, a repeat NOSS every three years is recommended.

6. HOW TO IMPLEMENT A NOSS

There are steps associated with getting good-quality data from observers (Data Collection), and steps associated with ensuring that accurate and meaningful data are given to management and line pilots (Data Analysis & Feedback). An ATS service provider can conduct its own NOSS by observing the following steps.

6.1 DATA COLLECTION

6.1.1 Form a NOSS steering committee and appoint a NOSS coordinator (safety manager)

6.1.2 Gather information and NOSS resources from other ATS units

Before conducting a NOSS for the first time, the committee and coordinator should seek out information from other ATS units that have already conducted a NOSS. Other ATS units may be able to share observer selection and training techniques, observation forms, scheduling tips, and other logistical aids.

6.1.3. Publicize NOSS within the ATS units and send information letter to the line managers

A first task is advance publicity via company publications to build line managers awareness and acceptance of the upcoming NOSS.
Next, the coordinator organizes and distributes information to all ATCo explaining the purpose of the NOSS.
This information specifies the purpose of the audit, the fact that all observations are of a non-jeopardy nature, and that all data will be kept strictly confidential. The information is signed by the highest level of management within AT operations
The information of announcement should precede the line audit by at least one month, with a follow-up alert one week before starting observations.
NOSS observers should have copies of the signed information to show shift members in case questions arise.

6.1.4 Decide the focus of the NOSS

The NOSS steering committee decides the focus of NOSS. One option is to sample broadly across the entire operation—this would be an effective strategy for a first NOSS. Alternately, the NOSS steering committee can focus on problems that have been identified by other data sources, this approach would schedule NOSS observations on particular units, in certain locations, or into particular airports that have been identified as problematic.
The committee can also focus a NOSS on new or other recent organizational changes.

6.1.5 Decide the number of observations

Most ATS service providers will find it cost effective to conduct a NOSS on a sample of their operation—the question is how big a sample?
As a general guideline for a full NOSS, match the number of observations per ATC unite to the relative number of operational task per shift.

6.1.6 Create an observation form

The observation form should be based on a conceptual framework that captures multiple aspects of normal operations, including the operating environment and ATCo performance. It should provide categories and codes to streamline observations and save the observer’s time, but it should also require a written description of the ATC unit operations that captures the full context.

6.1.7 Select observers

The observer team should have representatives from operational units, training, and safety.
The number of observers needed depends on the size of the audit and the observers’ workload. There is substantial work involved in completing an observation form and providing a detail-rich narrative for each ATC unit.

6.1.8 Train observers

NOSS observers must be educated about the purpose and rationale of NOSS, and trained in the use of the observation form.
Observers should practice with scripted scenarios or videos until they are confident they can use the observation form correctly.
Appendix b provides more detail on the objectives and content of observer training. Appendix b also addresses the standardization of observers and NOSS data.

6.1.9 Schedule observations

Plan no more than two observations per observer per day to allow sufficient time to complete the observation form and write a rich narrative. Schedule observers across ATS operational units regardless of their type rating to encourage a more general, cross ATS operational units perspective of operational controllers performance. Build some flexibility into the schedule to allow for the unexpected. Finally, do not let the observations continue indefinitely—schedule all observations within a 1-3 month period if possible, else the impact of NOSS will be lost.

6.1.10 Decide on a data repository

The NOSS coordinator organizes a secure site for the data and oversees the receipt of the observation forms. The coordinator must be able to protect the identity of the observers and the observed to ensure complete confidentiality and non-jeopardy conditions. Under no circumstances should it be possible to connect individuals with particular observations.
The observations can be kept in-house if data management and analysis expertise is available, and if data security can be assured.
The number of observers needed depends on the size of the audit and the observers’ workload. There is substantial work involved in completing an observation form and providing a detail-rich narrative for each ATS operational unit.

6.1.11 Provide logistical support

Give the observers the name of a contact person, most likely the NOSS coordinator, who can be reached if there are any problems with scheduling or data collection.

6.1.12 DATA ANALYSIS & FEEDBACK

6.1.12.1 Verify the data

Convene a meeting of “local experts”—operational personnel familiar with the operation of each ATS unit (possibly fleet managers or member of the steering committee, but not any of the observers). The group’s task is to review and verify the observations against current manuals, policies, and procedures. For example, an observer might log a procedural error for failure to make an approach callout when in fact there is no written procedure in the ATS unit's operations manual.
The data verification group would delete this particular ‘error’ from the database.
This step is a data integrity check in that it ensures that events are correctly recorded in line with each ATS operational unit's procedures and policies. It also builds ownership in the results and dispels any later criticism that the coding was not an accurate representation of the ATS unit's operations.

6.1 .12.2 Analyze data

NOSS data reveal strengths and vulnerabilities in an ATS unit's operations. The data analyst should investigate the prevalence and management of different events and errors.
Although certain types of comparisons will seem obvious, many analyses can and should be based upon hunches and theories derived from local knowledge of operations.
If the analyst knows how operations are managed, comparisons that reflect this structure can be made.
If the analyst knows the kinds of information that might be useful to training, safety, or to domestic or international operations, results can be tailored to these particular aspects of the operation.
Feedback from various stakeholders is critical during this stage of preparing the report.
The analyst should not hesitate to distribute early drafts to key people within the ATS UNIT familiar with NOSS to cross-verify the results. This helps validate derived trends.
Patterns will emerge as the data are analyzed. Certain errors occur more frequently than others, certain airports or events emerge as more problematic than others, certain SOPs are routinely ignored or modified, and certain maneuvers pose greater difficulty for adherence than others. These events and practices form the basis of suggested targets for enhancement.

6.1 .12.3 Prepare report

The last stage of a NOSS is a written report that presents the overall findings of the audit. With a large database like the one generated from a NOSS, it is easy to fall into the trap of trying to present everything. The report must be concise and present only the most significant trends from the data.
Along with the results, the report should provide an initial list of targets for enhancement. Targets need to be action-focused and data-driven.

6.1 .12.4 Brief management

The NOSS report should be presented to management in operations, training, standards, safety, and possibly other departments depending on the results.
Once the various departments are briefed on the report, they will likely want to investigate the data more deeply themselves. The data should be available in aggregated form for them to review.

6.1 .12.5 Brief line pilots

Unit managers should also be informed of the significant results in the NOSS report. To sustain the controllers’ interest in the NOSS project, make an announcement at the end of the data collection phase that the NOSS observations have been completed, stating how many and on what units, and advise when the unit managers can expect to see the results.
When the report is ready, the highlights should be presented to the unit managers, either as one NOSS debriefing event or spread over time in the safety periodical issues. Unit managers will want to know what changes will be undertaken as a result of the NOSS.

6.1.13 Monitor safety change process

6.1.13 .1 historically, organizational safety changes within ATS units have been driven by accident/incident investigation and intuition. Today, ATS units must deal proactively with accident and incident precursors. To be successful, the safety change process must be data-driven. Measurement of daily operations is fundamental.

unless an organization uses systematic measurement, the perspective it has on the strengths and weaknesses of its operations is largely based on anecdote and opinion.

6.1.13 .2 A NOSS provides specific and quantified results. To take full advantage of this specificity, the targets for enhancement that arise from the data analysis should go through a formal safety change process to produce improvement. A formal safety change process provides a principled approach to target limited resources and helps the ATS units avoid “turf” issues, by clearly defining and prioritizing the issues that impact ATS operations. The basic steps of a safety change process are:

�� Measurement (with NOSS) to obtain the targets;

�� Detailed analysis of targeted issues;

�� List of potential changes for improvement;

�� Risk analysis and prioritization of changes;

�� Selection and funding of changes;

�� Implementation of changes;

�� Time for changes to stabilize;

�� Re-measurement.

7. HOW TO USE NOSS DATA

A well-conducted and well-analyzed NOSS identifies strengths and vulnerabilities in an ATS’s operations. It provides this information in a quantifiable form against which targets can be specified and improvements can be measured.
Depending on the sophistication of an ATS Unit's safety management system, and the extent
to which different safety programs within the ATS UNIT are premised on the same conceptual
Framework, data from a NOSS can be cross-referenced with data from the other safety programms.
Each data source provides unique yet complementary evidence of the airline’s safety status.
NOSS presents a broad view of operations; a repeat NOSS can maintain that broad focus. For example, did the changes that were introduced after the first NOSS improve results in one area, only to cause problems in another? Checklist adherence may have improved, but did error detection—the super ordinate goal of improving checklist adherence—actually improve or is the new adherence simply cosmetic?

APPENDIX A: THREAT AND ERROR MANAGEMENT

1. Introduction

The Threat and Error Management (TEM) model is a conceptual framework for understanding operational performance in complex environments. Originally created to capture the ATCo’s task during operational shifts, the model is generic and can be applied to numerous work situations.

The added value that TEM brings to other performance models is that it focuses simultaneously on the operating environment and the humans working in that environment. Because the model captures ongoing performance in its “natural” or normal operating context, the resulting description is realistic, dynamic, and holistic. Because the model can also quantify the specifics of the environment and the effectiveness of performance in that environment, it is also highly diagnostic.

There are several ways of using the TEM model, from focusing on a single event (as is the case with accident/incident analysis) to understanding systemic patterns in a large set of events (as with NOSS). As a training tool, TEM can help individuals clarify their performance needs and vulnerabilities, and as part of a safety management system, TEM can help an organization measure and improve the effectiveness of its organizational defenses and safeguards.

2. Working Definitions

The Model

This section defines and provides examples of the various components of the Threat and

Error Management (TEM) model.

Threats

A threat is defined as external factors that increase the operational complexity and that need to be managed during a normal shift

There are threats from the environment—adverse weather, airport conditions, terrain, traffic, interruptions, or errors maintenance. Threats may be anticipated by the ATCo, , or they may be unexpected, occurring suddenly and without warning such as system malfunctions. Some threats are easily resolved and quickly dismissed from the controller’s workload, while other threats equire greater attention and management. A mismanaged threat is defined as a threat that is linked to or induces ATCo error.

Errors

ATC errors are defined as actions or inactions by controllers that lead to deviations from organisational or controller intentions or expectations. . Errors in the operational context tend to reduce the margin of safety and increase the probability of adverse events.there is:

• Proficiency - lack of knowledge or skills

• Inability to select appropriate functions in ATM system

• Operational decision - decision that unnecessarily increases risk

• Accepting traffic without separation assurance

• Communication – failure to communicate, miscommunication or misinterpretation of pertinent information

• Misunderstanding a coordination (adjacent sector)

• Procedural – intention is correct, execution is flawed

• Applying less than specified Wake Turbulence separation

• Intentional non-compliance - deliberate deviation from regulations/SOPs

• Using non-standard altitudes/flight levels

Understanding how the error was managed is as important, if not more important, than

understanding the prevalence of different types of error. It is of interest then if and when

Undesired States

An undesired state is defined as operational conditions where errors by controllers put traffic in a situation of unnecessary risk

It is a safety-compromising state that results from ineffective error management.

3. Threat and Error Countermeasures

3.1 A description of an ATC shift is not complete without noting what the ATCo was doing to anticipate threats and avoid errors, as well as managing those that occurred. The following ATCo behaviors are considered threat and error countermeasures:

�� Planning countermeasures—planning, preparation, briefings, contingency management—is essential for managing anticipated and unexpected threats

�� Execution countermeasures—monitor/cross-check, display management, workload and automation management—are essential for error detection and error response

�� Review/Modify countermeasures—evaluation of plans, inquiry—are essential for

managing the changing conditions of a traffic situation

3.2 In addition to ATCo behaviors, TEM countermeasures also include equipment and

procedural countermeasures. Warning systems such as STCA can be considered threat countermeasures, just as checklists and well-written procedures provide the means for error avoidance and error detection.

3.3 In sum, the TEM model captures the dynamic activity that is ATC planning and executing a traffic control in real time and under real conditions. The utility of the model is that it can be applied proactively or reactively, at the individual, organizational, and/or systemic levels.

4. Practical Applications of the TEM model

4.1 TEM as a training tool

It is important to clarify that TEM is not TRM and should not be considered a replacement for it. TEM and TRM refer to overlapping but not equivalent activities.
TRM refers specifically to activities conducted by the ATCo to optimize performance.
These activities include threat and error countermeasures such as briefing, contingency

planning, and monitor/cross-checking, but they also include higher-order concepts such

as leadership and establishing open communication during the shift. Similarly, TEM

includes controllers countermeasures, but it also encompasses equipment, procedural and

regulatory countermeasures.

As a training tool, TEM can help individuals clarify their performance needs and vulnerabilities from a different perspective. Hence, threat and error management concepts could be introduced and explored as one component of TRM training.

4.2 TEM as a reporting tool for incidents

Reporting forms structured to the TEM framework instruct the ATCo to describe the event at the level of threats and errors. The TEM format prompts ATCo to report information about the threats that were present, the errors they may have made, how well the event was managed, and how the event may have been avoided or handled better. Preliminary work has shown that even ATCo who have not had training in the TEM model are able to complete the reporting form, a fact that speaks to the intuitive nature of the TEM framework.
With NOSS, the benefit is that observers may detect threats and errors that the ATCo themselves do not detect.

4.3 TEM as a systematic observation tool

ICAO has instituted a formal group of ATC subject matter experts from across the world to develop the Normal Operations Safety Survey (NOSS), a formal protocol to observe normal operations in ATC, based on the TEM model and NOSS methodology.

4.4 TEM as a reactive analysis tool for accidents and incidents

TEM can be used as an analysis tool to understand rare events, such as accidents and

serious incidents based on its ease of use and utility of the extracted data.

4.5 TEM as a proactive analysis tool

When TEM is used as the framework for safety data collection, a wealth of information

can be extracted. An ATS service provider can use the data to understand patterns at the organizational level. The data can also be collected across the industry and analyzed for systemic trends. An analysis based on TEM can:

Quantify those aspects of the working environment that can pose a problem for

the efficiency or safety of the operation (threat prevalence);

Quantify the management of those threats as either effective or ineffective (threat

management);

Recognize high rates of threat prevalence and mismanagement as systemic vulnerabilities;
Codify and quantify the errors that ATCo commit (error prevalence);
Codify and quantify the error management process from diagnosis to response

and outcome (error management);

Recognize high rates of error prevalence and error mismanagement as systemic flaws in procedures, policies, training, workstation design, and or inter-agency coordination; and
Locate strengths as well as vulnerabilities in organizational safeguards.

APPENDIX B: TRAINING AND STANDARDIZING NOSS OBSERVERS

This appendix details the training and standardizing of NOSS observers. Some of this material appears throughout the advisory circular; this appendix draws that information

together and provides more in-depth information.

OBSERVER TRAINING

Observer training typically requires two to three days of classroom training, with a

follow-up session after one or two line observations. There are five topics that need to be

covered in Observer Training:

1. NOSS rationale and etiquette

2. Company policies and procedures

3. Observation form

4. Threat and Error Management concepts

5. Narratives

1. NOSS rationale and etiquette

1.1 Observers will likely have a rudimentary understanding of NOSS when selected for the project; however, they will need to:

· Fully understand the safety rationale for conducting a NOSS at their ATS operation unit. A “big picture” perspective will help observers understand the “why” of NOSS and will underscore the importance of their role in the NOSS process.

· Also, the observers will be ambassadors for NOSS while observing on the line shift and it is important that they be able to explain the process fully, to answer any questions that the line pilots may have, and allay any fears or concerns.

understand the safety rationale for normal operations monitoring—a discussion of

proactive vs. reactive safety strategies is recommended.

· know how the data collected from the NOSS will be used to

· Understand strengths and weaknesses in the operations. An overview of the whole process from observations to data cleaning and analysis, to the diagnostic report and the development of targets for enhancement is recommended.

· Know the “how” of NOSS, specifically the etiquette associated with being a NOSS observer.

· Learn how to approach a operational controllers, how to ask permission to observe a traffic control, how to walk away so that the controllers can discuss it, and to accept without question any operational shift’s decision to deny access to the control room.

· The observer should also carry a copy of the letter of endorsement jointly signed by management.

· Be unobtrusive, yet responsive to any queries the controllers may have.

1.2 The NOSS observers should be trained to accept their role as observers, not evaluators—they

are not Check controllers.

NOSS observers will observe errors and undesired states as part of their observations; however, they should only interrupt and take an active role if they perceive the safety of the traffic control to be seriously and immediately endangered.
The NOSS observer is the only person in the whole NOSS process who has access to

controller's identities. It is essential therefore, that observers are reminded throughout the

training of their responsibilities in this regard. Confidentiality is paramount and observed

controllers behavior should not be discussed with anyone—not even other observers.

Experience has shown that at the end of a shift, controllers will often ask the observer to

"debrief" their performance. In these circumstances, it is essential that the observer

politely decline the invitation. This emphasizes the concept that the observer is not there

to evaluate the controllers, merely to record events.

NOSS observers should act in an unobtrusive and consistent manner so that line manager has a similarly positive experience of NOSS, which in turn will favorably affect their receptivity to the final results and outcomes.

2. Company policies and procedures

Observers need to be current with company policies and procedures so as to observe procedural adherence on the line and detect any deviations. All observers should also be encouraged to review the manuals as homework.

3. Observation form

This sub-section refers to the mechanics of correctly completing and submitting a NOSS observation form. During the training, the observers work to develop the needed competencies as defined by the observation form. Hence, observers should see the observation form as soon as possible after the training begins so that they have a clear sense of what is expected of them.

4. Threat and Error Management concepts

Using Threat and Error Management as the basis of its NOSS, observers need to receive training in the framework. Specifically, observers need to be able to define, distinguish, and identify threats, errors, and undesired states. This is best achieved with a mixture of lecture, case studies, and review.
NOSS observers should be trained to accept their role as observers, not evaluators—they

are not Check controllers NOSS observers will observe errors and undesired states

as part of their observations; however, they should only interrupt and take an active role

if they perceive the safety of the controlled traffic to be seriously and immediately

endangered.

The NOSS observer is the only person in the whole NOSS process who has access to

controller's identities. It is essential therefore, that observers are reminded throughout the

training of their responsibilities in this regard.

NOSS observers should act in an unobtrusive and consistent manner so that line pilots

have a similarly positive experience of NOSS, which in turn will favorably affect their

receptivity to the final results and outcomes.

5. Narratives

Observers need training in writing the flight narrative. If they understand the concepts underlying the observation form as well as the diagnostic rationale for conducting a NOSS, the observers will realize that a good-quality narrative is imperative.
The observation form should contain several prompts to help the observer provide sufficient detail.
In particular, observers need to record events that happened, such as threats, errors or undesired states, the context in which they happened, and the controller's response and management of the event.
Observers are selected because they are experts at understanding traffic control operations and this expertise is best expressed in detailed narratives. As long as the observer provides a detailed narrative of the shift operation, any coding oversights can be remedied later in the data-cleaning process.

6. Training Objectives

In sum, at the end of NOSS observer training, an observer should be able to:

1. Knowledgeably and confidently explain the rationale and process for conducting a NOSS at the shift operation

2. Enact the NOSS observer etiquette in a professional and consistent manner

3. Demonstrate knowledge of company policies and procedures

4. Use the observation form accurately and comprehensively

5. Understand the theoretical framework of the observation form. If the tool is based on Threat and Error Management concepts, the observer should be able to define, distinguish, and identify threats, errors, and undesired states.

6. Write detailed and comprehensive shift operation narratives from which others will be able

to understand the full context of the shift operation and related events.

7. OBSERVER STANDARDIZATION

Standardization refers to the need to be sure that shift operation details are recorded in a systematic and consistent fashion. In NOSS, standardization is a multi-step process that involves standardizing the observers, and conducting follow-up data-cleaning and coding of completed observations:

The first step in any observer standardization is good-quality training. To be sure observers understand the concepts, group discussions are encouraged. Focusing on the finer points of the model and the observation form, these discussions help calibrate the observers to a common standard. A test can be administered at the end of the training to be sure that all observers have grasped the necessary knowledge and can demonstrate the required competencies as specified by the training objectives. If observers complete this test satisfactorily, they can be released to the line to complete one or two “trial” observations.
The NOSS project coordinator or trainer should schedule time with each observer to discuss their trial observations. If the observer does a good job, as evidenced particularly by the quality of the narrative, the observations can be retained and used in the NOSS, and the observer can be sent back out to complete their observations.
If the quality of the narrative is poor, e.g., lacking sufficient detail, the trainer can work with the observer to help draw out missing information. If the observer has forgotten details and cannot recreate the shift operation, the observation must be discarded, and the observer sent out to complete another trial observation. It is the NOSS coordinator’s decision to drop any observer from the observation team if that person fails to meet the required standard. For this reason, it can be a good idea to initially recruit and train more observers than needed, to allow for attrition, illness, and scheduling conflicts.

8. Data standardization

8.1 There are several supplementary techniques that ensure good-quality standardized data are used in a NOSS:

· First, observers are not asked to evaluate performance, but simply to observe it. From a data standpoint, this is the distinction between subjective judgment and objective observation. For example, observers are asked to note threats without any subjective judgment—if there is a thunderstorm, record it; if there is a system malfunction or maintenance problem, record it. It is the same for errors and undesired states.

Observers also note the controller’s response to the threats, errors, and undesired states and the outcome. The observer is not assessing controller's performance or providing a subjective evaluation—the observer is telling the story of the shift operation.

Second, observation forms submitted by observers are always checked by the data analyst against the narrative. Observers are expert at describing a shift operation —they are not necessarily expert at assigning codes to the various threats, errors, etc. especially if it is their first time as a NOSS observer. While training is given in the codes, nonetheless it is realistic to accept that observers won’t necessarily retain this information perfectly. The narrative is the “fail-safe” in the system in that it allows the analyst to read the events of the shift operation and match them to the observer’s codes. Because the analyst is the expert with the codes, he or she can add any codes that were missed and recode anything that might not be quite correct. Hence, a good-quality narrative is the ultimate key to standardized data. Observers provide a comprehensive narrative, and the analyst ensures consistent and accurate coding.
A third step in standardizing the NOSS data prior to analysis involves verifying the data with a team of local experts— shift operation personnel familiar with the operation of each duty shift. The group’s task is to review and verify the observations against current manuals, policies, and procedures. For example, an observer might log a procedural error for failure to make an approach callout when in fact there is no written procedure in the operations manual. The “error” would then be deleted from the database. The data verification group acts as a check on the analyst’s coding, ensuring events are correctly recorded in line with each procedures and policies. It also builds ownership in the results and dispels any later criticism that the coding was not accurate.

8.2 There are several methods that ensure that NOSS data are consistently and accurately recorded:

• Observers are trained, calibrated, tested, and recalibrated;

• Objective observation, not subjective evaluation, is the basis and outcome of the observation;

• The narrative is stressed as key to high-quality data;

• The data analyst applies consistent coding to the observations;

• The data verification group checks the analyst’s coding against specific procedures

8.3 Following the above steps will ensure reliability and validity of the data analyzed from

NOSS observations. Steering committee, but not any of the observers). The group’s task is to review and verify the observations against current manuals, policies, and procedures.

The data verification group acts as a check on the analyst’s coding, ensuring events are correctly recorded in line with each shift operation procedures and policies. It also builds ownership in the results and dispels any later criticism that the coding was not accurate.

Rev. Date

ITEM

1

NOSS – What is it?

2.

3.

4.

5.

6.

7.

APPENDIX A

APPENDIX B