FAA report raises questions about the oversight of maintenance repair stations

On 1 May 2013, the FAA Office of Inspector General office released a report (AV-2013-073) that detailed ongoing difficulties that the FAA has in overseeing the repair stations that provide airline maintenance services in the US and overseas. Like the airlines, organizations that provide FAA-approved repairs to airliner aircraft have to meet FAA regulatory standards.

While these organizations can be owned and controlled by an airline, increasingly these kinds of services, which could range from basic maintenance checks to complete aircraft overhauls, are being performed by contractors hired by the airliner rather than by an airline's own employees.

Based on a detailed study of a sample of certified repair stations in the US and elsewhere, Some of the key findings of the report was that while the FAA has invested significant efforts to create risk-based inspection procedures, those procedures are often ineffective, and in many cases inspectors don't even use the procedures when evaluating repair stations.

Aircraft maintenance responsibilities
Airlines are responsible for making sure that aircraft are maintained and repaired to the regulatory standards set forth by the FAA. Whether this work is done by the airline or an airline contractor, the organization doing the repairs has to do so according to FAA regulations. The FAA is responsible for making sure that the repair organizations meet FAA standards, and if they don't, the FAA is supposed to provide those organizations with guidance on how to stay in compliance.

Key issues raised by the report
According to the report, the ability of the FAA to fulfill their responsibilities has been hampered by the organization's lack of standardized processes, including missing or inadequate training of inspectors and not providing adequate guidance to repair stations.

What is a repair station?
The term 'repair station' refers to a maintenance facility that is certified by the FAA to perform maintenance, repair, inspection, overhaul, and alteration of aircraft and aircraft products. Not all repair stations are created equal, with only a small fraction of these stations authorized to maintain the kinds of large airliners used by most passengers.

Where the problems are
The report gives the impression that there is a serious problem with a wide range of reapir stations, and in fact that may be the case. However, the study looked at a very small sample of the repairs conducted by these repair stations.

As of September 2012, there were 4,788 total repair stations, of which 726 were outside the US. Of all of these repair stations, only 68 performed repairs on the kinds of aircraft that most US airline passengers may use.

The study covered the period from September 2008 to August 2011, and looked at documentation associated with work orders at 27 repair stations, of which 13 were outside the US. From these 27 stations, the FAA looked at a sample of 119 work orders from a population of 49,859 work orders.

How big is the problem?
The FAA found 57 errors within the 119 work orders studied. Further, after reviewing the tools and equipment at each facility and the 119 work orders, the FAA also found 92 of what they called "systemic deficiencies" at these 27 sampled repair stations. Based on their sample findings, the FAA estimated that from September 2008 to August 2011 there were deficiencies in about 37% of the estimated 589,573 work orders completed by the stations that the FAA inspects.

How does all this affect safety?
The answer is not clear for two reasons. First, because the definition of safety varies greatly depending on who you ask, and second, because the problems uncovered by the FAA do not necessarily imply that aircraft that are currently flying are being improperly repaired or that aircraft or their occupants are exposed to excessive risk or danger. The FAA study looked at documentation related to aircraft maintenance actions, and did not directly inspect the aircraft that were subject to the work orders that were reviewed.

One of the more popular safety measures used by the public is the number of airliner crashes that kill passenges. Using that measure, US airlines (specifically airlines offering passenger flights on aircraft operating under FAA Part 121 regulations) are experiencing the lowest ever number and rate of events causing airline passenger deaths, though maintenance-related issues still play a role in some crashes.

During the last ten years, there have been four events that have resulted in passenger deaths. One of those events was the December 2005 crash of a Chalk's Ocean Airways Grumman Turbo Mallard flying boat, where the NTSB cited FAA's failure to detect and correct deficiencies in the company's maintenance program as one of the probable causes. That maintenance would have taken place prior to the period covered by the latest FAA IG report.

Key definitions of risk and safety
While safety is a word that is often used to describe aviation situations involving potential harm, safety is not a concept that has a consistent definition. On the other hand, the concept of risk is much more well defined, and that concept can be used to help describe the situations uncovered by the FAA report. The FAA in its 2009 document FAA Risk Management Handbook (FAA-H-8083-2) has three key definitions that may help to clarify the key issues around the FAA Inspector General's report:

• Risk - The future impact of a hazard that is not eliminated or controlled.
• Risk Assessment - An approach to managing uncertainty. Risk assessment is a quantitative value assigned to a task, action, or event.
• Risk Management - The part of the decision-making process which relies on situational awareness, problem recognition, and good judgment to reduce risks associated with each flight.

In short, in the eyes of the FAA, risk is something that can be objectively defined, measured, and managed. In contrast, safety may seem like a related concept, and in fact some key FAA documents like FAA Order 8040.4A (effective 30 April 2012) explicitly state that the terms safety and risk are used interchangeably. The findings of the FAA IG report clearly imply that the FAA maintenance oversight process allows a significant fraction of work orders to be improperly completed. Because of that, it increases the amount of maintenance-related risks in the system.

Is safety different from risk?
Safety and risk may be the same in the context of some FAA documents, but that is not the case for the average passenger or even the average aviation professional. In general, safety doesn't represent something objective and measurable like the FAA's definition of risk, but rather something subjective like the acceptance or tolerance of risk or uncertainty.

Risk can be defined differently for different situations, but it typically uses two key elements, a specific hazard and some kind of frequency or probability of occurrence. Safety on the other hand could be described as a level of acceptability of a hazard, or the frequency of that hazard occurring.

To use a crude example, since 1960, the risk of a fatality of one or more occupants during an intended space flight is about one per 100 flights. In contrast, the likelihood of a fatality on airline passenger flight on a large transport aircraft has far smaller than one per million flights. In spite of the reality of fatal accidents for both kinds of flights, anyone who decides to fly in the air or in space would not do so unless he or she first decided that it was safe to do so (that the risk is acceptable) before strapping in.

Different points of view on the FAA report
Several prominent aviation safety professionals have had a number of comments about the report, and when reading these comments, keep in mind that they were likely talking about both risk and safety, though not making the kind of distinction made by AirSafe.com.

• John Goglia - This former NTSB board member with over 50 years of experience as a maintenance professional was quoted by USA Today on 6 May 2013 as saying that the discrepancies are a "major concern" because they mean that airlines aren't catching maintenance problems and the FAA isn't catching the airlines. Further, he said "That doesn't mean an airplane is going to fall out of the sky tomorrow, it means the system is broken."
• Christian Klein - Vice president of the Aeronautical Repair Station Association, an organization representing aircraft maintenance stations, said in the same article that "Shortcomings at the FAA don't translate into safety deficiencies in the industry."
• Rudy Quevedo - Director of global programs for the Flight Safety Foundation, was quoted in the same article as saying that "It's a perfectly safe system," but elsewhere he has also said that the problems identified need to be addressed as soon as possible to mitigate latent risk. Quevedo has also said that based on a standard risk management methodology of assessing probability and severity of what could happen because of the known deficiencies, that the problems, while serious, do not pose a "high level of danger" to air travel.

Quevedo's comments speak most directly to the way that the aviation industry may look at this situation. In the FAA Risk Management Handbook, there would be a high level of danger if the severity of the hazard were critical or catastrophic, combined with a likelihood that was occasional or probable. Based on the FAA IG report, it would appear that problems with work orders would be in the probable range of likelihood, but that the severity of the hazard is less than critical.