Performance evaluation and metrics projects

New Operational Procedures and Benefits Assessment for Achieving Fuel Savings in the Enroute and Descent Phases of Flight

speed controlSubstantial reductions in aircraft noise, emissions, and fuel usage can be achieved through the use of Continuous Descent Approaches (CDAs) and related concepts, e.g. Optimized Profile Descents (OPD’s), Tailored Arrivals, etc. However, it can be difficult to achieve these benefits in the near term because advanced arrival procedures are in many ways incompatible with current operational practice. This project developed new flight prioritization mechanisms and operational concepts to manage en route and terminal area traffic is such a way that much of the fuel savings and other benefits identified in various recent benefits analyses are achieved.

Under the general concept a controlled time of arrival (CTA) is set well in advance of the flight’s arrival in the terminal area of its destination airport. A suite of models was developed to do this so as to reduce congestion in the terminal area and allow flights to better follow their preferred approach trajectory.  In essence terminal area delay is transferred to the enroute phase of flight where is it much less costly.  The models developed seek to do this in a way treats flights/airlines in an equitable manner and also to take into account uncertainty associated with flight arrivals and airport capacity.

Sponsors: Federal Aviation Administration, Air Traffic Organization
Team: University of Maryland (UMD), Georgia Institute of Technology (GT). PI:  Michael Ball (UMD). CoPIs:  John Paul Clarke (GT), David Lovell (UMD). Researchers:  James Jones (UMD), Jim Brooks (GT)
Duration: 2011–2013
Documentation: | ATM 2013 Paper | ATM 2015 paper | Transportation Research Record paper | ICRAT 2012 paper | Transportation Science paper |

Analysis of Flight En Route Inefficiency

routingThe Federal Aviation Administration (FAA) and Eurocontrol publish metrics to evaluate flight horizontal en route inefficiency (HIE), which measures the excess distance flown with respect to the theoretical shortest distance route. It is a fundamental measure in the European performance scheme for air navigation services and is core to many fuel efficiency benefit assessments. In 2013, the average HIE was respectively 2.91% and 2.71% for Europe and the US, and it increased to 2.92% and 2.83% in 2015. Knowledge of the factors that drive variation in en route efficiency is, however, limited.

Under this project, advanced machine learning algorithms have been proposed and implemented to quantitatively understand how convective weather, wind, traffic management initiatives (TMIs) such as Miles-In-Trail (MIT) and Airspace Flow Programs (AFP), and Special Activity Airspace (SAA) contribute to the HIE. Our approach, which employs techniques such as trajectory clustering, feature engineering, trajectory prediction and deep neural networks, enables to produce numerical values of the contributions of the above contributing factors.

Sponsors: Federal Aviation Administration, Air Traffic Organization
Team: University of California, Berkeley (UCB); University of Maryland (UMD). PI: Mark Hansen (UCB); coPIs:  Michael Ball (UMD), David Lovell (UMD). Researcher: Yulin Liu (UCB)
Duration: 2015–2018
Documentation: TRB-2017-PAPER | ATM-2017-PAPER | ATM-2017-PRESENTATION | TRB-2017-PRESENTATION | ICRAT-2018-PAPER | ICRAT-2018-PRESENTATION | INFORMS-2018-PRESENTATION | TRB-2019-PAPER | TRB-2019-PRESENTATION | ArXiv-PAPER (online at Cornell) ArXiv-PAPER (PDF) |

Passenger Delay Analysis

Many of the existing methods for evaluating an airline’s on-time performance are based on flight-centric measures of delay. However, recent research has demonstrated that passenger delays depend on many factors in addition to flight delays. For instance, significant passenger delays result from flight cancellations and missed connections, which themselves depend on a significant number of factors. Unfortunately, lack of publicly available passenger travel data has made it difficult for researchers to explore the nature of these relationships.

Under this project, we developed methodologies to model historical travel and delays for U.S. domestic passengers. We developed a multinomial logit model for estimating historical passenger travel and extended a previously-developed greedy re-accommodation heuristic for estimating the resulting passenger delays. We reported and analyzed the estimated passenger delays for calendar year 2007, developing insights into factors that affect the performance of the National Air Transportation System in the United States. We also analyzed the causes and costs of U.S. passenger travel disruptions by applying data analysis and statistical modeling to historical flight and passenger data.

We demonstrated that as much as 50 percent of passenger delays are caused by passenger travel disruptions, either flight cancellations or missed connections. The propensity for disruptions varies significantly across airports and carriers, based on key factors such as scheduling practices, network structures, and passenger connections. We found that cancelations, which are the largest cause of disruption-related passenger delays, vary substantially across carriers, even when accounting for baseline variability across airports. Passenger and operational considerations also play a significant role in cancelation decisions. Regarding missed connections, much of the variability can be explained just by flight delays for the airport and carrier, though flight schedule construction is also a critical factor. Highly peaked (or banked) flight schedules tend to reduce connection times and therefore increase the risk of missed connections. Last, we demonstrated the importance of a variety of factors on the ease of re-accommodating disrupted passengers.

In addition to the analyses already performed in this research project, access to estimated passenger travel and delay data provided through this project has also enabled many other passenger-centric research opportunities that would not exist otherwise.

Sponsor: Air Traffic Organization, System Operations Services
Team: Massachusetts Institute of Technology, George Mason University

North Atlantic Alternate Benefits Study

In Phase 1 of this project we revised and validated many features of the NATSIM model developed by the FAA Tech Center. Virginia Tech delivered model changes in 23 weekly reports and design reviews accomplished in Phase 1. A final report on Phase 1 was delivered to the FAA in March 2012. In Phase 2 of this project we developed a new alternative benefits model called NATSAM benefits model to quantify cost/benefits of North Atlantic operations in the context of various ICAO/FAA equipage mandates. The NATSAM model development is complete and final modeling results will be presented at a future EFG meeting in Paris. Preliminary results of this model were presented at the October 4th INFORMS meeting in Myrtle Beach, SC. A final report for Phase 2 is due on December 2012.

This model will be used by FAA to support air navigation service provider decisions on future aircraft equipage mandates.

Sponsor: Air Traffic Organization
Team: Virginia Tech

Flight Predictability: Concepts, Metrics and Impacts

flight predictabilityIn this study, we analyze two benefit mechanisms by which improved flight time reliability could reduce airline operating cost. In one mechanism, reduced variability results in short scheduled block times, and airlines realize cost savings because scheduled block time is an important driver of several categories of airline costs. In the other, reduced variability results in less contingency fuel being loaded, and airlines save on the costs to carry the extra fuel. In examining both mechanisms, we employ econometric models to determine the relationship between flight time variability and the relevant behavior (scheduled block time setting or fuel loading) and additional models to estimate the cost impacts of changes in the behavior that could result from improving flight time reliability.

We first investigate how a change in actual block time distribution will affect scheduled block time (SBT) and system performance. This relationship is studied with empirical data and multiple regression models. The distribution of the historical block time for a flight is depicted by the difference between every 10th percentiles. We found that gate delay plays a minor role in setting SBT and that SBTs have decreasing sensitivity to historical flight times toward the right tail of the distribution. Using this behavior model, we generate different scenarios with hypothetical changes in actual block time distribution and evaluate the impacts of SBT, delay, and on-time performance. Focusing on the inner the right tail of the distribution would yield the most improvement in the system. It is also found that even an increase in average block time can still bring benefits such as reduced SBT and better on-time performance. Taking the heterogeneity across airlines into consideration, we also decompose the dataset to study specific airlines. Low cost carriers and United Airlines are found to be the more aggressive in setting SBT.

To measure the hypothetical economic benefits that such changes could have, Chapter 4 herewe use coefficients from the open literature to relate changes in scheduled block time to reductions in scheduled flights and crew staffing. Using fleet mix distributions and cost-of ownership data, plus pilot and cabin crew salary data, we monetize those benefits. Major carriers could save on the order of $20 million with fleet usage efficiencies; similar results could not be produced for the low-cost carriers because of missing data. For crew efficiencies, however, all categories of carriers could save millions of dollars, with about three quarters of that ascribed to cockpit crew and the remainder to cabin crew.

To study fuel loading, we analyze a large and recent dataset with flight-level fuel loading and consumption information from a major US airline. With thisthese data, firstly the relationship between the amount of loaded fuel and flight predictability performance is estimated using a statistical model. Then the impact on loaded fuel is translated into fuel consumption and ultimately, fuel cost for US domestic operations. We find that one minute of standard deviation in airborne time, capturing flight unpredictability, within a month for the same OD pair and shift of day would lead to 0.9 minute increase in loaded contingency fuel and 1.7 loaded contingency and alternate fuel.

We include a short chapter on preliminary investigations into predictability impacts of called rates for Ground Delay Programs. This effort was only gestational, so we are able to report only on some preliminary ideas for data analysis and presentation. This could be the starting point for a more exhaustive study at a later date.

Overall, we estimate annual cost savings to the US domestic airline industry on the order of $400 million from reductions in scheduled block time that could result from plausible changes in the block time distribution. We estimate fuel burn savings of $180-$240 million from eliminating flight time variability. We also find that different metrics are appropriate for these two mechanisms, with the scheduled block time based upon the inner right tail of the block time distribution (out to roughly the 70th percentile). Fuel loading considers the entire distribution, with particular focus on the outer right tail, making the standard deviation an appropriate metric.

Sponsor: Air Traffic Organization
Team: University of Maryland; University of California, Berkeley
Documentation: | ATM-2013-PAPER | TRB-2013-PAPER |

Flight Cancellation Behavior and Aviation System Performance

To better understand and mitigate flight delays, the FAA’s Office of Performance Analysis and Strategy (PAS) forecasts delays at the nation’s 30 major airports on a monthly basis. The purpose of this effort is to identify airports with significant delays, so that appropriate steps may be taken to prevent or mitigate such delays. The delay forecasts are performed six months in advance using demand projections based on published (look-ahead) airline schedules. The demand sets used for delay forecasts do not include flight cancellations made in response to operational contingencies. Therefore, in order to improve the fidelity of the FAA’s delay model, it will be necessary to formulate a suitable model for cancellations.

This study focuses on two pieces of analysis to fulfill this goal. First, the factors that lead to flight cancellations need to be understood. We use econometric choice models applied to a large historical dataset to infer the preferences and behaviors of airlines with respect to flight cancellations. We have found that flight characteristics are an important factor for cancellations, even when accounting for adverse weather effects. Airline heterogeneity is large and reveals different behavior across the industry.

Second, we will use queueing simulation techniques similar to that used by the current FAA model to determine the effect flight cancellations have on delay estimates. The cancellation model estimates are used to predict flight cancellations and evaluate their impact on queueing delay forecasts used by the FAA.

We examine the effect of cancellations on delays using a sample of 160 airport-days. Our results indicate that delays are over-predicted by the FAA queueing models by 30 percent when not considering flight cancellations. Even a very naïve model of flight cancellations, where only time-of-day and distance effects are considered, will reduce the delay estimates by 9 percent.

Finally, a case study was performed based on the current operating environment at SFO, where runway closures in the summer of 2014 will lower the airport’s capacity. Moreover, airlines are proposing schedules with 5% more demand. The increased schedule leads to an increase in queueing delay. We used a cancellation model with queueing delay as an explanatory variable to predict the increase in cancellations caused by the increase in demand. The results from our model indicate that departure cancellations will increase at an almost one-to-one ratio with the proposed demand increase, thus negating any benefit to airlines from a denser schedule.

Sponsor: Air Traffic Organization
Team: University of California, Berkeley
Duration: 2011–2014

Temporal Analysis of Operational Errors at Air Traffic Control Facilities

This project focuses on the occurrence of aircraft separation minimum violations as documented in the form of operational errors (OEs) at two types of air traffic controller facilities, terminal radar approach control (TRACON) facilities and air route traffic control centers (ARTCCs). Poisson regression was used to analyze the daily count of OEs at various facilities of both types. The occurrence of OEs was found to increase approximately with the square of daily traffic at TRACON facilities and slightly higher than the square of traffic at ARTCC facilities. At TRACON facilities, where separation violations are not automatically reported, an increase in reporting was seen after a new severity metric was introduced in 2007. It was also found that large, consolidated TRACON facilities tend to behave like a sum of several smaller facilities rather than a single larger facility with respect to the occurrence of OEs vs daily traffic. Weather effects such as visibility and wind were found to influence the occurrence of OEs as well. The model prediction for TRACON facilities is very good for the most severe OE types and very poor for the least severe OE types, indicating many unobserved factors contributing to the reporting of the least severe OE types in the terminal environment. Model prediction for the ARTCC facilities is very good for about half the facilities.

Sponsor: Air Traffic Organization
Team: University of California, Berkeley
Duration: 2010–2011

Air Connectivity Index

In this project, we investigate the measurement of air transport connectivity in the US air transportation network (ATN). In particular, we study the merit and applicability of the Air Connectivity Index (ACI) developed by researchers at the World Bank to measure the connectivity of the global air transport network. The ACI is derived using a generalized gravity model and accounts for both direct and indirect network effects. Our work has demonstrated that the ACI is also applicable to the US ATN, capturing the relevant aspects of connectivity and correlating with some econometric variables, but also presenting new challenges for correct implementation, especially in the proper estimation of regression parameters.

We determined how to adapt the original ACI calculation methodology to suit the US ATN. We applied the ACI at different levels of granularity (state, metropolitan, and airport) and conducted time series of the ACI at the metropolitan and airport level. We developed statistical models to predict ACI at the airport and metropolitan levels using facility and econometric variables. We investigated implications of results for the US ATN and how the index can be used by policy makers. We continue to substantiate the method for correct calculation in the US ATN and are collaborating with the original World Bank researchers to further improve and expand the global application of the ACI.

Sponsor: Air Traffic Organization, System Operations Services
Team: Purdue University

Optimized Profile Descent Modeling and Evaluation

The primary goal of this research was to evaluate the fuel saving/consumption of conflict-free Optimized Profile Descents (OPDs) under normal and heavy traffic conditions at different airport Metroplexes. In the research, we developed sequencing algorithms to strategically resolve conflicts for inbound traffic, and obtained fuel burn data using the FACET fast-time simulation model. The research results show that the fidelity of the fuel burn estimate strongly depends on how closely the simulation approximates the real operations. In a related previous research project, our research imposed several assumptions and constraints in the course of exercising the simulation, while in this NEXTOR II project, we employed more operationally realistic constraints to yield a more robust analytical result.

Sponsor: Air Traffic Organization, System Operations Services
Team: Purdue University. PI: Dengfeng Sun; Co-PI: Steven Landry.
Documentation: Evaluation of Continuous Descent Approach as a Standard Terminal Airspace Operation, Yi Cao, Tatsuya Kotegawa, Dengfeng Sun, Daniel DeLaurentis and Joseph Post, the 9th USA/Europe Air Traffic Management (ATM) Seminar, Berlin, Germany, June 14-17 2011.
Evaluation of fuel benefits due to continuous descent arrival procedures, Yi Cao, Li Jin, Nguyen V. P. Nguyen, Steven Landry, Dengfeng Sun and Joseph Post, submitted to Air Traffic Control Quarterly.

CNS Availability and NAS Performance: Controller-Pilot Data Link Communications (CPDLC)

In this study we investigate how unscheduled outages of Data Comm impact the National Airspace System (NAS) performance in the Next Generation Air Transportation System (NextGen) environment. A methodology is developed using analytical models and simulated data for translating impact of Data Comm outages on NAS performance. Fault tree analysis method and reliability block diagrams are used as engineering modeling techniques to describe Data Com system components and calculate system’s availability. Because Data Comm is not in operation yet, the developed methodology is based on simulated operational data. The overall methodology is based on several analytical models for estimating changes in capacity, safety and controller’s workload. The goal is to find the optimal balance between human psychological characteristics and strict engineering disciplines. Using the proposed methodology, calculated are delays, safety risks and excess fuel consumption that would occur due to a Data Comm outage. The conclusion is that the disadvantages caused by the Data Comm failure are negligible compared to the benefits of increased capacity and savings in delays that the implementation of Data Comm would bring. We especially explore any negative relationships between (1) the excess distance (difference between the filed route and the route actually flown) and (2) controller cognitive utilization. A more generalized concept of DataComm is also developed, providing an example to establish a standard methodology for other NAS systems in the NextGen environment. This is very important as it will provide a clear path for future analysis of other NAS systems.

Sponsor: Air Traffic Organization, NAS Systems Engineering Services Office
Team: University of California, Berkeley
Duration: 2011–2013

Survey and Analysis of Air Traffic Control Command Entries and the Effect on Decision Support Tool Performances

The Federal Aviation Administration (FAA) has developed a new Air Traffic Control (ATC) system to replace the existing Host Computer System (HCS) and User Request Evaluation Tool (URET) in the en route domain. This system is called En Route Automation Modernization (ERAM). Two key components of ERAM are the Trajectory Predictor (TP) and Conflict Probe (CP). While the TP predicts more accurate aircraft position, the CP predicts conflicts between two aircraft, or between an aircraft and a Special Use Airspace (SUA) using predicted flight paths (trajectories) generated by the TP. When the CP detects a conflict, the system alerts the air traffic controller of the problem in order to aid the controller to manage separation safely. The accuracy of these predictions is crucial to the performance of ERAM and the overall performance of en route airspace.

In order to support the implementation of improvements to the CP in ERAM, this study investigates and quantifies the impact and mitigation of CP prediction errors associated with missing aircraft intent data, attributable to the lack of automation input of verbal-only amendments by Air Traffic Control.

Hence, for the first time, a comprehensive methodology is developed to quantify the impact of missing aircraft intent information on the performance of the CP by analyzing the accuracy of the en route trajectory modeling capability and the usability of the CP. The proposed method focuses on the analysis of missing ATC clearances, while the application of the proposed method analyzes the impact of the 4th line ATC entries and then estimates the improvement of a complete, closed clearance of a 2-leg lateral maneuver. The dataset was obtained from the Washington, DC Air Route Traffic Control Center (ZDC) over 48 hours for January 26-27, 2010.

In the first step, a large database was created, comprised of the CMS, a record of air traffic controllers’ issued amendments, flight adherence measurement data, and weather information. Then, the amendments were interpreted and translated into new flight plans based on the amendments issued and the aircraft’s actual position. As a result, new CMS messages were created. In the second step, the new CMS messages were inserted into JEDI, a portable version of URET, to obtain error measurement metrics for trajectory accuracy.

There were four metrics measuring trajectory accuracy: horizontal, cross track, along track, and vertical errors. All were measured in nautical miles except for vertical error, which is measured in feet. The magnitudes of each treatment scenario’s error metrics were subtracted from the magnitudes of each baseline run’s error metrics to calculate error reduction (or improvement). Afterwards, the treatment scenarios were compared against the baseline run with a two sample paired t-test to determine if any statistically significant error reductions occurred, and study findings were discussed.

The study findings quantitatively demonstrate that (i) heading amendments increase trajectory accuracy, and (ii) speed amendments, noted in 4th line of the flight data block, decrease the overall number of false alerts generated by the Conflict Probe.

Sponsor: Air Traffic Systems Concept Development Group, Simulation & Analysis Team AJP-661; Federal Aviation Administration William J. Hughes Technical Center
Team: University of California, Berkeley
Duration: 2010

Advanced Stochastic Network Queuing Models of the Impact of 4D Trajectory Precision

diffusion modelsThe operation of the U.S. National Airspace System is migrating towards a paradigm of trajectory-based operations. Under four-dimensional trajectory (4DT) operations, aircraft fly an assigned 3D trajectory in space and meet scheduled times of arrival at points in airspace, with high precision. 4DT capability constitutes a cornerstone of FAA’s vision for the Next Generation Air Transportation System (NextGen), and particularly of an operational environment in which aircraft fly their preferred trajectories with reduced need for tactical air traffic control. The goal of this project was to assess the extent to which various sources of trajectory imprecision (and other sources of stochastic variance) would impact the ultimate system performance. The project involved the development of a suite of complex queuing network models, representing a spectrum of magnitudes of trajectory uncertainty, which in turn facilitate differing modeling techniques. Models developed include a robust deterministic queuing model, a continuum approximation based on a diffusion approximation, and a highly coupled network queuing model.

Sponsor: NASA Ames
Team: University of California, Berkeley (UCB); Massachusetts Institute of Technology; University of Maryland (UMD). PI:  Mark Hansen (UCB). CoPIs:  Amedeo Odoni (MIT), David Lovell (UMD), Alexandre Bayen (UCB). Researchers:  Shubham Gupta (MIT), Tasos Nikoleris (UCB), Kleoniki Vlachou (UMD), Tarek Rabbani (UCB), Nikolas Pyrgiotis (MIT).
Duration: 2007–2011
Documentation: | Transportation Research Part C paper (Modeling Delay) | Transportation Research Part C paper (Diffusion Approximation) | ATM 2009 paper | ICRAT 2010 paper (Queueing Models) | ICRAT 2010 paper (Diffusion Approximation) |