Implementing Augmented Reality in the Flight Deck for Single Pilot Operations

Abstract

Single Pilot Operations is a current topic with the potential to significantly affect the future of commercial aviation. While financially attractive for airlines, Single Pilot Operations bring forth important safety concerns, especially regarding the lack of human redundancy in the flight deck, an increased workload for the single pilot, reduced situational awareness and a higher risk of human error.

It is assumed that potential problems affecting Single Pilot Operations could be addressed by implementing an Augmented Reality (AR) device in the flight deck, by presenting additional information and supporting hints within the pilot’s field of view. Concretely, AR could be used to help reduce the single pilot’s workload, improve situational awareness and reduce the risk of human error.

This paper sets out to demonstrate two use cases for augmented reality in the flight deck. A system, called Pilot Assist, was developed that allows pilots to conduct checklists interactively with a Microsoft HoloLens. The system also provides a holographic Head-up-Display. Pilot Assist was developed and demonstrated with a fixed base Airbus A320 simulator at the Technical University of Wildau.

With the HoloLens’ spatial mapping capabilities – scanning and recognizing the environment around the user – it was possible to create a system that guides the pilot through the conduction of checklists. This is done by prompting the user towards the location of each checklist item in the cockpit, where information regarding necessary actions is projected. Furthermore, Pilot Assist is integrated with the aircraft systems, making it possible to obtain aircraft status data in real time, thus allowing error-checking of the pilot’s actions as well as automating the progress through checklists.

The holographic Head-up-Display allows the user to look at the surrounding environment while presenting critical flight data within the user’s field of view. The holographic Head-up-Display is intended to contribute to the pilot’s situational awareness.

Experts in the aviation field, including pilots, researchers and engineers had the chance to qualitatively assess the Pilot Assist tool. They pointed to limitations of both Pilot Assist and the HoloLens itself, but shared optimism as to how this technology and similar applications could indeed impact the future of flight operations. Concerns regarding the HoloLens’ weight, comfort and narrow field of view were expressed. However, continued development of head mounted devices (e.g. HoloLens 2) is expected in the coming years.

Further research into augmented reality applications in the flight deck is needed to advance this and other use cases. Nonetheless, the experts agreed Pilot Assist provides beneficial support during single pilot operation considering the current prototypical nature of the system.