Search

The Resilient Synthetic Vision for Advanced Control Tower Air Navigation Service Provision (RETINA) project is one of the selected Single European Sky ATM Research (SESAR) exploratory research projects on High Performing Airport Operations and it investigates the potential and applicability of Virtual/Augmented Reality (V/AR) technologies for the provision of Air Traffic Control (ATC) service by the airport control tower. The project assesses whether those concepts that stand behind tools such as Head-Mounted Displays (HUDs), Enhanced Vision Systems (EVSs) and Synthetic Vision Systems (SVS) can be transferred to ATC with relatively low effort and substantial benefits for controllers' Situational Awareness (SA). In doing so, two different augmented reality systems are investigated: Spatial Displays (SD) that, potentially, can be made to coincide with the tower windows and See-Through Head-Mounted Displays (ST-HMD). In this context the RETINA concept will enable the Air Traffic Controller to have a head-up view of the airport traffic even in low visibility conditions, similar to the vision currently provided in the cockpit with Head-Up displays. In the two-year project, the RETINA concept was developed, implemented and validated by means of human-in-the-loop simulations where the external view is provided to the user through a high fidelity 3D digital model in an immersive environment.

An increasing degree of automation in air traffic will also change the role of the air traffic controller (ATCO). ATCOs will fulfill significantly more monitoring tasks compared to today. However, this rather passive role may lead to Out-Of-The-Loop (OOTL) effects comprising vigilance decrement and less situation awareness. The project MINIMA (Mitigating Negative Impacts of Monitoring high levels of Automation) has conceived a system to control and mitigate such OOTL phenomena. In order to demonstrate the MINIMA concept, an experimental simulation set-up has been designed. This set-up consists of two parts: 1) a Task Environment (TE) comprising a Terminal Maneuvering Area (TMA) simulator as well as 2) a Vigilance and Attention Controller (VAC) based on neurophysiological data recording such as electroencephalography (EEG) and eye-tracking devices. The current vigilance level and the attention focus of the controller are measured during the ATCO’s active work in front of the human machine interface (HMI). The derived vigilance level and attention trigger adaptive automation functionalities in the TE to avoid OOTL effects. This paper describes the full-scale experimental set-up and the component development work towards it. Hence, it encompasses a pre-test whose results influenced the development of the VAC as well as the functionalities of the final TE and the two VAC’s sub-components., {"references":["ACARE, \"Flightpath 2050-Europe's Vision for Aviation,\" Advisory Council for Aeronautics Research in Europe, 2011.","SESAR JU, \"European ATM Master Plan,\" 2012.","HALA!, \"Deliverable, I. D. Position Paper,\" hala-sesar.net, 2012.","E. E. Jones, A. R. Carter-Sowell, J. R. Kelly, and K. D. Williams, \"I'm out of the loop': Ostracism through information exclusion,\" Group Processes & Intergroup Relations, 12(2), 2009, pp. 157–174.","P. Arico, G. Borghini, G. Di Flumeri, N. Sciaraffa, A. Colosimo, and F. Babiloni, \"Passive BCI in Operational Environments: Insights, Recent Advances and Future trends,\" IEEE Transactions on Biomedical Engineering, 2017.","M. I. Posner and S. J. Boies, \"Components of attention,\" Psychological review, 78(5), 391, 1971.","D. Kahneman, \"Attention and effort,\" Vol. 1063, Englewood Cliffs, NJ, Prentice-Hall, 1973.","R. Parasuraman, J. S. Warm, and J. E. See, \"Brain systems of vigilance,\" in The attentive brain, Cambridge, MA, US: The MIT Press, 1998, pp. 221–256.","M. Steriade, P. Gloor, R. R. Llinas, F. L. Da Silva, and M.-M. Mesulam, \"Basic mechanisms of cerebral rhythmic activities,\" Electroencephalogr. Clin. Neurophysiol., vol. 76, no. 6, pp. 481–508, 1990.\n[10]\tW. Sturm and K. Willmes, \"On the Functional Neuroanatomy of Intrinsic and Phasic Alertness,\" Neuroimage, vol. 14, no. 1, pp. S76–S84, Jul. 2001.\n[11]\tM. Singh and A. Sharma, \"Correlational study of attention task performance and EEG alpha power,\" Int. J. Inf. Technol. Knowl. Manag., vol. 8, no. 2, pp. 188–196, 2015.\n[12]\tF. M. Howells, D. J. Stein, and V. A. Russell, \"Perceived mental effort correlates with changes in tonic arousal during attentional tasks,\" in Behav. Brain Funct., vol. 6, p. 39, 2010.\n[13]\tE. Molina, Á. Correa, D. Sanabria, and T. P. Jung, \"Tonic EEG dynamics during psychomotor vigilance task,\" 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER), 2013, pp. 1382–1385.\n[14]\tR. T. Wilkinson and D. Houghton, \"Field test of arousal: a portable reaction timer with data storage,\" in Hum. Factors, vol. 24, no. 4, pp. 487–493, Aug. 1982.\n[15]\tD. Gould et al., \"Visual Analogue Scale (VAS),\" Journal of Clinical Nursing, 2001, 10, pp. 697–706.\n[16]\tT.-W. Lee, M. Girolami, and T. J. Sejnowski, \"Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources,\" Neural Comput., vol. 11, no. 2, pp. 417–441, Feb. 1999.\n[17]\tA. Delorme and S. Makeig, \"EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis,\" J. Neurosci. Methods, vol. 134, no. 1, pp. 9–21, Mar. 2004.\n[18]\tF. J. Harris, \"On the Use of Windows for Harmonic Analysis With the Discrete Fourier Transform,\" Proc. IEEE, vol. 66, no. 1, pp. 51–83, 1978.\n[19]\tW. Klimesch, \"EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis,\" Brain Res. Rev., vol. 29, no. 2–3, pp. 169–195, Apr. 1999.\n[20]\tP. Aricò, G. Di Borghini, G. Di Flumeri, S. Bonelli, A. Golfetti, I. Graziani, S. Pozzi, J.-P. Imbert, G. Granger, R. Benhacene, D. Schaefer, F. Babiloni, \"Human Factors and Neurophysiological Metrics in Air Traffic Control: a Critical Review,\" IEEE reviews in biomedical engineering, 2017.\n[21]\tP. Aricò, G. Borghini, G. Di Flumeri, A. Colosimo, S. Bonelli, A. Golfetti, and F. Babiloni, \"Adaptive automation triggered by EEG-based mental workload index: a passive brain-computer interface application in realistic air traffic control environment,\" Frontiers in human neuroscience, 10, 2016."]}

The SESAR (Single European Sky Air Traffic Management Research) Joint Undertaking has recently granted the Resilient Synthetic Vision for Ad- vanced Control Tower Air Navigation Service Provision project within the framework of the H2020 research on High Performing Airport Operations. Here- after, we describe the project motivations, the objectives, the proposed method- ology and the expected impacts, i.e. the consequences of using virtual/augmented reality technologies in the control tower.