Search

Geo-localisation from a single aerial image for Uncrewed Aerial Vehicles (UAVs) is an alternative to other vision-based methods, such as visual Simultaneous Localisation and Mapping (SLAM), seeking robustness under GPS failure. Due to the success of deep learning and the fact that UAVs can carry a low-cost camera, we can train a Convolutional Neural Network (CNN) to predict position from a single aerial image. However, conventional CNN-based methods adapted to this problem require off-board training that involves high computational processing time and where the model can not be used in the same flight mission. In this work, we explore the use of continual learning via latent replay to achieve online training with a CNN model that learns during the flight mission GPS coordinates associated with single aerial images. Thus, the learning process repeats the old data with the new ones using fewer images. Furthermore, inspired by the sub-mapping concept in visual SLAM, we propose a multi-model approach to assess the advantages of using compact models learned continuously with promising results. On average, our method achieved a processing speed of 150 fps with an accuracy of 0.71 to 0.85, demonstrating the effectiveness of our methodology for geo-localisation applications.

In this work, we present an approach to address the problem of warehouse inspection using a MicroAir Vehicle (MAV) that performs autonomous flight by following a set of waypoints in a GPS-denied environment. During the navigation, a second onboard camera is used to observe the inspection area where it is expected to observe QR codes attached to packages placed on shelves. To this end, we use a metric monocular simultaneous localization And Mapping (SLAM) system to estimate the MAVs position in metres, avoiding the need for an external positioning system. The onboard computer executes the detection and decoding of QR codes and the SLAM system. The MAV can also communicate with a Ground Control Station (GCS) to display telemetry, the MAVs position, images from the inspection camera with detected QR codes, and a list of the found packages. This approach was used to compete in the indoors competition of the International Micro Air Vehicle Competition (IMAV) 2019, where we received the special award: Best Flight Performance.

We present a new aerial image dataset, named ESPADA, intended for the training of deep neural networks for depth image estimation from a single aerial image. Given the difficulty of creating aerial image datasets containing image pairs of chromatic images related to their depth images, simulators such as AirSim have been proposed to generate synthetic images from photorealistic scenes. The latter enables the generation of thousands of images that can be used to train and evaluate neural models. However, we argue that synthetic photorealistic aerial image datasets can be improved by adding images generated from photogrammetric models imported into the simulator, thus enabling a less artificial generation of both chromatic and depth images. To assess the quality of these images, we compare the performance of 4 deep neural networks whose pre-trained models and code for re-training are publicly available. We also use ORB-SLAM, in its RGB-D version, to indirectly assess the estimated depth image. To accomplish this, chromatic images from 3 aerial videos and their depth images, estimated with the networks trained with ESPADA, are fed into ORB-SLAM. The estimated camera pose is compared against the trajectory retrieved from the GPS flight trajectory. Our results indicate that images generated from photogrammetric models improve the performance of depth estimation from a single aerial image.

Wind turbines require periodic inspection to ensure efficient power generation and a prolonged lifetime. Traditionally, inspection involves the risk of a person falling while abseiling from the top of the nacelle. To avoid this, drones have been controlled by operators to inspect the blades. However, this task requires expert pilots, who experience fatigue quickly. Alternatively, autonomous drones are not subject to human tiredness and can follow trajectories in a repeatable manner. Motivated by the latter, we introduce a vision-based blade detector capable of recognizing their orientation and relative position to generate a flight plan that allows it to safely collect image data. The proposed blade detector extracts line features with the camera, which are filtered to reduce the search space by using bounding boxes. They are obtained with a single-shot detector based on a convolutional neural network. Finally, a random sample consensus procedure finds the lines that best fit a geometrical model of the wind turbine. We compare our deep learning approach against a color segmentation method, showing that it is up to 6 times faster. We also compare against guided search during random sampling, which exploits the separate boxes detected by the network, seeking to reduce the number of outliers. We conclude with an illustrative example of how our proposed detector could be used for autonomous wind turbine inspection.

The first Autonomous Drone Racing (ADR) was launched in the IEEE IROS 2016, and continued to be organised in IROS 2017, 2018 and 2019. Inspired by this competition, other international competitions were launched: the AlphaPilot organised by Lockheed Martin in collaboration with the Drone Racing League, and the Game of Drones, organised by Microsoft and Stanford University. A distinctive feature in the IROS ADR and AlphaPilot competitions is that competing drones had to perform on-board processing only. Hence, along these years, teams have presented novel solutions for on-board processing based on a Graphic Processing Unit (GPU), a Field-Programmable Gate Array (FPGA), microcontrollers and embedded computers such as the Odroid or Intel Stick computers. Motivated by the variety of these solutions, the goal of this work is that of providing the reader with a detail description of the hardware used for the competitions, their benefits as much as limitations, including those microchips used in specialised sensors such as the RGB-D and stereo cameras whose data processing is carried out on the sensor itself. It is expected to conclude that GPU will stand out as the best hardware to compute complex processing on-board, in particular due to the use of deep learning not only to address the gate detection problem, but also to address the control and planning tasks involved in this challenge.

In autonomous drone racing, a drone flies through a gate track at high speed. Some solutions involve the use of camera localisation to control the drone. However, effective localisation is very demanding in processing time, which may compromise the flight speed. To address the latter, we propose a deep learning-based method for camera localisation that processes a small sequence of images of the scene at high-frequency operation. Our solution is a compact convolutional neural network based on the Inception network that uses a sequence of grey-scale images rather than colour images as input; we have called this network ‘GreySeqNet’. Our approach aims at leveraging the localisation process using a small stack of consecutive images fed as input to the network. To save computational effort, we explore the use of grey images instead of colour images, thus saving convolutional layers. We have conducted experiments in a simulated environment to measure the performance of GreySeqNet in different race tracks with variations in gate’s height and position. According to the results obtained in several test runs, our method achieves a camera pose estimation at an average operation frequency of 83 Hz running on GPU and 26 Hz on CPU, with an average camera pose error of 31 cm.

High-level structure (HLS) extraction recovers 3D elements on human-made surfaces (objects, buildings, ground, etc.). There are several approaches to HLS extraction. However, most of these approaches are based on processing two or more images captured from different camera views or on processing 3D data in the form of point clouds extracted from the camera images. In general, 3D point cloud and multiple views approaches have good performance for certain scenes with video sequences or image sequences, but they need sufficient parallax in order to guarantee accuracy. To address this problem, an alternative is to process a single RGB image seeking to interpret areas of the images where the human-made structure may be observed, thus removing parallax dependency, but adding the challenge of having to interpret image ambiguities correctly. Motivated by the latter, we propose a methodology for 3D volumetric structure extraction from a single image. Our strategy is to divide and simplify the 3D structure extraction process. For that, our methodology has three steps. First, the structure recognition step provides the segmentation, location, and delimitation of the urbanized structures in the scene. Second, we propose a graph analysis to classify and locate the boundaries between the different urbanized structures in the scene. Third, we use a proposed CNN and the pinhole camera model to extract the 3D volumetric structure. On the other hand, we evaluate this methodology in synthetic and public datasets.

This article aims at collecting and discussing the results reached by the research community regarding the study of the ground effect on small rotorcraft unmanned aerial vehicles, especially from the modeling and control point of view. Rotorcraft performance is affected by the presence of the ground or any other boundary that alters the flow into the rotors. Specifically, the ground effect can induce perturbations in the flight stability, when operating near the ground. For a rotorcraft, an accident is likely to happen when the vehicle leaves or enters the ground effect region, which may cause crashes and property damages. Today, the use of unmanned aerial vehicles has grown widespread, which raises safety concerns when they are flying at very low altitudes and near the ground. Consequently, studying the influence of the ground over rotorcrafts is of paramount importance for general safety. Also, these investigations can be used to design systems of guidance, navigation, and control. In this review, we break down the most relevant works to date. We discuss aspects related to modeling, control, and application of the ground effect for small-scale multirotors, as well as other aerodynamic proximity effects, such as the ceiling and wall effects. We conclude by mentioning potential avenues of research when studying the ground effect from the point of view of the robotics and artificial intelligence fields.

High-Level Structure (HLS) extraction in aerial images consists of recognizing Three-Dimensional (3D) elements on human-made surfaces (objects, buildings, ground, etc.). There are several approache...

Different imaging solutions have been proposed over the last few decades, aimed at three-dimensional (3D) space reconstruction and obstacle detection, either based on stereo-vision principles using active pixel sensors operating in the visible part of the spectra or based on active Near Infra-Red (NIR) illumination applying the time-of-flight principle, to mention just a few. If extremely low quantum efficiencies for NIR active illumination yielded by silicon-based detector solutions are considered together with the huge photon noise levels produced by the background illumination accompanied by Rayleigh scattering effects taking place in outdoor applications, the operating limitations of these systems under harsh weather conditions, especially if relatively low-power active illumination is used, are evident. If longer wavelengths for active illumination are applied to overcome these issues, indium gallium arsenide (InGaAs)-based photodetectors become the technology of choice, and for low-cost solutions, using a single InGaAs photodetector or an InGaAs line-sensor becomes a promising choice. In this case, the principles of Single-Pixel Imaging (SPI) and compressive sensing acquire a paramount importance. Thus, in this paper, we review and compare the different SPI developments reported. We cover a variety of SPI system architectures, modulation methods, pattern generation and reconstruction algorithms, embedded system approaches, and 2D/3D image reconstruction methods. In addition, we introduce a Near Infra-Red Single-Pixel Imaging (NIR-SPI) sensor aimed at detecting static and dynamic objects under outdoor conditions for unmanned aerial vehicle applications.

In this paper, we propose a method for single-pixel 2D/3D imaging to be applied in hyperspectral cameras operating in outdoor applications in unmanned aerial vehicles (UAVs). In rain rich environments, if camera systems operating in the visible range are used, alongside the atmospheric absorption that inevitably takes place, additional interactions occur between the light to be detected and the raindrops that produce Rayleigh and other scattering effects, thus limiting the visibility of the imagers mainly due to induced limitations in what radiation propagation is concerned. To solve this problem, we follow the approach that consists in using InGaAs based single photodetectors or line sensors that operate in the near-infrared (NIR) part of the spectra, in the wavelength range between 905 nm and 1600 nm. Due to the high costs of the few commercially available InGaAs based image sensors, we propose implementing a vision system based on the principle of single-pixel imaging (SPI) using active NIR illumination as a low-cost solution for image sensors in low-light scattering media. The SPI approach allows for the reconstruction of images from the electrical response signals of single InGaAs photodetectors to impinging active NIR illumination patterns created by an array of NIR LEDs. The latter is enabled by applying compression sensing (CS) algorithms. In this work, we aim at improving the quality of the reconstructed 2D images, defining the SSIM > 0.6 and RMSE

This paper presents a GPU-based real-time approach for generating high-definition (HD) aerial image mosaics. The cumbersome process of registering HD images is addressed by a parallel scheme that rapidly matches binary features. The proposed feature matcher takes advantage of the fast ORB (oriented FAST and rotated BRIEF) descriptor and its attainable arrangement into hash tables. By exploiting the best functionalities of binary descriptors and hashing-based data structures, the process of creating HD mosaics is accelerated. On average, real-time performance of 14.5 ms is achieved in a frame-to-frame process, for input images of 2.7 K resolution (2704 × 1521). For evaluation purposes in terms of robustness and speed, we selected two image registration methods for comparison. The first method uses the feature extractor and matcher modules of the well-known ORB-SLAM. The second comparison is carried out against the standard KNN-based matcher of OpenCV. The experiments were conducted under different conditions and scenarios, and the proposed approach exhibits a speed-up of 10.5 times compared to ORB-SLAM-based approach and 36.5 times compared to the OpenCV matcher. Therefore, this research widens the range of applications for aerial mosaicing, since the proposed system is capable of creating high-detail panoramas of large sites while acquiring data.

Object detection has become an essential tool in aerial robotics thanks to the use of onboard cameras in drones that enables find objects using techniques of vision. However, vision algorithms may become unreliable presenting drawback by the illumination changes. Deep learning has been used to solve tasks of classification, segmentation and detection using traditional Convolutional Neural Network (CNN) like VGG16, YOLO and AlexNet. This paper presents a gates detector system in a real-time using CNN based on a Single Shot Detector Network (SSD) for drone racing circuits. For the latter, we have adopted the SSD7 architecture to modified and present an implementation with five layers, reducing the prediction time and improve detection velocity in comparison with other architectures. For evaluation purpose, we selected three environments: simulation, indoors and outdoors to compare the prediction time, average fps and the confidence obtained in the detections of the gates.

Nowadays, depth estimation from a single image is a task that has been successfully addressed by Convolutional Neural Network (CNN) architectures. In this regard, several authors have taken advanta...

Recent advances in deep learning, in particular in convolutional neural networks (CNNs), have been widely used in robotics for object classification and action recognition, among others, with very high performance. Nevertheless, this high performance, mostly in classification tasks, is rarely accompanied by reasoning processes that consider the relationships between objects, actions, and effects. In this article, we used three CNNs to classify objects, actions, and effects that were trained with the CERTH-SOR3D dataset that has more than 20,000 RGB-D videos. This dataset involves 14 objects, 13 actions, and in this article was augmented with seven effects. The probabilistic vector output of each trained CNN was combined into a Bayesian network (BN) to capture the relationships between objects, actions, and effects. It is shown that by probabilistically combining information from the three classifiers, it is possible to improve the classification performance of each CNN or to level the same performance with less training data. In particular, the recognition performance improved from 71.2% to 79.7% for actions, 85.0%–86.7% for objects, and 77.0%–82.1% for effects. In the article, it is also shown that with missing information, the model can still produce reasonable classification performance. In particular, the system can be used for reasoning purposes in robotics, as it can make action planning with information from object and effects or it can predict effects with information from objects and actions.

Visual relocalisation is a well-known problem in the robotics community, where chromatic images are used to recognise a place that is being re-visited or re-observed again. Due to the success of deep neural networks in several computer vision tasks, convolutional neural networks have been proposed to address the visual relocalisation problem as well. However, these solutions follow the conventional off-line training in order to generate a model that can be used to regress a camera’s pose w.r.t to an input image. In this work, we present a methodology based on continual learning to address the visual relocalisation problem aiming at performing on-line model training, seeking to generate a model that is updated continuously to learn new acquired images associated with GPS coordinates. Moreover, we apply this methodology to the multi-camera case, where 8 images are acquired from a multi-rig camera, seeking to improve the localisation accuracy, this is, by using a multi-camera, we obtain a set of images observing different viewpoints of the scene for a given GPS position. Therefore, by using a voting scheme, our on-line learned model is capable of performing visual relocalisation with an accuracy of 0.78, performing at 50 fps.

The main goal in task planning is to build a sequence of actions that takes an agent from an initial state to a goal state. In robotics, this is particularly difficult because actions usually have several possible results, and sensors are prone to produce measurements with error. Partially observable Markov decision processes (POMDPs) are commonly employed, thanks to their ability to model the uncertainty of actions that modify and monitor the state of a system. However, since solving a POMDP is computationally expensive, their usage becomes prohibitive for most robotic applications. In this article, we propose a task planning architecture for service robotics. In the context of service robot design, we present a scheme to encode knowledge about the robot and its environment, that promotes the modularity and reuse of information. Also, we introduce a new recursive definition of a POMDP that enables our architecture to autonomously build a hierarchy of POMDPs, so that it can be used to generate and execute plans that solve the task at hand. Experimental results show that, in comparison to baseline methods, by following a recursive hierarchical approach the architecture is able to significantly reduce the planning time, while maintaining (or even improving) the robustness under several scenarios that vary in uncertainty and size.

We present a metric localisation approach for the NAO robot based on the methodology of depth estimation using optical flow in a frame-to-frame basis. We propose to convert optical flow into a 2-channel image from which images patches of \(60\times 60\) are extracted. Each patch is passed as input to a Convolutional Neural Network (CNN) with a regressor in the last layer, thus a depth value is estimated for such patch. A depth image is formed by putting together all the depth estimates obtained for each patch. The depth image is coupled with the RGB image and then passed to the well known ORB-SLAM system in its RGB-D version, this is, a visual simultaneous localisation and mapping approach that uses RGB and depth images to build a 3D map of the scene and use it to localise the camera. When using the depth images, the estimates are recovered with metric. Hence, the NAO’s position can be estimated in metres. Our approach aims at exploiting the walking motion of the robot, which produces image displacements in consecutive frames, and by taking advantage from the fact that the NAO’s walking motion could be programmed to be performed at constant speed. We mount a depth camera on the NAO’s head to produce a training dataset that associates patch RGB images with depth values. Then, a CNN can be trained to learn the patterns in between optical flow vectors and the scene depth. For evaluation, we use one of the in-built NAO’s camera. Our experiments show that this approach is feasible and could be exploited in applications where the NAO requires a localisation systems without depending on additional sensors or external localisation systems.

In this paper, a theoretical analysis of a vision system for 2D/3D single-pixel imaging using principles of compressed sensing and Indirect-Time-of-flight measurements is presented. The goal is to use this system for autonomous drone navigation in environments with adverse conditions where typical sensors used by RGB or RGB-Depth cameras typically fail. The harsh environments considered include smoke, rain, or fog. The performance evaluation is based on the signal-to-noise ratio considering different levels of background illumination, measured depths oscillating between 1 m and 10 m, different percentages of object reflectivity, and finally, the achievable spatial resolution understood as the standard deviation of the distance measured for a particular object in the illuminated scene under a set of defined border conditions. For the vision system proposed, we consider active illumination consisting of an array of NIR LEDs emitting Hadamard illumination patterns, and a pulsed laser diode used for Indirect-Time-of-flight. We propose using an InGaAs NIR sensitive photodiode as a single-pixel detector. Since the proposed vision system will be mounted on a drone, parameters such as weight, dimensions, power consumption, and processing time were considered to maximize its efficiency.

Physical contact with the surrounding environment is an essential and challenging task for unmanned aerial vehicles (UAV) to perform complex missions in the aerial manipulation field. Steady flight of an aerial manipulator is compromised due to the disturbances forces induced by the actuators of the manipulator, and the conditions of the indoor/outdoor environment. This paper focuses on the pose control performance of the aerial manipulator composed of an UAV and a 2-DOF arm for full contact of the system with a vertical surface in an outdoor scenario. The control technique is based on the Gain Scheduling (GS) approach incorporated into a proportional-integral-derivative (PID) algorithm to reduce the offset of the instability produced by the movement of the robotic arm, the wall effect and the external conditions like wind gusts. The visual simultaneous location and mapping (SLAM) method is implemented, exploiting the onboard sensing capabilities of the UAV without using another external caption motion method to estimate the pose of the system in the outdoor environment. The control design was implemented through extensive outdoor flight experiments to analyzed the behavior of the system in such conditions and obtained the gain value for the set of controllers. Experimental results of aerial contact with a surface show that the controller can suppress the disturbance mentioned above effectively, and make the system hover steadily with sufficient accuracy to complete aerial contact mission in an outdoor environment.

The compressive sensing (CS) technique is a novel tool used to reconstruct images using fewer samples, normally sparse in the transform domain, than those required by conventional imaging systems. However, the methods applied for signal reconstruction within the CS approach still present some problems in the implementation, mainly due to their intensive computational demand and high power consumption requirements. These drawbacks need addressing if this approach is followed in systems aimed at e.g. drone autonomous flying or other embedded applications that additionally require very short processing times. In this paper we evaluate the use of hardware based parallel processing architecture for the implementation of the Orthogonal Matching Pursuit (OMP) algorithm, one of the most efficient CS reconstruction algorithms developed so far. To improve the algorithm performance, we target different maximum allowed processing times to reach minimum image resolutions required by each system of interest using different sparse (16 and 64) amounts of single-pixel generated samples per image. We also target the final image resolution to be above 20 dB in terms of the peak signal-to-noise ratio (PSNR). To reduce the execution and processing times required to generate each image, we propose implementing parallel kernels in the hardware platform for each of the operations required by the algorithms under study. In the proposed implementation the reconstructed images are used to generate video streams that form the foundation on which decisions are to be made by the system in continuous time, whereby each single image (frame) reconstruction cannot overcome 30 ms in order to maintain the minimum amount of frames per second (fps) above 33 (minimum required for an acceptable video stream). The implementation of a variation of the OMP algorithm in a graphics processing unit (GPU) using parallel architecture approach allows obtaining processing times 4 or 5 times shorter than those obtained if central processing unit (CPU) based architecture implementation is used for the same purpose.

During the last decades the radio detecting and ranging (RADAR) technology underwent an evolution transiting from the linear-frequency-modulated (LFM) systems developed in the 1970s, up to the orthogonal frequency-division multiplexing (OFDM) systems developed in the early 2000s. In mid 2010s, systems were proposed that combined the radar principle with optical solutions developed for imaging and ranging tasks following a hyperspectral embedded systems approach. The idea was to profit on the one side from the possibility offered by RADAR systems to work in harsh environments using emitted radio waves and detect mainly metal objects placed far away (hundreds of meters or even kilometers) from the detection system with positioning spatial resolutions in tens of centimeters, even if there are non-metallic barriers such as e.g. walls in between; and expand this possibility by using optical systems (e.g. light detecting and ranging –LIDAR- systems), using visible light active illumination, capable of generating 2D and 3D images of objects placed at much smaller distances from the detector, but allowing for much higher spatial resolutions (in the millimeter range). To reduce the atmospheric absorption of the emitted active illumination and increase the emitted optical power allowed for these systems that can correctly function even in harsh environments, we propose shifting the active illumination wavelengths from the visible range to the near infra-red (NIR) range, e.g. to 1550 nm. Lacking affordable image sensors fabricated in InGaAs technology, capable of detecting NIR radiation, in this paper we propose a hyperspectral imaging system using a very low power consuming single commercially available InGaAs photodiode to generate 2D images using the single-pixel imaging (SPI) approach based on compressive sensing (CS) and an array of NIR light emitting LEDs, combined with an 80 GHz millimeter band RADAR. The system is conceived to deliver a maximum radar range of 150 m with a maximum spatial resolution of ≤ 5 cm and a RADAR cross-section (RCS) of 10 – 50 m2, combined with an optical system capable of generating 24 fps video streams based on SPI generated images yielding a maximum ranging depth of 10 m with a spatial resolution of < 1 cm. The proposed system will be used in unmanned ground vehicle (UGV) applications enabling decision making in continuous time. The power consumption, dimensions and weight of the hyperspectral ranging system will be adjusted to the UGV targeted applications.

We present a case study for a time-of-flight (ToF) 3D imaging system using single-pixel imaging (SPI) approach based on compressive sensing (CS), accompanied by the Time-of-flight (ToF) principle applied to four reference points of the 2D image created and then mapped to the rest of the SPI generated virtual pixels. In this analysis we have developed a mathematical model of the system and evaluated three different scenarios considering different performance issues based on signal-to-noise ratio, different levels of background illumination, distance, spatial resolution, and material reflectivity presented by the objects in the scene. To be able to reduce the background photon shot noise and enable the correct functionality of the system also in harsh environments (in presence of micrometer size particles such as rain, snow, fog or smoke) we propose using near infra-red (NIR) active illumination with a peak wavelength of 1550 nm. The SPI principle is based here on an array of NIR emitting LEDs and the Thorlab FGA015 InGaAs single photodiode. For the system modelling and analysis, we considered the maximum background illumination intensity of up to 100 klux, different reflection coefficients of the target material to be detected, and measurement distances between 1 and 10 m. Using the ToF principle, we evaluated the direct ToF using both, pulsed laser NIR source as well as an array of NIR emitting LEDs combined with a single InGaAs photodiode on the one side, and an InGaAs single-photon avalanche diode (SPAD) on the other. Using the model developed, we estimated the spatial resolution (standard deviation from the distance measured) the proposed system might reach for each of the ToF methods analyzed and combining different system elements. Finally, we propose a SPI-ToF 3D imaging and ranging system for drone outdoors applications.

In this work, we focus on the problem of feature-based 3D mapping of environments with large textureless regions, which generates sparse 3D maps that may not represent well the mapped scene. To deal with this problem, based on our previous work (Cruz Martinez et al. in 2016 IEEE international symposium on mixed and augmented reality (ISMAR) adjunct proceedings, IEEE, 2016), we propose to enhance sparse 3D maps by using a superpixel-based segmentation with the aim of generating denser 3D maps of the scene in real time. Superpixels are middle-level features, which represent similar regions in an image, which can be connected in order to segment textureless areas. We propose a graphics processor unit architecture for (1) superpixel extraction considering chromatic and depth information, (2) superpixel-based segmentation, generation of connectivity matrix to compute the connected components algorithm and (3) mapping of segmented regions to 3D points. We use the ORB-SLAM system (Mur-Artal et al. in IEEE Trans Robot 31(5):1147–1163, 2015) to generate a sparse 3D map and to project the textureless segments onto it at 27 frames per second. We assessed our approach in terms of segmentation and map quality. Regarding the latter, covered area by the generated map, depth accuracy, and computational performance are reported.

This paper presents an initial framework to extract high-level structures from man-made environments, by means of a novel methodology that combines stereo vision, binary descriptors and parallel processing implemented on a GPU. High-level structures such as planes, spheres and cubes provide vital information of the world, essential to perform applications in the field of robotics, augmented reality and computer vision. However, their extraction involves several computational challenges, especially because their application context requires solving real-time and environment operation constraints. Hence, stereo vision-based attempts have been proposed, without achieving real-time performance because they require a rectification stage running in the frame-to-frame basis, increasing the computational burden. Therefore, in contrast to typical stereo algorithms, the proposed methodology is developed on the basis of a semi-calibrated stereo rig, which means that rectification stage is avoiding, thus enabling to invest computational cost in critical stages and consequently achieving a frame rate up to 50 fps for the whole process.

Binary descriptors have won their place as efficient and effective visual descriptors in several vision tasks. In this context, one of the most widely used binary descriptors to date is the ORB descriptor. ORB is robust against rotation changes, and it uses a learning procedure to generate sampling pairwise tests to construct the descriptor. However, this construction involves a sequential memory access of as many steps as the binary string size. From the latter and motivated by the fact that modern computer vision tasks may require the construction of thousands, if not millions of binary descriptors, we propose to accelerate the construction process of the ORB descriptor via an FPGA-based hardware architecture. The latter is leveraged with a novel arrangement of pairwise tests, which takes advantage of a dual random access memory scheme achieving an acceleration of up to 17 times when compared against the sequential way. The empirical assessment indicates that ORB descriptors obtained from the proposed approach keep a similar performance to that of the original ORB.

In this work, we address the problem of target detection involved in an autonomous landing task for a Micro Aerial Vehicle (MAV). The challenge is to detect a flag located somewhere in the environment. The flag is posed on a pole, and to its right, a landing platform is located. Thus, the MAV has to detect the flag, fly towards it and once it is close enough, locate the landing platform nearby, aiming at centring over it to perform landing; all of this has to be carried out autonomously. In this context, the main problem is the detection of both the flag and the landing platform, whose shapes are known in advanced. Traditional computer vision algorithms could be used; however, the main challenges in this task are the changes in illumination, rotation and scale, and the fact that the flight controller uses the detection to perform the autonomous flight; hence the detection has to be stable and continuous on every camera frame. Motivated by this, we propose to use a Convolutional Neural Network optimised to be run on a small computer with limited computer processing budget. The MAV carries this computer, and it is used to process everything on board. To validate our system, we tested with rotated images, changes in scale and the presence of low illumination. Our method is compared against two conventional computer vision methods, namely, template and feature matching. In addition, we tested our system performance in a wide corridor, executing everything on board the MAV. We achieved a successful detection of the flag with a confidence metric of 0.9386 and 0.9826 for the Landing platform. In total, all the onboard computations ran at an average of 13.01 fps.

In recent years Autonomous Drone Racing has become a significant challenge due to the problems involved in developing an algorithm for autonomous navigation. One of the major problems is the estimation of the camera pose; several approaches can be founded to achieve the estimation of the camera pose. In particular, it is possible to estimates the position based on specific object detection. However, object detection at the same time of navigation entails the problem of a blind spot area when the camera is closest to the object. We propose a methodology that overcomes the blind spot in autonomous navigation based on CNN gate detection to perform pose estimation with a stochastic algorithm for distance estimation. We achieve over 95 % in gate detection and a mean error of around 35 cm in 1D pose estimation into the blind spot zone.