Search

Due to the severe distortion of the fisheye image, the rectangular bounding box contain a lot of invalid information. So the multi-point representation method are emerging. However, it will fail in some extreme cases especially when the center of the object is not in the instance. In this work, we propose a Convex Hull Regression Strategy for people detection on top-view fisheye images. It replaces the instance with its convex hull to solve the above challenging issue and can be pre-trained on regular datasets without additional processing. In addition, the mosaic and mixup data augmentation methods that perform well under rectangular boxes are applied to our representation. Finally, we improve the label assignment and propose a more reasonable loss function called PDIoU loss so as to focus on the overall IoU between ground truth polygon and predicted polygon. Experimental results demonstrate that our method outperforms state-of-the-art algorithms. Source code is available at https://github.com/xiaoxuebajie/CHRS. [ABSTRACT FROM AUTHOR]

Safe Micro Aerial Vehicle (MAV) navigation requires detecting and avoiding obstacles. For safe MAV navigation, expansion‐based algorithms are effective for detecting obstacles. However, accurate and real‐time obstacle detection is a fundamental challenge. Some traditional methods focus on extracting geometric features from images and applying geometric constraints to identify potential obstacles. Others may leverage machine learning algorithms for object detection and classification, using features such as texture, shape, and context to distinguish obstacles from background clutter. The choice of approach depends on factors such as the specific requirements of the application, the complexity of the scene, and the available computational resources. Since obstacles, in reality, take the form of objects (e.g., persons, walls, pillars, trees, automobiles, and other structures), it is preferable to represent them according to human comprehension and as objects. Therefore, the objective of this study is to reflect on the previous research and address the issues mentioned above by extracting objects from the fisheye image using a panoptic deep‐learning network. The extracted object regions are, then, used to identify obstacles with a novel area‐based expansion rate we developed in a previous study. We compared the accuracy of obstacle detection in our proposed method to the existing method when moving forward and to the right; thus, we improved it between 10% and 18%, respectively. In addition, compared with the existing method, and due to replacing a single object with multiple regions, obstacle‐detection runtime for forward and right direction is 15.71 and 25.5 times faster, respectively, and the required match points have decreased by 49% and 55%. [ABSTRACT FROM AUTHOR]

Images taken by fisheye cameras are characterized by wide field of view, geometric distortion and large scale variance, which bring great challenges to object detectors based on general convolutional networks. Existing object detection algorithms can be further improved with respect to network structure design, feature learning to be applicable to the distorted object detection task on fisheye images. To mitigate the effect of radial distortion on fisheye images, a multi-head attention module with multi-branch stacking structure is used in the YOLOv7 backbone to capture global contextual information. Meanwhile, a simple and efficient layer aggregation structure combining deformable convolutions is used on the Neck side of YOLOv7 to achieve effective multi-scale feature fusion. Experiments are conducted on the public comprehensive fisheye image dataset VOC_360, and the results show that the improved YOLOv7 fisheye image object detector effectively achieves detection accuracy of 84.3% and 70.4% for mAP50 and mAP50:95, respectively, which is 3.1 percentage points and 6.4 percentage points higher than the baseline model YOLOv7, respectively. [ABSTRACT FROM AUTHOR]

Fisheye cameras play a crucial role in various fields by offering a wide field of view, enabling the capture of expansive areas within a single frame. Nonetheless, the radial distortion characteristics of fisheye lenses lead to notable shape deformation, particularly at the edges of the image, posing a significant challenge for accurate object detection. In this paper, we introduce a novel method, 'VP-aided fine-tuning', which harnesses the strengths of the pretraining–fine-tuning paradigm augmented by visual prompting (VP) to bridge the domain gap between undistorted standard datasets and distorted fisheye image datasets. Our approach involves two key elements: the use of VPs to effectively adapt a pretrained model to the fisheye domain, and a detailed 24-point regression of objects to fit the unique distortions of fisheye images. This 24-point regression accurately defines the object boundaries and substantially reduces the impact of environmental noise. The proposed method was evaluated against existing object detection frameworks on fisheye images, demonstrating superior performance and robustness. Experimental results also showed performance improvements with the application of VP, regardless of the variety of fine-tuning method applied. [ABSTRACT FROM AUTHOR]

Fisheye camera is an important sensor in on-board systems and surveillance systems. However, the current fisheye image pedestrian detection still exists problems such as large distortions are difficult to detect, sparse datasets, and poor real-time performance. We propose an adversarial pedestrian detection model based on virtual fisheye image training. The fisheye transformation model is proposed to generate virtual image dataset, which provides a complete data base for network training. The DEMO-YOLO network is used to ensure real-time performance while improving accuracy, and the spatial transformation network is introduced to convert the features extracted from the backbone network into hard examples. In the process of training, adversarial learning is used so that the network is able to deal with objects with large deformations. The model is trained on the fisheye-transformed CityPersons and Kitti datasets and tested on the real fisheye dataset Woodscape. The experimental results show that compared with the traditional detection methods, the accuracy of the proposed model is improved by 18.45% and the real-time performance reaches an average of 318 frames/s. Therefore, the model can be applied to fisheye cameras. [ABSTRACT FROM AUTHOR]

Recently, object detection has become a research hotspot in computer vision, which often detects regular images with small viewing angles. In order to obtain a field of view without blind spots, fisheye cameras, which have distortions and discontinuities, have come into use. The fisheye camera, which has a wide viewing angle, and an unmanned aerial vehicle equipped with a fisheye camera are used to obtain a field of view without blind spots. However, distorted and discontinuous objects appear in the captured fisheye images due to the unique viewing angle of fisheye cameras. It poses a significant challenge to some existing object detectors. To solve this problem, this paper proposes a PGDS-YOLOv8s model to solve the issue of detecting distorted and discontinuous objects in fisheye images. First, two novel downsampling modules are proposed. Among them, the Max Pooling and Ghost's Downsampling (MPGD) module effectively extracts the essential feature information of distorted and discontinuous objects. The Average Pooling and Ghost's Downsampling (APGD) module acquires rich global features and reduces the feature loss of distorted and discontinuous objects. In addition, the proposed C2fs module uses Squeeze-and-Excitation (SE) blocks to model the interdependence of the channels to acquire richer gradient flow information about the features. The C2fs module provides a better understanding of the contextual information in fisheye images. Subsequently, an SE block is added after the Spatial Pyramid Pooling Fast (SPPF), thus improving the model's ability to capture features of distorted, discontinuous objects. Moreover, the UAV-360 dataset is created for object detection in fisheye images. Finally, experiments show that the proposed PGDS-YOLOv8s model on the VOC-360 dataset improves mAP@0.5 by 19.8% and mAP@0.5:0.95 by 27.5% compared to the original YOLOv8s model. In addition, the improved model on the UAV-360 dataset achieves 89.0% for mAP@0.5 and 60.5% for mAP@0.5:0.95. Furthermore, on the MS-COCO 2017 dataset, the PGDS-YOLOv8s model improved AP by 1.4%, AP 50 by 1.7%, and AP 75 by 1.2% compared with the original YOLOv8s model. [ABSTRACT FROM AUTHOR]

Fisheye cameras play a crucial role in various fields by offering a wide field of view, enabling the capture of expansive areas within a single frame. Nonetheless, the radial distortion characteristics of fisheye lenses lead to notable shape deformation, particularly at the edges of the image, posing a significant challenge for accurate object detection. In this paper, we introduce a novel method, ‘VP-aided fine-tuning’, which harnesses the strengths of the pretraining–fine-tuning paradigm augmented by visual prompting (VP) to bridge the domain gap between undistorted standard datasets and distorted fisheye image datasets. Our approach involves two key elements: the use of VPs to effectively adapt a pretrained model to the fisheye domain, and a detailed 24-point regression of objects to fit the unique distortions of fisheye images. This 24-point regression accurately defines the object boundaries and substantially reduces the impact of environmental noise. The proposed method was evaluated against existing object detection frameworks on fisheye images, demonstrating superior performance and robustness. Experimental results also showed performance improvements with the application of VP, regardless of the variety of fine-tuning method applied.

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Recently, object detection has become a research hotspot in computer vision, which often detects regular images with small viewing angles. In order to obtain a field of view without blind spots, fisheye cameras, which have distortions and discontinuities, have come into use. The fisheye camera, which has a wide viewing angle, and an unmanned aerial vehicle equipped with a fisheye camera are used to obtain a field of view without blind spots. However, distorted and discontinuous objects appear in the captured fisheye images due to the unique viewing angle of fisheye cameras. It poses a significant challenge to some existing object detectors. To solve this problem, this paper proposes a PGDS-YOLOv8s model to solve the issue of detecting distorted and discontinuous objects in fisheye images. First, two novel downsampling modules are proposed. Among them, the Max Pooling and Ghost’s Downsampling (MPGD) module effectively extracts the essential feature information of distorted and discontinuous objects. The Average Pooling and Ghost’s Downsampling (APGD) module acquires rich global features and reduces the feature loss of distorted and discontinuous objects. In addition, the proposed C2fs module uses Squeeze-and-Excitation (SE) blocks to model the interdependence of the channels to acquire richer gradient flow information about the features. The C2fs module provides a better understanding of the contextual information in fisheye images. Subsequently, an SE block is added after the Spatial Pyramid Pooling Fast (SPPF), thus improving the model’s ability to capture features of distorted, discontinuous objects. Moreover, the UAV-360 dataset is created for object detection in fisheye images. Finally, experiments show that the proposed PGDS-YOLOv8s model on the VOC-360 dataset improves mAP@0.5 by 19.8% and mAP@0.5:0.95 by 27.5% compared to the original YOLOv8s model. In addition, the improved model on the UAV-360 dataset achieves 89.0% for mAP@0.5 and 60.5% for mAP@0.5:0.95. Furthermore, on the MS-COCO 2017 dataset, the PGDS-YOLOv8s model improved AP by 1.4%, AP50 by 1.7%, and AP75 by 1.2% compared with the original YOLOv8s model.

This study evaluated the applicability of a shading matrix generated using hemispherical fisheye images to select the optimal parking space to maximize the photovoltaic (PV) power generation of solar electric vehicles. The study aims to identify the maximum power point (MPP) of a PV module in a parking space using current–voltage measuring equipment to determine whether the shading matrix reflects any shadow phenomenon. The experiment was conducted on a parking lot at Daeyeon Campus. The fisheye images were taken in seven parking spaces. The shading matrices were generated by tracking the position of the sun over 12 months and 24 h periods. A 50 W PV module was installed on the roof of the vehicle and the MPP was measured in five experiments (9 a.m., 11 a.m., 13 p.m., 15 p.m., and 17 p.m.) in seven parking spaces. In parking spaces with a shading factor of 0, the PV modules were not shaded at all during the experiment, and the average MPPs were 34, 43, 37, and 21 W at 9 a.m., 11 a.m., 13 p.m., and 15 p.m., respectively. The shading matrix calculated using the fisheye images accurately reflected the shadow phenomenon that decreases PV power generation. [ABSTRACT FROM AUTHOR]

The height of standing trees is an important index in forestry research. This index is not only hard to measure directly but also the environmental factors increase the measurement difficulty. Therefore, the measurement of the height of standing trees is always a problem that experts and scholars are trying to improve. In this study, improve fuzzy c-means algorithm to reduce the calculation time and improve the clustering effect, used on this image segmentation technology, a highly robust non-contact measuring method for the height of standing trees was proposed which is based on a smartphone with a fisheye lens. While ensuring the measurement accuracy, the measurement stability is improved. This method is simple to operate, just need to take a picture of the standing tree and determine the shooting distance to complete the measurement. The purpose of the fisheye lens is to ensure that the tree remains intact in the photograph and to reduce the shooting distance. The results of different stability experiments show that the measurement error ranged from -0.196m to 0.195m, and the highest relative error of tree measurement was 3.05%, and the average relative error was 1.45%. Analysis shows that compared with previous research, this method performs better at all stages. The proposed approach can provide a new way to obtain tree height, which can be used to analyze growing status and change in contrast height because of high accuracy and permanent preservation of images.

Conventional cameras and fisheye cameras are often used together to capture clear target images and large scene background images in many applications, such as mobile robotic telepresence systems and large scene monitoring systems. In this paper, we propose to stitch images from these cameras for offering remote operators a large field of view to perceive a local environment. To provide a clear view of targets for face-to-face communication and a complete view of a robot's surroundings for safe teleoperation of the robot, we stitch these images by keeping the original conventional image. The image stitching is formulated as a nonrigid motion estimation problem and images are stitched based on nonrigid warping, e.g., the thin-plate spline. To improve the algorithmic efficiency of image stitching, we exploit a region-based point correspondence selection method to reduce the number of point correspondences that are used for thin-plate spline interpolation. The experiments conducted on collected images and images captured from a telepresence system show the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Tree height is an essential indicator in forestry research. This indicator is difficult to measure directly, as well as wind disturbance adds to the measurement difficulty. Therefore, tree height measurement has always been an issue that experts and scholars strive to improve. We propose a tree height measurement method based on tree fisheye images to improve the accuracy of tree height measurements. Our aim is to extract tree height extreme points in fisheye images by proposing an improved lightweight target detection network YOLOX-tiny. We added CBAM attention mechanism, transfer learning, and data enhancement methods to improve the recall rate, F1 score, AP, and other indicators of YOLOX-tiny. This study improves the detection performance of YOLOX-tiny. The use of deep learning can improve measurement efficiency while ensuring measurement accuracy and stability. The results showed that the highest relative error of tree measurements was 4.06% and the average relative error was 1.62%. The analysis showed that the method performed better at all stages than in previous studies. [ABSTRACT FROM AUTHOR]

Thanks to the wide view field, the fisheye camera can get much more visual information. Thus, it is widely used in the field of computer vision. However, projection is often required for fisheye images to be used for object detection. Meanwhile, the projection will lead to distortion in fisheye images, and the discontinuous image edges will make the objects incomplete. Fisheye images are characterized by objects that are large near and small far. These problems are still challenges for the existing advanced object detector YOLOv7. Therefore, in this paper, we propose an improved YOLOv7 model. First, Modulated Deformable Convolution is introduced into the YOLOv7 model to automatically adapt to distortion changes of distorted objects in fisheye images. It not only adjusts the sampling position of the convolutional kernel but also further extends the deformation range. The improved model can efficiently extract features of distorted and edge-discontinuous objects. In addition, fisheye images are characterized by objects close to the fisheye lens being large, while objects farther away from the fisheye lens will be smaller. To further optimize the detection performance of small objects in fisheye images, Swin Transformer is also introduced into the YOLOv7 model, and Swin Transformer Block with Window Multi-head Self-Attention (W-MSA) Effectively enhances Network Local Perception. Finally, our proposed model achieves up to 2.4% improvement in mAP compared to the original YOLOv7 model on the ERP-360 dataset. Also, the proposed model achieves the best results compared to other state-of-the-art object detection methods for equirectangular projection images. On the VOC-360 dataset, our proposed model improves the mAP by up to 5.9% compared to the original YOLOv7 model. The experimental results show that the proposed models achieve good results for object detection in both fisheye images and equirectangular projection images. The ERP-360 dataset, source code and pre-trained models for related tasks can be found at https://github.com/xiaoxiaomichong/ERP-360dataset. • Modulated Deformable Convolution is introduced into YOLOv7 to effectively detect distorted objects in fisheye images. • Swin Transformer Block is introduced into YOLOv7 to effectively recognize small objects in fisheye images. • A SMDC-YOLOv7 model is proposed which can improve the overall detection performance of objects in fisheye images. [ABSTRACT FROM AUTHOR]

Tree height is an essential indicator in forestry research. This indicator is difficult to measure directly, as well as wind disturbance adds to the measurement difficulty. Therefore, tree height measurement has always been an issue that experts and scholars strive to improve. We propose a tree height measurement method based on tree fisheye images to improve the accuracy of tree height measurements. Our aim is to extract tree height extreme points in fisheye images by proposing an improved lightweight target detection network YOLOX-tiny. We added CBAM attention mechanism, transfer learning, and data enhancement methods to improve the recall rate, F1 score, AP, and other indicators of YOLOX-tiny. This study improves the detection performance of YOLOX-tiny. The use of deep learning can improve measurement efficiency while ensuring measurement accuracy and stability. The results showed that the highest relative error of tree measurements was 4.06% and the average relative error was 1.62%. The analysis showed that the method performed better at all stages than in previous studies.

This study presents a methodology for assessing the potential solar resource available at solar-powered parking machines installed in a typical city center. It is based on capturing fisheye images through an appropriate camera/lens. These images are then processed to classify the sky and non-sky pixels using a semi-automated sky detection method, also developed in this study. The processed images are then used to calculate the diffuse and beam solar potential of the site by estimating the visible fraction of the sky and the hours of the year during which the sun can be seen from the site, respectively. For the case study, 15 existing parking machines were analyzed in the central business district of Auckland (New Zealand). It was found that most of these machines were able to capture less than 50% of the available solar potential. Only two machines crossed the yearly irradiation threshold that represents the criterion for solar panel installation. If modern high-power solar-powered machines are used, selecting suitable locations for installation is crucial. • Method for assessing solar potential at the parking machines has been developed • It is based on capturing fisheye images and using semi-automated sky detection method • 15 parking machines installed on the sidewalk in a city center were analyzed • Most of the machines were able to receive 15–50% of the available solar potential • Assessment is crucial for the viability of modern solar-powered parking machines [ABSTRACT FROM AUTHOR]

For distortion problem of fisheye images collected by video monitoring system of coal mine, an automatic obtaining method of adjustment parameters was put forward based on image content according to line constrain of ideal spherical perspective projection model. The method detects image edge characteristic lines by preprocessing the fisheye distortion image using math morphologic and random Hough transform, then gets the optimal adjustment parameters through the statistic of horizontal and vertical lines proportion. In order to decrease effect of large image information on adjustment parameter obtaining, the ideal spherical perspective projection model was improved and an improved adjustment method of fisheye distortion images was proposed, which realized center transfer of the fisheye distortion image and magnifying of part regions. The experimental results show the method can get a better adjustment effect of fisheye distortion image.

The Gaussian kernel, its partial derivatives and the Laplacian kernel, applied at different image scales, play a very important role in image processing and in feature extraction from images. Although they have been extensively studied in the case of images acquired by projective cameras, this is not the case for cameras with fisheye lenses. This type of cameras is becoming very popular, since it exhibits a Field of View of 180 degrees. The model of fisheye image formation differs substantially from the simple projective transformation, causing straight lines to be imaged as curves. Thus the traditional kernels used for processing images acquired by projective cameras, are not optimal for fisheye images. This work uses the calibration of the acquiring fisheye camera to define a geodesic metric for distance between pixels in fisheye images and subsequently redefines the Gaussian kernel, its partial derivatives, as well as the Laplacian kernel. Finally, algorithms for applying in the spatial domain these kernels, as well as the Harris corner detector, are proposed, using efficient computational implementations. Comparative results are shown, in terms of correctness of image processing, efficiency of application for multi scale processing, as well as salient point extraction. Thus we conclude that the proposed algorithms allow the efficient application of standard processing and analysis techniques of fisheye images, in the spatial domain, once the calibration of the specific camera is available.

Crowd monitoring in public has become an important and challenging topic in the field of automatic surveillance system. Existing approaches for crowd monitoring cannot obtain desired results in dense crowd places, which have the limitation of small field-of-view. This paper proposes a novel approach for dense crowd counting in public using a fisheye camera. To segment the crowd from the background, the methods of setting region of interest and subtracting background are firstly proposed. In the process of feature extraction, a perspective weight model is generated according to the perspective of the fisheye image, and a novel mathematical morphology method for fisheye images is developed for feature enhancement. Some features for describing the crowd number are extracted based on the proposed model and the novel mathematical morphology method. Finally, the crowd is counted using three different classifiers including the multiple linear regression, the backpropagation neural network and the regression trees for comparison based on the extracted features. The experimental results show that the proposed approach is effective for dense crowd counting in fisheye images. [ABSTRACT FROM AUTHOR]

The fisheye lens has the characteristics of wide viewing angle and low overall cost. Panoramic images can be further generated based on fisheye camera. In this paper, we propose a high-quality fisheye image stitching algorithm(LLBI-AW). In correction part, traditional bilinear interpolation algorithm leads to edge blur, a latitude and longitude-based interpolation algorithm (LLBI) is proposed to address this problem. Direction of the target point 0.is introduced through the latitude and longitude, weights are determined by the distance which between each point and the direction, and then performs interpolation. In fusion part, for the ghost problem in the overlapped area of the image after stitching, a weighted fusion algorithm (AW) based on the arc function is proposed. We establish an arc function to determine weights of left and right images, and then perform fusion calculation. The experimental results show that the information entropy, standard deviation and Brenner gradient of LLBI-AW algorithm are improved compared with the traditional algorithm, and the central region of the stitching image has a natural transition and high image quality. [ABSTRACT FROM AUTHOR]

Affine-scale invariant feature transform (ASIFT) has performed very well for perspective images, but the intrinsic nonlinear distortion makes it difficult to utilize ASIFT directly for fisheye images to find a large amount of correspondences. We reuse ASIFT and propose a new feature matching method for uncorrected fisheye images. First, we employ the Gaussian hemi-image to divide fisheye images into patches. In this process, we introduce Plücker coordinates and side operator to establish the correspondences between the hemispherical projection model, Gaussian hemi-image, and fisheye images. Simultaneously, maximum stable extreme region algorithm is used to detect the target regions. After that, each patch in these regions is simulated at different orientation parameters in ASIFT, and SIFT algorithm is applied to all simulated patches. Experiments on real-world images show that the proposed method can achieve good performance: the numbers of the point correspondences increase greatly with satisfactory accuracy, reliability, and efficiency. [ABSTRACT FROM AUTHOR]

The height of standing trees is an important index in forestry research. This index is not only hard to measure directly but also the environmental factors increase the measurement difficulty. Therefore, the measurement of the height of standing trees is always a problem that experts and scholars are trying to improve. In this study, improve fuzzy c-means algorithm to reduce the calculation time and improve the clustering effect, used on this image segmentation technology, a highly robust non-contact measuring method for the height of standing trees was proposed which is based on a smartphone with a fisheye lens. While ensuring the measurement accuracy, the measurement stability is improved. This method is simple to operate, just need to take a picture of the standing tree and determine the shooting distance to complete the measurement. The purpose of the fisheye lens is to ensure that the tree remains intact in the photograph and to reduce the shooting distance. The results of different stability experiments show that the measurement error ranged from -0.196m to 0.195m, and the highest relative error of tree measurement was 3.05%, and the average relative error was 1.45%. Analysis shows that compared with previous research, this method performs better at all stages. The proposed approach can provide a new way to obtain tree height, which can be used to analyze growing status and change in contrast height because of high accuracy and permanent preservation of images.

We propose a fisheye camera calibration algorithm, in which the center of the distortion and equivalent focal length of the fisheye camera are obtained by the contour constraint, and the distortion parameters are calculated by linear constraint and Levenberg-Marquardt (LM) optimization algorithm. According to the obtained camera parameters, the fisheye image can be corrected by cubic-box-mapping-method (CBMM). The result shows that for checkerboard images the calibration algorithm demonstrates a high-accuracy of about 0.6727 pixels in the average reprojection root-mean-square error (RMSE). Based on the calibration algorithm, the ream-time fisheye video correction is implemented on an android smart-phone. In the implementation, a direct mapping method is used to simplify the cubic box mapping method, and a heterogeneous architecture with a graphics processing unit (GPU) is built to accelerate the algorithm. Lastly, a fisheye video stream of 1280×1280 resolution is timely corrected on the smart-phone with a speed of 25 frames/sec. [ABSTRACT FROM AUTHOR]

We present two new fisheye image datasets for training object and face detection models: VOC-360 and Wider-360. The fisheye images are created by post-processing regular images collected from two well-known datasets, VOC2012 and Wider Face, using a model for mapping regular to fisheye images implemented in Matlab. VOC-360 contains 39,575 fisheye images for object detection, segmentation, and classification. Wider-360 contains 63,897 fisheye images for face detection. These datasets will be useful for developing face and object detectors as well as segmentation modules for fisheye images while the efforts to collect and manually annotate true fisheye images are underway., (© 2019 The Author(s).)