Search

To achieve fast location, precise tracking and accurate identification over a large field of view (FOV), we have proposed a heterogeneous compound eye camera (HeCECam), which consists of a heterogeneous compound eye array, an optical relay system and a CMOS detector. However, the current HeCECam can hardly acquire high-precision 3D information of the targets to realize these applications. To solve this challenge, we propose a scheme on optimizing the structure of the HeCECam to improving the detection performance, including the optimization of the distribution uniformity of the sub-eyes with the proposed "Three-direction center-of-gravity subdivision (TGS)" and the enhancement of the compatibility between heterogeneous compound eyes and the optical relay system with the proposed compensation method for tilt. The TGS significantly reduces the distribution unevenness of sub-eyes down to 117% from the previous 152%, and provides symmetry to the heterogeneous compound eye array. The tilt compensation effectively addresses previous imaging defects, such as distortion of sub-images, increased stray light, and support structures being imaged, and it improves the imaging clarity of the system, especially in external FOV. Based on two proposed methods, we re-design and fabricate the heterogeneous compound eye array to obtain a high-performance prototype. To verify the imaging capacities of the optimized HeCECam, a series of comparison experiments are performed, including blank scene imaging, FOV tests, resolution verification and real-world scene imaging. The results show that the previous imaging defects have been well eliminated, and the optimized prototype has stronger resolving power and wider FOV. This allow the HeCECam to perform better in subsequent practical applications, such as wide-area surveillance, forewarning, and navigation.

Image-guided and robotic surgery based on endoscopic imaging technologies can enhance cancer treatment by ideally removing all cancerous tissue and avoiding iatrogenic damage to healthy tissue. Surgeons evaluate the tumor margins at the cost of impeding surgical workflow or working with dimmed surgical illumination, since current endoscopic imaging systems cannot simultaneous and real-time color and near-infrared (NIR) fluorescence imaging under normal surgical illumination. To overcome this problem, a bio-inspired multimodal 3D endoscope combining the excellent characteristics of human eyes and compound eyes of mantis shrimp is proposed. This 3D endoscope, which achieves simultaneous and real-time imaging of three-dimensional stereoscopic, color, and NIR fluorescence, consists of three parts: a broad-band binocular optical system like as human eye, an optical relay system, and a multiband sensor inspired by the mantis shrimp's compound eye. By introducing an optical relay system, the two sub-images after the broad-band binocular optical system can be projected onto one and the same multiband sensor. A series of experiments demonstrate that this bio-inspired multimodal 3D endoscope not only provides surgeons with real-time feedback on the location of tumor tissue and lymph nodes but also creates an immersive experience for surgeons without impeding surgical workflow. Its excellent characteristics and good scalability can promote the further development and application of image-guided and robotic surgery.