Search

With the rapid growth of video data on the internet, video summarization is becoming a very important AI technology. However, due to the high labelling cost of video summarization, existing studies have to be conducted on small-scale datasets, leading to limited performance and generalization capacity. In this work, we introduce the use of dense video captions as a supervision signal to train video summarization models. Motivated by this, we propose Cap2Sum, a model that learns to summarize videos by generating captions, to exploit dense video caption annotations. This weakly-supervised approach allows us to train the models on large-scale dense video caption datasets to achieve better performance and generalization capacity. To further improve the generalization capacity, we introduce a CLIP (a strong vision-language model) Prior mechanism to enhance the learning of important objects that captions may ignore in the videos. In practice, Cap2Sum can perform zero-shot video summarization or be fine-tuned by the ground-truth summary or video caption of the target dataset. To examine the performance of Cap2Sum after weakly-supervised fine-tuning by the video captions, we propose two new datasets, TVSum-Caption and SumMe-Caption, which are derived from two common video summarization datasets and will be publicly released. We conduct extensive experiments and the results demonstrate that our method achieves significant improvements in performance and generalization capacity compared with previous methods., Comment: 13 pages, 4 figures

Computational neuromorphic imaging (CNI) with event cameras offers advantages such as minimal motion blur and enhanced dynamic range, compared to conventional frame-based methods. Existing event-based radiance field rendering methods are built on neural radiance field, which is computationally heavy and slow in reconstruction speed. Motivated by the two aspects, we introduce Ev-GS, the first CNI-informed scheme to infer 3D Gaussian splatting from a monocular event camera, enabling efficient novel view synthesis. Leveraging 3D Gaussians with pure event-based supervision, Ev-GS overcomes challenges such as the detection of fast-moving objects and insufficient lighting. Experimental results show that Ev-GS outperforms the method that takes frame-based signals as input by rendering realistic views with reduced blurring and improved visual quality. Moreover, it demonstrates competitive reconstruction quality and reduced computing occupancy compared to existing methods, which paves the way to a highly efficient CNI approach for signal processing.

Neuromorphic imaging is a bio-inspired technique that imitates the human retina to sense variations in a dynamic scene. It responds to pixel-level brightness changes by asynchronous streaming events and boasts microsecond temporal precision over a high dynamic range, yielding blur-free recordings under extreme illumination. Nevertheless, such a modality falls short in spatial resolution and leads to a low level of visual richness and clarity. Pursuing hardware upgrades is expensive and might cause compromised performance due to more burdens on computational requirements. Another option is to harness offline, plug-in-play neuromorphic super-resolution solutions. However, existing ones, which demand substantial sample volumes for lengthy training on massive computing resources, are largely restricted by real data availability owing to the current imperfect high-resolution devices, as well as the randomness and variability of motion. To tackle these challenges, we introduce the first self-supervised neuromorphic super-resolution prototype. It can be self-adaptive to per input source from any low-resolution camera to estimate an optimal, high-resolution counterpart of any scale, without the need of side knowledge and prior training. Evaluated on downstream event-driven tasks, such a simple yet effective method can obtain competitive results against the state-of-the-arts, significantly promoting flexibility but not sacrificing accuracy. It also delivers enhancements for inferior natural images and optical micrographs acquired under non-ideal imaging conditions, breaking through the limitations that are challenging to overcome with traditional frame techniques. In the current landscape where the use of high-resolution cameras for event-based sensing remains an open debate, our solution serves as a cost-efficient and practical alternative, paving the way for more intelligent imaging systems., Comment: 11 pages, 13 figures, and 3 tables

The Shack-Hartmann wavefront sensor is widely employed in adaptive optics systems to measure optical aberrations. However, simultaneously achieving high sensitivity and large dynamic range is still challenging, limiting the performance of diagnosing fast-changing turbulence. To overcome this limitation, we propose neuromorphic Shack-Hartmann wave normal sensing (NeuroSH). NeuroSH is a unifying framework that harnesses the computational neuromorphic imaging paradigm to extract the high-dimensional wave normal from temporal diversity measurements. Both numerical analysis and experimental verification demonstrate the feasibility of NeuroSH. To the best of our knowledge, the proposed NeuroSH is the first scheme to surpass the optical dynamic range limitation under challenging dynamic scenarios, thereby advancing ultra-fast turbulence mitigation technology for cutting-edge imagers.

Bio-inspired neuromorphic cameras asynchronously record pixel brightness changes and generate sparse event streams. They can capture dynamic scenes with little motion blur and more details in extreme illumination conditions. Due to the multidimensional address-event structure, most existing vision algorithms cannot properly handle asynchronous event streams. While several event representations and processing methods have been developed to address such an issue, they are typically driven by a large number of events, leading to substantial overheads in runtime and memory. In this paper, we propose a new graph representation of the event data and couple it with a Graph Transformer to perform accurate neuromorphic classification. Extensive experiments show that our approach leads to better results and excels at the challenging realistic situations where only a small number of events and limited computational resources are available, paving the way for neuromorphic applications embedded into mobile facilities., Comment: 15 pages, 4 figures, and 7 tables. Accepted by Elsevier Neurocomputing

Neuromorphic imaging reacts to per-pixel brightness changes of a dynamic scene with high temporal precision and responds with asynchronous streaming events as a result. It also often supports a simultaneous output of an intensity image. Nevertheless, the raw events typically involve a large amount of noise due to the high sensitivity of the sensor, while capturing fast-moving objects at low frame rates results in blurry images. These deficiencies significantly degrade human observation and machine processing. Fortunately, the two information sources are inherently complementary -- events with microsecond-level temporal resolution, which are triggered by the edges of objects recorded in a latent sharp image, can supply rich motion details missing from the blurry one. In this work, we bring the two types of data together and introduce a simple yet effective unifying algorithm to jointly reconstruct blur-free images and noise-robust events in an iterative coarse-to-fine fashion. Specifically, an event-regularized prior offers precise high-frequency structures and dynamic features for blind deblurring, while image gradients serve as a kind of faithful supervision in regulating neuromorphic noise removal. Comprehensively evaluated on real and synthetic samples, such a synergy delivers superior reconstruction quality for both images with severe motion blur and raw event streams with a storm of noise, and also exhibits greater robustness to challenging realistic scenarios such as varying levels of illumination, contrast and motion magnitude. Meanwhile, it can be driven by much fewer events and holds a competitive edge at computational time overhead, rendering itself preferable as available computing resources are limited. Our solution gives impetus to the improvement of both sensing data and paves the way for highly accurate neuromorphic reasoning and analysis., Comment: 16 pages, 19 figures, and 3 tables. Accepted by IEEE Transactions on Image Processing

Phase recovery (PR) refers to calculating the phase of the light field from its intensity measurements. As exemplified from quantitative phase imaging and coherent diffraction imaging to adaptive optics, PR is essential for reconstructing the refractive index distribution or topography of an object and correcting the aberration of an imaging system. In recent years, deep learning (DL), often implemented through deep neural networks, has provided unprecedented support for computational imaging, leading to more efficient solutions for various PR problems. In this review, we first briefly introduce conventional methods for PR. Then, we review how DL provides support for PR from the following three stages, namely, pre-processing, in-processing, and post-processing. We also review how DL is used in phase image processing. Finally, we summarize the work in DL for PR and outlook on how to better use DL to improve the reliability and efficiency in PR. Furthermore, we present a live-updating resource (https://github.com/kqwang/phase-recovery) for readers to learn more about PR., Comment: 82 pages, 32 figures

Tuning the interaction between the bulk and edge states of topological materials is a powerful tool for manipulating edge transport behavior, opening up exciting opportunities for novel electronic and spintronic applications. This approach is particularly suited to topological crystalline insulators (TCI), a class of topologically nontrivial compounds that are endowed with multiple degrees of topological protection. In this study, we investigate how bulk-edge interactions can influence the edge transport in planar bismuthene, a TCI with metallic edge states protected by in-plane mirror symmetry, using first principles calculations and symmetrized Wannier tight-binding models. By exploring the impact of various perturbation effects, such as device size, substrate potentials, and applied transverse electric field, we examine the evolution of the electronic structure and edge transport in planar bismuthene. Our findings demonstrate that the TCI states of planar bismuthene can be engineered to exhibit either a gapped or conducting unconventional helical spin texture via a combination of substrate and electric field effects. Furthermore, under strong electric fields, the edge states can be stabilized through a delicate control of the bulk-edge interactions. These results open up new directions for discovering novel spin transport patterns in topological materials and provide critical insights for the fabrication of topological spintronic devices., Comment: 23 pages, 8 figures

With its monoelemental composition, various crystalline forms and an inherently strong spin-orbit coupling, bismuth has been regarded as an ideal prototype material to expand our understanding of topological electronic structures. In particular, two-dimensional bismuth thin films have attracted a growing interest due to potential applications in topological transistors and spintronics. This calls for an effective physical model to give an accurate interpretation of the novel topological phenomena shown by two-dimensional bismuth. However, the conventional semi-empirical approach of adapting bulk bismuth hoppings fails to capture the topological features of two-dimensional bismuth allotropes because the electronic band topology is heavily influenced by crystalline symmetries as well as atom spacings. Here we provide a new parameterization using localized Wannier functions derived from the Bloch states in first-principles calculations. We construct new tight-binding models for three types of two-dimensional bismuth allotropes: a Bi (111) bilayer, bismuthene and a Bi(110) bilayer. We demonstrate that our tight-binding models can successfully reproduce the band structures, symmetries and topological features of these two-dimensional allotropes. We anticipate that these models can be extended to other similar two-dimensional topological structures such as antimonene and arsenene. Moreover, these models can serve as a starting point for investigating the electron/spin transport and electromagnetic response in low-dimensional topological devices., Comment: 20 pages, 6 figures

Layered indium selenides ($In_{2}Se_{3}$) have recently been discovered to host robust out-of-plane and in-plane ferroelectricity in the $\alpha$ and $\beta$' phases, respectively. In this work, we utilise angle-resolved photoelectron spectroscopy to directly measure the electronic bandstructure of $\beta '-In_{2}Se_{3}$, and compare to hybrid density functional theory (DFT) calculations. In agreement with DFT, we find the band structure is highly two-dimensional, with negligible dispersion along the c-axis. Due to n-type doping we are able to observe the conduction band minima, and directly measure the minimum indirect (0.97 eV) and direct (1.46 eV) bandgaps. We find the Fermi surface in the conduction band is characterized by anisotropic electron pockets with sharp in-plane dispersion about the $\overline{M}$ points, yielding effective masses of 0.21 $m_{0}$ along $\overline{KM}$ and 0.33 $m_{0}$ along $\overline{\Gamma M}$. The measured band structure is well supported by hybrid density functional theory calculations. The highly two-dimensional (2D) bandstructure with moderate bandgap and small effective mass suggest that $\beta'-In_{2}Se_{3}$ is a potentially useful new van der Waals semiconductor. This together with its ferroelectricity makes it a viable material for high-mobility ferroelectric-photovoltaic devices, with applications in non-volatile memory switching and renewable energy technologies., Comment: 19 pages, 4 + 1 figures; typos corrected;added references; updated figures & discussion to reflect changes in model