Search

Light detection and ranging (LiDAR) is a ubiquitous tool to provide precise spatial awareness in various perception environments. A bionic LiDAR that can mimic human-like vision by adaptively gazing at selected regions of interest within a broad field of view is crucial to achieve high-resolution imaging in an energy-saving and cost-effective manner. However, current LiDARs based on stacking fixed-wavelength laser arrays and inertial scanning have not been able to achieve the desired dynamic focusing patterns and agile scalability simultaneously. Moreover, the ability to synchronously acquire multi-dimensional physical parameters, including distance, direction, Doppler, and color, through seamless fusion between multiple sensors, still remains elusive in LiDAR. Here, we overcome these limitations and demonstrate a bio-inspired frequency-modulated continuous wave (FMCW) LiDAR system with dynamic and scalable gazing capability. Our chip-scale LiDAR system is built using hybrid integrated photonic solutions, where a frequency-chirped external cavity laser provides broad spectral tunability, while on-chip electro-optic combs with elastic channel spacing allow customizable imaging granularity. Using the dynamic zoom-in capability and the coherent FMCW scheme, we achieve a state-of-the-art resolution of 0.012 degrees, providing up to 15 times the resolution of conventional 3D LiDAR sensors, with 115 equivalent scanning lines and 4D parallel imaging. We further demonstrate cooperative sensing between our adaptive coherent LiDAR and a camera to enable high-resolution color-enhanced machine vision.

Teasers are an effective tool for promoting content in entertainment, commercial and educational fields. However, creating an effective teaser for long videos is challenging for it requires long-range multimodal modeling on the input videos, while necessitating maintaining audiovisual alignments, managing scene changes and preserving factual accuracy for the output teasers. Due to the lack of a publicly-available dataset, progress along this research direction has been hindered. In this work, we present DocumentaryNet, a collection of 1,269 documentaries paired with their teasers, featuring multimodal data streams of video, speech, music, sound effects and narrations. With DocumentaryNet, we propose a new two-stage system for generating teasers from long documentaries. The proposed TeaserGen system first generates the teaser narration from the transcribed narration of the documentary using a pretrained large language model, and then selects the most relevant visual content to accompany the generated narration through language-vision models. For narration-video matching, we explore two approaches: a pretraining-based model using pretrained contrastive language-vision models and a deep sequential model that learns the mapping between the narrations and visuals. Our experimental results show that the pretraining-based approach is more effective at identifying relevant visual content than directly trained deep autoregressive models.

Recent years have seen many audio-domain text-to-music generation models that rely on large amounts of text-audio pairs for training. However, symbolic-domain controllable music generation has lagged behind partly due to the lack of a large-scale symbolic music dataset with extensive metadata and captions. In this work, we present MetaScore, a new dataset consisting of 963K musical scores paired with rich metadata, including free-form user-annotated tags, collected from an online music forum. To approach text-to-music generation, we leverage a pretrained large language model (LLM) to generate pseudo natural language captions from the metadata. With the LLM-enhanced MetaScore, we train a text-conditioned music generation model that learns to generate symbolic music from the pseudo captions, allowing control of instruments, genre, composer, complexity and other free-form music descriptors. In addition, we train a tag-conditioned system that supports a predefined set of tags available in MetaScore. Our experimental results show that both the proposed text-to-music and tags-to-music models outperform a baseline text-to-music model in a listening test, while the text-based system offers a more natural interface that allows free-form natural language prompts.

Schenkerian Analysis (SchA) is a uniquely expressive method of music analysis, combining elements of melody, harmony, counterpoint, and form to describe the hierarchical structure supporting a work of music. However, despite its powerful analytical utility and potential to improve music understanding and generation, SchA has rarely been utilized by the computer music community. This is in large part due to the paucity of available high-quality data in a computer-readable format. With a larger corpus of Schenkerian data, it may be possible to infuse machine learning models with a deeper understanding of musical structure, thus leading to more "human" results. To encourage further research in Schenkerian analysis and its potential benefits for music informatics and generation, this paper presents three main contributions: 1) a new and growing dataset of SchAs, the largest in human- and computer-readable formats to date (>140 excerpts), 2) a novel software for visualization and collection of SchA data, and 3) a novel, flexible representation of SchA as a heterogeneous-edge graph data structure.

The ''pretraining-and-finetuning'' paradigm has become a norm for training domain-specific models in natural language processing and computer vision. In this work, we aim to examine this paradigm for symbolic music generation through leveraging the largest ever symbolic music dataset sourced from the MuseScore forum. We first pretrain a large unconditional transformer model using 1.5 million songs. We then propose a simple technique to equip this pretrained unconditional music transformer model with instrument and genre controls by finetuning the model with additional control tokens. Our proposed representation offers improved high-level controllability and expressiveness against two existing representations. The experimental results show that the proposed model can successfully generate music with user-specified instruments and genre. In a subjective listening test, the proposed model outperforms the pretrained baseline model in terms of coherence, harmony, arrangement and overall quality.

Optical beamforming networks (OBFNs) based on optical true time delay lines (OTTDLs) are well-known as the promising candidate to solve the bandwidth limitation of traditional electronic phased array antennas (PAAs) due to beam squinting. Here we report the first monolithic 1x8 microwave photonic beamformer based on switchable OTTDLs on the silicon-on-insulator platform. The chip consists of a modulator, an eight-channel OBFN, and 8 photodetectors, which includes hundreds of active and passive components in total. It has a wide operating bandwidth from 8 to 18 GHz, which is almost two orders larger than that of electronic PAAs. The beam can be steered to 31 distinguishable angles in the range of -75.51{\deg} to 75.64{\deg} based on the beam pattern calculation with the measured RF response. The response time for beam steering is 56 {\mu}s. These results represent a significant step towards the realization of integrated microwave photonic beamformers that can satisfy compact size and low power consumption requirements for the future radar and wireless communication systems.