Search

A comprehensive understanding of interested human-to-human interactions in video streams, such as queuing, handshaking, fighting and chasing, is of immense importance to the surveillance of public security in regions like campuses, squares and parks. Different from conventional human interaction recognition, which uses choreographed videos as inputs, neglects concurrent interactive groups, and performs detection and recognition in separate stages, we introduce a new task named human-to-human interaction detection (HID). HID devotes to detecting subjects, recognizing person-wise actions, and grouping people according to their interactive relations, in one model. First, based on the popular AVA dataset created for action detection, we establish a new HID benchmark, termed AVA-Interaction (AVA-I), by adding annotations on interactive relations in a frame-by-frame manner. AVA-I consists of 85,254 frames and 86,338 interactive groups, and each image includes up to 4 concurrent interactive groups. Second, we present a novel baseline approach SaMFormer for HID, containing a visual feature extractor, a split stage which leverages a Transformer-based model to decode action instances and interactive groups, and a merging stage which reconstructs the relationship between instances and groups. All SaMFormer components are jointly trained in an end-to-end manner. Extensive experiments on AVA-I validate the superiority of SaMFormer over representative methods. The dataset and code will be made public to encourage more follow-up studies.

Spectrometers are a workhorse tool of optics, with applications ranging from scientific research to industrial process monitoring, remote sensing, and medical diagnostics. Although benchtop systems offer high performance and stability, alternative platforms offering reduced size, weight and cost could enable a host of new applications, e.g. in consumer personal electronics and field-deployable diagnostic platforms. To contribute to this trend towards miniaturised optical systems including spectrometers, this thesis presents the realisation of a visible spectrum microspectrometer using structurally coloured silicon nanowires and a reconstruction algorithm. We also experimentally demonstrate a plasmonic mid-infrared filter array-detector array microspectrometer that uses machine learning to determine the chemical compositions of a variety of liquids and solids. In this dissertation, we first present a reconstructive microspectrometer based on vertical silicon nanowire photodetectors. The nanowire photodetectors are designed to have absorption peaks across the visible spectrum. The spectral positions of these peaks are controlled by the radii of the nanowires. The nanowire detectors sit on substrate mesas that also serve as photodetectors for the light transmitted through the nanowires. We demonstrate the fabrication of this device, which has a footprint of a few millimetres. We use it as a spectrometer for the visible spectrum by implementing reconstructive algorithms. The identification of chemicals from their mid-infrared spectra has applications that include the industrial production of chemicals, food production, pharmaceutical manufacturing, and environmental monitoring. This is generally done using laboratory tools such as the Fourier transform infrared spectrometer. To address the need for fast and portable chemical sensing tools, we demonstrate the concept of a chemical classifier based on a filter array-detector array mid-infrared microspectrometer and a mach

In recent years there has been much interest concerning the development of modulators in the mid- to long-wave infrared, based on emerging materials such as graphene. These have been frequently pursued for optical communications, though also for other specialized applications such as infrared scene projectors. Here we investigate a new application for graphene modulators in the mid- to long-wave infrared. We demonstrate, for the first time, computational spectroscopy in the mid- to long-wave infrared using a graphene-based metasurface modulator. Furthermore, our metasurface device operates at low gate voltage. To demonstrate computational spectroscopy, we provide our algorithm with the measured reflection spectra of the modulator at different gate voltages. We also provide it with the measured reflected light power as a function of the gate voltage. The algorithm then estimates the input spectrum. We show that the reconstructed spectrum is in good agreement with that measured directly by a Fourier transform infrared spectrometer, with a normalized mean-absolute-error (NMAE) of 0.021.

In recent years there has been much interest concerning the development of modulators in the mid- to long-wave infrared, based on emerging materials such as graphene. These have been frequently pursued for optical communications, though also for other specialized applications such as infrared scene projectors. Here we investigate a new application for graphene modulators in the mid- to long-wave infrared. We demonstrate, for the first time, computational spectroscopy in the mid- to long-wave infrared using a graphene-based metasurface modulator. Furthermore, our metasurface device operates at low gate voltage. To demonstrate computational spectroscopy, we provide our algorithm with the measured reflection spectra of the modulator at different gate voltages. We also provide it with the measured reflected light power as a function of the gate voltage. The algorithm then estimates the input spectrum. We show that the reconstructed spectrum is in good agreement with that measured directly by a Fourier transform infrared spectrometer, with a normalized mean-absolute-error (NMAE) of 0.021.

Compared with the progress made on human activity classification, much less success has been achieved on human interaction understanding (HIU). Apart from the latter task is much more challenging, the main cause is that recent approaches learn human interactive relations via shallow graphical models, which is inadequate to model complicated human interactions. In this paper, we propose a consistency-aware graph network, which combines the representative ability of graph network and the consistency-aware reasoning to facilitate the HIU task. Our network consists of three components, a backbone CNN to extract image features, a factor graph network to learn third-order interactive relations among participants, and a consistency-aware reasoning module to enforce labeling and grouping consistencies. Our key observation is that the consistency-aware-reasoning bias for HIU can be embedded into an energy function, minimizing which delivers consistent predictions. An efficient mean-field inference algorithm is proposed, such that all modules of our network could be trained jointly in an end-to-end manner. Experimental results show that our approach achieves leading performance on three benchmarks.