Search

• A fusion mechanism for global feature and local object-level feature is proposed. • The mechanism relies on the spatial information of the objects in the image. • Global features and local features are adaptively fusion by attention. • The fused features are decoded captions during generating captioning. Although attention mechanisms are exploited widely in encoder-decoder neural network-based image captioning framework, the relation between the selection of salient image regions and the supervision of spatial information on local and global representation learning was overlooked, thereby degrading captioning performance. Consequently, we propose an image captioning scheme based on adaptive spatial information attention (ASIA), extracting a sequence of spatial information of salient objects in a local image region or an entire image. Specifically, in the encoding stage, we extract the object-level visual features of salient objects and their spatial bounding-box. We obtain the global feature maps of an entire image, which are fused with local features and the fused features are fed into the LSTM-based language decoder. In the decoding stage, our adaptive attention mechanism dynamically selects the corresponding image regions specified by an image description. Extensive experiments conducted on two datasets demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Keypoint-based object detection achieves better performance without positioning calculations and extensive prediction. However, they have heavy backbone, and high-resolution is restored using upsampling that obtain unreliable features. We propose a self-constrained parallelism keypoint-based lightweight object detection network (SCPNet), which speeds inference, drops parameters, widens receptive fields, and makes prediction accurate. Specifically, the parallel multi-scale fusion module (PMFM) with parallel shuffle blocks (PSB) adopts parallel structure to obtain reliable features and reduce depth, adopts repeated multi-scale fusion to avoid too many parallel branches. The self-constrained detection module (SCDM) has a two-branch structure, with one branch predicting corners, and employing entad offset to match high-quality corner pairs, and the other branch predicting center keypoints. The distances between the paired corners' geometric centers and the center keypoints are used for self-constrained detection. On MS-COCO 2017 and PASCAL VOC , SCPNet's results are competitive with the state-of-the-art lightweight object detection. https://github.com/mengdie-wang/SCPNet.git. • SCPNet can speed inference, drop parameters, and obtain more accurate boundary boxes. • PMFM for feature extraction can maintain high-resolution and widen receptive field. • SCDM proposes the distance constraint between geometric centers and center keypoints. [ABSTRACT FROM AUTHOR]