Search

The transformer-based encoder-decoder framework is becoming popular in scene text recognition, largely because it naturally integrates recognition clues from both visual and semantic domains. However, recent studies show that the two kinds of clues are not always well registered and therefore, feature and character might be misaligned in difficult text (e.g., with a rare shape). As a result, constraints such as character position are introduced to alleviate this problem. Despite certain success, visual and semantic are still separately modeled and they are merely loosely associated. In this paper, we propose a novel module called multi-domain character distance perception (MDCDP) to establish a visually and semantically related position embedding. MDCDP uses the position embedding to query both visual and semantic features following the cross-attention mechanism. The two kinds of clues are fused into the position branch, generating a content-aware embedding that well perceives character spacing and orientation variants, character semantic affinities, and clues tying the two kinds of information. They are summarized as the multi-domain character distance. We develop CDistNet that stacks multiple MDCDPs to guide a gradually precise distance modeling. Thus, the feature-character alignment is well build even though various recognition difficulties are presented. We verify CDistNet on ten challenging public datasets and two series of augmented datasets created by ourselves. The experiments demonstrate that CDistNet performs highly competitively. It not only ranks top-tier in standard benchmarks, but also outperforms recent popular methods by obvious margins on real and augmented datasets presenting severe text deformation, poor linguistic support, and rare character layouts. In addition, the visualization shows that CDistNet achieves proper information utilization in both visual and semantic domains. Our code is available at https://github.com/simplify23/CDistNet. [ABSTRACT FROM AUTHOR]

Image super-resolution (SR) is a fundamental research task in low-level vision. Recently it has been applied to digital pathology to build transformations from low-resolution (LR) to super-resolved high-resolution (HR) images, which benefits pathological image sharing, storage, management, etc. However, existing studies on pathological image SR are mostly carried out on simulated dataset. It cannot fully reveal the challenge of real-world SR. Meanwhile, these studies rarely investigate SR models from a pathological-tailored perspective. This paper aims to promote studies on pathological image SR from the two aspects. Firstly, we construct PathImgSR, a dataset containing real-captured paired LR-HR pathological images by leveraging the progressively imaging property of pathological images. Second, we develop MASRGAN, a GAN-based mix-attention network to implement the SR. It devises a mix-attention block that is featured by modeling the channel and spatial attentions in parallel. Therefore it better captures the discriminative feature from pathological images spatially and channel-wisely. Furthermore, by formulating the learning processing in an adversarial learning manner, it also improves the subjective perception quality of the reconstructed HR image. Experiments on PathImgSR demonstrate that MASRGAN outperforms popular CNN-based and GAN-based SR methods in both quantitative metrics and visual subjective perception. [ABSTRACT FROM AUTHOR]

Visual Question Answering(VQA), an important task to evaluate the cross-modal understanding capability of an Artificial Intelligence model, has been a hot research topic in both computer vision and natural language processing communities. Recently, graph-based models have received growing interest in VQA, for its potential of modeling the relationships between objects as well as its formidable interpretability. Nonetheless, those solutions mainly define the similarity between objects as their semantical relationships, while largely ignoring the critical point that the difference between objects can provide more information for establishing the relationship between nodes in the graph. To achieve this, we propose an object-difference based graph learner, which learns question-adaptive semantic relations by calculating inter-object difference under the guidance of questions. With the learned relationships, the input image can be represented as an object graph encoded with structural dependencies between objects. In addition, existing graph-based models leverage the pre-extracted object boxes by the object detection model as node features for convenience, but they are suffering from the redundancy problem. To reduce the redundant objects, we introduce a soft-attention mechanism to magnify the question-related objects. Moreover, we incorporate our object-difference based graph learner into the soft-attention based Graph Convolutional Networks to capture question-specific objects and their interactions for answer prediction. Our experimental results on the VQA 2.0 dataset demonstrate that our model gives significantly better performance than baseline methods. [ABSTRACT FROM AUTHOR]

In recent years, deep convolutional features have been applied to discriminative correlation filters-based methods, which have achieved impressive performance in tracking. Most of them utilize hierarchical features from a certain layer. However, this is not always sufficient to learn target appearance changes and to suppress the background interference in complicated interfering factors (e.g., deformation, fast motion, low resolution, and rotations). In this paper, we propose an adaptive multi-branch correlation filter tracking method, by constructing multi-branch models and using an adaptive selection strategy to improve the accuracy and robustness of visual tracking. Specially, the multi-branch models are introduced to tolerate temporal changes of the object, which can serve different circumstances. In addition, the adaptive selection strategy incorporates both foreground and background information to learn background suppression. To further improve the tracking performance, we propose a measurement method to handle tracking failures from unreliable samples. Extensive experiments on OTB-2013, OTB-2015, and VOT-2016 datasets show that the proposed tracker has comparable performance compared to state-of-the-art tracking methods. Especially, on the OTB-2015, our method significantly improves the baseline with a gain of 5.5% in overlap precision. [ABSTRACT FROM AUTHOR]

Weakly-supervised semantic segmentation with image-level labels has much significance as it facilitates related practical applications under lightweight manual annotations. Recent researches first infer the visual cues, referred as discriminative regions corresponding to each object category on images using deep convolutional classification networks. Then they expand visual cues to generate initial segmentation masks. Despite the remarkable progress, the segmentation performance still remains unsatisfactory due to the absence of complete visual cues. Using these low-quality visual cues as prior will have the limitation on improving segmentation performance. To overcome this problem, we propose a novel context propagation embedding network, i.e., the CPENet to generate high-quality visual cues, which focuses on learning semantic relationship between each region and its surrounding neighbors and selectively propagates discriminative information to non-discriminative, object related regions. Our methods can provide reliable initial segmentation masks for training subsequent segmentation network to generate final segmentation results. In addition, we refined convolutional block attention module (CBAM) [30] to hierarchically extract more category-aware features by capturing global context related information and further promote the propagation process. Experiments on benchmark demonstrate that our proposed method obtains superior performance over the state-of-the-arts. [ABSTRACT FROM AUTHOR]

This paper describes methods for automatically associating faces detected from multimedia documents with their names presented in the surrounding metadata. We consider the task in the image matching (IM) framework, where external Web facial images are automatically retrieved as the gallery face set of the names in advance, and a detected face is assigned to one of the names, or none of them, according to the association score between the two kinds of faces and constraints. Several important issues are investigated within the IM framework. In collecting Web facial images, beyond the basic scheme that use a celebrity name purely as the query to crawl facial images, a context-assisted image search method is proposed to enhance the relevance and discriminability of the retrieved faces. In constraint formulation, we propose an assigning-thresholding (AT) pipeline to uniformly ensure that the name-face correspondence is strictly one-to-one, and set low confidence associations as null assignments. In association score computation, we propose methods that jointly consider IM with the well-established graph-based association (GA) method at different stages, aiming at producing more accurate scores to benefit the association. Based on these efforts, an Accu-IM method performing the association as accurate as possible and a Fast-IM method performing the association in real-time are respective proposed. Extensive experiments on datasets of captioned News images and Web videos both demonstrate the advantages of the proposed efforts individually and jointly, which consistently provide improvement gains under different settings when compared with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In recent years, block compressed sampling (BCS) has emerged as a considerable attractive sampling technology for image acquisition. However, the general BCS approaches ignore the information distribution in the same image sub-block, and may lead to unfair allocation of sampling resources. In this paper, we propose a novel compressed sampling scheme by employing the idea of adaptive partition. In the proposed scheme, images are adaptively partitioned based on their saliency information through clustering, and pixels with similar saliency are gathered in the same sub-blocks. Sampling rates for those blocks, in turn, are computed on the basis of their saliency values, respectively. Therefore the sampling resources are allocated with fairer and more equitable sharing by all sub-blocks. Experimental results show that the proposed scheme has better visual effect and obtains higher image reconstruction accuracy than existing ones. [ABSTRACT FROM AUTHOR]

How to read Uyghur text from biomedical graphic images is a challenge problem due to the complex layout and cursive writing of Uyghur. In this paper, we propose a system that extracts text from Uyghur biomedical images, and matches the text in a specific lexicon for semantic analysis. The proposed system possesses following distinctive properties: first, it is an integrated system which firstly detects and crops the Uyghur text lines using a single fully convolutional neural network, and then keywords in the lexicon are matched by a well-designed matching network. Second, to train the matching network effectively an online sampling method is applied, which generates synthetic data continually. Finally, we propose a GPU acceleration scheme for matching network to match a complete Uyghur text line directly rather than a single window. Experimental results on benchmark dataset show our method achieves a good performance of F-measure 74.5%. Besides, our system keeps high efficiency with 0.5s running time for each image due to the GPU acceleration scheme. [ABSTRACT FROM AUTHOR]

Uyghur text localization in complex background images is a significant research for Uyghur image content analysis. In this paper, we propose a robust Uyghur text localization method in complex background images and provide a CPU-GPU heterogeneous parallelization scheme. Firstly, a multi-color-channel enhanced maximally stable extremal region is used to extract components in images, which is robust to blur and low contrast. Secondly, a two-stage component classification system is used to filter out non-text components. Finally, a component connected graph algorithm is proposed to construct text lines. Experiments on the proposed dataset demonstrate that our algorithm compares favorably with the state-of-the-art algorithms when handling Uyghur texts. Besides, the heterogeneous parallel implementation achieves 12.5 times speedup. [ABSTRACT FROM AUTHOR]

Uyghur text detection is crucial to a variety of real-world applications, while little researches put their attention on it. In this paper, we develop an effective and efficient region-based convolutional neural network for Uyghur text detection in complex background images. The characteristics of the network include: (1) Three region proposal networks are used to improve the recall, which simultaneously utilize feature maps from different convolutional layers. (2) The overall architecture of our network is in the form of fully convolutional network, and global average pooling is applied to replace the fully connected layers in the classification and bounding box regression layers. (3) To fully utilize the baseline information, Uyghur text lines are detected directly by the network in an end-to-end fashion. Experiment results on benchmark dataset show that our method achieves an F-measure of 0.83 and detection time of 0.6 s for each image in a single K20c GPU, which is much faster than the state-of-the-art methods while keeps competitive accuracy. [ABSTRACT FROM AUTHOR]

The proliferation of mobile devices is producing a new wave of mobile visual search applications that enable users to sense their surroundings with smart phones. As the particular challenges of mobile visual search, achieving high recognition bitrate becomes the consistent target of existed related works. In this paper, we explore to holistically exploit the deep learning-based hashing methods for more robust and instant mobile visual search. Firstly, we present a comprehensive survey of the existed deep learning based hashing methods, which showcases their remarkable power of automatic learning highly robust and compact binary code representation for visual search. Furthermore, in order to implement the deep learning hashing on computation and memory constrained mobile device, we investigate the deep learning optimization works to accelerate the computation and reduce the model size. Finally, we demonstrate a case study of deep learning hashing based mobile visual search system. The evaluations show that the proposed system can significantly improve 70% accuracy in MAP than traditional methods, and only needs less than one second computation time on the ordinary mobile phone. Finally, with the comprehensive study, we discuss the open issues and future research directions of deep learning hashing for mobile visual search. [ABSTRACT FROM AUTHOR]

Broadcast news video has been playing an increasingly important role in our daily life. However, how to effectively segment a broadcast news video into meaningful semantic story units is still a challenge issue. In this paper, we propose a novel unified video structure parsing approach, named multiple style exploration-based news story segmentation (MSE-NSS), to segment broadcast news videos into semantic story units. In MSE-NSS, we first investigate the appropriate methods to explore multiple kinds of style information inherent in broadcast news videos, including temporal style inferred from caption texts, boundary style represented by a wealth of multi-modal visual-audio features, and structural style known as the spanning duration of story units. Then the above multiple style information is integrated together and the task of story unit segmentation is accomplished through the following three steps: temporal style-based pre-location, boundary style-based description, and boundary-structural style-based segmentation, where the segmentation process is composed of a SVM-based detector and a dynamic programming-based refiner that considers the boundary style and the structural style collectively. Parallel to this, a news-oriented broadcast management system-NOBMs is implemented on top of the proposed MSE-NSS. Encouraging experimental results on a large broadcast news video dataset demonstrate the effectiveness of the proposed MSE-NSS, as well as its superiority over traditional story unit segmentation methods. [ABSTRACT FROM AUTHOR]