Search

Motivation: Retrosynthesis planning poses a formidable challenge in the organic chemical industry, particularly in pharmaceuticals. Single-step retrosynthesis prediction, a crucial step in the planning process, has witnessed a surge in interest in recent years due to advancements in AI for science. Various deep learning-based methods have been proposed for this task in recent years, incorporating diverse levels of additional chemical knowledge dependency. Results: This paper introduces UAlign, a template-free graph-to-sequence pipeline for retrosynthesis prediction. By combining graph neural networks and Transformers, our method can more effectively leverage the inherent graph structure of molecules. Based on the fact that the majority of molecule structures remain unchanged during a chemical reaction, we propose a simple yet effective SMILES alignment technique to facilitate the reuse of unchanged structures for reactant generation. Extensive experiments show that our method substantially outperforms state-of-the-art template-free and semi-template-based approaches. Importantly, our template-free method achieves effectiveness comparable to, or even surpasses, established powerful template-based methods. Scientific contribution: We present a novel graph-to-sequence template-free retrosynthesis prediction pipeline that overcomes the limitations of Transformer-based methods in molecular representation learning and insufficient utilization of chemical information. We propose an unsupervised learning mechanism for establishing product-atom correspondence with reactant SMILES tokens, achieving even better results than supervised SMILES alignment methods. Extensive experiments demonstrate that UAlign significantly outperforms state-of-the-art template-free methods and rivals or surpasses template-based approaches, with up to 5% (top-5) and 5.4% (top-10) increased accuracy over the strongest baseline. [ABSTRACT FROM AUTHOR]

Head generation with diverse identities is an important task in computer vision and computer graphics, widely used in multimedia applications. However, current full-head generation methods require a large number of three-dimensional (3D) scans or multi-view images to train the model, resulting in expensive data acquisition costs. To address this issue, we propose Head3D, a method to generate full 3D heads with limited multi-view images. Specifically, our approach first extracts facial priors represented by tri-planes learned in EG3D, a 3D-aware generative model, and then proposes feature distillation to deliver the 3D frontal faces within complete heads without compromising head integrity. To mitigate the domain gap between the face and head models, we present a dual-discriminator to guide the frontal and back head generation. Our model achieves cost-efficient and diverse complete head generation with photo-realistic renderings and high-quality geometry representations. Extensive experiments demonstrate the effectiveness of our proposed Head3D, both qualitatively and quantitatively. [ABSTRACT FROM AUTHOR]

The preparation method of integrated circuit lead frames has transitioned from stamping to etching, rendering them more sensitive to residual stress. Consequently, the dimensional deviations caused by residual stress become more pronounced, necessitating a thorough investigation into the copper strip processing process, particularly considering the high-precision requirements of the lead frame. A quasi in situ method was employed to monitor the deformation process, and quantitative analyses and graphical reconstructions of the residual stress were conducted. The results indicated that the orientation evolution did not exhibit a significant correlation with grain size or grain aspect ratio. However, the stored energy of the different grains was related to their orientations. Further analysis of slip traces revealed that single or multiple slipping may be activated in grain subdivisions, and the Schmid factor difference ratio (SFDR) value proved to be an effective tool for analyzing this deformation mode. An even more interesting finding was that the deformation mode directly affected the residual stress distribution in local regions. The relationship between residual stress, Schmid factor, and SFDR was further analyzed, and a clear correlation between SFDR and residual stress was found in this study. [ABSTRACT FROM AUTHOR]

Detecting component defects and attaching tiny-scaled foreign objects to the overhead transmission lines are critical to the national grid's safe operation and power distribution. This urgent task, however, faces challenges, such as the complex working environment and the considerable amount of workforce investment, for which we propose a deep-learning-aided object detection approach, YOLO-CSM, to address the issue. Combined with two attention mechanisms (Swin transformer and CBAM) and an extra detection layer, the proposed model can effectively capture global information and key visual features and promote its ability to identify tiny-scaled defects and distant objects in the visual fields. In order to validate this model, this work consolidates a dataset composed of public images and our field-taken picture samples. The experiment verifies YOLO-CSM as a suitable solution for small and distant object detection that outperforms several well-used algorithms, featuring a 16.3% faster detection speed than YOLOv5 and a 3.3% better detection accuracy than YOLOv7. Finally, this work conducts an interpretability experiment to reveal the similarity between YOLO-CSM's attention patterns and that of humans, aiming to explain YOLO-CSM's advantages in detecting small objects and minor defects in the working environments of power transmission lines. [ABSTRACT FROM AUTHOR]

Since the audio information is fully explored and leveraged in omnidirectional videos (ODVs), the performance of existing audio-visual saliency models has been improving dramatically and significantly. However, these models are still in their infancy stages, and there are two significant issues in modeling human attention between visual and auditory modalities: (1) Temporal non-alignment problem between auditory and visual modalities is rarely considered; (2) Most audio-visual saliency models are audio content attributes-agnostic. Thus, they need to learn audio features with fine details. This paper proposes a novel audio-visual aligned saliency (AVAS) model that can simultaneously tackle two issues as mentioned above in an effective end-to-end training manner. In order to solve the temporal non-alignment problem between the two modalities, a Hanning window method is employed on the audio stream to truncate the audio signal per unit time (frame-time interval) to match the visual information stream of the corresponding duration, which can capture the potential correlation of two modalities across time steps and facilitate audio-visual features fusion. Regarding the audio content attribute-agnostic issue, an effective periodic audio encoding method is proposed based on implicit neural representation (INR) to map audio sampling points to their corresponding audio frequency values, which can better discriminate and interpret audio content attributes. Comprehensive experiments and detailed ablation analyses are performed on the benchmark dataset to demonstrate the efficacy of the proposed model. The experimental results indicate that the proposed model consistently outperforms other competitors by a large margin. [ABSTRACT FROM AUTHOR]

Person search aims to simultaneously localize and identify a query person from uncropped images. To achieve this goal, state-of-the-art models typically add a re-id branch upon two-stage detectors like Faster R-CNN. Owing to the ROI-Align operation, this pipeline yields promising accuracy as re-id features are explicitly aligned with the corresponding object regions, but in the meantime, it introduces high computational overhead due to dense object anchors. In this work, we present an anchor-free approach to efficiently tackling this challenging task, by introducing the following dedicated designs. First, we select an anchor-free detector (i.e., FCOS) as the prototype of our framework. Due to the lack of dense object anchors, it exhibits significantly higher efficiency compared with existing person search models. Second, when directly accommodating this anchor-free detector for person search, there exist several misalignment issues in different levels (i.e., scale, region, and task). To address these issues, we propose an aligned feature aggregation module to generate more discriminative and robust feature embeddings. Accordingly, we name our framework as Feature-Aligned Person Search Network (AlignPS). Third, by investigating the advantages of both anchor-based and anchor-free models, we further augment AlignPS with an ROI-Align head, which significantly improves the robustness of re-id features while still keeping our model highly efficient. Our framework not only achieves state-of-the-art or competitive performance on two challenging person search benchmarks, but can be also extended to other challenging searching tasks such as animal and object search. All the source codes, data, and trained models are available at: https://github.com/daodaofr/alignps. [ABSTRACT FROM AUTHOR]

Environment perception is a key part of robot self-controlled motion. When using vision to accomplish obstacle detection tasks, it is difficult for deep learning methods to detect all obstacles due to complex environment and vision limitations, and it is difficult for traditional methods to meet real-time requirements when applied to embedded platforms. In this paper, a fast obstacle-detection process applied to RGB-D cameras is proposed. The process has three main steps, feature point extraction, noise removal, and obstacle clustering. Using Canny and Shi–Tomasi algorithms to complete the pre-processing and feature point extraction, filtering noise based on geometry, grouping obstacles with different depths based on the basic principle that the feature points on the same object contour must be continuous or within the same depth in the view of RGB-D camera, and then doing further segmentation from the horizontal direction to complete the obstacle clustering work. The method omits the iterative computation process required by traditional methods and greatly reduces the memory and time overhead. After experimental verification, the proposed method has a comprehensive recognition accuracy of 82.41%, which is 4.13% and 19.34% higher than that of RSC and traditional methods, respectively, and recognition accuracy of 91.72% under normal illumination, with a recognition speed of more than 20 FPS on the embedded platform; at the same time, all detections can be achieved within 1 m under normal illumination, and the detection error is no more than 2 cm within 3 m. [ABSTRACT FROM AUTHOR]

Neural network quantization has shown to be an effective way for network compression and acceleration. However, existing binary or ternary quantization methods suffer from two major issues. First, low bit-width input/activation quantization easily results in severe prediction accuracy degradation. Second, network training and quantization are always treated as two non-related tasks, leading to accumulated parameter training error and quantization error. In this work, we introduce a novel scheme, named Residual Quantization, to train a neural network with both weights and inputs constrained to low bit-width, e.g., binary or ternary values. On one hand, by recursively performing residual quantization, the resulting binary/ternary network is guaranteed to approximate the full-precision network with much smaller errors. On the other hand, we mathematically re-formulate the network training scheme in an EM-like manner, which iteratively performs network quantization and parameter optimization. During expectation, the low bit-width network is encouraged to approximate the full-precision network. During maximization, the low bit-width network is further tuned to gain better representation capability. Extensive experiments well demonstrate that the proposed quantization scheme outperforms previous low bit-width methods and achieves much closer performance to the full-precision counterpart. [ABSTRACT FROM AUTHOR]

Capturing the dynamics in user preference is crucial to better predict user future behaviors because user preferences often drift over time. Many existing recommendation algorithms – including both shallow and deep ones – often model such dynamics independently, i.e., user static and dynamic preferences are not modeled under the same latent space, which makes it difficult to fuse them for recommendation. This paper considers the problem of embedding a user's sequential behavior into the latent space of user preferences, namely translating sequence to preference. To this end, we formulate the sequential recommendation task as a dictionary learning problem, which learns: 1) a shared dictionary matrix, each row of which represents a partial signal of user dynamic preferences shared across users; and 2) a posterior distribution estimator using a deep autoregressive model integrated with Gated Recurrent Unit (GRU), which can select related rows of the dictionary to represent a user's dynamic preferences conditioned on his/her past behaviors. Qualitative studies on the Netflix dataset demonstrate that the proposed method can capture the user preference drifts over time and quantitative studies on multiple real-world datasets demonstrate that the proposed method can achieve higher accuracy compared with state-of-the-art factorization and neural sequential recommendation methods. [ABSTRACT FROM AUTHOR]

Graph matching involves combinatorial optimization based on edge-to-edge affinity matrix, which can be generally formulated as Lawler's quadratic assignment problem (QAP). This paper presents a QAP network directly learning with the affinity matrix (equivalently the association graph) whereby the matching problem is translated into a constrained vertex classification task. The association graph is learned by an embedding network for vertex classification, followed by Sinkhorn normalization and a cross-entropy loss for end-to-end learning. We further improve the embedding model on association graph by introducing Sinkhorn based matching-aware constraint, as well as dummy nodes to deal with unequal sizes of graphs. To our best knowledge, this is one of the first network to directly learn with the general Lawler's QAP. In contrast, recent deep matching methods focus on the learning of node/edge features in two graphs respectively. We also show how to extend our network to hypergraph matching, and matching of multiple graphs. Experimental results on both synthetic graphs and real-world images show its effectiveness. For pure QAP tasks on synthetic data and QAPLIB benchmark, our method can perform competitively and even surpass state-of-the-art graph matching and QAP solvers with notable less time cost. We provide a project homepage at http://thinklab.sjtu.edu.cn/project/NGM/index.html. [ABSTRACT FROM AUTHOR]

Despite pedestrian detectors having made significant progress by introducing convolutional neural networks, their performance still suffers degradation, especially in occlusion scenes with more false positives (FPs) and false negatives (FNs). To alleviate the problem, we propose a novel Sequential Attention-based Distinct Part Modeling (SA-DPM) for balanced pedestrian detection. It takes one step further in constructing more robust representation that supports detection with fewer FNs and FPs. Specifically, the Sequential Attention serves as one internal perception process that captures several distinct part areas step by step from each pedestrian proposal (full-body). Different from the previous either-or feature selection, the following Joint Learning attempts to seek a reasonable trade-off between part and full-body features, and combines both features for more accurate classification and regression. Evaluation on the widely used pedestrian datasets including Caltech and Citypersons shows that the proposed SA-DPM achieves promising performance for both non-occluded and occluded pedestrian detection tasks, especially on Caltech Heavy Occlusion set, which yields a new state-of-the-art miss rate by 30.18% and outperforms the second best detector by 6.32%. [ABSTRACT FROM AUTHOR]

Objective: The m6A modification is the most common ribonucleic acid (RNA) modification, playing a role in prompting the virus's gene mutation and protein structure changes in the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2). Nanopore single-molecule direct RNA sequencing (DRS) provides data support for RNA modification detection, which can preserve the potential $m^6A$ signature compared to second-generation sequencing. However, due to insufficient DRS data, there is a lack of methods to find m6A RNA modifications in DRS. Our purpose is to identify $m^6A$ modifications in DRS precisely. Methods: We present a methodology for identifying $m^6A$ modifications that incorporated mapping and extracted features from DRS data. To detect $m^6A$ modifications, we introduce an ensemble method called mixed-weight neural bagging (MWNB), trained with 5-base RNA synthetic DRS containing modified and unmodified $m^6A$. Results: Our MWNB model achieved the highest classification accuracy of 97.85% and AUC of 0.9968. Additionally, we applied the MWNB model to the COVID-19 dataset; the experiment results reveal a strong association with biomedical experiments. Conclusion: Our strategy enables the prediction of $m^6A$ modifications using DRS data and completes the identification of $m^6A$ modifications on the SARS-CoV-2. Significance: The Corona Virus Disease 2019 (COVID-19) outbreak has significantly influence, caused by the SARS-CoV-2. An RNA modification called $m^6A$ is connected with viral infections. The appearance of $m^6A$ modifications related to several essential proteins affects proteins’ structure and function. Therefore, finding the location and number of $m^6A$ RNA modifications is crucial for subsequent analysis of the protein expression profile. [ABSTRACT FROM AUTHOR]

In virtual reality (VR), correct and precise estimations of user’s visual fixations and head movements can enhance the quality of experience by allocating more computation resources for analysing and rendering on the areas of interest. However, there is insufficient research about understanding the visual exploration of users when modeling VR visual attention. To bridge the gap between the saliency prediction for traditional 2D content and omnidirectional content, we construct the visual attention dataset and propose the visual saliency prediction framework for panoramic videos. Around the instantaneous viewing behavior, we propose a traditional method to adapt 2D saliency models and design a CNN-based model to better predict visual saliency. In the proposed traditional model, mechanism of visual attention and viewing behaviors are considered in the computation of edge weights on graphs which are interpreted as Markov chains. The fraction of the visual attention that is diverted to each high-clarity vision (HCV) area is estimated through equilibrium distribution of this chain. We also propose the Graph-Based CNN model. The RGB channel and optical flow form the spatial-temporal units of HCVs, from which node feature vectors are extracted. Graph convolution is used to learn the mutual information between node feature vectors of HCVs and retain geometric information. Then feature vectors are aligned according to geometry structure of equirectangular format, and the feature decoder maps the aligned feature maps to the data distribution. We also construct the dynamic omnidirectional monocular (DOM) saliency dataset with 64 diverse videos evaluated by 28 people. The subjective results show that the instantaneous viewing behavior is important in the VR experience. Extensive experiments are conducted on the dataset and the results demonstrate the effectiveness of the proposed framework. The dataset will be released to facilitate the future studies related to visual saliency prediction for 360-degree contents. [ABSTRACT FROM AUTHOR]

In the era of multimedia and Internet, the quick response (QR) code helps people obtain information from offline to online quickly. However, the QR code is often limited in many scenarios because of its random and dull appearance. Therefore, this article proposes a novel approach to embed hyperlinks into common images, making the hyperlinks invisible for human eyes but detectable for mobile devices equipped with a camera. Our approach is an end-to-end neural network with an encoder to hide messages and a decoder to extract messages. To maintain the hidden message resilient to cameras, we build a distortion network between the encoder and the decoder to augment the encoded images. The distortion network uses differentiable 3-D rendering operations, which can simulate the distortion introduced by camera imaging in both printing and display scenarios. To maintain the visual attraction of the image with hyperlinks, a loss function conforming to the human visual system (HVS) is used to supervise the training of the encoder. Experimental results show that the proposed approach outperforms the previous work on both robustness and quality. Based on the proposed approach, many applications become possible, for example, “image hyperlinks” for advertisement on TV, website, or poster, and “invisible watermark” for copyright protection on digital resources or product packagings. [ABSTRACT FROM AUTHOR]

With the development of multimedia technology, Augmented Reality (AR) has become a promising next-generation mobile platform. The primary value of AR is to promote the fusion of digital contents and real-world environments, however, studies on how this fusion will influence the Quality of Experience (QoE) of these two components are lacking. To achieve better QoE of AR, whose two layers are influenced by each other, it is important to evaluate its perceptual quality first. In this paper, we consider AR technology as the superimposition of virtual scenes and real scenes, and introduce visual confusion as its basic theory. A more general problem is first proposed, which is evaluating the perceptual quality of superimposed images, i.e., confusing image quality assessment. A ConFusing Image Quality Assessment (CFIQA) database is established, which includes 600 reference images and 300 distorted images generated by mixing reference images in pairs. Then a subjective quality perception experiment is conducted towards attaining a better understanding of how humans perceive the confusing images. Based on the CFIQA database, several benchmark models and a specifically designed CFIQA model are proposed for solving this problem. Experimental results show that the proposed CFIQA model achieves state-of-the-art performance compared to other benchmark models. Moreover, an extended ARIQA study is further conducted based on the CFIQA study. We establish an ARIQA database to better simulate the real AR application scenarios, which contains 20 AR reference images, 20 background (BG) reference images, and 560 distorted images generated from AR and BG references, as well as the correspondingly collected subjective quality ratings. Three types of full-reference (FR) IQA benchmark variants are designed to study whether we should consider the visual confusion when designing corresponding IQA algorithms. An ARIQA metric is finally proposed for better evaluating the perceptual quality of AR images. Experimental results demonstrate the good generalization ability of the CFIQA model and the state-of-the-art performance of the ARIQA model. The databases, benchmark models, and proposed metrics are available at: https://github.com/DuanHuiyu/ARIQA. [ABSTRACT FROM AUTHOR]

Bit-depth enhancement (BDE) is a challenging task due to stubborn false contour artifacts and disappeared detailed information. Given the mixture of structural distortions and real edges in low bit-depth (LBD) images, both large and small receptive fields (RFs) are critical for BDE tasks. However, even powerful state-of-the-art CNN-based methods can hardly capture sufficient LBD features under multiple RFs. This paper proposes a residual-guided multiscale fusion network (RMFNet) to explore multiscale features in a residual manner. We find that the shuffling operation provides desired multiscale inputs for effectively distinguishing false contours from real edges without any loss of information. Therefore, we shuffle LBD images to multiple scales and then fully extract residual features under different RFs with corresponding subnets. To facilitate interscale guidance from the global context to the local context, we progressively transfer the encoded residual features between adjacent subnets from top to bottom. We further propose a dual-branch depthwise group fusion (DDGF) module to fully capture inter- and inner correlations of multiscale features with fewer parameters. Finally, extensive experiments show that our algorithm achieves excellent performance improvement both quantitatively and qualitatively, verifying its effectiveness. [ABSTRACT FROM AUTHOR]

A compression experiment with electron backscatter diffraction (EBSD) measurements was designed to characterize the effect of the microtexture on the grain rotation process. The rotation degrees of more than 180 grains before and after the compression were calculated. Results showed that grains with different crystallographic orientations experienced various rotation degrees. Furthermore, grains in certain microtexture regions also had varying degrees of rotation. The compression led to the lattice rotation and change in orientation of individual grains, but the relative misorientation between grains has not changed much in the microtexture region. The microtexture region, as a whole, participated in the compression process. The similar slipping behavior of the grains in the region promoted the slip transmission between the neighboring grains. Thus, the amount of piled-up dislocations at grain boundaries inside the microtexture region are less than that at grain boundaries outside the microtexture region, leading to a small stored energy density for grain boundaries inside the microtexture region. [ABSTRACT FROM AUTHOR]

In this paper, we address the issue of data imbalance in learning deep models for visual object tracking. Although it is well known that data distribution plays a crucial role in learning and inference models, considerably less attention has been paid to data imbalance in visual tracking. For the deep regression trackers that directly learn a dense mapping from input images of target objects to soft response maps, we identify their performance is limited by the extremely imbalanced pixel-to-pixel differences when computing regression loss. This prevents existing end-to-end learnable deep regression trackers from performing as well as discriminative correlation filters (DCFs) trackers. For the deep classification trackers that draw positive and negative samples to learn discriminative classifiers, there exists heavy class imbalance due to a limited number of positive samples when compared to the number of negative samples. To balance training data, we propose a novel shrinkage loss to penalize the importance of easy training data mostly coming from the background, which facilitates both deep regression and classification trackers to better distinguish target objects from the background. We extensively validate the proposed shrinkage loss function on six benchmark datasets, including the OTB-2013, OTB-2015, UAV-123, VOT-2016, VOT-2018 and LaSOT. Equipped with our shrinkage loss, the proposed one-stage deep regression tracker achieves favorable results against state-of-the-art methods, especially in comparison with DCFs trackers. Meanwhile, our shrinkage loss generalizes well to deep classification trackers. When replacing the original binary cross entropy loss with our shrinkage loss, three representative baseline trackers achieve large performance gains, even setting new state-of-the-art results. [ABSTRACT FROM AUTHOR]

Learning to generate continuous linguistic descriptions for multi-subject interactive videos in great details has particular applications in team sports auto-narrative. In contrast to traditional video caption, this task is more challenging as it requires simultaneous modeling of fine-grained individual actions, uncovering of spatio-temporal dependency structures of frequent group interactions, and then accurate mapping of these complex interaction details into long and detailed commentary. To explicitly address these challenges, we propose a novel framework Graph-based Learning for Multi-Granularity Interaction Representation (GLMGIR) for fine-grained team sports auto-narrative task. A multi-granular interaction modeling module is proposed to extract among-subjects’ interactive actions in a progressive way for encoding both intra- and inter-team interactions. Based on the above multi-granular representations, a multi-granular attention module is developed to consider action/event descriptions of multiple spatio-temporal resolutions. Both modules are integrated seamlessly and work in a collaborative way to generate the final narrative. In the meantime, to facilitate reproducible research, we collect a new video dataset from YouTube.com called Sports Video Narrative dataset (SVN). It is a novel direction as it contains $6K$ 6 K team sports videos (i.e., NBA basketball games) with $10K$ 10 K ground-truth narratives(e.g., sentences). Furthermore, as previous metrics such as METEOR (i.e., used in coarse-grained video caption task) DO NOT cope with fine-grained sports narrative task well, we hence develop a novel evaluation metric named Fine-grained Captioning Evaluation (FCE), which measures how accurate the generated linguistic description reflects fine-grained action details as well as the overall spatio-temporal interactional structure. Extensive experiments on our SVN dataset have demonstrated the effectiveness of the proposed framework for fine-grained team sports video auto-narrative. [ABSTRACT FROM AUTHOR]

Temporal point process (TPP) is an expressive tool for modeling the temporal pattern of event sequences. However, discovering temporal patterns for event sequences clustering is rarely studied in TPP modeling. To solve this problem, we take a reinforcement learning view whereby the observed sequences are assumed to be generated from a mixture of latent policies. The purpose is to cluster the sequences with different temporal patterns into the underlying policies while learning each of the policy model. The flexibility of our model lies in: i) all the components are networks including the policy network for modeling the temporal point process; ii) to handle varying-length event sequences, we resort to inverse reinforcement learning by decomposing the observed sequence into states (RNN hidden embedding of history) and actions (time interval to next event) in order to learn a reward function, it helps to achieve better performance or increasing efficiency compared to existing methods using rewards over the entire sequence such as log-likelihood or Wasserstein distance. We adopt an Expectation-Maximization algorithm, in E-step estimating the cluster labels for each sequence, in M-step aiming to learn the respective policy. Extensive experiments on synthetic and real-world datasets show the efficacy of our method against the state-of-the-arts. [ABSTRACT FROM AUTHOR]

Variations in the wind direction over time mean that it is essential to improve the directional adaptability of wind energy harvesters (WEHs) based on wind-induced vibration (WIV) to expand their application potential. Several multi-directional WIV WEHs have been reported in the literature but most of them are not omnidirectional. In particular, no mathematical model has been proposed for omnidirectional WIV WEHs to date. In this Letter, an in-plane omnidirectional piezoelectric WEH with a cylindrical shell, acting as a bluff body and supported by internal piezoelectric composite beams, is proposed. It is deduced that the omnidirectionality of wind energy harvesting can be enhanced by improving the isotropies of the aerodynamic force, stiffness, and electromechanical conversion. For a WEH with three semicircular-shaped supporting beams, a mathematical model suitable for arbitrary wind directions in the horizontal plane was derived. Simulations show that the WEH's stiffness and electromechanical conversion are approximately isotropic. Simulations and experiments demonstrate that the wind direction's effect on the total power is small. The ratio of the experimental minimum to maximum total power is 0.88 at 9 m/s, verifying that the device is an in-plane omnidirectional harvester. An omnidirectionality index including contributions from all directions is proposed with the value of 0.86 at 9 m/s for the prototype. The proposed device configuration and design method may serve as a reference for the development of omnidirectional WIV WEHs. [ABSTRACT FROM AUTHOR]

In this study, we used a difference-in-difference (DID) approach to analyze the effect of environmental regulation on corporate tax avoidance behavior based on China's carbon emissions trading pilot policy of 2013. Our findings were as follows: (1) Environmental regulation has led companies to adopt further tax evasion behaviors. Furthermore, the core conclusion was confirmed after a series of robust and endogenous tests, such as parallel trends and PSM-DID (propensity score matching-difference-in-difference). (2) Environmental regulations increase tax avoidance activities by reducing corporate cash flows. (3) The influence of environmental regulation on firm tax evasion is highly pronounced among non-state-owned enterprises, big-scale enterprises, and enterprises with a high degree of industry competition. [ABSTRACT FROM AUTHOR]

The power consumption of mobile devices is always a concern for users due to the constraints on the size and weight of mobile devices. Up to now, mobile video traffic has accounted for the majority of total traffic, which implies that viewing mobile videos has been the major activity when using mobile devices. Among all the subsystems involving in mobile video playback, the display is the most power consuming subsystem. To reduce the power consumption, dynamic backlight scaling (DBS) technique is developed by adjusting the backlight magnitude when playing the mobile video. However, the convenience of mobile devices makes lots of people watch mobile videos in various luminance environments, which makes the existing DBS methods ineffective since ambient luminance varies greatly. In this paper, we propose a novel DBS strategy to maximally enhance the battery power performance under various ambient luminance conditions through backlight magnitude adjusting, while without negatively impacting users’ quality of experience (QoE). In particular, we conduct a series of subject quality assessment experiments to uncover the quantitative relationship among QoE, ambient luminance, video content luminance, and backlight luminance. We then investigate whether the continuous playback of backlight-scaled videos using the proposed scaling magnitude under various luminance environments would cause flicker effect or not. Motivated by the findings of these studies, we implement a novel DBS strategy for mobile energy saving which is suitable for various ambient luminance conditions. The experimental results demonstrate that the proposed DBS strategy can save more than 40 percent power at most and can save 10 percent power even at a very high ambient luminance condition. We also show that the proposed DBS strategy can be easily adapted to different user preferences and different devices, and can be conveniently integrated into practical applications. [ABSTRACT FROM AUTHOR]

The high-temperature deformation behavior of the as-forged TC17 titanium alloy was studied based on the results of thermal compression simulation tests. Additionally, the effects of strain rate, temperature and true strain on the flow stress were analyzed using orthogonal experimentation and variance analysis. The results indicated that all deformation parameters exerted significant influence on flow stress, with temperature and strain having the largest and least influence, respectively. Thereafter, a constitutive equation based on modified nonlinear regression was proposed. The error analysis results showed that the flow stress of the as-forged TC17 alloy could be predicted by the obtained constitutive equation. The effect of deformation parameters on the microstructure was investigated by EBSD and TEM. [ABSTRACT FROM AUTHOR]

The emerging vision-and-language navigation (VLN) problem aims at learning to navigate an agent to the target location in unseen photo-realistic environments according to the given language instruction. The main challenges of VLN arise mainly from two aspects: first, the agent needs to attend to the meaningful paragraphs of the language instruction corresponding to the dynamically-varying visual environments; second, during the training process, the agent usually imitate the expert demonstrations, i.e., the shortest-path to the target location specified by associated language instructions. Due to the discrepancy of action selection between training and inference, the agent solely on the basis of imitation learning does not perform well. Existing VLN approaches address this issue by sampling the next action from its predicted probability distribution during the training process. This allows the agent to explore diverse routes from the environments, yielding higher success rates. Nevertheless, without being presented with the golden shortest navigation paths during the training process, the agent may arrive at the target location through an unexpected longer route. To overcome these challenges, we design a cross-modal grounding module, which is composed of two complementary attention mechanisms, to equip the agent with a better ability to track the correspondence between the textual and visual modalities. We then propose to recursively alternate the learning schemes of imitation and exploration to narrow the discrepancy between training and inference. We further exploit the advantages of both these two learning schemes via adversarial learning. Extensive experimental results on the Room-to-Room (R2R) benchmark dataset demonstrate that the proposed learning scheme is generalized and complementary to prior arts. Our method performs well against state-of-the-art approaches in terms of effectiveness and efficiency. [ABSTRACT FROM AUTHOR]

Recent years have witnessed rapid development of deep learning technology and its successful application in the saliency prediction of traditional 2D images. However, when using deep neural network (DNN) models to perform saliency prediction on omnidirectional images (ODIs), there are two critical issues: (1) The datasets for ODIs are small-scale that cannot support the training DNN-based models. (2) It is challenging to perform saliency prediction in that some ODIs contain complex background clutters. In order to solve these two problems, we propose a novel Attention-Aware Features Fusion Network (AAFFN) model which is first trained with traditional 2D images and then transferred to the ODIs for saliency prediction. Specifically, our proposed AAFFN model consists of three modules: a Part-guided Attention (PA) module, a Visibility Score (VS) module, and a Attention-Aware Features Fusion (AAFF) module. The PA module is used to extract precise features to estimate attention of the finer part on ODIs, and eliminate the influence of cluttered background. Meanwhile, the VS module is introduced to measure the proportion of the foreground and background parts and generate visibility scores in the feature learning process. Finally, in the AAFF module, we utilize the weighted fusion of attention maps and visibility scores to generate the final saliency map. Extensive experiments and ablation analysis demonstrate that the proposed model achieves superior performance and outperforms other state-of-the-art methods on public benchmark datasets. [ABSTRACT FROM AUTHOR]

Performance of blind image quality assessment (BIQA) models has been significantly boosted by end-to-end optimization of feature engineering and quality regression. Nevertheless, due to the distributional shift between images simulated in the laboratory and captured in the wild, models trained on databases with synthetic distortions remain particularly weak at handling realistic distortions (and vice versa). To confront the cross-distortion-scenario challenge, we develop a unified BIQA model and an approach of training it for both synthetic and realistic distortions. We first sample pairs of images from individual IQA databases, and compute a probability that the first image of each pair is of higher quality. We then employ the fidelity loss to optimize a deep neural network for BIQA over a large number of such image pairs. We also explicitly enforce a hinge constraint to regularize uncertainty estimation during optimization. Extensive experiments on six IQA databases show the promise of the learned method in blindly assessing image quality in the laboratory and wild. In addition, we demonstrate the universality of the proposed training strategy by using it to improve existing BIQA models. [ABSTRACT FROM AUTHOR]

Retinal screening contributes to early detection of diabetic retinopathy and timely treatment. To facilitate the screening process, we develop a deep learning system, named DeepDR, that can detect early-to-late stages of diabetic retinopathy. DeepDR is trained for real-time image quality assessment, lesion detection and grading using 466,247 fundus images from 121,342 patients with diabetes. Evaluation is performed on a local dataset with 200,136 fundus images from 52,004 patients and three external datasets with a total of 209,322 images. The area under the receiver operating characteristic curves for detecting microaneurysms, cotton-wool spots, hard exudates and hemorrhages are 0.901, 0.941, 0.954 and 0.967, respectively. The grading of diabetic retinopathy as mild, moderate, severe and proliferative achieves area under the curves of 0.943, 0.955, 0.960 and 0.972, respectively. In external validations, the area under the curves for grading range from 0.916 to 0.970, which further supports the system is efficient for diabetic retinopathy grading. As the leading cause of vision loss in working-age adults, diabetic retinopathy requires routinely retinal screening. Here the authors develop a deep learning system that can facilitate the screening by providing real-time image quality assessment, lesions detection, and grades across the disease spectrum. [ABSTRACT FROM AUTHOR]

Fractional interpolation has been extensively utilized in a series of video coding standards to generate fractional precision prediction to remove the temporal redundancy in consecutive frames. In the traditional interpolation filter based methods, the fractional samples are interpolated through a linear combination of the neighboring integer samples. This method is simple yet unable to accurately characterize the nonstationary video signals. Recently, convolutional neural network has been utilized in the fractional interpolation and shows superior performance compared with the traditional methods. However, only the reconstruction of the reference frame is used as the infer information source. All the other information contained in the bitstream or generated during the encoding/decoding procedure denoted as the compression prior is not utilized at all. In this paper, we give the first trial to involve some compression priors into the CNN to improve the performance of a in-loop coding tool. Specifically, we propose a Compression Priors assisted Convolutional Neural Network (CPCNN) to further improve the fractional interpolation efficiency. In addition to the reconstructed component, we additionally utilize two other compression priors – the corresponding residual component and col-located high quality component to boost the performance. Specifically, the residual component that indicates the prediction efficiency and contains effective texture information is utilized as a complementary input to the reconstructed one. While the col-located component provides more useful high quality information to help the reconstruction get rid of the quality fluctuation. Furthermore, a special network structure is designed to learn powerful representations of these triple input components. Comprehensive experiments have been conducted to demonstrate the effectiveness of our proposed CPCNN. The experimental results show that compared to HEVC, our proposed CPCNN achieves on average of 5.3%, 2.8% and 1.9% BD-Rate savings under LDP, LDB and RA configurations, respectively. [ABSTRACT FROM AUTHOR]

The simulation survey of TAE Technologies has demonstrated that high harmonic fast wave (HHFW) heating is a promising method for core electron heating of FRC plasma. This study mainly describes the HHFW antenna mechanical design and assembly on the basis of the results of electromagnetic simulations performed by Oak Ridge National Laboratory (ORNL), the available port dimensions, and antenna installation position of the LAPD. Compared to the original scheme, this antenna is also optimized in the design. It is found that the E field distribution of optimized antenna becomes even, and the maximum electric field decreases by approximately 14%. The current on the antenna box and FS is reduced after optimization, whereas the maximum J density decreases from 53.3 kA to 14.5 kA. The reflection performance of the port at 30 MHz is also improved after the structural optimization; The k// spectrum distribution is sharper at the monopole phase (0, 0, 0, 0) and dipole phase (0, π, 0, π) and (0, 90, 270,180) than other phases. The optimized antenna can obtain a maximum |k//| spectrum, which peaks about |k//| = 30 m−1 at the dipole phase (0, π, 0, π). The analysis results and assembly strategy can provide useful reference and guidance for the study of HHFW antenna design and fabrication in LAPD or other magnetic confined fusion devices. [ABSTRACT FROM AUTHOR]

In this paper, we propose an adaptive region proposal scheme with feature channel regularization to facilitate robust object tracking. We consider tracking as a linear regression problem and an ensemble of correlation filters is trained on-line to distinguish the foreground target from the background. Further, we integrate adaptively learned region proposals into an enhanced two-stream tracking framework based on correlation filters. For the tracking stream, we learn two-stage cascade correlation filters on deep convolutional features to ensure competitive tracking performance. For the detection stream, we employ adaptive region proposals, which are effective in recovering target objects from tracking failures caused by heavy occlusion or out-of-view movement. In contrast to traditional tracking-by-detection methods using random samples or sliding windows, we perform target re-detection over adaptively learned region proposals. Since region proposals naturally take the objectness information into account, we show that the proposed adaptive region proposals can handle the challenging scale estimation problem as well. In addition, we observe the channel redundancy and noisy of feature representation, especially for the convolutional features. Thus, we apply a channel regularization to the correlation filter learning. Extensive experimental validations on OTB, VOT and UAV-123 datasets demonstrate that the proposed method performs favorably against state-of-the-art tracking algorithms. [ABSTRACT FROM AUTHOR]

Video prediction is challenging as real-world motion dynamics are usually multi-modally distributed. Existing stochastic methods commonly formulate random noise input with simple prior distribution, which is insufficient to model highly complex motion dynamics. This work proposes a progressive multiple granularity analysis framework to tackle the above difficulty. Firstly, to achieve coarse alignment, the input sequence is matched to prototype motion dynamics in the training set, based on self-supervised auto-encoder learning via motion/appearance disentanglement. Secondly, motion dynamics is transferred from the matched prototype sequence to input sequence via adaptively learned kernel, and the predicted frames are further refined through a motion-aware prediction model. Extensive qualitative and quantitative experiments on three widely used video prediction datasets demonstrate that: (1) the proposed framework essentially decomposes the hard task into a series of more approachable sub-tasks where a better solution is easier to be sought and (2) our proposed method performs favorably against state-of-the-art prediction methods. [ABSTRACT FROM AUTHOR]

Stereoscopic 3D (S3D) geometric distortions can be introduced by mismatches among image capture, display, and viewing configurations. In previous work of S3D geometric models, geometric distortions have been analyzed from a third-person perspective based on the binocular depth cue (i.e., binocular disparity). A third-person perspective is different from what the viewer sees since monocular depth cues (e.g., linear perspective, occlusion, and shadows) from different perspectives are different. However, depth perception in a 3D space involves both monocular and binocular depth cues. Geometric distortions that are solely predicted by the binocular depth cue cannot describe what a viewer really perceives. In this paper, we combine geometric models and retinal disparity models to analyze geometric distortions from the viewer's perspective where both monocular and binocular depth cues are considered. Results show that binocular and monocular depth-cue conflicts in a geometrically distorted S3D space. Moreover, user-initiated head translations averting from the optimal viewing position in conventional S3D displays can also introduce geometric distortions, which are inconsistent with our natural 3D viewing condition. The inconsistency of depth cues in a dynamic scene may be a source of visually induced motions sickness. [ABSTRACT FROM AUTHOR]

Being able to accurately predict the common gaze point of a group of persons is of particular interest to precise marketing and automatic group attention assessment. Group gaze estimation faces challenges including small face/head size and outlier observers. To address these challenges, we proposed a novel framework called Multi-stream Group Gaze Learning and Estimation (MUGGLE). The MUGGLE infrastructure includes two inference streams: 1) a holistic stream which utilizes fused attention map as input to a global deep convolutional structure to explore the global geometric configurations and contexts of interesting persons in the scene; and 2) an aggregative stream which robustly aggregates individual gazes via a recurrent structure (e.g., LSTM) to obtain outlier-tolerant estimation. Both streams are seamlessly integrated via a fusion network. Extensive experiments are performed on a fully annotated group gaze image dataset with 8,000+ images and 100,000+ faces (which is publicly releasable). The results demonstrate the effectiveness of the proposed MUGGLE framework in group gaze estimation. [ABSTRACT FROM AUTHOR]

Background: Horseshoe kidney (HSK) is a common renal fusion anomaly, occurring in about 1 in 400-600 individuals. In addition, the incidence of duplicated collecting system is about 0.8%.Case Presentation: This report documents an extremely rare case, which was treated by multiple procedures in the same operative session to accomplish laparoscopic amputation of the HSK isthmus, resection of duplicate kidney and ureteroscopic lithotripsy.Conclusion: Results showed that minimally invasive surgery with use of multiple endoscopes may be a feasible choice for this patient population with complicated comorbid renal conditions. [ABSTRACT FROM AUTHOR]

To explore the advantages and limitations of holmium laser resection of the bladder tumor (HOLRBT) versus standard transurethral resection of the bladder tumor (TURBT) in the treatment of non-muscle-invasive bladder cancer (NMIBC), the eligible studies were selected from the following databases: PubMed, Cochrane Library, and Embase. Studies comparing HOLRBT and TURBT for patients with NMIBC were included. The outcomes of interest were time of operation, catheterization and hospitalization, rates of recurrence, and perioperative complications, including obturator nerve reflex, bladder perforation, bladder irritation, and urethral stricture. Results of all data were compared and analyzed by Review Manager 5.3. A total of 9 comparative studies were finally included for this analysis. Pooled data demonstrated that HOLRBT significantly reduced the time to catheterization and hospitalization, the rate of recurrence in 2 years of follow-up, obturator nerve reflex, bladder perforation, and bladder irritation, compared with those in TURBT, respectively. However, no significant difference found between HOLRBT and TURBT in the time of operation, rate of recurrence in 1-year follow-up, and urethral stricture. The results of this research reached that HOLRBT would be a better choice than TURBT for patients with NMIBC. [ABSTRACT FROM AUTHOR]

The Deep learning of optical flow has been an active area for its empirical success. For the difficulty of obtaining accurate dense correspondence labels, unsupervised learning of optical flow has drawn more and more attention, while the accuracy is still far from satisfaction. By holding the philosophy that better estimation models can be trained with better-approximated labels, which in turn can be obtained from better estimation models, we propose a self-taught learning framework to continually improve the accuracy using self-generated pseudo labels. The estimated optical flow is first filtered by bidirectional flow consistency validation and occlusion-aware dense labels are then generated by edge-aware interpolation from selected sparse matches. Moreover, by combining reconstruction loss with regression loss on the generated pseudo labels, the performance is further improved. The experimental results demonstrate that our models achieve state-of-the-art results among unsupervised methods on the public KITTI, MPI-Sintel and Flying Chairs datasets. [ABSTRACT FROM AUTHOR]

Video prediction refers to predicting and generating future video frames given a set of consecutive frames. Conventional video prediction methods usually criticize the discrepancy between the ground-truth and predictions frame by frame. As the prediction error accumulates recursively, these methods would easily become out of control and are often confined to the short-term horizon. In this paper, we introduce a retrospection process to rectify the prediction errors beyond criticizing the future prediction. The introduced retrospection process is designed to look back what have been learned from the past and rectify the prediction deficiencies. To this end, we build a retrospection network to reconstruct the past frames given the currently predicted frames. A retrospection loss is introduced to push the retrospection frames being consistent with the observed frames, so that the prediction error is alleviated. On the other hand, an auxiliary route is built by reversing the flow of time and executing a similar retrospection. These two routes interact with each other to boost the performance of retrospection network and enhance the understanding of dynamics across frames, especially for the long-term horizon. An adversarial loss is employed to generate more realistic results in both prediction and retrospection process. In addition, the proposed method can be used to extend many state-of-the-art video prediction methods. Extensive experiments on the natural video dataset demonstrate the advantage of introducing the retrospection process for long-term video prediction. [ABSTRACT FROM AUTHOR]

Owning to the recorded light ray distributions, light field contains much richer information and provides possibilities of some enlightening applications, and it has becoming more and more popular. To facilitate the relevant applications, many light field processing techniques have been proposed recently. These operations also bring the loss of visual quality, and thus there is need of a light field quality metric to quantify the visual quality loss. To reduce the processing complexity and resource consumption, light fields are generally sparsely sampled, compressed, and finally reconstructed and displayed to the users. We consider the distortions introduced in this typical light field processing chain, and propose a full-reference light field quality metric. Specifically, we measure the light field quality from three aspects: global spatial quality based on view structure matching, local spatial quality based on near-edge mean square error, and angular quality based on multi-view quality analysis. These three aspects have captured the most common distortions introduced in light field processing, including global distortions like blur and blocking, local geometric distortions like ghosting and stretching, and angular distortions like flickering and sampling. Experimental results show that the proposed method can estimate light field quality accurately, and it outperforms the state-of-the-art quality metrics which may be effective for light field. [ABSTRACT FROM AUTHOR]

Audio information has been bypassed by most of current visual attention prediction studies. However, sound could have influence on visual attention and such influence has been widely investigated and proofed by many psychological studies. In this paper, we propose a novel multi-modal saliency (MMS) model for videos containing scenes with high audio-visual correspondence. In such scenes, humans tend to be attracted by the sound sources and it is also possible to localize the sound sources via cross-modal analysis. Specifically, we first detect the spatial and temporal saliency maps from the visual modality by using a novel free energy principle. Then we propose to detect the audio saliency map from both audio and visual modalities by localizing the moving-sounding objects using cross-modal kernel canonical correlation analysis, which is first of its kind in the literature. Finally we propose a new two-stage adaptive audiovisual saliency fusion method to integrate the spatial, temporal and audio saliency maps to our audio-visual saliency map. The proposed MMS model has captured the influence of audio, which is not considered in the latest deep learning based saliency models. To take advantages of both deep saliency modeling and audio-visual saliency modeling, we propose to combine deep saliency models and the MMS model via a later fusion, and we find that an average of 5% performance gain is obtained. Experimental results on audio-visual attention databases show that the introduced models incorporating audio cues have significant superiority over state-of-the-art image and video saliency models which utilize a single visual modality. [ABSTRACT FROM AUTHOR]

This paper presents a novel approach for accurate barcodes detection in real and challenging environments using compact deep neural networks. Our approach is based on Convolutional Neural Network (CNN) and neural network compression, which can detect the four vertexes coordinates of a barcode accurately and quickly. Our approach consists of four stages: ($i$) feature extraction by a base network, ($ii$) region proposal network (RPN) training, ($iii$) barcode classification and coordinates regression, and ($iv$) weights pruning and recoding. The model is trained in the first three stages and then compressed in the fourth stage to reduce the size of the trained model. In order to remove the effect of geometric distortion during barcode decoding, we add a distortion removal module to the end of the trained model. In experiments, we validate our approach on a challenging large-scale dataset. Compared with previous methods, our method can locate the coordinates of a barcode accurately and quickly and enhance decoding rate through distortion removal. In addition, the storage and memory overheads of our model are reduced through model compression, which shows great potential in industrial applications. [ABSTRACT FROM AUTHOR]

This paper presents an aggregate wavelet-predominant algorithm to measure the distortions in THz security images. The algorithm integrates a spectral-based sharpness estimator, a noise estimator derived alpha-stable model and an overall viewing experience estimator based on free-energy principle. Among them, the greater weight is assigned to the spectral-based sharpness estimator considering that the main quality factor in THz security image is sharpness. To verify the feasibility of the proposed metric, we construct the THz security image dataset including a total of 181 THz security images, and each image has the mean opinion score (MOS) collected via subjective quality evaluation experiment. Quantitative experimental results on the constructed THz security image dataset show that the aggregate wavelet-predominant estimator produces the promising overall performance for the estimation of MOS values, with PLCC, SROCC, and RMSE of 0.900, 0.873, and 0.386, respectively. This performance is superior to other opinion-unaware approaches, viz., FISBLIM, SISBLIM, NIQE, CPBD, SINE, S3, FISH, and noise estimator. The determination coefficient (${R} ^{{2}}$) of linear regression between reference and predicted MOSs is 0.81. The result of Bland–Altman analysis further validates that the aggregate wavelet-predominant estimator can substitute for the subjective IQA of THz security image, with approximately 94.5% of data points locating within the limits of agreement. For usability identification, the wavelet-predominant estimator gives the satisfactory results, with accuracy, precision, recall rate, and false positive rate of 84.0%, 79.8%, 95.0%, and 29.6%, respectively. Furthermore, the potential application perspectives of the proposed metric can refer to commercial applications (guarantee THz security images of good quality) and scientific researches (assist in software development for THz security image analysis). The dataset is available at https://doi.org/10.6084/m9.figshare.7700123.v3. Possible researches on this dataset may include the development of THz quality standards, the selection of the best display mode, the enhancement of images, the modeling of image noise, and the detection of prohibited goods. [ABSTRACT FROM AUTHOR]

Hashing has been widely used in large-scale image retrieval based on approximate nearest neighbor search. Most learning-to-hashing methods adopt a two-stage algorithm to generate binary codes. First, original images are mapped into continuous visual features. Then, binary codes are generated by quantization step or separate projection. Nevertheless, these methods are sensitive to quantization operation, i.e., thresholding. To explicitly address this issue, this study proposes a novel feature quantization scheme with a loopy recurrent neural network, called loopy residual hashing, for the purpose of high accuracy in image retrieval. Instead of one-off thresholding-based feature binarization, the proposed approach performs an iterative threshold-then-approximate operation, which calculates the quantization residual after each thresholding step and then imitates another round of binarization to further approximate the coding residual. The resulting sequences of binary codes possess higher representation accuracy and extensive experiments on image retrieval demonstrate its superior discriminative capability over the prior art. In the meantime, theoretical approximation error analysis is given. [ABSTRACT FROM AUTHOR]

Hashing is a widely adopted method based on an approximate nearest neighbor search and is used in large-scale image retrieval tasks. Conventional learning-based hashing algorithms employ end-to-end representation learning, which is a one-off technique. Because of the tradeoff between efficiency and performance, conventional learning-based hashing methods must sacrifice code length to improve performance, which increases their computational complexity. To improve the efficiency of binary codes, motivated by the “nonsalient-to-salient” attention scheme of humans, we propose a recursive hashing mechanism that maps progressively expanded salient regions to a series of binary codes. These salient regions are generated by a conventional saliency model based on bottom-up saliency-driven attention and a semantic-guided saliency model based on top-down task-driven attention. After obtaining a series of salient regions, we perform long-range temporal modeling of salient regions using a graph-based recurrent deep network to obtain more refined representative features. The later output nodes inherit aggregated information from all previous nodes and extract discriminative features from more salient regions. Therefore, this network possesses more significant information and satisfactory scalability. The proposed recursive hashing neural network, optimized by a triplet ranking loss, is end-to-end trainable. Extensive experimental results from several image retrieval benchmarks show the scalability of our method and demonstrate its strong performance compared with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Visual tracking is challenging as target objects often undergo significant appearance changes caused by deformation, abrupt motion, background clutter and occlusion. In this paper, we propose to exploit the rich hierarchical features of deep convolutional neural networks to improve the accuracy and robustness of visual tracking. Deep neural networks trained on object recognition datasets consist of multiple convolutional layers. These layers encode target appearance with different levels of abstraction. For example, the outputs of the last convolutional layers encode the semantic information of targets and such representations are invariant to significant appearance variations. However, their spatial resolutions are too coarse to precisely localize the target. In contrast, features from earlier convolutional layers provide more precise localization but are less invariant to appearance changes. We interpret the hierarchical features of convolutional layers as a nonlinear counterpart of an image pyramid representation and explicitly exploit these multiple levels of abstraction to represent target objects. Specifically, we learn adaptive correlation filters on the outputs from each convolutional layer to encode the target appearance. We infer the maximum response of each layer to locate targets in a coarse-to-fine manner. To further handle the issues with scale estimation and re-detecting target objects from tracking failures caused by heavy occlusion or out-of-the-view movement, we conservatively learn another correlation filter, that maintains a long-term memory of target appearance, as a discriminative classifier. We apply the classifier to two types of object proposals: (1) proposals with a small step size and tightly around the estimated location for scale estimation; and (2) proposals with large step size and across the whole image for target re-detection. Extensive experimental results on large-scale benchmark datasets show that the proposed algorithm performs favorably against the state-of-the-art tracking methods. [ABSTRACT FROM AUTHOR]

There is a growing demand for monitors to provide high-quality visualization with more bits representing each rendered pixel. However, since most existing images and videos are of low bit-depth (LBD), transforming LBD images to visually pleasant high bit-depth (HBD) versions is of significant value. Most existing bit-depth enhancement methods generate unsatisfactory HBD images with annoying false contour artifacts or blurry details, and some algorithms are also time-consuming. To overcome these drawbacks, we propose a bit-depth enhancement framework via concatenating all level features of deep neural networks (DNNs). A novel deep learning network is proposed based on the deep convolutional variational auto-encoders (VAEs), and skip connections that concatenate every two layers are applied to pass low-level and high-level features to consequent layers, easing the gradient vanishing problem. Meanwhile, the proposed network is optimized to generate the residual between original images and its quantized ones, which performs better than recovering HBD images directly. The experimental results show that the proposed algorithm can eliminate false contour artifacts of the recovered HBD images with low time consumption, and can achieve dramatic restoration performance gains compared with state-of-the-art methods both subjectively and objectively. [ABSTRACT FROM AUTHOR]

The visual quality of perceptions is highly correlated with the mechanisms of the human brain and visual system. Recently, the free-energy principle, which has been widely researched in brain theory and neuroscience, is introduced to quantize the perception, action, and learning in human brain. In the field of image quality assessment (IQA), on one hand, the free-energy principle can resort to the internal generative model to simulate the visual stimulus of the human beings. On the other hand, abundant psychological and neurobiological studies reveal that different frequency and orientation components of one visual stimulus arouse different neurons in the striate cortex, and the striate cortex processes visual information in the cerebral cortex. Motivated by these two aspects, a novel reduce-reference IQA metric called the multi-channel free-energy based reduced-reference quality metric is proposed in this paper. First, a two-level discrete Haar wavelet transform is used to decompose the input reference and distorted images. Next, to simulate the generative model in the human brain, the sparse representation is leveraged to extract the free-energy-based features in subband images. Finally, the overall quality metric is obtained through the support vector regressor. Extensive experimental comparisons on four benchmark image quality databases (LIVE, CSIQ, TID2008, and TID2013) demonstrate that the proposed method is highly competitive with the representative reduced-reference and classical full-reference models. [ABSTRACT FROM AUTHOR]