Search

We present a solution to egocentric 3D body pose estimation from monocular images captured from downward looking fish-eye cameras installed on the rim of a head mounted VR device. This unusual viewpoint leads to images with unique visual appearance, with severe self-occlusions and perspective distortions that result in drastic differences in resolution between lower and upper body. We propose an encoder-decoder architecture with a novel multi-branch decoder designed to account for the varying uncertainty in 2D predictions. The quantitative evaluation, on synthetic and real-world datasets, shows that our strategy leads to substantial improvements in accuracy over state of the art egocentric approaches. To tackle the lack of labelled data we also introduced a large photo-realistic synthetic dataset. xR-EgoPose offers high quality renderings of people with diverse skintones, body shapes and clothing, performing a range of actions. Our experiments show that the high variability in our new synthetic training corpus leads to good generalization to real world footage and to state of theart results on real world datasets with ground truth. Moreover, an evaluation on the Human3.6M benchmark shows that the performance of our method is on par with top performing approaches on the more classic problem of 3D human pose from a third person viewpoint., 14 pages. arXiv admin note: substantial text overlap with arXiv:1907.10045

The popularity of egocentric cameras and their always-on nature has lead to the abundance of day long first-person videos. The highly redundant nature of these videos and extreme camera-shakes make them difficult to watch from beginning to end. These videos require efficient summarization tools for consumption. However, traditional summarization techniques developed for static surveillance videos or highly curated sports videos and movies are either not suitable or simply do not scale for such hours long videos in the wild. On the other hand, specialized summarization techniques developed for egocentric videos limit their focus to important objects and people. This paper presents a novel unsupervised reinforcement learning framework to summarize egocentric videos both in terms of length and the content. The proposed framework facilitates incorporating various prior preferences such as faces, places, or scene diversity and interactive user choice in terms of including or excluding the particular type of content. This approach can also be adapted to generate summaries of various lengths, making it possible to view even 1-minute summaries of one's entire day. When using the facial saliency-based reward, we show that our approach generates summaries focusing on social interactions, similar to the current state-of-the-art (SOTA). The quantitative comparisons on the benchmark Disney dataset show that our method achieves significant improvement in Relaxed F-Score (RFS) (29.60 compared to 19.21 from SOTA), BLEU score (0.68 compared to 0.67 from SOTA), Average Human Ranking (AHR), and unique events covered. Finally, we show that our technique can be applied to summarize traditional, short, hand-held videos as well, where we improve the SOTA F-score on benchmark SumMe and TVSum datasets from 41.4 to 46.40 and 57.6 to 58.3 respectively. We also provide a Pytorch implementation and a web demo at https://pravin74.github.io/Int-sum/index.html.

The performance of a convolutional neural network (CNN) heavily depends on its hyperparameters. However, finding a suitable hyperparameters configuration is difficult, challenging, and computationally expensive due to three issues, which are 1) the mixed-variable problem of different types of hyperparameters; 2) the large-scale search space of finding optimal hyperparameters; and 3) the expensive computational cost for evaluating candidate hyperparameters configuration. Therefore, this article focuses on these three issues and proposes a novel estimation of distribution algorithm (EDA) for efficient hyperparameters optimization, with three major contributions in the algorithm design. First, a hybrid-model EDA is proposed to efficiently deal with the mixed-variable difficulty. The proposed algorithm uses a mixed-variable encoding scheme to encode the mixed-variable hyperparameters and adopts an adaptive hybrid-model learning (AHL) strategy to efficiently optimize the mixed-variables. Second, an orthogonal initialization (OI) strategy is proposed to efficiently deal with the challenge of large-scale search space. Third, a surrogate-assisted multi-level evaluation (SME) method is proposed to reduce the expensive computational cost. Based on the above, the proposed algorithm is named surrogate-assisted hybrid-model EDA (SHEDA). For experimental studies, the proposed SHEDA is verified on widely used classification benchmark problems, and is compared with various state-of-the-art methods. Moreover, a case study on aortic dissection (AD) diagnosis is carried out to evaluate its performance. Experimental results show that the proposed SHEDA is very effective and efficient for hyperparameters optimization, which can find a satisfactory hyperparameters configuration for the CIFAR10, CIFAR100, and AD diagnosis with only 0.58, 0.97, and 1.18 GPU days, respectively.

In data-driven evolutionary optimization, most existing Gaussian processes (GPs)-assisted evolutionary algorithms (EAs) adopt stationary GPs (SGPs) as surrogate models, which might be insufficient for solving most optimization problems. This article finds that GPs in the optimization problems are nonstationary with great probability. We propose to employ a nonstationary GP (NSGP) surrogate model for data-driven evolutionary optimization, where the mean of the NSGP is allowed to vary with the decision variables, while its residue variance follows an SGP. In this article, the nonstationarity of GPs in the tested functions is theoretically analyzed. In addition, this article constructs an NSGP where the SGP is a degenerate case. Performance comparisons of the NSGP with the SGP and the NSGP-assisted EA (NSGP-MAEA) with the SGP-assisted EA (SGP-MAEA) are carried out on a set of benchmark problems and an antenna design problem. These comparison results demonstrate the competitiveness of the NSGP model.

Safety plays an important role in determining the applicability of energetic compounds, and the bond dissociation enthalpy (BDE) of the “trigger bond” X-NO2 provides useful information to evaluate various safety properties. Accurate and rapid calculation of the BDE of X-NO2 is of great significance to perform the high-throughput design of energetic compounds, which becomes an increasingly popular means of materials design. We conduct a benchmark BDE calculation for 44 X-NO2 samples extracted from the iBond database, with the accuracies of 55 quantum chemistry calculation levels evaluated by the experimentally measured values. Only four levels have the global mean-absolute deviation (MAD) less than 10 kJ/mol, but no calculation level can achieve that all the local MADs of each category less than 10 kJ/mol. We propose a simple correction strategy for the original calculation deviations, and apply it to 30 calculation levels screened out through a series of accuracy assessments and obtain the corrected MAD

Surrogate-assisted evolutionary algorithms (SAEAs) have been widely used for solving complex and computationally expensive optimization problems. However, most of the existing algorithms converge slowly in the later stage. This article proposes a novel two-stage data-driven evolutionary optimization (TS-DDEO) that meets the requirements of early exploration and later exploitation. In the first stage, a surrogate-assisted hierarchical particle swarm optimization method is used to find a promising area from the entire search space. In the second stage, we propose a best-data-driven optimization (BDDO) method with a strong exploitation ability to accelerate the optimization process. BDDO has a real-time update mechanism for the surrogate model and population and uses a predefined number of ranking-top solutions to update population and surrogates. BDDO combines three surrogate-assisted evolutionary sampling strategies: 1) surrogate-assisted differential evolution sampling; 2) surrogate-assisted local search; and 3) a surrogate-assisted full-crossover (FC) strategy which is proposed to integrate existing best genotypes in the population. Experiments and analysis have validated the effectiveness of the two-stage framework, the BDDO method, and the FC strategy. Moreover, the proposed algorithm is compared with five state-of-the-art SAEAs on high-dimensional benchmark functions. The result shows that TS-DDEO performs better both in effectiveness and robustness.

Detecting highly-overlapped objects in crowded scenes remains a challenging problem, especially for one-stage detector. In this paper, we extricate YOLOv4 from the dilemma in a crowd by fine-tuning its detection scheme, named YOLO-CS. Specifically, we give YOLOv4 the power to detect multiple objects in one cell. Center to our method is the carefully designed joint prediction scheme, which is executed through an assignment of bounding boxes and a joint loss. Equipped with the derived joint-object augmentation (DJA), refined regression loss (RL) and Score-NMS (SN), YOLO-CS achieves competitive detection performance on CrowdHuman and CityPersons benchmarks compared with state-of-the-art detectors at the cost of little time. Furthermore, on the widely used general benchmark COCO, YOLO-CS still has a good performance, indicating its robustness to various scenes.

Face parsing aims to assign pixel-wise semantic labels to different facial components (e.g., hair, brows, and lips) in given face images. However, directly predicting pixel-level labels for each facial component over the whole face image would obtain limited accuracy, especially for tiny facial components. To address this problem, some recent works propose to first crop tiny patches from the whole face image and then predict masks for each facial component. However, such cropping-and-segmenting strategy consists of two independent stages, which cannot be jointly optimized. Besides, as one valuable piece of information for parsing the highly structured facial components, context cues are not elaborately explored by the existing works. To address these issues, we propose a component-level refinement network (CLRNet) for precisely segmenting out each facial component. Specifically, we introduce an attention mechanism to bridge the two independent stages together and form an end-to-end trainable pipeline for face parsing. Furthermore, we incorporate the global context information into the refining process for each cropped facial component patch, providing informative cues for accurate parsing. Extensive experiments are carried out on two benchmark datasets, LFW-PL and HELEN. The results demonstrate the superiority of the proposed CLRNet over other state-of-the-art methods, especially for tiny facial components.

Camouflaged object detection is a challenging task that aims to identify objects having similar texture to the surroundings. This paper presents to amplify the subtle texture difference between camouflaged objects and the background for camouflaged object detection by formulating multiple texture-aware refinement modules to learn the texture-aware features in a deep convolutional neural network. The texture-aware refinement module computes the biased co-variance matrices of feature responses to extract the texture information, adopts an affinity loss to learn a set of parameter maps that help to separate the texture between camouflaged objects and the background, and leverages a boundary-consistency loss to explore the structures of object details. We evaluate our network on the benchmark datasets for camouflaged object detection both qualitatively and quantitatively. Experimental results show that our approach outperforms various state-of-the-art methods by a large margin.

Fault diagnostics involving the Internet of Things (IoT) sensors and edge devices is a challenging task due to their limited energy and computational capabilities. Another challenge concerning IoT sensors or devices is the incursion of noise when used in an industrial environment. The noisy samples affect the decision support system that could lead to financial and operational losses. This paper proposes a Noisy Encoder using Artificial Intelligence of Things (NEAT) architecture for fault diagnosis in IoT edge devices. NEAT combines autoencoders and Inception module to co-train the clean and noisy samples for solving the said problem. Experimental results on benchmark datasets reveal that the NEAT architecture is noise resilient in comparison to the existing works. Furthermore, we also show that the NEAT architecture has lightweight characteristics as it yields a lower number of parameters, weight storage, training, and testing times that support its real-life applicability in an Industrial IoT environment.

Unsupervised domain adaptation methods have been proposed to tackle the problem of covariate shift by minimizing the distribution discrepancy between the feature embeddings of source domain and target domain. However, the standard evaluation protocols assume that the conditional label distributions of the two domains are invariant, which is usually not consistent with the real-world scenarios such as long-tailed distribution of visual categories. In this article, the imbalanced domain adaptation (IDA) is formulated for a more realistic scenario where both label shift and covariate shift occur between the two domains. Theoretically, when label shift exists, aligning the marginal distributions may result in negative transfer. Therefore, a novel cluster-level discrepancy minimization (CDM) is developed. CDM proposes cross-domain similarity learning to learn tight and discriminative clusters, which are utilized for both feature-level and distribution-level discrepancy minimization, palliating the negative effect of label shift during domain transfer. Theoretical justifications further demonstrate that CDM minimizes the target risk in a progressive manner. To corroborate the effectiveness of CDM, we propose two evaluation protocols according to the real-world situation and benchmark existing domain adaptation approaches. Extensive experiments demonstrate that negative transfer does occur due to label shift, while our approach achieves significant improvement on imbalanced datasets, including Office-31, Image-CLEF, and Office-Home.

Continues field monitoring and searching sensor data remains an imminent element emphasizes the influence of the Internet of Things (IoT). Most of the existing systems are concede spatial coordinates or semantic keywords to retrieve the entail data, which are not comprehensive constraints because of sensor cohesion, unique localization haphazardness. To address this issue, we propose deep learning inspired sensor-rank consolidation (DLi-SRC) system that enables 3-set of algorithms. First, sensor cohesion algorithm based on Lyapunov approach to accelerate sensor stability. Second, sensor unique localization algorithm based on rank-inferior measurement index to avoid redundancy data and data loss. Third, a heuristic directive algorithm to improve entail data search efficiency, which returns appropriate ranked sensor results as per searching specifications. We examined thorough simulations to describe the DLi-SRC effectiveness. The outcomes reveal that our approach has significant performance gain, such as search efficiency, service quality, sensor existence rate enhancement by 91%, and sensor energy gain by 49% than benchmark standard approaches.

Current works on image emotion distribution learning typically extract visual representations from the holistic image or explore emotion-related regions in the image from a global-wise perspective. However, different regions of an image contribute differently to the arousal of each emotion. Existing works do not deeply explore corresponding emotion-aware regions of each emotion in the image, nor do they fully capture the relationship between each emotion-aware region and the emotion labels. In this paper, we propose a novel attention based emotion distribution learning method, which can explore the emotion-related regions of images from the perspective of each emotion category, and can conduct region relationship learning. Specifically, we introduce a semantic guided attention detection network to generate class-wise attention maps for each emotion and a global-wise attention map for the holistic image. Meanwhile, an emotional graph-based network is adopted to capture the correlation between each region and the emotion distribution. Experiments on several benchmark datasets demonstrate the superiority of the proposed method compared to related works.

In dealing with the expensive multiobjective optimization problem, some algorithms convert it into a number of single-objective subproblems for optimization. At each iteration, these algorithms conduct surrogate-assisted optimization on one or multiple subproblems. However, these subproblems may be unnecessary or resolved. Operating on such subproblems can cause server inefficiencies, especially in the case of expensive optimization. To overcome this shortcoming, we propose an adaptive subproblem selection (ASS) strategy to identify the most promising subproblems for further modeling. To better leverage the cross information between the subproblems, we use the collaborative multioutput Gaussian process surrogate to model them jointly. Moreover, the commonly used acquisition functions (also known as infill criteria) are investigated in this article. Our analysis reveals that these acquisition functions may cause severe imbalances between exploitation and exploration in multiobjective optimization scenarios. Consequently, we develop a new acquisition function, namely, adaptive lower confidence bound (ALCB), to cope with it. The experimental results on three different sets of benchmark problems indicate that our proposed algorithm is competitive. Beyond that, we also quantitatively validate the effectiveness of the ASS strategy, the CoMOGP model, and the ALCB acquisition function.

Unmanned Aerial Vehicles (UAV) have many applications in both commerce and recreation. However, irresponsibly operated UAVs will pose a threat to public safety. Therefore, developing our understanding of UAVs and their uses is of particular interest. This paper considers tracking UAVs, which provide multifaceted information around location, paths and trajectories. To facilitate research on this topic, we introduce a new benchmark, herein referred to as Anti-UAV, which provides a novel direction for UAV tracking with more than 300 video pairs containing over 580k manually annotated bounding boxes. Addressing anti-UAV research challenges could help to design anti-UAV systems, which in turn may improve surveillance. Accordingly, we have proposed a simple yet effective approach, called dual-flow semantic consistency (DFSC) is proposed for UAV tracking. Modulated by the semantic flow across video sequences, tracker learns more robust class-level semantic information and obtains more discriminative instance-level features. Experiments highlight significant performance gain with the proposed approach over state-of-the-art trackers and the challenging aspects of Anti-UAV. The Anti-UAV benchmark and the code for the proposed approach have been made publicly available at https://github.com/ucas-vg/Anti-UAV.

The state-of-the-art approaches to Targeted Aspect-Based Sentiment Analysis (TABSA) are mostly deep learning models based on attention mechanisms. One problem in most previous studies is that embeddings of targets and aspects are either pre-trained from large external corpora or randomly initialized. We argue that affective commonsense knowledge and words indicative of sentiment could be used to learn better target and aspect embeddings. We therefore propose an embedding refinement framework called RAEC (Refining Affective Embedding from Context), in which sentiment concepts extracted from affective commonsense knowledge and word relative location information are incorporated to derive context-affective embeddings. Furthermore, a sparse coefficient vector is exploited in refining the embeddings of targets and aspects separately. In this way, embeddings of targets and aspects can capture the highly relevant affective words. Experimental results on two benchmark datasets show that our framework can be easily integrated with existing embedding-based TABSA models and achieves state-of-the-art results compared to models relying on pre-trained word embeddings or built on other embedding refinement methods.

Model-free control approaches require advanced exploration-exploitation policies to achieve practical tasks such as learning to bipedal robot walk in unstructured environments. In this article, we first construct a comprehensive exploration-exploitation policy that carries quality knowledge about the long-term predictor and the control policy, and the control signal of the model-free algorithms. Therefore, the developed model-free algorithm continues exploration by adjusting its unknown parameters until the desired learning and control are accomplished. Second, we provide an utterly model-free adaptive law enriched with the exploration-exploitation policy and derived step-by-step using the exact analogy of the model-based solution. The obtained adaptive control law considers the control signal saturation and the control signal (input) delay. Performed Lyapunov stability analysis ensures the convergence of the adaptive law that can also be used for intelligent control approaches. Third, we implement the adaptive algorithm in real time on a challenging benchmark system: a fourth-order, coupled dynamics, input saturated, and time-delayed underactuated manipulator. The results show that the proposed adaptive algorithm explores larger state-action spaces and treats the vanishing gradient problem in both learning and control. Also, we notice from the results that the learning and control properties of the adaptive algorithm are optimized as required.

In power systems, Economic Power dispatch Problem (EPP) is an influential optimization problem which is a highly non-convex and non-linear optimization problem. In the current study, a novel version of Differential Evolution (NDE) is used to solve this particular problem. NDE algorithm enhances local and global search capability along with efficient utilization of time and space by making use of two elite features: selfadaptive control parameter and single population structure. The combined effect of these concepts improves the performance of Differential Evolution (DE) without compromising on quality of the solution and balances the exploitation and exploration capabilities of DE. The efficiency of NDE is validated by evaluating on three benchmark cases of the power system problem having constraints such as power balance and power generation along with nonsmooth cost function and is compared with other optimization algorithms. The Numerical outcomes uncovered that NDE performed well for all the benchmark cases and maintained a trade-off between convergence rate and efficiency.

The nonlinearity of the vehicle dynamics and the non-convexity of collision avoidance constraints pose major challenges for optimization-based trajectory planning of automated vehicles. Current solutions to these challenges are either tailored to specific traffic scenarios, simplify the vehicle dynamics, are computationally demanding, or may get stuck in local minima. This work presents a novel approach to address the aforementioned shortcomings by identifying collision-free driving corridors that represent spatio-temporal constraints for motion planning using set-based reachability analysis. We derive a suitable formulation of collision avoidance constraints from driving corridors that can be integrated into arbitrary nonlinear programs as well as (successive) convexification procedures. When combining our approach with existing motion planning methods based on continuous optimization, trajectories can be planned in arbitrary traffic situations in a computationally efficient way. We demonstrate the efficacy of our approach using scenarios from the CommonRoad benchmark suite.

Key semantics can come from everywhere on an image. Semantic alignment is a key part of few-shot learning but still remains challenging. In this paper, we design a Mixer-Based Semantic Spread (MBSS) algorithm that employs a {\it mixer} module to spread the key semantic on the whole image, so that one can directly compare the processed image pairs. We first adopt a convolutional neural network to extract features from both support and query images and separate each of them into multiple Local Descriptor-based Representations (LDRs). The LDRs are then fed into the {\it mixer} for semantic spread, where every LDR attracts complementary information from its peers. In this way, the objective semantic is made spread on the whole image in a data-driven manner. The overall pipeline is supervised by a voting-based loss, guaranteeing a good {\it mixer}. Visualization results validate the feasibility of our {\it mixer}. Comprehensive experiments on three benchmark datasets, miniImageNet, tieredImageNet, and CUB, show that our algorithm achieves the state-of-the-art performance in both $5$ -way $1$ -shot and $5$ -way $5$ -shot settings.

Customer privacy is increasingly important to marketers. High-profile breaches of databases containing sensitive customer information, and the growing need to build the infrastructure required to support analysis of big data, present nontrivial obstacles to researchers seeking individual-level customer data from firms. In this paper, we show that recent developments in machine learning may enable firms to transfer a generative model, instead of data, thus potentially obviating the process of anonymizing and sampling customer data for release, for use in a variety of analytic use cases. We demonstrate the efficacy of a specific deep learning model, Generative Adversarial Networks (GANs), in preserving desired characteristics of original data. In real-world settings, we find that GANs can double the accuracy as compared to the best benchmark methods. We also demonstrate that GANs can be used to solve marketing problems of price markups for optimal profits and customer targeting, and that a single GAN can tackle multiple marketing problems. Furthermore, GANs have volume and velocity advantages, as the size of informational transfer grows according to model complexity, and it can readily handle real-time data streams.

Weakly supervised object detection (WSOD aims to train object detectors by using only image-level annotations. Many recent works on WSOD adopt multiple instance detection networks (MIDN, which usually generate a certain number of proposals and regard proposal classification as a latent model learning within image classification. However, these methods tend to detect salient object, salient object parts and clustered objects due to lack of instance-level annotations during training. Thus a core issue is how to guarantee that the network learn as many objects with precise bounding boxes as possible. In this paper, we address this issue by exploiting the potential of proposal scores during training. We propose an adaptive instance refinement (AIR framework with three novel designs, which can be integrated with MIDN into a single network. Specifically, adaptive instance mining attempts to discover all positive instances according to the score distribution of proposals and their spatial similarity. Adaptive score modulation dynamically adjusts proposal scores to make the network focus more on instances with different difficulties in different training iterations. Adaptive knowledge refinement distills important information from all previous stages by the weighted average of proposal scores. The experimental results on the PASCAL VOC 2007 and 2012 benchmarks and the MS COCO benchmark demonstrate that AIR significantly improves the performance of the original MIDN and achieves the state-of-the-art results.

Learning from complementary labels (CLs) is a useful learning paradigm, where the CL specifies the classes that the instance does not belong to, instead of providing the ground truth as in the ordinary supervised learning scenario. In general, although it is less laborious and more efficient to collect CLs compared with ordinary labels, the less informative signal in the complementary supervision is less helpful to learn competent feature representation. Consequently, the final classifier's performance greatly deteriorates. In this article, we leverage generative adversarial networks (GANs) to derive an algorithm GAN-CL to effectively learn from CLs. In addition to the role in original GAN, the discriminator also serves as a normal classifier in GAN-CL, with the objective constructed partly with the complementary information. To further prove the effectiveness of our schema, we study the global optimality of both generator and discriminator for the GAN-CL under mild assumptions. We conduct extensive experiments on benchmark image datasets using deep models, to demonstrate the compelling improvements, compared with state-of-the-art CL learning approaches.

Mobile edge computing achieves better application responsiveness by offloading tasks from end devices to edge servers installed at the vicinity. Practical scenarios such as post-disaster rescuing and battlefield monitoring make it attractive to use end devices themselves as edge servers. This however introduces a new challenge: Due to mobility and power limitation, the set of edge servers becomes dynamic. As some servers fail, the tasks that run on them will also fail. This paper introduces a new dynamic edge computing model and conducts the first study on robust task offloading which is tolerant to h server failures. We propose online primal-dual algorithms that offload tasks as they arrive. We evaluate the performance of our robust task offloading solutions through extensive simulations based on real task sets. The results show that our proposed solutions can well handle edge dynamics and achieve near optimal throughput (above 95%) compared to the optimal offline benchmark algorithm.

Medical image report generation (MeIRG) aims at generating associated diagnosis descriptions with natural language sentences from medical images, which is essential in the computer-aided diagnosis system. Nevertheless, this task remains challenging in that medical images and linguistic expressions should be understood jointly which however show great discrepancies in the modality. To fill this visual-to-semantic gap, we propose a novel framework that follows the encoder-decoder pipeline. Our framework is characterized by encoding both deep visual and semantic embeddings through a triple-branch network (TriNet) during the encoding phase. The visual attention branch captures attended visual embeddings from medical images with the soft-attention mechanism. The medical report (MeRP) embedding branch predicts semantic report embeddings. The embedding branch of medical subject headings (MeSH) obtains semantic embeddings of related medical tags as complementary information. Then, outputs of these branches are fused and fed into a decoder for the report generation. Experimental results on two benchmark datasets have demonstrated the excellent performance of our method.

For face-related applications like face recognition and face verification, face detection is vital. Law enforcement agencies extensively use face recognition applications for the detection of known offenders. Security cameras are increasingly deployed everywhere, and these cameras’ captured data can be used for tagging offenders. This is done by offering the face detection system with input images, which are then processed. Later on, attempts will be done to tag the security cameras’ captured data with this processed input data. Effective utilization of deep learning techniques in order to efficiently carry out this task. Computer vision’s various tasks have been taken over by deep Convolutional Neural Networks (CNNs). Of late, the key paradigms for face detection are the region-based CNN (R-CNN) detection models due to their quicker processing speed and progressively better performance. Nevertheless, the huge feature set causes these models’ to be inefficient for real-time face detection. A Grammatical Evolution (GE) Region-Based Fully Conventional Neural Network (R-FCN) for face detection has been proposed in this work in which the GE heuristics are utilized for tuning the hyper-parameters. The WIDER face dataset and Face Detection Dataset and Benchmark (FDDB) dataset were employed for conducting the assessments. In comparison to other advanced paradigms, the proposed technique’s superiority is evident from the experiments conducted.

Zero-Shot Learning (ZSL) aims to recognize unseen classes that never appear during training. Recently, generative adversarial networks (GANs) have been introduced to convert ZSL into a supervised learning problem by synthesizing unseen visual features. However, since unseen classes are never experienced for the generator during training, the synthesized unseen visual features often become heavily biased towards seen classes, or sometimes there is even no meaningful class that can be assigned to them. This is known as the \textit{bias problem}. In this paper, we propose a novel method, namely Adaptive Bias-Aware GAN (ABA-GAN), to alleviate generating biased visual features. For this purpose, we build a semantic adversarial network to regularize the feature generator. Specifically, an adaptive adversarial loss is proposed to constrain the feature distributions, which avoids the generation of meaningless visual features. Meanwhile, a domain divider is presented to explicitly distinguish synthesized visual features between seen and unseen domains, such that the bias towards seen classes can be alleviated. Moreover, we propose a novel metric named bias score (BS) to explicitly quantify the degree of the strong bias. Extensive experiments on four widely used benchmark datasets demonstrate that our proposed method outperforms the state-of-the-art approaches under both ZSL and GZSL protocols.

Most existing light field saliency detection methods have achieved great success by exploiting unique light field data-focus information in focal slices. However, they process light field data in a slicewise way, leading to suboptimal results because the relative contribution of different regions in focal slices is ignored. How we can comprehensively explore and integrate focused saliency regions that would positively contribute to accurate saliency detection. Answering this question inspires us to develop a new insight. In this article, we propose a patch-aware network to explore light field data in a regionwise way. First, we excavate focused salient regions with a proposed multisource learning module (MSLM), which generates a filtering strategy for integration followed by three guidances based on saliency, boundary, and position. Second, we design a sharpness recognition module (SRM) to refine and update this strategy and perform feature integration. With our proposed MSLM and SRM, we can obtain more accurate and complete saliency maps. Comprehensive experiments on three benchmark datasets prove that our proposed method achieves competitive performance over 2-D, 3-D, and 4-D salient object detection methods. The code and results of our method are available at https://github.com/OIPLab-DUT/IEEE-TCYB-PANet.

Deep Neural Network (DNN) in recent era offers new opportunities to manage the production systems with increasing complexity. In this paper, DNN is trained with discrete event simulation and a process based association to make the job shop an autonomous one. This intelligent system operates well in complex environment with constrained time limits while making optimal decisions. The DNN is henceforth combined with constrained time limits for the process of production control. The system is implemented typically in semiconductor manufacturing industries on complex job shops of a wafer fab case. The DNN is trained in such a way that it operates in complex environment with timing constraints that ships the job in accurate way without flaws in operation. The DNN rewards the selection with reduced time constraint, which tends to operates with most critical batch list. The study therefore shows that the DNN manages well the timing constraints than the standard benchmark technique.

Constrained multiobjective optimization problems widely exist in real-world applications. To handle them, the balance between constraints and objectives is crucial, but remains challenging due to non-negligible impacts of problem types. In our context, the problem types refer particularly to those determined by the relationship between the constrained Pareto-optimal front (PF) and the unconstrained PF. Unfortunately, there has been little awareness on how to achieve this balance when faced with different types of problems. In this article, we propose a new constraint handling technique (CHT) by taking into account potential problem types. Specifically, inspired by the prior work, problems are classified into three primary types: 1) I; 2) II; and 3) III, with the constrained PF being made up of the entire, part and none of the unconstrained counterpart, respectively. Clearly, any problem must be one of the three types. For each possible type, there exists a tailored mechanism being used to handle the relationships between constraints and objectives (i.e., constraint priority, objective priority, or the switch between them). It is worth mentioning that exact problem types are not required because we just consider their possibilities in the new CHT. Conceptually, we show that the new CHT can make a tradeoff among different types of problems. This argument is confirmed by experimental studies performed on 38 benchmark problems, whose types are known, and a real-world problem (with unknown types) in search-based software engineering. Results demonstrate that within both decomposition-based and nondecomposition-based frameworks, the new CHT can indeed achieve a good tradeoff among different problem types, being better than several state-of-the-art CHTs.

In this work, we consider a network, where n distributed information sources whose states evolve according to a random process transmit their time-varying states to a remote estimator over a shared wireless channel. Each source generates packets in a decentralized manner and employs a slotted random access mechanism to transmit the packets. In particular, we are interested in networks with a large number of low-complexity devices that share low-capacity random access channels. Accordingly, we investigate update strategies for remote tracking of source states that require each update to constitute as few bits as possible. To that end, we develop update strategies requiring only one-bit of information per update. We first consider a natural benchmark update policy and reveal that the benchmark policy cannot guarantee stability under all conditions. We then introduce an improvement of the benchmark policy that employs a local cancellation strategy, which makes the system always stable. We further analyze and optimize the performance of the cancellation-enabled update policy to bound the estimation error at the receiver. Through simulations, we compare the proposed cancellation-enabled one-bit update policy with zero-wait sampling and threshold-based sampling policies that require more than one-bit of information per update. The comparisons show that the cancellation-enabled update policy at its optimal threshold level outperforms the multi-bit update policies. This suggests that the cancellation-enabled one-bit update policy could be greatly beneficial for applications where transmission power or shared channel capacity are limited.

RGB-D salient object detection aims to detect visually distinctive objects or regions from a pair of the RGB image and the depth image. State-of-the-art RGB-D saliency detectors are mainly based on convolutional neural networks but almost suffer from an intrinsic limitation relying on the labeled data, thus degrading detection accuracy in complex cases. In this work, we present a self-supervised self-ensembling network (S $^3$ Net) for semi-supervised RGB-D salient object detection by leveraging the unlabeled data and exploring a self-supervised learning mechanism. To be specific, we first build a self-guided convolutional neural network (SG-CNN) as a baseline model by developing a series of three-layer cross-model feature fusion (TCF) modules to leverage complementary information among depth and RGB modalities and formulating an auxiliary task that predicts a self-supervised image rotation angle. After that, to further explore the knowledge from unlabeled data, we assign SG-CNN to a student network and a teacher network, and encourage the saliency predictions and self-supervised rotation predictions from these two networks to be consistent on the unlabeled data. Experimental results on seven widely-used benchmark datasets demonstrate that our network quantitatively and qualitatively outperforms the state-of-the-art methods.

Deploying unmanned aerial vehicles (UAVs) in communication networks could achieve desirable degrees of freedom, but resulting from the high line-of-sight link attributes, security threats faced by UAV communications are serious. Taking that into consideration, we focus on a UAV swarm-enabled communication network in this article, where a fixed-wing UAV acts as a legitimate user and a neighboring UAV eavesdropper attempts to eavesdrop the confidential information forwarded by rotary-wing UAV relays. In particular, we jointly optimize transmit power on the base station and UAV relays, power allocation coefficient and beamforming on UAV relays, and trajectory of the fixed-wing UAV to maximize the average secrecy rate. Due to the nonconvexity of the optimization problem, the block coordinate descent method is utilized to solve it. Specifically, the overall optimizing problem is divided into four subproblems in which the transmit power, power allocation coefficient, beamforming vector, and the trajectory of user are optimized, respectively. By employing the successive convex approximate method, subproblems are transformed to convex problems which are numerically tractable. Numerical simulations validate the effectiveness of the proposed algorithm and show that compared with benchmark schemes, the considered transmission scheme with artificial noise and optimized network parameters can improve secrecy rate by about 13.8%.

Hand detection is a crucial technology for space human-robot interaction (SHRI), and the awareness of hand identities is particularly critical. However, most advanced works have three limitations: 1) the low detection accuracy of small-size objects; 2) insufficient temporal feature modeling between frames in videos; and 3) the inability of real-time detection. In the article, a temporal detector (called TA-RSSD) is proposed based on the SSD and spatiotemporal long short-term memory (ST-LSTM) for real-time detection in SHRI applications. Next, based on the online tubelet analysis, a real-time identity-awareness module is designed for multiple hand object identification. Several notable properties are described as follows: 1) the hybrid structure of the Resnet-101 and the SSD improves the detection accuracy of small objects; 2) three-level feature pyramidal structure retains rich semantic information without losing detailed information; 3) a group of the redesigned temporal attentional LSTM (TA-LSTM) is utilized for three-level feature map modeling, which effectively achieves background suppression and scale suppression; 4) low-level attention maps are used to eliminate in-class similarity between hand objects, which improves the accuracy of identity awareness; and 5) a novel association training scheme enhances the temporal coherence between frames. The proposed model is evaluated on the SHRI-VID dataset (collected according to the task requirements), the AU-AIR dataset, and the ImageNet-VID benchmark. Extensive ablation studies and comparisons on detection and identity-awareness capacities show the superiority of the proposed model. Finally, a set of actual testing is conducted on a space robot, and the results show that the proposed model achieves a real-time speed and high accuracy.

Understanding what is happening in the surveillance video is important for human-machine interface in transportation systems, where temporal language grounding is one of the key tasks, targeting at localizing the desired moment in an untrimmed video with a given sentence query that is relevant to the moment. This task is challenging due to the following reasons: 1) the requirement of understanding the video contents and query semantics comprehensively, and 2) building the bridge between the cross-modal semantics. To tackle these problems, early methods first sample video clips and then match them with the sentence to find the most relevant one. To reduce the computational complexity associated with video clip sampling, recent methods directly predict the temporal boundaries of the desired moment on the fused features of the sentence and the video frames. However, all the previous methods often learn the word-level or phrase-level features of the sentence, or directly generates the global sentence representation by attention mechanisms or graph network. However, we argue that applying only word-level or phrase-level semantic information and cross-modal interactions is not enough to fully capture the correspondence between the video and the query. To this end, we proposed a novel Multi-level Query Exploration and Interaction (MQEI) model, which explores the semantics in both the word- and phrase-level and captures the multi-level interactions between the video and the query through an attention module. Extensive experiments on two public benchmark datasets ActivityNet Captions and Charades-STA demonstrate that the proposed model can outperform all the state-of-the-art methods consistently.

Clustering algorithms based on deep neural networks have been widely studied for image analysis. Most existing methods require partial knowledge of the true labels, namely, the number of clusters, which is usually not available in practice. In this article, we propose a Bayesian nonparametric framework, deep nonparametric Bayes (DNB), for jointly learning image clusters and deep representations in a doubly unsupervised manner. In doubly unsupervised learning, we are dealing with the problem of ``unknown unknowns,'' where we estimate not only the unknown image labels but also the unknown number of labels as well. The proposed algorithm alternates between generating a potentially unbounded number of clusters in the forward pass and learning the deep networks in the backward pass. With the help of the Dirichlet process mixtures, the proposed method is able to partition the latent representations space without specifying the number of clusters a priori. An important feature of this work is that all the estimation is realized with an end-to-end solution, which is very different from the methods that rely on post hoc analysis to select the number of clusters. Another key idea in this article is to provide a principled solution to the problem of ``trivial solution'' for deep clustering, which has not been much studied in the current literature. With extensive experiments on benchmark datasets, we show that our doubly unsupervised method achieves good clustering performance and outperforms many other unsupervised image clustering methods.

It is a typical problem in the field of object detection to simultaneously detect objects with large scale variation in one image. Recently proposed state-of-the-art object detectors generally learn pyramidal feature representation to deal with the scale variation, which has been proved effective via various feature pyramid networks. However, the majority of the feature pyramid networks based on heuristic feature fusion strategies may be suboptimal, as excess human guidance will restrict the self-learning of deep neural networks. An adaptive feature pyramid is bound to provide a significant performance boost. In this paper, we propose a novel feature pyramid network named CATFPN that consists of Scale-Wise Feature Concatenation (SWFC) module and Global Context (GC) block. The SWFC module evenly distributes semantic features for each feature layer and the GC block introduces a self-attention mechanism. As a feature pyramid network, the CATFPN can be applied to any detector based on multi-scale features. We adopt the CATFPN in typical RetinaNet and Faster R-CNN detector models, without bells and whistles, achieving 1.1% AP and 0.7% AP improvements over FPN on the MS COCO benchmark, respectively. Our competitive performance reported on the test-dev subset of COCO achieves 42.3% AP.

To improve the performance of text classification, we propose text augmentation based on attention score (TABAS). We recognized that a criterion for selecting a replacement word rather than a random selection was necessary. Therefore, TABAS utilizes attention scores for text modification, processing only words with the same entity and part-of-speech tags to consider informational aspects. To verify this approach, we used two benchmark tasks. As a result, TABAS can significantly improve performance, both recurrent and convolutional neural networks. Furthermore, we confirm that it provides a practical way to develop deep-learning models by saving costs on making additional datasets.

Video frame interpolation is a challenging problem that involves various scenarios depending on the variety of foreground and background motions, frame rate, and occlusion. Therefore, generalizing across different scenes is difficult for a single network with fixed parameters. Ideally, one could have a different network for each scenario, but this will be computationally infeasible for practical applications. In this work, we propose MetaVFI, an adaptive video frame interpolation algorithm that uses additional information readily available at test time but has not been exploited in previous works. We initially show the benefits of test-time adaptation through simple fine-tuning of a network and then greatly improve its efficiency by incorporating meta-learning. Thus, we obtain significant performance gains with only a single gradient update without introducing any additional parameters. Moreover, the proposed MetaVFI algorithm is model-agnostic which can be easily combined with any video frame interpolation network. We show that our adaptive framework greatly improves the performance of baseline video frame interpolation networks on multiple benchmark datasets.

This paper develops parameter instability and structural change tests within predictive regressions for economic systems governed by persistent vector autoregressive dynamics. Specifically, in a setting where all – or a subset – of the variables may be fractionally integrated and the predictive relation may feature cointegration, we provide sup-Wald break tests that are constructed using the Local speCtruM (LCM) approach. The new tests cover both parameter variation and multiple structural changes with unknown break dates, and the number of breaks being known or unknown. We establish asymptotic limit theory for the tests, showing that it coincides with standard testing procedures. As a consequence, existing critical values for tied-down Bessel processes may be applied, without modification. We implement the new structural change tests to explore the stability of the fractionally cointegrating relation between implied- and realized volatility (IV and RV). Moreover, we assess the relative efficiency of IV forecasts against a challenging time-series benchmark constructed from high-frequency data. Unlike existing studies, we find evidence that the IV–RV cointegrating relation is unstable, and that carefully constructed time-series forecasts are more efficient than IV in capturing low-frequency movements in RV.

Multivariate time-series (MTS) clustering is a fundamental technique in data mining with a wide range of real-world applications. To date, though some approaches have been developed, they suffer from various drawbacks, such as high computational cost or loss of information. Most existing approaches are single-view methods without considering the benefits of mutual-support multiple views. Moreover, due to its data structure, MTS data cannot be handled well by most multiview clustering methods. Toward this end, we propose a consistent and specific non-negative matrix factorization-based multiview clustering (CSMVC) method for MTS clustering. The proposed method constructs a multilayer graph to represent the original MTS data and generates multiple views with a subspace technique. The obtained multiview data are processed through a novel non-negative matrix factorization (NMF) method, which can explore the view-consistent and view-specific information simultaneously. Furthermore, an alternating optimization scheme is proposed to solve the corresponding optimization problem. We conduct extensive experiments on 13 benchmark datasets and the results demonstrate the superiority of our proposed method against other state-of-the-art algorithms under a wide range of evaluation metrics.

Small object detection is the main challenge for image detection of unmanned aerial vehicles (UAVs), especially with small pixel ratios and blurred boundaries. In this paper, a one-stage detector (SF-SSD) is proposed with a new spatial cognition algorithm. The deconvolution operation is introduced to a feature fusion module, which enhances the representation of shallow features. These more representative features prove effective for small-scale object detection. Empowered by a spatial cognition method, the deep model can re-detect objects with less-reliable confidence scores. This enables the detector to improve detection accuracy significantly. Both between-class similarity and within-class similarity are fully exploited to suppress useless background information. This motivates the proposed model to take full use of semantic features in the detection process of multi-class small objects. A simplified network structure can improve the speed of object detection. The experiments are conducted on a newly collected dataset (SY-UAV) and the benchmark datasets (CARPK and PUCPR+). To further demonstrate the effectiveness of the spatial cognition module, a multi-class object detection experiment is conducted on the Stanford Drone dataset (SDD). The results show that the proposed model achieves high frame rates and better detection accuracies than the state-of-the-art methods, which are 90.1% (CAPPK), 90.8% (PUCPR+), and 91.2% (SDD).

In recent years, remarkable progress in zero-shot learning (ZSL) has been achieved by generative adversarial networks (GAN). To compensate for the lack of training samples in ZSL, a surge of GAN architectures have been developed by human experts through trial-and-error testing. Despite their efficacy, however, there is still no guarantee that these hand-crafted models can consistently achieve good performance across diversified datasets or scenarios. Accordingly, in this paper, we turn to neural architecture search (NAS) and make the first attempt to bring NAS techniques into the ZSL realm. Specifically, we propose a differentiable GAN architecture search method over a specifically designed search space for zero-shot learning, referred to as ZeroNAS. Considering the relevance and balance of the generator and discriminator, ZeroNAS jointly searches their architectures in a min-max player game via adversarial training. Extensive experiments conducted on four widely used benchmark datasets demonstrate that ZeroNAS is capable of discovering desirable architectures that perform favorably against state-of-the-art ZSL and generalized zero-shot learning (GZSL) approaches. Source code is at https://github.com/caixiay/ZeroNAS.

In plenty of real-life tasks, strongly supervised information is hard to obtain, such that there is not sufficient high-quality supervision to make traditional learning approaches succeed. Therefore, weakly supervised learning has drawn considerable attention recently. In this paper, we consider the problem of learning from incomplete and inaccurate supervision, where only a limited subset of training data is labeled but potentially with noise. This setting is challenging and of great importance but rarely studied in the literature. We notice that in many applications, the limited labeled data are usually with one-sided noise. For instance, considering the bug detection task in the software system, the identified buggy codes are indeed with defects whereas the codes that have been checked many times or newly fixed may still have other flaws due to the complexity of the system. We propose a novel method which is able to effectively alleviate the negative influence of one-sided label noise with the help of a vast number of unlabeled data. Excess risk analysis is provided as theoretical justifications on the usefulness of incomplete and one-sided inaccurate supervision. We conduct experiments on synthetic, benchmark datasets, and real-life tasks to validate the effectiveness of the proposed approach.

Recently, multi-task correlation filters has drawn much attention in the object tracking field, which utilizes the multi-task learning (MTL) approach to explore the interdependencies among deep features for object tracking. However, the existing multi-task correlation filters based method fails to consider the relations between the correlation filters. To address this problem, a novel correlation filters based visual tracking method is proposed in this paper, with the integration of multi-task convolution operators and object detection. In our method, convolution and correction filters are jointly learnt through using the MTL technique, with the purpose of exploring not only the interdependencies of deep features but also the internal relevance of the convolution filters. In addition, object detection is introduced into our algorithm to handle the problem of object missing to ensure a better performance of our tracking method. Experiments on five benchmark datasets demonstrate that the proposed visual tracking method outperforms existing state-of-the-art approaches.

Given an undirected connected graph G = ( V , E ) , the minimum linear arrangement problem (MinLA) consists in determining a permutation π ≔ ( π 1 , … , π n ) of the node-set V = { 1 , … , n } , which minimizes the sum of edge costs | π i − π j | over all edges { i , j } ∈ E . We introduce a compact quadratic formulation, prove its correctness, and generate new mixed-integer linear formulations that require a very small number of variables and constraints. The idea behind the way we model permutations allows the development of strong valid constraints to strengthen the new formulations. We also adapt some cutting plane procedures from the literature to one of the new models. Computational experiments with benchmark and new instances are very encouraging and improve state-of-the-art solution approaches from the literature.

Recent learning-based intrinsic image decomposition methods have achieved remarkable progress. However, they usually require massive ground truth intrinsic images for supervised learning, which limits their applicability on real-world images since obtaining ground truth intrinsic decomposition for natural images is very challenging. In this paper, we present an unsupervised framework that is able to learn the decomposition effectively from a single natural image by training solely with the image itself. Our approach is built upon the observations that the reflectance of a natural image typically has high internal self-similarity of patches, and a convolutional generation network tends to boost the self-similarity of an image when trained for image reconstruction. Based on the observations, an unsupervised intrinsic decomposition network (UIDNet) consisting of two fully convolutional encoder-decoder sub-networks, i.e., reflectance prediction network (RPN) and shading prediction network (SPN), is devised to decompose an image into reflectance and shading by promoting the internal self-similarity of the reflectance component, in a way that jointly trains RPN and SPN to reproduce the given image. A novel loss function is also designed to make effective the training for intrinsic decomposition. Experimental results on three benchmark real-world datasets demonstrate the superiority of the proposed method.

In this study, a self-learning discrete Jaya algorithm (SD-Jaya) is proposed to address the energy-efficient distributed no-idle flow-shop scheduling problem (FSP) in a heterogeneous factory system (HFS-EEDNIFSP) with the criteria of minimizing the total tardiness (TTD), total energy consumption (TEC), and factory load balancing (FLB). First, the mixed-integer programming model of HFS-EEDNIFSP is presented. An evaluation criterion of FLB combining the energy consumption and the completion time is introduced. Second, a self-learning operators selection strategy, in which the success rate of each operator is summarized as knowledge, is designed for guiding the selection of operators. Third, the energy-saving strategy is proposed for reducing the TEC. The energy-efficient no-idle FSP is transformed to be an energy-efficient permutation FSP to search the idle times. The speed of operations which adjacent are idle times is reduced. The effectiveness of SD-Jaya is tested on 60 benchmark instances. On the quality of the solution, the experimental results reveal that the efficacy of the SD-Jaya algorithm outperforms the other algorithms for addressing HFS-EEDNIFSP.

With the development of information and communication technologies, industrial cyber-physical systems (ICPSs) have accumulated a large amount of data, which enables us to convert data into industrial insight. However, since the industrial process of ICPS is always complicated and large-scale, the raw data only contains a few operation condition information, which brings challenges to process monitoring and control. Thus, an efficient operation condition division method for ICPS is necessary. Although many operation condition division methods have been proposed, they were mainly relying on the static characteristics and ignored how industrial process varies dynamically. Meanwhile, with industrial process running, there may exist some new operation conditions that make the operation condition division task even more difficult. In order to grasp the static and dynamic features simultaneously of industrial process and divide operation conditions accurately, we proposed an operation condition division method based on joint static and dynamic analysis with incremental learning. In detail, the slow feature analysis (SFA) and self-organizing map (SOM) network were proposed to extract the static and dynamic features jointly. Then, a division strategy was proposed to distinguish the operation condition changing points. For the new operating condition, we designed an incremental learning method based on SOM network, which can update the operation condition model in real-time. Extensive experiments including a numerical simulation, two benchmark processes and an industrial roasting process demonstrate that the proposed method can identify the operation conditions of the raw data in ICPS accurately and efficiently.

Few-shot learning, aiming to learn novel concepts from one or a few labeled examples, is an interesting and very challenging problem with many practical advantages. Existing few-shot methods usually utilize data of the same classes to train the feature embedding module and in a row, which is unable to learn adapting to new tasks. Besides, traditional few-shot models fail to take advantage of the valuable relations of the support-query pairs, leading to performance degradation. In this article, we propose a transductive relation-propagation graph neural network (GNN) with a decoupling training strategy (TRPN-D) to explicitly model and propagate such relations across support-query pairs, and empower the few-shot module the ability of transferring past knowledge to new tasks via the decoupling training. Our few-shot module, namely TRPN, treats the relation of each support-query pair as a graph node, named relational node, and resorts to the known relations between support samples, including both intraclass commonality and interclass uniqueness. Through relation propagation, the model could generate the discriminative relation embeddings for support-query pairs. To the best of our knowledge, this is the first work that decouples the training of the embedding network and the few-shot graph module with different tasks, which might offer a new way to solve the few-shot learning problem. Extensive experiments conducted on several benchmark datasets demonstrate that our method can significantly outperform a variety of state-of-the-art few-shot learning methods.