Search

Embedding learning (EL) and feature synthesizing (FS) are two of the popular categories of fine-grained GZSL methods. EL or FS using global features cannot discriminate fine details in the absence of local features. On the other hand, EL or FS methods exploiting local features either neglect direct attribute guidance or global information. Consequently, neither method performs well. In this paper, we propose to explore global and direct attribute-supervised local visual features for both EL and FS categories in an integrated manner for fine-grained GZSL. The proposed integrated network has an EL sub-network and a FS sub-network. Consequently, the proposed integrated network can be tested in two ways. We propose a novel two-step dense attention mechanism to discover attribute-guided local visual features. We introduce new mutual learning between the sub-networks to exploit mutually beneficial information for optimization. Moreover, we propose to compute source-target class similarity based on mutual information and transfer-learn the target classes to reduce bias towards the source domain during testing. We demonstrate that our proposed method outperforms contemporary methods on benchmark datasets.

Frost damage is one of the major concerns for crop growers as it can impact the growth of the plants and hence, yields. Early detection of frost can help farmers mitigating its impact. In the past, frost detection was a manual or visual process. Image-based techniques are increasingly being used to understand frost development in plants and automatic assessment of damage resulting from frost. This research presents a comprehensive survey of the state-of the-art methods applied to detect and analyse frost stress in plants. We identify three broad computational learning approaches i.e., statistical, traditional machine learning and deep learning, applied to images to detect and analyse frost in plants. We propose a novel taxonomy to classify the existing studies based on several attributes. This taxonomy has been developed to classify the major characteristics of a significant body of published research. In this survey, we profile 80 relevant papers based on the proposed taxonomy. We thoroughly analyse and discuss the techniques used in the various approaches, i.e., data acquisition, data preparation, feature extraction, computational learning, and evaluation. We summarise the current challenges and discuss the opportunities for future research and development in this area including in-field advanced artificial intelligence systems for real-time frost monitoring.

Missing data is a major problem in real-world datasets, which hinders the performance of data analytics. Conventional data imputation schemes such as univariate single imputation replace missing values in each column with the same approximated value. These univariate single imputation techniques underestimate the variance of the imputed values. On the other hand, multivariate imputation explores the relationships between different columns of data, to impute the missing values. Reinforcement Learning (RL) is a machine learning paradigm where the agent learns by taking actions and receiving rewards in response, to achieve its goal. In this work, we propose an RL-based approach to impute missing data by learning a policy to impute data through an action-reward-based experience. Our approach imputes missing values in a column by working only on the same column (similar to univariate single imputation) but imputes the missing values in the column with different values thus keeping the variance in the imputed values. We report superior performance of our approach, compared with other imputation techniques, on a number of datasets.

Considering the vulnerability of existing deep models in the adversarial scenario, the robustness of 3D target tracking is not guaranteed. In this paper, we present an efficient generation based adversarial attack, termed Adversary Distillation Network (AD-Net), which is able to distract a victim tracker in a single shot. In contrast to existing adversarial attacks derived from point perturbations, the proposed method designs a generative network to distill an adversarial example from a tracking template through point-wise filtration. A binary distribution encoding layer is specialized to filter points, which is modeled as a Bernoulli distribution and approximated in a differentiable formulation. To boost the performance of adversarial example generation, a feature extraction module is deployed, which leverages the PointNet++ architecture to learn hierarchical features for the template points as well as similarities with the search areas. Experimental results on the KITTI vision benchmark show that the proposed adversarial attack can effectively mislead popular deep 3D trackers.

Background Appropriate anticoagulant therapy for patients with atrial fibrillation (AF) requires assessment of stroke and bleeding risks. However, risk stratification schemas such as CHA2DS2-VASc and HAS-BLED have modest predictive capacity for patients with AF. Multilabel machine learning (ML) techniques may improve predictive performance and support decision-making for anticoagulant therapy. We compared the performance of multilabel ML models with the currently used risk scores for predicting outcomes in AF patients. Methods This was a retrospective cohort study of 9670 patients, mean age 76.9 years, 46% women, who were hospitalized with non-valvular AF, and had 1-year follow-up. The outcomes were ischemic stroke (167), major bleeding (430) admissions, all-cause death (1912) and event-free survival (7387). Discrimination and calibration of ML models were compared with clinical risk scores by area under the curve (AUC). Risk stratification was assessed using net reclassification index (NRI). Results Multilabel gradient boosting classifier chain provided the best AUCs for stroke (0.685 95% CI 0.676, 0.694), major bleeding (0.709 95% CI 0.703, 0.716) and death (0.765 95% CI 0.763, 0.768) compared to multi-layer neural networks and classifier chain using support vector machine. It provided modest performance improvement for stroke compared to AUC of CHA2DS2-VASc (0.652, NRI = 3.2%, p-value = 0.1), but significantly improved major bleeding prediction compared to AUC of HAS-BLED (0.522, NRI = 22.8%, p-value < 0.05). It also achieved greater discriminant power for death compared with AUC of CHA2DS2-VASc (0.606, p-value < 0.05). ML models identified additional risk features such as hemoglobin level, renal function. Conclusions Multilabel ML models can outperform clinical risk stratification scores for predicting the risk of major bleeding and death in non-valvular AF patients.

Biotic and abiotic plant stress (e.g., frost, fungi, diseases) can significantly impact crop production. It is thus essential to detect such stress at an early stage before visual symptoms and damage become apparent. To this end, this paper proposes a novel deep learning method, called Spectral Convolution and Channel Attention Network (SC-CAN), which exploits the difference in spectral responses of healthy and stressed crops. The proposed SC-CAN method comprises two main modules: (i) a spectral convolution module, which consists of dilated causal convolutional layers stacked in a residual manner to capture the spectral features; (ii) a channel attention module, which consists of a global pooling layer and fully connected layers that compute inter-relationship between feature map channels before scaling them based on their importance level (attention score). Unlike standard convolution, which focuses on learning local features, the dilated convolution layers can learn both local and global features. These layers also have long receptive fields, making them suitable for capturing long dependency patterns in hyperspectral data. However, because not all feature maps produced by the dilated convolutional layers are important, we propose a channel attention module that weights the feature maps according to their importance level. We used SC-CAN to classify salt stress (i.e., abiotic stress) on four datasets (Chinese Spring (CS), Aegilops columnaris (co(CS)), Ae. speltoides auchery (sp(CS)), and Kharchia datasets) and Fusarium head blight disease (i.e., biotic stress) on Fusarium dataset. Reported experimental results show that the proposed method outperforms existing state-of-the-art techniques with an overall accuracy of 83.08%, 88.90%, 82.44%, 82.10%, and 82.78% on CS, co(CS), sp(CS), Kharchia, and Fusarium datasets, respectively.

Eggplant is a popular vegetable crop. Eggplant yields can be affected by various diseases. Automatic detection and recognition of diseases is an important step toward improving crop yields. In this paper, we used a two-stream deep fusion architecture, employing CNN-SVM and CNN-Softmax pipelines, along with an inference model to infer the disease classes. A dataset of 2284 images was sourced from primary (using a consumer RGB camera) and secondary sources (the internet). The dataset contained images of nine eggplant diseases. Experimental results show that the proposed method achieved better accuracy and lower false-positive results compared to other deep learning methods (such as VGG16, Inception V3, VGG 19, MobileNet, NasNetMobile, and ResNet50).

Frost can cause irreversible damage to plant tissue and can significantly reduce yields and quality. A thorough understanding of the freezing dynamics is crucial to developing strategies for frost protection and the prevention of freezing damage. This study investigated artificial intelligence machine learning (ML) models to capture the thermodynamic patterns of freezing based on infrared thermography (IRT) imagery, which would help to automate the image analysis process in real-time or post frost events. A small-scale dataset of IRT images was collected to capture the freezing process from sample droplets containing ice-nucleating bacterium. We performed several ML experiments on the data to detect the transitions in temperatures from the images. We evaluated five popular ML models, namely support vector machines, random forest (RF), extreme gradient boosting (XGBoost), multi-layer perceptron, and convolutional neural networks. We analysed the dataset to classify adjacent temperature transitions. The results show that ML models can consistently capture the thermodynamics of frost events, i.e., ice-nucleation and freezing points on typical freezing curves. Amongst the ML models, RF and XGBoost achieved the best results, both with an average accuracy of 87–88% in classifying the temperatures. With 0.25 °C temperature transitions, RF model identified the ice nucleation and freezing points at around −2.25 °C to −2.75 °C and −4.25 °C to −4.75 °C, respectively. RF and XGBoost took about 7.3 ms and 5.5 ms time per image respectively, which indicates that these models can be used in real-time applications. Our study shows that ML models using IRT imagery can be used as an automatic real-time tool to accurately detect the critical temperatures for frost formation.

Context: Most weed species can adversely impact agricultural productivity by competing for nutrients required by high-value crops. Manual weeding is not practical for large cropping areas. Many studies have been undertaken to develop automatic weed management systems for agricultural crops. In this process, one of the major tasks is to recognise the weeds from images. However, weed recognition is a challenging task. It is because weed and crop plants can be similar in colour, texture and shape which can be exacerbated further by the imaging conditions, geographic or weather conditions when the images are recorded. Advanced machine learning techniques can be used to recognise weeds from imagery. Aims: In this paper, we have investigated five state-of-the-art deep neural networks, namely VGG16, ResNet-50, Inception-V3, Inception-ResNet-v2 and MobileNetV2, and evaluated their performance for weed recognition. Methods: We have used several experimental settings and multiple dataset combinations. In particular, we constructed a large weed-crop dataset by combining several smaller datasets, mitigating class imbalance by data augmentation, and using this dataset in benchmarking the deep neural networks. We investigated the use of transfer learning techniques by preserving the pre-trained weights for extracting the features and fine-tuning them using the images of crop and weed datasets. Key results: We found that VGG16 performed better than others on small-scale datasets, while ResNet-50 performed better than other deep networks on the large combined dataset. Conclusions: This research shows that data augmentation and fine tuning techniques improve the performance of deep learning models for classifying crop and weed images. Implications: This research evaluates the performance of several deep learning models and offers directions for using the most appropriate models as well as highlights the need for a large scale benchmark weed dataset.

Classification of sheep behaviour from a sequence of tri-axial accelerometer data has the potential to enhance sheep management. Sheep behaviour is inherently imbalanced (e.g., more ruminating than walking) resulting in underperforming classification for the minority activities which hold importance. Existing works have not addressed class imbalance and use traditional machine learning techniques, e.g., Random Forest (RF). We investigated Deep Learning (DL) models, namely, Long Short Term Memory (LSTM) and Bidirectional LSTM (BLSTM), appropriate for sequential data, from imbalanced data. Two data sets were collected in normal grazing conditions using jaw-mounted and ear-mounted sensors. Novel to this study, alongside typical single classes, e.g., walking, depending on the behaviours, data samples were labelled with compound classes, e.g., walking_grazing. The number of steps a sheep performed in the observed 10 s time window was also recorded and incorporated in the models. We designed several multi-class classification studies with imbalance being addressed using synthetic data. DL models achieved superior performance to traditional ML models, especially with augmented data (e.g., 4-Class + Steps: LSTM 88.0%, RF 82.5%). DL methods showed superior generalisability on unseen sheep (i.e., F1-score: BLSTM 0.84, LSTM 0.83, RF 0.65). LSTM, BLSTM and RF achieved sub-millisecond average inference time, making them suitable for real-time applications. The results demonstrate the effectiveness of DL models for sheep behaviour classification in grazing conditions. The results also demonstrate the DL techniques can generalise across different sheep. The study presents a strong foundation of the development of such models for real-time animal monitoring.

Context: Insects are a major threat to crop production. They can infect, damage, and reduce agricultural yields. Accurate and fast detection of insects will help insect control. From a computer algorithm point of view, insect detection from imagery is a tiny object detection problem. Handling detection of tiny objects in large datasets is challenging due to small resolution of the insects in an image, and other nuisances such as occlusion, noise, and lack of features. Aims: Our aim was to achieve a high-performance agricultural insect detector using an enhanced artificial intelligence machine learning technique. Methods: We used a YOLOv3 network-based framework, which is a high performing and computationally fast object detector. We further improved the original feature pyramidal network of YOLOv3 by integrating an adaptive feature fusion module. For training the network, we first applied data augmentation techniques to regularise the dataset. Then, we trained the network using the adaptive features and optimised the hyper-parameters. Finally, we tested the proposed network on a subset dataset of the multi-class insect pest dataset Pest24, which contains 25 878 images. Key results: We achieved an accuracy of 72.10%, which is superior to existing techniques, while achieving a fast detection rate of 63.8 images per second. Conclusions: We compared the results with several object detection models regarding detection accuracy and processing speed. The proposed method achieved superior performance both in terms of accuracy and computational speed. Implications: The proposed method demonstrates that machine learning networks can provide a foundation for developing real-time systems that can help better pest control to reduce crop damage.

Bidirectional mapping-based generalized zero-shot learning (GZSL) methods rely on the quality of synthesized features to recognize seen and unseen data. Therefore, learning a joint distribution of seen-unseen classes and preserving the distinction between seen-unseen classes is crucial for GZSL methods. However, existing methods only learn the underlying distribution of seen data, although unseen class semantics are available in the GZSL problem setting. Most methods neglect retaining seen-unseen classes distinction and use the learned distribution to recognize seen and unseen data. Consequently, they do not perform well. In this work, we utilize the available unseen class semantics alongside seen class semantics and learn joint distribution through a strong visual-semantic coupling. We propose a bidirectional mapping coupled generative adversarial network (BMCoGAN) by extending the concept of the coupled generative adversarial network into a bidirectional mapping model. We further integrate a Wasserstein generative adversarial optimization to supervise the joint distribution learning. We design a loss optimization for retaining distinctive information of seen-unseen classes in the synthesized features and reducing bias towards seen classes, which pushes synthesized seen features towards real seen features and pulls synthesized unseen features away from real seen features. We evaluate BMCoGAN on benchmark datasets and demonstrate its superior performance against contemporary methods.

The publisher regrets “Table 1, Table 2 and Table 3 should be inserted in Section 4.3”. The publisher would like to apologise for any inconvenience caused.

Deep learning models have been demonstrated vulnerable to adversarial attacks even with imperceptible perturbations. As such, the reliability of existing deep neural networks-based autonomous driving systems can suffer. However, deep 3D models have applications in various Cyber-Physical Systems (CPSs) with safety-critical requirements, particularly autonomous driving systems. In this paper, the robustness of deep 3D object detection models under adversarial point cloud perturbations has been investigated. A novel method is developed to generate 3D adversarial examples from point cloud perturbations, which are common due to the intrinsic characteristics of the data captured by 3D sensors, e.g., LiDAR. The generation of adversarial samples is supervised by a dual loss, which constitutes an adversarial loss and a perturbation loss. The adversarial loss produces a point cloud with the property of aggressiveness, while the perturbation loss enforces the produced point cloud subject to visual imperception. We demonstrate that the method can successfully attack 3D object detection models in most cases, and expose their vulnerability to physical-world attacks in the form of point cloud perturbations. We perform a thorough evaluation of popular deep 3D object detectors in an adversarial setting on the KITTI vision benchmark. Experimental results show that current deep 3D object detection models are susceptible to adversarial attacks in the context of autonomous driving, and their performances are degraded by a large margin in the presence of adversarial point clouds generated by the proposed method.

Our aim was to investigate the usefulness of machine learning approaches on linked administrative health data at the population level in predicting older patients’ one-year risk of acute coronary syndrome and death following the use of non-steroidal anti-inflammatory drugs (NSAIDs). Patients from a Western Australian cardiovascular population who were supplied with NSAIDs between 1 Jan 2003 and 31 Dec 2004 were identified from Pharmaceutical Benefits Scheme data. Comorbidities from linked hospital admissions data and medication history were inputs. Admissions for acute coronary syndrome or death within one year from the first supply date were outputs. Machine learning classification methods were used to build models to predict ACS and death. Model performance was measured by the area under the receiver operating characteristic curve (AUC-ROC), sensitivity and specificity. There were 68,889 patients in the NSAIDs cohort with mean age 76 years and 54% were female. 1882 patients were admitted for acute coronary syndrome and 5405 patients died within one year after their first supply of NSAIDs. The multi-layer neural network, gradient boosting machine and support vector machine were applied to build various classification models. The gradient boosting machine achieved the best performance with an average AUC-ROC of 0.72 predicting ACS and 0.84 predicting death. Machine learning models applied to linked administrative data can potentially improve adverse outcome risk prediction. Further investigation of additional data and approaches are required to improve the performance for adverse outcome risk prediction.

This paper proposes a novel image set classification technique based on the concept of linear regression. Unlike most other approaches, the proposed technique does not require any training. We represent the gallery image sets as subspaces in a high dimensional space. Class specific gallery subspaces are used to estimate regression models for each image in the test image set. Images of the test set are then projected onto the gallery subspaces. The residuals, calculated using the Euclidean distance between the original and the projected test images, are used as the distance metric. Three different strategies are devised to decide on the final class of the test image set. We extensively evaluated the proposed technique using both low resolution and noisy images and with less gallery data to assess the suitability of our technique for the tasks of surveillance and video-based face recognition. The experiments show that the proposed technique achieves superior classification accuracy and has a faster execution time compared with existing techniques, especially under the challenging conditions of low resolution and a limited amount of gallery and test data.

Domain adaptation aims to transfer knowledge from a domain with adequate labeled samples to a domain with scarce labeled samples. Prior research has introduced various open set domain adaptation settings in the literature to extend the applications of domain adaptation methods in real-world scenarios. This paper focuses on the type of open set domain adaptation setting where the target domain has both private (‘unknown classes’) label space and the shared (‘known classes’) label space. However, the source domain only has the ‘known classes’ label space. Prevalent distribution-matching domain adaptation methods are inadequate in such a setting that demands adaptation from a smaller source domain to a larger and diverse target domain with more classes. For addressing this specific open set domain adaptation setting, prior research introduces a domain adversarial model that uses a fixed threshold for distinguishing known from unknown target samples and lacks at handling negative transfers. We extend their adversarial model and propose a novel adversarial domain adaptation model with multiple auxiliary classifiers. The proposed multi-classifier structure introduces a weighting module that evaluates distinctive domain characteristics for assigning the target samples with weights which are more representative to whether they are likely to belong to the known and unknown classes to encourage positive transfers during adversarial training and simultaneously reduces the domain gap between the shared classes of the source and target domains. A thorough experimental investigation shows that our proposed method outperforms existing domain adaptation methods on a number of domain adaptation datasets.

Convolutional Neural Network’s (CNN’s) performance disparity on clean and corrupted datasets has recently come under scrutiny. In this work, we analyse common corruptions in the frequency domain, i.e., High Frequency corruptions (HFc, e.g., noise) and Low Frequency corruptions (LFc, e.g., blur). Although a simple solution to HFc is low-pass filtering, ReLU – a widely used Activation Function (AF), does not have any filtering mechanism. In this work, we instill low-pass filtering into the AF (LP-ReLU) to improve robustness against HFc. To deal with LFc, we complement LP-ReLU with Discrete Cosine Transform based augmentation. LP-ReLU, coupled with DCT augmentation, enables a deep network to tackle the entire spectrum of corruption. We use CIFAR-10-C and Tiny ImageNet-C for evaluation and demonstrate improvements of 5% and 7.3% in accuracy respectively, compared to the State-Of-The-Art (SOTA). We further evaluate our method’s stability on a variety of perturbations in CIFAR-10-P and Tiny ImageNet-P, achieving new SOTA in these experiments as well. To further strengthen our understanding regarding CNN’s lack of robustness, a decision space visualisation process is proposed and presented in this work.

We consider the 3D object recognition problem from the perspective of the lack of labelled data. In this paper, we propose a novel progressive conditional generative adversarial network (PC-GAN) for 3D object recognition by conditioning the input with progressive learning strategies. PC-GAN is a powerful adversarial model whose generator automatically produces realistic 3D objects with annotations, and the discriminator distinguishes them from the training distribution and recognizes their categories. We train the discriminative classifier simultaneously with the generator to predict the class label by embedding a SoftMax classifier. Progressive learning uses input samples from lower to higher resolutions to increase the generator performance gradually and produce informative objects for a certain class of objects. The key idea of adopting progressing learning is to mitigate overshoots issues of the discriminator and increase variations in the generated objects by learning progressively. This strategy helps the generator to produce more realistic synthetic objects and improve the active classification performance of the discriminator. Our proposed PC-GAN is trained for object classification in a supervised manner and the performance is evaluated on two public datasets. Experimental results demonstrate that our adversarial PC-GAN outperforms the existing volumetric discriminative classifiers in term of classification accuracy.

Hajj is a mass gathering event that takes place annually in Makkah, Saudi Arabia. Typically, around three million people participate in the event and perform rituals that involve their movements within strict space and time restrictions. Despite efforts by the Hajj organisers, such massive crowd gathering and movement cause overcrowding related problems at the Hajj sites. Several previous simulation studies on Hajj focused on the rituals individually. Tawaf, followed by Sayee, are two important rituals that are performed by all the pilgrims at the same venue on the same day. These events have a strong potential for crowd buildup and related problems. As opposed the previous works in the literature, in this paper we study these two events jointly, rather than separately. We use ExtendSim, a Discrete Event Simulation tool, to integrate the Tawaf and Sayee rituals into one model. The validated model was applied to a wide range of scenarios where different percentages of pilgrims were allocated to the various Tawaf and Sayee areas. The effect of such allocations on the time to complete Tawaf and Sayee indicate strategies for managing these two key Hajj rituals.

Missing data is a common problem faced with real-world datasets. Imputation is a widely used technique to estimate the missing data. State-of-the-art imputation approaches model the distribution of observed data to approximate the missing values. Such an approach usually models a single distribution for the entire dataset, which overlooks the class-specific characteristics of the data. Class-specific characteristics are especially useful when there is a class imbalance. We propose a new method for imputing missing data based on its class-specific characteristics by adapting the popular Conditional Generative Adversarial Networks (CGAN). Our Conditional Generative Adversarial Imputation Network (CGAIN) imputes the missing data using class-specific distributions, which can produce the best estimates for the missing values. We tested our approach on baseline datasets and achieved superior performance compared with the state-of-the-art and popular imputation approaches.

The rapid advances in Deep Learning (DL) techniques have enabled rapid detection, localisation, and recognition of objects from images or videos. DL techniques are now being used in many applications related to agriculture and farming. Automatic detection and classification of weeds can play an important role in weed management and so contribute to higher yields. Weed detection in crops from imagery is inherently a challenging problem because both weeds and crops have similar colours (‘green-on-green’), and their shapes and texture can be very similar at the growth phase. Also, a crop in one setting can be considered a weed in another. In addition to their detection, the recognition of specific weed species is essential so that targeted controlling mechanisms (e.g. appropriate herbicides and correct doses) can be applied. In this paper, we review existing deep learning-based weed detection and classification techniques. We cover the detailed literature on four main procedures, i.e., data acquisition, dataset preparation, DL techniques employed for detection, location and classification of weeds in crops, and evaluation metrics approaches. We found that most studies applied supervised learning techniques, they achieved high classification accuracy by fine-tuning pre-trained models on any plant dataset, and past experiments have already achieved high accuracy when a large amount of labelled data is available.

Generating textual descriptions of images has been an important topic in computer vision and natural language processing. A number of techniques based on deep learning have been proposed on this topic. These techniques use human-annotated images for training and testing the models. These models require a large number of training data to perform at their full potential. Collecting human generated images with associative captions is expensive and time-consuming. In this paper, we propose an image captioning method that uses both real and synthetic data for training and testing the model. We use a Generative Adversarial Network (GAN) based text to image generator to generate synthetic images. We use an attention-based image captioning method trained on both real and synthetic images to generate the captions. We demonstrate the results of our models using both qualitative and quantitative analysis on popularly used evaluation metrics. We show that our experimental results achieve two fold benefits of our proposed work: i) it demonstrates the effectiveness of image captioning for synthetic images, and ii) it further improves the quality of the generated captions for real images, understandably because we use additional images for training.

Weakly supervised semantic segmentation has been attracting increasing attention as it can alleviate the need for expensive pixel-level annotations through the use of image-level labels. Relevant methods mainly rely on the implicit object localization ability of convolutional neural networks (CNNs). However, generated object attention maps remain mostly small and incomplete. In this paper, we propose an Atrous Convolutional Feature Network (ACFN) to generate dense object attention maps. This is achieved by enhancing the context representation of image classification CNNs. More specifically, cascaded atrous convolutions are used in the middle layers to retain sufficient spatial details, and pyramidal atrous convolutions are used in the last convolutional layers to provide multi-scale context information for the extraction of object attention maps. Moreover, we propose an attentive fusion strategy to adaptively fuse the multi-scale features. Our method shows improvements over existing methods on both the PASCAL VOC 2012 and MS COCO datasets, achieving state-of-the-art performance.

Semantic segmentation is a challenging task in the absence of densely labelled data. Only relying on class activation maps (CAM) with image-level labels provides deficient segmentation supervision. Prior works thus consider pre-trained models to produce coarse saliency maps to guide the generation of pseudo segmentation labels. However, the commonly used off-line heuristic generation process cannot fully exploit the benefits of these coarse saliency maps. Motivated by the significant inter-task correlation, we propose a novel weakly supervised multi-task framework termed as AuxSegNet, to leverage saliency detection and multi-label image classification as auxiliary tasks to improve the primary task of semantic segmentation using only image-level ground-truth labels. Inspired by their similar structured semantics, we also propose to learn a cross-task global pixellevel affinity map from the saliency and segmentation representations. The learned cross-task affinity can be used to refine saliency predictions and propagate CAM maps to provide improved pseudo labels for both tasks. The mutual boost between pseudo label updating and cross-task affinity learning enables iterative improvements on segmentation performance. Extensive experiments demonstrate the effectiveness of the proposed auxiliary learning network structure and the cross-task affinity learning method. The proposed approach achieves state-of-the-art weakly supervised segmentation performance on the challenging PASCAL VOC 2012 and MS COCO benchmarks.

It is a challenging task to generate adversarial point clouds, considering the irregular structure of a point cloud, the large search space, and the requirement of imperception to humans. In this paper, a flexible adversarial point cloud generation method, named Point Dropping Network (PD-Net), is proposed, which can be trained to craft adversarial examples in a single forward pass. The network is designed to launch untargeted black-box attacks to deep 3D models through point dropping regularized by the L0 norm, in contrast to the widely adopted point perturbation methods. To enable incorporation into a deep neural network, the probability of a point to be dropped, which can be described by a Bernoulli distribution, is approximated by a hard concrete distribution. The network of PD-Net consists of an encoder and a decoder, where the former encodes geometric information of each point and the latter learns to drop points from their local features in an unsupervised way. Experiments on two popular deep 3D models (including PointNet and PointNet++) show that the proposed PD-Net degrades the recognition accuracy to a large extent and achieves a high flexibility at the same time.

An investigator needs to analyze a forensic timeline after a cybersecurity incident has occurred. Log entries from various sources are used to generate a forensic timeline. Finding the anomalous activities recorded in these log records is a difficult task if manual inspection or keyword searches are used. In this work, we propose a method for identifying anomalies in a forensic timeline. We use deep autoencoders as a machine learning technique to establish a baseline for normal activities in log files. Furthermore, we set an anomaly threshold of reconstruction value based on the constructed baseline. We then plot these anomalous events on a forensic timeline. Our experiments indicate that the proposed method achieves superior performance compared to other log anomaly detection methods with overall mean F1 score and accuracy of 94.036% and 96.720%, respectively.

Background Risk scores such as CHA2DS2-VASc and HAS-BLED are used to assess stroke and bleeding risk respectively and choose appropriate antithrombotic therapy in patients with atrial fibrillation (AF). The application of ML models may improve risk prediction and identification of potential risk factors. Objective To investigate the usefulness of ML methods in estimating one-year risk of ischemic stroke and major bleeding in patients after hospitalisation with AF. Methods We identified adults with a history of non-valvular AF or atrial flutter who were admitted to a tertiary or secondary hospital in Perth, Western Australia from 2009 to 2016 using linked clinical and administrative data. Based on all the available risk factors in the data including individual risk factors in the scores, we built ML models and compared their predictive performance [Area under the receiver operating characteristic curve (AUC)] with the standard risk scores. Results There were 9,634 patients in the study cohort with a mean age of 77 years and 46% were female. 2407 patients died (n=1636) or were readmitted for ischemic stroke (n=157) and major bleeding (n=614) within one year after the first admission. All-cause death was treated as a competing risk. Gradient Boosting Machine identified nonconventional risk factors and achieved the best prediction (ischemic stroke: AUC 0.67 vs 0.64 for CHA2DS2-VASc; major bleeding: AUC 0.66 vs 0.53 for HAS-BLED). Conclusion ML models can identify nonconventional risk factors and also outperform commonly used risk scores for predicting ischemic stroke and major bleeding in patients with AF.

The purpose of sentiment analysis is to detect an opinion or polarity in text data. We can apply such an analysis to detect negative sentiment, which represents the anomalous activities in operating system (OS) logs. Existing methods involve manual searching, predefined rules, or traditional machine learning techniques to detect such suspicious events. In this work, we propose a novel deep learning-based sentiment analysis technique to check whether there are anomalous activities in OS logs. Log messages are modeled as sentences and we identify the sentiments using the gated recurrent unit (GRU) networks. OS log datasets inherently have a class imbalance in the sense that the number of negative sentiment is much lower than that of the number of positive ones. In order to address the class imbalance, we build a GRU layer on top of a class imbalance solver using the Tomek link method. Experimental results demonstrate that the proposed method can detect anomalous events in OS logs with an overall F1 and accuracy of 99.84% and 99.93%, respectively.

In this paper, we propose a high performance Two-Stream spectral-spatial Residual Network (TSRN) for hyperspectral image classification. The first spectral residual network (sRN) stream is used to extract spectral characteristics, and the second spatial residual network (saRN) stream is concurrently used to extract spatial features. The sRN uses 1D convolutional layers to fit the spectral data structure, while the saRN uses 2D convolutional layers to match the hyperspectral spatial data structure. Furthermore, each convolutional layer is preceded by a Batch Normalization (BN) layer that works as a regularizer to speed up the training process and to improve the accuracy. We conducted experiments on three well-known hyperspectral datasets, and we compare our results with five contemporary methods across various sizes of training samples. The experimental results show that the proposed architecture can be trained with small size datasets and outperforms the state-of-the-art methods in terms of the Overall Accuracy, Average Accuracy, Kappa Value, and training time.

Geometric analysis of three-dimensional (3D) surfaces with local deformations is a challenging task, required by mobile devices. In this paper, we propose a new local feature-based method derived from diffusion geometry, including a keypoint detector named persistence-based Heat Kernel Signature (pHKS), and a feature descriptor named Heat Propagation Strips (HeaPS). The pHKS detector first constructs a scalar field using the heat kernel signature function. The scalar field is generated at a small scale to capture fine geometric information of the local surface. Persistent homology is then computed to extract all the local maxima from the scalar field, and to provide a measure of persistence. Points with a high persistence are selected as pHKS keypoints. In order to describe a keypoint, an intrinsic support region is generated by the diffusion area. This support region is more robust than its geodesic distance counterpart, and provides a local surface with adaptive scale for subsequent feature description. The HeaPS descriptor is then developed by encoding the information contained in both the spatial and temporal domains of the heat kernel. We conducted several experiments to evaluate the effectiveness of the proposed method. On the TOSCA Dataset, the HeaPS descriptor achieved a high performance in terms of descriptiveness. The feature detector and descriptor were then tested on the SHREC 2010 Feature Detection and Description Dataset, and produced results that were better than the state-of-the-art methods. Finally, their application to shape retrieval was evaluated. The proposed pHKS detector and HeaPS descriptor achieved a notable improvement on the SHREC 2014 Human Dataset.

In computer vision fields, 3D object recognition is one of the most important tasks for many real-world applications. Three-dimensional convolutional neural networks ( CNNs ) have demonstrated their advantages in 3D object recognition. In this paper, we propose to use the principal curvature directions of 3D objects ( using a CAD model ) to represent the geometric features as inputs for the 3D CNN. Our framework, namely CurveNet, learns perceptually relevant salient features and predicts object class labels. Curvature directions incorporate complex surface information of a 3D object, which helps our framework to produce more precise and discriminative features for object recognition. Multitask learning is inspired by sharing features between two related tasks, where we consider pose classification as an auxiliary task to enable our CurveNet to better generalize object label classification. Experimental results show that our proposed framework using curvature vectors performs better than voxels as an input for 3D object classification. We further improved the performance of CurveNet by combining two networks with both curvature direction and voxels of a 3D object as the inputs. A Cross-Stitch module was adopted to learn effective shared features across multiple representations. We evaluated our methods using three publicly available datasets and achieved competitive performance in the 3D object recognition task.

The aim of this paper is to classify conductive material corrosion by eddy current pulsed thermography. Thermal transient images generate a large of amount of data which is difficult for accurate detection and classification of the different corrosion materials, especially with the hidden corrosion. We apply deep Boltzmann machines (DBM) network to automatically extract and classify features from the whole measured area. Corrosion classification is tested with several different machine learning based algorithms including: clustering, PCA, multi-layer DBM classifier. The performance of the proposed framework is measured in terms of accuracy, sensitivity, specificity and precision. Several experiments are performed on a dataset of eddy current signal samples for four different corrosion degrees. The results show that our method outperforms the existing algorithms in classification accuracy (97.9%), sensitivity (96.1%), precision (97.1%), and especially specificity (98.4%).

Sound event detection (SED) has been widely applied in real world applications. Convolutional recurrent neural network based SED approaches have achieved state-of-the-art performance. However, the convolution process is typically performed by using a fixed sized kernel, which adversely affects the detection accuracy especially when the acoustic features of different event classes are characterized by high variations. To deal with this, this article proposes a sound event detection technique using a convolutional recurrent neural network framework with multiple convolutional kernels of different sizes. The top performing kernels are selected from a kernel pool based on the unsupervised clustering errors and the accuracies of the temporarily trained models. Afterwards, the selected kernels are fed to multiple convolution layers to deal with the acoustic feature variations. Experimental results on different subsets of AudioSet, namely the DCASE Challenge 2017 Task 4 and DCASE Challenge 2018 Task 4, demonstrate the performance of the proposed approach compared to state-of-the-art systems.

A forensic investigator creates a timeline from a forensic disk image after an occurrence of a security incident. This procedure aims to acquire the time for all events identified from the investigated artifacts. An investigator usually looks for events of interest by manually searching the timeline. One of the sources from which to build a timeline is log files, and these events are often found in log messages. In this paper, we propose a sentiment analysis technique to automatically extract events of interest from log messages in the forensic timeline. We use a deep learning technique with a context and content attention model to identify aspect terms and the corresponding sentiments in the forensic timeline. Terms with negative sentiments indicate events of interest and are highlighted in the timeline. Therefore, the investigator can quickly examine the events and other activities recorded within the surrounding time frame. Experimental results on four public forensic case studies show that the proposed method achieves 98.43% and 99.64% for the F1 score and accuracy, respectively.

Weakly supervised semantic segmentation with image-level labels is of great significance since it alleviates the dependency on dense annotations. However, as it relies on image classification networks that are only capable of producing sparse object localization maps, its performance is far behind that of fully supervised semantic segmentation models. Inspired by the successful use of multi-scale features for an improved performance in a wide range of visual tasks, we propose a Scale-Aware Feature Network (SAFN) for generating object localization maps. The proposed SAFN uses an attention module to learn the relative weights of multi-scale features in a modified fully convolutional network with dilated convolutions. This approach leads to efficient enlargements of the receptive fields of view and produces dense object localization maps. Our approach achieves mIoUs of 62.3% and 66.5% on the PASCAL VOC 2012 test set using VGG16 based and ResNet based segmentation models, respectively, outperforming other state-of-the-art methods for the weakly supervised semantic segmentation task.

The use of virtual environments as pedagogical tools within the domain of education has become an increasingly popular phenomenon over the last few decades especially. However, the uses and effects of virtual environments as pedagogical tools within the specific domain of engineering education have not been explored in extensive detail. The limited body of current research regarding the uses of virtual environments in engineering education affords a unique opportunity. This review provides an overview and synthesis of the uses of virtual reality within the domain of engineering education through the systematic analysis of journal articles and conference proceedings published across two databases (IEEE Xplore and Scopus) between 2015 and 2019. In summary, 17 studies were analysed utilising a systematic approach and using a coding scheme including four main categories and 12 binary items. The findings indicate an escalation in the use of virtual classroom environments as supplementary to traditional teaching environments, as well as reported substantial benefits to cognitive and skill‐based learning outcomes. The findings suggest that evaluation metrics and processes that lack clarity, together with unrealistic virtual scenarios and small evaluation sample sizes, may confound comparison and, hence, the reported substantive and beneficial use of virtual reality environments in engineering education. Scholars and educators within the domain of engineering education should continue investigating the uses and merits of virtual reality environments as pedagogical tools, taking these issues into consideration.

Acoustic event detection deals with the acoustic signals to determine the sound type and to estimate the audio event boundaries. Multi-label classification based approaches are commonly used to detect the frame wise event types with a median filter applied to determine the happening acoustic events. However, the multi-label classifiers are trained only on the acoustic event types ignoring the frame position within the audio events. To deal with this, this paper proposes to construct a joint learning based multi-task system. The first task performs the acoustic event type detection and the second task is to predict the frame position information. By sharing representations between the two tasks, we can enable the acoustic models to generalize better than the original classifier by averaging respective noise patterns to be implicitly regularized. Experimental results on the monophonic UPC-TALP and the polyphonic TUT Sound Event datasets demonstrate the superior performance of the joint learning method by achieving lower error rate and higher F-score compared to the baseline AED system.

Oceanographers rely on advanced digital imaging systems to assess the health of marine ecosystems. The majority of the imagery collected by these systems do not get annotated due to lack of resources. Consequently, the expert labeled data is not enough to train dedicated deep networks. Meanwhile, in the deep learning community, much focus is on how to use pre-trained deep networks to classify out-of-domain images and transfer learning. In this paper, we leverage these advances to evaluate how well features extracted from deep neural networks transfer to underwater image classification. We propose new image features (called ResFeats) extracted from the different convolutional layers of a deep residual network pre-trained on ImageNet. We further combine the ResFeats extracted from different layers to obtain compact and powerful deep features. Moreover, we show that ResFeats consistently perform better than their CNN counterparts. Experimental results are provided to show the effectiveness of ResFeats with state-of-the-art classification accuracies on MLC, Benthoz15, EILAT and RSMAS datasets.

Across the globe, remote image data is rapidly being collected for the assessment of benthic communities from shallow to extremely deep waters on continental slopes to the abyssal seas. Exploiting this data is presently limited by the time it takes for experts to identify organisms found in these images. With this limitation in mind, a large effort has been made globally to introduce automation and machine learning algorithms to accelerate both classification and assessment of marine benthic biota. One major issue lies with organisms that move with swell and currents, such as kelps. This paper presents an automatic hierarchical classification method local binary classification as opposed to the conventional flat classification to classify kelps in images collected by autonomous underwater vehicles. The proposed kelp classification approach exploits learned feature representations extracted from deep residual networks. We show that these generic features outperform the traditional off-the-shelf CNN features and the conventional hand-crafted features. Experiments also demonstrate that the hierarchical classification method outperforms the traditional parallel multi-class classifications by a significant margin (90.0% vs. 57.6% and 77.2% vs. 59.0%) on Benthoz15 and Rottnest datasets respectively. Furthermore, we compare different hierarchical classification approaches and experimentally show that the sibling hierarchical training approach outperforms the inclusive hierarchical approach by a significant margin. We also report an application of our proposed method to study the change in kelp cover over time for annually repeated AUV surveys.

Digital pathology has attracted significant attention in recent years. Analysis of Whole Slide Images (WSIs) is challenging because they are very large, i.e., of Giga-pixel resolution. Identifying Regions of Interest (ROIs) is the first step for pathologists to analyse further the regions of diagnostic interest for cancer detection and other anomalies. In this paper, we investigate the use of RCNN, which is a deep machine learning technique, for detecting such ROIs only using a small number of labelled WSIs for training. For experimentation, we used real WSIs from a public hospital pathology service in Western Australia. We used 60 WSIs for training the RCNN model and another 12 WSIs for testing. The model was further tested on a new set of unseen WSIs. The results show that RCNN can be effectively used for ROI detection from WSIs.

An administrator needs to examine operating system log files for any anomalous events. In real-life log data, the number of anomalies is often smaller than the normal ones. This imbalance situation affects the performance of the anomaly detectors because a large number of normal events feed the training of the classifier. In this paper, we evaluate popular machine learning methods and consider this problem of data imbalance. We compare data oversampling and undersampling approaches before inputting them to the classifier. Experimental results demonstrate that by taking data imbalance into consideration, there is an improvement in the method performance in terms of precision and recall scores.

ion of event logs is the creation of a template that contains the most common words representing all members in a group of event log entries. Abstraction helps the forensic investigators to obtain an overall view of the main events in a log file. Existing log abstraction methods require user input parameters. This manual input is time consuming due to the need to identify the best parameters, especially when a log file is large. We propose an automatic method to facilitate event log abstraction avoiding the need for the user to manually identify suitable parameters. We model event logs as a graph and propose a new graph clustering approach to group log entries. The abstraction is then extracted from each cluster. Experimental results show that the proposed method achieves superior performance compared to existing approaches with an F-measure of 95.35%.

Human actions represented with 3D skeleton sequences are robust to clustered backgrounds and illumination changes. In this paper, we investigate skeleton-based action prediction, which aims to recognize an action from a partial skeleton sequence that contains incomplete action information. We propose a new Latent Global Network based on adversarial learning for action prediction. We demonstrate that the proposed network provides latent long-term global information that is complementary to the local action information of the partial sequences and helps improve action prediction. We show that action prediction can be improved by combining the latent global information with the local action information. We test the proposed method on three challenging skeleton datasets and report state-of-the-art performance.

The use of virtual environments as pedagogical tools within the domain of education has become an increasingly popular phenomenon over the last few decades especially. However, the uses and effects of virtual environments as pedagogical tools within the specific domain of engineering education have not been explored in extensive detail. The limited body of current research regarding the uses of virtual environments in engineering education affords a unique opportunity. This review provides an overview and synthesis of the uses of virtual reality within the domain of engineering education through the systematic analysis of journal articles and conference proceedings published across two databases (IEEE Xplore and Scopus) between 2015 and 2019. In summary, 17 studies were analysed utilising a systematic approach and using a coding scheme including four main categories and 12 binary items. The findings indicate an escalation in the use of virtual classroom environments as supplementary to traditional teaching environments, as well as reported substantial benefits to cognitive and skill‐based learning outcomes. The findings suggest that evaluation metrics and processes that lack clarity, together with unrealistic virtual scenarios and small evaluation sample sizes, may confound comparison and, hence, the reported substantive and beneficial use of virtual reality environments in engineering education. Scholars and educators within the domain of engineering education should continue investigating the uses and merits of virtual reality environments as pedagogical tools, taking these issues into consideration.

We consider the recent challenges of 3D shape analysis based on a volumetric CNN that requires a huge computational power. This high-cost approach forces to reduce the volume resolutions when applying 3D CNN on volumetric data. In this context, we propose a multiorientation volumetric deep neural network (MV-DNN) for 3D object classification with octree generating low-cost volumetric features. In comparison to conventional octree representations, we propose to limit the octree partition to a certain depth to reserve all leaf octants with sparsity features. This allows for improved learning of complex 3D features and increased prediction of object labels at both low and high resolutions. Our auxiliary learning approach predicts object classes based on the subvolume parts of a 3D object that improve the classification accuracy compared to other existing 3D volumetric CNN methods. In addition, the influence of views and depths of the 3D model on the classification performance is investigated through extensive experiments applied to the ModelNet40 database. Our deep learning framework runs significantly faster and consumes less memory than full voxel representations and demonstrate the effectiveness of our octree-based auxiliary learning approach for exploring high resolution 3D models. Experimental results reveal the superiority of our MV-DNN that achieves better classification accuracy compared to state-of-art methods on two public databases.