Search

Manufacturing systems envisioned for factories of the future will promote human-centricity for close collaboration in a shared working environment towards better overall productivity within the context of Industry 5.0. Robust and accurate recognition and prediction of human intentions are crucial to reliable and safe collaborative operations between humans and robots. For this purpose, this paper proposed a safety-aware human-centric collaborative assembly approach driven by function blocks, human action recognition for intention detection, and collision avoidance for safe robot control. Within the context, a deep learning-based recognition system is developed for high-accuracy human intention recognition and prediction, and an assembly feature-based approach driven by function blocks is presented for assembly execution and control. Thus, assembly features and human behaviours during assembly are formulated to support safe assembly actions. Skeleton-based human behaviours are defined as control inputs to an adaptive safety-aware scheme. The scheme includes collaborative and parallel mode-based pre-warning and obstacle avoidance approaches for a human-centric collaborative assembly system. The former is to monitor and regulate robot control modes when working in parallel with humans, and the latter uses a position-based approach to control robot actions by adaptively adjusting obstacle avoidance trajectories in a dynamic collaborative environment. The findings of this paper reveal the effectiveness of the developed system, as experimentally validated through an engine-assembly case study. [ABSTRACT FROM AUTHOR]

• The domain ontology for aircraft assembly is constructed with six main categories. • Ontology is converted into a graph structure to extract geometric information between key nodes. • Spatial attention is deployed to fuse the structural characteristics of the equipment and products. • Onto-SAGCN model is proposed to explore relationship between operating data and quality indicators. Aircraft assembly is an essential stage in the aircraft manufacturing industry, and the increasing complexity of aircraft functionality has put higher requirements for assembly quality. Various deep learning methods based on image or structural data have been used to predict assembly quality. However, these methods focus more on the structural relationships and interactions between assembled products and are difficult to adapt to scenarios where products and equipment may change. This paper proposes a graph convolutional neural network based on ontology modeling and spatial attention mechanism (Onto-SAGCN) for assembly quality analysis and prediction. Specifically, formal representations of assembly equipment, products, and assembly processes are established using ontologies, enabling the unified modeling of spatial and temporal relationships between entities and resolving the issues of data representation inconsistency. Then, the ontologies are transformed into undirected graphs, where nodes represent entities, node attributes represent feature data, and edges represent relationships or constraints between nodes. Designed to process data from the assembly process, the Onto-SAGCN model is subsequently applied to the assembly of aircraft moving wings. It predicts the diameters of holes by gathering operational data from equipment during the assembly and benchmarks these predictions against other established methods. Experimental outcomes affirm the method's elevated accuracy and dependability in forecasting assembly quality. [ABSTRACT FROM AUTHOR]

Facade parsing stands as a pivotal computer vision task with far-reaching applications in areas like architecture, urban planning, and energy efficiency. Despite the recent success of deep learning-based methods in yielding impressive results on certain open-source datasets, their viability for real-world applications remains uncertain. Real-world scenarios are considerably more intricate, demanding greater computational efficiency. Existing datasets often fall short in representing these settings, and previous methods frequently rely on extra detection models to enhance accuracy, which requires much computation cost. In this paper, we first introduce Comprehensive Facade Parsing (CFP), a dataset meticulously designed to encompass the intricacies of real-world facade parsing tasks. Comprising a total of 602 high-resolution street-view images, this dataset captures a diverse array of challenging scenarios, including sloping angles and densely clustered buildings, with painstakingly curated annotations for each image. Moreover, a new pipeline is proposed known as Revision-based Transformer Facade Parsing (RTFP). This marks the pioneering utilization of Vision Transformers (ViT) in facade parsing, and our experimental results definitively substantiate its merit. We also design Line Acquisition, Filtering, and Revision (LAFR), an efficient yet accurate revision algorithm that can improve the segment result solely from simple line detection using prior knowledge of the facade. In ECP 2011, RueMonge 2014, and our CFP, the experimental results verify the superiority of our method. The dataset and code are available at https://github.com/wbw520/RTFP. [ABSTRACT FROM AUTHOR]

Digital twin has become a critical technical solution for satellite in-orbit service and maintenance. The majority of existing research in this field has focused on constructing system-level digital twins, with various typical applications that primarily describe the operating condition of satellite subsystems. However, achieving a more meticulous portrayal of in-orbit satellites necessitates the inclusion of physical field information, e.g., the temperature field serves as a fundamental basis for characterizing the thermodynamic state and plays a crucial role in precise thermal management of a satellite. The main challenge for realizing thorough state detection and real-time response lies in the cross-level model fusion of system simulation and physical field calculation. To overcome this problem, this paper proposes a Hybrid Digital Twin (HDT) framework that seamlessly integrates the digital twin at the system-level with its physical field-level counterpart, enabling a comprehensive and detailed representation of modeling objects. Specifically, at the physical field-level, deep learning methods are utilized to construct an accurate surrogate model, offering a lightweight, computationally efficient and accelerated execution alternative to the Finite Element Analysis models. At the system-level, multi-domain unified modeling technology is employed to ensure both efficiency and accuracy of coupling simulation. Moreover, a co-simulation approach is developed on the basis of the Functional Mock-up Interface, overcoming the bottleneck of cross-level information interaction and computing time synchronization. The practical utility of the proposed framework is verified by a real-world engineering problem, i.e., satellite temperature field perception and attitude control in the context of the harsh space environment. Whole process engineering experiments demonstrate the effectiveness of the HDT, with mean absolute errors below 1 K for temperature field reconstruction and within 1° for attitude control. [ABSTRACT FROM AUTHOR]

[Display omitted] • Winning strategies from the CRDDC, with over 90 teams from 20+ countries, are analyzed. • Cross-country collaboration in the challenge develops effective road damage detection models for 6 countries. • Multidimensional analysis of dataset and model selection provides valuable insights for future research. • Information fusion of multi-country datasets and multiple views is found crucial for robust model training. • The findings empower researchers to optimize approaches for automatic road inspection globally. Monitoring road conditions is crucial for safe and efficient transportation infrastructure, but developing effective models for automatic road damage detection is challenging requiring large-scale annotated datasets. Cross-country collaboration provide access to diverse datasets and insights into factors affecting road damage detection models. This paper presents a review of winning strategies of the Crowdsensing-based Road Damage Detection Challenge (CRDDC) held in 2022 as a Big Data Cup, with 90+ teams from 20+ countries proposing solutions for six countries: India, Japan, the Czech Republic, Norway, the United States, and China. The best solution achieved an F1-score of 77 % for all six countries, which is 2.7 % better than the 2nd ranked solution. This study explores the impact of factors influencing dataset and model selection by CRDDC winners. The study's insights can guide future research in making data-related choices and developing more effective road damage detection models accounting for the diverse road conditions across different countries. [ABSTRACT FROM AUTHOR]

Turbopump is the key component of liquid rocket engine (LRE), and its working status directly affects the performance of the launch system. However, it is usually difficult to obtain abundant training samples for turbopump, which has become a major factor limiting the application of deep learning based fault diagnosis technologies. In this paper, a two-stage strategy towards turbopump fault diagnosis under both limited labeled and unlabeled data based on variational auto-encoder (VAE) and subdomain adaptation joint attention network (SAJANet) is developed. In the suggested strategy, to better coordinate the importance of different layers during domain adaptation, the joint attention mechanism that can take the relationship between different layers into consideration is proposed and embedded into a subdomain adaptation network to developed the SAJANet. Subsequently, by combining the developed SAJANet with VAE, the "Generation-Transfer" two-stage diagnosis strategy for LRE turbopump under both limited labeled and unlabeled data is developed. In the suggested two-stage strategy, the function of VAE is to preliminary expand the data volume utilizing extremely few labeled data, while the SAJANet followed by is utilized to perform transfer learning from the VAE manual samples to the actual samples. Through the above strategy, the SAJANet can be well pre-trained by the VAE data, and the introduction of the adaptation stage can further improve the model performance against actual data by fully utilizing the limited actual data. The superiority of the proposed method is verified by the data of an actual LRE turbopump. [ABSTRACT FROM AUTHOR]

Given the pervasive use of deep learning models across various domains, ensuring model security has emerged as a critical concern. This paper examines backdoor attacks, a form of security threat that compromises model output by poisoning the training data. Our investigation specifically addresses backdoor attacks on object detection models, vital for security-sensitive applications like autonomous driving and smart city systems. Consequently, such attacks on object detection models could pose significant risks to human life and property. Consequently, backdoor attacks on object detection could pose serious threats to human life and property. To elucidate this security risk, we propose and experimentally evaluate five backdoor attack methods for object detection models. The key findings are: (1) Unnecessary Object Generation : a globally embedded trigger creating false objects in the target class; (2) Partial Misclassification : a trigger causing specific class misclassification; (3) Global Misclassification : a trigger reclassifying all objects into the target class; (4) Specific Object Vanishing : a trigger causing non-detection of certain objects; (5) Object Position Shifting : a trigger causing bounding box shifts for a specific class. To assess attack effectiveness, we introduced the Attack Success Rate (ASR), which can surpass 1 in object detection tasks, thus providing a more accurate reflection of the attack impact. Experimental outcomes indicate that the ASR values of these varied backdoor attacks frequently approach or surpass 1, demonstrating our method's capacity to impact multiple objects simultaneously. Additionally, to augment trigger stealth, we introduce Backdoor Attack with Wavelet Embedding (BAWE), which discreetly embeds triggers as image watermarks in training data. This embedding method yields more natural triggers with enhanced stealth. Highly stealthy triggers are less detectable, significantly increasing the likelihood of attack success and efficacy. We have developed a Transformer-based network architecture, diverging from traditional neural network frameworks. Our experiments across various object detection datasets highlight the susceptibility of these models and the high success rate of our approaches. This vulnerability poses significant risks to digital twin systems utilizing object detection technology. Our methodology not only enhances trigger stealth but also suits dense predictive tasks and circumvents current neural network backdoor attack detection methods. The experimental findings expose key challenges in the security of object detection models, particularly when integrated with digital twins, offering new avenues for backdoor attack research and foundational insights for devising defense strategies against these attacks. • We embed watermarks as invisible triggers in training data using wavelet transform. • Our method is suited for dense prediction tasks like object detection. • We use watermarks as triggers and are not limited to standard neural network. [ABSTRACT FROM AUTHOR]

The goal of human image generation (HIG) is to synthesize a human image in a novel pose. HIG can potentially benefit various computer vision applications and engineering tasks. The recently-developed CNN-based approach applies the attention architecture to vision tasks. However, owing to the locality in CNNs, extracting and maintaining the long-range pixel interactions input images is difficult. Thus, existing human image generation methods face limited content representation. In this paper, we propose a novel human image generation framework called HIGSA that can utilize the position information from the input source image. The proposed HIGSA contains two complementary self-attention blocks to generate photo-realistic human images, named as stripe self-attention block (SSAB) and content attention block (CAB), respectively. In SSAB, this paper establishes global dependencies of human images and computes the attention map for each pixel based on its relative spatial positions concerning other pixels. In CAB, this paper introduces an effective feature extraction module to interactively enhance both person's appearance and shape feature representations. Therefore, the HIGSA framework inherently preserves the better appearance consistency and shape consistency with sharper details. Extensive experiments on mainstream datasets demonstrate that HIGSA achieves the state-of-the-art (SOTA) results. [ABSTRACT FROM AUTHOR]

Even though analog circuits comprise a small minority among all circuits, they are responsible for the vast majority of all faults. Hence, analog circuit fault diagnosis and prognosis is crucial in preventing failure and reducing unplanned downtime in industrial electronics. There are a multitude of ways that any analog circuit can fail, which leads to proportional scaling in the number of possible fault classes with circuit complexity. This paper presents an advanced design of experiments-based approach, using supersaturated and space-filling designs, to account for components that degrade in both an individual and interacting fashion, to narrow down the number of possible fault classes that must be considered. Next, a wavelet-based deep learning network called WavePHMNet is developed that can localize the circuit component(s) that is the source of degradation and estimate the value of the degraded component(s), all based solely on the output waveforms produced by the circuit. This degraded value can be used in conjunction with component degradation models to predict circuit remaining useful life. An implementation of this approach is demonstrated on three circuits: a Sallen-Key bandpass filter (7 components), a two-switch forward convertor (25 components), and a digital to analog convertor (260 components). The approach is also demonstrated experimentally on the two-switch forward convertor circuit. [ABSTRACT FROM AUTHOR]

Deep learning-based visual quality inspection methods have been increasingly adopted in product assembly process. However, the inspection for small component remains a challenging issue due to the small size and difficulty in feature extraction. Meanwhile, due to the large number of assembly scenarios of small components, the dependency on abundant manually labeled training dataset limits the further application of these inspection methods. To mitigate the situation, this paper designs a contrastive and transfer learning-based visual inspection approach to inspect the assembly state of small component. An attention-guided lightweight feature extraction and reconstruction encoder is designed to extract features of small component, of which the deconvolution operation is leveraged to recover the resolution of the feature map, and the bidirectional feature pyramid network algorithm is introduced to effectively fuse multi-scale feature information. Moreover, an unsupervised contrastive pre-training method for the designed encoder is proposed to effectively learn the visual representation of small component. With the use of a little annotated data, the pre-trained encoder can be flexibly fine-tuned and transferred to different small component assembly scenarios. The comparative experiments demonstrate the effectiveness of our method. [ABSTRACT FROM AUTHOR]

A large proportion of construction accidents are caused by workers' unsafe actions. Due to the complexity of the work environment and excessive demands of safety supervision on construction sites, many modern information technologies, such as computer vision (CV) technologies, have been increasingly applied and gradually replaced the traditional manual supervision to automatically identify unsafe behaviors from surveillance video data. Current models focus on high resolution (HR) video containing high quality features. However, challenges remain for more specific real-world construction scenarios, including but not limited to lack of high-quality data, far-field video recognition, privacy protection, and limited computing resources. To address the challenges, in this paper, a simple but effective method relying on the Teacher–Student Learning Architecture is proposed, where the teacher network assists the student network in effectively capturing useful features for downstream action recognition from extreme low resolution (eLR) video which can be easily transferred to other tasks. In addition, a Knowledge Learning Model and a similarity loss function are proposed to better guide the training of student network. In numerical experiments and case studies, our proposed training strategy achieves an top-1 accuracy of 28.53% and 71.81% on eLR HMDB dataset and eLR CMA dataset, 5.96% and 3.89% higher than the baseline model. The proposed strategy can make the automatic safety monitoring of construction works more accurate and reliable, thereby effectively reducing the accident rate and management cost. In addition, the performance of proposed strategy depends on efficient knowledge distillation methods, which may inspire future applications of CV-based deep learning models in construction management to achieve higher recognition results while maintaining low computational cost. [ABSTRACT FROM AUTHOR]

• Proposed a knowledge framework for 3D reconstruction in civil engineering. • Provided a technical framework of image-based 3D reconstruction with deep learning. • Summarized multi-scale state-of-the-practice of 3D reconstruction in civil engineering. • Pointed out eight research directions such as automated modeling, model fusion, etc. As a bridge between physical objects and as-built models, image-based 3D reconstruction performs a vital role by generating point cloud models, mesh models, textured models, and eventually BIMs from images. This study provides a quantitative and qualitative summary of image-based 3D reconstruction for civil engineering projects in the last decade. A bibliometric analysis of 286 journal papers suggested that 3D reconstruction is an interdisciplinary field that integrates photogrammetry, 3D point cloud analysis, semantic segmentation, and deep learning. Based on the analysis, we proposed a 3D reconstruction knowledge framework with three dimensions — essential elements, use phases, and reconstruction scales. The "essential elements" dimension is a technical framework of visual geometry and deep learning methods for 3D model generation. The "use phases" emphasize using 3D reconstruction techniques during the construction, operation, and maintenance phases, which are driven by the demands of visual inspection in various contexts. The "reconstruction scales" dimension synthesizes 3D reconstruction applications from the component level to the city scale with highlights of their opportunities and challenges. This 3D reconstruction knowledge framework sheds light on eight future research directions: automated modeling, model fusion, performance optimization, data fusion, enhanced virtual experience, real-time modeling, standardized reference, and in-depth deep learning research. This review can help scholars understand the present status and highlight research trends of image-based 3D reconstruction in civil engineering associated with the integration of deep learning methods. [ABSTRACT FROM AUTHOR]

Domain generalization-based fault diagnosis (DGFD) has garnered significant attention due to its ability to generalize prior diagnostic knowledge to unseen working conditions or machines. However, existing DGFD methods are generally implemented under the premise of class balance, which may not accurately reflect real-world diagnosis scenarios since fault data collected in practical engineering often exhibits severe class imbalance. To address this challenge, this paper proposes a semantic-discriminative augmentation-driven network for imbalanced domain generalization fault diagnosis. A semantic regularization-based mixup strategy is devised to synthesize sufficient reliable samples to compensate for minority classes. Subsequently, discriminative representations are acquired by minimizing the triplet loss, thereby enhancing the model's generalization capabilities. Extensive evaluations, including cross-working condition and cross-machine tasks, demonstrate the effectiveness and superiority of the proposed method. [ABSTRACT FROM AUTHOR]

Risk monitoring is a critical task in numerous industrial fields, including construction engineering. Existing approaches primarily focus on identifying the immediate incidents of unsafe events, often neglecting the latent risks arising from the high uncertainty of agents' motions. To bridge this gap and lay the basis for preemptive risk monitoring, this paper proposes a novel and effective digital twin-based motion forecasting framework that leverages cloud-edge collaboration to predict the future trajectories of workers on construction sites, while ensuring data privacy and reducing computational burden. The edge-end acquires multi-view images about the physical scene and extracts multi-scale features with a deep learning-based backbone. In the cloud-end, where the digital world is hosted, a Transformer-based trajectory tracking model is developed to construct a behavioral digital replica comprising of multiple workers' historical trajectories from the bird's-eye view perspective. Subsequently, a LSTM-based model, combined with the social pooling mechanism, is deployed to obtain spatiotemporal interactive features from the historical trajectories and predict workers' future motions. Experiments are conducted on a dataset (WHUT-MF) collected from real-world constructions sites. Our framework achieves 0.528 m in ADE and 0.84 m in FDE on entire WHUT-MF, which demonstrate that the proposed digital twin-based motion forecasting model accurately predicts the future trajectories of workers in various scenarios with potential risks. [ABSTRACT FROM AUTHOR]

A novel data exploration framework (PredMaX) for predictive maintenance is introduced in the present paper. PredMaX offers automatic time period clustering and efficient identification of sensitive machine parts by exploiting hidden knowledge in high-dimensional, unlabeled temporal data. Condition monitoring systems often provide such data, which is further analyzed by human experts or used for training predictive models. PredMaX reduces data dimensionality in two steps: An explainable deep convolutional autoencoder is applied on the data first, followed by principal component analysis. The automatic clustering is performed in the latent space of the autoencoder, ensuring higher accuracy than the clustering in the space of principal components. If clusters of normal and abnormal operation are known, the reasoning module is able to reveal the measurement channels that contributed the most to the latent representation moving from normal to abnormal operation. Beyond the detailed presentation of the PredMaX approach, the paper presents the case study of identifying the most important signals that can be used for predicting oil degradation in an industrial gearbox. The case study is performed on a data-driven basis with minimal human assistance and without preliminary knowledge of the machine. [Display omitted] • High-dimensional temporal data is clustered in the latent space of an autoencoder. • Explainable neural networks are used to identify sensitive machine parts. • Clustered data are visualized after sequential dimension reduction in a concise way. • The effectiveness of the methodology is presented in a real-life case study. [ABSTRACT FROM AUTHOR]

Surface defect detection is very important to guarantee the quality of ceramic tiles production. At present, this process is usually performed manually in the ceramic tile industry, which is low efficiency and time-consuming. For small surface defects detection of high-resolution ceramic tiles image, an intelligent detection method for surface defects of ceramic tiles based on an improved you only look once version 5 (YOLOv5) algorithm is presented. Firstly, the high-resolution ceramic tile images are cropped into slices, and the Bottleneck module in the YOLOv5s network is optimized by introducing depthwise convolution and replaced in the whole network. Then, feature extraction is performed using the improved Shufflenetv2 backbone, and an attention mechanism is added to the backbone network to improve the feature extraction ability. The path aggregation network (PAN) and Feature Pyramid Networks (FPN) neck are used to enhance the feature extraction, and finally, the YOLO head is used to identify and locate the ceramic tile defects. The multiple sliding windows detection method is proposed to detect the original ceramic tile image which is faster than the single sliding window detection method. The experimental results show that compared with the original YOLOv5s detection algorithm, the parameters of the model are reduced by 20.46 %, the floating point operations are reduced by 26.22 %, and the mean average precision (mAP) of the proposed method is 96.73 % in the ceramic tile image slice test set which has 1.93 % improvement in mAP than the original YOLOv5s. Compare with other object detection methods, the method proposed in this paper also has certain advantages. In the high-resolution ceramic tile images test set, the mAP of the proposed algorithm is 86.44 % by using the multiple sliding window detection method. The ceramic defect detection experiment has verified the feasibility of the method proposed in this paper. [ABSTRACT FROM AUTHOR]

This paper presents the application of a deep learning based model for the short-term forecasting of the electric demand in a heating, ventilation, and air conditioning system (HVAC) for the demand response programs of utility companies. The deep learning model is applied through two different approaches comparing their merits. The approaches consist of: (i) a monolithic approach that applies a single large model to forecast the target variables, and (ii) a sequential approach that consists of multiple deep learning models coupled together each targeting a specific energy load within the HVAC system. The model accuracy of both approaches is explored over two case studies applied to the same institutional building; however, the case studies differ in their data source. The first case study uses synthetic data obtained from an eQuest simulation, while the second case study uses measurement data obtained from the building automation system. Results show that the difference in forecasting error of these approaches is negligible; however, the monolithic approach required the least amount of calibration time. Next, this paper explores the application of off-site weather data applied to a building model calibrated with on-site data. The experiments demonstrated that the off-site weather data can be applied with a slight reduction in forecasting performance. [ABSTRACT FROM AUTHOR]

The current deep learning-based fault diagnosis methods have achieved excellent classification performance under supervised training, but the existence of domain discrepancy under different working conditions will greatly affect data relationship mining and fault diagnosis accuracy. Graph convolutional network (GCN) with topological structured relationships is excellent in solving these issues. In this paper, structural discrepancy and domain adversarial fusion network (SDDAFN) is proposed to learn deep high-dimensional features and eliminate distribution discrepancy across different domains. In data structure construction, the high-dimensional mapping features are first extracted accurately from the input raw signals by a feature encoder, then the output of encoder is converted into an instance graph through graph construction layer (GCL) for the convenient modeling by multi-channel kernel GCN (MCK-GCN). In network optimization, the classification loss function is introduced to optimize the discrimination boundaries of different fault types, and the integrated loss function based on structural discrepancy and domain adversarial are jointly adopted to minimize the domain distribution discrepancy. Experimental cases show the proposed method outperforms the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

• A novel method is developed for real-time monitoring of crack dynamic expansion. • An efficient benchmarking method is established to determine crack expansion. • The YOLOv7 algorithm is improved to enhance the accuracy of crack detection. • A new attention mechanism is introduced in the DeepLabv3+ algorithm to improve crack segmentation. Crack monitoring has been a hot research topic in structural health monitoring. However, the current research on deep learning-based crack image focuses more on cracks at a certain moment and ignores the full-time crack expansion details, which are crucial for more reasonable evaluation and safety quantification of concrete structures. This paper proposes a new method based on the combination of improved You Only Look Once v7 (YOLOv7) algorithm, crack expansion benchmark method, improved DeepLabv3+ algorithm, and image processing technology to monitor the whole process of crack development, including real-time crack recognition and real-time monitoring of crack dynamic expansion. The precision of the improved detection algorithm can be improved by a maximum of 5.34%, and the mean intersection over union (mIoU) of the improved segmentation algorithm can be improved by 0.15%, resulting in better segmentation results. The experimental results show that this method can efficiently and accurately achieve real-time tracking of crack dynamic expansion, especially for monitoring of tiny cracks. [ABSTRACT FROM AUTHOR]

In the Architecture, Engineering, and Construction (AEC) sector, digital twins rely on precise 3D models to convey digital information about physical structures in a virtual space. However, due to the vulnerability to measurement errors in weak-textured regions, 3D point clouds generated by conventional photometric consistency or deep learning-based Multi-View Stereo (MVS) algorithms are often incomplete or inaccurate. Therefore, this paper integrates the consistency constraint across multiple views and the inferential capacity of deep learning to propose a novel approach for refining the missing regions of the depth maps generated by photo-consistency based MVS algorithms. The proposed solution involves a cost volume pyramid-based depth completion (CVP-DC) network with three multi-level pyramid structures, which sequentially estimates and completes depth maps in a coarse-to-fine manner. A dataset that consists of input images and the corresponding depth maps generated by photo-consistency based MVS algorithms, along with output ground truth depth maps, is developed using an open DTU MVS dataset. CVP-DC demonstrates competitive performance when tested on the public DTU MVS dataset, outperforming existing MVS algorithms in terms of both completeness and accuracy. Additionally, experimental studies are conducted utilizing UAV-collected RTK (Real-Time Kinematic) images of an outdoor bridge pier to reconstruct point clouds with absolute scales. Experimental validations demonstrate the effectiveness and applicability of the proposed approach in filling uneven and incomplete depth maps, thereby enhancing the completeness of the generated point clouds. The proposed approach holds promise for establishing precise 3D models for the digital twin of the AEC sector. [ABSTRACT FROM AUTHOR]

With the constantly generation of process data in industries, process knowledge reuse and learning from process dataset has shown promising potential in machining process planning. However, existing methods cannot effectively extract different granularity process intents, and thus the process data cannot be effectively utilized. In this paper, an effective process design intent inference method of process data via integrating deep learning and grammar parsing is proposed. First, based on the projection of cutter location segments, an association approach between machining operations and machining features is proposed to extract the process intent of each machining operation. Then, a deep learning based working step process intent sequence decision-making model is proposed to estimate the probability distributions of alternative working step process intents at each time. Moreover, according to the working step process intent sequences of existing parts in an industry, the process knowledge And-Or graph (PK-AOG) is constructed by introducing the probabilistic grammar graph model. Finally, the optimal process intents of each working step are jointly obtained based on grammar parsing by taking the PK-AOG as a guide. In the experiments, the effectiveness of our approach is validated by developing a prototype system, and the results show that our approach is practical in process intents extraction. [ABSTRACT FROM AUTHOR]

Although substantial progress in safety management performance has been made in the construction industry, continuing fatalities and injuries at workplaces hinder sustainable development of this labor-intensive industry. Many machine learning approaches using different types of data such as text, image, video, and audio were adopted for safety risk analysis at construction sites. Our paper aimed to implement a hybrid review of construction safety research based upon machine learning. This hybrid review focused on various attributes from three perspectives: Nature of dataset, machine learning approach, and research topic. After the review of individual attributes, intra-relationships between attributes in each perspective and inter-relationships between attributes across the three perspectives were determined. According to risk recognition, risk prediction, and risk control, feasible research paths were developed from both intra-relationships and inter-relationships between multiple attributes for reference in future studies. Finally, gaps and opportunities were discussed in detail for research agendas on this subject. This hybrid review contributes to outlining the framework of construction safety management based upon machine learning. It is able to provide new entrants with a systematic idea of promising research trends for the future. Research findings are helpful for academia and industry to fill in the gaps between study and practice in the area of construction safety, in order to assist in sustainable development of the construction industry by use of machine learning. [ABSTRACT FROM AUTHOR]

In the latest years, deep learning has been massively used to face problems that have not been solved by means of classical approaches. In particular, an autoencoder is a popular unsupervised artificial neural network that learns efficient data representations (encoding) by training the network to ignore features with a small content of information. Even though autoencoders over-perform classical techniques in several applications like anomaly detection, dimensionality reduction, features denoising, and missing values imputation, the literature does not provide a commonly accepted methodology to define the optimal amount of data needed to train the model. This paper proposes a procedure to determine the optimal train-set size to minimize the reconstruction error of an autoencoder with pre-defined structure and hyper-parameters that will be trained to encode the normal behavior of energy generation systems. This procedure exploits the outcome of learning curves, a powerful tool to track algorithms performance while the train-set dimension varies. Afterward, the procedure is applied to three real case studies where two types of autoencoders are trained to learn the normal behavior of a YANMAR combined heat and power unit with the scope of detecting incoming anomalies. In the end, the outcomes of the procedure are explained and, under the constraint of a daily retraining frequency, 6 weeks are identified as the optimal train-set size for both autoencoders. • Autoencoder maps input data to a latent representation. • Autoencoder learns the normal behavior of a system and detects anomalies. • Optimal autoencoder's train-set size can be defined using the proposed methodology. • The proposed methodology is applied to three real case studies: three YANMAR CHPs. [ABSTRACT FROM AUTHOR]

This paper proposes a new model, called rotor finite element mimetic neural network (RFEMNN), for diagnosing rotor unbalance and shaft crack faults. RFEMNN uses a convolution long short-term memory networks (CNN-LSTM) architecture to achieve fault recognition and localization in the rotor structure. The term "Mimetic" refers to the topology of the rotor finite element model and solution process by custom designing NN structure and data flow, which is based on the physics informed structure paradigm. The proposed "Mimetic" theory enhances the physics consistency of the RFEMNN model without requiring complete analytic physical knowledge or all parameter values. The proposed model's effectiveness is validated through experimental tests, compared to 9 benchmark models from the literature and one deep learning model with the same depth but no physics embedding. Results showed that RFEMNN outperformed the benchmark models with a comprehensive diagnostics accuracy of 94.7%, indicating its potential for practical industrial applications. [ABSTRACT FROM AUTHOR]

• An innovative partly explainable deep learning model has been proposed for tool wear intelligent monitoring. • The proposed Conv-PhyFormer model is a novel model driven by both physical and data. • The embedded hard physical constraint and soft physical constraint in the model make the model partially interpretable and credible. • The proposed model shows a significantly superior high prediction accuracy and robustness. • The proposed model has superior ability to learn global and local sensitive tool wear features. With the trend of lightweight in the field of intelligent electric vehicles and 3C, the demand for high precision machining of aluminum alloy parts is growing. And tool condition monitoring (TCM) is very important for quality control of parts, so intelligent high-accuracy wear prediction of aluminum alloy high precision machining tools has great industrial application value at present and in the future. This paper presents a novel TCM model (Conv-PhyFormer) of Transformer with physics informed. The model has excellent ability to capture short-term and long-term dependencies from nonlinear cutting time series data when there are few training samples. The embedded hard physical constraint and soft physical constraint in the model make the model partially interpretable. Soft physical constraint in the form of one-dimensional causal convolution can help the proposed model better learn the local context. Hard physical constraint in the form of the mathematical equation representing cutting physical knowledge are embedded, thus the model does not need to learn this knowledge from time series data from scratch. A large number of analysis results of aluminum alloy machining experimental data show that the proposed Conv-PhyFormer has significantly superior prediction accuracy and robustness compared with the current three popular deep learning models for TCM. Embedded soft and hard physical constraints can significantly reduce the training epochs of Transformer prediction model. [ABSTRACT FROM AUTHOR]

Prognostics and health management (PHM) of turbine cutting tools, particularly the remaining useful life (RUL) prediction is a Gordian technique to maintain the reliability and availability of the turbine, which is also a method to reduce unexpected downtime and total machined costs. Nowadays, most of the recent works in RUL prediction neglect the quality of data used for model training, assuming that the collected sensor data and extracted features can both reflect the degradation process well, which is not always true. This paper firstly introduces a novel cumulative descriptor to better represent the degradation process of cutting tools and further proposes a well-designed stacking-based deep neural network (S-DNN) that contains several different neuron architectures and an ensemble model to construct deep learning layer and ensemble learning layer. Cumulative descriptors are constructed only using directly measured values extracted from raw sensor data, and then the constructed novel features will be fed into S-DNN to extract abstract features from sensor data in multiple ways and automatically learn the relationship between the outputs of deep learning layer and the ground truth by training the ensemble model iteratively. Case studies on experimental data set and NASA Ames milling data set indicate that the proposed method has great effectiveness and generalization in RUL prediction tasks. [ABSTRACT FROM AUTHOR]

Machine learning (ML) and deep learning (DL) are both branches of AI. As a form of AI, ML automatically adapts to changing datasets with minimal human interference. Deep learning is a subset of ML that uses artificial neural networks to imitate the learning process of the human brain. The 'black box' nature of ML and DL makes their inner workings difficult to understand and interpret. Deploying explainable artificial intelligence (XAI) can help explain why and how the output of ML and DL models are generated. As a result, understanding a model's functioning, behavior, and outputs can be garnered, reducing bias and error and improving confidence in decision-making. Despite providing an improved understanding of model outputs, XAI has received limited attention in construction. This paper presents a narrative review of XAI and a taxonomy of precepts and models to raise awareness about its potential opportunities for use in construction. It is envisaged that the opportunities suggested can stimulate new lines of inquiry to help alleviate the prevailing skepticism and hesitancy toward AI adoption and integration in construction. [ABSTRACT FROM AUTHOR]

Malignancy is the tendency of a medical condition to progressively become fatal and generally refers to the presence of cancerous cells that can spread, invade, and destroy tissues. In the United States, a new cancer diagnosis is given every 30 s, and the rates of diagnosis are not indicating any slowdown. The number of cancer cases is projected to increase by 35% over the next decade. Increasing cancer cases have attracted researchers to find new approaches for their identification and characterization. Skin diseases are a significant healthcare burden, affecting millions of people globally. The early detection and diagnosis of skin diseases are essential to prevent adverse health consequences, including the development of malignant skin tumors. Dermoscopy-based diagnosis of skin diseases is a time-consuming process, particularly in resource-limited healthcare settings. Recent advancements in computer vision algorithms have led to the development of several smartphone-based applications for early skin disease prognosis. In this paper, we have utilized a set of 16 distinctive convolutional neural network models, based on deep learning, to process over 45,000 high-quality images from the HAM10000 dataset, encompassing seven categories of skin diseases. Our models were able to effectively identify and classify skin diseases using lesion images as input. Dermoscopy-based diagnosis of skin diseases is a time-consuming process, particularly in resource-limited healthcare settings. Our proposed application for early skin disease detection aims to prevent unintended health consequences, which can result in wastage of operating hours, resources, and, in some cases, compromised patient health. We have benchmarked our models against one another to evaluate their effectiveness and efficiency, using performance measures such as accuracy, precision, sensitivity, specificity, F-score, and G-Mean. Our results demonstrate that most deployed models achieve an accuracy of up to 99%. [ABSTRACT FROM AUTHOR]

Many existing automated compliance checking (ACC) systems require the processes of extracting regulatory information from natural-language building-code requirements and transforming the extracted information into computer-processable semantic representations. These processes could, however, be jeopardized by the ambiguous nature of the natural language and the hierarchically complex structures of building-code requirements. To address this problem, this paper proposes the concept of intelligent building code for bypassing the error-prone information extraction and transformation processes. In the proposed intelligent code, the natural-language requirements in the code are connected with highly structured computer-understandable semantic information, which is represented in the form of semantic requirement hierarchies and can be readily used by computers for ACC. The paper also proposes a deep learning-based method to automatically generate such intelligent code. The method leverages the requirement hierarchy representation, a proposed deep learning unit-to-text model for generating requirement sentence segments, and a proposed semantic correspondence score for configuring the segments into requirement sentences. The method was implemented and tested on a dataset from multiple regulatory documents. The generated intelligent requirements were evaluated in terms of both natural-language requirement comprehensibility and correspondence between the natural language and the semantic representation, with the results indicating high performance for the proposed representation and method. The proposed intelligent code will help reduce ACC errors, improve requirement comprehensibility, and facilitate intelligent code analytics. [ABSTRACT FROM AUTHOR]

Defect inspection plays an essential role in ensuring quality of industrial products. The most widely used human visual inspection method has some drawbacks such as high cost and low efficiency, which bring an eager demand for the application of automatic defect inspection algorithm in actual production. However, few industrial production lines use automatic detection devices due to the gap between data collected in the actual production environment and ready-made datasets. Lace is one of the industrial products which completely depends on manual defect inspection. The complex and fine texture of lace makes it difficult to extract regular patterns using the existing image-based defect inspection methods. In this paper, we propose to collect lace videos in the weaving stage and design a deep-learning-based anomaly detection framework to detect lace defects. The framework contains three stages, namely video pre-processing stage, pixel reconstruction stage and pixel classification stage. In the offline phase, only defect-free lace videos are needed to train the pixel reconstruction model and calculate the detection threshold by our adaptive thresholding method. In the online phase, the proposed framework reconstructs lace videos and performs defect inspection using reconstruction error and the pre-set threshold. As far as we know, this paper the first to detect fabric defects by videos. Experimental results on artificial defect videos demonstrate the effectiveness of the proposed framework. [ABSTRACT FROM AUTHOR]

Highlights • Develop a deep learning network (DLN) for immediate assessment of stiffness losses. • Validate the DLN model against a full-scale 3-story real-world building structure. • Methods comparison demonstrates the capability of DLN in longer term monitoring. Abstract Hysteresis loop analysis (HLA) has proven an effective indicator of damage detection in civil engineering structural health monitoring (SHM). In this paper, the histogram of stiffness (HOS) features are extracted from segregated half cycles of hysteresis loops reconstructed from measured response. A deep learning network (DLN) is proposed with the use of the HOS to classify the damage index (DI) based on stiffness degradation for damage identification. Training data are obtained using numerical simulations of 30,000 realistic, randomly created hysteresis loops, including a wide range of typical linear and nonlinear structural behaviours. Performance of the trained DLN model is assessed using both 1800 additional simulated 3-story "virtual" buildings and experimental data from a 3-story full-scale real building. Results are compared to the validated HLA method. Validation on simulated virtual building data yields prediction accuracy for 97.2% and 91.6% samples without and with 10% added noise, respectively. The comparison shows a good match of trend and percentage stiffness drop between DLN and HLA identification with the average difference for all cases within 1.1–4.6%, indicating a good accuracy of the proposed DLN prediction model for real structures. The overall results show its potential to provide a rapid, and real-time alarm or other notice on damage states and mitigation to emergency response using DLN and thus without detailed engineering analysis. [ABSTRACT FROM AUTHOR]

Graphical abstract Abstract Increasing sustainability requirements make evaluating different design options for identifying energy-efficient design ever more important. These requirements demand simulation models that are not only accurate but also fast. Machine Learning (ML) enables effective mimicry of Building Performance Simulation (BPS) while generating results much faster than BPS. Component-Based Machine Learning (CBML) enhances the capabilities of the monolithic ML model. Extending monolithic ML approach, the paper presents deep-learning architectures, component development methods and evaluates their suitability for space exploration in building design. Results indicate that deep learning increases the performance of models over simple artificial neural network models. Methods such as transfer learning and Multi-Task Learning make the component development process more efficient. Testing the deep-learning model on 201 new design cases indicates that its cooling energy prediction (R2: 0.983) is similar to BPS, while errors for heating energy predictions (R2: 0.848) are higher than BPS. Higher heating energy prediction error can be resolved by collecting heating data using better design space sampling methods that cover the heating demand distribution effectively. Given that the accuracy of the deep-learning model for heating predictions can be increased, the major advantage of deep-learning models over BPS is their high computation speed. BPS required 1145 s to simulate 201 design cases. Using the deep-learning model, similar results can be obtained in 0.9 s. High computation speed makes deep-learning models suitable for design space exploration. [ABSTRACT FROM AUTHOR]

The colored paintings on the surfaces of ancient Chinese buildings have suffered from the wind and the sun and produced many defects such as paint loss, blurring, and color distortion. Previous methods have not been able to virtually repair them well. Therefore, this paper proposed a virtual restoration method for the weathered beams in the Forbidden City using multiple deep learning algorithms. Instead of using only one technology to restore paintings, this paper divided the painting into 3 parts, i.e., the background, the golden edges, and the dragon patterns, and restored them in different technology. For the background, this paper transformed the problem of unrecognizable color restoration into a semantic segmentation problem using U-Net MobileNet. For the golden edges, this paper used traditional image processing technology to obtain them from the color maps generated by the semantic segmentation algorithm. For the dragon patterns, after sketching the skeletons according to the dragon patterns, the image translation algorithm Pix2pix was applied to generate a realistic dragon pattern. Finally, the three repair results are superimposed to complete the repair. The virtually restored paintings can provide reference and guidance for traditional manual restoration and help the restorers to imagine what they looked like before oxidation, thus alleviating some repetitive work and reducing the complexity of restoration. Besides, each step of the method could form a layer, so that the restorers could overlay or modify them as they wish. [ABSTRACT FROM AUTHOR]

For facility management, photography is an efficient and accurate method of recording the physical state of infrastructure. However, without an effective organizational scheme, the difficulty of retrieving relevant photos from historical databases can become overly burdensome for highly complex or long-lived assets. To make strategic decisions, it is crucial to retrieve the right information from a plurality of sources in a timely manner. The main objective of this paper is to present a method for organizing and retrieving photos from massive facility management photo databases using photo-metadata: photographed location, camera perspective, and image semantic content information. Indoor localization experiments were performed using Bluetooth technology to infer the location information. Perspective is inferred from the device's on-board inertial measurement unit (IMU). Image semantic content is inferred using a Convolutional Neural Network (CNN)-based deep learning algorithm. Fusing these three features, seven query options were provided for the user when retrieving images. Leveraging Building Information Modeling (BIM) as a process and Geographic Information Systems (GIS) as a framework, this paper also envisions a federated information management by connecting 2D and 3D facility assets with our real-world map which can be smoothly bridged with our image retrieval system. The realization of the integrated application with BIM and GIS is significantly beneficial for the facility management domain by advancing the understanding of projects in a broader view with a federated data platform. In this research, the framework is illustrated with 21 institutional buildings within the University of Texas at Austin's main campus, and the authors conclude that the proposed metadata-based image retrieval system can ultimately enhance the better-informed decision-making process through rapid information retrieval. [ABSTRACT FROM AUTHOR]

A worldwide challenge is to provide medical resources required for COVID-19 detection. They must be effective tools for fast detection and diagnose of the virus using a large number of tests; besides, they should be low-cost developments. While a chest X-ray scan is a powerful candidate tool, if several tests are carried out, the images produced by the devices must be interpreted accurately and rapidly. COVID-19 induces longitudinal pulmonary parenchymal ground-glass and consolidates pulmonary opacity, in some cases with rounded morphology and peripheral lung distribution, which is very difficult to predict in an early stage. In this paper, we aim to develop a robust model to extract high-level features of COVID-19 from chest X-ray (CXR) images to help in rapid diagnosis. In specific, this paper proposes an optimization model for COVID-19 diagnosis based on adaptive Fuzzy C-means (AFCM) and improved Slime Mould Algorithm (SMA) based on Lévy distribution, namely AFCM-LSMA. The SMA optimizer is proposed to adapt weights in oscillation mode and to mimic the process of generating positive and negative feedback from the propagation wave to shape the optimum path for food connectivity. Lévy motion is used as a permutation to perform a local search and to adapt SMA optimizer (LSMA) by generating several solutions that are apart from current candidates. Furthermore, it permits the optimizer to escape from local minima, examine large search areas and reach optimal solutions in fewer iterations with high convergence speed. The FCM algorithm is used to segment pulmonary regions from CXR images and is adapted to reduce time and amount of computations using histogram of the image intensities during the clustering process. The performance of the proposed AFCM-LSMA has been validated on CXR images and compared with different conventional machine learning and deep learning techniques, meta-heuristics methods, and different chaotic maps. The accuracies achieved by the proposed model are around (ACC = 0.96, RMSE = 0.23, Prec. = 0.98, F1_score = 0.98, MCC = 0.79, and Kappa = 0.79). The experimental findings indicate that the proposed new method outperforms all other methods, which will be beneficial to the clinical practitioner for the early identification of infected COVID-19 patients. [ABSTRACT FROM AUTHOR]

Soft sensors have been widely used in industrial processes over the past two decades because they use easy-to-measure process variables to predict difficult-to-measure ones. Some success has been achieved by the dominant traditional methods of modeling soft sensors based on statistics, such as principal components analysis (PCA) and partial least square (PLS), but such sensors usually become inaccurate and inefficient when processing strong nonlinear data. In this paper, a new soft sensor modeling approach is proposed based on a deep learning network. First, stacked auto-encoders (SAEs) are employed to extract high-level feature representations of the input data. In the process of training each layer of a SAE, the Limited-memory Broyden-Fletcher-Goldfarb-Shanno algorithm (L-BFGS) is adopted to optimize the weights parameters. Then, a support vector regression (SVR) is added to predict the target value on the basis of the features obtained from the SAE. To improve the model performance, Genetic Algorithm (GA) is used to obtain the optimal parameters of the SVR. To evaluate the proposed method, a soft sensor model for estimating the rotor deformation of air preheaters in a thermal power plant boiler is studied. The experimental results demonstrate that the soft sensor based on the SAE-SVR algorithm is more effective than the existing methods are. [ABSTRACT FROM AUTHOR]

Detection of building objects in airborne LiDAR data is an essential task in many types of geospatial data applications such as urban reconstruction and damage assessment. Traditional approaches used in building detection often rely on shape primitives that can be detected by 2D/3D computer vision techniques. These approaches require carefully engineered features which tend to be specific to building types. Furthermore, these approaches are often computationally expensive with the increase of data size. In this paper, we propose a novel approach that employs a deep neural network to recognize and extract residential building objects in airborne LiDAR data. This proposed approach does not require any pre-defined geometric or texture features, and it is applicable to airborne LiDAR data sets with varied point densities and with damaged building objects. The latter makes our approach particularly useful in damage assessment applications. The research results show that the proposed approach is capable of achieving the state -of-the- art accuracy in building detection in these different types of point cloud data sets. [ABSTRACT FROM AUTHOR]

This paper proposes a novel hybrid scheme through read-first-LSTM (RLSTM) encoder-decoder and broad learning system (BLS) for bearings degradation monitoring and remaining useful life (RUL) estimation, which aims to describe the nonlinear characteristics of the degradation process. Firstly, the raw signals are processed premier by complete ensemble empirical mode decomposition with adaptive noise (CEEMDAN) and a novel dimensionality reduction method composed of t-distribution stochastic neighbor embedding (t-SNE) and density-based spatial clustering of application with noise algorithm (DBSCAN). Then, the health indicator is constructed with the Hilbert-Huang transform (HHT) corresponding to the bearings' natural fault frequency, which can be employed as the hybrid scheme training label. Linear rectification technology (LRT) and exponentially weighted moving average (EWMA) control chart are adapted to define the exact process of the degradation. Secondly, a novel RLSTM is proposed. And simultaneously, an encoder-decoder model, where RLSTM is utilized as an encoder, and LSTM is adopted as a decoder, is designed for degradation monitoring. Finally, a broad learning system (BLS), which differs from deep learning with a deeper structure, is established in a flat network to estimate the RUL of bearings. Compared with the state-of-the-art techniques, the better efficacy of the proposed hybrid scheme is illustrated using the PRONOSTIA platform dataset. [ABSTRACT FROM AUTHOR]

This paper presents a novel denoising approach based on deep learning and signal processing to improve communication efficiency. Construction activities take place when different trades come to the site for overlapped periods to perform their works, which may easily produce hazardous noise levels. The existence of noise affects workers' health issues, especially hearing and rhythm of the heart, and impacts communication efficiency between workers. The proposed approach employs signal processing technique to transform the noisy audio into image and utilize neural networks to extract noisy features and denoise the image. The denoised image is then converted to obtain the denoised audio. Experiments on reducing the side effect of several common noises in construction sites were conducted, compared with the performance of denoising using conventional wavelet transform. Standard objective measures, such as signal-to-noise ratio (SNR), and subjective measures, such as listening tests are used for evaluations. Our experimental results show that the proposed algorithm achieved significant improvements over the traditional method, as evidenced by the following quantitative results of median value: MSE of 0.002, RMSE of 0.049, SNR of 5.7 dB, PSNR of 25.8 dB, and SSR of 8.Results indicate that the proposed algorithm outperforms conventional denoising methods in terms of both objective and subjective evaluation metrics and have the potential to facilitate communication between site workers when facing different noise sources inevitably. [ABSTRACT FROM AUTHOR]

Accurately detect vehicles or pedestrians from 3D point clouds (3D object detection) is a fast developing research topic in autonomous driving and other domains. The fundamental component for feature extraction in 3D object detection is Set Abstraction (SA), which can downsample points while aggregating points to extract features. However, the current SA ignores the geometric and semantic properties of point clouds and may miss to detect remote small objects. In this paper, FocusSA is proposed, which consists two modules for enhancing useful feature extraction in the SA layer to improve 3D object detection accuracy. At first, Focused FPS (FocFPS) is proposed to evaluate the foreground and boundary scores of the points and reweighs the Furthest Point Sampling (FPS) using the evaluated scores to retain more contextual points in downsampling. Then a Geometry-aware Feature Extraction (GeoFE) module is proposed to add geometric information to enrich the awareness of geometric structure in feature aggregation. To evaluate the performances of the proposed methods, we conduct extensive experiments on three difficulty levels of Car class in KITTI dataset. The experimental results show that on "moderate" instances, our results outperform the state-of-the-art method by 1.08%. Moreover, FocusSA is easy to be plugged in popular architectures. [ABSTRACT FROM AUTHOR]

Autonomous driving and the machine learning (ML) models developed to achieve it have grown rapidly in importance and complexity respectively, and with them has also grown their carbon footprint due to long training times. Given the importance of climate change, it should be necessary to include the CO 2 impact of ML models explicitly to encourage competition on more than just model quality. This work presents the implementation of two different Convolutional Neural Network (CNN) training approaches, used for autonomous driving by behavioral cloning in a simulated environment and compares their impact on the CO 2 footprint. Using a cloud execution environment and driving data, previously obtained by applying an end-to-end deep learning technique, the first implemented training approach is a classical approach that uses an image generator that carries out the pre-processing and data augmentation during training. The contribution proposed in this paper is a second approach that improves the training by decreasing the training time, carrying out the data augmentation and pre-processing tasks before training the model, storing the result in RAM and then starting the training. The new approach to training presented in this article finishes the training approximately 38 times faster and reduces the carbon footprint impact by approximately 96%. In absolute values, this is a reduction from an average value of approximately 0.1643 CO 2 e q (kg) to 0.007 CO 2 e q (kg). To estimate the impact of CO 2 , the hardware used for the project, the training time and the cloud service provider were all taken into account. [ABSTRACT FROM AUTHOR]

How to design System of Systems has been widely concerned in recent years, especially in military applications. This problem is also known as SoS architecting, which can be boiled down to two subproblems: selecting a number of systems from a set of candidates and specifying the tasks to be completed for each selected system. Essentially, such a problem can be reduced to a combinatorial optimization problem. Traditional exact solvers such as branch-bound algorithm are not efficient enough to deal with large scale cases. Heuristic algorithms are more scalable, but if input changes, these algorithms have to restart the searching process. Re-searching process may take a long time and interfere with the mission achievement of SoS in highly dynamic scenarios, e.g., in the Mosaic Warfare. In this paper, we combine artificial intelligence with SoS architecting and propose a deep reinforcement learning approach DRL-SoSDP for SoS design. Deep neural networks and actor–critic algorithms are used to find the optimal solution with constraints. Evaluation results show that the proposed approach is superior to heuristic algorithms in both solution quality and computation time, especially in large scale cases. DRL-SoSDP can find great solutions in a near real-time manner, showing great potential for cases that require an instant reply. DRL-SoSDP also shows good generalization ability and can find better results than heuristic algorithms even when the scale of SoS is much larger than that in training data. [ABSTRACT FROM AUTHOR]

Complete and high-quality deformation monitoring data are critical for shield tunnel construction safety and quality. In engineering practices, data missing frequently occurs during instrumentation, adversely impacting further analysis and decision making. Existing imputation methods either ignore the crucial interactions between different parameters during shield tunneling or focus on the global characteristics of deformation data while neglect their local difference. This paper proposes a novel hybrid model, MCCB, combining multi-view matrix completion algorithms, convolutional neural network (CNN), and bidirectional long short-term memory (BiLSTM) algorithms to impute missing deformation values in shield tunnel monitoring data. The performance of the proposed method is verified using bridge deformation data from a practical project in Beijing. Different missing patterns of the bridge deformation data are filled. The experiment results show that the proposed model can effectively learn the various characteristics of the deformation data and outperforms the four selected models and its two sub-models, and can be used to improve the accuracy of the deformation prediction through data imputation. The novelty of this study includes two aspects. First is that the complicated interaction between different parameters and local difference of the data are considered simultaneously. They have not been addressed before by existing studies. Second is that the innovative combination of matrix completion and deep learning algorithm for application in missing deformation values imputation. To our best knowledge, no research on engineering construction has implemented this technique before. [ABSTRACT FROM AUTHOR]

The shop floor has always been an important application object for the digital twin. It is well known that production, process, and product are the core business of the shop floor. Therefore, the digital twin shop floor covers multi-dimensional information and multi-scale application scenarios. In this paper, the digital twin shop floor is constructed according to the modeling method of the complex digital twin proposed in Part I. The digital twin shop floor is firstly divided into several simple digital twins that focus on scenarios of different scales. Two simple application scenarios are constructed, including tool wear prediction and spindle temperature prediction. Main functions in different application scenarios, such as data acquisition, data processing, and data visualization, are implemented and encapsulated as components to construct simple digital twins. Secondly, ontology models, knowledge graphs, and message queues are used to assemble these simple digital twins into the complex digital twin shop floor. And two complex application scenarios are constructed, including machining geometry simulation considering spindle temperature and production scheduling considering tool wear. The implementation of the complex digital twin shop floor demonstrates the feasibility of the proposed modeling method. [ABSTRACT FROM AUTHOR]

Accurate estimating the machine health indicator is an essential part of industrial intelligence. Despite having considerable progress, remaining useful life (RUL) prediction based on deep learning still confronts the following two challenges. Firstly, the length of condition monitoring data obtained from sensors is inconsistent, and the existing fixed window data processing method cannot fully adapt to all individual samples. Secondly, it is challenging to extract local and global features for long-series prediction tasks. To address these issues, this paper proposes a Multi-task Spatio-Temporal Augmented Net(MTSTAN) for industrial RUL prediction, which enhances the local features of different sensors data through channel attention mechanism, and proposes a skip connected causal augmented convolution network to enhance the global feature extraction in time series. For the industrial scenario of inconsistent data lengths, a multi-window multi-task sharing mechanism is set up to capture various time dependencies among different time scales. The robustness and universality of the model are increased by sharing information among tasks and multi-task window mechanism. Finally, a large number of experiments were carried out on the turbofan aircraft engine run-to-failure prognostic benchmark dataset (C-MAPSS) to evaluate the proposed model, and compared with the existing 14 state-of-the-art approaches. The results show that the enhancement of local and global time series features can effectively improve the prediction accuracy. The Multi-task learning strategy has excellent applicability in dealing with the problem of inconsistent data length. [ABSTRACT FROM AUTHOR]

As a representative deep learning network, Convolutional Neural Network (CNN) has been extensively used in bearing fault diagnosis and many good results have been reported. In Prognostics and Health Management (PHM) field, the CNN's input size is usually designed as a 1D vector or 2D square matrix, and the convolution kernel size is also defined as a square shape like 3 × 3 and 5 × 5, which are directly adopted from the image recognition. Though satisfying results can be obtained, CNN with such parameter specifications is not optimal and efficient. To this end, this paper elaborated the physical characteristics of bearing acceleration signals to guide the CNN design. First, the fault period under different fault types and shaft rotation frequency were used to determine the size of CNN's input. Next, an exponential function was involved in fitting the envelope of decaying acceleration signal during each fault period, and signal length within different decaying ratios was used to define the CNN's kernel size. Finally, the designed CNN was validated with the Case Western Reserve University bearing dataset and Paderborn University bearing dataset. Results confirm that the physics-guided CNN (PGCNN) with rectangular input shape and rectangular convolution kernel works better than the baseline CNN with higher accuracy and smaller uncertainty. The feasibility of designing CNN parameters with physics-guided rules derived from bearing fault signal analysis has also been verified. [ABSTRACT FROM AUTHOR]

Surface defect detection plays an increasing role in intelligent manufacturing and product life-cycle management, such as quality inspection, process monitoring, and preventive maintenance. The existing intelligent methods almost adopt convolution architecture, and the limited receptive field hinders performance improvement of defect detection. In general, a larger receptive field can bring richer contextual information, resulting in better performance. Although operations such as dilated convolution can expand the receptive field, this improvement is still limited. Recently, benefitting from the ability to model long-range dependencies, Transformer-based models achieve great success in computer vision and image processing. However, applying Transformer-based models without modification is not desirable because there is no awareness and pertinence to defects. In this paper, an intelligent method is proposed by using defect-aware Transformer network (DAT-Net). In DAT-Net, Transformer replaces convolution in encoder to overcome the difficulty of modeling long-range dependencies. Defect-aware module assembled by basic weight matrixes is incorporated into Transformer to perceive and capture geometry and characteristic of defect. Graph position encoding by constructing a dynamic graph on tokens is designed to provide auxiliary positional information, which brings desired improved performance and fine adaptability. Specially, we carry out field experiments and painstakingly construct blade defect and tool wear datasets to compare DAT-Net with other methods. The comprehensive experiments demonstrate that DAT-Net has superior performance with 90.19 mIoU on blade defect dataset and 87.24 mIoU on tool wear dataset. [ABSTRACT FROM AUTHOR]

In complex working site, bearings used as the important part of machine, could simultaneously have faults on several positions. Consequently, multi-label learning approach considering fully the correlation between different faulted positions of bearings becomes the popular learning pattern. Deep reinforcement learning (DRL) combining the perception ability of deep learning and the decision-making ability of reinforcement learning, could be adapted to the compound fault diagnosis while having a strong ability extracting the fault feature from the raw data. However, DRL is difficult to converge and easily falls into the unstable training problem. Therefore, this paper integrates the feature extraction ability of DRL and the knowledge transfer ability of transfer learning (TL), and proposes the multi-label transfer reinforcement learning (ML-TRL). In detail, the proposed method utilizes the improved trust region policy optimization (TRPO) as the basic DRL framework and pre-trains the fixed convolutional networks of ML-TRL using the multi-label convolutional neural network method. In compound fault experiment, the final results demonstrate powerfully that the proposed method could have the higher accuracy than other multi-label learning methods. Hence, the proposed method is a remarkable alternative when recognizing the compound fault of bearings. • The proposed method is based on the DRL and the TL. • A pre-trained ML-CNN is designed to enhance the ability of ML-TRL. • An end-to-end mode is developed by training directly the 1D signal. [ABSTRACT FROM AUTHOR]

• A deep neural network enhanced by transfer learning is designed for joint failure load prediction. • A fruit fly optimisation algorithm is developed to achieve the best joint parameters. • Case studies are conducted to demonstrate the effectiveness of the approach. Adhesively bonded joints have been extensively employed in the aeronautical and automotive industries to join thin-layer materials for developing lightweight components. To strengthen the structural integrity of joints, it is critical to estimate and improve joint failure loads effectually. To accomplish the aforementioned purpose, this paper presents a novel deep neural network (DNN) model-enabled approach, and a single lap joint (SLJ) design is used to support research development and validation. The approach is innovative in the following aspects: (i) the DNN model is reinforced with a transfer learning (TL) mechanism to realise an adaptive prediction on a new SLJ design, and the requirement to re-create new training samples and re-train the DNN model from scratch for the design can be alleviated; (ii) a fruit fly optimisation (FFO) algorithm featured with the parallel computing capability is incorporated into the approach to efficiently optimise joint parameters based on joint failure load predictions. Case studies were developed to validate the effectiveness of the approach. Experimental results demonstrate that, with this approach, the number of datasets and the computational time required to re-train the DNN model for a new SLJ design were significantly reduced by 92.00% and 99.57% respectively, and the joint failure load was substantially increased by 9.96%. [ABSTRACT FROM AUTHOR]

Bearing fault diagnosis plays an important role in rotating machinery equipment's safe and stable operation. However, the fault sample collected from the equipment is seriously absent, which obstacles the establishment of the diagnostic model. In this paper, a novel false-real data synthesis method combined bearing dynamic model with a generated adversarial network is proposed to solve the problem of zero-shot in new condition. Firstly, the bearing dynamic model is constructed to simulate vibration data in different conditions. Secondly, the conversion model is trained by simulation data in different conditions, which will be employed to convert real-world data in the old condition into the conversion data in the new condition. Thirdly, the GAN model is trained by simulation data and real-world data in old condition and finetuned by simulation data and conversion data in the new condition. Finally, simulation data in the new condition are inputted to the finetuned GAN model to obtain generated data in the new condition, and the fault diagnosis model is trained by it. To validate the performance of the proposed method, various comparative experiments are carried out on one rolling bearing dataset. The results indicate that the proposed method can solve the problem of zero-shot in new condition with excellent diagnosis performance. [ABSTRACT FROM AUTHOR]