Search

Shop scheduling is deeply involved in manufacturing. In order to improve the efficiency of scheduling and fit dynamic scenarios, many Deep Reinforcement Learning (DRL) methods are studied to solve scheduling problems like job shop and flow shop. But most studies focus on using the latest algorithms while ignoring that the environment plays an important role in agent learning. In this paper, we design an effective, robust and size-agnostic environment for job shop scheduling. The proposed design of environment uses centralised training and decentralised execution (CTDE) to implement a multi-agent architecture. Together with the observation space we design, environmental information that is irrelevant to the current decision is eliminated as much as possible. The proposed action space enlarges the decision space of agents, which performs better than the traditional way. Finally, Soft Actor-Critic (SAC) algorithm is adapted to learning within this environment. By comparing with traditional scheduling rules, other reinforcement learning algorithms, and relevant literature, the superiority of the results obtained in this study is demonstrated. [ABSTRACT FROM AUTHOR]

The cloud-edge collaborative manufacturing system (CCMS) connects distributed factories to a cloud centre through cloud-edge collaborative communication, introducing both opportunities and challenges to conventional dynamic job scheduling. Enhancing each factory's scheduling performance by sharing general scheduling knowledge among heterogeneous factories under the consideration of data privacy protection remains challenging. To this end, this paper proposes to solve the dynamic job scheduling in the context of CCMS with a novel federated deep reinforcement learning (FDRL) approach. Within each factory, the scheduling objective involves minimising the makespan and energy consumption, accounting for machine warm-up procedures and real-time dynamics. To handle heterogeneous policy structures, we aggregate their hidden parameters through FDRL, with states, actions, and rewards designed to facilitate the aggregation. The two-phase algorithm, comprising iterative local training and global aggregation, trains the scheduling policies. Constraint items are introduced to the loss functions to smooth local training, and the global aggregation considers production scales and obtained objectives. The proposed approach enhances the solution quality and generalisation of each factory's scheduling policy without exposing original production data. Numerical experiments conducted on sixty scheduling instances validate the superiority of the proposed approach compared to twelve dynamic scheduling methods. Compared to independently trained DRL-based approaches, the proposed FDRL-based approach achieves up to an 8.9% reduction in makespan and a 22.3% decrease in energy consumption through knowledge sharing. [ABSTRACT FROM AUTHOR]

The increasing vessel size and automation level have shifted the productivity bottleneck of automated container terminals from the terminal side to the yard side. Operating an automated container terminal (ACT) yard with a big number of automated guided vehicles (AGV) is challenging due to the complexity and dynamics of the system, severely affecting the operational efficiency and energy use efficiency. In this paper, a hybrid multi-AGV scheduling algorithm is proposed to minimise the energy consumption and the total makespan of AGVs in an ACT yard. This framework first models the AGV scheduling process as a Markov decision process (MDP). Furthermore, a novel scheduling algorithm called MDAS is proposed based on multi-agent deep deterministic policy gradient (MADDPG) to facilitate online real-time scheduling decision-making. Finally, simulation experiments show that the proposed method can effectively enhance the operational efficiency and energy use performance of AGVs in ACT yards of various scales by comparing with benchmarking methods. [ABSTRACT FROM AUTHOR]

Nowadays industrial robots have been increasingly widely deployed across various sectors and assembly tasks are seen as one of the dominant application fields. The assembly tasks, as the critical process in manufacturing, are still challenging for the robot because of the complex contact state between the robot and the environment (i.e. assembly components). In the assembly task, the robot should switch its controller from a non-contact mode to a contact-rich mode when the contact condition changes, where compliant behaviour is necessary for robustness to uncertain contact and safety of physical interaction. This paper proposes a deep reinforcement learning (DRL) method to achieve such compliance using variable stiffness compliant control. Concretely, a Cartesian compliant controller is built on a virtual dynamics model with a variable non-diagonal stiffness matrix to derive desired motion that reacts with the external force/torque. Upon that, a deep deterministic policy gradient (DDPG)-based agent is deployed to fine-tune such a non-diagonal stiffness matrix. After error-and-trial learning, robots can handle changes in contact state in assembly tasks without any pause and switching its controller mode, thus increasing task efficiency and compliance. The simulation and experiments show that our method allows robots to complete assembly tasks safely and efficiently under noisy observation. [ABSTRACT FROM AUTHOR]

This document discusses the importance of carbon reduction engineering in combating global climate change. It highlights the role of production and supply chain management in achieving carbon neutrality and reducing carbon emissions. The document presents 20 selected research papers that cover various aspects of production optimization in carbon reduction engineering, including carbon reduction in the production process, low-carbon and sustainable supply chains, assessment and optimization methodologies, and policy instruments. The papers provide insights into topics such as scheduling optimization, renewable energy production, carbon emission efficiency, sustainable consumption, government subsidizing arrangements, and the impact of policy instruments on carbon reduction. The document expresses gratitude to the authors and reviewers for their contributions and acknowledges the support of the International Journal of Production Research. [Extracted from the article]

The semiconductor industry is a resource-intensive sector that heavily relies on energy, water, chemicals, and raw materials. Within the semiconductor manufacturing process, the diffusion furnace, ion implantation machine, and plasma etching machine exhibit high energy demands or operate at extremely high temperatures, resulting in significant electricity consumption, which is usually carbon-intensive. To address energy conservation concerns, the industry adopts batch production technology, which allows for the simultaneous processing of multiple products. The energy-efficient parallel batch scheduling problem arises from the need to optimise product grouping and sequencing. In contrast to existing heuristics, meta-heuristics, and exact algorithms, this paper introduces the Deep Q-Network (DQN) algorithm as a novel approach to address the proposed problem. The DQN algorithm is built upon the agent's systematic learning of scheduling rules, thereby enabling it to offer guidance for online decision-making regarding the grouping and sequencing of products. The efficacy of the algorithm is substantiated through extensive computational experiments. [ABSTRACT FROM AUTHOR]

This study conducts a comprehensive analysis of deep reinforcement learning (DRL) algorithms applied to supply chain inventory management (SCIM), which consists of a sequential decision-making problem focussed on determining the optimal quantity of products to produce and ship across multiple capacitated local warehouses over a specific time horizon. In detail, we formulate the problem as a Markov decision process for a divergent two-echelon inventory control system characterised by stochastic and seasonal demand, also presenting a balanced allocation rule designed to prevent backorders in the first echelon. Through numerical experiments, we evaluate the performance of state-of-the-art DRL algorithms and static inventory policies in terms of both cost minimisation and training time while varying the number of local warehouses and product types and the length of replenishment lead times. Our results reveal that the Proximal Policy Optimization algorithm consistently outperforms other algorithms across all experiments, proving to be a robust method for tackling the SCIM problem. Furthermore, the (s, Q)-policy stands as a solid alternative, offering a compromise in performance and computational efficiency. Lastly, this study presents an open-source software library that provides a customisable simulation environment for addressing the SCIM problem, utilising a wide range of DRL algorithms and static inventory policies. [ABSTRACT FROM AUTHOR]

We propose a scheme for achieving basic quantum gates using ultracold polar molecules in pendular states. The qubits are encoded in the YbF molecules trapped in an electric field with a certain gradient and coupled by the dipole–dipole interaction. The time-dependent control sequences consisting of multiple pulses are considered to interact with the pendular qubits. To achieve high-fidelity quantum gates, we map the control problem for the coupled molecular system into a Markov decision process and deal with it using the techniques of deep reinforcement learning (DRL). By training the agents over multiple episodes, the optimal control pulse sequences for the two-qubit gates of NOT, controlled NOT, and Hadamard are discovered with high fidelities. Moreover, the population dynamics of YbF molecules driven by the discovered gate sequences are analyzed in detail. Furthermore, by combining the optimal gate sequences, we successfully simulate the quantum circuit for entanglement. Our findings could offer new insights into efficiently controlling molecular systems for practical molecule-based quantum computing using DRL. [ABSTRACT FROM AUTHOR]

Thermophotovoltaic (TPV) systems can be used to harvest thermal energy for thermoelectric conversion with much improved efficiency and power density compared with traditional photovoltaic systems. As the key component, selective emitters (SEs) can re-emit tailored thermal radiation for better matching with the absorption band of TPV cells. However, current designs of the SEs heavily rely on empirical design templates, particularly the metal-insulator-metal (MIM) structure, and lack of considering the overall performance of TPV systems and optimization efficiency. Here, we utilized a deep reinforcement learning (DRL) method to perform a comprehensive design of a 2D square-pattern metamaterial SE, with simultaneous optimization of material selections and structural parameters. In the DRL method, only the database of refractory materials with gradient refraction indexes needs to be prepared in advance, and the whole design roadmap will automatically output the SE with optimal Figure-of-Merit (FoM) efficiently. The optimal SE is composed of a novel material combination of TiO2, Si, and W substrate, with its thickness and structure precisely optimized. Its emissivity spectra match well with the external quantum efficiency curve of the GaSb cell. Consequently, the overall performance of TPV is significantly enhanced with an output power density of 5.78 W/cm2, an energy conversion efficiency of 38.26%, and a corresponding FoM of 2.21, surpassing most existing designs. The underlying physics of optimal SE is explained by the coupling effect of multiple resonance modes. This work advances the practical application potential of TPV systems and paves the way for addressing other multi-physics optimization problems and metamaterial designs. [ABSTRACT FROM AUTHOR]

Photogrammetry systems are widely used in industrial manufacturing applications as an assistance measurement tool. Not only does it provide high-precision feedback for assembly process inspection and product quality assessment, but also it can improve the flexibility and robustness of manufacturing systems and production lines. However, with growing global competition and demands, companies are forced to enhance production efficiency, shorten production lifecycle and increase product variety by incorporating reconfigurable factory design that can meet challenging timeline and requirements. Although dynamic facility layout is widely investigated, the position selection for the photogrammetry system in dynamic manufacturing environment is usually overlooked. In this paper, dynamic layout of the V-STARS photogrammetry system is investigated and optimised in a digital-twin environment using deep reinforcement learning. The learning objectives are derived from the field of view (FoV) evaluation from point clouds 3D reconstruction, and collision detection from the digital twin simulated in Visual Components. The application feasibility of the proposed dynamic layout optimisation of the V-STARS photogrammetry system is verified with a real world industrial application. [ABSTRACT FROM AUTHOR]

Dynamic task scheduling problem in cloud manufacturing (CMfg) is always challenging because of changing manufacturing requirements and services. To make instant decisions for task requirements, deep reinforcement learning-based (DRL-based) methods have been broadly applied to learn the scheduling policies of service providers. However, the current DRL-based scheduling methods struggle to fine-tune a pre-trained policy effectively. The resulting training from scratch takes more time and may easily overfit the environment. Additionally, most DRL-based methods with uneven action distribution and inefficient output masks largely reduce the training efficiency, thus degrading the solution quality. To this end, this paper proposes an improved DRL-based approach for dynamic task scheduling in CMfg. First, the paper uncovers the causes behind the inadequate fine-tuning ability and low training efficiency observed in existing DRL-based scheduling methods. Subsequently, a novel approach is proposed to address these issues by updating the scheduling policy while considering the distribution distance between the pre-training dataset and the in-training policy. Uncertainty weights are introduced to the loss function, and the output mask is extended to the updating procedures. Numerical experiments on thirty actual scheduling instances validate that the solution quality and generalization of the proposed approach surpass other DRL-based methods at most by 32.8% and 28.6%, respectively. Additionally, our method can effectively fine-tune a pre-trained scheduling policy, resulting in an average reward increase of up to 23.8%. [ABSTRACT FROM AUTHOR]

Herein, we present a real-time multi-agent deep reinforcement learning model as a disassembly planning framework for human-robot collaboration. This disassembly plan optimises sequences to minimise operation time and the disassembling costs of end-of-life (EoL) products. Combining different data-driven decision-making tools, the plan aims to handle the complexities and uncertainties of disassembly tasks. Based on the physical features and geometric limitations of EoL product components, we calculate product disassembly difficulty scores. Subsequently, the deep reinforcement learning model integrates these scores into planning process. The model allocates tasks in real time according to the online conditions of the human operator, cobot, and product, enabling the model to cope with uncertainties that may change the process routine. We also present different scenarios wherein a cobot collaborates with human operators with different skill levels. To evaluate model performance, we compare it with baseline models in terms of the convergence time and incorporated disassembly features. The analysis indicates that our model converges three times faster than a baseline model applied to the same case study. Moreover, our model includes more features of the disassembly problem in its decision-making process than any other baseline model. [ABSTRACT FROM AUTHOR]

Unlike mass production manufacturing systems, where configurations are rarely changed after the initial design, reconfigurable cyber-physical systems (RCPMS) self-change their structures throughout missions and thus self-adjust production in response to demand requirements. Accordingly, such a paradigm requires enhancing selective maintenance strategy to optimise scheduling maintenance actions, selecting configuration layouts for capacity and product family changes, and achieving maintenance cost reduction and reliability maximisation. This paper is the first to propose a robust model for a selective maintenance problem with imperfect repairs in the RCPMS context. The model also integrates uncertainties originating from the imperfect observations of components' health status. The model's objectives are to maximise the expected reliability and minimise the variance and maintenance cost under maintenance resource constraints. Moreover, we propose a new deep reinforcement learning framework for solving the resulting multi-objective and combinatorial optimisation problem. In addition, we use decision values to enhance the scalarisation process by permitting the priorities of specific objectives to be adjusted after the learning process. Furthermore, we employ Analytical Hierarchy Process to adjust the static priorities with respect to the objective functions and the actual learning context. Finally, broad experiments are conducted to highlight the performance of the proposed model and resolution framework. [ABSTRACT FROM AUTHOR]

In nowadays production, fluctuations in demand, shortening product life-cycles, and highly configurable products require an adaptive and robust control approach to maintain competitiveness. This approach must not only optimise desired production objectives but also cope with unforeseen machine failures, rush orders, and changes in short-term demand. Previous control approaches were often implemented using a single operations layer and a standalone deep learning approach, which may not adequately address the complex organisational demands of modern manufacturing systems. To address this challenge, we propose a hyper-heuristics control model within a semi-heterarchical production system, in which multiple manufacturing and distribution agents are spread across pre-defined modules. The agents employ a deep reinforcement learning algorithm to learn a policy for selecting low-level heuristics in a situation-specific manner, thereby leveraging system performance and adaptability. We tested our approach in simulation and transferred it to a hybrid production environment. By that, we were able to demonstrate its multi-objective optimisation capabilities compared to conventional approaches in terms of mean throughput time, tardiness, and processing of prioritised orders in a multi-layered production system. The modular design is promising in reducing the overall system complexity and facilitates a quick and seamless integration into other scenarios. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of stochastic jointly scheduling of resumable jobs and preventive maintenance on a single machine, subject to random breakdowns, to minimise the earliness-tardiness cost. The main objective is to investigate using trending machine learning-based methods compared to stochastic optimisation approaches. We propose two different methods from both fields as we solve the same problem firstly with a stochastic dynamic programming model in an approximation way, then with an attention-based deep reinforcement learning model. We conduct a detailed experimental study according to solution quality, run time, and robustness to analyse their performances compared to those of an existing approach in the literature as a baseline. Both algorithms outperform the baseline. Moreover, the machine learning-based algorithm outperforms the stochastic dynamic programming-based heuristic as we report up to 30.5% saving in total cost, a reduction of computational time from 67 min to less than $ 1s $ 1 s on big instances, and a better robustness. These facts highlight clearly its potential for solving such problems. [ABSTRACT FROM AUTHOR]

This paper proposes a novel intelligent scheduling method based on deep reinforcement learning (DRL) to solve the multi-objective steelmaking-continuous casting (SCC) scheduling problem, under time-of-use (TOU) tariffs for the first time. The intelligent scheduling system architecture is designed, and a mathematical model is established to minimise the total sojourn time and electricity cost. To effectively reduce production costs by avoiding peak periods of electricity consumption, the 'start time' of the system is generated based on the Markov Decision Process (MDP), and heuristic scheduling rules related to power cost are used as the action space, with corresponding reward functions designed according to the characteristics of these two objectives. To satisfy the continuous casting which is a particular SCC constraint, a backward strategy is developed. Additionally, a branching duelling double deep Q-network (BD3QN) is adapted to guide action selection and avoid blind search in the iteration process, and then applied to real-time scheduling. Numerical experiments demonstrate that the proposed method outperforms comparison algorithms in terms of solution quality and CPU times by a large margin. [ABSTRACT FROM AUTHOR]

China's cotton textile industry is undergoing a critical period of digital transformation and upgrading to cope with pressure and challenges such as rising labour costs and large fluctuations in raw material prices. Developing a cost-based competitive advantage while ensuring a high-quality product is a critical problem in intelligent manufacturing. From the perspective of big data and reinforcement learning, the authors designed a reward value combining transaction, interaction, and measurement data by combining the reward mechanism and Markov decision for a combination of different raw materials in the intelligent textile factory. The authors propose a big data-driven application to the depth of the reinforcement learning to solve problems and build a big-data-driven matching model based on deep reinforcement learning to cotton matching. The offline strategy is designed to construct a memory bank and neural network, and the incentive mechanism of reinforcement learning is used to iterate the optimal yarn matching scheme to achieve the goal of intelligent cotton matching. The results show that deep reinforcement learning can be optimised using big data on the premise of quality assurance. Manufacturing costs can be optimised using a matching model of big data based on a deep reinforcement learning model. [ABSTRACT FROM AUTHOR]

This paper studies a Deep Q-Learning (DQL) method for transaction sequencing problems in an automated warehousing system, Shuttle-based Storage and Retrieval System (SBSRS), in which shuttles can move between tiers flexibly. Here, the system is referred to as tier-to-tier SBSRS (t-SBSRS), developed as an alternative design to tier-captive SBSRS (c-SBSRS). By the flexible travel of shuttles between tiers in t-SBSRS, the number of shuttles in the system may be reduced compared to its simulant c-SBSRS design. The flexible travel of shuttles makes the operation decisions more complex in that system, motivating us to explore whether integration of a machine learning approach would help to improve the system performance. We apply the DQL method for the transaction selection of shuttles in the system to attain process time advantage. The outcomes of the DQN are confronted with the well-applied heuristic approaches: first-come-first-serve (FIFO) and shortest process time (SPT) rules under different racking and numbers of shuttles scenarios. The results show that DQL outperforms the FIFO and SPT rules promising for the future of smart industry applications. Especially, compared to the well-applied SPT rule in industries, DQL improves the average cycle time per transaction by roughly 43% on average. [ABSTRACT FROM AUTHOR]

Copyright of Telecommunication Engineering is the property of Telecommunication Engineering and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Telecommunication Engineering is the property of Telecommunication Engineering and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The generation of quadruped robot gait through deep reinforcement learning (DRL) has emerged as a prominent topic in recent years. However, DRL methods face challenges in generating specific and energy-efficient gait patterns, as well as in achieving gait transition. These methods often require imposing restrictions on movement or on the learning process to achieve proper gait specification and generation, thereby limiting the quadruped model movement achievable with DRL training. In this paper, we propose a new DRL reward design method, called FootStep Reward, to successfully generate specific gait patterns that exhibit animal-like energetic properties. Our method is capable of generating Walk, Trot, and Gallop gait patterns at any velocity while demonstrating better energy efficiency compared to similar gait specification methods. Additionally, we employ our method to analyze the impact of two body parameters: model mass and joint stiffness, increasing our understanding of how these parameters influence Walk, Trot and Gallop gait efficiency. Finally, we implement the FootStep Reward method to generate gait transitions and successfully achieve the transition from Walk to Trot gait, optimizing locomotion at low and medium velocities. [ABSTRACT FROM AUTHOR]

Increasing data transmission volumes are causing more frequent and more severe network congestion. In order to handle spikes in network traffic, a substantially bigger buffer has been included into the system. Bufferbloat, which happens when a bigger buffer is implemented, exacerbates network congestion. Using the transfer control protocol (TCP) congestion management strategy with active queue management (AQM) can fix this issue. As congestion increases, it becomes increasingly difficult to forecast and fine‐tune dynamic AQM/TCP systems in order to achieve acceptable performance. To shed new light on the AQM system, we plan to use deep reinforcement learning (DRL) techniques. It is possible that AQM can learn about the appropriate drop policy the same way people do when using a model‐free technique like DRL‐AQM. After training in a simple network scenario, DRL‐AQM is able to recognize complex patterns in the data traffic model and apply them to improve performance in a wide variety of scenarios. Offline training precedes deployment in our approach. In many cases, the model does not require any further parameter tweaks after training. Even in the most complicated networks, AQM algorithms have proven to be effective, regardless of the network's complexity. Minimizing buffer capacity use is an important goal of DRL‐AQM. It automatically and continually adjusts to changes in network connectivity. [ABSTRACT FROM AUTHOR]