Search

China's cotton textile industry is undergoing a critical period of digital transformation and upgrading to cope with pressure and challenges such as rising labour costs and large fluctuations in raw material prices. Developing a cost-based competitive advantage while ensuring a high-quality product is a critical problem in intelligent manufacturing. From the perspective of big data and reinforcement learning, the authors designed a reward value combining transaction, interaction, and measurement data by combining the reward mechanism and Markov decision for a combination of different raw materials in the intelligent textile factory. The authors propose a big data-driven application to the depth of the reinforcement learning to solve problems and build a big-data-driven matching model based on deep reinforcement learning to cotton matching. The offline strategy is designed to construct a memory bank and neural network, and the incentive mechanism of reinforcement learning is used to iterate the optimal yarn matching scheme to achieve the goal of intelligent cotton matching. The results show that deep reinforcement learning can be optimised using big data on the premise of quality assurance. Manufacturing costs can be optimised using a matching model of big data based on a deep reinforcement learning model. [ABSTRACT FROM AUTHOR]

Treatment planning for chronic diseases is a critical task in medical artificial intelligence, particularly in traditional Chinese medicine (TCM). However, generating optimized sequential treatment strategies for patients with chronic diseases in different clinical encounters remains a challenging issue that requires further exploration. In this study, we proposed a TCM herbal prescription planning framework based on deep reinforcement learning for chronic disease treatment (PrescDRL). PrescDRL is a sequential herbal prescription optimization model that focuses on long-term effectiveness rather than achieving maximum reward at every step, thereby ensuring better patient outcomes. We constructed a high-quality benchmark dataset for sequential diagnosis and treatment of diabetes and evaluated PrescDRL against this benchmark. Our results showed that PrescDRL achieved a higher curative effect, with the single-step reward improving by 117% and 153% compared to doctors. Furthermore, PrescDRL outperformed the benchmark in prescription prediction, with precision improving by 40.5% and recall improving by 63%. Overall, our study demonstrates the potential of using artificial intelligence to improve clinical intelligent diagnosis and treatment in TCM. [ABSTRACT FROM AUTHOR]

With the increasing enrollment of higher education in China, teaching quality evaluation is an urgent problem to be studied. Colleges and universities should also have their own set of teaching quality evaluation system with the evaluation activities, and thus pre-evaluate their teaching and ensure the teaching quality between the two teaching evaluations. At present, big data is more and more widely used, and distributing these big data to corresponding scores through content centric data center networks (CCDCNs) provides Quality of Experience (QoE) for English teaching quality assessment. Therefore, one of the challenges faced by the current network is to enhance QoE under content distribution. In this work, in order to solve this problem, we schedule the cached teaching feature vector to the corresponding score level. Three cache scheduling algorithms are constructed in CCDCNs. Firstly, an approximate dynamic algorithm is proposed, which has high complexity. Then, based on the characteristics of node centralization, we propose an improved approximate dynamic scheduling algorithm. Although the algorithm includes the scheduling of cached content and the scheduling of content transmission rate, it has low complexity in processing scheduling. In addition, we propose a cache scheduling algorithm based on deep reinforcement learning (DRL). Although the algorithm has high complexity, the scheduling accuracy is also high. Experiments show that the method proposed in this work can obtain higher QoE, and excellent performance in English teaching evaluation, about 7.8% improvement degree. [ABSTRACT FROM AUTHOR]

To improve automatic generation control (AGC) performance and reduce the wastage of regulation resources in interconnected grids including highproportion renewable energy, a multi-area integrated AGC (MAI-AGC) framework is proposed to solve the coordination problem of secondary frequency regulation between different areas. In addition, a cocktail exploration multi-agent deep deterministic policy gradient (CE-MADDPG) algorithm is proposed as the framework algorithm. In this algorithm, the controller and power distributor of an area are combined into a single agent which can directly output the power generation command of different units. Moreover, the cocktail exploration strategy as well as various other techniques are introduced to improve the robustness of the framework. Through centralized training and decentralized execution, the proposed method can nonlinearly and adaptively derive the optimal coordinated control strategies for multiple agents and is verified on the two-area LFC model of southwest China and the four-area LFC model of the China Southern Grid (CSG). [ABSTRACT FROM AUTHOR]

Inventory management is one of the most important components of Alibaba's business. Traditionally, human buyers make replenishment decisions: although artificial intelligence (AI) algorithms make recommendations, human buyers can choose to ignore these recommendations and make their own decisions. The company has been exploring a new replenishment system in which algorithmic recommendations are final. The algorithms combine state-of-the-art deep reinforcement learning techniques with the framework of fictitious play. By learning the supplier's behavior, we are able to address the important issues of lead time and fill rate on order quantity, which have been ignored in the extant literature of stochastic inventory control. We present evidence that our algorithms outperform human buyers in terms of reducing out-of-stock rates and inventory levels. More interestingly, we have seen additional benefits amid the pandemic. Over the last two years, cities in China partially and intermittently locked down to mitigate COVID-19 outbreaks. We have observed panic buying from human buyers during lockdowns, leading to the bullwhip effect. By contrast, panic buying and the bullwhip effect can be mitigated using our algorithms due to their ability to recognize changes in the supplier's behavior during lockdowns. [ABSTRACT FROM AUTHOR]

The marine ranching industry in China is transitioning from traditional farming to a digital and intelligent model. The use of new technologies, algorithms, and models in the era of artificial intelligence (AI) is a key focus to enhance the efficiency, sustainability, and resilience of marine ranch operations, particularly in risk and disaster management. This study proposes a methodology for applying deep reinforcement learning to decision making in this domain. The approach involves creating an environmental model based on decision objects and scenarios, determining the number of decision makers, and selecting a single or multi-agent reinforcement learning algorithm to optimize decision making in response to randomly generated disasters. Three core innovations are presented: the development of a disaster simulator for marine ranching scenarios, the application of reinforcement learning algorithms to address risk and disaster management problems in marine ranching. Future research could focus on further refining the methodology by integrating different data sources and sensors and evaluating the social and economic impacts of AI-driven marine ranching. Overall, this study provides a foundation for further research in this area, which is expected to play an increasingly important role in global food production, environmental sustainability, and energy efficiency. [ABSTRACT FROM AUTHOR]

Machine learning algorithms, including supervised and unsupervised learning ones, have been widely used in mineral prospectivity mapping. Supervised learning algorithms require the use of numerous known mineral deposits to ensure the reliability of the training results. Unsupervised learning algorithms can be applied to areas with rare or no known deposits. Reinforcement learning (RL) is a type of machine learning algorithm that differs from supervised and unsupervised learning models in that the learning process is performed interactively by the agent and environment. The environment feeds the agent with reward signals and states, and the agent synthetically evaluates the mineralization potential of each state based on these rewards. In this study, a deep RL framework was constructed for mineral prospectivity mapping, and a case study for mapping gold prospectivity in northwest Hubei Province, China, was used to test the framework. The deep RL agent extracted the information of known mineralization by automatically interacting with the environment while simultaneously mining potential mineralization information from the unlabeled dataset. Its comparison with random forest and isolation forest models demonstrates that deep RL performs better regardless of the number of known mineral deposits because of its unique reward and feedback mechanism. The delineated high-potential areas show a strong spatial correlation with known gold deposits and can therefore provide significant clues for future prospecting in the study area. [ABSTRACT FROM AUTHOR]

Water supply systems that use rivers with high sedimentation levels may experience issues such as reservoir siltation. The suspended sediment concentration (SSC) of rivers experiences interannual variation and high nonlinearity due to its close relationship with meteorological factors, which increase the mismatch between the river water source and urban water demand. The raw water system scheduling problem is expressed as a reservoir and pump station control problem that involves real-time SSC changes. To lower the SSC of the water intake and lower the pumping station's energy consumption, a deep reinforcement learning (DRL) model based on SSC prediction was developed. The framework consists of a DRL model, a hydraulic model for simulating the raw water system, and a neural network for predicting river SSC. The framework was tested using data from a Yellow River water withdrawal pumping station in China with an average capacity of 400,000 m3/d. The strategy created in this study can reduce the system energy consumption per unit of water withdrawal by 8.33% and the average annual water withdrawal SSC by 37.01%, when compared to manual strategy. Meanwhile, the deep reinforcement learning algorithm had good response robustness to uncertain imperfect predictive data. [ABSTRACT FROM AUTHOR]

It has been widely certified that hyperspectral images can be effectively used to monitor soil organic matter (SOM). Though numerous bands reveal more details in spectral features, information redundancy and noise interference also come accordingly. Due to the fact that, nowadays, prevailing dimensionality reduction methods targeted to hyperspectral images fail to make effective band selections, it is hard to capture the spectral features of ground objects quickly and accurately. In this paper, to solve the inefficiency and instability of hyperspectral feature selection, we proposed a feature selection framework named reinforcement learning for feature selection in hyperspectral regression (RLFSR). Specifically, the Markov Decision Process (MDP) was used to simulate the hyperspectral band selection process, and reinforcement learning agents were introduced to improve model performance. Then two spectral feature evaluation methods were introduced to find internal relationships between the hyperspectral features and thus comprehensively evaluate all hyperspectral bands aimed at the soil. The feature selection methods—RLFSR-Net and RLFSR-Cv—were based on pre-trained deep networks and cross-validation, respectively, and achieved excellent results on airborne hyperspectral images from Yitong Manchu Autonomous County in China. The feature subsets achieved the highest accuracy for most inversion models, with inversion R2 values of 0.7506 and 0.7518, respectively. The two proposed methods showed slight differences in spectral feature extraction preferences and hyperspectral feature selection flexibilities in deep reinforcement learning. The experiments showed that the proposed RLFSR framework could better capture the spectral characteristics of SOM than the existing methods. [ABSTRACT FROM AUTHOR]

In recent studies, deep reinforcement learning (RL) methods have been used for the real‐time control of urban drainage systems (UDSs). However, the training process of an RL agent is computationally expensive since collecting the training data requires multiple simulations of a UDS model. An effective solution to this issue is to replace the original UDS model with an emulator, a simpler model which emulates the specified output of the system. However, the adaptability and interpretability of the existing UDS emulators still require further improvements. To address these challenges, a new emulator with increased adaptability and interpretability, called Koopman emulator, was developed by using the Koopman operator and a deep learning technique in this study. Then, the Koopman emulator was used to replace the UDS model in RL to establish an emulator‐based RL framework for fast and efficient training. This framework was used to train emulator‐based RL agents for flooding and overflow mitigation in a UDS located in Eastern China. Baseline emulators were also employed for comparison. All the methods were evaluated through simulations. The following results were obtained: (a) The Koopman emulator demonstrated a better emulation performance than the baseline emulators. (b) The RL agents based on the Koopman emulator achieved a similar control effect with a faster training process and more efficient data usage compared with the RL agents based on the storm water management model. (c) The uncertainty analysis showed that the RL agents based on the Koopman emulator exhibited acceptable robustness when facing different rainfall events and input perturbations. Key Points: Koopman emulator demonstrates better performance and better interpretability in nonlinear emulationThe emulator‐based reinforcement learning (RL) framework achieves similar control effect with faster training process and more efficient data usageThe RL agents based on Koopman emulator exhibited acceptable robustness when facing different rainfall events and input perturbations [ABSTRACT FROM AUTHOR]

This study introduces a novel approach to adaptive traffic signal control (ATSC) by leveraging multi-objective deep reinforcement learning (DRL) techniques. The proposed scheme aims to optimize control strategies at intersections while concurrently addressing the objectives of safety, efficiency, and decarbonization. Traditional ATSC schemes primarily emphasize traffic efficiency and often lack the ability to adapt to real-time dynamic traffic conditions. To overcome these limitations, the study proposes a DRL-based ATSC algorithm that integrates the Dueling Double Deep Q Network (D3QN) framework. The performance of the proposed algorithm is evaluated through a simulated intersection in Changsha, China. Specifically, the proposed ATSC algorithm outperforms both traditional ATSC and ATSC with efficiency optimization only algorithms by achieving more than a 16% reduction in traffic conflicts and a 4% reduction in carbon emissions. In terms of traffic efficiency, waiting time reduces by 18% compared to traditional ATSC, but slightly increases (0.64%) compared to DRL-based ATSC algorithm that integrates D3QN framework. This small increase indicates a trade-off between efficiency and other objectives such as safety and decarbonization. Moreover, the proposed scheme demonstrates superior performance specifically in highly traffic-demand scenarios in terms of all three objectives. The findings of this study contribute to the advancement of traffic control systems by providing a practical and effective solution for optimizing signal control strategies in real-world traffic scenarios., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2023. Published by Elsevier Ltd.)

This paper develops a deep reinforcement learning (DRL) framework for intelligence operation of cascaded hydropower reservoirs considering inflow forecasts, in which two key problems of large discrete action spaces and uncertainty of inflow forecasts are addressed. In this study, a DRL framework is first developed based on a newly defined knowledge sample form and a deep Q-network (DQN). Then, an aggregation-disaggregation model is used to reduce the multi-dimension spaces of state and action for cascaded reservoirs. Following, three DRL models are developed respectively to evaluate the performance of the newly defined decision value functions and modified decision action selection approach. In this paper, the DRL methodologies are tested on China's Hun River cascade hydropower reservoirs system. The results show that the aggregation-disaggregation model can effectively reduce the dimensions of state and action, which also makes the model structure simpler and has higher learning efficiency. The Bayesian theory in the decision action selection approach is useful to address the uncertainty of inflow forecasts, which can improve the performance to reduce spillages during the wet season. The proposed DRL models outperform the comparison models (i.e., stochastic dynamic programming) in terms of annual hydropower generation and system reliability. This study suggests that the DRL has the potential to be implemented in practice to derive optimal operation strategies. [ABSTRACT FROM AUTHOR]

Due to the stricter emission reduction and renewable energy consumption goals in China, the carbon emissions trading (CET) market and tradable green certificate (TGC) market, as key means to achieve these goals, urgently need to adjust the current operating mode timely and maximize market effectiveness by introducing adaptive incentive mechanisms. Therefore, taking the power industry as an example, an incentive-oriented power-CET-TGC integrated market simulation framework based on multi-agent deep reinforcement learning is constructed. The impact of consignment auction mechanism, traditional auction mechanism, and voluntary TGC trading mechanism on the transaction situation of each market is prospectively analyzed. The results demonstrate that with the introduction of various incentive mechanisms, the transaction scale and prices of CET and TGC markets increase, and the effectiveness of carbon reduction and renewable energy consumption is significant. Among them, auction mechanisms are valid way to promote the explicit cost of carbon emissions, and the voluntary TGC trading mechanism is an important means to enhance the clean value of green energy. In addition, it is recommended to adopt consignment auction mechanism under the short-term goal "carbon peak" of pursuing economic benefits and stable emission reduction, while traditional auction mechanisms are recommended under the long-term goal of "carbon neutrality" of urgently requiring large-scale emission reduction. Moreover, the voluntary TGC trading mechanism can achieve synergy and connection between the TGC market and the green electricity market to a certain extent. These insights can provide mechanism reference for the sustainable development and construction of CET and TGC markets. • An incentive-oriented multi-market framework is constructed. • A novel tradable green certificate market mechanism is proposed. • Incomplete market information and learning ability of traders are considered. • Transaction strategies are simulated by on multi-agent deep reinforcement learning. • Consignment auction is the transition between free quota and traditional auction. [ABSTRACT FROM AUTHOR]

In this study, the start time of teleconsultations is optimized for the clinical departments of class A tertiary hospitals to improve service quality and efficiency. For this purpose, first, a general teleconsultation scheduling model is formulated. In the formulation, the number of services (NS) is one of the objectives because of demand intermittency and service mobility. Demand intermittency means that demand has zero size in several periods. Service mobility means that specialists move between clinical departments and the National Telemedicine Center of China to provide the service. For problem-solving, the general model is converted into a Markov decision process (MDP) by elaborately defining the state, action, and reward. To solve the MDP, deep reinforcement learning (DRL) is applied to overcome the problem of inaccurate transition probability. To reduce the dimensions of the state-action space, a semi-fixed policy is developed and applied to the deep Q network (DQN) to construct an algorithm of the DQN with a semi-fixed policy (DQN-S). For efficient fitting, an early stop strategy is applied in DQN-S training. To verify the effectiveness of the proposed scheduling model and the model solving method DQN-S, scheduling experiments are carried out based on actual data of teleconsultation demand arrivals and service arrangements. The results show that DQN-S can improve the quality and efficiency of teleconsultations by reducing 9%-41% of the demand average waiting time, 3%-42% of the number of services, and 3%-33% of the total cost of services., Competing Interests: Declaration of competing interest None., (Copyright © 2024 Elsevier B.V. All rights reserved.)

• Deep reinforcement learning (DRL) based models are proposed for HVAC system energy consumption prediction. • Autoencoder (AE) algorithm is used to extract the high-level features of state space of DDPG. • The DDPG based models are applied for energy consumption forecasting over short-term time horizon. • Improvement in prediction performance is observed when combine AE with DDPG. • The DDPG based prediction models, outperform common supervised models like BP neural network and support vector machine. Short-term energy consumption prediction has fundamental importance in many HVAC system management tasks, such as demand-side management, short-term maintenance, etc. Currently, the prevailing data-driven techniques, especially supervised machine learning methods, are widely applied for short-term energy consumption prediction. Deep reinforcement learning (DRL), as the state-of-the-art machine learning techniques, have been applied for HVAC system control, but rarely for energy consumption prediction. In this paper, a DRL algorithm, namely Deep Deterministic Policy Gradient (DDPG), is firstly introduced for short-term HVAC system energy consumption prediction. Moreover, Autoencoder (AE), which is powerful in processing data in their raw form, is incorporated into DDPG method to extract the high-level features of state space and optimize the prediction model. The operation data of the ground source heat pump (GSHP) system of an office building in Henan province, China is used to train and assess the proposed models. The results demonstrate that the proposed DDPG based models can achieve better prediction performance than common supervised models like BP Neural Network and Support Vector Machine. This study is an enlightening work which may inspire other researchers to tap the potential of DRL algorithms in this field. [ABSTRACT FROM AUTHOR]

Timely, effective, and robust artificial intelligence (AI) technology is urgently needed to improve decision-making efficiency in the presence of renewable energy with high penetration by elevating the level of power grid intelligence. However, at this stage, AI technology lacks reliability and transparency, making it unable to play a full role in application areas with high-security requirements such as power systems. This paper presents a multi-hierarchical interpretable method for power system dispatch and operation based on the graph deep Q-network (GDQN) model to achieve active power corrective control. The multi-hierarchical interpretable method combined with an improved sample-balanced deep shapley additive explanation (SE-DSHAP) method and a subgraph explainer can promote a more intuitive and comprehensive explanation of decision-making for power systems with complex topology. Operators will obtain more noteworthy power grid areas through the proposed interpretable method as the basis of auxiliary decision-making to realize efficient and accurate control. Then, two cases studied in a modified 36-bus system in the IEEE 118-bus system and the 300-bus network of the China power grid validate the effectiveness of the proposed method. • A graph deep Q-network model achieve active power corrective in power systems. • The SE-DSHAP method improve the computational speed and accuracy. • A subgraph explainer bring a comprehensive understanding of the model predictions. • Two cases validate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

[Display omitted] • A machine learning tool solve the single-track rail scheduling problem is proposed. • On a gridworld model, optimal solutions are obtained under constraint application. • The algorithm uses deep convolutional neural networks to process image-like input information. This study proposes a deep reinforcement learning approach for bi-direction single-track railway scheduling called DRLA-eTGM. The goal is to define the departure times and track allocation without conflict for all trains on the line given their speed, priority, origin, and destination while minimizing the total priority-weighted dwelling time. A novel time–space-capacity gridworld model (TGM), which contains a two-dimensional state representation to dynamically describe the distribution of constraints and trains in single-track railway scheduling, is proposed. Moreover, DRLA-eTGM innovatively extracts and exploits features from the TGM state automatically with deep convolutional neural networks to enhance scheduling decision strategies. Further, a shaping technique inspired by dynamic programming is adopted in DRLA-eTGM. Computational experiments are presented to verify the effectiveness of the proposed approach using hypothesized instances and a busy passenger-cargo single-track railway corridor in China. The experimental results show that DRLA-eTGM outperforms some heuristic and reinforcement learning algorithms, especially in real-world instances. Compared to some deep reinforcement learning algorithms, DRLA-eTGM shows robustness and efficiency owing to its partial scalable observation and parameter sharing mechanisms. This study provides a new approach to exploiting constraint information in railway scheduling and a prototype system for automatic single-track railway scheduling. [ABSTRACT FROM AUTHOR]

With the explosive growth of electric vehicles (EVs), it is an urgent task to incorporate low-carbon EVs with advanced optimization strategies to achieve orderly charging–discharging and economical operations for EVs. Nevertheless, current charging–discharging optimization strategies for EVs may be impractical since their energy consumption uncertainty and the driving trip cost are generally not considered. Therefore, a collaborative optimization model for large-scale EV charging–discharging with energy consumption uncertainty in this paper is proposed to simultaneously maximize passenger revenue and reduce the costs of the driving, charging–discharging, and battery depletion. Subsequently, a data-driven approach is developed to tackle the model. In this approach, an uncertainty predictor based on wavelet transform, deep deterministic policy gradient, and quantile regression is first applied to estimate the energy consumption uncertainty. Then, an adaptive learning rate firefly algorithm is presented to identify the most satisfactory solution for the optimization model. Finally, taking the actual data of 300 EVs in a city in China as case studies, the simulation results reveal that the proposed method is effective and has high application significance. • A new collaborative optimization model for EV charging–discharging is formulated. • A deep reinforcement learning method for energy consumption prediction is proposed. • The impact of driving behavior on energy consumption prediction is studied. • A new adaptive learning rate Firefly algorithm is presented and applied. • The effectiveness of the proposed methodology is verified by simulations of 300 EVs. [ABSTRACT FROM AUTHOR]

• Learning-based method continuously controls energy management system. • Soft actor-critic retrenches hyperparameter adjustment of energy management. • On-line health and thermal models of fuel cell and battery are incorporated into reward. • Fuel economy and health increments of fuel cell and battery. • Temperatures of fuel cell and battery can be effectively regulated. Temperature can significantly affect the water equilibrium, electrochemical kinetics and mass transmission in a proton exchange membrane fuel cell (PEMFC) stack, meanwhile it also impacts the lifespan and safety of a lithium-ion battery (LIB). Yet, energy management strategy (EMS) is rarely to synchronously study the durability performances of the LIB and PEMFC stack with their thermal effects in fuel cell vehicles (FCVs) under real-world driving scenarios. Thus, this study proposes a deep reinforcement learning (DRL)-based EMS to minimize transient costs of the LIB and PEMFC stack, which include their state-of-health (SOH) descents and overtemperature penalties. Meanwhile, the transient costs are incorporated into the overall cost, which comprises the hydrogen consumption rate of the PEMFC stack, and penalty of maintaining the LIB state-of-charge (SOC). Moreover, the soft actor-critic (SAC) is applied to the DRL-based EMS due to its advantage of stability across different random environments and no meticulous hyperparameter calibration. Specifically, the proposed EMS intelligently allocates the direct current (DC) bus power of FCVs in real time to maximize a multi-objective reward in accordance with FCV states, in which the reward is the negative overall cost. Then, long-term real-world driving scenarios in Chongqing city, China, are used for off-line training and real-time control to advance the adaptability of the proposed EMS. The results show that in comparison with the deep Q-network (DQN)-based EMS considering the powertrain temperature and durability, and the SAC-based EMS neglecting the powertrain temperature and durability, the proposed strategy can actualize overall SOH increments of the powertrain up to 14.01 % and 3.45 %, respectively, and restrict the maximum temperatures of the PEMFC stack and LIB. In addition, the generalization of the proposed EMS is verified, in which the trained model of the proposed EMS is tested in other FCV and driving cycles, and it can acquire similar effectiveness. Thus, the proposed strategy can enforce the lifespan durability and thermal stability of the powertrain system. [ABSTRACT FROM AUTHOR]

• The proposed approach considers multi-uncertainties from load and generation sides. • Leverage deep reinforcement learning to automatically handle multiple uncertainties. • Build an energy/matter co-transfer model, i.e. hydrothermal simultaneous transmission. • Simulation tests were carried out on an actual island IES in North China. Multi-uncertainties from power sources and loads have brought significant challenges to the stable demand supply of various resources at islands. To address these challenges, a comprehensive scheduling framework is proposed by introducing a model-free deep reinforcement learning (DRL) approach based on modeling an island integrated energy system (IES). In response to the shortage of freshwater on islands, in addition to the introduction of seawater desalination systems, a transmission structure of "hydrothermal simultaneous transmission" (HST) is proposed. The essence of the IES scheduling problem is the optimal combination of each unit's output, which is a typical timing control problem and conforms to the Markov decision-making solution framework of deep reinforcement learning. Deep reinforcement learning adapts to various changes and timely adjusts strategies through the interaction of agents and the environment, avoiding complicated modeling and prediction of multi-uncertainties. The simulation results show that the proposed scheduling framework properly handles multi-uncertainties from power sources and loads, achieves a stable demand supply for various resources, and has better performance than other real-time scheduling methods, especially in terms of computational efficiency. In addition, the HST model constitutes an active exploration to improve the utilization efficiency of island freshwater. [ABSTRACT FROM AUTHOR]

• A control framework based on deep reinforcement learning is designed to realize the real-time power system emergency generator tripping control. • Considering the generator tripping actions as continuous actions, a deep reinforcement learning algorithm, deep deterministic policy gradient, is employed in the framework. • Convolutional neural network is adopted in the deep deterministic policy gradient algorithm to improve the training speed and the generalization ability of the model. In case of faults or severe disturbances, the power system will enter an emergency operation state. After the system instability is detected, oscillation and blackout will occur in the system if effective control measures are not taken in time. Generator tripping control (GTC) is the most effective emergency control measure. In view of the mismatch between the traditional GTC algorithm and the transient stability assessment method based on machine learning, a new real-time GTC method is needed. In this paper, a three-part control framework is designed for the GTC problem. The control agent is endowed with decision-making ability by interacting with the simulation environment in the offline pre-learning part. Then the trained agent is transplanted to the online application which can help system operators make decisions. Meanwhile, the agent is updated with real data to be better adapted to the actual system in the online learning part. A deep reinforcement learning algorithm, deep deterministic policy gradient (DDPG) is employed to train the control agent in this framework. A modified DDPG algorithm and the corresponding reward function are designed for the GTC problem. Convolution neural network (CNN) is added to the DDPG network, by which the training time of the agent is shortened and the generalization ability of the algorithm is improved. Trained with simulation data and real system experience, the control agent can determine control strategies timely according to the system operating conditions. Simulation results on the IEEE-39 bus system and the realistic regional power system of Eastern China show the effectiveness, generalizability, and timeliness of the decision algorithm. [ABSTRACT FROM AUTHOR]

Traffic crashes, heavy congestion, and discomfort often occur on rough pavements due to human drivers' imperfect decision-making for vehicle control. Autonomous vehicles (AVs) will flood onto urban roads to replace human drivers and improve driving performance in the near future. With the development of the cooperative vehicle infrastructure system (CVIS), multi-source road and traffic information can be collected by onboard or roadside sensors and integrated into a cloud. The information is updated and used for decision-making in real-time. This study proposes an intelligent speed control approach for AVs in CVISs using deep reinforcement learning (DRL) to improve safety, efficiency, and ride comfort. First, the irregular and fluctuating road profiles of rough pavements are represented by maximum comfortable speeds on segments via vertical comfort evaluation. A DRL-based speed control model is then designed to learn safe, efficient, and comfortable car-following behavior based on road and traffic information. Specifically, the model is trained and tested in a stochastic environment using data sampled from 1341 car-following events collected in California and 110 rough pavements detected in Shanghai. The experimental results show that the DRL-based speed control model can improve computational efficiency, driving efficiency, longitudinal comfort, and vertical comfort in cars by 93.47%, 26.99%, 58.33%, and 6.05%, respectively, compared to a model predictive control-based adaptive cruise control. The results indicate that the proposed intelligent speed control approach for AVs is effective on rough pavements and has excellent potential for practical application.

• Weak horizontal and vertical transmission channels are defined. • A power flow adjustment strategy that can fasten the adjustment processing is formulated. • The prior probability of action is given to improve learning efficiency. • A deep reinforcement learning model suitable for power flow adjustment is established. To solve the problem of human resources and time consumption caused by unsolvable power flow of weak power grid, an adjustment method for unsolvable power flow based on weak transmission channels and deep reinforcement learning (DRL) is proposed. Weak horizontal and vertical channels are defined, and their relationship with unsolvable power flow is demonstrated. Then, based on the idea of reducing the load level first and then gradually recovering it, actionable device and action margin are determined based on transmission channel positioning, adjustment sensitivity and solvability degree, so as to form a power flow adjustment strategy. Next, the Markov decision-making process of power flow adjustment is constructed, the prior probability of action is given by adjustment sensitivity of the actionable device, and the action is mapped to the adjustment amounts of generators and capacitors/reactors. Based on soft actor-critic, a DRL model suitable for power flow adjustment is established. Finally, the simulation results show that the proposed method performs well both in the New England 39-bus standard system and an actual power grid of China. Compared with the results obtained by other methods, the superiority of the proposed method is verified. [ABSTRACT FROM AUTHOR]

• Plug-in hybrids with dual-mode engines greatly reduce fuel consumption and emissions. • A hierarchical energy management strategy is proposed for close-to-optimal control. • Compared with mainstream control methods under standard and real-world drive cycles. • Generalization and robustness to random initial and terminal conditions are proved. The dedicated hybrid engines (DHEs) with dual-mode combustion technology can drastically reduce the fuel consumption and emissions while guarantee the power density. This paper aims to investigate the optimal control of such DHE-based plug-in hybrid electric vehicles (PHEVs) under real driving conditions, with minimum fuel penalties caused by transient engine dynamics. For this purpose, the benefits brought by artificial intelligent control and traffic preview in terms of energy efficiency can be combined with the advantages of advanced combustion engine. This paper presents a hierarchical energy management strategy (HEMS) to realize the synergy of global and instantaneous optimization. At the cloud level of HEMS, dynamic programming is applied to obtain optimal combustion mode and state of charge reference trajectories in a receding horizon. At the powertrain level, deep reinforcement learning with a ranking-prioritized experience replay algorithm is used to output optimal engine power and combustion mode for the energy management. To evaluate the proposed strategy, a dual-mode engine with homogeneous charge compression ignition and spark ignition systems is tested and mapped, with which the PHEV is modeled in GT-Suite and Matlab/Simulink. Comprehensive experiments are carried out to verify the optimality, generalization and robustness based on a standard driving cycle and a real-world driving cycle in China with GPS data recorded. The results show that the HEMS avoids frequent switching of combustion modes and outperforms the conventional methods by more than 4% and 10% in terms of fuel economy and NOx emissions, respectively, with random initial and terminal conditions. [ABSTRACT FROM AUTHOR]

In recent years, machine learning for trading has been widely studied. The direction and size of position should be determined in trading decisions based on market conditions. However, there is no research so far that considers variable position sizes in models developed for trading purposes. In this paper, we propose a deep reinforcement learning model named LSTM-DDPG to make trading decisions with variable positions. Specifically, we consider the trading process as a Partially Observable Markov Decision Process, in which the long short-term memory (LSTM) network is used to extract market state features and the deep deterministic policy gradient (DDPG) framework is used to make trading decisions concerning the direction and variable size of position. We test the LSTM-DDPG model on IF300 (index futures of China stock market) data and the results show that LSTM-DDPG with variable positions performs better in terms of return and risk than models with fixed or few-level positions. In addition, the investment potential of the model can be better tapped by the reward function of the differential Sharpe ratio than that of profit reward function. [ABSTRACT FROM AUTHOR]

With the continuous improvement of the power system and the deepening of electricity market reform, the trend of users' active participation in power distribution is more and more significant. Demand response has become the promising focus of smart grid research. Providing reasonable incentive strategies for power grid companies and demand response strategies for customers plays a crucial role in maximizing the benefits of different participants. To meet different expectations of multiple agents in the same environment, deep reinforcement learning was adopted. The generative model of residential demand response strategy under different incentive policies can be trained iteratively through real-time interactions with the environmental conditions. In this paper, a novel optimization model of residential demand response strategy, based on a deep deterministic policy gradient (DDPG) algorithm, was proposed. The proposed work was validated with the actual electricity consumption data of a certain area in China. The results showed that the DDPG model could optimize residential demand response strategy under certain incentive policies. In addition, the overall goal of peak load-cutting and valley filling can be achieved, which reflects promising prospects of the electricity market. [ABSTRACT FROM AUTHOR]

Humanoid robots are equipped with humanoid arms to make them more acceptable to the general public. Humanoid robots are a great challenge in robotics. The concept of digital twin technology complies with the guiding ideology of not only Industry 4.0, but also Made in China 2025. This paper proposes a scheme that combines deep reinforcement learning (DRL) with digital twin technology for controlling humanoid robot arms. For rapid and stable motion planning for humanoid robots, multitasking-oriented training using the twin synchro-control (TSC) scheme with DRL is proposed. For switching between tasks, the robot arm training must be quick and diverse. In this work, an approach for obtaining a priori knowledge as input to DRL is developed and verified using simulations. Two simple examples are developed in a simulation environment. We developed a data acquisition system to generate angle data efficiently and automatically. These data are used to improve the reward function of the deep deterministic policy gradient (DDPG) and quickly train the robot for a task. The approach is applied to a model of the humanoid robot BHR-6, a humanoid robot with multiple-motion mode and a sophisticated mechanical structure. Using the policies trained in the simulations, the humanoid robot can perform tasks that are not possible to train with existing methods. The training is fast and allows the robot to perform multiple tasks. Our approach utilizes human joint angle data collected by the data acquisition system to solve the problem of a sparse reward in DRL for two simple tasks. A comparison with simulation results for controllers trained using the vanilla DDPG show that the designed controller developed using the DDPG with the TSC scheme have great advantages in terms of learning stability and convergence speed. [ABSTRACT FROM AUTHOR]