Search

Federated Reinforcement Learning (FedRL) encourages distributed agents to learn collectively from each other's experience to improve their performance without exchanging their raw trajectories. The existing work on FedRL assumes that all participating agents are homogeneous, which requires all agents to share the same policy parameterization (e.g., network architectures and training configurations). However, in real-world applications, agents are often in disagreement about the architecture and the parameters, possibly also because of disparate computational budgets. Because homogeneity is not given in practice, we introduce the problem setting of Federated Reinforcement Learning with Heterogeneous And bLack-box agEnts (FedRL-HALE). We present the unique challenges this new setting poses and propose the Federated Heterogeneous Q-Learning (FedHQL) algorithm that principally addresses these challenges. We empirically demonstrate the efficacy of FedHQL in boosting the sample efficiency of heterogeneous agents with distinct policy parameterization using standard RL tasks., Preprint. Under review

Interaction between the background vehicles (BVs) and automated vehicles (AVs) in scenario-based testing plays a critical role in evaluating the intelligence of the AVs. Current testing scenarios typically employ predefined or scripted BVs, which inadequately reflect the complexity of human-like social behaviors in real-world driving scenarios, and also lack a systematic metric for evaluating the comprehensive intelligence of AVs. Therefore, this paper proposes an evolving scenario generation method that utilizes deep reinforcement learning (DRL) to create human-like BVs for testing and intelligence evaluation of AVs. Firstly, a class of driver models with human-like competitive, cooperative, and mutual driving motivations is designed. Then, utilizing an improved "level-k" training procedure, the three distinct driver models acquire game-based interactive driving policies. And these models are assigned to BVs for generating evolving scenarios in which all BVs can interact continuously and evolve diverse contents. Next, a framework including safety, driving efficiency, and interaction utility are presented to evaluate and quantify the intelligence performance of 3 systems under test (SUTs), indicating the effectiveness of the evolving scenario for intelligence testing. Finally, the complexity and fidelity of the proposed evolving testing scenario are validated. The results demonstrate that the proposed evolving scenario exhibits the highest level of complexity compared to other baseline scenarios and has more than 85% similarity to naturalistic driving data. This highlights the potential of the proposed method to facilitate the development and evaluation of high-level AVs in a realistic and challenging environment., Comment: 18 pages,17 figures

Recent neural methods for vehicle routing problems always train and test the deep models on the same instance distribution (i.e., uniform). To tackle the consequent cross-distribution generalization concerns, we bring the knowledge distillation to this field and propose an Adaptive Multi-Distribution Knowledge Distillation (AMDKD) scheme for learning more generalizable deep models. Particularly, our AMDKD leverages various knowledge from multiple teachers trained on exemplar distributions to yield a light-weight yet generalist student model. Meanwhile, we equip AMDKD with an adaptive strategy that allows the student to concentrate on difficult distributions, so as to absorb hard-to-master knowledge more effectively. Extensive experimental results show that, compared with the baseline neural methods, our AMDKD is able to achieve competitive results on both unseen in-distribution and out-of-distribution instances, which are either randomly synthesized or adopted from benchmark datasets (i.e., TSPLIB and CVRPLIB). Notably, our AMDKD is generic, and consumes less computational resources for inference., Comment: Accepted at NeurIPS 2022

With the development of online information sharing, high-tech equipment for collaborative production management of power enterprises emerges endlessly. Therefore, it is necessary to design the collaborative production management system of power enterprises based on online information sharing to meet the information sharing needs. In terms of the hardware, the B/S structure was built, and the computer was debugged with Cascading Style Sheet (CSS). In terms of the software, Hadoop horizontal architecture technology framework was designed, the physical deployment was carried out, the production management center module was designed, and the production operation chain was monitored and managed to realize the collaborative production management of power enterprises. The experimental results showed that the designed collaborative production management system of power enterprise had high reliability and friendliness, the highest reliability is 97.2%, the highest friendliness is 99.8%, which meets the current demand.

Drought is a major natural disaster that causes a reduction in rain-fed maize yield. Agricultural drought risk assessment is conducive to improving regional disaster management ability, thereby reducing food security risks and economic losses. Considering the complexity of risk assessment research, an increasing number of researchers are focusing on the multiple-criteria decision-making (MCDM) method. However, the applicability of the MCDM method to agro-meteorological disaster risk assessments is not clear. Therefore, this study comprehensively evaluated hazard, exposure, vulnerability, and emergency response and recovery capability using the TOPSIS and VIKOR models to generate a maize drought risk map in mid-western Jilin Province and ranked the drought risk of each county. The results showed that: (1) maize drought risk in the middle and west of Jilin province showed an increasing trend. Spatially, the risk diminished from west to east. The drought risks faced by Tongyu, Changchun, and Dehui were more severe; (2) the evaluation results of the two models were verified using the yield reduction rate. The VIKOR model was found to be more suitable for agrometeorological drought risk assessments; (3) according to the damage degree of drought disaster to maize, the cluster analysis method was used to divide the study area into three sub-regions: safe, moderate drought, and severe drought. Combined with the characteristics of different regions, suggestions on disaster prevention and mitigation are proposed. The results of this study can provide a basis for formulating strategies to alleviate drought, reduce losses, and ensure the sustainable development of agriculture.

Existing deep reinforcement learning (DRL) based methods for solving the capacitated vehicle routing problem (CVRP) intrinsically cope with homogeneous vehicle fleet, in which the fleet is assumed as repetitions of a single vehicle. Hence, their key to construct a solution solely lies in the selection of the next node (customer) to visit excluding the selection of vehicle. However, vehicles in real-world scenarios are likely to be heterogeneous with different characteristics that affect their capacity (or travel speed), rendering existing DRL methods less effective. In this paper, we tackle heterogeneous CVRP (HCVRP), where vehicles are mainly characterized by different capacities. We consider both min-max and min-sum objectives for HCVRP, which aim to minimize the longest or total travel time of the vehicle(s) in the fleet. To solve those problems, we propose a DRL method based on the attention mechanism with a vehicle selection decoder accounting for the heterogeneous fleet constraint and a node selection decoder accounting for the route construction, which learns to construct a solution by automatically selecting both a vehicle and a node for this vehicle at each step. Experimental results based on randomly generated instances show that, with desirable generalization to various problem sizes, our method outperforms the state-of-the-art DRL method and most of the conventional heuristics, and also delivers competitive performance against the state-of-the-art heuristic method, i.e., SISR. Additionally, the results of extended experiments demonstrate that our method is also able to solve CVRPLib instances with satisfactory performance., This paper has been accepted at IEEE Transactions on Cybernetics

Recently, Transformer has become a prevailing deep architecture for solving vehicle routing problems (VRPs). However, it is less effective in learning improvement models for VRP because its positional encoding (PE) method is not suitable in representing VRP solutions. This paper presents a novel Dual-Aspect Collaborative Transformer (DACT) to learn embeddings for the node and positional features separately, instead of fusing them together as done in existing ones, so as to avoid potential noises and incompatible correlations. Moreover, the positional features are embedded through a novel cyclic positional encoding (CPE) method to allow Transformer to effectively capture the circularity and symmetry of VRP solutions (i.e., cyclic sequences). We train DACT using Proximal Policy Optimization and design a curriculum learning strategy for better sample efficiency. We apply DACT to solve the traveling salesman problem (TSP) and capacitated vehicle routing problem (CVRP). Results show that our DACT outperforms existing Transformer based improvement models, and exhibits much better generalization performance across different problem sizes on synthetic and benchmark instances, respectively., Comment: Accepted at NeurIPS 2021. Changes in v3: updated CVRP performance due to more accurate feasibility masks (reduced training time); improved comparison analysis in Table 1 and Section 5; fixed minor typos/inaccuracies in the CPE equation, Figure 6, and other sentences

The growing literature of Federated Learning (FL) has recently inspired Federated Reinforcement Learning (FRL) to encourage multiple agents to federatively build a better decision-making policy without sharing raw trajectories. Despite its promising applications, existing works on FRL fail to I) provide theoretical analysis on its convergence, and II) account for random system failures and adversarial attacks. Towards this end, we propose the first FRL framework the convergence of which is guaranteed and tolerant to less than half of the participating agents being random system failures or adversarial attackers. We prove that the sample efficiency of the proposed framework is guaranteed to improve with the number of agents and is able to account for such potential failures or attacks. All theoretical results are empirically verified on various RL benchmark tasks., Comment: Published at NeurIPS 2021. Extended version with proofs and additional experimental details and results. New version changes: reduced file size of figures; added a diagram illustrating the problem setting; added link to code on GitHub; modified proof for Theorem 6 (highlighted in red)

Abstract: Ecological region in southern China has been perennially affected by monsoon climate and anthropogenic emissions, resulting in complex aerosol components and frequent long-range transport. In this study, a Santa Barbara DISORT Atmospheric Radiative Transfer (SBDART) model is applied to estimate aerosol radiative forcing (ARF) and multiple aerosol observation datasets is used to estimate the aerosol chemical components and optical properties. The aerosol loading and the radiative effects in the ecological region exhibited strong seasonal changes. The average major components (NH4+, NO3−, SO42−) in Total water soluble ionic (TWSI) ,organic carbon (OC) concentration, the ratio of organic carbon to element carbon (OC/EC) and biogenic secondary organic aerosol (BSOA) tracers were 3.20±1.22 μg·m-3, 2.19±1.39 μg·m-3, 3.17 and 74.00±35.23 ng·m-3 in the dry season and 2.22±0.91 μg·m-3, 3.14±1.62 μg·m-3, 7.13 and 186.34±113.82 ng·m-3 in the wet season, respectively. The average radiative forcing at the top of atmosphere (TOA) is -11.73±11.36 W/m2 and -0.41±10.08 W/m2 in dry and wet season. When the aerosol single scattering albedo (SSA) less than 0.9, the retrieve frequency in wet season reached account for 75%. The increase of OC and BSOA transformed by forests in the wet season weaken the cooling effects. However, the dry season is mainly composed of anthropogenic inorganic aerosols, which enhances the scattering effect. The aerosol observation baseline also verified the seasonal variation of ARF in the ecological region. Driven by multiple factors such as meteorological conditions, emission sources, and the mixed state of particulate matter, the transport patterns of air masses in ecological area exhibits completely opposite affects to ARF.

A new, low-cost and locally available sorbent, pomelo peel (PP), was tested for its ability to remove spilled oil from simulated seawater. The experiments were conducted to evaluate the oil sorption capacities of PP modified by fatty acids (oleic acid and stearic acid). The effects of temperature, salinity, and oscillation frequency on the oil sorption capacity of the PP were studied in simulated seawater. It was found that the oil sorption capacity of the PP was greatly enhanced by the surface modification. The results showed that the PP modified by oleic acid had better oil sorption capacity than that treated with stearic acid. The sorption kinetics of unmodified and modified PP were well described by the pseudo-second-order kinetic model. The results indicated that PP was an attractive candidate for removing oily pollutants from seawater.

Graphene (GN) is a two-dimensional carbon material with highly hydrophobic and oleophilic properties, and it is prepared by a modified Hummers Method. In this study, superhydrophobic and superoleophilic polyurethane (PU) sponge was fabricated by coating graphene onto it. The contact angle of the coated sponge was as large as 156.1° for water, and 0° for pump oil, soybean oil, and diesel oil. The absorption capability of the coated sponges was up to 60 times their own weight. Additionally, the coated PU sponges could be reused for oil–water separation for more than 60 cycles without losing their superhydrophobic-superoleophilic properties. Therefore, the sponges can be used for the cleanup of oil spills from a water surface.