Bi-level hierarchical model with deep reinforcement learning-based extended horizon scheduling for integrated electricity-heat systems.

• An innovative scheduling scheme with an extended decision horizon is proposed for the IEHS, which considers the optimal status of the HST at the end of each dispatch period. • A bi-level hierarchical model integrating an upper-level DRL-based model with a lower-level mathematical programming-based model is developed. • The determination of EHC is formulated as a Markov decision process in the upper-level model and solved through DRL methods. • The beta policy is introduced to the proximal policy optimization algorithm to obtain the optimal DRL policy. The integration of heat storage tank (HST) in the integrated electricity–heat system (IEHS) has been widely adopted to facilitate the accommodation of wind power. However, existing studies generally put a simplified end-of-period heat capacity (EHC) constraints of the HST, which limits the utilization of potential heat supply in subsequent dispatch periods. To addresses the issue, this paper proposes an innovative scheduling scheme with an extended decision horizon for the IEHS, considering the optimal status of the HST at the end of each dispatch period. To implement this scheme, a bi-level hierarchical model is developed, which integrates a high-level planning model based on deep reinforcement learning with a low-level scheduling model based on mathematical programming. The high-level planning model aims to learn the optimal policy for determining a suitable EHC in accordance with the heat release needs of subsequent dispatch periods. Meanwhile, the low-level scheduling model incorporates the EHC as a constraint and formulates a mixed-integer quadratic programming subproblem, wherein the reward is computed and fed back to the high-level model to guide the policy learning. Experimental results conducted on the modified 6-bus test system verified the effectiveness and economic benefits of the proposed scheduling scheme, which significantly enhances the wind power accommodation capacity of the IEHS. [ABSTRACT FROM AUTHOR]