Search

Online Reinforcement learning (RL) typically requires high-stakes online interaction data to learn a policy for a target task. This prompts interest in leveraging historical data to improve sample efficiency. The historical data may come from outdated or related source environments with different dynamics. It remains unclear how to effectively use such data in the target task to provably enhance learning and sample efficiency. To address this, we propose a hybrid transfer RL (HTRL) setting, where an agent learns in a target environment while accessing offline data from a source environment with shifted dynamics. We show that -- without information on the dynamics shift -- general shifted-dynamics data, even with subtle shifts, does not reduce sample complexity in the target environment. However, with prior information on the degree of the dynamics shift, we design HySRL, a transfer algorithm that achieves problem-dependent sample complexity and outperforms pure online RL. Finally, our experimental results demonstrate that HySRL surpasses state-of-the-art online RL baseline.

Multi-objective reinforcement learning (MORL) excels at handling rapidly changing preferences in tasks that involve multiple criteria, even for unseen preferences. However, previous dominating MORL methods typically generate a fixed policy set or preference-conditioned policy through multiple training iterations exclusively for sampled preference vectors, and cannot ensure the efficient discovery of the Pareto front. Furthermore, integrating preferences into the input of policy or value functions presents scalability challenges, in particular as the dimension of the state and preference space grow, which can complicate the learning process and hinder the algorithm's performance on more complex tasks. To address these issues, we propose a two-stage Pareto front discovery algorithm called Constrained MORL (C-MORL), which serves as a seamless bridge between constrained policy optimization and MORL. Concretely, a set of policies is trained in parallel in the initialization stage, with each optimized towards its individual preference over the multiple objectives. Then, to fill the remaining vacancies in the Pareto front, the constrained optimization steps are employed to maximize one objective while constraining the other objectives to exceed a predefined threshold. Empirically, compared to recent advancements in MORL methods, our algorithm achieves more consistent and superior performances in terms of hypervolume, expected utility, and sparsity on both discrete and continuous control tasks, especially with numerous objectives (up to nine objectives in our experiments)., Comment: 27 pages, 8 figues. In Submission to a conference

This work examines the conditions for asymptotic and exponential convergence of saddle flow dynamics of convex-concave functions. First, we propose an observability-based certificate for asymptotic convergence, directly bridging the gap between the invariant set in a LaSalle argument and the equilibrium set of saddle flows. This certificate generalizes conventional conditions for convergence, e.g., strict convexity-concavity, and leads to a novel state-augmentation method that requires minimal assumptions for asymptotic convergence. We also show that global exponential stability follows from strong convexity-strong concavity, providing a lower-bound estimate of the convergence rate. This insight also explains the convergence of proximal saddle flows for strongly convex-concave objective functions. Our results generalize to dynamics with projections on the vector field and have applications in solving constrained convex optimization via primal-dual methods. Based on these insights, we study four algorithms built upon different Lagrangian function transformations. We validate our work by applying these methods to solve a network flow optimization and a Lasso regression problem.

We address the problem of characterizing the aggregate flexibility in populations of electric vehicles (EVs) with uncertain charging requirements. Extending upon prior results that provide exact characterizations of aggregate flexibility in populations of electric vehicle (EVs), we adapt the framework to encompass more general charging requirements. In doing so we give a characterization of the exact aggregate flexibility as a generalized polymatroid. Furthermore, this paper advances these aggregation methodologies to address the case in which charging requirements are uncertain. In this extended framework, requirements are instead sampled from a specified distribution. In particular, we construct robust aggregate flexibility sets, sets of aggregate charging profiles over which we can provide probabilistic guarantees that actual realized populations will be able to track. By leveraging measure concentration results that establish powerful finite sample guarantees, we are able to give tight bounds on these robust flexibility sets, even in low sample regimes that are well suited for aggregating small populations of EVs. We detail explicit methods of calculating these sets. Finally, we provide numerical results that validate our results and case studies that demonstrate the applicability of the theory developed herein., Comment: 10 pages, conference

As a critical component of modern infrastructure, data centers account for a huge amount of power consumption and greenhouse gas emission. This paper studies the electricity purchase strategy for a data center to lower its energy cost while integrating local renewable generation under uncertainty. To facilitate efficient and scalable decision-making, we propose a two-layer hierarchy where the lower layer consists of the operation of all electrical equipment in the data center and the upper layer determines the procurement and dispatch of electricity. At the lower layer, instead of device-level scheduling in real time, we propose to exploit the inherent flexibility in demand, such as thermostatically controlled loads and flexible computing tasks, and aggregate them into virtual batteries. By this means, the upper-layer decision only needs to take into account these virtual batteries, the size of which is generally small and independent of the data center scale. We further propose an online algorithm based on Lyapunov optimization to purchase electricity from the grid with a manageable energy cost, even though the prices, renewable availability, and battery specifications are uncertain and dynamic. In particular, we show that, under mild conditions, our algorithm can achieve bounded loss compared with the offline optimal cost, while strictly respecting battery operational constraints. Extensive simulation studies validate the theoretical analysis and illustrate the tradeoff between optimality and conservativeness.

Gradient Descent Ascent (GDA) methods for min-max optimization problems typically produce oscillatory behavior that can lead to instability, e.g., in bilinear settings. To address this problem, we introduce a dissipation term into the GDA updates to dampen these oscillations. The proposed Dissipative GDA (DGDA) method can be seen as performing standard GDA on a state-augmented and regularized saddle function that does not strictly introduce additional convexity/concavity. We theoretically show the linear convergence of DGDA in the bilinear and strongly convex-strongly concave settings and assess its performance by comparing DGDA with other methods such as GDA, Extra-Gradient (EG), and Optimistic GDA. Our findings demonstrate that DGDA surpasses these methods, achieving superior convergence rates. We support our claims with two numerical examples that showcase DGDA's effectiveness in solving saddle point problems.

Electricity markets typically operate in two stages, day-ahead and real-time. Despite best efforts striving efficiency, evidence of price manipulation has called for system-level market power mitigation (MPM) initiatives that substitute noncompetitive bids with default bids. Implementing these policies with a limited understanding of participant behavior may lead to unintended economic losses. In this paper, we model the competition between generators and inelastic loads in a two-stage market with stage-wise MPM policies. The loss of Nash equilibrium and lack of guarantee of stable market outcome in the case of conventional supply function bidding motivates the use of an alternative market mechanism where generators bid an intercept function. A Nash equilibrium analysis for a day-ahead MPM policy leads to a Stackelberg-Nash game with loads exercising market power at the expense of generators. A comparison of the resulting equilibrium with the standard market (not implementing any MPM policy) shows that a day-ahead policy completely mitigates the market power of generators. On the other hand, the real-time MPM policy increases demand allocation to real-time, contrary to current market practice with most electricity trades in the day-ahead market. Numerical studies illustrate the impact of the slope of the intercept function on the standard market.

The main goal of a sequential two-stage electricity market -- e.g., day-ahead and real-time markets -- is to operate efficiently. However, the price difference across stages due to inadequate competition and unforeseen circumstances leads to undesirable price manipulation. To mitigate this, some Independent System Operators (ISOs) proposed system-level market power mitigation (MPM) policies in addition to existing local policies. These policies aim to substitute noncompetitive bids with a default bid based on estimated generator costs. However, these policies may lead to unintended consequences when implemented without accounting for the conflicting interest of participants. In this paper, we model the competition between generators (bidding supply functions) and loads (bidding quantity) in a two-stage market with a stage-wise MPM policy. An equilibrium analysis shows that a real-time MPM policy leads to equilibrium loss, meaning no stable market outcome (Nash equilibrium) exists. A day-ahead MPM policy, besides, leads to a Stackelberg-Nash game with loads acting as leaders and generators as followers. In this setting, loads become winners, i.e., their aggregate payment is always less than competitive payments. Moreover, comparison with standard market equilibrium highlights that markets are better off without such policies. Finally, numerical studies highlight the impact of heterogeneity and load size on market equilibrium.

In constrained reinforcement learning (C-RL), an agent seeks to learn from the environment a policy that maximizes the expected cumulative reward while satisfying minimum requirements in secondary cumulative reward constraints. Several algorithms rooted in sampled-based primal-dual methods have been recently proposed to solve this problem in policy space. However, such methods are based on stochastic gradient descent ascent algorithms whose trajectories are connected to the optimal policy only after a mixing output stage that depends on the algorithm's history. As a result, there is a mismatch between the behavioral policy and the optimal one. In this work, we propose a novel algorithm for constrained RL that does not suffer from these limitations. Leveraging recent results on regularized saddle-flow dynamics, we develop a novel stochastic gradient descent-ascent algorithm whose trajectories converge to the optimal policy almost surely.

This paper focuses on the operation of an electricity market that accounts for participants that bid at a sub-minute timescale. To that end, we model the market-clearing process as a dynamical system, called market dynamics, which is temporally coupled with the grid frequency dynamics and is thus required to guarantee system-wide stability while meeting the system operational constraints. We characterize participants as price-takers who rationally update their bids to maximize their utility in response to real-time schedules of prices and dispatch. For two common bidding mechanisms, based on quantity and price, we identify a notion of alignment between participants' behavior and planners' goals that leads to a saddle-based design of the market that guarantees convergence to a point meeting all operational constraints. We further explore cases where this alignment property does not hold and observe that misaligned participants' bidding can destabilize the closed-loop system. We thus design a regularized version of the market dynamics that recovers all the desirable stability and steady-state performance guarantees. Numerical tests validate our results on the IEEE 39-bus system.

This paper proposes a market mechanism for multi-interval electricity markets with generator and storage participants. Drawing ideas from supply function bidding, we introduce a novel bid structure for storage participation that allows storage units to communicate their cost to the market using energy-cycling functions that map prices to cycle depths. The resulting market-clearing process--implemented via convex programming--yields corresponding schedules and payments based on traditional energy prices for power supply and per-cycle prices for storage utilization. We illustrate the benefits of our solution by comparing the competitive equilibrium of the resulting mechanism to that of an alternative solution that uses prosumer-based bids. Our solution shows several advantages over the prosumer-based approach. It does not require a priori price estimation. It also incentivizes participants to reveal their truthful cost, thus leading to an efficient, competitive equilibrium. Numerical experiments using New York Independent System Operator (NYISO) data validate our findings.

This paper investigates the efficiency loss in social cost caused by strategic bidding behavior of individual participants in a supply-demand balancing market, and proposes a mechanism to fully recover equilibrium social optimum via subsidization and taxation. We characterize the competition among supply-side firms to meet given inelastic demand, with linear supply function bidding and the proposed efficiency recovery mechanism. We show that the Nash equilibrium of such a game exists under mild conditions, and more importantly, it achieves the underlying efficient supply dispatch and the market clearing price that reflects the truthful system marginal production cost. Further, the mechanism can be tuned to guarantee self-sufficiency, i.e., taxes collected counterbalance subsidies needed. Extensive numerical case studies are run to validate the equilibrium analysis, and we employ individual net profit and a modified version of Lerner index as two metrics to evaluate the impact of the mechanism on market outcomes by varying its tuning parameter and firm heterogeneity.

This paper proposes a certificate, rooted in observability, for asymptotic convergence of saddle flow dynamics of convex-concave functions to a saddle point. This observable certificate directly bridges the gap between the invariant set and the equilibrium set in a LaSalle argument, and generalizes conventional conditions such as strict convexity-concavity and proximal regularization. We further build upon this certificate to propose a separable regularization method for saddle flow dynamics that makes minimal requirements on convexity-concavity and yet still guarantees asymptotic convergence to a saddle point. Our results generalize to saddle flow dynamics with projections on the vector field and have an immediate application as a distributed solution to linear programs.

Two-stage electricity market clearing is designed to maintain market efficiency under ideal conditions, e.g., perfect forecast and nonstrategic generation. This work demonstrates that the individual strategic behavior of inelastic load participants in a two-stage settlement electricity market can deteriorate efficiency. Our analysis further implies that virtual bidding can play a role in alleviating this loss of efficiency by mitigating the market power of strategic load participants. We use real-world market data from New York ISO to validate our theory., Comment: IEEE Conference on Decision and Control 2019

In Part I of this paper we formulate an optimal scheduling problem for battery swapping that assigns to each electric vehicle (EV) a best station to swap its depleted battery based on its current location and state of charge. The schedule aims to minimize a weighted sum of total travel distance and generation cost over both station assignments and power flow variables, subject to EV range constraints, grid operational constraints and AC power flow equations. We propose there a centralized solution based on the second-order cone programming (SOCP) relaxation of optimal power flow (OPF) and generalized Benders decomposition that is suitable when global information is available. In this paper we propose two distributed solutions based on the alternating direction method of multipliers (ADMM) and dual decomposition respectively that are suitable for cases where the distribution grid, battery stations and EVs are managed by separate entities. Our algorithms allow these entities to make individual decisions but coordinate through privacy-preserving information exchanges to jointly solve an approximate version of the joint battery swapping scheduling and OPF problem. We evaluate our algorithms through simulations.

We formulate an optimal scheduling problem for battery swapping that assigns to each electric vehicle (EV) a best station to swap its depleted battery based on its current location and state of charge. The schedule aims to minimize total travel distance and generation cost over both station assignments and power flow variables, subject to EV range constraints, grid operational constraints and AC power flow equations. To deal with the nonconvexity of power flow equations and the binary nature of station assignments, we propose a solution based on second-order cone programming (SOCP) relaxation of optimal power flow (OPF) and generalized Benders decomposition. When the SOCP relaxation is exact, this approach computes a globally optimal solution. We evaluate the performance of the proposed algorithm through simulations. The algorithm requires global information and is suitable for cases where the distribution network, stations, and EVs are managed centrally by the same operator. In Part II of the paper, we develop distributed solutions for cases where they are operated by different organizations that do not share private information.