Search

Energy storage scheduling problems, where a storage is operated to maximize its profit in response to a price signal, are essentially infinite-horizon optimization problems as storage systems operate continuously, without a foreseen end to their operation. Such problems can be solved to optimality with a rolling-horizon approach, provided that the planning horizon over which the problem is solved is long enough. Such a horizon is termed a forecast horizon. However, the length of the planning horizon is usually chosen arbitrarily for such applications. We introduce an easy-to-check condition that confirms whether a planning horizon is a forecast horizon, and which can be used to derive a bound on suboptimality when it is not the case. By way of an example, we demonstrate that the existence of forecast horizons is not guaranteed for this problem. We also derive a lower bound on the length of the minimum forecast horizon. We show how the condition introduced can be used as part of an algorithm to determine the minimum forecast horizon of the problem, which ensures the determination of optimal solutions at the lowest computational and forecasting costs. Finally, we provide insights into the implications of different planning horizons for a range of storage system characteristics.

The growing uncertainty from renewable power and electricity demand brings significant challenges to unit commitment (UC). While various advanced forecasting and optimization methods have been developed to predict better and address this uncertainty, most previous studies treat forecasting and optimization as separate tasks. This separation can lead to suboptimal results due to misalignment between the objectives of the two tasks. To overcome this challenge, we propose a robust UC framework that integrates the forecasting and optimization processes while ensuring statistical guarantees. In the forecasting stage, we combine multiple predictions derived from diverse data sources and methodologies for an improved prediction, aiming to optimize the UC performance. In the optimization stage, the combined prediction is used to construct an uncertainty set with statistical guarantees, based on which the robust UC model is formulated. The optimal robust UC solution provides feedback to refine the forecasting process, forming a closed loop. To solve the proposed integrated forecasting-optimization framework efficiently and effectively, we develop a neural network-based surrogate model for acceleration and introduce a reshaping method for the uncertainty set based on the optimization result to reduce conservativeness. Case studies on modified IEEE 30-bus and 118-bus systems demonstrate the advantages of the proposed approach., Comment: 11 pages, 9 figures

Complex distribution shifts are the main obstacle to achieving accurate long-term time series forecasting. Several efforts have been conducted to capture the distribution characteristics and propose adaptive normalization techniques to alleviate the influence of distribution shifts. However, these methods neglect the intricate distribution dynamics observed from various scales and the evolving functions of distribution dynamics and normalized mapping relationships. To this end, we propose a novel model-agnostic Evolving Multi-Scale Normalization (EvoMSN) framework to tackle the distribution shift problem. Flexible normalization and denormalization are proposed based on the multi-scale statistics prediction module and adaptive ensembling. An evolving optimization strategy is designed to update the forecasting model and statistics prediction module collaboratively to track the shifting distributions. We evaluate the effectiveness of EvoMSN in improving the performance of five mainstream forecasting methods on benchmark datasets and also show its superiority compared to existing advanced normalization and online learning approaches. The code is publicly available at https://github.com/qindalin/EvoMSN.

As seasonal thermal energy storage emerges as an efficient solution to reduce CO2 emissions of buildings, challenges appear related to its optimal operation. In a system including short-term electricity storage, long-term heat storage, and where electricity and heat networks are connected through a heat pump, it becomes crucial to operate the system on two time scales. Based on real data from a university building, we simulate the operation of such a system over a year, comparing different strategies based on model predictive control (MPC). The first objective of this paper is to determine the minimum prediction horizon to retrieve the results of the full-horizon operation problem with cost minimization. The second objective is to evaluate a method that combines MPC with setting targets on the heat storage level at the end of the prediction horizon, based on historical data. For a prediction horizon of 6 days, the suboptimality gap with the full-horizon results is 4.31%, compared to 11.42% when using a prediction horizon of 42 days and fixing the final level to be equal to the initial level, which is a common approach.

The risk of missing data and subsequent incomplete data records at wind farms increases with the number of turbines and sensors. We propose here an imputation method that blends data-driven concepts with expert knowledge, by using the geometry of the wind farm in order to provide better estimates when performing Nearest Neighbor imputation. Our method relies on learning Laplacian eigenmaps out of the graph of the wind farm through spectral graph theory. These learned representations can be based on the wind farm layout only, or additionally account for information provided by collected data. The related weighted graph is allowed to change with time and can be tracked in an online fashion. Application to the Westermost Rough offshore wind farm shows significant improvement over approaches that do not account for the wind farm layout information., Comment: 16 pages, 5 figures, 7 tables

This paper proposes a novel capacity credit evaluation framework to quantify the contribution of generalized energy storage (GES) to resource adequacy, considering both decision-independent uncertainty (DIU) and decision-dependent uncertainty (DDU). To this end, we establish a market-oriented risk-averse re-dispatch method to capture the cross-market reliable operation of GES. The proposed method is sequentially implemented along with the Monte Carlo simulation process, coordinating the pre-dispatched price arbitrage and capacity withholding in the energy market with adequacy-oriented re-dispatch during capacity market calls. In addition to DIUs in operational states and baseline behavior, we explicitly address the inherent DDU of GES (i.e., the uncertainty of available discharge capacity affected by the incentives and accumulated discomfort) during the re-dispatch stage using the proposed adaptive data-driven chance-constrained approach. Furthermore, a capacity credit metric called equivalent storage capacity substitution is introduced to quantify the equivalent deterministic storage capacity of uncertain GES. Simulations on the modified IEEE RTS-79 benchmark system demonstrate that the proposed method yields accurate capacity credit and improved economic performance. Key factors impacting GES's capacity credit are discussed with insights into capacity market decision-making., Comment: This is a manuscript submitted to IEEE Transcations on Power Systems

Improving the future of healthcare starts by better understanding the current actual practices in hospital settings. This motivates the objective of discovering typical care pathways from patient data. Revealing typical care pathways can be achieved through clustering. The difficulty in clustering care pathways, represented by sequences of timestamped events, lies in defining a semantically appropriate metric and clustering algorithms. In this article, we adapt two methods developed for time series to the clustering of timed sequences: the drop-DTW metric and the DBA approach for the construction of averaged time sequences. These methods are then applied in clustering algorithms to propose original and sound clustering algorithms for timed sequences. This approach is experimented with and evaluated on synthetic and real-world data.

Machine learning techniques have been successfully used in probabilistic wind power forecasting. However, the issue of missing values within datasets due to sensor failure, for instance, has been overlooked for a long time. Although it is natural to consider addressing this issue by imputing missing values before model estimation and forecasting, we suggest treating missing values and forecasting targets indifferently and predicting all unknown values simultaneously based on observations. In this paper, we offer an efficient probabilistic forecasting approach by estimating the joint distribution of features and targets based on a generative model. It is free of preprocessing, and thus avoids introducing potential errors. Compared with the traditional "impute, then predict" pipeline, the proposed approach achieves better performance in terms of continuous ranked probability score., Comment: 8 pages, to be presented at Power Systems Computation Conference (PSCC) 2024

This paper studies how to aggregate prosumers (or large consumers) and their collective decisions in electricity markets, with a focus on fairness. Fairness is essential for prosumers to participate in aggregation schemes. Some prosumers may not be able to access the energy market directly, even though it would be beneficial for them. Therefore, new companies offer to aggregate them and promise to treat them fairly. This leads to a fair resource allocation problem. We propose to use acceptability constraints to guarantee that each prosumer gains from the aggregation. Moreover, we aim to distribute the costs and benefits fairly, taking into account the multi-period and uncertain nature of the problem. Rather than using financial mechanisms to adjust for fairness issues, we focus on various objectives and constraints, within decision problems, that achieve fairness by design. We start from a simple single-period and deterministic model, and then generalize it to a dynamic and stochastic setting using, e.g., stochastic dominance constraints.

Energy forecasting is pivotal in energy systems, by providing fundamentals for operation, with different horizons and resolutions. Though energy forecasting has been widely studied for capturing temporal information, very few works concentrate on the frequency information provided by forecasts. They are consequently often limited to single-resolution applications (e.g., hourly). Here, we propose a unified energy forecasting framework based on Laplace transform in the multi-resolution context. The forecasts can be seamlessly produced at different desired resolutions without re-training or post-processing. Case studies on both energy demand and supply data show that the forecasts from our proposed method can provide accurate information in both time and frequency domains. Across the resolutions, the forecasts also demonstrate high consistency. More importantly, we explore the operational effects of our produced forecasts in the day-ahead and intra-day energy scheduling. The relationship between (i) errors in both time and frequency domains and (ii) operational value of the forecasts is analysed. Significant operational benefits are obtained.

Data markets facilitate decentralized data exchange for applications such as prediction, learning, or inference. The design of these markets is challenged by varying privacy preferences as well as data similarity among data owners. Related works have often overlooked how data similarity impacts pricing and data value through statistical information leakage. We demonstrate that data similarity and privacy preferences are integral to market design and propose a query-response protocol using local differential privacy for a two-party data acquisition mechanism. In our regression data market model, we analyze strategic interactions between privacy-aware owners and the learner as a Stackelberg game over the asked price and privacy factor. Finally, we numerically evaluate how data similarity affects market participation and traded data value., Comment: Submitted to IEEE Transactions on Neural Networks and Learning Systems

Although machine learning tasks are highly sensitive to the quality of input data, relevant datasets can often be challenging for firms to acquire, especially when held privately by a variety of owners. For instance, if these owners are competitors in a downstream market, they may be reluctant to share information. Focusing on supervised learning for regression tasks, we develop a regression market to provide a monetary incentive for data sharing. Our mechanism adopts a Bayesian framework, allowing us to consider a more general class of regression tasks. We present a thorough exploration of the market properties, and show that similar proposals in literature expose the market agents to sizeable financial risks, which can be mitigated in our setup., Comment: 35 pages, 11 figures, 3 tables. Published in Journal of Machine Learning Research (2024)

Despite widespread adoption of machine learning throughout industry, many firms face a common challenge: relevant datasets are typically distributed amongst market competitors that are reluctant to share information. Recent works propose data markets to provide monetary incentives for collaborative machine learning, where agents share features with each other and are rewarded based on their contribution to improving the predictions others. These contributions are determined by their relative Shapley value, which is computed by treating features as players and their interactions as a characteristic function game. However, in its standard form, this setup further provides an incentive for agents to replicate their data and act under multiple false identities in order to increase their own revenue and diminish that of others, restricting their use in practice. In this work, we develop a replication-robust data market for supervised learning problems. We adopt Pearl's do-calculus from causal reasoning to refine the characteristic function game by differentiating between observational and interventional conditional probabilities. By doing this, we derive Shapley value-based rewards that are robust to this malicious replication by design, whilst preserving desirable market properties., Comment: 16 pages, 5 figures

The paper introduces a new methodology for assessing on-line the prediction risk of short-term wind power forecasts. The first part of this methodology consists in computing confidence intervals with a confidence level defined by the end-user. The resampling approach is used for this purpose since it permits to avoid a restrictive hypothesis on the distribution of the errors. It has been however appropriately adapted for the wind power prediction problem taking into account the dependency of the errors on the level of predicted power through appropriately defined fuzzy sets. The second part of the proposed methodology introduces two indices, named as MRI and PRI, that quantify the meteorological risk by measuring the spread of multi-scenario Numerical Weather Predictions and wind power predictions respectively. The multi-scenario forecasts considered here are based on the 'poor man's' ensembles approach. The two indices are used either to fine-tune the confidence intervals or to give signals to the operator on the prediction risk, i.e. the probabilities for the occurrence of high prediction errors depending on the weather stability. A relation between these indices and the level of prediction error is shown. Evaluation results over a three-year period on the case of a wind farm in Denmark and over a one-year period on the case of several farms in Ireland are given. The proposed methodology has an operational nature and can be applied to all kinds of wind power forecasting models

We consider a new framework where a continuous, though bounded, random variable has unobserved bounds that vary over time. In the context of univariate time series, we look at the bounds as parameters of the distribution of the bounded random variable. We introduce an extended log-likelihood estimation and design algorithms to track the bound through online maximum likelihood estimation. Since the resulting optimization problem is not convex, we make use of recent theoretical results on Normalized Gradient Descent (NGD) for quasiconvex optimization, to eventually derive an Online Normalized Gradient Descent algorithm. We illustrate and discuss the workings of our approach based on both simulation studies and a real-world wind power forecasting problem., Comment: 43 pages, 9 figures

This study proposes a simulation framework of procurement operations in the container logistics industry that can support the development of dynamic procurement strategies. The idea is inspired by the success of Passenger Origin-Destination Simulator (PODS) in the field of airline revenue management. By and large, research in procurement has focused on the optimisation of purchasing decisions, i.e., when-to-order and supplier selection, but a principled approach to procurement operations is lacking. We fill this gap by developing a probabilistic model of a procurement system. A discrete-event simulation logic is used to drive the evolution of the system. In a small case study, we use the simulation to deliver insights by comparing different supplier selection policies in a dynamic spot market environment. Policies based on contextual multi-armed bandits are seen to be robust to limited access to the information that determines the distribution of the outcome. This paper provides a pool of modelling ideas for simulation and observational studies. Moreover, the probabilistic formulation paves the way for advanced machine learning techniques and data-driven optimisation in procurement.

Despite their growing popularity, data-driven models of real-world dynamical systems require lots of data. However, due to sensing limitations as well as privacy concerns, this data is not always available, especially in domains such as energy. Pre-trained models using data gathered in similar contexts have shown enormous potential in addressing these concerns: they can improve predictive accuracy at a much lower observational data expense. Theoretically, due to the risk posed by negative transfer, this improvement is however neither uniform for all agents nor is it guaranteed. In this paper, using data from several distributed energy resources, we investigate and report preliminary findings on several key questions in this regard. First, we evaluate the improvement in predictive accuracy due to pre-trained models, both with and without fine-tuning. Subsequently, we consider the question of fairness: do pre-trained models create equal improvements for heterogeneous agents, and how does this translate to downstream utility? Answering these questions can help enable improvements in the creation, fine-tuning, and adoption of such pre-trained models.

Should the organization, design and functioning of electricity markets be taken for granted? Definitely not. While decades of evolution of electricity markets in countries that committed early to restructure their electric power sector made us believe that we may have found the right and future-proof model, the substantially and rapidly evolving context of our power and energy systems is challenging this idea in many ways. Actually, that situation brings both challenges and opportunities. Challenges include accommodation of renewable energy generation, decentralization and support to investment, while opportunities are mainly that advances in technical and social sciences provide us with many more options in terms of future market design. We here take a holistic point of view, by trying to understand where we are coming from with electricity markets and where we may be going. Future electricity markets should be made fit for purpose by considering them as a way to organize and operate a socio-techno-economic system.

Energy market designs with non-merchant storage have been proposed in recent years, with the aim of achieving optimal market integration of storage. In order to handle the time-linking constraints that are introduced in such markets, existing works commonly make simplifying assumptions about the end-of-horizon storage level, e.g., by imposing an exogenous level for the amount of energy to be left for the next time horizon. This work analyzes market properties under such assumptions, as well as in their absence. We find that, although they ensure cost recovery for all market participants, these assumptions generally lead to market inefficiencies. Therefore we consider the design of markets with non-merchant storage without such simplifying assumptions. Using illustrative examples, as well as detailed proofs, we provide conditions under which market prices in subsequent market horizons fail to reflect the value of stored energy. We show that this problem is essential to address in order to preserve market efficiency and cost recovery. Finally, we propose a method for restoring these market properties in a perfect-foresight setting.

The increasing use of renewable energy sources (RESs) and responsive loads has made power systems more uncertain. Meanwhile, thanks to the development of advanced metering and forecasting technologies, predictions by RESs and load owners are now attainable. Many recent studies have revealed that pooling the predictions from RESs and loads can help the operators predict more accurately and make better dispatch decisions. However, how the prediction purchase decisions are made during the dispatch processes needs further investigation. This paper fills the research gap by proposing a novel robust generation dispatch model considering the purchase and use of predictions from RESs and loads. The prediction purchase decisions are made in the first stage, which influence the accuracy of predictions from RESs and loads, and further the uncertainty set and the worst-case second-stage dispatch performance. This two-stage procedure is essentially a robust optimization problem with decision-dependent uncertainty (DDU). A mapping-based column-and-constraint generation (C&CG) algorithm is developed to overcome the potential failures of traditional solution methods in detecting feasibility, guaranteeing convergence, and reaching optimal strategies under DDU. Case studies demonstrate the effectiveness, necessity, and scalability of the proposed model and algorithm., Comment: 13 pages, 11 figures

Convex optimization finds many real-life applications, where--optimized on real data--optimization results may expose private data attributes (e.g., individual health records, commercial information), thus leading to privacy breaches. To avoid these breaches and formally guarantee privacy to optimization data owners, we develop a new privacy-preserving perturbation strategy for convex optimization programs by combining stochastic (chance-constrained) programming and differential privacy. Unlike standard noise-additive strategies, which perturb either optimization data or optimization results, we express the optimization variables as functions of the random perturbation using linear decision rules; we then optimize these rules to accommodate the perturbation within the problem's feasible region by enforcing chance constraints. This way, the perturbation is feasible and makes different, yet adjacent in the sense of a given distance function, optimization datasets statistically similar in randomized optimization results, thereby enabling probabilistic differential privacy guarantees. The chance-constrained optimization additionally internalizes the conditional value-at-risk measure to model the tolerance towards the worst-case realizations of the optimality loss w.r.t. the non-private solution. We demonstrate the privacy properties of our perturbation strategy analytically and through optimization and machine learning applications.

Renewable energy generation is offered through electricity markets, quite some time in advance. This then leads to a problem of decision-making under uncertainty, which may be seen as a newsvendor problem. Contrarily to the conventional case for which underage and overage penalties are known, such penalties in the case of electricity markets are unknown, and difficult to estimate. In addition, one is actually only penalized for either overage or underage, not both. Consequently, we look at a slightly different form of a newsvendor problem, for a price-taker participant offering in electricity markets, which we refer to as Bernoulli newsvendor problem. After showing that its solution is consistent with that for the classical newsvendor problem, we then introduce distributionally robust versions, with ambiguity possibly about both the probabilistic forecasts for power generation and the chance of success of the Bernoulli variable. Both versions of the distributionally robust Bernoulli newsvendor problem admit closed-form solutions. We finally use simulation studies, as well as a real-world case-study application, to illustrate the workings and benefits from the approach.

We propose and develop a new algorithm for trading wind energy in electricity markets, within an online learning and optimization framework. In particular, we combine a component-wise adaptive variant of the gradient descent algorithm with recent advances in the feature-driven newsvendor model. This results in an online offering approach capable of leveraging data-rich environments, while adapting to the nonstationary characteristics of energy generation and electricity markets, also with a minimal computational burden. The performance of our approach is analyzed based on several numerical experiments, showing both better adaptability to nonstationary uncertain parameters and significant economic gains.

This paper considers a market for trading Internet of Things (IoT) data that is used to train machine learning models. The data, either raw or processed, is supplied to the market platform through a network and the price of such data is controlled based on the value it brings to the machine learning model. We explore the correlation property of data in a game-theoretical setting to eventually derive a simplified distributed solution for a data trading mechanism that emphasizes the mutual benefit of devices and the market. The key proposal is an efficient algorithm for markets that jointly addresses the challenges of availability and heterogeneity in participation, as well as the transfer of trust and the economic value of data exchange in IoT networks. The proposed approach establishes the data market by reinforcing collaboration opportunities between device with correlated data to avoid information leakage. Therein, we develop a network-wide optimization problem that maximizes the social value of coalition among the IoT devices of similar data types; at the same time, it minimizes the cost due to network externalities, i.e., the impact of information leakage due to data correlation, as well as the opportunity costs. Finally, we reveal the structure of the formulated problem as a distributed coalition game and solve it following the simplified split-and-merge algorithm. Simulation results show the efficacy of our proposed mechanism design toward a trusted IoT data market, with up to 32.72% gain in the average payoff for each seller., Comment: 15 pages. 12 figures. This paper has been accepted for publication in IEEE Internet of Things Journal. Copyright may change without notice

We describe an architecture for a decentralised data market for applications in which agents are incentivised to collaborate to crowd-source their data. The architecture is designed to reward data that furthers the market's collective goal, and distributes reward fairly to all those that contribute with their data. We show that the architecture is resilient to Sybil, wormhole, and data poisoning attacks. In order to evaluate the resilience of the architecture, we characterise its breakdown points for various adversarial threat models in an automotive use case., Comment: 13 pages, 7 figures

We present a data-driven approach for probabilistic wind power forecasting based on conditional normalizing flow (CNF). In contrast with the existing, this approach is distribution-free (as for non-parametric and quantile-based approaches) and can directly yield continuous probability densities, hence avoiding quantile crossing. It relies on a base distribution and a set of bijective mappings. Both the shape parameters of the base distribution and the bijective mappings are approximated with neural networks. Spline-based conditional normalizing flow is considered owing to its non-affine characteristics. Over the training phase, the model sequentially maps input examples onto samples of base distribution, given the conditional contexts, where parameters are estimated through maximum likelihood. To issue probabilistic forecasts, one eventually maps samples of the base distribution into samples of a desired distribution. Case studies based on open datasets validate the effectiveness of the proposed model, and allows us to discuss its advantages and caveats with respect to the state of the art., Comment: The second revision to IEEE Transactions on Sustainable Energy

In many areas of industry and society, e.g., energy, healthcare, logistics, agents collect vast amounts of data that they deem proprietary. These data owners extract predictive information of varying quality and relevance from data depending on quantity, inherent information content, and their own technical expertise. Aggregating these data and heterogeneous predictive skills, which are distributed in terms of ownership, can result in a higher collective value for a prediction task. In this paper, we envision a platform for improving predictions via implicit pooling of private information in return for possible remuneration. Specifically, we design a wagering-based forecast elicitation market platform, where a buyer intending to improve their forecasts posts a prediction task, and sellers respond to it with their forecast reports and wagers. This market delivers an aggregated forecast to the buyer (pre-event) and allocates a payoff to the sellers (post-event) for their contribution. We propose a payoff mechanism and prove that it satisfies several desirable economic properties, including those specific to electronic platforms. Furthermore, we discuss the properties of the forecast aggregation operator and scoring rules to emphasize their effect on the sellers' payoff. Finally, we provide numerical examples to illustrate the structure and properties of the proposed market platform.

Wind power forecasting is essential to power system operation and electricity markets. As abundant data became available thanks to the deployment of measurement infrastructures and the democratization of meteorological modelling, extensive data-driven approaches have been developed within both point and probabilistic forecasting frameworks. These models usually assume that the dataset at hand is complete and overlook missing value issues that often occur in practice. In contrast to that common approach, we rigorously consider here the wind power forecasting problem in the presence of missing values, by jointly accommodating imputation and forecasting tasks. Our approach allows inferring the joint distribution of input features and target variables at the model estimation stage based on incomplete observations only. We place emphasis on a fully conditional specification method owing to its desirable properties, e.g., being assumption-free when it comes to these joint distributions. Then, at the operational forecasting stage, with available features at hand, one can issue forecasts by implicitly imputing all missing entries. The approach is applicable to both point and probabilistic forecasting, while yielding competitive forecast quality within both simulation and real-world case studies. It confirms that by using a powerful universal imputation method like fully conditional specification, the proposed approach is superior to the common approach, especially in the context of probabilistic forecasting., Comment: revision to International Journal of Forecasting

Compared with large-scale physical batteries, aggregated and coordinated generic energy storage (GES) resources provide low-cost, but uncertain, flexibility for power grid operations. While GES can be characterized by different types of uncertainty, the literature mostly focuses on decision-independent uncertainties (DIUs), such as exogenous stochastic disturbances caused by weather conditions. Instead, this manuscript focuses on newly-introduced decision-dependent uncertainties (DDUs) and considers an optimal GES dispatch that accounts for uncertain available state-of-charge (SoC) bounds that are affected by incentive signals and discomfort levels. To incorporate DDUs, we present a novel chance-constrained optimization (CCO) approach for the day-ahead economic dispatch of GES units. Two tractable methods are presented to solve the proposed CCO problem with DDUs: (i) a robust reformulation for general but incomplete distributions of DDUs, and (ii) an iterative algorithm for specific and known distributions of DDUs. Furthermore, reliability indices are introduced to verify the applicability of the proposed approach with respect to the reliability of the response of GES units. Simulation-based analysis shows that the proposed methods yield conservative, but credible, GES dispatch strategies and reduced penalty cost by incorporating DDUs in the constraints and leveraging data-driven parameter identification. This results in improved availability and performance of coordinated GES units., Comment: Decision-dependent uncertainty; Generic Energy Storage

Current reserve procurement approaches ignore the stochastic nature of reserve asset availability itself and thus limit the type and volume of reserve offers. This paper develops a reliability-aware probabilistic approach that allows renewable generators to offer reserve capacity with reliability attributes. Offers with low reliability are priced at lower levels. The original non-convex market clearing problem is approximated by a MILP reformulation. The proposed probabilistic reserve procurement allows restricted reserve providers to enter the market, thereby increases liquidity and has the potential to lower procurement costs in power systems with high shares of variable renewable energy sources.

This paper proposes a regression market for wind agents to monetize data traded among themselves for wind power forecasting. Existing literature on data markets often treats data disclosure as a binary choice or modulates the data quality based on the mismatch between the offer and bid prices. As a result, the market disadvantages either the data sellers due to the overestimation of their willingness to disclose data, or the data buyers due to the lack of useful data being provided. Our proposed regression market determines the data payment based on the least absolute shrinkage and selection operator (lasso), which not only provides the data buyer with a means for selecting useful features, but also enables each data seller to individualize the threshold for data payment. Using both synthetic data and real-world wind data, the case studies demonstrate a reduction in the overall losses for wind agents who buy data, as well as additional financial benefits to those who sell data., Comment: Accepted to PSCC 2022. Will be included in a special issue of the journal Electric Power Systems Research (EPSR)

Energy forecasting has attracted enormous attention over the last few decades, with novel proposals related to the use of heterogeneous data sources, probabilistic forecasting, online learn-ing, etc. A key aspect that emerged is that learning and forecasting may highly benefit from distributed data, though not only in the geographical sense. That is, various agents collect and own data that may be useful to others. In contrast to recent proposals that look into distributed and privacy-preserving learning (incentive-free), we explore here a framework called regression markets. There, agents aiming to improve their forecasts post a regression task, for which other agents may contribute by sharing their data for their features and get monetarily rewarded for it.The market design is for regression models that are linear in their parameters, and possibly sep-arable, with estimation performed based on either batch or online learning. Both in-sample and out-of-sample aspects are considered, with markets for fitting models in-sample, and then for improving genuine forecasts out-of-sample. Such regression markets rely on recent concepts within interpretability of machine learning approaches and cooperative game theory, with Shapley additive explanations. Besides introducing the market design and proving its desirable properties, application results are shown based on simulation studies (to highlight the salient features of the proposal) and with real-world case studies.

The disturbances from variable and uncertain renewable generation propagate from power systems to natural gas networks, causing gas network operators to adjust gas supply nominations to ensure operational security. To alleviate expensive supply adjustments, we develop control policies to leverage instead the flexibility of linepack -- the gas stored in pipelines -- to balance stochastic gas extractions. These policies are based on multi-stage linear decision rules optimized on a finite discrete horizon to guide controllable network components, such as compressors and valves, towards feasible operations. Our approach offers several control applications. First, it treats the linepack as a main source of flexibility to balance disturbances from power systems without substantial impacts on nominal gas supply. Second, these policies can be optimized to minimize the variability (due to intermittency of renewables) and variance (due to their uncertainty) of network state variables, such as pressures. Finally, it enables topology optimization to decouple network parts and prevent uncertainty propagation through the network. This is demonstrated using illustrative numerical experiments.

Taking concrete steps towards a carbon-free society, the Danish Parliament has recently made an agreement on the establishment of the world's first two offshore energy hubs, one on the island of Bornholm and one on an artificial island in the North Sea. Being the two first-of-their-kind projects, several aspects related to the inclusion of these "energy islands" in the current market setup are still under discussion. To this end, this paper presents the first large-scale impact analysis of offshore hubs on the whole European power system and electricity market. The detailed models used for such analysis are publicly released with the paper. Our study shows that energy hubs in the North Sea have a positive impact, and overall increase economic welfare in EU. However, when considering the impact on each country, benefits are not shared equally. In order to help the development of such projects, we focus on the identification of market challenges and system needs arising from the hubs. From a market perspective, we show how exporting countries are negatively affected by the lower electricity prices and we point at potential strategic behaviors induced by the large amount of new transmission capacity installed in the North Sea. From a system point of view, we show how the large amount of wind energy stresses conventional generators, which are required to become more flexible, and national grids, which cannot always accommodate large imports from the hubs., Comment: Submitted to Energy Policy on March 31, 2021 - Revised on February 7, 2022

We propose a new forward electricity market framework that admits heterogeneous market participants with second-order cone strategy sets, who accurately express the nonlinearities in their costs and constraints through conic bids, and a network operator facing conic operational constraints. In contrast to the prevalent linear-programming-based electricity markets, we highlight how the inclusion of second-order cone constraints improves uncertainty-, asset- and network-awareness of the market, which is key to the successful transition towards an electricity system based on weather-dependent renewable energy sources. We analyze our general market-clearing proposal using conic duality theory to derive efficient spatially-differentiated prices for the multiple commodities, comprising of energy and flexibility services. Under the assumption of perfect competition, we prove the equivalence of the centrally-solved market-clearing optimization problem to a competitive spatial price equilibrium involving a set of rational and self-interested participants and a price setter. Finally, under common assumptions, we prove that moving towards conic markets does not incur the loss of desirable economic properties of markets, namely market efficiency, cost recovery and revenue adequacy. Our numerical studies focus on the specific use case of uncertainty-aware market design and demonstrate that the proposed conic market brings advantages over existing alternatives within the linear programming market framework., Comment: Final accepted manuscript with supplementary material (revised 15.12.2022)

There is increasing interest in very short-term and higher-resolution wind power forecasting (from minutes to hours ahead), especially offshore. Statistical methods are of utmost relevance, since weather forecasts cannot be informative for those lead times. Those approaches ought to account for the fact that wind power generation as a stochastic process is nonstationary, double-bounded (by zero and the nominal power of the turbine) and non-linear. Accommodating those aspects may lead to improving both point and probabilistic forecasts. We propose here to focus on generalized logit-normal distributions, which are naturally suitable and flexible for double-bounded and non-linear processes. Relevant parameters are estimated via maximum likelihood inference. Both batch and online versions of the estimation approach are described -- the online version permitting to additionally handle non-stationarity through the variation of distribution parameters. The approach is applied and analysed on the test case of the Anholt offshore wind farm in Denmark, with emphasis placed on 10-min-ahead point and probabilistic forecasts., Comment: 6 pages, 6 figures, submitted to the 14th IEEE PowerTech 2021 Conference

When energy customers schedule loads ahead of time, this information, if acquired by their energy retailer, can improve the retailer's load forecasts. Better forecasts lead to wholesale purchase decisions that are likely to result in lower energy imbalance costs, and thus higher profits for the retailer. Therefore, this paper monetizes the value of the customer schedulable load data by quantifying the retailer's profit gain from adjusting the wholesale purchase based on such data. Using a cooperative game theoretic approach, the retailer translates their increased profit in expectation into the value of cooperation, and redistributes a portion of it among the customers as monetary incentives for them to continue providing their load data. Through case studies, this paper demonstrates the significance of the additional profit for the retailer from using the proposed framework, and evaluates the long-term monetary benefits to the customers based on different payoff allocation methods., Comment: 6 pages, 6 figures, accepted to and presented at the 14th IEEE PowerTech 2021 Conference

Forecasting has always been at the forefront of decision making and planning. The uncertainty that surrounds the future is both exciting and challenging, with individuals and organisations seeking to minimise risks and maximise utilities. The large number of forecasting applications calls for a diverse set of forecasting methods to tackle real-life challenges. This article provides a non-systematic review of the theory and the practice of forecasting. We provide an overview of a wide range of theoretical, state-of-the-art models, methods, principles, and approaches to prepare, produce, organise, and evaluate forecasts. We then demonstrate how such theoretical concepts are applied in a variety of real-life contexts. We do not claim that this review is an exhaustive list of methods and applications. However, we wish that our encyclopedic presentation will offer a point of reference for the rich work that has been undertaken over the last decades, with some key insights for the future of forecasting theory and practice. Given its encyclopedic nature, the intended mode of reading is non-linear. We offer cross-references to allow the readers to navigate through the various topics. We complement the theoretical concepts and applications covered by large lists of free or open-source software implementations and publicly-available databases.

We propose stochastic control policies to cope with uncertain and variable gas extractions in natural gas networks. Given historical gas extraction data, these policies are optimized to produce the real-time control inputs for nodal gas injections and for pressure regulation rates by compressors and valves. We describe the random network state as a function of control inputs, which enables a chance-constrained optimization of these policies for arbitrary network topologies. This optimization ensures the real-time gas flow feasibility and a minimal variation in the network state up to specified feasibility and variance criteria. Furthermore, the chance-constrained optimization provides the foundation of a stochastic pricing scheme for natural gas networks, which improves on a deterministic market settlement by offering the compensations to network assets for their contribution to uncertainty and variance control. We analyze the economic properties, including efficiency, revenue adequacy and cost recovery, of the proposed pricing scheme and make them conditioned on the network design., Comment: for associated GitHub repository, see https://github.com/anubhavratha/ng_stochastic_control_and_pricing

This paper develops a novel differentially private framework to solve convex optimization problems with sensitive optimization data and complex physical or operational constraints. Unlike standard noise-additive algorithms, that act primarily on the problem data, objective or solution, and disregard the problem constraints, this framework requires the optimization variables to be a function of the noise and exploits a chance-constrained problem reformulation with formal feasibility guarantees. The noise is calibrated to provide differential privacy for identity and linear queries on the optimization solution. For many applications, including resource allocation problems, the proposed framework provides a trade-off between the expected optimality loss and the variance of optimization results.

We develop a stochastic equilibrium model for an electricity market with asymmetric renewable energy forecasts. In our setting, market participants optimize their profits using public information about a conditional expectation of energy production but use private information about the forecast error distribution. This information is given in the form of samples and incorporated into profit-maximizing optimizations of market participants through chance constraints. We model information asymmetry by varying the sample size of participants' private information. We show that with more information available, the equilibrium gradually converges to the ideal solution provided by the perfect information scenario. Under information scarcity, however, we show that the market converges to the ideal equilibrium if participants are to infer the forecast error distribution from the statistical properties of the data at hand or share their private forecasts.

The large shares of wind power generation in electricity markets motivate higher levels of operating reserves. However, current reserve sizing practices fail to account for important topological aspects that might hinder their deployment, thus resulting in high operating costs. Zonal reserve procurement mitigates such inefficiencies, however, the way the zones are defined is still open to interpretation. This paper challenges the efficiency of predetermined zonal setups that neglect the location of stochastic power production in the system, as well as the availability, cost and accessibility of flexible generating units. To this end, we propose a novel reserve procurement approach, formulated as a two-stage stochastic bilevel model, in which the upper level identifies a number of contiguous reserve zones using dynamic grid partitioning and sets zonal requirements based on the total expected operating costs. Using two standard IEEE reliability test cases, we show how the efficient partitioning of reserve zones can reduce expected system cost and promote the integration of stochastic renewables., Comment: Submitted to IEEE Transactions on Power Systems on the 20th of March 2020

Cooperation between different data owners may lead to an improvement in forecast quality - for instance by benefiting from spatial-temporal dependencies in geographically distributed time series. Due to business competitive factors and personal data protection questions, said data owners might be unwilling to share their data, which increases the interest in collaborative privacy-preserving forecasting. This paper analyses the state-of-the-art and unveils several shortcomings of existing methods in guaranteeing data privacy when employing Vector Autoregressive (VAR) models. The paper also provides mathematical proofs and numerical analysis to evaluate existing privacy-preserving methods, dividing them into three groups: data transformation, secure multi-party computations, and decomposition methods. The analysis shows that state-of-the-art techniques have limitations in preserving data privacy, such as a trade-off between privacy and forecasting accuracy, while the original data in iterative model fitting processes, in which intermediate results are shared, can be inferred after some iterations.

Although distribution grid customers are obliged to share their consumption data with distribution system operators (DSOs), a possible leakage of this data is often disregarded in operational routines of DSOs. This paper introduces a privacy-preserving optimal power flow (OPF) mechanism for distribution grids that secures customer privacy from unauthorised access to OPF solutions, e.g., current and voltage measurements. The mechanism is based on the framework of differential privacy that allows to control the participation risks of individuals in a dataset by applying a carefully calibrated noise to the output of a computation. Unlike existing private mechanisms, this mechanism does not apply the noise to the optimization parameters or its result. Instead, it optimizes OPF variables as affine functions of the random noise, which weakens the correlation between the grid loads and OPF variables. To ensure feasibility of the randomized OPF solution, the mechanism makes use of chance constraints enforced on the grid limits. The mechanism is further extended to control the optimality loss induced by the random noise, as well as the variance of OPF variables. The paper shows that the differentially private OPF solution does not leak customer loads up to specified parameters.

Large deployment of distribute energy resources and the increasing awareness of end-users towards their energy procurement are challenging current practices of electricity markets. A change of paradigm, from a top-down hierarchical approach to a more decentralized framework, has been recently researched, with market structures relying on multi-bilateral trades among market participants. In order to guarantee feasibility in power system operation, it is crucial to rethink the interaction with system operators and the way operational costs are shared in such decentralized markets. We propose here to include system operators, both at transmission and distribution level, as active actors of the market, accounting for power grid constraints and line losses. Moreover, to avoid market outcomes that discriminate agents for their geographical location, we analyze loss allocation policies and their impact on market outcomes and prices., Comment: Submitted to "IEEE Transactions on Power Systems" on January 15, 2020 - Revised on May 6, 2020 and on August 6, 2020 - Accepted on September 13, 2020