Search

Time series analysis by state-space models is widely used in forecasting and extracting unobservable components like level, slope, and seasonality, along with explanatory variables. However, their reliance on traditional Kalman filtering frequently hampers their effectiveness, primarily due to Gaussian assumptions and the absence of efficient subset selection methods to accommodate the multitude of potential explanatory variables in today's big-data applications. Our research introduces the State Space Learning (SSL), a novel framework and paradigm that leverages the capabilities of statistical learning to construct a comprehensive framework for time series modeling and forecasting. By utilizing a regularized high-dimensional regression framework, our approach jointly extracts typical time series unobservable components, detects and addresses outliers, and selects the influence of exogenous variables within a high-dimensional space in polynomial time and global optimality guarantees. Through a controlled numerical experiment, we demonstrate the superiority of our approach in terms of subset selection of explanatory variables accuracy compared to relevant benchmarks. We also present an intuitive forecasting scheme and showcase superior performances relative to traditional time series models using a dataset of 48,000 monthly time series from the M4 competition. We extend the applicability of our approach to reformulate any linear state space formulation featuring time-varying coefficients into high-dimensional regularized regressions, expanding the impact of our research to other engineering applications beyond time series analysis. Finally, our proposed methodology is implemented within the Julia open-source package, ``StateSpaceLearning.jl"., Comment: 22 pages, 28 figures

Sequential Decision Making under Uncertainty (SDMU) is ubiquitous in many domains such as energy, finance, and supply chains. Some SDMU applications are naturally modeled as Multistage Stochastic Optimization Problems (MSPs), but the resulting optimizations are notoriously challenging from a computational standpoint. Under assumptions of convexity and stage-wise independence of the uncertainty, the resulting optimization can be solved efficiently using Stochastic Dual Dynamic Programming (SDDP). Two-stage Linear Decision Rules (TS-LDRs) have been proposed to solve MSPs without the stage-wise independence assumption. TS-LDRs are computationally tractable, but using a policy that is a linear function of past observations is typically not suitable for non-convex environments arising, for example, in energy systems. This paper introduces a novel approach, Two-Stage General Decision Rules (TS-GDR), to generalize the policy space beyond linear functions, making them suitable for non-convex environments. TS-GDR is a self-supervised learning algorithm that trains the nonlinear decision rules using stochastic gradient descent (SGD); its forward passes solve the policy implementation optimization problems, and the backward passes leverage duality theory to obtain closed-form gradients. The effectiveness of TS-GDR is demonstrated through an instantiation using Deep Recurrent Neural Networks named Two-Stage Deep Decision Rules (TS-DDR). The method inherits the flexibility and computational performance of Deep Learning methodologies to solve SDMU problems generally tackled through large-scale optimization techniques. Applied to the Long-Term Hydrothermal Dispatch (LTHD) problem using actual power system data from Bolivia, the TS-DDR not only enhances solution quality but also significantly reduces computation times by several orders of magnitude., Comment: 10 Main Pages, 6 Appendices, 6 Figures

Simulating long-term hydrothermal bid-based markets considering strategic agents is a challenging task. The representation of strategic agents considering inter-temporal constraints within a stochastic framework brings additional complexity to the already difficult single-period bilevel, thus, non-convex, optimal bidding problem. Thus, we propose a simulation methodology that effectively addresses these challenges for large-scale hydrothermal power systems. We demonstrate the effectiveness of the framework through a case study with real data from the large-scale Brazilian power system. In the case studies, we show the effects of market concentration in power systems and how contracts can be used to mitigate them. In particular, we show how market power might affect the current setting in Brazil. The developed method can strongly benefit policy makers, market monitors, and market designers as simulations can be used to understand existing power systems and experiment with alternative designs.

Wildfires can severely damage electricity grids leading to long periods of power interruption. Climate change will exacerbate this threat by increasing the frequency of dry climate conditions. Under these climate conditions, human-related actions that initiate wildfires should be avoided, including those induced by power systems operation. In this paper, we propose a novel optimization model that is capable of determining appropriate network topology changes (via switching actions) to alleviate the levels of power flows through vulnerable parts of the grid so as to decrease the probability of wildfire ignition. Within this framework, the proposed model captures the relationship between failure probabilities and line-flow decisions by explicitly considering the former as a function of the latter. The resulting formulation is a two-stage model with endogenous decision-dependent probabilities, where the first stage determines the optimal switching actions and the second stage evaluates the worst-case expected operation cost. We propose an exact iterative method to deal with this intricate problem and the methodology is illustrated with a 54-bus and a 138-bus distribution system.

In this technical report, we compare multiple reformulation techniques and solvers that can be used with the Julia package BilevelJuMP. We focus on the special case of Hyperparameter Tuning for Support Vector Regression. We describe a bilevel model for the problem in question. Then we present code for generating data and models that solve the problem. Finally, we present results and a brief analysis., Comment: arXiv admin note: substantial text overlap with arXiv:2205.02307

In this paper we present BilevelJuMP, a new Julia package to support bilevel optimization within the JuMP framework. The package is a Julia library that enables the user to describe both upper and lower-level optimization problems using the JuMP algebraic syntax. Due to the generality and flexibility our library inherits from JuMP's syntax, our package allows users to model bilevel optimization problems with conic constraints in the lower level and all JuMP supported constraints in the upper level (Conic, Quadratic, Non-Linear, Integer, etc.). Moreover, the user-defined problem can be subsequently solved by various techniques relying on mathematical program with equilibrium constraints (MPEC) reformulations. Manipulations on the original problem data are possible due to MathOptInterface.jl's structures and Dualization.jl features. Hence, the proposed package allows quickly model, deploy, and thereby experiment bilevel models based on off-the-shelf mixed integer linear programming and nonlinear solvers.

The solution of multistage stochastic linear problems (MSLP) represents a challenge for many application areas. Long-term hydrothermal dispatch planning (LHDP) materializes this challenge in a real-world problem that affects electricity markets, economies, and natural resources worldwide. No closed-form solutions are available for MSLP and the definition of non-anticipative policies with high-quality out-of-sample performance is crucial. Linear decision rules (LDR) provide an interesting simulation-based framework for finding high-quality policies for MSLP through two-stage stochastic models. In practical applications, however, the number of parameters to be estimated when using an LDR may be close to or higher than the number of scenarios of the sample average approximation problem, thereby generating an in-sample overfit and poor performances in out-of-sample simulations. In this paper, we propose a novel regularized LDR to solve MSLP based on the AdaLASSO (adaptive least absolute shrinkage and selection operator). The goal is to use the parsimony principle, as largely studied in high-dimensional linear regression models, to obtain better out-of-sample performance for LDR applied to MSLP. Computational experiments show that the overfit threat is non-negligible when using classical non-regularized LDR to solve the LHDP, one of the most studied MSLP with relevant applications. Our analysis highlights the following benefits of the proposed framework in comparison to the non-regularized benchmark: 1) significant reductions in the number of non-zero coefficients (model parsimony), 2) substantial cost reductions in out-of-sample evaluations, and 3) improved spot-price profiles., Comment: European Journal of Operational Research, 2022. See the published version at (EJOR) through the DOI link https://doi.org/10.1016/j.ejor.2022.12.039

This paper exploits the decomposition structure of the large-scale hydrothermal generation expansion planning problem with an integrated modified Benders Decomposition and Progressive Hedging approach. We consider detailed and realistic data from the Brazilian power system to represent hourly chronological constraints based on typical days per month and year. Also, we represent the multistage stochastic nature of the optimal hydrothermal operational policy through co-optimized linear decision rules for individual reservoirs. Therefore, we ensure investment decisions compatible with a nonanticipative (implementable) operational policy. To solve the large-scale optimization problem, we propose an improved Benders Decomposition method with multiple instances of the master problem, each of which strengthened by primal cuts and new Benders cuts generated by each master's trial solution. Additionally, our new approach allows using Progressive Hedging penalization terms for accelerating the convergence of the method. We show that our method is 60\% faster than the benchmark. Finally, the consideration of a nonanticipative operational policy can save 7.64\% of the total cost (16.18\% of the investment costs) and significantly improve spot price profiles.

The sustainable utilization of hydro energy relies on accurate estimates of the opportunity cost of the water. This value is calculated through long-term hydrothermal dispatch problems (LTHDP), and the recent literature has raised awareness about the consequences of modeling simplifications in these problems. The inaccurate representation of Kirchhoff's voltage law under the premise of a DC power flow is an example. Under a non-linear AC model, however, the LTHDP becomes intractable, and the literature lacks an accurate evaluation method of different modeling alternatives. In this paper, we extend the state-of-the-art cost-assessment framework of network approximations for LTHDP and bring relevant and practical new insights. First, we increase the quality of the assessment by using an AC power flow to simulate and compare the performance of five policies based on different network approximations. Second, we find that the tightest network relaxation (based on semidefinite programming) is not the one exhibiting the best performance. Results show that the DC {power flow} with quadratic losses approximation exhibits the lowest expected cost and inconsistency gaps. Finally, its computational burden is lower than that exhibited by the semidefinite relaxation, whereas market distortions are significantly reduced in comparison to previously published benchmarks based on DC power flow.

Forecasting and decision-making are generally modeled as two sequential steps with no feedback, following an open-loop approach. In this paper, we present application-driven learning, a new closed-loop framework in which the processes of forecasting and decision-making are merged and co-optimized through a bilevel optimization problem. We present our methodology in a general format and prove that the solution converges to the best estimator in terms of the expected cost of the selected application. Then, we propose two solution methods: an exact method based on the KKT conditions of the second-level problem and a scalable heuristic approach suitable for decomposition methods. The proposed methodology is applied to the relevant problem of defining dynamic reserve requirements and conditional load forecasts, offering an alternative approach to current ad hoc procedures implemented in industry practices. We benchmark our methodology with the standard sequential least-squares forecast and dispatch planning process. We apply the proposed methodology to an illustrative system and to a wide range of instances, from dozens of buses to large-scale realistic systems with thousands of buses. Our results show that the proposed methodology is scalable and yields consistently better performance than the standard open-loop approach.

Score-driven models, also known as generalized autoregressive score models, represent a class of observation-driven time series models. They possess powerful properties, such as the ability to model different conditional distributions and to consider time-varying parameters within a flexible framework. In this paper, we present ScoreDrivenModels.jl, an open-source Julia package for modeling, forecasting, and simulating time series using the framework of score-driven models. The package is flexible with respect to model definition, allowing the user to specify the lag structure and which parameters are time-varying or constant. It is also possible to consider several distributions, including Beta, Exponential, Gamma, Lognormal, Normal, Poisson, Student's t, and Weibull. The provided interface is flexible, allowing interested users to implement any desired distribution and parametrization.

The number of Covid-19 cases is increasing dramatically worldwide. Therefore, the availability of reliable forecasts for the number of cases in the coming days is of fundamental importance. We propose a simple statistical method for short-term real-time forecasting of the number of Covid-19 cases and fatalities in countries that are latecomers -- i.e., countries where cases of the disease started to appear some time after others. In particular, we propose a penalized (LASSO) regression with an error correction mechanism to construct a model of a latecomer in terms of the other countries that were at a similar stage of the pandemic some days before. By tracking the number of cases and deaths in those countries, we forecast through an adaptive rolling-window scheme the number of cases and deaths in the latecomer. We apply this methodology to Brazil, and show that (so far) it has been performing very well. These forecasts aim to foster a better short-run management of the health system capacity.

Modern decision-making processes require uncertainty-aware models, especially those relying on non-symmetric costs and risk-averse profiles. The objective of this work is to propose a dynamic model for the conditional non-parametric distribution function (CDF) to generate probabilistic forecasts for a renewable generation time series. To do that, we propose an adaptive non-parametric time-series model driven by a regularized multiple-quantile-regression (MQR) framework. In our approach, all regression models are jointly estimated through a single linear optimization problem that finds the global-optimal parameters in polynomial time. An innovative feature of our work is the consideration of a Lipschitz regularization of the first derivative of coefficients in the quantile space, which imposes coefficient smoothness. The proposed regularization induces a coupling effect among quantiles creating a single non-parametric CDF model with improved out-of-sample performance. A case study with realistic wind-power generation data from the Brazilian system shows: 1) the regularization model is capable to improve the performance of MQR probabilistic forecasts, and 2) our MQR model outperforms five relevant benchmarks: two based on the MQR framework, and three based on parametric models, namely, SARIMA, and GAS with Beta and Weibull CDF., Comment: 17 pages, 7 figures

We present a distributionally robust optimization (DRO) approach for the transmission expansion planning problem, considering both long- and short-term uncertainties on the system demand and non-dispatchable renewable generation. On the long-term level, as is customary in industry applications, we address the deep uncertainties arising from social and economic transformations, political and environmental issues, and technology disruptions by using long-term scenarios devised by experts. In this setting, many exogenous long-term scenarios containing partial information about the random parameters, namely, the average and the support set, can be considered. For each long-term scenario, a conditional ambiguity set models the incomplete knowledge about the probability distribution of the uncertain parameters in the short-term operation. Consequently, the mathematical problem is formulated as a DRO model with multiple conditional ambiguity sets. The resulting infinite-dimensional problem is recast as an exact, although very large, finite mixed-integer linear programming problem. To circumvent scalability issues, we propose a new enhanced-column-and-constraint-generation (ECCG) decomposition approach with an additional Dantzig--Wolfe procedure. In comparison to existing methods, ECCG leads to a better representation of the recourse function and, consequently, tighter bounds. Numerical experiments based on the benchmark IEEE 118-bus system are reported to corroborate the effectiveness of the method.

Two-stage robust unit commitment (RUC) models have been widely used for day-ahead energy and reserve scheduling under high renewable integration. The current state of the art relies on budget-constrained polyhedral uncertainty sets to control the conservativeness of the solutions. The associated lack of interpretability and parameter specification procedures, as well as the high computational burden exhibited by available exact solution techniques call for new approaches. In this work, we use an alternative scenario-based framework whereby uncertain renewable generation is characterized by a polyhedral uncertainty set relying on the direct specification of its vertexes. Moreover, we present a simple, yet efficient, adaptive data-driven procedure to dynamically update the uncertainty set vertexes with observed daily renewable-output profiles. Within this setting, the proposed data-driven RUC ensures protection against the convex hull of realistic scenarios empirically capturing the complex and time-varying intra-day spatial and temporal interdependences among renewable units. The resulting counterpart features advantageous properties from a computational perspective and can be effectively solved by the column-and-constraint generation algorithm until $\epsilon$-global optimality. Out-of-sample experiments reveal that the proposed approach is capable of attaining efficient solutions in terms of cost and robustness while keeping the model tractable and scalable.