Search

Direct policy search (DPS) is emerging as one of the most effective and widely applied reinforcement learning (RL) methods to design optimal control policies for multiobjective Markov decision processes (MOMDPs). Traditionally, DPS defines the control policy within a preselected functional class and searches its optimal parameterization with respect to a given set of objectives. The functional class should be tailored to the problem at hand and its selection is crucial, as it determines the search space within which solutions can be found. In MOMDPs problems, a different objective tradeoff determines a different fitness landscape, requiring a tradeoff-dynamic functional class selection. Yet, in state-of-the-art applications, the policy class is generally selected a priori and kept constant across the multidimensional objective space. In this work, we present a novel policy search routine called neuro-evolutionary multiobjective DPS (NEMODPS), which extends the DPS problem formulation to conjunctively search the policy functional class and its parameterization in a hyperspace containing policy architectures and coefficients. NEMODPS begins with a population of minimally structured approximating networks and progressively builds more sophisticated architectures by topological and parametrical mutation and crossover, and selection of the fittest individuals concerning multiple objectives. We tested NEMODPS for the problem of designing the control policy of a multipurpose water system. Numerical results show that the tradeoff-dynamic structural and parametrical policy search of NEMODPS is consistent across multiple runs, and outperforms the solutions designed via traditional DPS with predefined policy topologies.

Uncertainty arising from climate change poses a central challenge to the long‐term performance of many engineered water systems. Water supply infrastructure projects can leverage different types of flexibility, in planning, design, or operations, to adapt infrastructure systems in response to climate change over time. Both flexible planning and design enable future capacity expansion if‐and‐when needed, with flexible design proactively incorporating physical design changes that enable retrofits. All three forms of flexibility have not previously been analyzed together to explicitly assess their relative value in mitigating cost and water supply reliability risk. In this paper, we propose a new framework to evaluate combinations of flexible planning, design, and operations. We develop a nested stochastic dynamic optimization approach that jointly optimizes dam development and operating policies under dynamic climate uncertainty. We demonstrate this approach on a reservoir project near Mombasa, Kenya. Our results find that flexible operations have the greatest potential to reduce costs. Flexible design and flexible planning can amplify the value of flexible operations under higher discounting scenarios and when initial infrastructure capacities are undersized. This approach provides insight on the climate change and techno‐economic conditions under which flexible planning, design, and operations can be best leveraged individually or in combination to reduce climate change uncertainty risks in water supply infrastructure projects. We develop a framework to compare the value of combinations of flexibility in water supply infrastructure planning, design, and operationsFlexible operations are more effective than flexible planning or design in addressing climate uncertainty in water‐limited environmentsFlexible planning and design are more useful when initial reservoir capacity is low, discounting is high, or the cost of shortages is high We develop a framework to compare the value of combinations of flexibility in water supply infrastructure planning, design, and operations Flexible operations are more effective than flexible planning or design in addressing climate uncertainty in water‐limited environments Flexible planning and design are more useful when initial reservoir capacity is low, discounting is high, or the cost of shortages is high

This study presents a novel approach to combine a data-driven control strategy with an emulator model of the climate system in order to make the optimal control of water systems more flexible and adaptive to the increasing frequency and intensity of extreme events. These latter are often associated with global climate anomalies, which are difficult to model and incorporate into optimal control algorithms. In this paper, we compare a traditional control policy conditioned only on the reservoir storage with an informed controller that enlarges the state space to include the emulated dynamics of global Sea Surface Temperature anomalies. The multi-purpose operations of Lake Como in Italy, accounting for flood control and water supply, is used as a case study. Numerical results show that the proposed approach provides a 59% improvement in system performance with respect to traditional solutions. This gain further increases during extreme drought episodes, which are influenced by global climate oscillations.

Increasingly variable hydrologic regimes combined with more frequent and intense extreme events are challenging water systems management worldwide. These trends emphasize the need of accurate medium‐ to long‐term predictions to timely prompt anticipatory operations. Despite in some locations global climate oscillations and particularly the El Niño Southern Oscillation (ENSO) may contribute to extending forecast lead times, in other regions there is no consensus on how ENSO can be detected, and used as local conditions are also influenced by other concurrent climate signals. In this work, we introduce the Climate State Intelligence framework to capture the state of multiple global climate signals via artificial intelligence and improve seasonal forecasts. These forecasts are used as additional inputs for informing water system operations and their value is quantified as the corresponding gain in system performance. We apply the framework to the Lake Como basin, a regulated lake in northern Italy mainly operated for flood control and irrigation supply. Numerical results show the existence of notable teleconnection patterns dependent on both ENSO and the North Atlantic Oscillation over the Alpine region, which contribute in generating skilful seasonal precipitation and hydrologic forecasts. The use of this information for conditioning the lake operations produces an average 44% improvement in system performance with respect to a baseline solution not informed by any forecast, with this gain that further increases during extreme drought episodes. Our results also suggest that observed preseason sea surface temperature anomalies appear more valuable than hydrologic‐based seasonal forecasts, producing an average 59% improvement in system performance. We introduce the Climate State Intelligence framework to capture the state of multiple climate signals and improve seasonal forecastStrong influence of divergent SST patterns dependent on ENSO and NAO phases is detected over the Alpine regionPreseason SST anomalies appear more valuable than hydrologic‐based seasonal forecasts for informing reservoir operations

In highly regulated water systems droughts are complex, basin-specific phenomena. The identification of drought drivers is challenged by the coexistence of possibly relevant processes with inconsistent dynamics and origins (natural or anthropic). FRIDA is a fully automated data-driven approach developed to extract relevant drought drivers from a pool of candidate hydro-meteorological predictors at different time aggregations. Selected predictors are then combined into a basin-specific drought index to monitor the state of water resources in highly regulated contexts. The operational value of this index in improving water systems operations is quantified by designing a control policy informed by the index, and contrasting its performance with that of a baseline policy conditioned on basic information only. The approach is demonstrated on Lake Como, Italy, a multipurpose regulated lake operated for flood control and irrigation supply. Results show that the designed index is accurate in representing basin drought conditions, and the overall system performance can improve by nearly 20% when operations are informed with the basin-tailored drought index. The proposed framework is portable across different contexts, where basin-specific drought indexes can support drought characterization and control in a fully data-driven fashion.

Most water reservoir operators make use of forecasts to inform their decisions and enhance water systems flexibility and resilience by anticipating hydrological extremes. Yet, despite numerous candidate hydro‐meteorological variables and forecast horizons may potentially be beneficial to operations, the best information set for a given problem is often not evident. Additionally, in multipurpose systems characterized by multiple demands with varying vulnerabilities and temporal scales, this information set might change according to the objective tradeoff. In this work, we contribute a novel method to learn the optimal policy representation (i.e., policy input set) by combining a feature selection routine with a multiobjective Direct Policy Search framework in order to retrieve the best policy input set online (i.e., while learning the policy) and dynamically with the objective trade‐off. The selected policy search routine is the Neuro‐Evolutionary Multi‐Objective Direct Policy Search (NEMODPS) which generates flexible policy shapes adaptive to online changes in the input set. This approach is demonstrated on the case study of Lake Como (Italy), where the operating objectives are highly heterogeneous in their dynamics (fast and slow) and vulnerabilities (wet and dry extremes). We show how varying objectives, and tradeoffs therein, benefit from a different policy representation, ultimately yielding remarkable results in terms of conflict mitigation between different users. More informed policies, moreover, show higher robustness when re‐evaluated across a suite of different hydrological conditions. We introduce a novel method to define an optimal input set for a multipurpose dam operating policy that varies with the objective trade‐offBetter informed policies are able to mitigate conflicts between water users and achieve system‐wide benefitsThe addition of information in policy design increases the policies robustness toward extreme hydrological conditions We introduce a novel method to define an optimal input set for a multipurpose dam operating policy that varies with the objective trade‐off Better informed policies are able to mitigate conflicts between water users and achieve system‐wide benefits The addition of information in policy design increases the policies robustness toward extreme hydrological conditions