Search

With the continuous growth of the global economy and markets, resource imbalance has risen to be one of the central issues in real logistic scenarios. In marine transportation, this trade imbalance leads to Empty Container Repositioning (ECR) problems. Once the freight has been delivered from an exporting country to an importing one, the laden will turn into empty containers that need to be repositioned to satisfy new goods requests in exporting countries. In such problems, the performance that any cooperative repositioning policy can achieve strictly depends on the routes that vessels will follow (i.e., fleet deployment). Historically, Operation Research (OR) approaches were proposed to jointly optimize the repositioning policy along with the fleet of vessels. However, the stochasticity of future supply and demand of containers, together with black-box and non-linear constraints that are present within the environment, make these approaches unsuitable for these scenarios. In this paper, we introduce a novel framework, Configurable Semi-POMDPs, to model this type of problems. Furthermore, we provide a two-stage learning algorithm, “Configure & Conquer” (CC), that first configures the environment by finding an approximation of the optimal fleet deployment strategy, and then “conquers” it by learning an ECR policy in this tuned environmental setting. We validate our approach in large and real-world instances of the problem. Our experiments highlight that CC avoids the pitfalls of OR methods and that it is successful at optimizing both the ECR policy and the fleet of vessels, leading to superior performance in world trade environments.

Distribution drift is an important issue for practical applications of machine learning (ML). In particular, in streaming ML, the data distribution may change over time, yielding the problem of concept drift, which affects the performance of learners trained with outdated data. In this article, we focus on supervised problems in an online nonstationary setting, introducing a novel learner-agnostic algorithm for drift adaptation, namely importance weighting for drift adaptation (IWDA), with the goal of performing efficient retraining of the learner when drift is detected. IWDA incrementally estimates the joint probability density of input and target for the incoming data and, as soon as drift is detected, retrains the learner using importance-weighted empirical risk minimization. The importance weights are computed for all the samples observed so far, employing the estimated densities, thus, using all available information efficiently. After presenting our approach, we provide a theoretical analysis in the abrupt drift setting. Finally, we present numerical simulations that illustrate how IWDA competes and often outperforms state-of-the-art stream learning techniques, including adaptive ensemble methods, on both synthetic and real-world data benchmarks.

Reinforcement Learning (RL) is an effective approach to solve sequential decision making problems when the environment is equipped with a reward function to evaluate the agent’s actions. However, there are several domains in which a reward function is not available and difficult to estimate. When samples of expert agents are available, Inverse Reinforcement Learning (IRL) allows recovering a reward function that explains the demonstrated behavior. Most of the classic IRL methods, in addition to expert’s demonstrations, require sampling the environment to evaluate each reward function, that, in turn, is built starting from a set of engineered features. This paper is about a novel model-free IRL approach that does not require to specify a function space where to search for the expert’s reward function. Leveraging on the fact that the policy gradient needs to be zero for an optimal policy, the algorithm generates an approximation space for the reward function, in which a reward is singled out employing a second-order criterion. After introducing our approach for finite domains, we extend it to continuous ones. The empirical results, on both finite and continuous domains, show that the reward function recovered by our algorithm allows learning policies that outperform those obtained with the true reward function, in terms of learning speed.

Smart Grids are the evolution of traditional electric grids and allow two-way flows of electricity and information between different actors. At the edge of this network, customers can both produce and consume energy. Due to the intermittent nature of renewable energy sources, customers are characterized by moments of energy surplus and deficit. To solve this problem, customers are connected to the power grid, and, usually, they are also provided with Lithium-Ion battery packs positioned near the energy source used to store energy in excess for later use, reducing expensive energy exchanges with the grid. On the one hand, using the battery at its full capabilities produces significant economic savings, and, on the other hand, massive use of the battery leads to degradation and consequently to a more frequent substitution of the battery. Therefore, depending on the cost of energy and batteries, one should carefully choose when it is favorable to use it. To avoid inefficiencies, it is common to design controllers that regulate the energy flow within the battery packs, deciding whether to exchange energy with the network or store it in the battery. In this work, a Reinforcement Learning controller optimizing energy flow is developed. The controller’s goal is to balance the costs of exchanges with the power grid and those derived from the degradation due to battery usage. A synthetic experimental campaign conducted using real-world data demonstrates that the policy learned shows an improvement in the worst-caseof 3% w.r.t. state-of-the-art baselines.

The adoption of Nash equilibrium (NE) in real–world settings is often impractical due to its too restrictive assumptions. Game theory and artificial intelligence provide alternative (relaxed) solution concepts. When knowledge about opponents’ utilities and types are not available, the appropriate solution concept for extensive–form games is the self–confirming equilibrium(SCE), which relaxes NE allowing agents to have wrong beliefs off the equilibrium path. In this paper, we provide the first computational and learning study of the situations in which a two–agent extensive–form game is played by heterogeneous populations of individuals that repeatedly match (e.g., eBay): we extend the SCE concept, we study the equilibrium computation problem, and we study how these equilibria affect learning dynamics. We show that SCEs are crucial for characterizing both stable states of learning dynamics and the dynamics themselves.

In recent years, the request for batteries to employ in emerging technologies like smart grids or electric vehicles shows constant growth. To maintain these systems over time, it is crucial to have a mechanism to monitor the battery State of Health (SoH), determine when it is not of use for the current application, and eventually reuse it in another context a.k.a. battery second life. However, standard techniques from the literature provide an accurate estimation of the State of Health mainly by performing offline tests or with a prioriknowledge of hyper-parameters. This paper proposes a novel algorithm, namely State of Health Estimator (SHE), that infers the battery model online, i.e., during its operational life, and uses this characterization to provide a reliable and accurate estimation of both actual battery capacity and internal resistance, considering both ohmic and polarization components. The experimental campaign, performed on real-world data, shows satisfactory performance, with an average error of 1.2 % and of 4 % in the estimate of the maximum battery capacity and internal resistance, respectively.

In multi-task reinforcement learning (MTRL), the objective is to simultaneously learn multiple tasks and exploit their similarity to improve the performance w.r.t. single-task learning. In this paper we investigate the case when all the tasks can be accurately represented in a linear approximation space using the same small subset of the original (large) set of features. This is equivalent to assuming that the weight vectors of the task value functions are jointly sparse, i.e., the set of their non-zero components is small and it is shared across tasks. Building on existing results in multi-task regression, we develop two multi-task extensions of the fitted Q-iteration algorithm. While the first algorithm assumes that the tasks are jointly sparse in the given representation, the second one learns a transformation of the features in the attempt of finding a more sparse representation. For both algorithms we provide a sample complexity analysis and numerical simulations.

In recent years, the use of Lithium-ion batteries in smart power systems and hybrid/electric vehicles has become increasingly popular since they provide a flexible and cost-effective way to store and deliver power. Their full integration into more complex systems requires an accurate estimate of the energy a battery is currently storing, a.k.a. State of Charge (SoC). However, the standard techniques present in the literature provide an accurate estimation of the SoC only having a priori knowledge about the battery. Moreover, their accuracy degrades if the battery working conditions (e.g.,external temperature) are variable over time, or battery measurements necessary for the SoC estimation are affected by offset or gain biases. To overcome these limitations, this paper proposes a novel data-driven optimization based methodology for battery SoC estimation, namely VDB-SE. The proposed methodology provides accurate SoC estimations without knowing battery model parameters, such as capacity and internal resistance, whose characterization would require complex and long laboratory tests. Experimental verification and comparisons demonstrate that VDB-SE performance are comparable to the state-of-the-art algorithms over a wide range of working conditions. Indeed, the difference in terms of performance is smaller than 0.2%. Moreover, experimental results showed that on a real energy storage system the proposed method provides a SoC estimation with an error of less than 2.1%.

In behavior-based robotics, robots are controlled by behavioral modules that are triggered on data perceived by sensors or according to some model of the situation to be faced. However, data are acquired by imperfect sensors and, whenever a world modeler exists, are usually interpreted by modelling algorithms that abstract the relevant features inevitably loosing information. In this paper, we present our approach to data reliability and to its propagation through the path from data to actuation. We show how reliability is different from truth value or from approximation and how this impacts on the agent's performance

Stuck-pipe phenomena can have disastrous effects on drilling performance, with outcomes that can range from time delays to loss of expensive machinery. In this work, we develop three indicators based on mudlog data, which aim to detect three different physical phenomena associated with the insurgence of a sticking. In particular, two indices target respectively the detection of translational and rotational motion issues, while the third index concerns the wellbore pressure. A statistical model that relates these features to documented stuck-pipe events is then developed using advanced machine learning tools. The resulting model takes the form of a depth-based map of the risk of incurring into a stuck-pipe, updated in real-time. Preliminary experimental results on the available dataset indicate that the use of the proposed model and indicators can help mitigate the stuck-pipe issue.