RL-based Optimization in Battery Pools for DA flexibility

Recently, I published a paper on marketing energy flexibility, available in groups of Photovoltaic (PV)-battery-storage systems, in the Day-ahead (whole-sale) electricity market. This paper is part of the Flex+ project: https://www.flexplus.at/.

Such optimization problem consists of two subproblems: a) which amounts of energy should we trade over the next day (usually in 15min granularity), b) which batteries should be used to generate this flexibility (having in mind that the use of the batteries incurs additional costs due to their limited lifetime). One of the major problems addressing such complex optimization problem is the induced complexity due to the underlying dynamics of the battery-storage systems, and the uncertainty both in PV generation and electricity load consumption (i.e., the behavior of the residents). The question emerging is how shall we trade the available flexibility during the next day, in order for the users to reduce their electricity costs.

Reinforcement learning could provide a framework for addressing such complex (two-level) optimization problem, since it is a combinatorial optimization problem with uncertainties. However, the limited availability of historical data and the size of the problem (given that the number of the involved batteries could be in the range of thousands) do not allow for a black-box-based design implementation of reinforcement learning.

For this reason, we follow a more problem-specific design of the reinforcement learning implementation, since this problem allows for a rather accurate design of the policy function. Given that the type of the policy function is known, training allows for minimizing the impact of the uncertainties. The performance is also compared with standard linear-programming-based methodology that provides the theoretical optimum. The details of this design are available in the following paper: https://idss-tech.blog/wp-content/uploads/2020/11/chasparis-and-lettner-reinforcement-learning-based-optimization-for-day-.pdf

A presentation is also available in the following youtube link: