Deep Reinforcement Learning Approach for HVAC Control

I. Vikström — M.Sc. thesis, Aalto University, December 2020 (with TietoEVRY)

Part of Reinforcement learning and control.

Heating, ventilation and air conditioning (HVAC) accounts for roughly a fifth of global energy consumption. There are two ways to reduce that: build machinery that draws less power, or control the existing machinery better. This thesis takes the second. Most HVAC plant is still controlled by simple rule-based schemes, which are reliable and easy to maintain but blind to electricity price, weather and occupancy. Model-based alternatives need a calibrated building model that is rarely available. The question here is whether an agent can learn a control policy from interaction alone.

The learning problem

Reinforcement learning does not decompose quite like supervised learning: there is no fixed dataset and no labels. The useful substitutions are experience for data, policy for model, and reward alongside loss. The last two are distinct objects that are easy to conflate.

Data points. There is no corpus. Experience is generated by interaction: the agent acts, an EnergyPlus building simulation responds, and the resulting transition is stored in replay memory and sampled from later. One timestep is five simulated minutes, so a simulated year is roughly 105,000 interactions. Agents were trained on Helsinki weather for 2013, a full year so that all four seasons appear, and validated on the first half of 2014.

Features. The state vector read from the simulator each step: indoor temperature, current heating rate, current cooling rate, number of occupants present, wind speed, wind direction, and solar diffuse radiation rate — plus the Nord Pool electricity spot price, which is what lets the agent learn to shift load towards cheap hours.

Actions (in place of labels). Not power output directly, but heating and cooling setpoints between +15 °C and +25 °C at 0.2 °C resolution — 51 discrete actions per step. The agent manipulates the thermostat, exactly as a human operator would.

Hypothesis space. A Deep Q-Network: an artificial neural network approximating the action-value function, which is what makes the high-dimensional state space tractable. The architecture, meaning the number and width of hidden layers, was chosen by a coarse grid search, because a single train-and-validate cycle took about four and a half hours. Exploration used linear epsilon decay.

Reward, and loss. These are two different things and the thesis keeps them separate. The reward combines energy cost (power drawn × spot price at that moment) with a work-performance cost drawn from a model of how productivity degrades as indoor temperature moves away from the comfortable zone. That second term is what stops the agent from discovering that the cheapest building is an unoccupiable one. The loss actually minimised by gradient descent is the DQN temporal-difference error, the gap between predicted and bootstrapped action values. It is not the reward.

What it found

The best agent cut energy costs by 24.4% and 37.1% against two baseline control strategies supplied by the simulator, while holding indoor conditions acceptable for simulated occupants.

The binding constraint is a systems problem rather than a learning one: communication overhead between the reinforcement learning program and EnergyPlus meant simulating a single year took nearly three hours, which is what limited the hyperparameter search. The reward design is the part most likely to transfer to other building-control problems, specifically the choice to price thermal discomfort in the same units as electricity.

Watch the defence

Full text

The thesis record is at Aaltodoc. The full text is not open access; the recorded defence above is the fullest public account.