Reinforcement learning · HVAC control

Teaching an AI to keep a
data center cool.

A simulated data-center cooling controller trained with Soft Actor-Critic. The agent watches the room temperature and server load, then decides how hard to cool — holding the target temperature, dodging thermal violations, and burning as little energy as possible.

Try the live simulation View source
Gymnasium Stable-Baselines3 SAC PyTorch NumPy Matplotlib
300K
training timesteps
24 h
episode horizon
24 °C
target temperature
2 → 1
obs → action dims
The problem

Cooling is a control problem

Data centers spend a huge share of their power just on cooling. Too little and servers overheat; too much and you waste energy. It's a continuous balancing act — a perfect fit for reinforcement learning.

An environment

A custom Gymnasium env models room temperature with a heat-balance equation: IT load heats it up, cooling pulls it down, ambient air leaks in.

An agent

A Soft Actor-Critic policy learns from scratch by trial and error — no hand-coded rules — discovering when and how hard to run the cooling.

A reward

Every minute, the agent is scored on how close it held the target, how much power it used, and whether it crossed the thermal safety line.

Architecture

The control loop

Each 60-second tick, the agent picks a cooling effort, the simulated room reacts, and a reward flows back. Repeat 1,440 times for one simulated day.

SAC agent learns the policy cooling env physics simulation action a ∈ [0,1] [T, IT_load] + reward INSIDE THE ENVIRONMENT heat in 15 kW × IT_load cooling out 20 kW × action ambient leak drift → 28 °C ΔT = k(heat − cool) + leak reward signal penalize err², power, overheat
agent heat source cooling reward neutral / state
The physics

Heat balance

Room temperature evolves with a simple first-order heat equation. Servers add heat, the agent's cooling removes it, and the warmer outside air constantly leaks in.

ΔT = k ·( heatIT − coolaction ) + leakambient
heatIT rises with server load (0–1) · coolaction rises with the agent's effort (0–1) · leak is a slow drift toward the 28 °C outside · k scales how fast the room responds.
The objective

Reward function

Each step returns a negative cost: squared distance from target, a small energy term, and a heavy penalty for crossing the 30 °C safety ceiling.

r = −( (T − 24)² + 0.02·power )
if T > 30 °C → r −= 10
Because being even 1 °C off costs 1.0 while max energy costs only 0.4, the agent prioritizes holding temperature, then economizes on power once it's close.
Interactive

Run the simulation yourself

This is the project's physics, re-implemented live in your browser with the exact constants from SimpleCoolingCfg. Drive the cooling yourself, or hand it to the auto-pilot and watch it hold the line.

LIVE Server room · thermal sandbox
Temperature
24.0 °C
Cooling effort
60 %
Power draw
12.0 kW
Cumulative reward
0
Sim time: 0.0 h Violations: 0
Note — this sandbox reproduces the environment's dynamics for intuition. The auto-pilot is a hand-tuned reference controller; the actual project trains a Soft Actor-Critic network to discover a policy like it from scratch over 300K steps.
Get started

Run it locally

Pure Python — clone, install, and you can sanity-check the sim, train an agent, and watch it work in three commands.

01 / SANITY

Test the environment

Runs the sim with random actions and live plots — no RL needed. Confirms everything works.

02 / TRAIN

Train the SAC agent

Normalizes observations, trains for 300K steps, then saves the policy and normalization stats.

03 / EVALUATE

Watch it cool

Loads the trained model, runs one full day deterministically, and renders the result to a plot.

# 1 — install dependencies
pip install -r requirements.txt
cd environment

# 2 — sanity-check the simulation (random actions)
python test_env_run.py

# 3 — train the Soft Actor-Critic agent (writes sac_dc.zip + vecnorm.pkl)
python train_sac.py

# 4 — load the policy and render one episode → eval_plot.png
python eval_and_render.py
data_center_cooling_v0.pyThe custom Gymnasium environment — physics, observations, reward, and live rendering.
train_sac.pyWraps the env in VecNormalize and trains the SAC policy for 300K timesteps.
eval_and_render.pyLoads the trained model and plays one deterministic episode with live plots.
test_env_run.pyThe simplest entry point — runs the sim with random actions, no RL.