With the rapid proliferation of battery EV cars, the distribution network is facing unprecedented pressure from charging loads. Traditional management methods, such as fixed scheduling or heuristic algorithms, struggle to handle the randomness and spatiotemporal fluctuations of these loads. This challenge threatens grid stability,调度 flexibility, and the efficient integration of renewable energy sources. As a researcher in this field, I have explored advanced techniques to address these issues, focusing on how to optimize charging schedules while ensuring user satisfaction and grid safety. In this article, I present a comprehensive study on a multi-agent reinforcement learning (MARL) framework for collaborative charging optimization of battery EV cars. The goal is to develop a strategy that not only reduces peak loads and costs but also adapts to dynamic environments and large-scale deployments.
The core of my approach lies in modeling the charging coordination problem as a partially observable Markov decision process (POMDP) extended to multi-agent settings. I consider a residential area with multiple households, each equipped with a battery EV car and possibly rooftop photovoltaic (PV) systems. These battery EV cars are connected to a transformer that interfaces with the main grid, allowing for energy exchange. The system aims to manage charging and discharging decisions to minimize costs, alleviate user range anxiety, and prevent transformer overloads. To visualize this setup, I include a representation of the residential EV charging control model below.
In this model, each battery EV car acts as an intelligent agent, making decisions based on local observations and shared information. The environment state at time \( t \) is represented as a joint state set \( S_t \), comprising three key components: the charging states of all battery EV cars \( s^{\text{EV}}_t \), the grid conditions \( s^{\text{Grid}}_t \), and the status of distributed energy resources like PV generation \( s^{\text{DER}}_t \). Formally, this is expressed as:
$$ s_t = \{ s^{\text{EV}}_t, s^{\text{Grid}}_t, s^{\text{DER}}_t \} $$
For each battery EV car \( i \), the action \( a_{i,t} \) corresponds to the chosen charging or discharging power at time \( t \). The joint action of all \( N \) battery EV cars is denoted as \( \mathbf{a}_t = (a_{1,t}, a_{2,t}, \dots, a_{N,t}) \). These actions are constrained by physical limits, such as maximum charging and discharging rates, ensuring practical feasibility:
$$ -a_{\text{min}} \leq a_{i,t} \leq a_{\text{max}} $$
where \( a_{\text{max}} \) and \( a_{\text{min}} \) are the upper and lower bounds for power flow, typically set based on the battery EV car’s capabilities. To guide the learning process, I design a reward function that balances multiple objectives. The total reward for agent \( i \) at time \( t \), \( r_{i,t} \), integrates four components: cost reduction, user anxiety mitigation, transformer overload prevention, and PV utilization. Each component is detailed below with formulas.
First, the cost reward \( r_{p,i,t} \) is tied to electricity prices, encouraging charging during low-price periods and discharging when prices are high to gain revenue:
$$ r_{p,i,t} = -\lambda_t \cdot a_{i,t}, \quad \text{for } t_{i,a} \leq t \leq t_{i,d} $$
where \( \lambda_t \) is the real-time electricity price, and \( t_{i,a} \) and \( t_{i,d} \) are the arrival and departure times of the battery EV car \( i \). Second, to address user range anxiety—a critical factor for battery EV car adoption—I incorporate a time-sensitive anxiety weight \( \omega(t) \), which increases as the departure time approaches. This weight is calculated as:
$$ \omega(t) = \frac{e^{c(t – t_{\text{arr}})/(t_{\text{dep}} – t_{\text{arr}})} – 1}{e^c – 1}, \quad \forall t \in T $$
where \( c \) is an anxiety growth factor, and \( T \) is the time horizon from arrival to departure. The anxiety reward \( r_{\text{ra},i,t} \) is then defined as:
$$ r_{\text{ra},i,t} = -\alpha \times \omega(t) \times \Delta \text{SOC}(t) $$
with \( \Delta \text{SOC}(t) \) being the difference between the maximum state of charge (SOC) and the current SOC, and \( \alpha \) a weighting coefficient. Third, to avoid transformer overloads, a penalty reward \( r_{o,i,t} \) is applied when the total load \( W(t) \) exceeds a threshold \( W_{\text{th}} \):
$$ r_{o,i,t} = \begin{cases}
-\vartheta \frac{p_{i,t}}{W(t)} [W(t) – W_{\text{th}}]^2, & \text{if } W(t) \geq W_{\text{th}} \\
0, & \text{if } W(t) < W_{\text{th}}
\end{cases} $$
where \( \vartheta \) is a penalty coefficient, and \( p_{i,t} \) is the power consumption of household \( i \). Finally, to promote renewable energy use, a PV curtailment penalty \( r_{\text{PV},i,t} \) is included:
$$ r_{\text{PV},i,t} = -\lambda_{\text{PV}} \sum_t (P^c_{\text{PV},t} – P_{\text{PV},t}) \Delta t $$
where \( \lambda_{\text{PV}} \) is a penalty coefficient, \( P^c_{\text{PV},t} \) is the available PV generation, and \( P_{\text{PV},t} \) is the actual output. The overall reward for each battery EV car agent is the sum of these components:
$$ r_{i,t} = r_{p,i,t} + r_{\text{ra},i,t} + r_{o,i,t} + r_{\text{PV},i,t} $$
This reward structure ensures that the multi-agent system optimizes for economic efficiency, user satisfaction, grid safety, and environmental sustainability simultaneously.
To solve this complex optimization problem, I propose the Multi-Agent Trust Region Policy Optimization with Variational Information (MATRPO-VI) algorithm. This algorithm builds on traditional trust region policy optimization (TRPO) but enhances it with a variational information bottleneck (VIB) mechanism and a multi-scale attention mechanism. The goal is to improve training efficiency, policy robustness, and inter-agent coordination for battery EV car fleets. The MATRPO-VI framework operates in a centralized training with decentralized execution manner, where each battery EV car agent learns its policy while considering the actions of others.
The TRPO component ensures stable policy updates by constraining the Kullback-Leibler (KL) divergence between the old and new policies. For a multi-agent setting, the optimization objective for all agents is formulated as:
$$ \max_{\theta} \mathbb{E}_{s \sim d^{\pi_{\theta_{\text{old}}}}, a \sim \pi_{\theta_{\text{old}}}} \left[ \frac{\pi_\theta(a | s)}{\pi_{\theta_{\text{old}}}(a | s)} A^{\pi_{\theta_{\text{old}}}}(s, a) \right] $$
subject to \( D_{\text{KL}}(\pi_{\theta_{\text{old}}} \| \pi_\theta) \leq \delta \), where \( \theta \) represents the policy parameters, \( d^{\pi_{\theta_{\text{old}}}} \) is the state visitation distribution, \( A \) is the advantage function, and \( \delta \) is a trust region threshold. This constraint prevents large policy updates that could destabilize learning, which is crucial when coordinating multiple battery EV cars.
The VIB mechanism is introduced to filter redundant information from the state observations. It learns a latent representation \( Z \) that preserves only the information relevant to predicting rewards, thereby improving learning efficiency. The VIB objective is to minimize the mutual information between \( Z \) and the state \( S \) while maximizing the mutual information between \( Z \) and the reward \( R \):
$$ \min I(Z; S) – \beta I(Z; R) $$
where \( \beta \) is a hyperparameter that balances compression and relevance. This helps each battery EV car agent focus on critical features, such as electricity prices or SOC levels, ignoring noisy data.
Additionally, I incorporate a multi-scale attention mechanism to enhance interaction among battery EV car agents. This mechanism computes attention weights across different time scales, allowing agents to capture both short-term and long-term dependencies. For agent \( i \) and agent \( j \) at scale \( k \), the attention weight \( \alpha^{(k)}_{ij} \) is computed as:
$$ \alpha^{(k)}_{ij} = \frac{\exp(e^{(k)}_{ij})}{\sum_m \exp(e^{(k)}_{im})}, \quad e^{(k)}_{ij} = \text{LeakyReLU}\left( \mathbf{W}^{(k)}_q s_i \cdot \mathbf{W}^{(k)}_k s_j \right) $$
where \( \mathbf{W}^{(k)}_q \) and \( \mathbf{W}^{(k)}_k \) are weight matrices for queries and keys, and \( s_i \) and \( s_j \) are input features. The final attention weight \( \alpha_{ij} \) is a weighted sum across scales:
$$ \alpha_{ij} = \sum_k w_k \alpha^{(k)}_{ij} $$
with \( w_k \) being learnable weights. This enables dynamic adjustment of interactions, improving coordination in scenarios like peak shaving or valley filling for battery EV car charging.
The overall MATRPO-VI algorithm流程 is summarized in the table below, outlining key steps from initialization to policy updates.
| Step | Description | Key Operations |
|---|---|---|
| 1 | Initialization | Set policy parameters \( \theta \), value network parameters \( \phi \), KL constraint \( \delta \), VIB parameter \( \beta \), and experience replay buffer \( D \). |
| 2 | Environment Interaction | For each battery EV car agent \( i \), observe state \( s_{t,i} \), compute attention weights, apply VIB to get latent state \( z_{t,i} \), select action \( a_{i,t} \) from policy \( \pi_\theta \), execute action, receive reward \( r_{i,t} \) and next state \( s_{t+1,i} \), store transition \( (s_{t,i}, a_{i,t}, r_{i,t}, s_{t+1,i}) \) in \( D \). |
| 3 | Policy Update | Sample batches from \( D \), compute advantage estimates \( A \), update policy network using TRPO with KL constraint, update value network \( V_\phi \), copy old parameters \( \theta_{\text{old}} \leftarrow \theta \), clear buffer \( D \) periodically. |
| 4 | Convergence Check | Repeat steps 2-3 until policy performance stabilizes or maximum iterations reached. |
To validate the effectiveness of MATRPO-VI, I conduct extensive experiments in simulated environments. The experimental setup mimics a residential community with varying numbers of battery EV cars. Key parameters are derived from real-world data: hourly electricity prices follow a dynamic pattern, PV generation profiles are based on typical solar irradiance, and household load data reflects daily consumption trends. For user behavior, I model arrival times \( t_{\text{arr}} \), departure times \( t_{\text{dep}} \), and initial SOC values using normal distributions, as summarized in the table below.
| Variable | Probability Distribution | Range |
|---|---|---|
| \( t_{\text{arr}} \) | \( \mathcal{N}(18, 1) \) | [15, 21] hours |
| \( t_{\text{dep}} \) | \( \mathcal{N}(8, 1) \) | [5, 11] hours |
| Initial SOC | \( \mathcal{N}(0.5, 0.25) \) | [0.2, 0.8] |
Each battery EV car is assumed to have a battery capacity of 40 kWh, with charging and discharging power limits of ±7 kW, and an energy conversion efficiency of 0.9. The discount factor for selling electricity back to the grid is set to 0.6. Simulations are implemented in Python 3.9 using PyTorch for deep learning components.
I first evaluate the training performance of MATRPO-VI in a small-scale scenario with 50 battery EV cars. The transformer capacity is set to 60N kVA, where \( N \) is the number of households. The mean squared error (MSE) loss of the policy network during training shows a rapid decline in the first 200 episodes, indicating effective learning of charging patterns for battery EV cars. By episode 100, the loss drops below 2.0 on both training and test sets, demonstrating quick convergence. After around 250 episodes, the loss stabilizes, suggesting that the policy has matured and can reliably coordinate charging actions.
To benchmark MATRPO-VI, I compare it against three established reinforcement learning algorithms: Deep Q-Network (DQN), Trust Region Policy Optimization (TRPO), and Multi-Agent Deep Deterministic Policy Gradient (MADDPG). These algorithms are tested over a 100-day simulation period, and results are aggregated in the following table, highlighting training time and cumulative reward.
| Algorithm | Training Time (seconds) | Total Reward |
|---|---|---|
| DQN | 442 | -4042 |
| TRPO | 264 | -3607 |
| MADDPG | 87 | -3040 |
| MATRPO-VI | 62 | -2819 |
MATRPO-VI achieves the shortest training time and the highest cumulative reward (least negative), indicating superior efficiency and optimization capability. The reward components analysis reveals that MATRPO-VI significantly reduces charging costs, minimizes user anxiety, and avoids transformer overloads better than other methods. This is crucial for practical deployment where battery EV car fleets must balance economic and grid constraints.
Further, I analyze the power调度 effects of each algorithm on a typical day. The plot of total charging load over time shows that TRPO and MADDPG can shift some load to off-peak hours, but their curves exhibit noticeable fluctuations. In contrast, MATRPO-VI produces a smoother power profile with minimal peaks, effectively flattening the load curve. This is attributed to the combined benefits of VIB, which filters out noise, and multi-scale attention, which enhances coordination among battery EV cars. For instance, during high-price periods, MATRPO-VI schedules more discharging actions, while in low-price intervals, it maximizes charging, all while maintaining SOC levels to satisfy user departure needs.
To assess scalability, I extend the simulation to a large-scale community with 500 battery EV cars. The results demonstrate that MATRPO-VI maintains robust performance even in this expanded setting. The average charging and discharging power over a day align well with real-time electricity prices: during low-price hours (e.g., 0:00–5:00), charging activity increases, whereas during high-price periods (e.g., 16:00–18:00), discharging is encouraged to feed energy back to the grid. The SOC trajectories show that battery EV cars maintain high charge levels before morning departures (5:00–8:00), ensuring users have sufficient range for their commutes. This adaptability underscores the algorithm’s potential for real-world applications where the number of battery EV cars can vary widely.
The key formulas used in the reward function and algorithm are consolidated below for reference:
$$ \text{Reward for battery EV car } i: \quad r_{i,t} = -\lambda_t a_{i,t} – \alpha \omega(t) \Delta \text{SOC}(t) + r_{o,i,t} – \lambda_{\text{PV}} \sum_t (P^c_{\text{PV},t} – P_{\text{PV},t}) \Delta t $$
$$ \text{Anxiety weight:} \quad \omega(t) = \frac{e^{c(t – t_{\text{arr}})/(t_{\text{dep}} – t_{\text{arr}})} – 1}{e^c – 1} $$
$$ \text{TRPO objective:} \quad \max_\theta \mathbb{E} \left[ \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)} A(s,a) \right] \text{ s.t. } D_{\text{KL}}(\pi_{\theta_{\text{old}}} \| \pi_\theta) \leq \delta $$
$$ \text{VIB objective:} \quad \min I(Z; S) – \beta I(Z; R) $$
$$ \text{Attention weight:} \quad \alpha_{ij} = \sum_k w_k \frac{\exp(\text{LeakyReLU}(\mathbf{W}^{(k)}_q s_i \cdot \mathbf{W}^{(k)}_k s_j))}{\sum_m \exp(\text{LeakyReLU}(\mathbf{W}^{(k)}_q s_i \cdot \mathbf{W}^{(k)}_k s_m))} $$
In conclusion, my research presents a novel multi-agent reinforcement learning framework for optimizing the charging of battery EV cars. By integrating trust region policy optimization, variational information bottlenecks, and multi-scale attention mechanisms, the proposed MATRPO-VI algorithm effectively addresses the challenges of dynamic environments, user satisfaction, and grid stability. Experimental results confirm its advantages in training speed, load management, and scalability compared to existing methods. This work contributes to the broader goal of sustainable energy systems, where battery EV cars play a pivotal role. Future directions include incorporating more realistic battery degradation models and testing in real-world smart grid deployments to further validate the approach.