🏎️ Reinforcement Learning Car Simulator (PPO with PyTorch)

This project implements a compact yet powerful reinforcement learning environment for training an autonomous car to drive on a 2D procedural track.
The agent is trained using Proximal Policy Optimization (PPO) — a modern policy-gradient algorithm.
Simulation and rendering are handled by Pygame, while the neural policy is implemented in PyTorch.

---

🧠 Model Architecture

The agent (AIDriver) is a two-headed neural network:

• Policy head (π) — outputs action logits (steering/brake/accelerate)
• Value head (V) — estimates the expected return from the current state

Both heads share a deep feature extractor (three ReLU layers, each with 128 units).

Observation → [MLP Body] → ┬─> π(s)
                           └─> V(s)

---

🏋️‍♂️ Training Process

Run training

python -m ML.train

Training pipeline

The training script automatically:

• Collects N_RUNS_FOR_HISTORY simulation rollouts
• Computes Generalized Advantage Estimates (GAE) per episode
• Performs PPO updates on minibatches
• Periodically saves model weights to
  ML/weights/model.pth

Training setup

Parameter	Description	Value
epoch_number	Total epochs	10,000
N_RUNS_FOR_HISTORY	Rollouts per PPO update	10
episode_rerun	PPO epochs per batch	10
MINIBATCH_SIZE	Batch size	128
Learning rate	0.0001
Clip ε	0.2
Entropy coeff.	0.005
Value coeff.	0.5

Training schedule

1. Stage 1 – Stable Learning (Easy Track)
   The agent trains for 10,000 epochs on a single fixed track (one_track=True)
   to develop basic control, stability, and cornering behavior.

2. Stage 2 – Generalization (Dynamic Tracks)
   After pretraining, the environment switches track generation every 2,000 iterations.
   This exposes the agent to varying shapes, widths, and corner placements —
   encouraging it to generalize its driving policy.

---

🎮 Evaluation

For evaluation, you can run the following:

python -m ML.eval

This loads the current trained weights and visualizes how the model drives.
By default, the evaluation model uses the file:

ML/weights/unified.pth

If you start a new training session, new weights will be saved as:

ML/weights/model.pth

---

👾 Try It Yourself

You can also take manual control of the car and experience the environment:

python main.py

Use the keyboard to drive (WASD by default).
This allows you to see how the physics, sensors, and collision detection behave.

---

⚙️ Requirements

pygame>=2.6.0
torch>=2.2.0
tqdm
numpy

---

📈 Typical Learning Behavior

During training, average episode rewards fluctuate widely due to exploration, but the general trend increases over time.
Once trained, the car successfully follows both the default and randomized tracks with smooth acceleration and controlled steering.

---

🏁 Final result