Autonomous Driving W/ Deep Reinforcement Learning in Lane Keeping

SAC with KINEMATICS INPUT TRAINING RESULTS

SAC-KINEMATICS.mp4

DDQN-PER with KINEMATICS INPUT TRAINING RESULTS

DDQN-PER-KINAMATICS.mp4

---

Comparative Analysis of SAC and DDQN in 2D Autonomous Vehicle Simulation: Insights From Grayscale Bird View And Kinematics Inputs
Mehmet Yaşar Osman Özturan Computer Engineering Department, Bursa Uludağ University, Türkiye
1. International Black Sea Scientific Research and Innovation Congress 2023

---

Overview

A comprehensive comparative analysis of two formidable deep reinforcement learning algorithms: Soft Actor-Critic (SAC) and Double Deep Q-Network with Prioritized Experience Replay (DDQN with PER). Our primary goal was to discern how the choice of observation space influences the performance of these algorithms.

Our primary goal was to discern how the choice of observation space influences the performance of these algorithms and to offer an alternative to end-to-end deep learning studies carried out with raw sensor data and to show that processed data is much more successful in terms of reinforcement learning algorithms in the autonomous driving system, compared to raw data.

Method

Simulation Environment

• Using Highway-Env simulation.
• The simulated environment was designed to mimic a racetrack scenario.
• Vehicle tasked with lane-keeping and maintaining target speed on a racetrack.
• Testing two different deep reinforcement learning algorithms (SAC and DDQN-PER) with two different observation types (Kinematics and Birdview Images)

Action Space :

Both steering and throttle can be controlled. In fact, "one_act" file contains code for the situation where agents control steering only, and "two_acts" file contains code for the situation where agents control both steering and throttle. This doc focused on "two_acts".

Action spaces are continuous between [-1,1] values. Continuous action space is supported in SAC. For DDQN-PER, we discretize action space to 55 different action.

Observation Spaces

Two different observation types are testes:

1. Kinematics

3. Birdview Images

Reward Funtion

Designed to Promote:

• On-road behavior
• Distance to lane centering
• Target speed maintenance

** FOR TARGET SPEED MAINTENANCE WE USE GAUSSIAN FUNCTION

Terminal conditions:

• Agent is off road
• Agent reaches maximum number of steps
• Agent reaches maximum time to run

Deep Networks for Algorithms

For Kinematics Input

For Birdview Input

RESULS

Performance Graphs

Citation

If you find this work useful, please cite:

@INPROCEEDINGS{11329257,
  author={Osman Özturan, Mehmet Yaşar},
  booktitle={1. International Black Sea Scientific Research and Innovation Congress.}, 
  title={Comparative Analysis of SAC and DDQN in 2D Autonomous Vehicle Simulation: Insights From Grayscale Bird View And Kinematics Inputs}, 
  year={2023},
  keywords={Training;Soft Actor-Critic (SAC);Double Deep Q-Network;Navigation;Filtering;Deep reinforcement learning;Driver behavior;Autonomous vehicles;Tuning},

---

License

This project is licensed under the MIT License. See LICENSE for details.