MRDrive

An Open Source Mixed Reality Driving Simulator for Automotive User Research

---

Table of Contents

• Environment
• AI Agents
• Traffic Lights
• Dashboard
• In-Vehicle Infotainment System
• Running the Simulation
• Setup
• Troubleshooting / Known Issues

Project Origin

This project is a heavily modified version of a coupled simulator originally developed by Dr. Pavlo Bazilinskyy at TU Eindhoven for research on driver-pedestrian interactions.

Our adaptation combines a fixed-based driving simulator with a virtual environment into a Mixed-Reality solution powered by the Varjo XR-3 HMD. It features two additional screens (dashboard and IVIS) and has been migrated to the HDRP in Unity. This setup provides a platform for in-depth research on driver-interface interaction under realistic simulated conditions.

Goal of the Project

This project aims to serve as a valuable contribution to the open-source landscape directed at researchers and developers interested in advancing mixed reality applications for mobility and interaction studies. Further contributions, collaborations, and discussions are encouraged to enhance the simulator's capabilities and research applicability.

---

Environment

The simulator replicates a realistic urban environment, integrating roads, buildings, trees, and animated pedestrians. The layout of the roads as well as the street lights and some pedestrians were adopted from Dr. Pavlo Bazilinskyy's coupled simulator. Buildings, props, and trees have been adjusted. It features:

• Network of 2-lane roads.
• Loop of 4-lane road (partially surrounded by buildings).
• Loop of 6-lane road (partially surrounded by buildings).
• Half-clover interchange for the motorway.
• 10 intersections with traffic lights.
• 34 zebra crossings.
• Static objects (buildings, parked cars and bikes, trees, park benches, bins, flowers).
• Dynamic objects (pedestrians, cars).

AI Agents

There are agents in form of pedestrians and other cars, some active, some idle in the scene. Active agents follow specific waypoint cycles while avoiding collisions using colliders and raycasts. If a pedestrian crosses the street on a crosswalk, cars will stop until the street is clear. Cars try to avoid crashing into other cars by slowing down based on proximity to the car in front.

Traffic Lights

There is a traffic light system implemented at all crossings (the traffic lights themselves are free unity assets by Healthbar Games). Each crossing features a Traffic Light Manager script which controls two phases, NS (north & south) and EW (east & west). When a traffic light is red, AI cars will stop at a specified stop line (yellow line) in front of the pedestrian crosswalk. Similarily, cars will stop here if they detect a pedestrian on the crosswalk. To facilitate gradual braking, AI cars check the state of an upcoming traffic light once they enter a crosswalk zone (yellow zone) and adjust their speed in relation to their current velocity and distance to the stop line.

---

Dashboard

The dashboard display shows key driving information:

• Top: Time, date, temperature (mock).
• Left side: Brake and throttle input are displayed as vertical bars; Current gear is highlighted, along with inactive gears.
• Center: The steering wheel angle and current speed.
• Right side: Additional driving metrics including speed, elapsed time, distance traveled, and consumption (mock).

There is a range of buttons, mostly for decoration. The left and right indicator icons are mapped to inputs; on start up, all buttons shortly light up, then fade out.

---

In-Vehicle Infotainment System

The IVIS features a Birds-Eye-View of the Player Car in a simplified environment. Pedestrians and other vehicles are marked in light green. There are buttons to play/pause and mute/unmute music, as well as an assistant button that is interactive during the experiment. The Pull Over button is not functional.

There are two additional screens: A Start screen, and a Home screen. The start ride button starts the AI driving function and log during the experiment. Clicking buttons on the Home screen will toggle the Home screen visibility.

The Home screen is inactive on default. To activate it, the HomeButton should be enabled in the project hierarchy. It is also recommended to disable the Start screen for manual driving.

---

Running the Simulation

The Experiment Config is the first element in the scene hierarchy and contains the overall setup of the simulation.

Variable	Meaning
Is Automated	Checked means the player car is being controlled by AI and follows a waypoint trajectory. Unchecking enables manual driving.
Is Logging	Checking this enables the logging of button inputs. Specify a directory in the input field below.
Is Trial Route	Check this to select a short trial run without any condition
Condition	Select an experimental condition: Lumo (optional explanations), Coda (proactive explanations), Nelo (no explanations)
Route	Select a waypoint route [A, B, C, T]. T is the trial route.

The remainder of inputs and assignments in this menu are preset and can be left as is.

You can run the simulation directly from the editor; however, Unity can only render one display at a time. If you want to make full use of all displays, compile a build of the project (make sure development build is checked in the build settings; otherwise, certain features won't work).

---

Setup

Tech Stack

Software:

• Unity 2022.3.5f1

• Varjo Base & Varjo Lab Tools

• Steam & SteamVR

• Varjo SDK for Unity (pre-installed)

• DirectX11

• vJoy

---

Prerequisites

Hardware:

• PC with Windows 10 or 11

• GPU with two ports connected directly (Varjo XR-3 requirement)

• Four SteamVR Base Stations 2.0

• Fixed-Based Driving Simulator (Ergoneers)

Software:

Following assets require licenses:

• Realistic Car Controller v3 (Bonecracker Games) -- Mandatory; Basic vehicle control and physics asset. Link to Unity Asset Store
• German Emergency Vehicles (VIS Games) -- Optional; Collection of car assets. Only needed for Route C of the experiment; you can remove it from the project if you only want to use the simulation. Link to Unity Asset Store

This simulation was build for use with the Ergoneers Fixed-Based Driving Simulator. If you do not have this, the simulation can be adapted to run with a different setup by modifiying the input system and adjusting the Mixed-Reality set up.

---

Varjo Software Installation

1. Install Varjo Base from Varjo Website.
2. Download Varjo Lab Tools from Varjo Website.

These applications are essential for managing the headset and configuring the mixed reality settings.

Steam and SteamVR

1. Install Steam and SteamVR. SteamVR is necessary for utilizing the base stations, which enable motion tracking. Follow the internal instructions for installation and setup.

Further tracking solutions

Motion and Hand Tracking can also be done with the Inside-Out-Tracking feauture of Varjo XR-3 (which is still in Beta).
For this go in Varjo Base to System, and enable Inside-Out-Tracking with Varjo (Beta).

Hardware Connections

1. Connect the Varjo XR-3 headset to your PC/Laptop with the included connector, ensuring you use the correct ports or adapters specified for your hardware configuration.

Varjo and Unity integration

More info is to be found on the Varjo developer page.
Make sure to check all Menu Elements, as navigation on Varjo page isn't very clear.

Unity Project Configuration

Varjo SDK installation

Follow the instructions on Varjo page if Varjo SDK is not already pre-installed or corrupted in the project. You need to install git for it to work.

Follow all the steps on Varjo Page until the segment about Converting the Main Camera to an XR-Rig.

!Deviation from Varjo Tutorial!

A one-click conversion of our scene is not possible, because we have multiple cameras in the scene. Here is a detailed instruction how to setup an XR-Rig for Varjo XR-3 when having multiple cameras in the scene:

1. in Unity ➡️ (located at left bottom) Project ➡️ Assets ➡️ Assets ➡️ Locate the DrivableCommonObject using search function. This object is the car model. It is the modified to integrating the real car model with the virtual car in the virtual driving environment.

2. Under DrivableCommonObject, navigate to Driver Logic, open it up, manually insert an XR Origin component by clicking Right Click on the mouse ➡️ XR ➡️ XR Origin(Mobile AR).

3. Place XR Origin-Element under CameraCounter in DriverLogic.

4. Place existing Main Camera and its children under the XR Origin. This step makes the Main Camera an XR Origin camera

Head Tracking Configuration

Now we after placing the Main Camera as a child of XR Origin, we need to implement the head tracking functionality, allowing us to look around and move in the scene.

1. Click on Main Camera, scroll down and click Add Component. Search for Tracked Pose Driver specifically. NOT Tracked Pose Driver (Input System). As Device choose Generic XR Device. As Pose Source choose Head or Center Eye - HMD Reference.
2. For Tracking Type choose Rotation only or Rotation and Position. For update type we chose Update And Before Render.

Initially, we set it to track only the rotation. The reason being, that the Inside-Out-Tracking from the Varjo headset didn't allow for really precise and repeatable starting position calculation, because each time the starting point in real world was defined after calibration. Position tracking will be added once the real car model is integrated, to ensure the virtual and real-world align accurately. This will be done using the SteamVR Basestation 2.0.

If you still decie to track the head position using the Inside-Out-Tracking, adjust the position of the CameraPositionSet in the scene. To see how the Camera is placed in the scene, press #Scene, left of the Game icon at the top middle-right. As the camera's position is measured relative to the floor it may be necessary to place the origin further down than expected.

Post-Processing Bugs

When using the Varjo XR-3 many Post-Processing settings can't be used, as they create visual artifacts in the players filed of view. Follow the instructions by Varjo and disable all the settings listed, which can't be used with Varjo XR-3. There is a YouTube video by user "FowardX" which could help visualise the issues for our use-case.

Testing the Setup for VR

With the above configuration, the simulator should now be playable in full VR mode. For it to work correctly run the programms in the following order:

1. Varjo Base
2. Calibrate Inside-Out-Tracking in Varjo Base.
3. Run the simulation in Unity.

Test the setup to ensure that the virtual environment is correctly rendered through the Varjo XR-3 headset and that motion tracking functions as expected.

Eye tracking / Logging

We want to track the point on which the eyes of the user focus, projected on their view of the Mixed-Reality environment. For this there is a possibility to extract the data from Unity, but that is a non-efficient workaround.
There is a much better, easier and more visual implementation made directly by Varjo. On Varjo's website you can find an extensive guide on it. This implementation uses Varjo Base Software to record the view from the HMD. The output is a video with the view from the HMD and a point showing where the eyes of the user are looking at. We also get an extensive eye tracking file with multiple variables ranging from gaze coordinates to quality of eye tracking.

---

Troubleshooting / Known Issues

Pedestrians Not Moving

Problem: Pedestrians in the scene are not performing animations.

Possible Causes & Fixes:

1. Animation Controller Issue

• Ensure animations are connected to the "Entry" Node.

  • Delete and re-drag animations from the character folder into the Animation Controller.

• Check Character's Rig settings:

  1. Go to Assets → Models → People → <Character_Name>.
  2. Select the character’s .fbx.
  3. In the Inspector, open the Rig tab.
     • Animation Type: Humanoid
     • Avatar Definition: Create From This Model
  4. Re-apply these settings.

• Check Animation's Rig settings:

  1. In the Project window, select the animation
  2. In the Rig tab:
     • Animation Type: Humanoid
     • Avatar Definition: Copy From Other Avatar
     • Source: The character’s avatar
  3. Re-apply these settings.

2. Missing Script
   • Only affects characters prefixed with "Pedestrian_".
   • Idle_Characters use Animator only.
   • Compare inspector to reference screenshot to confirm setup.

   

Figure 1: This is what the Inspector of your pedestrian should look like. If any scripts are missing, make sure to find them in the project folder and add them to the Inspector. This is also important if you want to add new characters to the scene.

Tip: Multiple pedestrian failures usually indicate a controller problem; single failures may be rig or missing script issues.