NewSight Quest 3 - Unity VR Application

A Unity VR application for Meta Quest 3 that provides vision accessibility features through mixed reality passthrough.

Project Overview

This is a Unity port of the NewSight Android application, designed specifically for Meta Quest 3 with controller-based interaction and passthrough camera integration.

Requirements

• Unity Version: 2022.3.62f1 LTS (or later 2022.3.x)
• Target Platform: Meta Quest 3
• Meta XR SDK: v69.0.1 or later

Setup Instructions

1. Open the Project

1. Open Unity Hub
2. Click "Open" and navigate to NewSight-Quest3-Unity folder
3. Select Unity 2022.3.62f1 LTS as the Unity version
4. Wait for package resolution (may take a few minutes)

2. Install Meta XR SDK

If the Meta XR SDK doesn't auto-import:

1. Go to Window > Package Manager
2. Click the + button and select "Add package by name"
3. Enter: com.meta.xr.sdk.all
4. Click "Add"

Alternatively, import from Asset Store:

1. Go to Asset Store (Window > Asset Store)
2. Search for "Meta XR All-in-One SDK"
3. Download and import

3. Configure Build Settings

1. Go to File > Build Settings
2. Select Android platform
3. Click Switch Platform
4. Click Player Settings
5. Configure the following:
   • Company Name: NewSight
   • Product Name: NewSight Quest3
   • Package Name: com.newsight.quest3
   • Minimum API Level: Android 10.0 (API 29)
   • Target API Level: Android 12 (API 32)
   • Scripting Backend: IL2CPP
   • Target Architectures: ARM64

4. Configure XR Settings

1. Go to Edit > Project Settings > XR Plug-in Management
2. Select the Android tab
3. Enable Oculus checkbox
4. Go to XR Plug-in Management > Oculus
5. Set Target Devices to Quest 3

5. Configure OVR Manager

For each scene that uses VR:

1. Create an empty GameObject named "OVRCameraRig"
2. Add the OVR Camera Rig component
3. Add the OVR Manager component
4. Enable Passthrough in OVR Manager settings

6. Create Scenes

Create the following scenes in Assets/Scenes/:

Scene Name	UI Script
Login	LoginUI
Signup	SignupUI
ForgotPassword	(similar to Login)
Home	HomeUI
Settings	SettingsUI
UserProfile	UserProfileUI
TrustedContacts	(similar to Settings)
Privacy	(similar to Settings)
Help	(similar to Settings)
Navigation	NavigationUI
GetDirections	(similar to Navigation)
Emergency	EmergencyUI
EmergencySetup	(similar to Emergency)
DetectObjects	ObjectDetectionUI
ReadText	TextDetectionUI
DetectFaces	FaceDetectionUI
DetectColors	ColorDetectionUI
Communicate	ASLDetectionUI
VoiceCommand	VoiceCommandUI

7. Scene Setup Template

For each scene:

1. Create a new scene
2. Add OVRCameraRig prefab
3. Create an empty GameObject named "UI"
4. Add the appropriate UI script component
5. Add Canvas component to UI (World Space)
6. Configure Canvas:
   • Render Mode: World Space
   • Width: 800, Height: 1000 (varies by screen)
   • Scale: 0.001, 0.001, 0.001
7. Build your UI elements as children

Project Structure

NewSight-Quest3-Unity/
├── Assets/
│   ├── Scenes/           # Unity scenes for each screen
│   ├── Scripts/
│   │   ├── API/          # Backend communication
│   │   │   ├── ApiManager.cs
│   │   │   ├── BackendConfig.cs
│   │   │   └── WebSocketManager.cs
│   │   ├── Core/         # Core app functionality
│   │   │   ├── AppManager.cs
│   │   │   ├── SceneLoader.cs
│   │   │   └── VRInputHandler.cs
│   │   ├── Passthrough/  # Quest 3 passthrough camera
│   │   │   └── PassthroughManager.cs
│   │   └── UI/           # Screen-specific UI scripts
│   │       ├── UITheme.cs
│   │       ├── VRUIPanel.cs
│   │       ├── UIComponentFactory.cs
│   │       ├── LoginUI.cs
│   │       ├── HomeUI.cs
│   │       ├── SettingsUI.cs
│   │       ├── NavigationUI.cs
│   │       ├── EmergencyUI.cs
│   │       ├── DetectionUI.cs
│   │       ├── VoiceCommandUI.cs
│   │       └── ...
│   ├── Prefabs/          # Reusable UI prefabs
│   ├── Materials/        # UI materials
│   ├── Fonts/            # TextMeshPro fonts
│   └── Resources/        # Runtime-loaded assets
├── Packages/
│   └── manifest.json     # Package dependencies
└── ProjectSettings/      # Unity project settings

Backend Configuration

The app connects to the NewSight backend hosted on Render.com:

Base URL: https://cis4398-project-newsight-backend.onrender.com

REST Endpoints

Endpoint	Purpose
/auth/login	User login
/auth/signup	User registration
/voice/transcribe	Voice to text
/voice/wake-word	Wake word detection
/emergency_alert/{id}	Emergency SOS
/detect/objects	Object detection
/detect/text	OCR text reading
/detect/faces	Face recognition
/detect/colors	Color detection
/detect/asl	ASL translation
/navigation/directions	Get directions

WebSocket Endpoints

Endpoint	Purpose
/ws/verify	Auth verification
/location/ws	Real-time location
/navigation/ws	Navigation updates
/ws/camera	Camera frame streaming

UI Design System

The UI matches the original Android app design:

Colors

Name	Hex	Usage
Primary	#0b61da	Buttons, accents
Background	#f8f9fa	Screen backgrounds
Foreground	#15181d	Primary text
Card	#ffffff	Card backgrounds
Surface1	#f2f4f6	Input backgrounds
Surface2	#e6eaee	Secondary surfaces
MutedForeground	#5b6470	Secondary text
Red	#ef4042	Emergency, errors
Green	#3bd8ad	Success states

Typography

• Title: 28sp bold
• Header: 32sp bold
• Normal: 16sp
• Small: 14sp
• Tiny: 12sp

Building for Quest 3

1. Connect Quest 3 via USB (Developer Mode enabled)
2. Go to File > Build Settings
3. Ensure Android platform is selected
4. Click Build and Run

Or for APK:

1. Click Build
2. Choose save location
3. Install APK via adb install NewSight.apk

Features

Home Screen

• Rewards & points display
• Voice search bar
• Feature grid:
  • Emergency SOS
  • Navigation
  • Read Text (OCR)
  • Identify Objects
  • Faces Recognition
  • ASL Translation
  • Color Detection

Navigation

• Enter destination
• Get turn-by-turn directions
• Real-time location updates via WebSocket

Emergency

• One-tap SOS button
• 5-second countdown (cancelable)
• Sends location to trusted contacts

Camera Features

• Uses Quest 3 passthrough cameras
• Sends frames to backend for processing
• Displays detection results with TTS

Voice Commands

• Press and hold to record
• Sends to backend for transcription
• Processes commands like "Navigate to...", "What do you see?", etc.

Controller Mapping

Button	Action
Trigger	Select/Click
B Button	Back
Thumbstick	Scroll

Troubleshooting

"Package not found" errors

• Check internet connection
• Try Assets > Reimport All
• Manually add packages via Package Manager

Passthrough not working

• Ensure Quest 3 is selected as target device
• Enable passthrough in OVR Manager settings
• Check Quest 3 has latest firmware

UI not appearing

• Check Canvas render mode is World Space
• Verify Canvas scale is 0.001
• Ensure UI is positioned in front of camera

Backend connection issues

• Check internet connectivity on Quest 3
• Verify backend is running on Render.com
• Check console for specific error messages

License

© 2024 NewSight Team. All rights reserved.