Implement QOI Compression for Raw RGB Data

[samuelm2]

🎯 Problem Statement

Currently, raw RGB camera data is stored as uncompressed YUV files (.yuv), which results in very large file sizes. A typical 1-3 minute capture session can generate several gigabytes of data, making it:

• Slow to export/transfer from the Quest device
• Challenging to store multiple sessions on device
• Inefficient for bandwidth when uploading to cloud services

Current Data Flow:

Camera (YUV_420_888) → Direct write to .yuv file → Large file sizes

Location: QuestCameraLib/app/src/main/java/com/samusynth/questcamera/io/ImageReaderSurfaceProvider.kt

---

💡 Proposed Solution

Implement QOI (Quite OK Image Format) compression at capture time on the Quest side to reduce file size by ~50% with negligible quality loss.

QOI Benefits:

• Fast: Extremely fast encode/decode (faster than PNG, comparable to raw writes)
• Lossless: No quality degradation
• Simple: Pure Kotlin library available on Maven Central - no native code needed
• Efficient: Typical compression ratio of 2x for real-world images
• Mobile-friendly: Low CPU overhead, perfect for Quest 3 real-time encoding

Proposed Data Flow:

Camera (YUV_420_888) → Convert to RGB → QOI Encode → Write .qoi file → 80% smaller

---

📝 Current Implementation (Needs Modification)

Image Capture Code

The image data is currently saved in ImageReaderSurfaceProvider.kt:

// Current implementation (lines 168-178)
// Save image data (we only reach here if capture was signaled)
val fileName = "${computeUnixTime(image.timestamp)}.yuv"
val file = File(directory, fileName)

saveExecutor.execute {
    try {
        BufferedOutputStream(FileOutputStream(file)).use { it.write(data) }
    } catch (e: Exception) {
        e.printStackTrace()
    }
}

File: QuestCameraLib/app/src/main/java/com/samusynth/questcamera/io/ImageReaderSurfaceProvider.kt

Data Format Details

The YUV data is processed through dumpImageUnsafe() method around line 189, which extracts raw bytes from Android's Image object:

fun dumpImageUnsafe(image: Image, reusedBuffer: ByteArray): ByteArray {
    val requiredSize = calculateDumpBufferSize(image)
    val outputBuffer = if (reusedBuffer.size >= requiredSize) {
        // Buffer reuse logic...

---

🔨 Implementation Plan

Phase 1: Add QOI Library (Kotlin/Android)

Good news: Mature Kotlin QOI library already exists!

Use qoi-kotlin library (Recommended):

1. Add QOI dependency to Android project:

// QuestCameraLib/app/build.gradle.kts
dependencies {
    // Add qoi-kotlin from Maven Central
    implementation("io.github.mzgreen:qoi-kotlin:1.0.1")
}

This is a Kotlin Multiplatform library with:

• ✅ MIT licensed
• ✅ Published on Maven Central
• ✅ Uses Okio for efficient file I/O
• ✅ Well-tested with QOI reference images
• ✅ Both encoder and decoder support

Phase 2: Update Kotlin Code

Option A: Using qoi-kotlin library (Recommended)

Modify ImageReaderSurfaceProvider.kt:

import com.github.mzgreen.qoi.QOIImage
import com.github.mzgreen.qoi.QOIWriter
import okio.buffer
import okio.sink
import java.io.File

class ImageReaderSurfaceProvider(
    private val width: Int,
    private val height: Int,
    imageFileDirPath: String,
    formatInfoFilePath: String,
    private val bufferPoolSize: Int = 5,
    private val useQoiCompression: Boolean = true  // NEW: Feature flag
): ISurfaceProvider, AutoCloseable {
    
    private val qoiWriter = QOIWriter() // Initialize QOI writer
    
    private fun processImage(image: Image) {
        // ... existing code ...
        
        val data = dumpImageUnsafe(image, buffer)
        bufferPool[nextBufferPoolIndex] = data
        latestBufferPoolIndex = nextBufferPoolIndex
        
        // MODIFIED: Save image data with QOI compression
        val timestamp = computeUnixTime(image.timestamp)
        
        saveExecutor.execute {
            try {
                if (useQoiCompression) {
                    // Convert YUV to RGB
                    val rgb = convertYuv420ToRgb(data, width, height)
                    
                    // Create QOI image
                    val qoiImage = QOIImage(
                        width = width,
                        height = height,
                        channels = 3, // RGB
                        colorspace = 1, // sRGB
                        pixels = rgb
                    )
                    
                    // Write to file using qoi-kotlin library
                    val fileName = "$timestamp.qoi"
                    val file = File(directory, fileName)
                    file.sink().buffer().use { sink ->
                        qoiWriter.write(qoiImage, sink)
                    }
                    
                    val qoiSize = file.length()
                    Log.d(TAG, "Saved QOI: ${file.name}, " +
                          "original: ${data.size}, compressed: $qoiSize, " +
                          "ratio: ${String.format("%.1f", 100.0 * qoiSize / data.size)}%")
                } else {
                    // Original YUV path (for compatibility)
                    val fileName = "$timestamp.yuv"
                    val file = File(directory, fileName)
                    BufferedOutputStream(FileOutputStream(file)).use { 
                        it.write(data) 
                    }
                }
            } catch (e: Exception) {
                Log.e(TAG, "Failed to save image: ${e.message}")
                e.printStackTrace()
            }
        }
    }
    
    /**
     * Convert YUV420 (NV12) to RGB for QOI encoding
     */
    private fun convertYuv420ToRgb(yuv: ByteArray, width: Int, height: Int): ByteArray {
        val rgb = ByteArray(width * height * 3)
        val frameSize = width * height
        val uvStart = frameSize
        
        for (j in 0 until height) {
            for (i in 0 until width) {
                val yIndex = j * width + i
                val uvIndex = uvStart + (j / 2) * width + (i and 1.inv())
                
                val y = (yuv[yIndex].toInt() and 0xff) - 16
                val u = (yuv[uvIndex].toInt() and 0xff) - 128
                val v = (yuv[uvIndex + 1].toInt() and 0xff) - 128
                
                val r = (1.164f * y + 1.596f * v).toInt().coerceIn(0, 255)
                val g = (1.164f * y - 0.392f * u - 0.813f * v).toInt().coerceIn(0, 255)
                val b = (1.164f * y + 2.017f * u).toInt().coerceIn(0, 255)
                
                val rgbIndex = yIndex * 3
                rgb[rgbIndex] = r.toByte()
                rgb[rgbIndex + 1] = g.toByte()
                rgb[rgbIndex + 2] = b.toByte()
            }
        }
        
        return rgb
    }
}

That's it! Just add one dependency and you're done. No JNI, no native code, just a clean Kotlin library.

Option B: Using custom implementation (if you prefer no dependencies)

If you want to avoid external dependencies, you could implement the QOI compression yourself. Its a very simple format.

// Use custom encoder instead:
val qoiData = QoiEncoder.encodeYuvToQoi(data, width, height)
val fileName = "$timestamp.qoi"
val file = File(directory, fileName)
BufferedOutputStream(FileOutputStream(file)).use { 
    it.write(qoiData) 
}

Phase 3: Update Processing Pipeline

Update Python processing code to handle QOI files:

File: quest-3d-reconstruction/scripts/dataio/image_data_io.py

import qoi  # pip install qoi

class ImageDataIO:
    def load_image(self, side: Side, timestamp: int) -> np.ndarray:
        """Load image, supporting both QOI and YUV formats"""
        base_dir = self.image_path_config.get_image_dir(side=side)
        
        # Try QOI first (new format)
        qoi_path = base_dir / f'{timestamp}.qoi'
        if qoi_path.exists():
            return qoi.read(str(qoi_path))
        
        # Fallback to YUV (legacy format)
        yuv_path = base_dir / f'{timestamp}.yuv'
        if yuv_path.exists():
            raw_data = np.fromfile(yuv_path, dtype=np.uint8)
            format_info = self.load_image_format_info(side=side)
            return convert_yuv420_888_to_bgr(raw_data, format_info)
        
        raise FileNotFoundError(f"No image found for {side}/{timestamp}")

Update file detection in path config:

# quest-3d-reconstruction/scripts/config/project_path_config.py
class ImagePathConfig:
    def get_image_paths(self, side: Side) -> list[Path]:
        """Get all image paths, preferring QOI over YUV"""
        image_dir = self.get_image_dir(side=side)
        
        # Get QOI files
        qoi_files = sorted(image_dir.glob('*.qoi'))
        if qoi_files:
            return qoi_files
        
        # Fallback to YUV
        return sorted(image_dir.glob('*.yuv'))

---

✅ Acceptance Criteria

• QOI compression is implemented on Quest side using qoi-kotlin library
• Dependency added to build.gradle.kts: io.github.mzgreen:qoi-kotlin:1.0.1
• File sizes are reduced by ~75-85% compared to raw YUV
• Encoding overhead is < 10ms per frame on Quest 3 (doesn't impact capture FPS)
• Files are saved with .qoi extension instead of .yuv
• Python processing pipeline can read .qoi files
• Backward compatibility: Python code can still read old .yuv files
• Feature flag allows toggling between QOI and raw YUV (for debugging)
• Update documentation in README with new file format
• Include performance metrics in logs (compression ratio, encode time)
• No JNI/NDK configuration required - pure Kotlin implementation

---

📚 Additional Resources

• QOI Specification: https://qoiformat.org/
• QOI Spec PDF: https://qoiformat.org/qoi-specification.pdf
• QOI Original C Library: https://github.com/phoboslab/qoi
• Kotlin/Java QOI Libraries:
  • qoi-kotlin (Recommended): https://github.com/mzgreen/qoi-kotlin
    • Maven: io.github.mzgreen:qoi-kotlin:1.0.1
    • Kotlin Multiplatform, MIT licensed, well-tested
  • https://github.com/saharNooby/qoi-java (Alternative)
  • Multiple C# implementations available on NuGet
• Python QOI Library: pip install qoi (https://pypi.org/project/qoi/)
• QOI Performance Benchmarks: Shows 3-4x faster encoding than PNG with similar compression

Why Pure Kotlin Works:

• QOI is designed to be simple to implement (original C is ~300 lines)
• No native code needed - pure JVM/Kotlin implementation
• qoi-kotlin is a mature, tested library on Maven Central
• Easier to debug and maintain than JNI
• Faster development time
• Uses efficient Okio for I/O operations

---

🧪 Testing Plan

1. Library Integration Test:

   • Verify qoi-kotlin dependency resolves correctly
   • Build Android project successfully
   • No runtime errors on Quest device

2. Compression Ratio Test:

   • Capture a 1-minute session with QOI
   • Compare file sizes with equivalent YUV session
   • Verify ~80% reduction

3. Performance Test:

   • Measure encoding time per frame on Quest 3
   • Ensure it doesn't drop below target 3 FPS
   • Profile CPU usage during capture
   • Compare with raw YUV write performance

4. Quality Test:

   • Reconstruct scene from QOI-compressed data
   • Compare 3D reconstruction quality with YUV baseline
   • Verify no visible artifacts or degradation (QOI is lossless)

5. Compatibility Test:

   • Process old YUV sessions with new Python code
   • Process new QOI sessions with updated Python code
   • Verify both work correctly

6. Decode Test (Python side):

   • Install Python QOI library: pip install qoi
   • Verify QOI files can be decoded correctly
   • Compare decoded RGB with original YUV conversion

---

💭 Alternative Approaches Considered

1. PNG Compression:

   • ❌ Too slow for real-time encoding on Quest (30-50ms per frame)
   • ✅ QOI is 3-4x faster

2. JPEG Compression:

   • ❌ Lossy compression, not suitable for 3D reconstruction
   • ❌ Artifacts affect feature detection in reconstruction
   • ✅ QOI is lossless

3. Server-side compression:

   • ❌ Still requires large file transfer from Quest
   • ❌ Slower export times
   • ✅ QOI compresses at capture time

4. Hardware HEVC/H.265:

   • ❌ Lossy compression
   • ❌ Requires decoding infrastructure
   • ✅ QOI is simpler and lossless

---

🔗 Related Files

• QuestCameraLib/app/src/main/java/com/samusynth/questcamera/io/ImageReaderSurfaceProvider.kt (lines 168-178)
• quest-3d-reconstruction/scripts/dataio/image_data_io.py (lines 39-42)
• quest-3d-reconstruction/scripts/processing/yuv_conversion/convert_yuv_dir.py
• README.md (lines 136-140) - Documentation update needed