Predicto — Real-Time Emotion-Aware Music Discovery Platform

An intelligent web platform bridging client-side edge computer vision and cloud media streaming — translating real-time facial expressions into curated, mood-resonant playlists.

---

Table of Contents

• Executive Summary
• System Architecture
• Edge Computer Vision & Inference Pipeline
• Media Ingestion & Token Normalization Pipeline
• Web Audio Synchronization & Playback Engine
• Authentication, Session Security & Revocation
• Domain Model & Data Architecture
• API Architecture & Contract Specifications
• Frontend Architecture & Design Systems
• Technology Stack
• Repository Organization
• Implementation Status & Roadmap
• License & Attribution

---

Executive Summary

Predicto addresses the friction of manual music discovery by transforming a user's live facial expression into an instant, mood-tailored listening experience. Rather than requiring users to manually browse genres or complete subjective mood surveys, Predicto evaluates continuous facial landmark regressions directly within the client browser at display refresh rates.

Core Engineering Distinctions

• Edge Computer Vision via WebAssembly: Executes facial landmark and 52+ blendshape regressions locally using Google's MediaPipe Tasks Vision WebAssembly runtime. Video frames are processed in volatile client memory and never leave the device, eliminating streaming bandwidth bottlenecks and preserving user privacy.
• Deterministic Heuristic Expression Engine: Analyzes five composite geometric signals (smile intensity, jaw openness, brow movement, and frown depth) through an auditable, sub-frame decision chain that resolves expressions into indexed mood buckets.
• Automated Media Parsing & Ingestion: Implements an in-memory binary processing pipeline (node-id3) that parses MP3 buffers, extracts embedded APIC album artwork, dispatches parallel cloud uploads to ImageKit CDN, and sanitizes titles via an 8-stage regex normalization engine.
• Stateless Authentication with Distributed Revocation: Couples HTTP-only JWT sessions and salted bcrypt credential hashing with asynchronous Redis-backed token revocation on logout.
• Decoupled CDN Streaming: Offloads binary audio delivery from the Node.js application server directly to ImageKit's global edge network, maintaining high backend throughput while streaming to an event-driven HTML5 audio controller.

---

System Architecture

Predicto implements a multi-tier decoupled topology spanning client-side edge inference, authenticated REST services, distributed persistence, and global CDN delivery.

graph TD
    subgraph Client ["Client Tier · React 19 + Vite 7"]
        UI["Web UI (Outfit & Inter Typography)"]
        Cam["Device Camera Stream"]
        MP["MediaPipe Face Landmarker<br/>(WASM · VIDEO Mode)"]
        Router["React Router 7<br/>(Declarative Route Guards)"]
        AuthCtx["Auth Context & Custom Hook"]
        Player["HTML5 Audio Engine<br/>(Timeline Scrubbing & Queue)"]
    end

    subgraph API ["Backend Gateway · Node.js & Express 5"]
        CORS["CORS & Cookie Parser"]
        AuthMW["Auth Middleware<br/>(JWT Verification)"]
        MulterMW["Multer In-Memory Storage<br/>(20MB Payload Ceiling)"]
        AuthCtrl["Auth Controller"]
        SongCtrl["Song Controller &<br/>Regex Normalization Engine"]
        ID3["node-id3 Binary Parser"]
        StoreSvc["Storage Service Abstraction"]
    end

    subgraph Data ["Persistence & Delivery Tier"]
        Mongo[("MongoDB<br/>Users & Songs Collections")]
        Redis[("Redis<br/>Token Revocation Blacklist")]
        IK["ImageKit Global CDN<br/>(Binary Audio & Cover Artwork)"]
    end

    Cam --> MP
    MP --> UI
    UI --> Router
    Router --> AuthCtx
    Router --> Player
    AuthCtx -->|"HTTP-Only Cookies / Axios"| CORS
    Player -.->|"Direct Audio Byte Stream"| IK

    CORS --> AuthMW
    AuthMW --> AuthCtrl
    AuthMW --> MulterMW
    MulterMW --> ID3
    ID3 --> SongCtrl
    AuthCtrl -->|"Bcrypt Hash & Tokens"| Mongo
    AuthCtrl -->|"Blacklist on Logout"| Redis
    SongCtrl -->|"Metadata & Mood Key"| Mongo
    SongCtrl --> StoreSvc
    StoreSvc -->|"Parallel Upload"| IK

Loading

Architectural Layer Breakdown

Architectural Tier	Subsystem	Responsibility & Implementation Patterns
Client Presentation	React 19 + Sass	Single Page Application (SPA) structured around domain-driven feature slices, GPU-accelerated glassmorphic styling, and responsive HUD layouts.
Edge Vision Inference	@mediapipe/tasks-vision	In-browser Face Landmarker executing in WebAssembly via requestAnimationFrame, extracting per-frame facial blendshape scores without server-side compute.
Navigation & Guarding	React Router 7	Declarative routing with asynchronous session verification wrappers (<Protected>), preventing layout shifts or unauthorized render flashes.
API Gateway	Express 5	Modular routing with strict origin-validated CORS, URL-encoded/JSON payload parsers, and router-level authentication middleware chains.
Authentication & Revocation	JWT + bcrypt + Redis	Stateless bearer token generation (1-hour expiration); salted password hashing (10 salt rounds); distributed token revocation tracking in Redis.
Media Ingestion Engine	Multer + node-id3	Synchronous binary buffer parsing in system RAM (20 MB ceiling, zero disk I/O); APIC album cover extraction; concurrent cloud dispatch.
Persistence Layer	MongoDB + Mongoose 9	Schema-enforced document storage for user accounts and song records with categorical mood enum validation.
Content Delivery Network	ImageKit CDN	Low-latency edge caching and byte streaming directly to browser audio elements, isolating the API tier from media traffic.

---

Edge Computer Vision & Inference Pipeline

The vision pipeline processes live camera input entirely within the client runtime, achieving real-time inference without network round-trips.

sequenceDiagram
    autonumber
    actor User
    participant Browser as Browser UI / Video Element
    participant MP as MediaPipe WASM Runtime
    participant Classifier as Heuristic Expression Engine
    participant Backend as Express API Gateway
    participant DB as MongoDB
    participant CDN as ImageKit Edge CDN

    User->>Browser: Enters /predict (Guarded Route)
    Browser->>Browser: Captures camera stream via getUserMedia
    loop Synchronized with requestAnimationFrame
        Browser->>MP: detectForVideo(videoElement, timestamp)
        MP-->>Browser: 52+ Facial Blendshape Coefficients
        Browser->>Classifier: Aggregate composite metrics (Smile, Jaw, Brow, Frown)
        Classifier-->>Browser: Update reactive emotion state & resolved mood key
    end
    User->>Browser: Selects "Find Music for Mood"
    Browser->>Browser: Navigate to /songsbymood?mood={key}
    Browser->>Backend: GET /api/songs?mood={key} (Cookie Transport)
    Backend->>DB: Song.find({ mood: key })
    DB-->>Backend: Matched Song Documents
    Backend-->>Browser: 200 OK with track list (CDN URLs & metadata)
    Browser->>CDN: Request binary audio stream & poster assets
    CDN-->>Browser: Direct edge media delivery
    Browser->>User: Audio playback with scrubbed timeline & active queue

Loading

1. WebAssembly Runtime Initialization

• Binary Resolution: FilesetResolver.forVisionTasks dynamically loads the WebAssembly runtime from CDN mirrors.
• Model Graph: Google's face_landmarker.task float16 model initializes in VIDEO mode with outputFaceBlendshapes: true and numFaces: 1.
• Stream Binding: navigator.mediaDevices.getUserMedia({ video: true }) streams live video into a CSS-mirrored <video> element (scaleX(-1) for natural mirror feedback).
• Resource Cleanup: On route unmount, cancelAnimationFrame halts the inference cycle and MediaStreamTrack.stop() explicitly releases hardware camera access.

2. Mathematical Blendshape Aggregation

MediaPipe outputs 52 continuous blendshape coefficients ($0.0$ to $1.0$). Predicto aggregates them into five composite geometric metrics:

Composite Metric	Source Blendshapes	Mathematical Formulation
Smile Intensity ($S$)	mouthSmileLeft, mouthSmileRight	$S = \max(S_{\text{left}}, S_{\text{right}})$
Jaw Openness ($J$)	jawOpen	$J = \text{score}$
Brow Raised ($B_{\text{up}}$)	browOuterUpLeft, browOuterUpRight	$B_{\text{up}} = \max(B_{\text{upLeft}}, B_{\text{upRight}})$
Brow Pressed ($B_{\text{down}}$)	browDownLeft, browDownRight	$B_{\text{down}} = \max(B_{\text{downLeft}}, B_{\text{downRight}})$
Frown Depth ($F$)	mouthFrownLeft, mouthFrownRight	$F = \max(F_{\text{left}}, F_{\text{right}})$

3. Priority Decision Hierarchy

The composite metrics are evaluated through an ordered decision chain:

Evaluation Order	Condition	Output Expression	Resolved Mood Key
1	$S > 0.75 \land J > 0.30$	😁 Very Happy	very happy
2	$S > 0.45$	😊 Happy	happy
3	$J > 0.50 \land B_{\text{up}} > 0.38$	😲 Surprise	surprised
4	$B_{\text{down}} > 0.45 \land F > 0.30$	😠 Angry	surprised
5	$F > 0.0045$	😢 Sad	sad
6	Fallback	😐 Neutral	neutral

---

Media Ingestion & Token Normalization Pipeline

The backend features an automated pipeline for administrative track uploads (POST /api/songs/upload):

flowchart TD
    A["Binary Audio Buffer (MP3 ≤ 20MB)"] --> B["Multer In-Memory Storage"]
    B --> C["node-id3 Tag Extraction"]
    C --> D["Extract ID3 Title Tag"]
    C --> E{"Embedded APIC Artwork?"}
    
    E -->|Present| F["Extract Poster Image Buffer"]
    E -->|Absent| G["Assign /default_poster.png Reference"]
    
    D --> H["8-Stage Regex Sanitization Engine<br/>• Strip BOM & Zero-Width Markers<br/>• Normalize Underscores to Word Boundaries<br/>• Strip Audio Bitrate & Quality Suffixes<br/>• Purge Bracketed Video/Audio Descriptors<br/>• Remove Standalone Marketing Keywords<br/>• Collapse Redundant Delimiters<br/>• Trim Outer Boundary Punctuation"]
    
    B --> I["ImageKit Audio Dispatch<br/>(/Predicto/songs)"]
    F --> J["ImageKit Artwork Dispatch<br/>(/Predicto/posters)"]
    
    I & J --> K["Promise.all Parallel Resolution"]
    K & H & G --> L["Persist Song Document in MongoDB<br/>{ title, url, posterUrl, mood }"]

Loading

8-Stage Title Sanitization Engine (cleanTitle)

Audio files sourced from diverse libraries often feature erratic naming artifacts. The sanitization routine executes eight sequential transformations:

1. BOM & Zero-Width Stripping: Removes non-printing Unicode characters ([\u200B-\u200D\uFEFF]) that distort string indexes.
2. Underscore Boundary Normalization: Replaces double underscores with " - " and single underscores with whitespace.
3. Quality Suffix Elimination: Identifies and removes bitrate markers (e.g., 320k, 320kbps, MP3_320K).
4. Bracketed Metadata Purging: Strips enclosed marketing tokens (e.g., [Official Video], (Lyrical Song), [HD], (4K)).
5. Standalone Keyword Removal: Cleans unbracketed media tags (e.g., official music video, video song).
6. Delimiter Deduplication: Collapses multi-hyphen sequences (---) into single hyphens.
7. Separator Uniformity: Normalizes chaotic delimiter spacing (e.g., " - - " or " // ") into clean " - " separators.
8. Boundary Trimming: Removes extraneous punctuation and collapses multi-space runs into normalized single spacing.

---

Web Audio Synchronization & Playback Engine

The music player (SongByMood.jsx) encapsulates audio playback through React state bindings directly coupled to the native HTML5 <audio> element:

• Reactive State Bindings: Real-time synchronization of currentTime, duration, and isPlaying state across the UI.
• Non-Blocking Timeline Scrubbing: Controlled <input type="range"> timeline slider providing interactive seek capabilities with dynamic $M:SS$ formatting.
• Queue Navigation & Circular Indexing: Seamless previous/next track selection with modulo boundary wrapping ($(i \pm 1) \pmod N$).
• Continuous Auto-Advancement: Native subscription to the audio element's onEnded lifecycle event triggers automatic queue progression.
• Resilient Poster Art Fallbacks: Client-side onError handlers replace missing or unreachable CDN artwork with /default_poster.png.
• GPU-Accelerated Visual Feedback: Dynamic CSS class assignment (.playing) activates glowing aura animations and live queue indicators.

---

Authentication, Session Security & Revocation

Security Layer	Implementation Mechanism	Architectural Justification
Password Storage	bcrypt (10 salt rounds)	Adaptive work-factor hashing resisting rainbow table and brute-force attacks.
Token Delivery	HTTP-Only, Secure Cookies	Eliminates client-side JavaScript access to JWTs, preventing Cross-Site Scripting (XSS) token exfiltration.
Client Route Guarding	React Router Guard (<Protected>)	Evaluates hydrated authentication state, displaying loading states during verification and preventing unauthorized render flashes.
API Route Authorization	verifyToken Middleware	Router-level middleware verifying JWT signatures before granting access to song catalog endpoints.
CORS Policy	Explicit Origin Allowlist	Restricts cross-origin requests to configured domains (FRONTEND_URL, localhost, and predicto.skramizraza.tech) with credentials: true.
Buffer Protection	Multer In-Memory Limit	Restricts incoming multipart audio buffers to 20 MB in RAM, mitigating disk exhaustion attacks.
Distributed Revocation	Redis Blacklist Record	On logout, tokens are persisted in Redis with an explicit 3,600-second TTL (EX 3600).

Architectural Note on Token Revocation: On user logout, req.cookies.token is written to Redis as "Blacklisted" with a 1-hour TTL matching the token's lifetime. The current verifyToken middleware validates cryptographic integrity and expiry. Extending the middleware to query Redis on every request enables immediate distributed revocation across multi-instance clusters.

---

Domain Model & Data Architecture

Data persistence is managed via Mongoose schemas enforcing structural integrity at the database layer.

User Entity (users Collection)

Field	Type	Validation / Constraints	Indexing & Projection Behavior
username	String	Required (true)	Unique index; primary account handle
email	String	Required (true)	Unique index; account recovery and login
password	String	Required (true)	select: false (excluded from query projections to prevent accidental exposure)

Song Entity (songs Collection)

Field	Type	Validation / Constraints	Description
title	String	Required (true)	Sanitized track title generated by the normalization engine
url	String	Required (true)	Direct ImageKit CDN URL for binary audio streaming
posterUrl	String	Required (true)	Direct ImageKit CDN URL for extracted album artwork
mood	String	enum: ["neutral", "", "happy", "sad", "surprised", "very happy"]	Database-enforced categorical mood taxonomy

---

API Architecture & Contract Specifications

All API routes are prefixed under /api.

Base Health Check

• GET / — Public endpoint returning service status:

  { "message": "Welcome to the Predicto" }

Authentication Endpoints (/api/auth)

Endpoint	Method	Access Level	Request Payload	Response Contract
/api/auth/register	POST	Public	{ username, email, password }	201 Created Sets token cookie { message, user }
/api/auth/login	POST	Public	{ email | username, password }	200 OK Sets token cookie { message, user }
/api/auth/profile	GET	Private	Cookie: token=<JWT>	200 OK { message, user }
/api/auth/logout	GET	Private	Cookie: token=<JWT>	200 OK Clears token cookie Writes token to Redis blacklist { message }

Song Catalog Endpoints (/api/songs)

Note: Router-level verifyToken middleware applies to all /api/songs routes.

Endpoint	Method	Access Level	Parameters / Form	Response Contract
/api/songs	GET	Private	Query: ?mood=<mood_key>	200 OK { message, songs: [...] }
/api/songs/upload	POST	Private	Multipart: song (MP3 file, $\le 20\text{MB}$) Field: mood (String enum)	201 Created { message, song: {...} }

---

Frontend Architecture & Design Systems

The frontend client is engineered around four domain-driven feature modules:

• Auth: Manages registration, login, the <Protected> route guard, global session hydration via AuthContext, and credentialed Axios services.
• Expression: Houses the emotion detection view, the mirrored camera viewport, and the emotion.js WebAssembly inference and heuristic classification engine.
• Home: Contains the landing showcase, mood taxonomy catalog (songCatalog.js), and expression-to-mood mapping logic.
• Song: Houses the interactive audio player, custom useSong hook, SongContext provider, and catalog retrieval services.

Design System Architecture

• Visual Palette: Deep void canvas (#0D1B2A), translucent card backgrounds (#1B3A5C), electric violet accent (#9D4EDD), and crimson action indicators (#BF092F).
• Typography Tokens: Display headings set in Outfit (700–800 weight); body text and timestamps set in Inter (400–600 weight).
• Glassmorphic Surface Design: Custom Sass mixins providing backdrop-filter: blur(...) with translucent borders, radial background auras, and HUD-inspired grid textures.
• Responsive Layout Adaptation: Dual-column desktop HUD collapsing into a single-column stacked view on viewport widths $\le 992\text{px}$.

---

Technology Stack

Layer	Technology	Version	Purpose in Predicto
Client Presentation	React	^19.2.0	Declarative UI, lifecycle hooks, and context state management
Tooling & Bundling	Vite	^7.2.4	Development server and production bundling
Client-Side Routing	React Router DOM	^7.18.2	Declarative routing and guarded component wrappers
Edge Computer Vision	MediaPipe Tasks Vision	^0.10.35	Client-side Face Landmarker and blendshape regression
CSS Preprocessor	Sass (Dart Sass)	^1.89.2	Design tokens, variables, and feature-scoped stylesheets
HTTP Communication	Axios	^1.19.0	API client configured with automatic credential transport
Runtime Environment	Node.js	>= 18	Server-side JavaScript runtime
API Framework	Express	^5.2.1	REST API routing and middleware pipelines
Primary Database	MongoDB	Local / Cloud	Document store for user credentials and song catalog
Object Data Modeling	Mongoose	^9.8.0	Schema validation and query construction
Distributed Caching	Redis via ioredis	^5.11.1	Fast key-value store for session revocation tracking
Session Security	JSON Web Tokens	^9.0.3	Cryptographically signed bearer token assertions
Credential Hashing	bcrypt	^6.0.0	Adaptive salted password hashing
Multipart Streaming	Multer	^2.2.0	In-memory audio buffer ingestion (20 MB ceiling)
Binary ID3 Parsing	node-id3	^0.2.9	Synchronous extraction of audio metadata and album artwork
Media CDN Delivery	@imagekit/nodejs	^7.10.0	Programmatic cloud media storage and global CDN streaming

---

Repository Organization

Predicto-main/
├── Backend/
│   ├── server.js                        # Server entry point & database initialization
│   ├── package.json                     # Server dependencies & scripts
│   └── src/
│       ├── app.js                       # Express configuration, CORS & route mounts
│       ├── config/
│       │   ├── database.js              # Mongoose MongoDB connection handler
│       │   └── cache.js                 # Redis client instance via ioredis
│       ├── controllers/
│       │   ├── auth.controller.js       # Register, login, profile & logout handlers
│       │   └── song.controller.js       # Track upload, ID3 parsing & title cleaning
│       ├── middlewares/
│       │   ├── auth.middleware.js       # JWT cookie verification middleware
│       │   └── upload.middleware.js     # Multer 20MB in-memory storage config
│       ├── models/
│       │   ├── user.model.js            # User schema with select: false password protection
│       │   └── song.model.js            # Song schema with mood enum constraint
│       ├── routes/
│       │   ├── auth.routes.js           # Public & private auth route declarations
│       │   └── song.routes.js           # Guarded song routes (verifyToken applied)
│       └── services/
│           └── storage.service.js       # ImageKit Node SDK abstraction layer
│
└── Frontend/
    ├── index.html                       # Document template with Outfit & Inter typography
    ├── vite.config.js                   # Vite bundler configuration
    ├── package.json                     # Frontend dependencies & scripts
    ├── public/
    │   ├── default_poster.png           # Fallback album cover asset
    │   └── vite.svg                     # Application favicon
    └── src/
        ├── main.jsx                     # React root bootstrap
        ├── App.jsx                      # App root with AuthProvider & RouterProvider
        ├── app.routes.jsx               # Declarative route tree with protected guards
        ├── styles/
        │   └── main.scss                # Global resets & scrollbar theming
        └── Features/
            ├── Auth/
            │   ├── auth.context.jsx     # User session state & silent profile hydration
            │   ├── components/Protected.jsx # Route guarding wrapper
            │   ├── pages/               # Login & Register views
            │   ├── services/auth.api.js # Credentialed Axios auth endpoints
            │   └── styles/Auth.scss     # Authentication view styling
            ├── Expression/
            │   ├── pages/EmotionDetectorPage.jsx # Camera viewport & live classification
            │   ├── components/Expression.jsx     # Video element presentation
            │   ├── utils/emotion.js     # MediaPipe init, loop & threshold engine
            │   └── styles/Expression.scss
            ├── Home/
            │   ├── pages/Home.jsx       # Landing showcase & marketing presentation
            │   ├── domain/songCatalog.js# Mood catalog taxonomy & emotion mapper
            │   └── styles/              # Design tokens (_variables, _mixins) & styles
            └── Song/
                ├── pages/SongByMood.jsx # Full-featured audio player & queue interface
                ├── song.context.jsx     # Active song state management
                ├── services/song.api.js # Credentialed song retrieval API
                └── styles/Song.scss     # Player controls, progress bar & glow styling

---

Implementation Status & Roadmap

Implemented Architecture

• Client-side edge facial landmark and blendshape regression via MediaPipe Tasks Vision.
• Composite 5-signal geometric reduction and deterministic threshold expression classifier.
• Declarative route protection with silent profile hydration on initial load.
• Cookie-based JWT authentication with salted bcrypt credential encryption.
• Distributed Redis-based token revocation record storage on logout.
• In-memory multipart audio parsing with synchronous ID3 tag and APIC artwork extraction.
• Parallel audio and artwork CDN ingestion via ImageKit Node SDK.
• 8-stage regex track title normalization and sanitization pipeline.
• Event-driven HTML5 audio player (timeline scrubbing, timecode formatting, queue wrap-around, auto-advance).
• Responsive glassmorphic HUD interface with GPU-accelerated glow animations.

Architectural Roadmap

• Request-Time Redis Revocation Verification: Enforce active Redis blacklist checks within verifyToken on every incoming API request for instantaneous multi-device session revocation.
• Refresh Token Rotation: Implement dual-token authentication (short-lived access token + rotating refresh token) to extend sessions securely without frequent re-authentications.
• Multi-Frame Temporal Smoothing: Apply a rolling-window average across consecutive blendshape frames to eliminate single-frame expression jitter.
• Continuous Mood Adaptation: Dynamically transition playlists when facial mood shifts over sustained listening intervals.
• Catalog Admin Dashboard: Dedicated administrative interface for audio uploads, category re-indexing, and catalog curation.

---

License & Attribution

This project is licensed under the GNU General Public License v3.0 — see the LICENSE file for complete license terms.

Engineered by Raza — Full-Stack & Systems Engineer

• Live Deployment: predicto.skramizraza.tech
• Core Focus: Edge computer vision, distributed backend services, cloud media streaming, and interactive web audio systems.

---

Predicto — where your face picks the playlist.