One chat interface. Web search. YouTube transcription. Document Q&A. Persistent memory. All powered by Nemotron Core AI.
Nemetron is a full-stack AI chat platform that routes every user message through an intelligent multi-layer pipeline before generating a final answer. You don't pick a tool — Nemetron figures out whether to search the web, scrape a page, transcribe a YouTube video, query your uploaded files, or answer from memory. One message. The right action. Every time.
Built with: Next.js 16 App Router · React 19 · TypeScript · Tailwind CSS v4 · Supabase (PostgreSQL + pgvector + Private Storage)
| Feature | Description |
|---|---|
| 🧠 Multi-Layer Intent Router | 6-layer orchestration pipeline classifies every prompt before any tool is invoked |
| 🌐 Real-Time Web Search | Dual provider (Tavily → SerpApi Google) with automatic fallback |
| 🎬 YouTube Video Search & Transcription | Search videos by topic or drop a raw YouTube URL — Nemetron transcribes it instantly |
| 📚 Document Q&A (RAG) | Upload PDFs, DOCX, TXT, MD, or JSON — Nemetron indexes them and answers questions from inside |
| 🔍 Hybrid Retrieval (Exact + FTS + Vector) | Reciprocal Rank Fusion combining exact match, full-text, and 384-dim local MiniLM embeddings |
| ⚡ Resilient AI Fallback Chain | Primary Nemotron Core → Backup Key → Groq llama-3.3-70b → Groq llama-3.1-8b-instant |
| 💾 Per-User Persistent Memory | Maintains conversation context across sessions at the per-user level |
| 🛡️ Rate Limiting & Security | Per-user prompt limits with rolling window enforcement and admin bypass |
| 📱 Mobile-First Chrome Layout | Dynamic viewport (100dvh) with keyboard-aware input positioning |
| 🎯 Workload Profile Router | Classifies the user into a professional, developer, student, or researcher profile to tailor response style |
| 🎙️ Voice Assistant (STT & TTS) | Real-time conversational voice mode using Groq Whisper (Speech-to-Text) and ElevenLabs (Text-to-Speech) with dynamic markdown stripping |
Every message you send flows through a 6-layer orchestration pipeline in milliseconds before a single token is generated. Here is the complete flow:
graph TD
%% --------------------------------
%% 1. Core Request & Security
%% --------------------------------
subgraph Input [" 🔐 Request Intake "]
Voice([" 🎙️ Voice Input "]) -->|" Groq Whisper STT "| User([" 👤 User Prompt "])
Text([" ⌨️ Text Input "]) --> User
User --> Security{" 🛡️ Security & Rate Limit Gate "}
Security -->|" ✗ Failed "| Reject([" ⛔ Reject / Countdown Timer "])
Security -->|" ✓ Passed "| L1{" Layer 1: Context Relevance Gate "}
end
%% --------------------------------
%% 2. Fast Direct Path & Identity
%% --------------------------------
L1 ==>|" Greetings / Direct Chat "| FastPath((" ⚡ FAST DIRECT PATH "))
FastPath ==> NIM_Pri
L1 ==>|" Identity Queries "| IdentityHub(((" 🪪 IDENTITY HUB ")))
IdentityHub -->|" Nemetron Self-Awareness Data "| NIM_Pri
%% URL Fast Bypass
L1 ==>|" Raw YouTube URL detected "| URLBypass(((" 🔗 URL FAST BYPASS ")))
URLBypass -->|" Clean URL extracted "| Transcribe
%% --------------------------------
%% 3. Advanced Orchestration Brain
%% --------------------------------
subgraph Brain [" 🧠 Nemetron Orchestration Brain "]
L1 -->|" Complex Prompt "| WPR{" Layer 2: Workload Profile Router\n(Gemini → Groq Fallback) "}
WPR -->|" Profile + Style Metadata "| L3[" Layer 3: Deterministic Supervisor "]
L3 -->|" Requires Clarification / Direct Action "| FastTrack[" Fast Track Output "]
FastTrack --> Aggregator
L3 -->|" Complex Intent "| L4{" Layer 4: Intent Router "}
L4 -->|" Classified Intent JSON "| L5{" Layer 5: Execution Planner "}
L3 -->|" Filters History "| Memory[(" 💾 Memory Manager ")]
L5 -->|" Builds Parallel Tool Graph "| L6[" Layer 6: Tool Executor "]
end
%% --------------------------------
%% 4. Parallel Execution Pipelines
%% --------------------------------
subgraph Execution [" ⚙️ Parallel Execution Pipelines "]
direction TB
subgraph Web [" 🌐 Web Search "]
L6 -->|" Query "| WebPri[" Tavily Search "]
WebPri -->|" Fallback "| WebSec[" SerpApi Google "]
end
subgraph Video [" 🎬 Video Search & Transcription "]
L6 -->|" Topic "| VidPri[" SerpApi YouTube "]
VidPri -->|" Fallback "| VidSec[" Tavily site:youtube "]
Transcribe[" Supadata Transcriber "] -->|" Fallback "| FireTr[" Firecrawl Scrape "]
end
subgraph RAG [" 📚 Hybrid RAG Pipeline "]
L6 -->|" Query "| HybridR[" Hybrid Retriever "]
HybridR --> ExactM[" Exact Term Match "]
HybridR --> FTS[" Full-Text Search (FTS) "]
HybridR --> VecS[" Local MiniLM\nvector(384) Cosine Search "]
ExactM & FTS & VecS --> RRF[" RRF Score Fusion "]
end
subgraph ReadURL [" 🔗 URL Reader "]
L6 -->|" URL "| Scrape[" Firecrawl Scraper "]
end
end
%% --------------------------------
%% 5. AI Generation & Fallback Chain
%% --------------------------------
subgraph AIChain [" 🤖 AI Generation — Resilient Fallback Chain "]
NIM_Pri[" 🟢 Nemotron Core AI\n(Primary Key) "]
NIM_Pri -->|" ResourceExhausted / Timeout "| NIM_Bk[" 🟡 Nemotron\n(Backup Key) "]
NIM_Bk -->|" Exhausted "| Groq1[" 🔵 Groq llama-3.3-70b-versatile "]
Groq1 -->|" Fallback "| Groq2[" 🔵 Groq llama-3.1-8b-instant "]
end
%% --------------------------------
%% 6. Context Merge & Final Output
%% --------------------------------
WebPri & WebSec & VidPri & VidSec & RRF & Scrape --> Merge{" 🔀 Context Merge & Deduplication "}
Merge --> Aggregator[" 📦 Unified Context Aggregator "]
Memory --> Aggregator
Aggregator --> NIM_Pri
Groq2 -->|" Stream Response "| FinalOutput([" ✅ Final Streamed Answer "])
NIM_Pri -->|" Stream Response "| FinalOutput
FinalOutput -->|" If Voice Triggered\n(Markdown Stripper) "| TTS[" 🔊 ElevenLabs TTS "]
TTS --> AudioOut([" 🎧 Spoken Audio Response "])
%% --------------------------------
%% Styling
%% --------------------------------
classDef primary fill:#76B900,stroke:#5e9400,color:#000,stroke-width:2px,font-weight:bold;
classDef bypass fill:#1a1a2e,stroke:#76B900,color:#76B900,stroke-width:2px;
class L1,WPR,L3,L4,L5,L6 primary;
class FastPath,URLBypass,IdentityHub bypass;
Nemetron features an advanced, structure-preserving RAG engine to parse and retrieve dense grid documents (such as timetable, reports, and academic papers) and YouTube transcripts:
- Layout-Aware PDF Coordinate Extraction — Leverages
pdfjs-distpixel coordinates (x,y,width,height) to reconstruct column grids and tabular layouts instead of flattening raw characters. - Visual Fallback Pipeline — Automatically detects low-density or corrupted pages and extracts tabular grids via Gemini multimodal PDF analysis, strictly validated against Zod schemas.
- Structured DOCX Parsing — Converts DOCX files to HTML using Mammoth and parses them into hierarchical block elements preserving heading structure.
- Timestamped YouTube Segmenting — Extracts and chunk-binds captions using time offsets for precise timestamp citations in answers.
- Local MiniLM Embeddings — Generates 384-dimensional dense vectors using
Xenova/all-MiniLM-L6-v2entirely on-server with no remote embedding API dependency. - RRF Hybrid Search RPC (
match_hybrid_chunks_v2) — Performs reciprocal rank fusion combining Postgres exact term matching, full-text indexes, and pgvector cosine similarity into a single composite score.
All database tables, Row-Level Security policies, and Supabase RPCs are consolidated into a single file for easy setup.
Run via Supabase Dashboard SQL Editor or Supabase CLI:
psql -h <host> -U postgres -d postgres -f supabase/schema.sqlsupabase/schema.sql— Full unified schema
Nemetron is genuinely impressive and handles the vast majority of queries with precision. However, as this is an active development and testing build, a few rough edges remain that we want to be transparent about:
- Occasional Context Confusion — In complex multi-document sessions, the intent router may occasionally misattribute which source a question is referring to, pulling from the wrong document.
- Semantic Search Intermittency — The local MiniLM embedding model requires a warm-up on first load. Cold-start requests may temporarily fall back to exact/FTS search instead of vector similarity search.
- Long Document Processing Latency — Very large PDFs or DOCX files (approaching the 5 MB cap) may experience delayed ingestion as the batched embedding pipeline works through chunks progressively.
- Rate Limit Cascades — Under sustained heavy usage, Nvidia Nemotron API limits can be triggered, causing the system to cascade through its fallback chain. While the system recovers gracefully, response times may temporarily increase to 30–60 seconds during fallback.
- Video Transcription Edge Cases — Some YouTube videos with auto-generated captions in non-English or mixed-language content may produce transcripts with reduced accuracy.
- Routing Quirks on Ambiguous Queries — Every so often, an intentionally abstract or multi-intent question may route to the wrong tool. The model occasionally misinterprets the scope of the request.
We are actively improving each of these areas. Contributions, bug reports, and feedback are warmly welcome.
MIT License — see LICENSE for full details.
Author — Mirza Haseeb ur Rahman
AI Engineer · Computer Science Student @ FAST University, Pakistan
Powered by Nemotron Core AI