VISION is a multimodal AI learning assistant for students, educators, and researchers. It bridges the gap between passive consumption and active recall by transforming long-form content — YouTube lectures, PDFs, and raw text — into structured summaries and context-aware interactive quizzes.
VISION follows a modular, three-layer pipeline:
[ Ingestion Layer ] [ Processing Layer ] [ Persistence Layer ]
- YouTube URLs -> - Audio extraction (Whisper) -> - MongoDB Atlas
- PDF uploads - Summarization (T5) (metadata, transcripts,
- Raw text - Q&A generation (LLaMA 3/Groq) summaries, quiz state)
- Ingestion Layer — Accepts diverse input types: video URLs, document uploads, and raw text.
- Processing Layer — Runs a multi-model pipeline. Video audio is transcribed with OpenAI Whisper, text is compressed into hierarchical summaries with T5, and interactive quizzes are generated dynamically with LLaMA 3 via the Groq API.
- Persistence Layer — A schema-flexible MongoDB Atlas layer stores processing state, user history, and compiled study materials for fast, asynchronous retrieval.
| Feature | Description |
|---|---|
| Multi-input ingestion | Supports YouTube audio scraping, PDF parsing, and raw text input |
| Automated transcription | Converts lecture audio into high-fidelity text using OpenAI Whisper |
| Smart summarization | Generates dense, hierarchical bullet-point summaries with a fine-tuned T5 pipeline |
| Interactive Q&A | Dynamically synthesizes customizable study quizzes with LLaMA 3, built around active recall |
| Persistent session history | Maps outputs and metadata to MongoDB Atlas for historical retrieval |
| Analytics-ready state | Decoupled state tracking, designed to feed future data dashboards |
| Layer | Technology |
|---|---|
| Frontend UI | Streamlit |
| Backend API | FastAPI, Pydantic v2 |
| Orchestration | LangChain, Groq Cloud API |
| Speech-to-text | OpenAI Whisper |
| NLP summarization | T5 (Text-to-Text Transfer Transformer) |
| LLM reasoning | LLaMA 3 (8B / 70B) |
| Database | MongoDB Atlas |
VISION/
├── backend/ # FastAPI routes, schemas, and API controllers
├── frontend/ # Streamlit UI layouts, pages, and components
├── models/ # Pipeline scripts (T5 config, Whisper wrapper, LLaMA 3 prompts)
├── .env # Environment secrets (API keys, DB connection)
├── requirements.txt # Python dependency list
└── README.md
- Python 3.10 or higher
- An active MongoDB Atlas cluster and connection string
- A Groq Cloud API key
git clone https://github.com/Priyankshu-07/VISION.git
cd VISIONpython -m venv venv
# Windows
venv\Scripts\activate
# macOS / Linux
source venv/bin/activatepip install -r requirements.txtCreate a .env file in the project root:
MONGO_URL=your_mongodb_atlas_connection_string
GROQ_API_KEY=your_groq_api_keyNever commit your .env file. Confirm it is listed in .gitignore before pushing any changes.
Start the backend and frontend in separate terminal windows.
Terminal 1 — Backend
uvicorn backend.main:app --reloadTerminal 2 — Frontend
streamlit run frontend/app.pyThe application will be available at http://localhost:8501.
- Analytics dashboard — Visualize learning progression metrics over time
- Model benchmarking — Evaluate local T5 summaries against cloud LLMs for quality and density
- Docker support — Containerize the stack for deployment to Render, AWS, or Railway
- RAG integration — Enable semantic search across historical summaries using Retrieval-Augmented Generation
Contributions are welcome. To get started:
- Fork the repository
- Create a feature branch:
git checkout -b feature/YourFeature - Commit your changes:
git commit -m "Add YourFeature" - Push to your branch:
git push origin feature/YourFeature - Open a pull request with a clear description of your changes
Distributed under the MIT License. See LICENSE for full details.