novai

Local RAG with FastAPI + LlamaIndex + Ollama (deepseek-r1:1.5b). JWT auth, multi-session chat, attachable documents in Chroma, three working function-calls.

---

What this is

Nov.AI is a local RAG (Retrieval Augmented Generation) system built with FastAPI (Python), leveraging partially llamaindex and ollama to run a local Large Language Model (deepseek-r1:1.5b). It also features:

• JWT-based authentication (user registration & login).
• Multi-session chat with persistent memory.
• Document management (upload, attach, remove).
• RAG-based retrieval from a local Chroma database (using manual chunking/embedding).
• Function calling/tools system.

Table of Contents

1. Project Status
2. Features
3. Project Structure
4. RAG Implementation Details & LlamaIndex
5. Function Calling
6. Installation & Setup
7. Running the Project
8. Backend Endpoints Overview
9. Frontend
10. Screenshots

---

Project Status

• In Development
  • The system runs locally with ollama on port 11434.
  • Accomplished:
    • Chain-of-thought (CoT) model running locally.
    • RAG with persistent ChromaDB storage.
    • Function calling with three working tools (local time, weather via OpenWeather, currency conversion via Exchangerates).
    • Security layer (JWT-based).
    • Document management (upload, attach/detach).
    • Session management (multiple chat sessions, persistent conversations).
  • In Progress:
    • More granular message management inside each session (edit, update, remove messages).
  • Missing/Planned Improvements:
    • Better error handling in streaming (currently requires a page refresh).
    • Env-based API key management (e.g. .env or Docker env vars).
    • Blocking UI during document upload until the backend responds.
    • Auto-refresh when sessions expire (no manual reload).
    • Add test.

---

RAG Implementation & LlamaIndex

Originally planned to let LlamaIndex handle all the chunking, embedding, and retrieval for a typical RAG setup. However, due to the complexity of requirements, multiple user sessions, documents that can be attached/detached at runtime, plus a persistent SQL-based & persistent vector-storage document manager, opted for a more manual approach:

• Manual Chunking & Embedding: Split each uploaded document into chunks and embed them with a HuggingFace model. These chunks/embeddings are then stored in a local Chroma database.
• Selective Retrieval: During chat inference, if a user’s session has documents attached, only query the subset of chunks whose doc_id is in the session’s attached list.
• LlamaIndex Usage: Still rely on LlamaIndex for:
  • The Ollama integration, which lets stream local LLM responses (our LLMService).
  • The HuggingFace embedding model class (HuggingFaceEmbedding) for generating embeddings.
  • The message memory system.

By handling chunking and storage without llamaindex,attach or detach documents per session is more direct. This does limit some of LlamaIndex’s built-in features like dynamic re-chunking, rewriting, and hierarchical indexes. In future projects, leverage LlamaIndex’s full indexing pipeline for a simpler end-to-end RAG solution will be considered.

RAG Debug Endpoints

Exposed an internal set of routes under /rag/ purely for debugging and research. These endpoints let inspect exactly how storing and retrieving text chunks/embeddings in is stored in Chroma. Not intended for production use!

---

Function Calling

Function calling implemented, three function calls:

1. Local Time (getLocalTime):

   • No external API; simply returns the current system time (hour:minute) plus a phrase (morning/afternoon/night).

2. Weather Forecast (getWeather):

   • Uses the OpenWeatherMap API.
   • Requires an API key you place inside tools/plugins/weather.py.
   • Returns current weather conditions (temperature, description).

3. Currency Converter (convertCurrency):

   • Uses the Exchangerates API.
   • Also needs an API key, stored in tools/plugins/exchange.py.
   • Converts an amount from one currency to another (e.g. USD to EUR).

Note: Currently, only one function can be called per user prompt. If a single prompt requests multiple tools (“What's the time, the weather, and convert 50 USD to EUR?”), the LLM typically picks one. In the future, thew system can be extended to handle multi-call requests.

Plan to move all API keys and configuration into an .env file or a centralized config soon, along with Dockerization and advanced message management. For now, these three calls demonstrate the full end-to-end flow:

1. User requests a function.
2. LLM outputs JSON specifying "use_tool": "yes", a tool_id, and parameters.
3. Backend calls the matching plugin.
4. The final result is returned to the chat session.

---

Features

1. User System

   • Registration & login using JWT tokens.
   • Basic session expiration/renewal flow.

2. Chat & Memory

   • Users can create multiple chat sessions.
   • Each session stores chat messages (user & assistant).
   • Messages are saved in a local database, so they persist across restarts.

3. Documents (RAG)

   • Users can upload documents.
   • Documents can be attached/detached from any chat session.
   • RAG is integrated via a local Chroma database.
   • Documents are chunked and embedded upon upload, and then retrieved during inference.

4. Tools (Function Calling)

   • Basic structure to load available tools (e.g., Weather, Internet, etc.) Dummy for now.
   • Tools can be attached/detached to sessions (symbolic for now).

5. Local LLM

   • Powered by llamaindex + ollama for local inference.
   • Uses an open-source CoT (chain-of-thought) model: deepseek-r1:1.5b.

6. Frontend

   • Streams tokens in real-time, including thinking content.
   • Supports Markdown, tables, code snippet blocks, and a responsive UI.
   • Implements add/remove/list for sessions, documents, and (static) function tools.

---

Project Structure

Main directories and files:

• main.py <-- FastAPI entry point
• config.py <-- Configuration parameters
• security.py <-- JWT token creation/verification
• database.py <-- SQLAlchemy engine & session
• rag/ <-- RAG system (database modificarion, query, etc.)
• models/ <-- Database models (User, Session, Documents, etc.)
• routes/ <-- FastAPI route definitions (auth, chat, sessions, tools, documents)
• services/ <-- Core logic: chat_service, memory_service, LLM service, RAG service
• utils/logger.py <-- Central logger setup
• create_tables.py <-- Utility script to initialize DB schema
• seed_tools.py <-- Template for seeding function-calling tools

---

Installation & Setup

Prerequisites:

• Python 3.9+ (recommend using Conda environment).
• ollama installed and running locally (on default port 11434).

1) Create and activate a Conda environment (example command):

conda create -n novai python=3.9
conda activate novai

2) Install required Python packages:

pip install fastapi uvicorn sqlalchemy pydantic python-jose llamaindex ...

(Check the requirements.txt or your internal reference for the complete list.)

3) Initialize the database:

python create_tables.py

This creates the necessary tables in the local database.db.

4) (Optional) Seed Tools:

python seed_tools.py

This inserts some sample dummy tools for the function-calling demonstration.

---

Running the Project

1) Start 'ollama' (in a separate terminal/window) to serve the local model:

ollama serve

2) Launch the FastAPI backend:

uvicorn main:app --reload --host 0.0.0.0 --port 8000

3) Access the API:
Default backend URL: http://localhost:8000/

---

Backend Endpoints Overview

1. Authentication (/auth)

   • POST /auth/register
   • POST /auth/login
   • POST /auth/logout
   • GET /auth/profile

2. Chat (/chat)

   • POST /chat/stream (streams inference tokens)
   • POST /chat/stop (stops ongoing inference)

3. Sessions (/sessions)

   • GET /sessions (list user sessions)
   • POST /sessions (create session)
   • DELETE /sessions/{session_id} (delete session)
   • GET /sessions/{session_id}/messages (list messages)
   • POST /sessions/{session_id}/messages (add new message)
   • PUT /sessions/{session_id}/messages/{message_id} (edit message)
   • DELETE /sessions/{session_id}/messages/{message_id} (remove message)

4. Documents (/documents)

   • GET /documents/list (list user documents)
   • POST /documents/upload (upload and ingest doc)
   • POST /documents/delete (delete a document)
   • POST /documents/attach (attach doc to session)
   • POST /documents/detach (detach doc from session)

5. Tools (/tools)

   • GET /tools/list (lists all available tools)
   • POST /tools/attach (attach a tool to a session)
   • POST /tools/detach (detach a tool from a session)

---

Frontend

Technologies:

• Built with React, Vite, and MUI.

Running:

cd frontend
npm install
npm run dev

The frontend is configured to talk to http://localhost:8000 by default (adjust if needed).

---

Screenshots

Register / Log In

Chat / Inference

Chat / Inference - Light theme

Sessions Manager

Managers

---

Enjoy experimenting with Nov.AI!
Feel free to explore the code and modify it to suit your research and development needs.

---

License

MIT for original code in this repository (FastAPI backend, React/Vite/MUI frontend, scripts, configs). Third-party libraries and services pulled at runtime (llamaindex, Ollama, Chroma, HuggingFace embeddings, deepseek-r1:1.5b, OpenWeather, Exchangerates) retain their own upstream licenses and terms of service; this repository does not redistribute them.