LLM RAG Project - Embedded Systems Subject Matter Expert

Local offline-capable LLM system for embedded C development assistance.

---

Quick Start Checklist

Every time you start working on this project, run through this checklist:

1. Check Ollama Service

# Check if Ollama is running
systemctl status ollama

# If not running, start it
sudo systemctl start ollama

# Verify models are available
ollama list
# Should show: deepseek-coder:6.7b-instruct and others

2. Check Open WebUI Service

# Check if Open WebUI is running
sudo systemctl status open-webui

# If not running, start it
sudo systemctl start open-webui

# Verify it's accessible
# Open browser to: http://localhost:8080

3. Activate Project Virtual Environment

# Navigate to project directory
cd ~/apps/llm-rag-project

# Activate the venv
source venv/bin/activate

# Your prompt should now show: (venv)

4. You're Ready!

• Open WebUI running at http://localhost:8080
• Ollama serving models at localhost:11434
• Python environment activated for document processing

---

System Architecture

┌─────────────────────────────────────────┐
│  Open WebUI (localhost:8080)            │
│  Service: open-webui.service            │
│  Location: /opt/open-webui/             │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│  Ollama (localhost:11434)               │
│  Service: ollama.service                │
│  Models: deepseek-coder:6.7b-instruct   │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│  ChromaDB (./chroma_db)                 │
│  - Standards documents                  │
│  - Code samples                         │
│  - Embeddings: bge-small-en-v1.5        │
└─────────────────────────────────────────┘

---

Project Structure

~/apps/llm-rag-project/
├── README.md              # This file
├── venv/                  # Python virtual environment (DO NOT COMMIT)
├── standards_pdfs/        # Input: Your PDF standards (J1939, ISO, etc.)
├── processed_chunks/      # Output: JSONL files from Docling
├── scripts/               # Python processing scripts
│   ├── process_standards.py
│   ├── load_to_chromadb.py
│   └── create_code_index.py
├── sample_projects/       # Known Good C code samples
├── chroma_db/            # Vector database (created after processing)
└── requirements.txt       # Python dependencies for this project

---

Common Commands

Start/Stop Services

# Start Ollama
sudo systemctl start ollama

# Stop Ollama
sudo systemctl stop ollama

# Start Open WebUI
sudo systemctl start open-webui

# Stop Open WebUI
sudo systemctl stop open-webui

# Check service status
systemctl status ollama
sudo systemctl status open-webui

Enable Auto-Start on Boot (Optional)

# Enable Ollama to start on boot
sudo systemctl enable ollama

# Enable Open WebUI to start on boot
sudo systemctl enable open-webui

Ollama Model Management

# List installed models
ollama list

# Pull a new model
ollama pull 

# Remove a model
ollama rm 

# Run a model (keeps it loaded in VRAM)
ollama run deepseek-coder:6.7b-instruct

Virtual Environment

# Activate venv
cd ~/apps/llm-rag-project
source venv/bin/activate

# Deactivate venv
deactivate

# Install new package in venv
pip install 

# Save current dependencies
pip freeze > requirements.txt

# Reinstall all dependencies (if venv recreated)
pip install -r requirements.txt

---

Document Processing Workflow

1. Add PDFs to Process

# Copy your standards PDFs
cp /path/to/standards/*.pdf ~/apps/llm-rag-project/standards_pdfs/

2. Process PDFs with Docling

cd ~/apps/llm-rag-project
source venv/bin/activate
python scripts/process_standards.py

3. Load Chunks into ChromaDB

python scripts/load_to_chromadb.py

4. Verify in Open WebUI

• Open browser to http://localhost:8080
• Navigate to Settings → Documents
• Verify your documents are loaded
• Test a query referencing your standards

---

Troubleshooting

Open WebUI won't start

# Check logs
sudo journalctl -u open-webui -f

# Check if port 8080 is in use
sudo lsof -i :8080

# Restart the service
sudo systemctl restart open-webui

Ollama not responding

# Check if running
systemctl status ollama

# Check GPU availability
nvidia-smi

# Restart Ollama
sudo systemctl restart ollama

# Check logs
journalctl -u ollama -f

Virtual environment issues

# If packages seem broken, recreate venv
cd ~/apps/llm-rag-project
deactivate
rm -rf venv
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

GPU not being used

# Monitor GPU during query
nvidia-smi -l 1

# Should show:
# - GPU utilization > 80% during inference
# - Memory usage appropriate for model

# If not using GPU, check Ollama installation
which ollama
ollama run llama2 --verbose  # Will show if CUDA detected

---

Installed Python Packages (in venv)

Current packages (run pip list to see versions):

• docling - PDF processing
• chromadb - Vector database
• sentence-transformers - Embeddings
• (add more as you install them)

---

Important Notes

DO NOT:

• ❌ Use pip install without activating the venv first
• ❌ Use sudo pip install ever
• ❌ Commit the venv/ directory to git
• ❌ Commit the chroma_db/ directory to git (large files)
• ❌ Use --break-system-packages flag

DO:

• ✅ Always activate venv before running Python scripts
• ✅ Keep requirements.txt updated when adding packages
• ✅ Check services are running before starting work
• ✅ Use deactivate when done with Python work
• ✅ Document any new scripts or workflows in this README

---

Useful Keyboard Shortcuts

Terminal

• Ctrl+C - Stop running process (e.g., a Python script)
• Ctrl+Z - Suspend process (use fg to resume)
• Ctrl+D - Exit terminal (or deactivate venv)

Open WebUI

• Access at: http://localhost:8080
• Stop service: sudo systemctl stop open-webui

---

System Requirements

• OS: Arch Linux (Omarchy Hyperland config)
• GPU: NVIDIA RTX 3050 Ti (4GB VRAM)
• RAM: 64GB
• Storage: 1TB SSD
• Network: 400 Mbps (for initial setup only)

---

Offline Operation

Once everything is set up, this system runs completely offline:

• ✅ Ollama models stored locally
• ✅ Open WebUI runs locally
• ✅ ChromaDB vector database local
• ✅ Embedding model cached locally
• ✅ No external API calls required

To verify offline capability:

# Disable network
sudo systemctl stop NetworkManager

# Test queries in Open WebUI
# Should still work!

# Re-enable network
sudo systemctl start NetworkManager

---

Resources & Documentation

• Ollama: https://ollama.ai/docs
• Open WebUI: https://docs.openwebui.com
• ChromaDB: https://docs.trychroma.com
• Docling: https://github.com/DS4SD/docling
• Arch Wiki - Python: https://wiki.archlinux.org/title/Python

---

Project Goals

This system is designed to be a subject matter expert for:

• Embedded C programming for automotive ECUs
• J1939 CAN bus protocol implementation
• ISO 15765-2 (CAN transport protocol)
• Infineon TC399 microcontroller programming
• STW controller application development
• Mobile machinery control systems
• "Known Good" coding patterns from sample projects

---

Quick Test Queries

Use these to verify the system is working correctly:

1. Basic LLM Test:

   "Explain the difference between a mutex and a semaphore in RTOS"
   

2. Standards Knowledge Test (after RAG setup):

   "What is the purpose of J1939 PGN 64982?"
   

3. Code Pattern Test (after code indexing):

   "Show me the recommended CAN message filtering pattern"
   

4. Hardware Specific Test:

   "What are the DMA channel limitations on the TC399?"
   

---

Version History

• 2025-12-30: Initial setup
  • Installed Ollama via pacman
  • Installed Open WebUI via pacman
  • Pulled deepseek-coder:6.7b-instruct
  • Created project venv
  • Installed docling, chromadb, sentence-transformers

---

Next Steps

• Process first PDF standard with Docling
• Set up ChromaDB vector database
• Index sample C projects
• Test RAG retrieval accuracy
• Create custom prompt templates in Open WebUI
• Fine-tune chunking strategy based on results

---