PathFinderShip

PathFinderShip is a multimodal AI assistant with a structured FastAPI backend that routes user requests across local CPU inference, retrieval-augmented generation, optional web retrieval, API-based generation, browser camera actions, and ONNX object detection.

The default generation path uses a project-specific fine-tuned T5 model exported to quantized ONNX encoder/decoder artifacts and executed locally on CPU. The same pipeline can switch to Gemini through a provider abstraction without changing the orchestration layer.

PathFinderShip can also integrate with Diagent to observe real pipeline behavior through runs, spans, retrievals, tool calls, and application-specific policy evidence.

Highlights

• Central PipelineOrchestrator for request execution
• Typed Pydantic contracts for intent, routing, retrieval, generation, detection, client actions, and final results
• Fine-tuned local T5 generation through ONNX Runtime on CPU
• Optional Gemini API provider behind the same generation interface
• ONNX MiniLM intent classification with confidence-aware routing
• Hybrid RAG with Chroma semantic search and SQLite FTS5/BM25
• Optional web retrieval with explicit execution evidence
• PDF, DOCX, TXT, Markdown, and HTML upload/indexing
• YOLO-NAS ONNX object detection on CPU
• Browser-owned camera flow with backend-owned inference
• Optional fail-open Diagent observability and policy checks

Model Development Evidence

The local models used by this project were developed by fine-tuning pretrained checkpoints on project-curated datasets; they were not trained from scratch. The evidence package preserves successful, intermediate, and rejected runs instead of presenting only the best checkpoint.

The read-only source audit currently records:

• 459 relevant dataset, notebook, configuration, metric, model, and visual-evidence files (18.46 GiB)
• 32 model artifacts and 16 training-resume artifacts
• 34 training/evaluation notebooks, including executed outputs
• 4 saved Trainer states and 52 SHA-256 duplicate groups
• a frozen experiment registry with professional IDs mapped to the original folder names such as kötü, 1.2x chat, 2x rag, and My Class

Retraining Evaluation v2

Six updated Flan-T5 Large LoRA adapters were evaluated on the same 300-example Chat and 160-example project RAG suites. The original folder aliases 1.2x Chat and 2x RAG describe task-loss weights; they do not mean model scaling or dataset duplication.

Experiment	Deliberate training change	Chat token-F1	RAG token-F1	RAG EM	Decision
LoRA q/v — kötü	q/v-only LoRA, r=16, alpha=32	0.3942	0.8127	0.7063	Rejected: Chat remained weak
RAG-loss-weight 2.0 — step 1320	RAG loss weight 2.0; Chat 1.0	0.3778	0.5864	0.4625	Weighting did not improve the shared evaluation
Chat-loss-weight 1.2 — step 1485	Chat loss weight 1.2; RAG 1.0	0.4695	0.6719	0.5563	Intermediate checkpoint
Chat-loss-weight 1.2 — step 1980	Same weighting, more optimization steps	0.4684	0.6587	0.5438	Slight regression versus step 1485
My Class — First Try	First custom trainer/loss iteration	0.4937	0.8421	0.7438	Strong runner-up
My Class — Second Try	Chat weight 1.7, task smoothing, partial R-Drop	0.5216	0.8894	0.7938	Selected final; best on every reported metric

The results show why the rejected and intermediate attempts are retained: RAG weight 2.0 did not help, Chat weight 1.2 improved Chat, and extending the same run from step 1485 to 1980 caused a small regression. The custom Second Try configuration produced the strongest balance. There is no cross-task composite score, and values are reported with their metric names rather than as generic “accuracy.”

MiniLM intent result

The unchanged MiniLM-L6 INT8 ONNX classifier scored accuracy, macro-F1, and weighted-F1 of 1.0000 on the frozen balanced 1,000-example intent suite; ECE was 0.1813. The perfect label score applies only to this project stress set, while ECE shows that confidence calibration is not perfect.

The v2 result card links the machine-readable JSON, editable CSV, auditable workbook, updated figures, and claims boundary. Historical notebook losses remain available in the experiment record; they are not relabeled as v2 evaluation metrics. See also the benchmark protocol, dataset card, artifact inventory, and split-leakage audit.

Published Model Artifacts

The selected artifacts are published on Hugging Face so developers, technical reviewers, and hiring engineers can inspect the model cards, configurations, evaluation context, and checksums or download the files for independent review. PathFinderShip is presented as a student portfolio and reproducible engineering study, not as a hosted commercial model service.

Artifact	What it demonstrates	Hugging Face
PathFinder Flan-T5 Large — Second Try	Final selected Chat + RAG LoRA adapter; loaded together with google/flan-t5-large	Model card and files
PathFinder Flan-T5 Large — Second Try INT8	Quantized ONNX encoder/decoder deployment artifacts for local CPU inference	Model card and files
PathFinder MiniLM Intent INT8	Five-class ONNX intent classifier used by the structured routing pipeline	Model card and files

An artifact repository can be downloaded with the Hugging Face CLI:

hf download Fatihaybasn/pathfinder-flan-t5-large-second-try-lora --local-dir ./pathfinder-lora

The benchmark results and experiment history were produced and documented by this project. Hugging Face provides artifact hosting and versioning; these links should not be read as independent metric certification by Hugging Face.

Architecture

flowchart TD
    UI[Browser UI] --> API[FastAPI API]
    API --> PIPE[PipelineOrchestrator]

    PIPE --> NLU[MiniLM ONNX Intent Classifier]
    NLU --> ROUTE[Route Decision]

    ROUTE --> CHAT[Chat]
    ROUTE --> RAG[RAG]
    ROUTE --> DETECT[Detection]
    ROUTE --> ACTION[Client Action]

    CHAT --> GEN[Generation Provider]
    RAG --> LOCAL[Local Hybrid Retrieval]
    RAG --> WEB[Optional Web Retrieval]
    LOCAL --> CHROMA[Chroma Semantic Search]
    LOCAL --> FTS[SQLite FTS5 / BM25]
    RAG --> GEN

    DETECT --> YOLO[YOLO-NAS ONNX / CPU]
    YOLO --> GEN

    ACTION --> CAMERA[Browser Camera]

    GEN --> T5[Fine-tuned T5 / ONNX CPU]
    GEN --> GEMINI[Gemini API]

    PIPE --> RESULT[Structured RunResult]
    RESULT --> UI

    API -. optional telemetry .-> DIAGENT[Diagent]

Loading

Structured Pipeline

The primary backend entry point is:

POST /api/run

A request flows through:

user message
    ↓
intent classification
    ↓
route decision
    ↓
selected execution path
    ├── chat
    ├── RAG
    ├── detection
    └── browser client action
    ↓
structured RunResult

The classifier does not directly execute services. It produces an IntentResult; a separate route layer decides what actually runs. This keeps prediction, orchestration, and side effects distinct.

The shared pipeline models are:

• IntentResult
• RouteDecision
• RetrievedChunk
• RetrievalResult
• IndexingResult
• GenerationResult
• DetectionResult
• ClientAction
• RunResult

A RunResult can preserve the selected route, retrieval evidence, generation metadata, detection output, fallbacks, warnings, errors, and total duration instead of returning only a final answer string.

Generation Providers

PathFinderShip separates orchestration from text generation:

BaseGenerationProvider
├── LocalT5Provider
└── GeminiProvider

Local fine-tuned T5

The default provider is:

GENERATION_PROVIDER=local_t5

The local path uses the project's fine-tuned T5 model as quantized ONNX encoder/decoder artifacts. Inference runs through ONNX Runtime with:

CPUExecutionProvider

It supports:

• chat
• RAG answers
• model-only fallback answers
• camera action narration
• detection narration

Structured generation results preserve metadata such as model, runtime, device, prompt type, latency, token counts when available, truncation state, fallback state, and errors.

Gemini API

The same pipeline can use Gemini:

GENERATION_PROVIDER=gemini
GEMINI_API_KEY=...
GEMINI_MODEL=gemini-2.5-flash

The Gemini provider supports configurable timeout, maximum output tokens, and temperature while returning the same GenerationResult contract used by the local provider.

Intent Classification and Routing

Intent classification uses an ONNX MiniLM model with a local tokenizer and CPU inference.

Canonical intent labels include:

• open_camera
• close_camera
• take_photo
• object_detect
• chat

The classifier returns confidence, threshold state, raw scores, latency, and errors.

Routing is handled separately in route_decision.py. It:

• applies confidence thresholds
• normalizes intent aliases
• handles classifier failures
• distinguishes predicted intent from executed route
• uses a narrow document-question heuristic for RAG routing
• records fallback reasons
• emits browser client actions when user-side capabilities are required

Execution routes include:

• chat
• rag
• detect
• camera_action

For example, object detection without an available image can return a capture_photo client action rather than pretending that the backend owns a browser camera.

Hybrid RAG

PathFinderShip combines semantic and keyword retrieval.

Semantic search

ChromaDB stores document chunks and embeddings generated with:

sentence-transformers/all-MiniLM-L6-v2

Queries use the same embedding model and explicit query embeddings.

Keyword search

SQLite FTS5 provides keyword retrieval with BM25 ranking.

Score fusion

Semantic and keyword scores are normalized and combined with configurable weights:

VECTOR_WEIGHT=0.75
BM25_WEIGHT=0.25

Results are returned as structured chunks with source, score, rank, retrieval type, and metadata.

Relevant defaults include:

RAG_SCORE_THRESHOLD=0.40
RAG_TOP_K=4
RAG_MAX_CTX_TOKENS=512

When retrieval does not provide usable context, the pipeline uses an explicit model-only fallback path.

Optional Web Retrieval

Web retrieval can be requested per run. The flow includes:

DDGS search
    ↓
parallel page fetch
    ↓
HTML/text extraction
    ↓
chunking
    ↓
relevance scoring
    ↓
web-strength gate

The implementation deduplicates domains, fetches pages concurrently, filters unsupported content types, extracts readable text, and ranks candidate chunks.

The structured retrieval result explicitly separates:

web search attempted
    ≠
web result survived final filtering

RetrievalResult records:

• web_search_attempted
• web_search_status
• web_candidate_count
• web_error_type

This avoids inferring that web search did not run merely because no web chunk survived the final strength gate.

Document Upload and Indexing

Documents can be uploaded through:

POST /api/upload

Supported formats:

• PDF
• DOCX
• TXT
• Markdown
• HTML / HTM

The indexing flow is:

upload
    ↓
type and size validation
    ↓
SHA-256 content digest
    ↓
stable document ID
    ↓
text extraction
    ↓
token-aware chunking
    ↓
Chroma indexing
    ↓
SQLite FTS5 indexing

If the local tokenizer is unavailable, chunking falls back to a word-based strategy.

Content-derived document IDs support replacement of prior chunks in both Chroma and SQLite when the same document is indexed again.

Object Detection and Camera Flow

Camera ownership and model inference are separated.

Browser responsibilities

• camera permission
• live preview
• frame capture
• image upload

Backend responsibilities

• image decoding
• YOLO-NAS ONNX inference
• confidence filtering
• IoU filtering
• non-maximum suppression
• coordinate restoration
• structured detection results

YOLO inference runs through ONNX Runtime on CPU. The service handles letterbox preprocessing, BGR-to-RGB conversion, normalized NCHW input, different score layouts, and box scaling back to the original image.

When configured, the selected generation provider can turn the detection summary into a natural-language narration.

Frontend

The browser frontend includes:

• conversational chat
• camera preview and capture
• image upload
• document upload
• web-search toggle
• source rendering from structured retrieval results
• route and intent indicators
• browser speech recognition
• browser speech synthesis
• local chat history
• light/dark theme
• Markdown export
• response regeneration

API

Method	Endpoint	Purpose
GET	/api/health	Liveness check
GET	/api/readiness	Configuration and asset readiness
POST	/api/run	Primary structured pipeline
POST	/api/intent	Intent classification
POST	/api/chat	Direct generation
POST	/api/rag	Direct RAG flow
POST	/api/detect	Image detection and optional narration
POST	/api/photo	Photo persistence and background email flow
POST	/api/upload	Document upload and indexing

With the backend running, FastAPI exposes interactive API documentation at:

http://127.0.0.1:8000/docs

Observability with Diagent

PathFinderShip can optionally send structured telemetry to Diagent, an observability, evaluation, and diagnosis backend for AI-agent and RAG workflows.

The integration is disabled by default:

DIAGENT_ENABLED=false

When enabled, PathFinderShip uses a lazy, fail-open bridge around the Diagent tracer. A telemetry failure does not need to become an application failure.

The integration maps pipeline behavior into explicit telemetry:

PathFinderShip behavior	Diagent telemetry
Intent classification	pathfindership.intent system span
Route decision	pathfindership.route_decision system span
Retrieval evidence	retrieval telemetry
Generation	pathfindership.generation LLM span
Web search execution	pathfindership.web_search tool call
YOLO inference	pathfindership.yolo.detect tool call
Browser action request	pathfindership.client_action system span
Policy evaluation	pathfindership.policy_check system span

Retrieval telemetry is bounded and metadata is sanitized. Raw image bytes, base64 image payloads, authorization headers, API keys, and full provider responses are not required by the integration.

Application-specific policy checks

PathFinderShip evaluates deterministic expected-vs-actual behavior for cases such as:

• RAG route without recorded retrieval
• grounded answer without source evidence
• required web fallback not executed
• unexpected YOLO execution for a non-vision route
• detection attempted without a received image
• empty generation output
• completed run without a final answer

These checks remain distinct from generic Diagent anomaly detection:

telemetry tells what happened
policy defines what should have happened

A policy span is not automatically an alert and is not automatically a diagnosis trigger.

The worked Diagent-side reference integration is available at:

https://github.com/fatihaybsn/Diagent/tree/example/pathfindership-integration

That branch demonstrates bounded consumption of failed pathfindership.policy_check evidence by the diagnostician while preserving alert and trigger semantics.

Project Structure

PathFinder-Ship/
├── backend/
│   ├── schemas/
│   │   └── pipeline.py
│   ├── services/
│   │   ├── generation/
│   │   ├── observability/
│   │   ├── rag_backend/
│   │   ├── document_indexing.py
│   │   ├── nlu_classifier.py
│   │   ├── pipeline_orchestrator.py
│   │   ├── rag.py
│   │   ├── route_decision.py
│   │   ├── t5.py
│   │   └── yolo.py
│   ├── tests/
│   ├── utils/
│   ├── web/
│   │   └── app.py
│   ├── config.py
│   └── main.py
├── frontend/
│   ├── app.js
│   ├── index.html
│   └── styles.css
├── .env.example
├── requirements.txt
└── README.md

Setup

1. Create a virtual environment

Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1

macOS/Linux:

python -m venv .venv
source .venv/bin/activate

Install dependencies:

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

2. Create local configuration

Windows PowerShell:

Copy-Item .env.example backend\.env

macOS/Linux:

cp .env.example backend/.env

Keep secrets out of source control.

3. Choose a generation provider

Local T5:

GENERATION_PROVIDER=local_t5

Default local asset paths:

T5_TOKENIZER_DIR=assets/models/t5/tokenizer
T5_ENCODER=assets/models/t5/encoder_model_int8.onnx
T5_DECODER=assets/models/t5/decoder_model_int8.onnx

Gemini:

GENERATION_PROVIDER=gemini
GEMINI_API_KEY=...
GEMINI_MODEL=gemini-2.5-flash

4. Run the backend

cd backend
python main.py

The API runs on:

http://127.0.0.1:8000

5. Run the frontend

From the repository root:

cd frontend
python -m http.server 5173

Open:

http://localhost:5173

Readiness

With the backend running:

GET /api/readiness

PowerShell example:

Invoke-RestMethod http://127.0.0.1:8000/api/readiness

The readiness response reports expected T5, NLU, YOLO, RAG corpus, Chroma, SQLite, generation-provider, feature-flag, and Diagent configuration state.

It verifies configuration and expected assets; it does not perform full model inference.

Optional Diagent Setup

Enable observability in backend/.env:

DIAGENT_ENABLED=true
DIAGENT_API_URL=http://localhost:8001
DIAGENT_AGENT_NAME=pathfindership
DIAGENT_FAIL_OPEN=true

Use the actual Diagent backend URL. If PathFinderShip and Diagent run on the same machine, assign them different ports.

The Diagent package must be importable by the PathFinderShip process. With sibling repositories:

workspace/
├── Diagent/
└── PathFinder-Ship/

install Diagent in editable mode:

python -m pip install -e ../Diagent

Run the Diagent backend separately and point DIAGENT_API_URL to it.

Reference integration:

https://github.com/fatihaybsn/Diagent/tree/example/pathfindership-integration

Configuration

Selected environment variables:

Variable	Default	Purpose
GENERATION_PROVIDER	local_t5	Select local_t5 or gemini
CLS_ROUTE_THRESHOLD	0.60	Intent confidence threshold
RAG_SCORE_THRESHOLD	0.40	Local retrieval threshold
RAG_TOP_K	4	Number of local retrieval candidates
VECTOR_WEIGHT	0.75	Semantic score weight
BM25_WEIGHT	0.25	Keyword score weight
RAG_WEB_MIN_STRENGTH	0.75	Web result strength gate
ENABLE_EMAIL	false	Enable email-related behavior
DIAGENT_ENABLED	false	Enable Diagent telemetry
DIAGENT_MAX_RETRIEVAL_CHUNKS	5	Bound retrieval telemetry
DIAGENT_MAX_CHUNK_CHARS	1200	Bound telemetry chunk text

See .env.example for the full configuration surface.

Testing

Run the Python test suite from the repository root:

pytest -q

The suite covers pipeline schemas, route decisions, orchestration, structured RAG evidence, generation providers, local T5 results, upload/indexing, YOLO results, Diagent telemetry mapping, policy checks, and readiness behavior.

Frontend source extraction has a focused JavaScript test:

node frontend/test_extract_sources.js