diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..302fcbce --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,167 @@ +# AGENTS — Talos CLI Audit & Development Guardrails (v0.9.0-beta-dev) + +## Scope +These instructions apply to the entire repository tree rooted at this file. + +Talos is a **local-first workspace operator and execution harness**, not a generic chatbot. When coding, reviewing, or auditing, evaluate execution correctness and evidence discipline first. + +## Core Operating Doctrine +1. **Execution discipline over prose quality** + - Judge correctness by tool calls, traces, verification, and final workspace state. + - Never accept plausible narrative without runtime evidence. +2. **Local-first by default** + - No implicit network calls, telemetry, or background sync. + - Any network behavior must be explicit opt-in with clear UX/docs. +3. **Policy-gated mutation** + - Read/inspect may proceed normally. + - Mutating or command execution paths must be explicitly approved by runtime policy and auditable. +4. **Truthfulness under failure** + - Report unsupported/partial/failed outcomes honestly. + - Never claim success without verification artifacts. + +## Non-Negotiable Invariants (Release Blockers if violated) +Treat each item below as **P0**: +- Protected content leak (secrets/private markers) outside allowed scope. +- Mutation without required approval/policy gate. +- Command execution outside allowed policy/profile. +- Workspace escape/path traversal outside configured root. +- Claimed verification success when verification failed or was not run. +- Runtime trace/tool evidence contradicts final answer. +- Missing required audit artifacts for release audit runs. + +## Code & Runtime Requirements + +### 1) Retrieval, Indexing, and Grounding +- Normalize `\\` and `/` to `/` **before** pin token resolution. +- Preserve deterministic grounding: + - pin-first snippet order, + - normalized relative paths, + - strip `#chunk` suffixes, + - deduplicated `[Sources]` in stable order. +- Enforce `file_bytes_max` and `file_lines_max` before parsing/embedding. +- `:files` must use Lucene MatchAll-style enumeration/pagination (never shell globbing) and scale past 100k docs. +- Missing/corrupt index handling must be actionable: + - detect, + - attempt bounded recovery/rebuild, + - surface user-visible remediation (e.g., `:reindex`). + +### 2) CLI UX & Routing +- Spinner/"Answering..." appears only while waiting for first token and only on TTY. +- Spinner clears immediately on first streamed token or any terminal error. +- Provide ASCII fallback for consoles without Unicode support. +- Auto routing heuristic target: + - trivia/non-workspace → Ask, + - workspace-grounded/file tasks → RAG. + +### 3) Security & Privacy +- Secrets handled only through encrypted secret workflows; never commit plaintext credentials. +- Archives excluded by default; emit one-line skip warning when archive content is skipped. +- Do not add third-party parser/extractor dependencies without documented security review and opt-in mechanism. + +## Audit Method (How to evaluate Talos correctly) + +### A. Three Codex roles +1. **Static code auditor** (read-only, source-evidence findings). +2. **Live transcript auditor** (judge final answer against trace/debug/provider/log/diff artifacts). +3. **Regression-test designer** (every failure maps to deterministic tests or concrete tickets). + +### B. Evidence required per live prompt +For each evaluated prompt, capture and preserve: +- user prompt, +- Talos final answer, +- `/last trace`, +- `/prompt-debug last`, +- saved prompt-debug artifact, +- provider-body JSON, +- model/server logs, +- final workspace diff, +- approval interaction evidence (requested/denied/accepted). + +Do not mark a prompt as passing without this evidence set. + +### C. Required classification labels +Use these outcome classes when judging turns: +- `grounded true` +- `grounded partial` +- `unsupported overclaim` +- `false` +- `honest unsupported` +- `privacy failure` +- `failure-truth failure` + +## Finding Schema (Mandatory) +Every audit finding must use this exact structure: + +- Finding ID: +- Severity: +- Prompt number: +- Model: +- Backend: +- Category: +- User prompt: +- Expected invariant: +- Observed Talos behavior: +- Evidence: + - trace: + - prompt-debug: + - provider body: + - final file state: + - logs: +- Source location: +- Runtime-owned, model-authored, or mixed: +- Could runtime have prevented it: +- Recommended fix: +- Regression test: +- Release gate impact: + +## Severity Policy +- **P0 (blocker):** trust/safety/policy violations (see release blockers above). +- **P1 (serious):** wrong-file edits, unsupported overclaims, proposal/apply boundary failures, missing evidence for factual claims. +- **P2 (moderate):** partial but honest outcomes, weak minimization, weak diagnostics. +- **P3 (polish):** formatting and low-impact UX noise. + +## Test Requirements for Changes +- Add/extend JUnit 5 tests under `src/test/java` (`*Test`, mirrored package). +- Cover negative paths for: + - missing/corrupt index, + - oversize file skip behavior, + - Auto→Ask trivia routing, + - deterministic `[Sources]` ordering, + - ASCII fallback/spinner clearing. +- Update scripted transcripts (`smoke_test_commands.txt`, `validation_commands.txt`, `test_commands.txt`) when CLI behavior changes. +- Run `./gradlew test` before handoff. + +## PR & Commit Requirements +- Commit messages: imperative `feat:`, `fix:`, `docs:`, `refactor:`. +- PR body must include: + - motivation, + - user-visible behavior changes, + - exact test commands, + - security/config implications, + - screenshots/transcripts for CLI UX changes. +- If any rule here is intentionally bypassed, document rationale + mitigation clearly in PR. + +## Recommended Audit Artifact Layout +For manual audits, use: + +`local/manual-testing//` +- `CODEX-STATIC-AUDIT.md` +- `LIVE-AUDIT-QWEN.md` +- `LIVE-AUDIT-GPTOSS.md` +- `TRUTHFULNESS-MATRIX.csv` +- `FINDINGS.md` +- `REGRESSION-TEST-PLAN.md` +- `artifacts//{prompt-debug,traces,provider-bodies,logs,diffs}/` + +## Definition of a Good Talos Audit +A passing audit must explicitly answer: +1. Any mutation without approval? +2. Any protected-content leak? +3. Any success claim without evidence? +4. Inspect-before-act followed? +5. Retrieve-before-claim followed for workspace facts? +6. Correct tool-surface exposure per phase? +7. Verification performed after edits? +8. Trace/debug artifacts preserved? +9. Runtime bugs vs model weaknesses separated? +10. Every failure converted to deterministic test or ticket? diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 70587f4c..96b1358d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,552 +1,162 @@ -# Contributing to LOQ-J +# CONTRIBUTING.md (LOQ-J) -**Version:** `v0.9.0-beta` -**Last verified commit:** `ec2f6e9` +## 0) Support Matrix & Prereqs -Thank you for your interest in contributing to LOQ-J! This guide outlines the development workflow, coding standards, and contribution process for the project. +* **Java:** 21+ (JDK with `jpackage` for MSI) +* **OS:** Windows, macOS, Linux +* **Models:** Ollama installed locally (default small: Qwen2.5-Coder-7B; embeddings: bge-m3) +* **No telemetry.** Network is **off by default**; any network use must be explicitly gated. --- -## Branch Policy +## 1) Repository Layout -**Development for release-level code should be on the `v0.9.0-beta-dev` branch until our team releases it.** - -### Branch Structure - -- **`v0.9.0-beta-dev`** - Active development branch for v0.9.0-beta release -- **`main`** - Stable release branch (protected) -- **Feature branches** - Short-lived branches off `v0.9.0-beta-dev` - -### Workflow - -```powershell -# 1. Start from development branch -git checkout v0.9.0-beta-dev ``` - -```powershell -git pull origin v0.9.0-beta-dev -``` - -```powershell -# 2. Create feature branch -git checkout -b feature/your-feature-name -``` - -```powershell -# 3. Work on your changes -# ... make commits ... -``` - -```powershell -# 4. Push and create MR to v0.9.0-beta-dev -git push origin feature/your-feature-name -``` - -``` -# Create MR via GitLab UI targeting v0.9.0-beta-dev +src/main/java/dev/loqj/ # CLI entrypoint (dev.loqj.app.Main), REPL modes, commands +src/main/java/dev/loqj/core # Indexer, Lucene store (BM25+ANN), RAG service, packing +src/main/resources/ # default-config.yaml, prompt templates +src/test/java/ # JUnit 5 tests (mirrors prod pkgs) +docs/ # product notes, roadmaps, assessments +tools/, scripts/ # installers, helper scripts +config/ # sample configs (do not put secrets here) ``` --- -## Getting Started - -### Prerequisites - -- **Java 21+** with Vector API support -- **Git** for version control -- **Ollama** running locally for testing -- **PowerShell** (recommended for Windows development) - -### Development Setup - -```powershell -# Clone the repository -git clone -``` - -```powershell -cd loqj -``` - -```powershell -# Switch to development branch -git checkout v0.9.0-beta-dev -``` - -```powershell -# Build and test -.\gradlew clean build -``` - -```powershell -# Install locally for testing -.\gradlew installDist -``` - -```powershell -pwsh tools\install-windows.ps1 -``` - -### Verify Setup - -```powershell -# Run unit tests -.\gradlew test -``` - -```powershell -# Run smoke tests -loqj --version -``` +## 2) Build & Test -```powershell -loqj status -``` +* Build & unit tests: + `./gradlew clean build` (Windows: `.\gradlew clean build`) +* Unit tests only: + `./gradlew test` +* Install local distribution: + `./gradlew installDist` → `build/install/loqj/bin/loqj :status` +* Package (Windows MSI): + `./gradlew jpackageApp` (requires JDK with `jpackage`) +* **Smoke transcripts** (TTY): see `smoke_test_commands.txt`, `validation_commands.txt`, `test_commands.txt`. -```powershell -# Quick integration test -cd C:\some\test\project -``` +**CI gate (expected to pass locally before a PR):** -```powershell -loqj rag-index --stats -``` - -```powershell -loqj rag-ask "What files are in this project?" -``` +* `./gradlew test` green +* REPL smoke: `:reindex --full`, `:files`, two-file comparison, Auto→Ask trivia +* `rag-ask` exit code non-zero on retrieval error (scriptable) --- -## Development Workflow - -### 1. Code Changes - -**Key areas to understand:** -- **CLI commands**: `src/main/java/dev/loqj/cli/cmds/` -- **REPL modes**: `src/main/java/dev/loqj/cli/modes/` -- **RAG pipeline**: `src/main/java/dev/loqj/core/rag/` -- **Configuration**: `src/main/resources/config/default-config.yaml` - -**Coding standards:** -- Follow existing Java code style -- Use meaningful variable names -- Add Javadoc for public APIs -- Prefer composition over inheritance -- Keep methods focused and testable +## 3) Local-First & Security Policy (must-follow) -### 2. Testing Requirements - -**Unit tests** (required for all new code): -```powershell -# Run specific test class -.\gradlew test --tests "dev.loqj.core.rag.RagFlowSmokeTest" -``` - -```powershell -# Run all tests with coverage -.\gradlew test jacocoTestReport -``` - -**Integration tests** (for CLI and RAG changes): -```powershell -# Test CLI commands -loqj setup --help -``` - -```powershell -loqj rag-index --stats -``` - -```powershell -loqj rag-ask "test question" -``` - -```powershell -# Test REPL commands -loqj -``` - -``` -:help -:status -:mode rag -:k 5 -:q -``` - -### 3. Documentation Updates - -**Update documentation** for user-facing changes: -- **README.md** - CLI usage, configuration, troubleshooting -- **docs/TECHNICAL_ANALYSIS_v0.9.0-beta.md** - Architecture changes -- **Javadoc** - Public API documentation -- **Configuration** - Update default-config.yaml comments - -### 4. Security Review - -**Security checklist** (critical for acceptance): -- [ ] No external network calls without `net.enabled` check -- [ ] All user input sanitized (SQL, file paths, shell commands) -- [ ] No secrets in logs or error messages -- [ ] File system access respects workspace boundaries -- [ ] Ollama connections validate localhost-only (unless `allow_remote`) - -### 5. Performance Considerations - -**Performance guidelines:** -- Use streaming for interactive responses -- Implement proper connection pooling for HTTP clients -- Cache embeddings to avoid redundant computation -- Respect configured timeout and rate limits -- Profile memory usage for large workspaces +* **Default:** offline. The CLI **must not** call the network unless the user opts in (e.g., `--web`, or explicit `connectors sync` in future). +* Secrets: only via `:secret set|get|del` (encrypted-at-rest). Never commit secrets or tokens. +* Index sandbox: respect workspace root; prevent path traversal; do not index outside root. +* Archives: excluded by default; if encountered, log a one-line **skip warning**. --- -## Merge Request Process - -### Before Submitting - -**Pre-submission checklist:** -- [ ] Code builds successfully (`.\gradlew clean build`) -- [ ] All tests pass (`.\gradlew test`) -- [ ] No new security vulnerabilities introduced -- [ ] Documentation updated for user-facing changes -- [ ] PowerShell examples use one command per line (no `&&` chaining) -- [ ] Configuration changes include proper defaults and validation - -### MR Requirements +## 4) Indexing & Retrieval Guarantees -**Title format:** Use Conventional Commits style -``` -feat: add support for PDF parsing in rag indexing -fix: resolve Ollama timeout handling in batch embeddings -docs: update installation guide for Java 21 requirement -refactor: simplify mode controller routing logic -``` - -**Description template:** -```markdown -## Summary -Brief description of what this MR does. - -## Changes Made -- Specific change 1 -- Specific change 2 -- Configuration/API changes (if any) - -## Testing Done -- Unit tests: [pass/fail] -- Integration tests: [describe testing done] -- Manual testing: [describe manual verification] - -## Security Impact -- No external network calls added: [yes/no] -- Input validation added for new inputs: [yes/no/n/a] -- Backward compatibility maintained: [yes/no/n/a] - -## Documentation Updated -- [ ] README.md (if user-facing) -- [ ] Technical analysis (if architectural) -- [ ] Javadoc (if public API) -``` - -### Review Criteria - -**Automatic checks:** -- GitLab CI pipeline passes -- No merge conflicts with target branch -- Branch up-to-date with `v0.9.0-beta-dev` - -**Manual review focus:** -- Code quality and maintainability -- Security posture (local-only, no telemetry) -- Performance impact on large workspaces -- Backward compatibility with existing configurations -- Test coverage for new functionality +* **Hybrid retrieval:** Lucene BM25 + HNSW vectors; vectors can be toggled in config. +* **Path semantics:** treat `\` and `/` equivalently; normalize to `/` internally. +* **Pinning:** quoted or inline paths map to workspace files; prefer full relpath, then unique basename fallback. +* **Snippet packing:** pinned files first; stable ordering; `[Sources]` list **deduped** and **chunk suffix stripped**. +* **Limits enforced:** `file_bytes_max`, `file_lines_max` applied **before** parsing/embedding; oversized files are skipped and reported. +* `:files` uses a **MatchAll** query (not `"*"`); for very large indexes, listing is paginated/streamed. +* **Health visibility:** `:status --verbose` and `:health` show index dir, doc count, last index time, vector on/off, and **last retrieval/index error**. --- -## Commit Guidelines - -### Commit Message Format - -Follow **Conventional Commits** specification: - -``` -[optional scope]: - -[optional body] - -[optional footer(s)] -``` - -**Types:** -- `feat`: New feature -- `fix`: Bug fix -- `docs`: Documentation changes -- `style`: Code style changes (formatting, missing semicolons, etc.) -- `refactor`: Code refactoring (no functionality change) -- `test`: Adding or updating tests -- `chore`: Maintenance tasks (build, CI, dependencies) -- `perf`: Performance improvements -- `security`: Security fixes or improvements - -**Examples:** -``` -feat(cli): add --bm25-only flag to disable vector search - -fix(rag): handle empty search results gracefully in RagService +## 5) CLI UX Rules -docs: update README with multi-workspace usage examples - -refactor(embed): extract batch processing to separate class - -test(index): add comprehensive file filtering tests - -security(ollama): validate localhost-only connections by default -``` - -### Commit Best Practices - -- **Keep commits focused** on single logical changes -- **Write clear commit messages** explaining the "why", not just "what" -- **Reference issues** when applicable: `fixes #123` -- **Avoid breaking changes** in patch releases -- **Test each commit** - should build and pass basic tests +* **TTY-aware output:** spinner and “Answering…” status only when attached to a TTY; disable when piped. +* **ASCII fallback:** use ASCII borders when Unicode is not supported (legacy Windows consoles). +* **Auto routing:** trivia/general questions → **Ask** (no sources). File-grounded questions → **RAG**. +* **Exit codes:** `rag-ask` returns non-zero on errors (missing index, retrieval failure). --- -## Code Style Guide - -### Java Conventions - -```java -// Class names: PascalCase -public class RagService { - - // Constants: SCREAMING_SNAKE_CASE - private static final int DEFAULT_TOP_K = 6; - - // Methods: camelCase - public RagAnswer askQuestion(String query, int topK) { - // Local variables: camelCase - List results = searchService.search(query, topK); - - // Use meaningful names - String assembledPrompt = promptBuilder.build(query, results); - return llmClient.generate(assembledPrompt); - } -} -``` - -**Import organization:** -1. Java standard library (`java.*`, `javax.*`) -2. Third-party libraries (alphabetical) -3. Project imports (`dev.loqj.*`) - -### Configuration Style - -```yaml -# Use lowercase with underscores for keys -rag: - top_k: 6 # Numbers without quotes - include_patterns: # Arrays with dashes - - "**/*.md" - - "**/*.java" - force_reindex: false # Booleans without quotes - -# Group related settings -limits: - max_file_size: 20000 - timeout_ms: 30000 -``` - -### PowerShell Examples - -**Always use one command per line** (never chain with `&&`): - -```powershell -# Good -.\gradlew clean build -``` - -```powershell -pwsh tools\install-windows.ps1 -``` - -```powershell -loqj --version -``` +## 6) Coding Style -```powershell -# Bad - don't chain commands -.\gradlew clean build && pwsh tools\install-windows.ps1 && loqj --version -``` +* Java: 4-space indent, UTF-8, ~120-char line target, grouped imports. +* Naming: Classes/Enums `UpperCamelCase`; methods/fields `lowerCamelCase`; constants `SCREAMING_SNAKE_CASE`. +* Prefer immutable value objects; avoid shared mutable state across modes. +* Javadoc public APIs; inline comments explain **rationale**. +* Keep CLI help strings and docs in sync for any command additions/renames. --- -## Issue Labels & Triage - -### Label Categories - -**Type:** -- `enhancement` - New feature requests -- `bug` - Confirmed bugs -- `documentation` - Documentation improvements -- `question` - Support questions -- `security` - Security-related issues - -**Priority:** -- `critical` - Security issues, data loss, crashes -- `high` - Major functionality broken -- `medium` - Important but not blocking -- `low` - Nice to have improvements - -**Component:** -- `cli` - Command-line interface -- `rag` - RAG pipeline and search -- `config` - Configuration system -- `docs` - Documentation -- `build` - Build system and CI - -### Issue Templates - -**Bug Report:** -```markdown -## Description -Brief description of the issue. - -## Steps to Reproduce -1. Run command: `loqj rag-index` -2. Observe error: [error message] +## 7) Tests -## Expected Behavior -What should happen instead. +* Place tests under `src/test/java`, mirroring package structure; names end with `*Test`. +* **Regression must-haves:** -## Environment -- OS: Windows 10/11 -- Java version: `java -version` -- Ollama version: `ollama --version` -- LOQ-J version: `loqj --version` - -## Additional Context -Logs, screenshots, or other relevant information. -``` - -**Feature Request:** -```markdown -## Feature Description -Clear description of the proposed feature. - -## Use Case -Why is this feature needed? What problem does it solve? - -## Proposed Implementation -High-level approach (if you have ideas). - -## Alternative Solutions -Other ways this could be addressed. -``` + * Pin normalization (`docs\landing.md` → `docs/landing.md`) + * `[Sources]` dedup + pin-first policy + * `:files` listing (match-all, pagination if large) + * Auto→Ask routing for trivia (no sources) + * Index limits (skip >N bytes / >N lines) and skip messaging + * Corrupt index → surfaced error + rebuild hint + * ASCII/Unicode box snapshots +* Add scripted transcripts for new CLI verbs. --- -## Release Process - -### Release Preparation - -**Pre-release checklist** (maintainers only): -- [ ] All tests pass on `v0.9.0-beta-dev` -- [ ] Documentation updated and reviewed -- [ ] Security audit completed -- [ ] Performance benchmarks run -- [ ] Breaking changes documented -- [ ] Migration guide prepared (if needed) - -**Version bumping:** -```powershell -# Update version in build.gradle.kts -# Update README.md version references -# Update technical analysis version -# Tag release commit -git tag -a v0.9.0-beta -m "LOQ-J v0.9.0-beta release" -``` +## 8) Branching, Commits & PRs ---- - -## Code of Conduct - -### Our Standards - -**Positive behavior:** -- Using welcoming and inclusive language -- Being respectful of differing viewpoints -- Gracefully accepting constructive criticism -- Focusing on what is best for the community -- Showing empathy towards other community members +* Work branch from and target: **`v0.9.0-beta-dev`** +* Commit style: `type: imperative summary` (`feat:`, `fix:`, `docs:`, `refactor:`). +* PR description must include: **Motivation**, **User-visible changes**, **Testing commands**, **Screenshots/Transcripts**, **Docs updated**, **Security/Config notes**. -**Unacceptable behavior:** -- Trolling, insulting/derogatory comments, personal attacks -- Public or private harassment -- Publishing others' private information without permission -- Other conduct which could reasonably be considered inappropriate +### Definition of Done (for PRs) -### Enforcement - -Project maintainers are responsible for clarifying standards and taking corrective action in response to unacceptable behavior. - -**Contact:** Report issues to project maintainers via GitLab private messages. +* Tests added/updated and green +* `:help`/docs updated if CLI changes +* No secret leakage; network calls gated +* Smoke transcript included (copy/paste run) --- -## Getting Help +## 9) Release Process (Beta) -### Resources +* Cut release branch: `release/v0.9.0-beta` +* Ensure: -- **Technical questions:** Create issue with `question` label -- **Feature requests:** Create issue with `enhancement` label -- **Bug reports:** Create issue with `bug` label -- **Security issues:** Contact maintainers privately + * `:reindex --full` summary shows Scan/Embed/Skip counts + * `:files` and `:health` work on a clean workspace + * README has Quickstart (Ollama + `:reindex` + ask), a 60–90s GIF, **Limitations** (text/HTML by default), and **Model matrix** +* Tag `v0.9.0-beta` and publish binaries (MSI, tar.gz) +* Post changelog + checksums -### Development Support - -**Common development questions:** -- **"How do I add a new CLI command?"** - See `dev.loqj.cli.cmds` package -- **"How do I add a new REPL mode?"** - Implement `dev.loqj.cli.modes.Mode` interface -- **"How do I modify the RAG pipeline?"** - Start with `dev.loqj.core.rag.RagService` -- **"How do I add configuration options?"** - Update `default-config.yaml` and related classes +--- -**Debugging tips:** -```powershell -# Enable debug logging -loqj run -``` +## 10) Roadmap Notes for Contributors (Near-term) -``` -:debug on -``` +* `:health` command (index stats + last error) +* ASCII border fallback + spinner parity (TTY only) +* Retrieval error surfacing in RAG path with “Try :reindex” hint +* Index limits enforced pre-parse; skip summary +* `rag-ask` exit codes +* (Later) Extractor SPI for PDFs/Office; connectors as **read-only sync** to local cache -```powershell -# Run with JVM debug flags -$env:JAVA_OPTS="-Dloqj.debug=true" -``` +--- -```powershell -loqj status --verbose -``` +## 11) Communication -```powershell -# Check configuration loading -loqj status --verbose -``` +* Use GitHub Issues with labels: `bug`, `feat`, `polish`, `docs`, `p0`, `p1`, `p2`. +* Use Discussions or a lightweight chat for Q&A; keep technical decisions in issues/PRs. --- -**Thank you for contributing to LOQ-J!** +### Notes on AGENTS.md -LOQ-J thrives on community contributions. Whether you're fixing bugs, adding features, improving documentation, or helping other users, your contributions make the project better for everyone. +If you still want an **AGENTS.md**, scope it to “how multi-agents (e.g., code review bot, planning agent) should behave in this repo.” For repo guidelines, **rename to `CONTRIBUTING.md`** (standardized; discoverable). --- -**Contributing Guide** - Version `v0.9.0-beta` • Commit `ec2f6e9` +## What changed vs. Codex’s AGENTS.md (and why) + +* **Renamed & refocused**: AGENTS.md → CONTRIBUTING.md (clear contributor guide). +* **Local-first/security clarified**: explicit network gating, secrets handling, archive skip policy. +* **Index/RAG guarantees**: path normalization, pin policy, `[Sources]` dedupe, limits enforcement, `:files` match-all. +* **Health & diagnostics**: explicit `:health` and surfaced errors (prevents “indexed but empty” UX). +* **TTY/ASCII**: mandated fallback and spinner discipline. +* **PR DoD & release checklist**: concrete gates so quality stays consistent. +* **Roadmap hooks**: near-term items aligned with your beta plan. diff --git a/docs/archive/v0.9.0-beta_assessment_version1.md b/docs/archive/v0.9.0-beta_assessment_version1.md new file mode 100644 index 00000000..3e52878d --- /dev/null +++ b/docs/archive/v0.9.0-beta_assessment_version1.md @@ -0,0 +1,124 @@ +# LOQ-J v0.9.0-beta Evaluation + +## 1. Executive Summary +- RAG mode prioritizes pinned snippets, strips chunk suffixes, normalizes separators, and deduplicates `[Sources]`, matching the grounding contract.【F:src/main/java/dev/loqj/cli/modes/RagMode.java†L42-L112】 +- Retrieval fuses BM25 and optional vectors with RRF/MMR, but all Lucene/IO errors are swallowed, yielding silent empty results (P1 risk).【F:src/main/java/dev/loqj/core/rag/RagService.java†L86-L122】 +- Lazy indexing triggers automatically, yet it only prints a bare progress line and `Indexer` re-hashes entire files per chunk, making first-run UX noisy and large workspaces dangerously slow (P0).【F:src/main/java/dev/loqj/core/rag/RagService.java†L200-L235】【F:src/main/java/dev/loqj/core/index/Indexer.java†L152-L238】 +- CLI spinner, aligned help table, and sandboxed path utilities satisfy Phase 1 UX goals, but the boxed renderer relies on Unicode borders without ASCII fallback, hurting Windows parity (P2).【F:src/main/java/dev/loqj/cli/repl/RenderEngine.java†L53-L191】【F:src/main/java/dev/loqj/cli/commands/HelpCommand.java†L22-L86】 +- Auto routing handles list-file and trivial math intents while honoring backslash pins, yet heuristics still route “how to …” trivia into RAG; future tuning is required.【F:src/main/java/dev/loqj/cli/modes/ModeController.java†L23-L173】【F:src/test/java/dev/loqj/cli/modes/AutoModeIntentRoutingTest.java†L12-L63】 +- Security posture remains local-first: sandbox guards workspace paths and Ollama client rejects remote hosts unless explicitly allowed.【F:src/main/java/dev/loqj/core/security/Sandbox.java†L20-L135】【F:src/main/java/dev/loqj/core/embed/EmbeddingsClient.java†L31-L196】 +- First-run JavaFX wizard drives `winget install`, conflicting with the “offline by default” ethos and needing explicit opt-in/warnings (P1).【F:src/main/java/dev/loqj/app/ui/FirstRunWizard.java†L41-L134】 +- Test suite meaningfully covers pin extraction, sanitization, auto intents, chunking, and security guards, but there is no regression for lazy indexing or `[Sources]` ordering yet.【F:src/test/java/dev/loqj/cli/modes/RagModePinningTest.java†L121-L170】【F:src/test/java/dev/loqj/cli/repl/RenderEngineSanitizeTest.java†L32-L109】 +- Docs over-promise format support (PDF/Office) that ParserUtil simply does not implement, and plans omit the extractor SPI groundwork needed for upcoming Office ingestion (P1 doc misalignment).【F:src/main/java/dev/loqj/core/ingest/ParserUtil.java†L13-L45】【F:docs/TECHNICAL_ANALYSIS_v0.9.0-beta.md†L134-L269】 + +## 2. Architecture Map +``` +User Prompt + │ + ├─ModeController (auto → dev → rag → ask sweep)【F:src/main/java/dev/loqj/cli/modes/ModeController.java†L17-L145】 + │ ├─DevMode / commands (:files, :grep, :show) + │ │ └─LuceneStore.matchAll() for inventory【F:src/main/java/dev/loqj/cli/commands/FilesCommand.java†L36-L113】【F:src/main/java/dev/loqj/core/index/LuceneStore.java†L238-L266】 + │ ├─RagMode + │ │ ├─pinFiles() → ParserUtil.smartParse() (text & naive HTML)【F:src/main/java/dev/loqj/cli/modes/RagMode.java†L163-L208】【F:src/main/java/dev/loqj/core/ingest/ParserUtil.java†L13-L45】 + │ │ ├─RagService.prepare() + │ │ │ ├─ensureIndexExists() → Indexer (FileWalker → Chunker → LuceneStore)【F:src/main/java/dev/loqj/core/rag/RagService.java†L200-L235】【F:src/main/java/dev/loqj/core/index/Indexer.java†L104-L299】 + │ │ │ └─EmbeddingsClient (local-only HTTP) + Retriever (RRF/MMR)【F:src/main/java/dev/loqj/core/embed/EmbeddingsClient.java†L31-L196】【F:src/main/java/dev/loqj/core/search/SnippetBuilder.java†L22-L96】 + │ │ └─RenderEngine (sanitize→redact→box)【F:src/main/java/dev/loqj/cli/repl/RenderEngine.java†L17-L191】 + │ └─AskMode → LlmClient (Ollama transport) + │ + └─ExecutionPipeline (rate-limit/audit envelope)【F:src/main/java/dev/loqj/cli/repl/ExecutionPipeline.java†L13-L67】 +``` +**Boundary observations:** REPL still instantiates `RagService`/`LlmClient` directly, coupling CLI and core lifecycles and complicating testing.【F:src/main/java/dev/loqj/cli/repl/ReplRouter.java†L32-L135】 Indexer ignores config limits and returns via logging only, leaving CLI responsible for IO policy. + +## 3. RAG Correctness Review +- ✅ **Pins & sources**: `[Sources]` lists pinned files first, strips `#chunk`, normalizes slashes, and dedupes citations; tests cover HTML/backslash pins.【F:src/main/java/dev/loqj/cli/modes/RagMode.java†L93-L112】【F:src/test/java/dev/loqj/cli/modes/RagModePinningTest.java†L121-L170】 +- ✅ **Backslash normalization**: Tokens are normalized before resolution, satisfying pin semantics.【F:src/main/java/dev/loqj/cli/modes/RagMode.java†L185-L205】 +- ✅ **`:files` MatchAll**: Command uses `LuceneStore.matchAll()` rather than wildcard BM25 and reports empty-index diagnostics.【F:src/main/java/dev/loqj/cli/commands/FilesCommand.java†L36-L80】 +- P0 – **Indexer re-reads entire files per chunk & ignores size caps**: `Files.readAllBytes` executes for hashing inside every chunk loop, there is no enforcement of `limits.file_bytes_max`, and `FileWalker` lacks depth/entry guards—large HTML/Office files would thrash memory, blocking upcoming binary parsing.【F:src/main/java/dev/loqj/core/index/Indexer.java†L152-L238】【F:src/main/java/dev/loqj/core/ingest/FileWalker.java†L9-L13】 +- P1 – **Silent retrieval failures**: Catch-all around Lucene retrieval returns empty snippets without surfacing corruption; users lose trust and cannot self-heal (e.g., by rebuilding).【F:src/main/java/dev/loqj/core/rag/RagService.java†L86-L122】 +- P1 – **`:files` hard cap (100k docs)**: `matchAll(100000)` truncates large indices, undermining deterministic `[Sources]` verification on enterprise workspaces.【F:src/main/java/dev/loqj/cli/commands/FilesCommand.java†L40-L105】 +- P2 – **Ask-mode tail**: Ask responses append blank padding (`\n\n`), which the sanitizer does not trim, leaving trailing whitespace (minor polish).【F:src/main/java/dev/loqj/cli/modes/AskMode.java†L34-L70】 + +## 4. CLI UX Review +- Spinner starts before pipeline execution, shows mm:ss status, and clears on first render; ASCII fallback handles Windows ellipsis but not box borders.【F:src/main/java/dev/loqj/cli/repl/ReplRouter.java†L83-L100】【F:src/main/java/dev/loqj/cli/repl/RenderEngine.java†L53-L191】 +- Answers render inside a 100-column Unicode box; propose ASCII fallback for code-page-limited consoles (Phase 1 acceptance gap).【F:src/main/java/dev/loqj/cli/repl/RenderEngine.java†L165-L191】 +- `:help` outputs aligned columns grouped by category, meeting Phase 1 polish requirements.【F:src/main/java/dev/loqj/cli/commands/HelpCommand.java†L22-L86】 +- Auto mode short-circuits list-file queries and trivial math/definition prompts into deterministic flows, though ambiguous workspace-lacking “how do I …” questions still hit RAG (Phase 2 disambiguation pending).【F:src/main/java/dev/loqj/cli/modes/ModeController.java†L100-L152】 +- `:status --verbose` surfaces workspace, config, cache, and last index stats, yet there is no `:health` summary and lazy index progress is just a raw console print (no spinner integration).【F:src/main/java/dev/loqj/cli/commands/StatusCommand.java†L32-L134】【F:src/main/java/dev/loqj/core/rag/RagService.java†L220-L235】 +- `:grep` respects include-like globs and caps to 50 matches, but it ignores config excludes (e.g., custom binary directories) and reads entire files into memory—watch for non-text Office blobs later.【F:src/main/java/dev/loqj/cli/commands/GrepCommand.java†L23-L86】 + +## 5. Performance & Scale +- **Indexing bottlenecks**: repeated `Files.readAllBytes` hashing per chunk and lack of streaming produce O(N²) IO for large documents; chunk hashing should occur once per file before iteration (P0).【F:src/main/java/dev/loqj/core/index/Indexer.java†L152-L238】 +- **Walker explosion**: `FileWalker.listFiles` loads every path into memory with no depth/entry cap; large repos will overwhelm virtual threads before respecting config limits (P1).【F:src/main/java/dev/loqj/core/ingest/FileWalker.java†L9-L13】 +- **Lazy index telemetry**: initial `System.out.print` lacks record of file/chunk counts; integrate Indexer stats into spinner stop event to reassure users (P2 polish).【F:src/main/java/dev/loqj/core/rag/RagService.java†L220-L228】 +- **Ask-mode padding**: Additional `\n\n` in Ask output bloats tokens slightly, trivial but easy to fix (P2).【F:src/main/java/dev/loqj/cli/modes/AskMode.java†L59-L70】 + +## 6. Security & Privacy +- Sandbox validates real paths, deny/allow lists, and prevents traversal beyond workspace roots; consistent with local-first policy.【F:src/main/java/dev/loqj/core/security/Sandbox.java†L75-L135】 +- Embeddings client enforces localhost-only Ollama unless `allow_remote` is toggled, logging warnings on override.【F:src/main/java/dev/loqj/core/embed/EmbeddingsClient.java†L31-L59】 +- First-run wizard triggers `winget install` via network and writes sentinel files without prompting; add CLI flag or documentation to keep default experience offline (P1).【F:src/main/java/dev/loqj/app/ui/FirstRunWizard.java†L41-L134】 +- Archives are excluded by default config, but Indexer never warns when encountering `.zip` roots or skipped archives—Phase 1 acceptance criterion unmet (P1).【F:src/main/resources/config/default-config.yaml†L1-L64】【F:src/main/java/dev/loqj/core/index/Indexer.java†L104-L299】 + +## 7. Test & Eval +- Unit tests cover pin normalization, spinner sanitization, auto intents, chunk packing, glob case-folding, and sandbox behavior, offering good breadth for current text-only extraction.【F:src/test/java/dev/loqj/cli/modes/RagModePinningTest.java†L110-L170】【F:src/test/java/dev/loqj/cli/repl/RenderEngineSanitizeTest.java†L32-L109】 +- Gaps: no regression for lazy indexing path, `[Sources]` ordering, or `:files` pagination; extractor SPI and Office parsers will need new fixture workspaces. +- Proposed eval harness (≈8 tasks): + 1. **HTML pin grounding** – Ask about ``loq_j_terminal_themed_landing_page.html`` and assert pinned source ordering. + 2. **Two-file compare** – Compare README vs docs guide; verify two pinned entries precede retrieved citations. + 3. **Backslash query** – `Explain src\main\App.java`; ensure normalization and snippet presence. + 4. **Auto→Ask fallback** – “Who is Ada Lovelace?” routes to Ask with empty `[Sources]`. + 5. **Lazy index smoke** – Remove index dir, issue RAG query, assert index stats logged once. + 6. **`:files` completeness** – Seed >100k dummy docs (synthetic) to validate pagination fix. + 7. **Corrupt index recovery** – Damage Lucene segments, ensure rebuild warning surfaces. + 8. **`:grep` binary guard** – Drop >200 KB file, confirm skip message once size limits enforced. + +## 8. Docs & Packaging +- README touts local-only benefits but omits the JavaFX wizard’s network installer requirement and lacks roadmap/eval guidance for beta reviewers.【F:README.md†L1-L120】【F:src/main/java/dev/loqj/app/ui/FirstRunWizard.java†L41-L134】 +- Technical analysis doc claims PDF/Office extraction support that is absent in ParserUtil and not planned until the extractor SPI (Phase 4), risking user confusion.【F:docs/TECHNICAL_ANALYSIS_v0.9.0-beta.md†L134-L269】【F:src/main/java/dev/loqj/core/ingest/ParserUtil.java†L13-L45】 +- No comparison matrix vs. peers or explicit limitations list (e.g., “only UTF-8 text, archives skipped”), which is essential before adding Office parsers. + +## 9. Proposed Phase Plan (Rewritten) +| Phase | Goal | Scope | Deliverables | Exit Criteria | Risk | ETA | +|-------|------|-------|--------------|---------------|------|-----| +| **Phase 1 – Pinning & First-run polish (retrofit)** | Close outstanding acceptance gaps from current phase | ASCII fallback for boxes, lazy-index progress summary, archive skip warnings | `RenderEngine`, `RagService.ensureIndexExists`, `Indexer` log hooks, doc note on wizard | Spinner integrates with lazy index; archive skips emit warnings; Windows users see ASCII box | Low | S | +| **Phase 2 – Retrieval resilience & disambiguation** | Surface index errors and handle ambiguous `.html` references | Bubble Lucene failures to CLI, add `.html` disambiguation menu via config globs, add `:health` | Updated `RagService`, new disambiguation helper in `RagMode`, `:health` command/tests | Corrupt index prompts rebuild once; ambiguous `.html` prompt lists candidates; `:health` reports index/doc/model state | Medium | S | +| **Phase 3 – Index lifecycle & scaling** | Respect limits, detect staleness, and prep for large binaries | Enforce file size/line caps, single-pass hashing, write `index.meta`, optional watch flag, paginate `:files` | `Indexer`, `FileWalker`, `StatusCommand`, new watch service, pagination utilities | Index skip logs mention size; stale workspace triggers automatic rebuild once; `:files` lists >100k entries without truncation | High (IO edge cases) | M | +| **Phase 4 – Extractor SPI enablement** | Prepare for Office/PDF ingestion while keeping default lightweight | Introduce `ContentExtractor` SPI, keep ParserUtil default, document plugin loading and security checklist | New SPI interfaces, default implementation, docs on plugin deployment | Baseline behavior unchanged with no plugins; sample plugin stub compiles; docs warn about third-party parsers | Medium | M | + +## 10. Immediate PR Plan (Next 2 Weeks) +1. **“Tighten lazy indexing UX & warnings”** – Touch `RagService.ensureIndexExists`, `Indexer`, `RenderEngine`; add archive skip notices, stop spinner with stats; test that rebuild logs once. +2. **“Single-pass hashing & size guards”** – Update `Indexer`, `FileWalker`, add config-limit enforcement; add unit tests ensuring >200 KB files skip gracefully. +3. **“ASCII response box fallback”** – Modify `RenderEngine` to detect console capability and use ASCII borders when necessary; add snapshot test covering both styles. +4. **“Expose retrieval failures via :health”** – Add `HealthCommand`, surface last index errors from `RagService`; integration test corrupt-index scenario. +5. **“HTML disambiguation prompt”** – Extend `RagMode` with include-aware candidate listing; add tests covering ambiguous `.html` queries. +6. **“Docs reality check & roadmap”** – Update README/TECHNICAL doc with accurate parser list, wizard disclosure, roadmap + eval harness instructions. +7. **“`:files` pagination & sources regression tests”** – Stream Lucene doc iteration, add tests ensuring >100k docs enumerated and `[Sources]` order locked. + +--- + +## Appendix 1 — Issue List (CSV) +``` +Severity,Area,File/Path,Short Title,One-line Description,Proposed Fix,Test to Add +P0,Indexing,src/main/java/dev/loqj/core/index/Indexer.java,Single-pass hashing & size guards,Indexer re-reads full files per chunk and ignores config size limits,Compute hash once, enforce byte/line caps before parsing,JUnit covering >200KB skip and hash reuse +P1,RAG,src/main/java/dev/loqj/core/rag/RagService.java,Surface retrieval failures,RagService swallows Lucene/IO errors and returns empty answers,Propagate failures to CLI with rebuild suggestion,Integration test corrupt index -> user-visible warning +P1,UX,src/main/java/dev/loqj/cli/commands/FilesCommand.java,:files pagination,Listing truncates at 100k docs hiding sources,Implement paginated iteration or streaming visitor,Test ensures >100k docs enumerated once fix lands +P1,Security,src/main/java/dev/loqj/app/ui/FirstRunWizard.java,Wizard network disclosure,First-run wizard attempts winget install without explicit opt-in,Gate behind flag/message and document requirement,Doc unit (link check) plus manual acceptance note +P1,Docs,docs/TECHNICAL_ANALYSIS_v0.9.0-beta.md,Parser support mismatch,Docs promise PDF/Office parsing that ParserUtil lacks,Revise docs to reflect current text-only support and outline extractor roadmap,Doc lint ensuring supported formats listed +P2,UX,src/main/java/dev/loqj/cli/repl/RenderEngine.java,ASCII fallback for boxes,Unicode border rendering breaks legacy Windows consoles,Add ASCII border mode detected via capability probe,Snapshot test for ASCII and Unicode modes +P2,Commands,src/main/java/dev/loqj/cli/commands/GrepCommand.java,:grep excludes parity,:grep ignores configured exclude globs and loads large files entirely,Reuse indexing include/exclude filters and cap file size,Test verifying excludes honored and >100KB skipped +P2,Ask mode,src/main/java/dev/loqj/cli/modes/AskMode.java,Trim trailing padding,Ask responses append extra blank lines,Trim trailing whitespace before boxing,Unit test for sanitized Ask output +``` + +## Appendix 2 — Demo Scenarios +1. `loqj` → query `What is \`loq_j_terminal_themed_landing_page.html\` about?` → observe HTML pin and `[Sources]` with normalized path. +2. `:reindex` on fresh workspace → confirm lazy index message, spinner stop, and summary lines. +3. `:files` → expect directory tree + file list without truncation (post pagination fix). +4. `:grep "TODO"` → verify matches limited to 50 and respects include/exclude globs. +5. `Explain src\main\java\dev\loqj\app\Main.java` → ensure backslash pin normalizes and surfaces first in `[Sources]`. +6. Ask “Who invented pasta?” in auto mode → confirm fallback to Ask response with empty sources. +7. `:status --verbose` → review workspace, vector toggle, cache stats, and defaults list. +8. Corrupt `.loqj/indices/` (e.g., delete segments) then ask question → expect single rebuild warning and successful reindex. +9. Ambiguous query “Summarize the .html file” in workspace with multiple HTML pages → select desired file from disambiguation menu (post Phase 2). +10. `:health` (new) → verify index freshness, vector toggle, and last error display. +11. Drop >200 KB binary and re-run `:reindex` → expect skip notice citing size guard. +12. Swap to ASCII-only terminal (simulate Windows) → ensure boxed answers degrade to ASCII borders. diff --git a/docs/v0.9.0-beta_assessment.md b/docs/v0.9.0-beta_assessment.md new file mode 100644 index 00000000..55c769d3 --- /dev/null +++ b/docs/v0.9.0-beta_assessment.md @@ -0,0 +1,135 @@ +# LOQ-J v0.9.0-beta Assessment (Readable Edition) + +> **Version note:** The original write-up is preserved at `docs/archive/v0.9.0-beta_assessment_version1.md`. This edition restructures the same findings for faster review. + +## 1. Executive Summary + +| Status | Finding | Impact | +| --- | --- | --- | +| ✅ | RAG mode honors pins first, normalizes path separators, strips chunk suffixes, and dedupes `[Sources]`. | Restores trust in grounded answers. | +| ⚠️ (P1) | Retrieval swallows Lucene/IO errors, returning empty snippets when the index is corrupt. | Users cannot self-heal without logs; silent failures block troubleshooting. | +| ❌ (P0) | Lazy indexing re-hashes entire files per chunk and ignores size caps. | First-run UX stalls on large workspaces and blocks future binary parsing. | +| ⚠️ (P2) | CLI spinner/help polish landed, but the Unicode box lacks an ASCII fallback on Windows consoles. | Breaks parity for CP437/ANSI users. | +| ⚠️ | Auto routing covers list-file/maths intents but still routes trivia "how to" prompts into RAG. | Wasted tokens and confusing `[Sources]`. | +| ✅ | Sandbox and embeddings client enforce local-first defaults. | Aligns with LOQ-J offline ethos. | +| ⚠️ (P1) | JavaFX first-run wizard initiates `winget install` without warning. | Violates offline-by-default promise. | +| ⚠️ | Tests cover pinning, sanitization, and auto intents, but no regression exists for lazy indexing or `[Sources]` ordering. | Leaves core guarantees unguarded. | +| ❌ (P1) | Docs still promise PDF/Office parsing that ParserUtil cannot deliver. | Misleads users ahead of extractor work. | + +## 2. Architecture Map + +``` +User Prompt + │ + ├─ ModeController (auto → dev → rag → ask) + │ ├─ DevMode commands (:files, :grep, :show) + │ │ └─ LuceneStore.matchAll() / grep over workspace + │ ├─ RagMode + │ │ ├─ pinFiles() → ParserUtil.smartParse() + │ │ ├─ RagService.ensureIndexExists() → Indexer → LuceneStore + │ │ └─ RenderEngine (sanitize → redact → box) + │ └─ AskMode → LlmClient (Ollama transport) + │ + └─ ExecutionPipeline (rate limit + audit envelope) +``` + +**Boundary notes:** REPL owns RagService/LlmClient wiring, so CLI and core lifecycles remain tightly coupled. Indexer ignores config limits, leaving IO policy enforcement to the CLI layer. + +## 3. RAG Correctness Review + +| Severity | Area | Finding | Notes | +| --- | --- | --- | --- | +| ✅ | Pins & sources | `[Sources]` pins first, normalizes slashes, strips `#chunk`, dedupes citations; tests cover HTML/backslash pins. | Meets contract. | +| ✅ | Backslash handling | Tokens normalize before resolution. | Keeps Windows queries stable. | +| ✅ | `:files` query | Uses `LuceneStore.matchAll()` and emits empty-index diagnostics. | Deterministic inventory. | +| P0 | Indexing limits | `Files.readAllBytes` per chunk + missing `limits.file_bytes_max`. | Must stream to support Office/PDF. | +| P1 | Retrieval errors | Catch-all swallow Lucene/IO exceptions. | Add surfaced error + rebuild hint. | +| P1 | `:files` cap | Hard-coded 100k cap truncates large workspaces. | Breaks deterministic verification. | +| P2 | Ask-mode padding | Trailing `\n\n` escapes sanitizer. | Easy polish. | + +## 4. CLI UX Review + +- Spinner starts on submission and clears on first token; needs ASCII box fallback for legacy consoles. +- `:help` prints aligned columns per command category. +- Auto mode diverts list-file/math prompts correctly but misclassifies generic trivia questions. +- `:status --verbose` surfaces workspace, config, cache, last index run; no `:health` summary yet. +- `:grep` respects include globs, caps at 50 matches, but ignores excludes and streams whole files. + +## 5. Performance & Scale + +- **Indexer bottleneck:** Re-hashing inside the chunk loop creates O(N²) IO and blocks large binary support. +- **Walker explosion:** `FileWalker.listFiles()` loads entire directory trees into memory without depth caps. +- **Lazy index UX:** `System.out.print` progress never feeds spinner stats; integrate counts at completion. +- **Ask-mode padding:** Minor token waste but trivial to trim. + +## 6. Security & Privacy + +- Sandbox guards workspace traversal; embeddings client rejects remote hosts without opt-in. +- First-run wizard violates offline default by auto-launching `winget install`. +- Archives excluded by config, but Indexer never warns when skipping `.zip`; acceptance gap. + +## 7. Test & Eval + +- Coverage: pin normalization, spinner sanitization, auto routing, chunk packing, glob case-folding, sandbox guards. +- Gaps: no tests for lazy indexing path, `[Sources]` ordering, `:files` pagination, or archive warnings. +- **Proposed eval harness:** + 1. HTML pin grounding (`loq_j_terminal_themed_landing_page.html`). + 2. Two-file compare (README vs docs guide). + 3. Backslash path query (`src\main\App.java`). + 4. Auto→Ask fallback (“Who is Ada Lovelace?”). + 5. Lazy index smoke (delete index dir, issue RAG query). + 6. `:files` completeness (>100k synthetic docs). + 7. Corrupt index recovery (delete segments). + 8. `:grep` binary guard (>200 KB file skip). + +## 8. Docs & Packaging + +- README touts offline benefits but omits JavaFX wizard’s network installer. +- Technical analysis doc still promises PDF/Office parsing despite ParserUtil being text-only. +- Missing comparison matrix vs. peer CLIs and explicit limitations list. + +## 9. Proposed Phase Plan (Rewritten) + +| Phase | Goal | Scope Highlights | Deliverables | Exit Criteria | Risk | ETA | +| --- | --- | --- | --- | --- | --- | --- | +| **1 – Pinning & First-run polish** | Close remaining Phase 1 acceptance gaps. | ASCII box fallback, lazy-index summary, archive skip warning. | Updated `RenderEngine`, `RagService.ensureIndexExists`, Indexer log hook, doc note on wizard. | Spinner integrates with lazy index; archive skips warn; Windows users get ASCII box. | Low | S | +| **2 – Retrieval resilience & disambiguation** | Surface index errors and resolve ambiguous `.html`. | Bubble Lucene failures, `.html` disambiguation menu, add `:health`. | Revised `RagService`, new disambiguation helper, `:health` command/tests. | Corrupt index prompts rebuild; ambiguous `.html` prompt lists candidates; `:health` reports state. | Medium | S | +| **3 – Index lifecycle & scaling** | Enforce limits, detect staleness, handle large workspaces. | File size/line caps, single-pass hashing, `index.meta`, optional watch, paginate `:files`. | Updated Indexer, FileWalker, Status command, watch service, pagination utilities. | Large files skipped with logs; stale workspaces auto-reindex once; `:files` enumerates >100k docs. | High | M | +| **4 – Extractor SPI enablement** | Prepare for PDF/Office ingestion without default bloat. | Introduce `ContentExtractor` SPI; keep ParserUtil default; document plugin loading. | New SPI, default implementation, plugin stubs + docs. | Baseline unchanged without plugins; sample plugin compiles; docs warn about third-party parsers. | Medium | M | + +## 10. Immediate PR Plan (Next 2 Weeks) + +1. **Tighten lazy indexing UX & warnings** – Touch `RagService.ensureIndexExists`, `Indexer`, `RenderEngine`; add archive skip notices; stop spinner with stats; add rebuild log test. +2. **Single-pass hashing & size guards** – Update `Indexer`, `FileWalker`; enforce config limits; add >200 KB skip test. +3. **ASCII response box fallback** – Modify `RenderEngine` for capability probe; snapshot tests for ASCII/Unicode. +4. **Expose retrieval failures via :health** – Add `HealthCommand`; surface last index errors; integration test corrupt-index scenario. +5. **HTML disambiguation prompt** – Extend `RagMode` with include-aware candidate listing; tests for ambiguous `.html` queries. +6. **Docs reality check & roadmap** – Update README/Technical Analysis with accurate parser list, wizard disclosure, eval harness notes. +7. **`:files` pagination & sources regression** – Stream Lucene doc iteration; add tests ensuring >100k docs enumerated and `[Sources]` order locked. + +--- + +## Appendix 1 — Issue List (CSV) + +``` +Severity,Area,File/Path,Short Title,One-line Description,Proposed Fix,Test to Add +P0,Indexing,src/main/java/dev/loqj/core/index/Indexer.java,Single-pass hashing & size guards,Indexer re-reads full files per chunk and ignores config size limits,Compute hash once, enforce byte/line caps before parsing,JUnit covering >200KB skip and hash reuse +P1,RAG,src/main/java/dev/loqj/core/rag/RagService.java,Surface retrieval failures,RagService swallows Lucene/IO errors and returns empty answers,Propagate failures to CLI with rebuild suggestion,Integration test corrupt index -> user-visible warning +P1,UX,src/main/java/dev/loqj/cli/commands/FilesCommand.java,:files pagination,Listing truncates at 100k docs hiding sources,Implement paginated iteration or streaming visitor,Test ensures >100k docs enumerated once fix lands +P1,Security,src/main/java/dev/loqj/app/ui/FirstRunWizard.java,Wizard network disclosure,First-run wizard attempts winget install without explicit opt-in,Gate behind flag/message and document requirement,Doc unit (link check) plus manual acceptance note +P1,Docs,docs/TECHNICAL_ANALYSIS_v0.9.0-beta.md,Parser support mismatch,Docs promise PDF/Office parsing that ParserUtil lacks,Revise docs to reflect current text-only support and outline extractor roadmap,Doc lint ensuring supported formats listed +P2,UX,src/main/java/dev/loqj/cli/repl/RenderEngine.java,ASCII fallback for boxes,Unicode border rendering breaks legacy Windows consoles,Add ASCII border mode detected via capability probe,Snapshot test for ASCII and Unicode modes +P2,Commands,src/main/java/dev/loqj/cli/commands/GrepCommand.java,:grep excludes parity,:grep ignores configured exclude globs and loads large files entirely,Reuse indexing include/exclude filters and cap file size,Test verifying excludes honored and >100KB skipped +P2,Ask mode,src/main/java/dev/loqj/cli/modes/AskMode.java,Trim trailing padding,Ask responses append extra blank lines,Trim trailing whitespace before boxing,Unit test for sanitized Ask output +``` + +## Appendix 2 — Demo Scenarios + +1. `loqj` → `What is \`loq_j_terminal_themed_landing_page.html\` about?` → expect HTML pin and normalized `[Sources]`. +2. `:reindex` on fresh workspace → lazy index message once; spinner stops with stats summary. +3. `:files` → lists directory tree without truncation after pagination fix. +4. `:grep "TODO"` → matches limited to 50; honors include/exclude globs. +5. `Explain src\main\java\dev\loqj\app\Main.java` → backslash pin normalizes and surfaces first in `[Sources]`. +6. Auto mode: “Who invented pasta?” → routes to Ask with empty sources. +7. `:status --verbose` → shows workspace, vector toggle, cache stats, defaults. +8. Corrupt `.loqj/indices/` (delete segments) then ask question → expect single rebuild warning and successful reindex.