feat: local retrieval and inspectable source selection with full regression evidence - #4
Merged
Merged
Conversation
…uard, demo ## Embedding (Aşama 2) - OllamaEmbedding: zero-dependency HTTP/1.0 to 127.0.0.1:11434/api/embed with L2 normalization and offline HashEmbedding fallback - default_provider_from_env() supports RI_EMBEDDING_PROVIDER env + --embedding CLI flag - Index::save_to / load_from serialize provider+dimension; reject dimension mismatch with io::ErrorKind::InvalidData ## Answer Generation (Aşama 3-4) - OllamaProvider: stdin(Stdio::null()) + --nowordwrap + ANSI strip_ansi() - Strengthened prompt: repository text explicitly marked as untrusted data; never instructions; indirect prompt injection neutralized - Citation guard in main.rs: validates [path:line] / [path:start-end] against evidence spans; zero verified citations OR model says 'Insufficient...' → exact refusal ## Evaluation (Aşama 5) - 42-question benchmark (30 answerable, 12 trap/adversarial) - New corpus files: retrieval.rs, storage.rs, security.rs - Results (hybrid_nomic, heldout MRR 0.9750): Lexical: Recall@5 1.000 MRR 1.0000 p50 73ms Semantic hash: Recall@5 1.000 MRR 0.8428 p50 73ms Hybrid hash: Recall@5 1.000 MRR 0.9667 p50 74ms Semantic nomic: Recall@5 1.000 MRR 0.9178 p50 484ms Hybrid nomic: Recall@5 1.000 MRR 0.9667 p50 485ms Trap refusal: 12/12 (100%) ## Regression Tests (Aşama 6) - 5 new tests: dimension_mismatch, dirty_worktree, file_deletion_purge, unanswerable_refusal, ollama_embedding_roundtrip_or_offline_fallback - Total: 25 tests (13 unit + 12 integration) — all pass ## Demo & Docs - scripts/demo.sh: 7-step reproducible end-to-end demo (Ollama health, lexical, semantic, hybrid, answer, refusal+injection, HTTP) - README.md: full rewrite with benchmark table, model details, demo instructions Ollama models: nomic-embed-text:latest (768-dim), qwen2.5-coder:1.5b Zero external crate dependencies (Cargo.toml [dependencies] = empty)
- Create src/citation.rs with strict parser and verifier: * Support Evidence IDs [E1], [E2] alongside span citations [path:start-end] * Strict line-level citation enforcement; zero uncited claims policy * Reject out-of-bounds, reversed ranges (start > end), non-numeric, and mixed valid/invalid citations on single line - Make EmbeddingProvider return io::Result<Vec<f32>>: * Check expected dimension, reject NaN/inf vectors * Reject HTTPS endpoint on local stream; enforce connect (5s), read (30s), write (10s) timeouts * Escape tabs and all control characters in JSON serialization - Guard child processes in OllamaProvider: * 60s timeout with explicit kill, bounded stdout (64KB) and stderr (4KB) - Prevent index file from indexing itself by ignoring .ri extension
- Add comprehensive regression tests in tests/ri_regression.rs: * Mixed valid and invalid citation rejection * Out of bounds and reversed range rejection * Single line uncited claim rejection * Prompt injection fixture neutralization * HTTPS endpoint rejection and provider dimension mismatch - Add scripts/reproducible_lifecycle_demo.sh: * 8-step lifecycle demo using isolated git workspace * Initial indexing, hybrid retrieval, grounded real Qwen answer with citation verification * Exact refusal on unanswerable trap * In-place file modification with old terms purged * File deletion with chunks purged * Clean teardown and git snapshot verification - Add evaluation/v2/ suite: * 52-question dataset (10 dev, 42 heldout, 12 traps) * 4-mode comparative benchmarking (lexical, hash_baseline, neural_nomic, hybrid_nomic) * Real LLM answering and citation guard evaluation
…t attribution - Implement resolve_citation for application-side source mapping of Evidence IDs and spans - Implement evidence_supports_claim to verify substantive token overlap between assertion and cited chunk: * Detect and refuse irrelevant citations where span exists but does not support claim (e.g. reload endpoint citing block_symlink_traversal) * Ensure zero uncited claims policy * Prevent counter corruption on mixed valid/invalid citations on single line - Add regression test answer_command_rejects_irrelevant_citation_even_if_span_exists - Update evaluation/v2 results with 100% trap refusal
rustfuture
changed the title
…tegrity tooling Follow-up to the extractive-selection change. The old 12/12 trap-refusal figure was still reachable through evaluate_modes.py and the smoke scripts, and the v3 report was a bare summary dump. - evaluation/evaluate_modes.py: trap outcomes are now classified as pre_model_refusal (no lexical anchor, model never called), model refusal, guard rejection, accepted-on-unanswerable, or provider error. No single 'refusal accuracy' number is reported; the script skips when Ollama is absent instead of silently falling back to an external provider. results_modes.json regenerated: 12 traps -> 8 pre-model refusals, 4 guard rejections, 0 false accepts. - evaluation/v3/evaluate.py: reports are rendered from raw.jsonl with a per-question verdict table, true numerators/denominators, and derived false_accepts / false_rejects / model_called / pre_model_refusals. Added --render-only to re-render a run with no model calls; run-01 regenerated. - tests/ri_regression.rs: assert every evidence chunk's stored text matches its declared path:start-end span (quotation integrity). - scripts/validate_citations.py: validates the extractive contract (accepted decision, in-range citations, exact verbatim quotation) instead of requiring concept words in free-form output; corpus-relative path base fixed. - scripts/llm_answer_smoke.sh, scripts/prompt_injection_smoke.sh, scripts/demo.sh: check only model-authored output for the injected sentinel and stop presenting concept-word presence or a single fixture as injection immunity. - docs/llm-validation.md rewritten for extractive-selection-v1; README, RELEASE_NOTES.md, evaluation/v2/README.md and evaluation/v3/README.md record that v1/v2 claims are historical and must not be pooled with v3.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and behavior
The previous citation guard treated word overlap as factual support. This PR
adds local neural retrieval and now limits CLI answers to source selection:
the model returns evidence IDs, the application renders verbatim excerpts,
and malformed or mixed prose/ID responses are refused atomically.
--answer-jsonrecords model calls, raw selection, resolved sources and refusalreasons. Quotation integrity is explicitly separate from semantic relevance.
The experimental heuristic API is retained but is not used to approve CLI answers.
Validation
qwen2.5-coder:1.5b and nomic-embed-text, with model digests and source hashes.
3 false refusals. 12 traps: 9 correct refusals, 3 false selections.
Limits
This is an inspectable source-selection product, not verified free-form synthesis.
Incorrect or malicious repository claims can still be selected and quoted as data.
No universal prompt-injection defense, 100% refusal, automatic hash fallback,
or semantic entailment guarantee is claimed. Historical v1/v2 metrics do not
describe the current answering contract. The corpus is synthetic regression data.
PR remains open for review; no merge or visibility change.