Skip to content

feat(ai): site-knowledge index for retrieval evals (group 1) - #386

Open
arutsh wants to merge 11 commits into
mainfrom
AI/feat/Issue-381/site-assistant-retrieval-evals-group1
Open

arutsh wants to merge 11 commits into
mainfrom
AI/feat/Issue-381/site-assistant-retrieval-evals-group1

Conversation

@arutsh

@arutsh arutsh commented Oct 3, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adds the site_chunks table and an idempotent ingestion pipeline that indexes published site-content sections, user-guide docs, and PRODUCT.md into full-text-searchable rows, synced on service startup.
  • Fixes a config bug found while testing locally: SITE_CONTENT_DIR/GUIDE_DOCS_DIR/PRODUCT_DOC_PATH defaulted to container-only /app/... paths, so bare-metal (non-Docker) dev runs silently indexed zero rows. Defaults now fall back to the real repo-root source paths when the container paths don't exist.
  • Adds types-PyYAML (first direct yaml import in this service) and a black-formatting fix needed to pass pre-push checks.

Note for reviewers: this branch was created before PR #376 (public-site-content group 1) merged, so the diff currently includes #376's commits too. That will shrink automatically to just this group's changes once #376 merges — no action needed here.

Test plan

  • pytest tests/test_site_content_ingestion.py tests/test_guide_doc_ingestion.py tests/test_site_knowledge_startup.py — 26 passed
  • Verified SITE_CONTENT_DIR/GUIDE_DOCS_DIR/PRODUCT_DOC_PATH resolve to real, existing paths when running bare-metal
  • Confirmed site_chunks populates after restarting the local ai service with the path fix
  • black --check, mypy, flake8 all clean on services/ai/app

🤖 Generated with Claude Code

Closes #381

arutsh and others added 8 commits October 1, 2026 13:54
…ages (#373)

Introduce the markdown+frontmatter content contract and loader so public
copy lives in src/content/site instead of JSX, with build-time validation
that aggregates every file's errors (including fence/YAML/BOM issues).

Add a shared PublicLayout (header with mobile menu, footer, hash scroll)
and route every public marketing page through it, including / and /legal,
so footer links to /legal#privacy and #terms land on the right section.

Add a frontend lint ratchet (lint:ci, BASELINE=87) enforced in CI and the
pre-push hook, so the existing ESLint backlog can only shrink.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add About and Contact pages on the site content loader, with unconfirmed
claims (CIC status, pilot schedule, time commitment) shipped as drafts.

Move the demo-request form into DemoRequestForm with organisation type,
reporting-time and pilot-interest fields, still posting to web3forms.

Define the brand palette as Tailwind @theme tokens and use brand-* classes
in the public site instead of inline styles; brand.ts now references the
CSS variables so the hex values live in one place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…structure

- Created HomePage component to replace the deprecated LandingPage.
- Implemented legacy hash redirects for navigation.
- Added tests for HomePage to ensure proper rendering and functionality.
- Removed LandingPage.tsx and its associated tests.
- Verified that all relevant assertions were migrated to the new tests.
- Conducted a production build check to ensure stability and performance.
… design, specs and tasks

Resolves the five open questions from the skeleton proposal (reindex
trigger, guide-doc chunking/publishing, golden dataset ownership, gate
tolerance, unanswerable-question scoring) and adds design.md, delta specs
for site-knowledge-index/site-retrieval/retrieval-evaluation, and tasks.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…into change dir, add tracking block

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…x (group 1)

Adds the site_chunks table and ingestion pipeline that indexes published
site-content sections, guide docs, and PRODUCT.md into full-text-searchable
rows, synced idempotently on startup. Falls back to repo-root source paths
when the Dockerfile's container-only /app/... paths aren't present, so
bare-metal dev runs index real content instead of silently skipping it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…pe checks

black-format guide_doc_ingestion.py and add types-PyYAML, the stub package
mypy needs now that this service imports yaml directly for the first time.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
grant-flow Ready Ready Preview Oct 5, 2026 5:07pm UTC

…ic route (group 3)

Guide-doc chunks cite /guides/<doc-slug>#<heading-slug>, so the site now
renders docs/user-guide/*.md and docs/PRODUCT.md (as `product`) with H2
anchors slugified exactly like the ai service ingester. GitHub-relative doc
links are rewritten to sibling guide routes or to the repo on GitHub.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… persona boost (group 2)

ts_rank_cd-ranked full-text retrieval over site_chunks, OR-ing query
lexemes so natural-language questions needn't contain every word. CI's
ai-test job gains a postgres:15 service since ts_rank_cd can't run on
sqlite; corrects the design's claim that it already had one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…retrieval eval gate (group 4)

58-question golden dataset validated against the real corpus, a Recall@k/MRR/
unanswerable-accuracy harness with a token baseline, and baseline.yaml floors
that fail CI on any retrieval regression. Ingesters gain parse-only loaders and
the throwaway-schema Postgres helper is shared between retrieval and eval tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 7e66ac0f Deployed Oct 5, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

site-assistant-retrieval-evals: Data model and ingestion (group 1)

1 participant