docs: demonstrate retrieved-learning attribution and quality

[yyiilluu]

Summary

• Make reporting every injected profile/playbook the recommended notebook integration pattern, so retrieval quality can be evaluated rather than inferred from session success.
• Complete the retrieve → inject → publish stable IDs → grade → inspect loop using the same retained context for the prompt and retrieved_learnings.
• Explain relevance, positive/negative/neutral impact, judge reasons, sampling, coverage, and ungraded or missing results.

Changes

• Update Quickstart, Interactions, Profiles, Playbooks, Configuration, and Simulation, plus the notebook README.
• Generate context-aware assistant responses before publishing their actual learning references; leave seed turns without attribution.
• Demonstrate grade_on_demand and get_retrieved_learning_evaluation_results without assuming all learnings helped.
• Correct the quickstart package name and simulation minimum cluster size; format notebooks and clear stale outputs.

The raw diff exceeds 500 lines because these are standalone notebook tutorials and include JSON formatting/output cleanup. No backend API behavior changes.

Test Plan

• Notebook JSON/schema validation and compilation of every code cell.
• Ruff lint and formatting for all changed notebooks.
• 16 focused evaluation tests covering the real SQLite runner, verdict readback, and on-demand grading/cache contracts (LLM calls mocked).
• 12 OSS notebook workflow checks in a temporary harness using the real client/schema layer with mocked transport and generation, covering complete, degraded, and empty retrieval scenarios where applicable.
• Full live LLM notebook runs were not performed. Generation and grading incur provider cost; run against a disposable demo database because existing cleanup cells delete connected data.

Summary by CodeRabbit

• New Features

  • Added notebook examples for retrieving approved profiles and playbooks, using them in generated responses, and recording attribution.
  • Added retrieved-learning quality grading, including status, coverage, relevance, impact, and per-learning verdicts.
  • Enhanced the real-world simulation with scoped retrieval, improved filtering, agent version tracking, and quality reporting.

• Documentation

  • Expanded notebook guidance for retrieval, attribution, evaluation, sampling, costs, and cleanup.
  • Updated setup instructions to install notebook-specific extras.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: dd5bfb46-275a-47d4-a0e9-e57b4588d011

📥 Commits

Reviewing files that changed from the base of the PR and between 81004c7 and a488a29.

📒 Files selected for processing (7)

• notebooks/00_quickstart.ipynb
• notebooks/01_interactions.ipynb
• notebooks/02_profiles.ipynb
• notebooks/03_playbook.ipynb
• notebooks/04_configuration.ipynb
• notebooks/06_real_world_simulation.ipynb
• notebooks/README.md

---

📝 Walkthrough

Walkthrough

The notebooks now demonstrate profile and playbook retrieval, OpenAI-generated responses, retrieved_learnings attribution, response publishing, synchronous grading, verdict inspection, and updated evaluation configuration guidance.

Changes

Retrieval and evaluation demonstrations

Layer / File(s)	Summary
Profile retrieval and response evaluation notebooks/00_quickstart.ipynb, notebooks/01_interactions.ipynb, notebooks/02_profiles.ipynb	The notebooks retrieve profiles, inject profile context into OpenAI prompts, publish generated responses with stable learning IDs, and inspect grading status, coverage, relevance, and impact verdicts.
Playbook-grounded simulation notebooks/03_playbook.ipynb, notebooks/06_real_world_simulation.ipynb	The notebooks retrieve playbooks or filtered profiles, generate conversation responses, publish shared learning references, and report persisted evaluation results.
Evaluation configuration and notebook guidance notebooks/04_configuration.ipynb, notebooks/README.md	The configuration example sets retrieved-learning sampling to 1.0. The README documents installation, attribution, grading, monitoring, costs, and cleanup.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Notebook
  participant Retrieval
  participant OpenAI
  participant Reflexio
  Notebook->>Retrieval: retrieve profiles or playbooks
  Retrieval-->>Notebook: context and stable learning IDs
  Notebook->>OpenAI: generate response with retrieved context
  OpenAI-->>Notebook: response text
  Notebook->>Reflexio: publish response with retrieved_learnings
  Notebook->>Reflexio: run grade_on_demand and read verdicts

Loading

Suggested reviewers: yilu331

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/retrieved-learning-examples

Warning

Your free Security trial is over. An organization admin can activate Security or dismiss this notice.

---

_{Comment @coderabbitai help to get the list of available commands.}