From 082b40766416c74874114012b72425edc1a5433a Mon Sep 17 00:00:00 2001 From: legrab Date: Sun, 9 Aug 2026 11:13:18 +0200 Subject: [PATCH] chore: reduce ambiguity for deterministic learning flow --- AGENTS.md | 1 + CHANGELOG.md | 12 ++++ MANIFEST.txt | 18 +++--- README.md | 4 +- docs/AGENTIC_WORKFLOW_SANITY.md | 33 +++++++++++ docs/DESIGN_NOTES.md | 19 ++++++- docs/EXAMPLE_WALKTHROUGH.md | 84 ++++++++++++++++++++++++++++ docs/README.md | 1 + sample/common/agentic-flow/AGENTS.md | 10 ++++ sample/common/agentic-flow/README.md | 21 +++++++ skill-evals/agentic-cases.yaml | 17 ++++++ 11 files changed, 209 insertions(+), 11 deletions(-) create mode 100644 docs/EXAMPLE_WALKTHROUGH.md diff --git a/AGENTS.md b/AGENTS.md index 32aefa1..e40fc84 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,6 +38,7 @@ Keep the exchange conversational. Build a compact model, use one useful example - Treat installer update and preservation behavior as public contract. - Keep the framework usable across languages, build systems, repository sizes, and industries. - Never persist inferred distress, health, identity, secrets, customer data, or sensitive operational evidence. +- Record change rationale only in `docs/DESIGN_NOTES.md` (why, chronological, one section per version) and `CHANGELOG.md` (what, per release). Do not create a separate status, hardening, or implementation-tracking document for a change — fold the rationale into a `DESIGN_NOTES.md` section and the entries into `CHANGELOG.md`. A standalone tracking document goes stale the moment the next change lands; these two files are already the maintained, chronological record. ## External references diff --git a/CHANGELOG.md b/CHANGELOG.md index f54bc00..2d7bbdd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,18 @@ ## Unreleased +### Added + +- Two behavioral routing rules in `agentic-flow/AGENTS.md`: treat a user's proposed implementation or architecture as a hypothesis to check against repository evidence rather than a specification to endorse, and distinguish repository-resolvable ambiguity (inspect first) from user-intent ambiguity (ask the smallest useful question). +- A compact visual decision model for that routing in `agentic-flow/README.md`. +- Two regression scenarios (proposed design, open-ended ambiguity) in `docs/AGENTIC_WORKFLOW_SANITY.md`, and matching cases in `skill-evals/agentic-cases.yaml`. +- `docs/EXAMPLE_WALKTHROUGH.md`, a single compact worked example showing the new routing in practice. +- A "v1.0 hardening: proposal and ambiguity routing" entry in `docs/DESIGN_NOTES.md` recording the tightened, applied 1.0 behavioral contract in place — change rationale and current-state tracking live only in `DESIGN_NOTES.md` and this changelog, not in a separate status document. + +### Changed + +- README wording from "learning is the default behavior" to "learning-aware behavior is enabled by default," with routine work stated explicitly as staying routine. + ## 1.0.0 First tagged production release. `v0.1.0`/`v0.1.1` on GitHub were release-automation smoke tests and are not part of this line; this release picks up the version sequence documented below, starting at `0.8.0`. diff --git a/MANIFEST.txt b/MANIFEST.txt index 95961c8..6d372db 100644 --- a/MANIFEST.txt +++ b/MANIFEST.txt @@ -1,13 +1,13 @@ .gitattributes 40 .gitignore 324 -AGENTS.md 2524 -CHANGELOG.md 17573 -docs/AGENTIC_WORKFLOW_SANITY.md 7795 +AGENTS.md 2995 +CHANGELOG.md 18727 +docs/AGENTIC_WORKFLOW_SANITY.md 9207 docs/ARCHITECTURE.md 6009 -docs/DESIGN_NOTES.md 27644 +docs/DESIGN_NOTES.md 31120 docs/EDUCATION_MODEL.md 5452 docs/INITIALIZE_LEARNING_FLOW.md 10880 -docs/README.md 4185 +docs/README.md 4320 docs/references/REFERENCE_INTEGRATION.md 4551 docs/references/REFERENCE_REVIEW_AWESOME_AGENT_SKILLS.md 2317 docs/references/REFERENCE_REVIEW_BEST_OF_AGENT_HARNESSES.md 2540 @@ -16,7 +16,7 @@ docs/references/REFERENCE_REVIEW_LEARNING_FLOW_ADJUSTMENT.md 9505 docs/references/REFERENCE_REVIEW_LITT.md 2069 docs/references/REFERENCE_REVIEW_POCOK.md 2155 LICENSE 2213 -README.md 8553 +README.md 9012 sample/common/.agents/skills/agentic-workflow/SKILL.md 2686 sample/common/.agents/skills/learn-anything/agents/openai.yaml 246 sample/common/.agents/skills/learn-anything/SKILL.md 1756 @@ -33,13 +33,13 @@ sample/common/.agents/skills/structured-change/templates/exploration.md 368 sample/common/agentic-flow/.managed-files 170 sample/common/agentic-flow/.managed-skills 86 sample/common/agentic-flow/.template-version 6 -sample/common/agentic-flow/AGENTS.md 4121 +sample/common/agentic-flow/AGENTS.md 5328 sample/common/agentic-flow/CONFIGURE.md 4475 sample/common/agentic-flow/DECISIONS.md 963 sample/common/agentic-flow/EDUCATION.md 5232 sample/common/agentic-flow/LEARN.md 1822 sample/common/agentic-flow/LOCAL.md 3513 -sample/common/agentic-flow/README.md 2039 +sample/common/agentic-flow/README.md 2907 sample/common/agentic-flow/REFERENCE_INTEGRATION.md 2183 sample/common/agentic-flow/ROOT_INTEGRATION.md 3086 sample/common/agentic-flow/SETTINGS.md 544 @@ -100,7 +100,7 @@ scripts/install.bat 1477 scripts/install.ps1 40236 scripts/install.sh 34910 scripts/README.md 6771 -skill-evals/agentic-cases.yaml 7109 +skill-evals/agentic-cases.yaml 8332 skill-evals/conversation-cases.yaml 3653 skill-evals/full-cases.yaml 4248 skill-evals/minimal-cases.yaml 3208 diff --git a/README.md b/README.md index 1ea8299..caf185c 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ Codebase Learning Flow configures a host coding agent. It provides repository-local instructions, focused skills, learning surfaces, and private local continuity. It does **not** provide an agent runtime, sandbox, retry engine, or background worker. > [!IMPORTANT] -> **Learning is the default behavior after installation.** Users do not need to explain the learning method. A request such as `help me understand ` automatically selects the appropriate learning route, keeps the exchange conversational, builds a compact model, uses a useful example or experiment, and checks back for understanding when that adds value. +> **Learning-aware behavior is enabled by default after installation** — that does not mean every task becomes a lesson. Routine work stays routine. A request such as `help me understand ` automatically selects the appropriate learning route, keeps the exchange conversational, builds a compact model, uses a useful example or experiment, and checks back for understanding when that adds value. A proposed design or approach is treated as a hypothesis worth checking against repository evidence, not a specification to endorse. > [!WARNING] > The regulatory extension is a reasoning and workflow aid, not a compliance determination or substitute for qualified regulatory or quality expertise. @@ -46,6 +46,8 @@ The installed `AGENTS.md` connects the host repository to `agentic-flow`. The co - `help me understand ...`, `explain ...`, `teach me ...` → `learn-anything` for general topics - questions about the current codebase → repository learning - implementation work → normal delivery, with learning reinforcement when useful +- a proposed design or approach → treated as a hypothesis, checked against repository evidence before being endorsed or implemented +- an open-ended consequential question → repository evidence is inspected first; the user is asked only the smallest question evidence can't answer - one consequential or ambiguous change → `structured-change` alongside the active route - regulatory reasoning → the optional regulatory lens when installed diff --git a/docs/AGENTIC_WORKFLOW_SANITY.md b/docs/AGENTIC_WORKFLOW_SANITY.md index f632379..990ef5f 100644 --- a/docs/AGENTIC_WORKFLOW_SANITY.md +++ b/docs/AGENTIC_WORKFLOW_SANITY.md @@ -164,6 +164,39 @@ Expected behavior: Failure signal: automatic dumping of the entire session into `learning-history.md`, or silently creating durable knowledge without a reuse justification. +### 10. Proposed design or approach + +**Prompt:** Here's a task, and I'm planning to implement it by doing X. Anything to add? + +Expected behavior: + +- recognize the proposal as a hypothesis, not a specification; +- inspect relevant repository evidence before responding; +- name the assumptions the proposal depends on and which are unverified; +- surface a missing boundary, risk, or credible alternative when one exists; +- recommend or implement only after that check, and only within the requested scope. + +Failure signal: polishing or implementing the proposal as stated, or turning the check into a generic architecture lecture unconnected to repository evidence. + +### 11. Open-ended consequential ambiguity + +**Prompt:** What's the best way to redesign this subsystem? + +Expected behavior: + +- identify that the answer depends on the user's intent, scope, or tradeoffs; +- ask the smallest question that separates the materially different answers; +- do not invent the missing preference. + +**Contrast prompt:** Should this validation live in the controller or the service? + +Expected behavior when repository convention already answers it: + +- inspect existing ownership and validation conventions; +- answer from that evidence instead of asking. + +Failure signal: asking a broad clarifying question the repository could already answer, or silently choosing an architecture when the user's intent was actually required. + ## Review checklist For each scenario, review the agent interaction for: diff --git a/docs/DESIGN_NOTES.md b/docs/DESIGN_NOTES.md index d7d84ba..669bbc7 100644 --- a/docs/DESIGN_NOTES.md +++ b/docs/DESIGN_NOTES.md @@ -138,6 +138,21 @@ Regulatory-specific knowledge (traceability, validation, risk management, audit Engineering judgment content the source proposed (anti-overengineering guidance, AI-collaboration indicators, modernization, maintainability, testing, architecture, dependency management, documentation) is general-purpose, not regulatory-specific, and lives inside `structured-change`'s own knowledge folder instead. +## v1.0 hardening: proposal and ambiguity routing + +A pre-1.0 sanity review found the remaining gap to be behavioral routing, not architecture. The framework already had strong repository discovery, progressive learning, evidence-based reasoning, structured change, and local/shared learning continuity; what it lacked was an explicit rule distinguishing a request to implement from a request to challenge a proposed approach, and an explicit rule distinguishing an ambiguity the repository can resolve from one only the user can resolve. Without those, a proposal offered together with a task could be interpreted as a request to execute rather than a request to challenge, and an open-ended consequential question ("what's the best way to redesign X?") had no rule stopping the agent from silently picking an interpretation. + +Two rules closed that gap, added once to `agentic-flow/AGENTS.md` (the routing contract every task reads first) rather than duplicated into each skill: + +- **Proposals are hypotheses, not specifications.** When the user offers a proposed implementation, architecture, or approach and asks for feedback, identify its material assumptions, check them against repository evidence, and surface a missing boundary, risk, or alternative before recommending or implementing it. An explicit, narrowly scoped instruction ("do exactly X") does not require this challenge. +- **Repository ambiguity vs. user-intent ambiguity.** If evidence can resolve a consequential ambiguity, inspect first. If the missing piece is the user's intent, scope, tradeoff, or authority, ask the smallest useful question instead of deciding for them. + +Both are phrased as behavioral rules applied when a condition is encountered, not a sequence of steps to complete: new evidence can change the route mid-task, including abandoning the initial proposal. This is the same distinction the rest of this document already makes for delivery work — deterministic behavior, not a deterministic trajectory. + +The rest of the pass fit the same "one canonical owner" discipline as everything else in this section: a compact visual decision model landed once in `agentic-flow/README.md` rather than being restated in prose; two new regression scenarios landed in `docs/AGENTIC_WORKFLOW_SANITY.md` (proposed design, open-ended ambiguity) with matching cases in `skill-evals/agentic-cases.yaml`; one compact worked example landed in `docs/EXAMPLE_WALKTHROUGH.md`; and the root `README.md`'s framing changed from "learning is the default behavior" to "learning-aware behavior is enabled by default," making explicit that routine work stays routine. + +A larger set of proposed improvements (lightweight learning artifacts, incremental learning rounds, an explicit inspectable system model, a structure step between design and planning, branching alternative designs) was reviewed and deliberately deferred rather than folded in. Each would require real design and evaluation work to avoid becoming ceremony, which is exactly what this document's "smallest coherent hardening pass" standard rules out for a single pass; the reasoning and file-level implementation detail for each lives in the post-1.0 backlog document delivered alongside this release, not inside this repository, so an optional future improvement doesn't read as a committed roadmap. + ## General agentic loop ```text @@ -254,7 +269,9 @@ Design decisions specific to this boundary: - committed personal transcripts and hypothesis diaries; - a dashboard, database, vector store, orchestration service, or LMS; - a proliferating `explorations/`/`designs/`/`decisions/`-per-item directory scaffold in place of the existing durable-file surfaces; -- mandatory evaluation-matrix scoring or commit traceability blocks for ordinary, low-risk work. +- mandatory evaluation-matrix scoring or commit traceability blocks for ordinary, low-risk work; +- a design challenge for a one-line or obviously reversible change; +- a clarifying question about something repository evidence already answers. ## Final review checklist diff --git a/docs/EXAMPLE_WALKTHROUGH.md b/docs/EXAMPLE_WALKTHROUGH.md new file mode 100644 index 0000000..c161f4a --- /dev/null +++ b/docs/EXAMPLE_WALKTHROUGH.md @@ -0,0 +1,84 @@ +# Worked example: discovery and design challenge + +This is a single compact, fictional example showing what the behavior in +`AGENTIC_WORKFLOW_SANITY.md` scenario 10 looks like end to end. It is not +installed into a target repository; it exists here to make the abstract +routing rules concrete. + +## Setting + +A legacy .NET order-approval subsystem. A developer asks the agent to add +retry handling around an external ERP call and says: *"I'll just wrap the +existing client call with retries and move on. Anything to add?"* + +## What repository learning establishes + +```mermaid +flowchart LR + Operator --> Endpoint[Approval endpoint] + Endpoint --> Service[OrderService] + Service --> Rules[ApprovalRules] + Rules --> Proc[SQL approval procedure] + Proc --> Audit[Audit record] + Proc --> ERP[External ERP] +``` + +- **Invariant:** an approved order cannot be modified without re-entering approval. +- **Evidence:** `OrderService`, `ApprovalRules`, the approval stored procedure, regression tests, the ERP client. +- **Unknown:** whether the ERP client is idempotent, i.e. whether a retried submission can duplicate an order. +- **Control boundary:** production deployment and approval stay human-controlled. + +## Why this is not just implemented as proposed + +The proposal ("wrap the call with retries") is treated as a hypothesis, not +a specification: + +```text +Proposal: + Retry the ERP call on failure. + +Known: + ERP submission can fail transiently. + +Unknown: + Whether the ERP operation is idempotent. + +Consequence if wrong: + A retry could duplicate an order submission. + +Evidence needed: + ERP contract, request-identifier behavior, existing retry handling elsewhere + in the codebase. + +Design implication: + Retry cannot be considered safe until duplicate-submission semantics are + established. This is a repository-evidence question, not a user-intent + question, so it gets inspected before anything is asked or built. +``` + +## Resulting response shape + +```text +Before adding retries, one assumption needs checking: does a repeated ERP +submission create a duplicate order? I found evidence that submission +includes an order ID [evidence], but no confirmation the ERP treats it as +a dedupe key. + +If the ERP is idempotent, wrapping the call is enough. If not, retries need +a submission-identity check first, which is a larger change than "wrap the +call." + +I can inspect the ERP contract now, or if you already know it's idempotent, +say so and I'll proceed with the narrow retry. +``` + +Note what did *not* happen: no learning session was created, no mandatory +design document, no interrogation of unrelated parts of the system. One +material assumption was identified, checked against evidence where possible, +and only the remaining consequential gap was raised. + +## Related + +- `docs/AGENTIC_WORKFLOW_SANITY.md` — scenario 10 (proposed design) and 11 (open-ended ambiguity). +- `sample/common/agentic-flow/README.md` — the general routing diagram this example follows. +- `sample/common/agentic-flow/AGENTS.md` — the rule text ("Proposals are hypotheses, not specifications"). diff --git a/docs/README.md b/docs/README.md index 777be71..0871f8a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -40,6 +40,7 @@ be adopted into an existing agentic workflow without replacing that workflow. | understand the educational direction | [`EDUCATION_MODEL.md`](EDUCATION_MODEL.md) | | understand ownership and historical architecture decisions | [`DESIGN_NOTES.md`](DESIGN_NOTES.md) | | reconstruct or adapt the framework | [`INITIALIZE_LEARNING_FLOW.md`](INITIALIZE_LEARNING_FLOW.md) | +| see the proposal-challenge and ambiguity routing behavior in a worked example | [`EXAMPLE_WALKTHROUGH.md`](EXAMPLE_WALKTHROUGH.md) | | integrate an external source | [`references/REFERENCE_INTEGRATION.md`](references/REFERENCE_INTEGRATION.md) | | inspect installer behavior | [`../scripts/README.md`](../scripts/README.md) | diff --git a/sample/common/agentic-flow/AGENTS.md b/sample/common/agentic-flow/AGENTS.md index 7889774..5b6e5c8 100644 --- a/sample/common/agentic-flow/AGENTS.md +++ b/sample/common/agentic-flow/AGENTS.md @@ -66,6 +66,16 @@ Ask only when a consequential choice is unresolved, evidence cannot distinguish An explicit implementation request permits ordinary reversible work within scope. It does not permit destructive Git operations, publishing, release actions, secret access, or unrelated changes. +### Proposals are hypotheses, not specifications + +When the user presents a proposed implementation, architecture, or approach and asks for feedback, additions, or validation, treat it as a hypothesis rather than endorsing, polishing, or implementing it outright. Identify the assumptions it depends on, check the ones repository evidence can confirm or falsify, and surface a missing boundary, risk, or credible alternative before recommending it. Use `structured-change` when the decision is consequential or hard to reverse; otherwise fold the check into the active route. + +An explicit, narrowly scoped instruction ("do exactly X") does not require this challenge; treat it as sufficient intent and proceed. + +### Repository ambiguity vs. user-intent ambiguity + +When a consequential question has more than one materially different answer, do not silently pick one. Inspect first if repository evidence can settle it. Ask only the smallest useful question when what's missing is the user's intent, scope, tradeoff, or authority — not because the repository is unfamiliar. This is a routing rule, not a fixed sequence: new evidence can change the route mid-task, including abandoning an initial proposal. + ## Planning and records - Small tasks need no formal plan. diff --git a/sample/common/agentic-flow/README.md b/sample/common/agentic-flow/README.md index ea7605d..de5c60f 100644 --- a/sample/common/agentic-flow/README.md +++ b/sample/common/agentic-flow/README.md @@ -29,6 +29,27 @@ flowchart TB Start with `AGENTS.md`. Use balanced defaults from `SETTINGS.md` unless configuration matters. +## Proposal and ambiguity routing + +A proposed design and an open-ended consequential question are handled the same way: check what evidence can settle, ask only what it can't. + +```mermaid +flowchart TD + Q[Proposal or consequential question] --> E{Evidence can resolve it?} + E -->|Yes| I[Inspect, then proceed] + E -->|No| U{Depends on user intent, scope, or authority?} + U -->|Yes| A[Ask the smallest useful question] + U -->|No| I + I --> D{Is it a proposed design or approach?} + D -->|Yes| H[Treat as hypothesis: test assumptions, surface alternatives] + D -->|No| Act[Continue the active route] + H --> R{Consequential or hard to reverse?} + R -->|Yes| SC[structured-change] + R -->|No| Act +``` + +New evidence can change the route mid-task; this is a set of behavioral rules, not a plan to complete. See `AGENTS.md` for the full rule text. + ## Supporting guides | Guide | Purpose | diff --git a/skill-evals/agentic-cases.yaml b/skill-evals/agentic-cases.yaml index c8568b7..e1b1b1a 100644 --- a/skill-evals/agentic-cases.yaml +++ b/skill-evals/agentic-cases.yaml @@ -108,3 +108,20 @@ cases: expected_behavior: safe_framework_update must_include: [add managed EDUCATION.md, refresh managed instructions and skills, preserve MAP.md TAKEAWAYS.md SETTINGS.md .local and unrelated skills] must_not: [overwrite repository-authored learning, delete private state, downgrade full to minimal] + + - prompt: > + We need to add retry handling to the legacy integration. I'm planning to wrap the + existing client call with retries and move on. Anything to add? + expected_behavior: challenge_proposed_design + must_include: [inspect current retry and idempotency semantics, challenge assumptions, identify missing failure or ownership boundary, distinguish repository evidence from inference, recommend only after checking evidence] + must_not: [simply endorse the proposal, immediately implement it, generic architecture advice without repository evidence] + + - prompt: What's the best way to redesign this authentication subsystem? + expected_behavior: ask_clarifying_question + must_include: [identify the missing decision dimension, ask the smallest useful question] + must_not: [silently choose an architecture, invent requirements, begin implementation] + + - prompt: Should this validation live in the controller or the service? + expected_behavior: inspect_before_asking + must_include: [inspect existing ownership and validation conventions, use repository evidence if decisive, explain the decision from evidence] + must_not: [ask the user unnecessarily, invent a product preference]