Skip to content

Make the consultation reach its LLM, document the golden's refresh rule, head config sections by path - #1832

Merged
seangalliher merged 6 commits into
mainfrom
fix/bf983-consultation-golden-config-reference-20261118
Nov 18, 2026
Merged

seangalliher merged 6 commits into
mainfrom
fix/bf983-consultation-golden-config-reference-20261118

Conversation

@seangalliher

@seangalliher seangalliher commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Refs #1676

What changes

Items 2, 3, 4 and 1 of #1676 are repaired or settled here. Item 5 moves to #1658 (a note is posted there) and item 6 is decided: the config-reader check stays report-only. The four changes are independent of each other and share only the issue.

  • Item 2: a consultation reaches its LLM. CognitiveAgent._answer_consultation (src/probos/cognitive/cognitive_agent.py) built LLMRequest(system_prompt=..., user_message=..., tier="fast"). LLMRequest has no user_message field and its prompt has no default, so every consultation raised TypeError inside the handler's try and answered the low-confidence fallback (0.2). The request is now built with prompt=user_message; system_prompt and tier are unchanged. A response the client marks failed (error set) or that holds no text now raises inside the same try, so the one existing fallback answers it, where it used to be returned as an empty answer at 0.6. The path is dormant: nothing in src calls consult, so no running code changes behaviour today. tests/test_ad594_consultation_reaches_the_llm.py adds 6 cases that cross the whole seam (the asking agent, the real ConsultationProtocol, the expert's registered handler and its client).
  • Item 3: the AD-1152 golden says how it is refreshed. tests/test_ad1152_agentic_correlation.py gains _DEFAULT_CONFIG_RULE (the failure message of the default_config_sha256 comparison) and assertion lines in _off_observation that name each deletion and the changed default since BF-924. docs/development/agentic-event-correlation.md states the rule and the history: the digest was refreshed 14 times since the capture, eight config changes of October that left no line in the test and six later ones that are now named. The golden file is not changed by a byte.
  • Item 1: the limits are listed. The docstring of tests/test_bf927_boot_claims.py lists all seven limitations, (a) to (g), of its call-site finder, each with its direction; the finder's code and its 12 production states are unchanged.
  • Item 4: one section per path. scripts/gen_config_reference.py keys and heads each section by its dotted path from SystemConfig. The seen set that was keyed on the leaf name dropped three sections (security.memory, proactive_cognitive.duty_schedule and the top-level discovery) and left 32 nested headings with a name that other sections share. docs/development/config-reference.md is regenerated (+69 -32). tests/test_config_reference_current.py adds one test that walks the config models without the generator and pins one heading per path, in order.
  • Trackers. DECISIONS.md, PROGRESS.md and the regenerated AD ledger pair. The rest of the diff is the contract (prompts/bf-983-1676-consultation-golden-config-reference.md).

Commits

Commit Subject
b2c8c819 Add the contract for the consultation, golden and config reference repairs
4d37049c Build a consultation's LLM request with prompt and answer a failed call with the fallback
66a867a6 State the golden's refresh rule and backfill its deleted fields; list the boot-claims limits
be4bc747 Head each config reference section with its path so that no section is dropped
3162deab Record the consultation, golden and config reference changes in the trackers
60df6f85 Repair the lineage page: name three kinds of golden refresh, count all 17, label the checksum

Review and the broad gate

Review. Two rounds of the Diff Reviewer on gpt-6.1-sol (the model read back from the session log's usage records, including the first dispatched model). Round 1 (the whole candidate, head 3162deab, 20 minutes) held properties 1 to 5 and 7 to 11, each by execution: the request constructs against the real LLMRequest with the same system prompt text and fields but prompt; a normal response answers its text at 0.6, an error response with or without text, an empty answer and a whitespace answer each take the fallback at 0.2 with one warning carrying a RuntimeError, a raising client and a str return behave as before, and the stop check and admission hold are untouched (AST compare); the admission byte-equality test holds on two runs; the new file fails 5 cases and passes its guard on the base source; the golden is byte-identical, a changed default in a scratch model changes only default_config_sha256 and the documented refresh works, and putting two deleted fields back fails the new lines; each of the boot-claims limits (a) to (g) is shown by a minimal example against the real finder, with the 12 production states confirmed; --check passes, an independent walk of SystemConfig matches all 228 headings, with 32 renames, the three restored sections and unchanged introductions; no link in the repository targets a changed anchor; the ledger is current and 13 paths changed; an AST enumeration of 1,096 production files and 80 LLMRequest( calls found no other invalid keyword and no other generator that keys sections by leaf name. Property 6 did not hold: two P3 findings, both ordinary reach, on docs/development/agentic-event-correlation.md. The page named two kinds of golden refresh where there are three (AD-1321, 0debd77b, added four keys holding null under persisted.endpoint.work_item), and it gave the original capture's observation checksum 3a95dbc8 without saying whose it is. I confirmed both by enumerating every revision of the golden with a sentinel probe (my first probe treated a missing key and a null as equal and could not see the four additions: its premise is now asserted). Repair 60df6f85: the page and the contract now name the three kinds, count the 17 refreshes since the capture as 14, two and one, and label the checksum as the initial capture's; the repair edits one page and the contract, no code or test. Round 2 (the delta, 8 minutes): READY, 0 findings. Every revision was classified with a sentinel probe (17 = 14 + 2 + 1, each named AD or BF on the right commit, the observation digest changed in every one), the checksum was enumerated across 2,124 tracked files (each occurrence is the capture's, labelled or dated), the contract's replacement blocks are byte-equal to the page, and the delta is exactly two files.

Size findings: 0 (no size finding in either round)

The broad gate. The canonical wrapper ran twice on the frozen head 60df6f85f9b84031984801aa98ef6b4b76901f00 (tree 8b79ff2844c20e08346b12a14832bc8f0baf6634), after the last review and with no change between the runs: scripts/run_test_gate.py --label <label> --receipt logs/gates/<label>.receipt.json, 3 workers, on a host with nothing else running.

  • Run 1 (label bf983, 17:53Z to 19:27Z): 1 failed, 54199 passed, 184 skipped, 245 warnings, 7 subtests passed in 1:32:50, exit 1. The one failure is tests/test_ad1328_s1_shell_oracle.py::test_live_shell_agrees[S07-pwsh] (the probe started 1 times, the row records 2), a test this change does not touch. The cause is in that test's probe script (ad1328_argv_probe.py), not in the code under test. The row starts two probe processes in one PowerShell pipeline and each appends one line to a shared log; on Windows an append is a seek to the end and then a write, so two writers that seek before either writes can overwrite each other. The probe's own code, run by two processes at an aligned instant, lost a line in 5 of 120 trials against a shared log and in none of 120 against a log per process (the control shows the harness loses nothing on its own). The case passes alone (3 passed in 2.6 s). It is recorded as item bf-983 F1 of Follow-up: residual findings from the #1817/#1796 (BF-982, BF-965), #1803/#1806 (BF-972, BF-975), #1808 (BF-977), #1804 (BF-973), #1809 (BF-978) and #1807 (BF-976) waves #1820 (P3, test-only; its evidence, anchors and smallest repair are there), and the Architect decided that it is not repaired inside this change.
  • Run 2 (label bf983b, 19:33Z to 20:49Z), the same commit and tree: 54200 passed, 184 skipped, 248 warnings, 7 subtests passed in 1:15:00, exit 0. The receipt validates (54,384 tests collected and reported on 3 workers) and binds the committed tree; the live-data manifest taken before and after the run is identical (54,107 files, none added, removed or changed). The count is the previous gate's (bf980: 54193 passed) plus the 7 new cases (6 for the consultation and 1 for the config reference).

origin/main was 596d0d87 at the start of both runs and at the push. Run 1 is superseded by run 2.

Evidence

  • Tests first, on the base source (each new file ran before the change it tests):
    • tests/test_ad594_consultation_reaches_the_llm.py: 5 of 6 cases fail and the guard passes. C1 fails at its first assertion after the premise (the fallback records name TypeError: LLMRequest.__init__() got an unexpected keyword argument 'user_message'); C2 (three params) and C3 fail at len(llm.requests) == 1 (0: the request raised before the client was reached); C4, the no-client guard, passes.
    • tests/test_config_reference_current.py with the new test, on the base generator and reference: 1 failed, 11 passed. R1 fails at index 6, 'attention' != 'memory.attention', and the expected list holds three more paths than the reference.
  • Mutation campaign of record (8 rows over the tree that holds the three code milestones; each file is restored byte for byte and its blob checked; the unmutated files pass 6, 41 and 12 cases first). Every row is killed, every named killer fails, and no other case fails:
    • E-1 (the request passes user_message= again): C1, C2 (three params) and C3, 5 cases.
    • E-2 (the failed-response test is if False:): C2 (three params).
    • E-3 (the error test is dropped): C2[error-with-text]. E-4 (the blank-text test is dropped): C2[empty-answer].
    • E-5 (a deleted field is put back in the observation): the three AD-1152 cases that run _off_observation.
    • E-6 and E-7 (the generator recurses with, or heads a section by, the leaf name): test_the_reference_matches_the_models. E-8 (one heading of the reference put back): R1 and that test.
  • Focused gates on the real tree: 568 passed, 1 skipped (the POSIX-only skip) in 11 minutes, over the 18 files of the four milestones' gates; the gate preflight (config facade, ledger, seam contracts, architecture principles, store registry, compile) exits 0.
  • Blob identity. Every blob of the three code milestones equals a tool-applied oracle's (the contract's SEARCH/REPLACE blocks applied by a script to a clean scratch tree), on the base the batch was drafted on and again on the merged base; the committed trees equal the oracle's trees but for the regenerated ledger pair.
  • Golden file. ad1152_agentic_off_golden.json is byte-identical to the base.

What this pull request does not do

seangalliher and others added 6 commits November 18, 2026 09:51
…pairs

The contract covers three items of the issue: a consultation builds its LLM
request with the field the request has, the default-off golden's refresh rule is
written where the next author reads it, and the config reference heads every
section with its dotted path. It adds no code.

Refs #1676.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ll with the fallback

The consultation handler built its request with a field the request does not
have, so every consultation raised an error and answered the low-confidence
fallback. It now passes the question as the prompt. A response that carries an
error or no text takes the same fallback instead of an empty answer at the
higher confidence. A new test file adds six cases that cross the whole seam,
from the asking agent through the protocol to the expert's client.

Refs #1676.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… the boot-claims limits

The default-off golden test now says how its digest is refreshed (by
measurement, never by a new capture) and names, in the test, each field the
defaults lost and each default that changed since the last refresh it records.
The lineage document is corrected to match. The docstring of the boot-claims
test lists all seven limits of its call-site finder with the direction of each.
The golden file does not change.

Refs #1676.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…s dropped

The generator keyed its sections on the leaf name of the model, so it skipped
three sections (the memory security flags, the duty schedule and the discovery
settings) and headed 32 more with a name that other sections share. It now uses
the dotted path from the system config as both the key and the heading. A new
test walks the config models without the generator and pins one heading per
path, in order. The reference is regenerated.

Refs #1676.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rackers

Adds the progress and decisions entries for the three repairs and regenerates
the AD ledger.

Refs #1676.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…l 17, label the checksum

Review found two statements about the golden's history that the history does
not support. The page named two kinds of refresh where there are three: the
commit that persists the work item's value and stakes context added four keys
that hold null. It also gave the original capture's observation checksum
without saying whose it is. The page now names the three kinds, counts the 17
refreshes since the capture (14, two and one), and labels the checksum. The
contract is corrected to match.

Refs #1676.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant