Repository navigation
Make the consultation reach its LLM, document the golden's refresh rule, head config sections by path - #1832
Merged
seangalliher merged 6 commits intoNov 18, 2026
Conversation
…pairs The contract covers three items of the issue: a consultation builds its LLM request with the field the request has, the default-off golden's refresh rule is written where the next author reads it, and the config reference heads every section with its dotted path. It adds no code. Refs #1676. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ll with the fallback The consultation handler built its request with a field the request does not have, so every consultation raised an error and answered the low-confidence fallback. It now passes the question as the prompt. A response that carries an error or no text takes the same fallback instead of an empty answer at the higher confidence. A new test file adds six cases that cross the whole seam, from the asking agent through the protocol to the expert's client. Refs #1676. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… the boot-claims limits The default-off golden test now says how its digest is refreshed (by measurement, never by a new capture) and names, in the test, each field the defaults lost and each default that changed since the last refresh it records. The lineage document is corrected to match. The docstring of the boot-claims test lists all seven limits of its call-site finder with the direction of each. The golden file does not change. Refs #1676. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…s dropped The generator keyed its sections on the leaf name of the model, so it skipped three sections (the memory security flags, the duty schedule and the discovery settings) and headed 32 more with a name that other sections share. It now uses the dotted path from the system config as both the key and the heading. A new test walks the config models without the generator and pins one heading per path, in order. The reference is regenerated. Refs #1676. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rackers Adds the progress and decisions entries for the three repairs and regenerates the AD ledger. Refs #1676. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…l 17, label the checksum Review found two statements about the golden's history that the history does not support. The page named two kinds of refresh where there are three: the commit that persists the work item's value and stakes context added four keys that hold null. It also gave the original capture's observation checksum without saying whose it is. The page now names the three kinds, counts the 17 refreshes since the capture (14, two and one), and labels the checksum. The contract is corrected to match. Refs #1676. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1676
What changes
Items 2, 3, 4 and 1 of #1676 are repaired or settled here. Item 5 moves to #1658 (a note is posted there) and item 6 is decided: the config-reader check stays report-only. The four changes are independent of each other and share only the issue.
CognitiveAgent._answer_consultation(src/probos/cognitive/cognitive_agent.py) builtLLMRequest(system_prompt=..., user_message=..., tier="fast").LLMRequesthas nouser_messagefield and itsprompthas no default, so every consultation raisedTypeErrorinside the handler'stryand answered the low-confidence fallback (0.2). The request is now built withprompt=user_message;system_promptandtierare unchanged. A response the client marks failed (errorset) or that holds no text now raises inside the sametry, so the one existing fallback answers it, where it used to be returned as an empty answer at 0.6. The path is dormant: nothing insrccallsconsult, so no running code changes behaviour today.tests/test_ad594_consultation_reaches_the_llm.pyadds 6 cases that cross the whole seam (the asking agent, the realConsultationProtocol, the expert's registered handler and its client).tests/test_ad1152_agentic_correlation.pygains_DEFAULT_CONFIG_RULE(the failure message of thedefault_config_sha256comparison) and assertion lines in_off_observationthat name each deletion and the changed default since BF-924.docs/development/agentic-event-correlation.mdstates the rule and the history: the digest was refreshed 14 times since the capture, eight config changes of October that left no line in the test and six later ones that are now named. The golden file is not changed by a byte.tests/test_bf927_boot_claims.pylists all seven limitations, (a) to (g), of its call-site finder, each with its direction; the finder's code and its 12 production states are unchanged.scripts/gen_config_reference.pykeys and heads each section by its dotted path fromSystemConfig. Theseenset that was keyed on the leaf name dropped three sections (security.memory,proactive_cognitive.duty_scheduleand the top-leveldiscovery) and left 32 nested headings with a name that other sections share.docs/development/config-reference.mdis regenerated (+69 -32).tests/test_config_reference_current.pyadds one test that walks the config models without the generator and pins one heading per path, in order.DECISIONS.md,PROGRESS.mdand the regenerated AD ledger pair. The rest of the diff is the contract (prompts/bf-983-1676-consultation-golden-config-reference.md).Commits
b2c8c8194d37049c66a867a6be4bc7473162deab60df6f85Review and the broad gate
Review. Two rounds of the
Diff Reviewerongpt-6.1-sol(the model read back from the session log's usage records, including the first dispatched model). Round 1 (the whole candidate, head3162deab, 20 minutes) held properties 1 to 5 and 7 to 11, each by execution: the request constructs against the realLLMRequestwith the same system prompt text and fields butprompt; a normal response answers its text at 0.6, anerrorresponse with or without text, an empty answer and a whitespace answer each take the fallback at 0.2 with one warning carrying aRuntimeError, a raising client and astrreturn behave as before, and the stop check and admission hold are untouched (AST compare); the admission byte-equality test holds on two runs; the new file fails 5 cases and passes its guard on the base source; the golden is byte-identical, a changed default in a scratch model changes onlydefault_config_sha256and the documented refresh works, and putting two deleted fields back fails the new lines; each of the boot-claims limits (a) to (g) is shown by a minimal example against the real finder, with the 12 production states confirmed;--checkpasses, an independent walk ofSystemConfigmatches all 228 headings, with 32 renames, the three restored sections and unchanged introductions; no link in the repository targets a changed anchor; the ledger is current and 13 paths changed; an AST enumeration of 1,096 production files and 80LLMRequest(calls found no other invalid keyword and no other generator that keys sections by leaf name. Property 6 did not hold: two P3 findings, both ordinary reach, ondocs/development/agentic-event-correlation.md. The page named two kinds of golden refresh where there are three (AD-1321,0debd77b, added four keys holding null underpersisted.endpoint.work_item), and it gave the original capture's observation checksum3a95dbc8without saying whose it is. I confirmed both by enumerating every revision of the golden with a sentinel probe (my first probe treated a missing key and a null as equal and could not see the four additions: its premise is now asserted). Repair60df6f85: the page and the contract now name the three kinds, count the 17 refreshes since the capture as 14, two and one, and label the checksum as the initial capture's; the repair edits one page and the contract, no code or test. Round 2 (the delta, 8 minutes): READY, 0 findings. Every revision was classified with a sentinel probe (17 = 14 + 2 + 1, each named AD or BF on the right commit, the observation digest changed in every one), the checksum was enumerated across 2,124 tracked files (each occurrence is the capture's, labelled or dated), the contract's replacement blocks are byte-equal to the page, and the delta is exactly two files.Size findings: 0 (no size finding in either round)
The broad gate. The canonical wrapper ran twice on the frozen head
60df6f85f9b84031984801aa98ef6b4b76901f00(tree8b79ff2844c20e08346b12a14832bc8f0baf6634), after the last review and with no change between the runs:scripts/run_test_gate.py --label <label> --receipt logs/gates/<label>.receipt.json, 3 workers, on a host with nothing else running.bf983, 17:53Z to 19:27Z):1 failed, 54199 passed, 184 skipped, 245 warnings, 7 subtests passed in 1:32:50, exit 1. The one failure istests/test_ad1328_s1_shell_oracle.py::test_live_shell_agrees[S07-pwsh](the probe started 1 times, the row records 2), a test this change does not touch. The cause is in that test's probe script (ad1328_argv_probe.py), not in the code under test. The row starts two probe processes in one PowerShell pipeline and each appends one line to a shared log; on Windows an append is a seek to the end and then a write, so two writers that seek before either writes can overwrite each other. The probe's own code, run by two processes at an aligned instant, lost a line in 5 of 120 trials against a shared log and in none of 120 against a log per process (the control shows the harness loses nothing on its own). The case passes alone (3 passed in 2.6 s). It is recorded as itembf-983 F1of Follow-up: residual findings from the #1817/#1796 (BF-982, BF-965), #1803/#1806 (BF-972, BF-975), #1808 (BF-977), #1804 (BF-973), #1809 (BF-978) and #1807 (BF-976) waves #1820 (P3, test-only; its evidence, anchors and smallest repair are there), and the Architect decided that it is not repaired inside this change.bf983b, 19:33Z to 20:49Z), the same commit and tree:54200 passed, 184 skipped, 248 warnings, 7 subtests passed in 1:15:00, exit 0. The receipt validates (54,384 tests collected and reported on 3 workers) and binds the committed tree; the live-data manifest taken before and after the run is identical (54,107 files, none added, removed or changed). The count is the previous gate's (bf980: 54193 passed) plus the 7 new cases (6 for the consultation and 1 for the config reference).origin/mainwas596d0d87at the start of both runs and at the push. Run 1 is superseded by run 2.Evidence
tests/test_ad594_consultation_reaches_the_llm.py: 5 of 6 cases fail and the guard passes. C1 fails at its first assertion after the premise (the fallback records nameTypeError:LLMRequest.__init__() got an unexpected keyword argument 'user_message'); C2 (three params) and C3 fail atlen(llm.requests) == 1(0: the request raised before the client was reached); C4, the no-client guard, passes.tests/test_config_reference_current.pywith the new test, on the base generator and reference: 1 failed, 11 passed. R1 fails at index 6,'attention' != 'memory.attention', and the expected list holds three more paths than the reference.user_message=again): C1, C2 (three params) and C3, 5 cases.if False:): C2 (three params).errortest is dropped):C2[error-with-text]. E-4 (the blank-text test is dropped):C2[empty-answer]._off_observation.test_the_reference_matches_the_models. E-8 (one heading of the reference put back): R1 and that test.ad1152_agentic_off_golden.jsonis byte-identical to the base.What this pull request does not do
consult: the path stays dormant, and once something calls it the answer is raw model text at a constant 0.6 (recorded in the contract as R-B3-1, and as itembf-983 L1of Follow-up: residual findings from the #1817/#1796 (BF-982, BF-965), #1803/#1806 (BF-972, BF-975), #1808 (BF-977), #1804 (BF-973), #1809 (BF-978) and #1807 (BF-976) waves #1820).ad1152_agentic_off_golden.json), and it does not put a deleted default back to keep an old digest.GET /api/disclosure-clearancesanswers"active"while nothing calls the filter): that is a note on AD-1352: verification and alert producers that nothing calls: AD-528 ground-truth verification, AD-555/566c Bridge alert evaluators and AD-451 reconciliation get a caller (observe-first) or are marked inert #1658. It does not makescripts/check_config_readers.pya gate (item 6).bf-983 F1of Follow-up: residual findings from the #1817/#1796 (BF-982, BF-965), #1803/#1806 (BF-972, BF-975), #1808 (BF-977), #1804 (BF-973), #1809 (BF-978) and #1807 (BF-976) waves #1820; its repair is test-only and touches files this pull request does not.