feat(wiki): harden the maintainer/writer loop for long runs - #23
Merged
Merged
Conversation
…ed long writes LiteLLM's `request_timeout` defaults to the sentinel 6000, and chat `completion()` maps that sentinel down to COMPLETION_HTTP_FALLBACK_SECONDS (600) whenever the caller passes no explicit timeout. We passed none, so every LLM call inherited a 600s client deadline. Ample for recall, but a wiki writer regenerating a 50k-char body on a self-hosted 27B exceeds it: LiteLLM times out, retries twice (3 x 600 = 1800s), and the scheduler logs `maintain: failed` on a 30-minute cadence while vLLM is still completing the request. The work is computed and then discarded — vLLM reported 271 requests that began prefill with no finish and zero aborts. Fixed at the BrainDB layer rather than via the REQUEST_TIMEOUT env var, which is global and has a trap: setting it to exactly 6000 is a no-op because that IS the sentinel. `ModelSettings.extra_args` is forwarded verbatim into the LiteLLM call, so `timeout` resolves ahead of the fallback. `extra_args` is a transport deadline only — it never enters the request body and cannot steer the model, unlike `output_type` / `tool_choice`, which stay unset deliberately (see the module docstring). Hosted providers finish far inside 4800s and are unaffected; the value is a ceiling, not a delay.
Neither agent could see how big a wiki was. The maintainer's catalog was
{id, canonical_name}, so every candidate target looked equally empty and
`attach` was always a reasonable answer. The writer's prompt named only its
own page, so "keep this dense, that detail belongs elsewhere" was not
expressible -- there was no elsewhere it could name.
Add size, and give the writer a short list of neighbouring pages:
- list_active_wikis(): + length(e.content) AS char_count, so each catalog
entry carries its current size.
- list_related_wikis(): wikis one hop from the members being written,
bounded by LIMIT. Deliberately NOT the full catalog -- the writer only
needs to know which neighbours exist, and the catalog grows without bound.
Returns names and sizes only, never ids: the writer must not hold a peer
id it could cite, since reconcile_summarises_additive() would then mint a
wiki->wiki relation.
- Skipped entirely in consolidate mode, where entity_ids holds the DUPLICATE
WIKI ids -- listing them would hand the writer the pages it must absorb
and invite it to link out instead of merging.
- Writer prompt gains %%RELATED_WIKIS%%: name a neighbour in prose rather
than restating it; do not open one; if none fits, leave the detail out and
it returns on its own later.
- Density nudge folded into the existing _body_block_or_stub() above 30000
chars -- advisory text on the stub it already returns, no new gate.
- Maintainer step 4 gains a size exception: when the target is already very
large AND the seed is a coherent narrower subject, prefer `create`. Step 3
gains the matching carve-out so a deliberately narrow page is not treated
as a fragment and consolidated straight back.
Triage, routing, claiming and the incremental one-entity-at-a-time flow are
unchanged. No new entity types, relation types, tools, schema fields, job
types or migrations.
A budget set far above where it can fire silently disables the successor path: on large pages the writer keeps accumulating context until it hits its turn limit instead of handing off to a fresh successor. 30000 keeps handoff live without firing on routine work.
Tool results arrive as `function_call_output` items, which keep their payload under `output`, not `content`. The estimate read only `content`, so the items that dominate a writer's context contributed zero and the handoff nudge never fired at any budget, leaving long writes to grow unbounded. Lookup only; the summing logic already handled both shapes `output` can take.
A 70h bench run spent 58h (83%) on ONE page whose work was already done: all 39 members of all 39 stuck jobs were already cited. `summarises` had been frozen at 114 for three days while the body grew 44K -> 73.5K, and `writer no-op accepted` -- the exit that closes those jobs -- had fired zero times. Root cause: an append-shaped job forced through a replace-shaped tool. `edit_wiki_section` appended only when the section NAME was new; for an existing section it always replaced, so adding one citation to a 23,463 char `references` section meant re-emitting all of it. Reads are capped at MAX_OUTPUT_CHARS (8000) while writes are uncapped, so the writer was ordered to preserve a section it could only see a third of. Its only escape was to rebuild the section from the body's [[ref:UUID]] tokens, one get_entity per citation. Subagents, told by the docstring they had "all the same BrainDB tools" but given none of the wiki tools, retyped whole bodies through update_entity instead -- flipping one hex digit of a cited UUID, after which reconcile raised a FK violation, `finish_jobs` (the next line) never ran, and every job re-queued forever, uncapped. Changes, all plumbing -- nothing judges, gates or authors content (C2/C3): - reconcile_summarises_additive skips a cited id with no entity row and reports it, instead of raising. Content can no longer block bookkeeping. Still additive-only; skipped ids reach the activity log via the caller's existing `**rel` spread, so it is visible, not silent. - wiki_sections.append_to_section: add to a section keeping what is there byte-for-byte, resolved from the body we already hold. Reuses parse_sections + splice_section. Surfaced as edit_wiki_section(mode=). - read_wiki_section takes offset/limit and pages via the existing slice_content helper, with the same content_meta/next_offset contract get_entity uses. No longer silently _truncate'd -- hiding that content exists past the cap is how a "preserve everything" replace loses data. - check_members_cited: the exact computation the router already runs, as a tool, so the writer can answer "is there any work left?" in one call rather than thirty turns of re-reading. - update_entity refuses a wiki body, generalising the existing datasource guard, and points at the section tools. - Subagents get the wiki READ tools; the docstring and system prompt now say what they actually have. Edit/delete stay writer-only so one writer per wiki keeps the revision CAS meaningful. - _claimable() bounds the lease-reclaim branch by attempts (pending stays unconditional), and the lease goes 20 -> 120 min: its stated premise, "AGENT_TIMEOUT ~10 min", expired three bumps ago. - The handoff successor is seeded with the rendered job prompt as well as the brief, making "You have the SAME prompt" true -- it was running on the generic system prompt with no members and no preservation rule. - The token estimator counts function_call `arguments`, the largest term in a writer's context and previously invisible. Budget stays 30000. - Writer prompt: "leave that detail out" is scoped to non-members (for a MEMBER it is false -- a dropped member is recorded as covered and never returns), and the section protocol now leads with check_members_cited and append. Tests: 126 -> 159 passing, no regressions. New coverage pins the properties that failed live -- append preserves an >8000-char section intact, the reclaim ceiling bounds only the reclaim branch, a dangling ref does not block the good refs, the subagent has read tools and no write tools, and delegation depth is isolated between concurrent runs.
Round two on the writer-loop fixes: the post-fix audit (and the user) found 90acc76 damaged two core principles. This restores both and makes the tests able to catch a revert. Self-healing (code): - A job past the lease AND the reclaim ceiling was un-claimable and never dispositioned -- it wedged in `assigned` forever, and since `assigned` counts as an active job in _orphan_conditions(), its entity was permanently barred from re-triage: silent, unbounded loss. run_cron now flips such rows to `failed` -- the status that, by design, returns the entity to the orphan pool -- and re-enqueues a fresh triage job in the SAME tick. Count reported as `assigned_expired_failed` in the cron activity log. Bounded per job, self-healing per entity. - ASSIGNED_LEASE_MIN comment rewritten as an operator invariant (lease > your WIKI_AGENT_TIMEOUT / agent_request_timeout) instead of a number the bench overlay (18000s) falsifies. One citation predicate (code): - New wiki_jobs.uncited_members(conn, body, member_ids) -> (missing, gone) built on the existing parse_refs + liveness SELECT. The router's no-op gate and the check_members_cited tool both call it, so they can never drift; a member DELETED since triage is reported `gone` and no longer wedges its job in a fail/retry loop (same premise as reconcile's dangling-ref skip). Three private copies of the arithmetic collapse to one. The LLM's editing judgement (prompts/docstrings -- the quality fix): - 90acc76's wording made append "the normal way", said "you do NOT need to read the section first", and forbade rebuilding. Measured live: 26 append / 1 replace, ref-stacking collapsed 33-40% -> 10%, filler sentences 3x, and a section narrating an accepted offer then appending "the user is applying". All three directives removed. New protocol: choose by CONTENT -- corroborating member -> revise the sentence and stack the citation; genuinely new -> append; changed story -> rewrite freely (the snapshot makes it reversible). Coherence rule added; the UUID-retyping warning moved to where it belongs (full rewrites). - system_prompt/docstrings: the subagent "cannot write" claim corrected -- it has no WIKI write tools; saves/relations/update_entity remain. - "byte for byte" corrected to the true guarantee (content preserved; trailing blank lines collapse to one; markers re-emitted canonically). Integrity guards (code): - update_entity: content="" is now a warned no-op instead of silent destruction (observed wiping a thought cited inside a wiki); real rewrites and delete_entity untouched. - edit_wiki_section activity-log key "appended" -> "created" (was contradictory beside the new "mode" field). Tests that fail on a revert: - tests/test_wiki_selfheal_db.py (new, DB-backed): the self-heal loop end-to-end (wedged job -> failed -> fresh triage in one cron tick); claim_jobs honours the ceiling behaviourally; update_entity exercised through the REAL tool via on_invoke_tool (wiki blocked + DB unchanged, fact updated, wiki notes/importance applied, ""-wipe warned); uncited_members split + tool `gone` reporting. - test_wiki_writer_guards.py reworked: toolsets asserted on the BUILT agents (dropping extra_tools= now fails -- verified by actually reverting it: 1 failed, then restored: green); delegation depth proven behaviourally (two CONCURRENT delegations proceed, max_active==2, while a NESTED one is refused -- a process-global counter fails this test); the vacuous _router_missing copies deleted. - CI allow-list extended with the four wiki test files (the workflow's Postgres service + alembic schema already support the DB-backed ones). Suite: 213 passed, 0 failed (LLM-provider smoke test excluded locally).
…restart self-clean
Every fix targets a CLASS observed live across three audits of real agent
loops; nothing is keyed to one page, one model's typo, or bench timing.
No new gates, no content-shaping code (C2/C3 hold).
A. Reversibility + self-healing
- Snapshots move to CLAIM time (pure move — content was always the
claim-time body; only the write was deferred to persist). A run that
dies after a section edit now always has a durable pre-run snapshot.
Prompt reworded to the true promise: reversible to the run's starting
revision. Failed/no-op runs leave a truthful "claimed at rev N" row.
- release_stale_assigned(): on API startup, jobs still `assigned` from
before the process existed return to `pending` (agents run only
in-process; scheduler/watcher are HTTP clients — the invariant is
documented, incl. where it breaks: multi-worker). Observed cost of not
having this: a restart-orphaned consolidate dark for hours while
writers re-derived the identity it would have settled.
- Crashed maintainer/writer runs and the handoff-depth cap now write
activity rows mirroring their success shapes — a DB-only auditor no
longer sees a suspiciously clean run.
B. The header becomes editable (the quality centrepiece)
- Reserved section name "header" on the EXISTING tools — no new tool:
wiki_sections.replace_header (3-line reuse of parse_sections+_rebuild),
read/edit dispatch, delete refused, outline surfaces "header: Nch".
Root cause closed: on any page >4000ch the Summary/Disambiguation/meta
froze at creation (observed: "is applying" against a recorded
acceptance; friend labelled "son"; revision=1 on rev-41 pages).
Persist propagation needed zero plumbing — the section path already
falls through to extract_summary_disambig + finalize_wiki_write.
- check_grammar gains one advisory check (missing wiki:meta line) — the
real risk of a header replace is silently zeroing keywords_from_meta.
- `revision=` dropped from the prompt's meta template (verified: no
consumer anywhere) — the perpetual stale token dies with zero code.
C. Idempotency
- create dedupe keys on lower(proposed_name): the 65-second same-name
double-create (reproduced three runs in a row) now collapses; the
second orphan re-enters the pool via cron and attaches to the page the
first run built. No canonical_name uniqueness added (that would be
code gating LLM naming).
D. Honesty + judgement alignment
- Router density nudge member-scoped (a writer obeying the old text was
rejected by our own uncited_members gate).
- Successor seed: trust the brief for STATE, still read any section you
replace; body="" restated as ATTACH-only; "choose by content" echoed.
- References ledger: scoped compaction carve-out reconciled with the
superset rule; the "append is preferred" leftover removed.
- One uniform wiki-not-found message across all six tools with the
general recovery ("copy the id exactly"); search_sql docstring gains
the real schema + PG dialect (6 dead queries observed from guessing);
dead used_section_edits removed; stale "gated body" docstring fixed.
E. Observability
- Per-run log tag (read-downward ContextVar, same pattern as the
delegation depth) on the "Running typed query" line and every TOOL
line — concurrent runs no longer need argument-fingerprint forensics.
- VERBOSE_PREVIEW_CHARS=1500 replaces the three bare [:500]s.
Tests: 230 passed (was 213). New coverage is behavioural: header
replace via the real tool (only the header changes, CAS holds,
extract_summary_disambig reads the new values), append/delete on header
refused, uniform not-found across all six tools, restart release honours
the cutoff and preserves attempts, name-keyed create dedupe collapses
then re-admits after completion, and two router-level tests that
monkeypatch run_typed to die: the claim-time snapshot already exists
when the writer crashes, and both crash paths leave activity rows.
Revert spot-check performed (header dispatch removed -> test failed ->
restored).
The Qwen3 chat template resolves `reasoning_effort|default('xhigh')`, so
sending nothing runs EVERY request at the model's maximum reasoning
setting and injects a "think carefully, validate assumptions, consider
alternatives" instruction into each system block. Meanwhile the SDK only
replays reasoning for DeepSeek/Claude models (reasoning_content_replay.py
returns False otherwise), so on Qwen every reasoning token is generated,
paid for, and dropped before the next turn — it buys no cross-turn
consistency.
Measured on the bench box: ~2750 output tokens/turn, GPU generating in
99.7% of samples, sum of LLM gaps = 99.3% of wall clock, tool bodies at
0.01-0.06s. Wall clock is ~entirely generation, so trimming reasoning
trims latency almost linearly. Probe against the live server, same prompt
and tools: default 66 completion tokens vs 45 with effort=low, both
returning a valid tool call.
Adds nothing new — every mechanism already existed:
- `agent_wiki_reasoning_effort: str = ""` beside the other agent_* knobs
(agent_writer_handoff_* is the precedent for a wiki-scoped model knob).
Blank = send nothing = byte-identical behaviour; verified no-op.
- One keyword arg through _build/_cached, exactly the shape extra_tools
and extra_stop_tools already use, adding one key to the extra_args dict
that already carries `timeout`. LiteLLM folds unknown kwargs into
extra_body, so it reaches an OpenAI-compatible server and is ignored by
one that doesn't know it.
- Passed by the maintainer, writer and subagent only.
NOT by get_agent(): that cache key is shared by /agent/query AND the
ingest watcher, whose extraction runs (max_turns 40/30) are the most
reasoning-dependent work in the stack. A test asserts that scoping and
fails if anyone later threads it there.
Valid values for this template are low/medium/none — minimal/high pass
vLLM's Literal but raise in the template's own validator.
Suite: 160 DB-free passed (incl. 2 new scoping tests); the 70 errors are
the DB-backed files with no local Postgres running — 160+70 = 230, the
same total as before.
The previous route, ModelSettings(extra_args={"reasoning_effort": ...}),
raised litellm.UnsupportedParamsError on every call and failed 61 triage
jobs live before it was reverted.
Root cause is a layer above LiteLLM: the SDK's LitellmModel lifts the key
reasoning_effort out of reasoning / extra_body / extra_args and promotes it
to a top-level kwarg on litellm.acompletion(), popping it back out of
extra_body so it cannot also ride in the body. LiteLLM then validates that
kwarg against a per-provider allow-list; the openai provider, which every
OpenAI-compatible profile resolves to, does not list it, so the call raises
before a request is ever sent. All three of the SDK's documented routes
converge on that one rejected kwarg.
Nesting the value under chat_template_kwargs avoids the interception
entirely: the SDK only special-cases the literal top-level key, so the dict
is copied verbatim into LiteLLM's extra_body and forwarded into the request
JSON unfiltered - which is where vLLM reads chat-template variables, and
reasoning_effort is one (chat_template.jinja resolves it with default
'xhigh', so sending nothing runs at the maximum setting).
Verified through the full stack rather than against vLLM directly - probing
vLLM alone is what hid the bug last time. Same prompt, same profile:
reasoning drops 155 -> 44 chars and completion_tokens 55 -> 42, with the
answer unchanged.
Scoped to the wiki agents; the general agent, shared with the ingest
watcher, is untouched. Blank default keeps this inert, and the body key is
self-hosted vLLM only - a hosted provider may reject it. Profiles and model
resolution are not touched.
Adds a regression test pinning that reasoning_effort appears at neither top
level, so the outage cannot recur silently.
Nine commits that take the wiki maintainer/writer loop from "works if you watch it" to "runs unattended for days": the writer loop always terminates, jobs past their lease re-enter triage instead of wedging, an interrupted run is always reversible, and the page header is editable rather than frozen after the first write. Plus a wiki-scoped reasoning-effort knob and a per-call LLM timeout so slow self-hosted writes are no longer abandoned while the server is still completing them. Release bookkeeping in this commit: - Version 0.9.0 -> 0.10.0 in pyproject.toml and braindb/main.py (the latter was missed in the 0.3.0-0.6.0 releases; both are bumped here). - CHANGELOG section for 0.10.0. - `vllm_workstation_qwen` pointed at the Qwen model and port the wiki pipeline's tuning constants were actually measured against, so selecting it needs no AGENT_MODEL override. `deepinfra` remains the default profile in config.py, .env.example and docker-compose.yml — no existing setup changes. - .env.example: the profile list claimed four profiles when there are seven; corrected, and AGENT_REQUEST_TIMEOUT documented for the first time. - docker-compose.yml: env passthrough for AGENT_REQUEST_TIMEOUT and AGENT_WIKI_REASONING_EFFORT, per the recipe in CONTRIBUTING.md. - .gitignore: integrations/*/.state/ (per-caller transcripts written at run time), plus *.log, *.bak, *.sql plus a !scripts/*.sql negation, and backups/ — local operational artefacts that are one `git add .` away from a public repo. No DB migration. No public response field renamed. Both new knobs default to the previous behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Everything here is plumbing and bookkeeping. Nothing added judges, gates or authors content; that stays with the model.
Validation
Suite 126 -> 230 passing, including two new DB-backed test files. Each fix targets a class of failure observed across repeated audits of real agent loops, not a single page or one model's quirk. The reasoning-effort delivery route was verified through the full provider stack rather than against the server alone — an earlier attempt passed a direct probe and still failed in production, which is how the wrong route got shipped and reverted once.
Also in this release
AGENT_REQUEST_TIMEOUTandAGENT_WIKI_REASONING_EFFORTdocumented and passed through compose.