sync_filter: bypass the LLM for /sync_* commands (inject assistant message directly)

[dvystrcil]

Why

After PR #38 added the <filter-result> marker tag + emphatic wording, /sync_down behaviour improved dramatically:

	Before #38	After #38
Wall time	spinner for minutes	2.4s
Reasoning size	20,037 chars circular loop	44 output tokens, no looping
Visible response	empty	"Pulled latest changes — already up to date."

But qwen3.6:35b still double-checks the filter's work by calling run_command("cd ~/fiction/standing-patrol && git pull") + get_process_status after the filter has already injected the SUCCESS result. The model gets the same answer ("Already up to date") and reports the right thing — but the redundant shell calls defeat the point of the filter intercepting in the first place. Prompt-only fixes haven't been enough to fully eliminate this.

Two prompt iterations applied to fiction-writer today:

1. "DO NOT call run_command for /sync_* commands"
2. " tag = authoritative outcome, do not re-do the work"

Both helped; neither got us to zero tool calls.

Proposal: bypass the LLM entirely

For /sync_* commands, the LLM adds no value — the filter already knows the outcome. Instead of injecting a role: system message and routing through the model, the filter should:

1. Inject a role: assistant message directly with the result text
2. Short-circuit the request so OWUI doesn't call the model at all

Two outcomes:

• Zero shell tool calls (impossible to misuse — the model never runs)
• Sub-second response (no model latency; user sees the result the moment the filter finishes the git operation)

Implementation sketch

async def inlet(self, body: dict, user=None) -> dict:
    ...
    if self._is_sync_down(user_message):
        result = self._sync_down(project)
        log.info("[sync] SYNC DOWN result: " + result)
        # Append an assistant message directly + signal short-circuit
        messages.append({
            "role": "assistant",
            "content": self._format_user_visible(command="/sync_down", result=result),
        })
        body["messages"] = messages
        # OWUI honors a sentinel that tells it to skip the LLM call entirely
        # (need to look up the exact mechanism — possibly body["_skip_completion"]=True
        # or a return-shape signal)
        body["_short_circuit"] = True   # ← stand-in; verify the actual key
        return body

The user-visible content would be exactly what the model produces today:

Pulled latest changes — already up to date.

For FAILURE cases:

Sync down failed: <error from git output>. You may want to investigate or retry.

Open questions before implementing

1. What's the actual OWUI mechanism to short-circuit? Need to inspect OWUI's filter dispatch source. Candidates:
   • Return body with a special key OWUI checks
   • Raise a specific exception type
   • Set a done: True flag on the appended assistant message
2. Does the OWUI frontend handle a chat that gets a response without a model round-trip? It might not animate the same way; might need extra fields (id, model name, usage stats) to look "real".
3. What about title generation and tagging? OWUI fires those as separate background tasks. They'll still hit ollama-small-cuda for the chat title. That's fine — they're independent of the main chat completion.
4. Does this break the outlet filter? outlet() runs after the assistant message; if it expects a real model response shape, the synthesized assistant message needs to match.

Acceptance criteria

• /sync_down in a fiction-writer chat produces the success/failure sentence in <500ms
• kubectl logs deploy/ollama shows NO chat completion request for the /sync_* command
• kubectl logs deploy/owui-pipelines still shows the filter chain running (sync_filter SUCCESS log line)
• No JSON.parse error on the frontend
• Title generation / tag generation still work (those hit the small lane independently)
• Same behaviour for /sync_up and /sync_pr

Related

• fix(sync_filter): wrap result in marker tag + emphatic wording #38 — the marker-tag + emphatic-wording approach (current state)
• fiction-writer system prompt has the /sync_* filter-awareness section + <filter-result> recognition (DB UPDATE applied 2026-05-25, prompt 7672 → 10080 chars)
• Trigger context: same chat where qwen3.6:35b wedged ollama-big for 30 min — diagnosis was a gfx1151 wedge, not the filter; pod-rolled to clear. The filter change is sound; this issue is the next level of polish.

[dvystrcil]

Investigated question 1 ("what's the actual OWUI mechanism to short-circuit?") against the live open-webui pod's real source (utils/middleware.py, utils/filter.py).

Finding: there isn't one, in this OWUI version. process_chat_payload calls process_filter_functions(...) and gets back (form_data, flags) — but process_filter_functions (in utils/filter.py) hardcodes return form_data, {} at every return path. flags is a vestigial/unused extension point right now, not a real signal channel. The only thing an inlet filter can currently influence through this path is form_data itself (messages, tools, features) — there's no sentinel, exception type, or done-flag OWUI's dispatch code checks for "don't call the model."

This means the proposal as sketched (inject an assistant message + a short-circuit flag) isn't directly buildable against this OWUI version's actual filter API — it would need patching OWUI itself (a fork, same shape as tonight's open-terminal-app-fork work) to add a genuine short-circuit path, not just a filter-side change. That's a materially bigger lift than this issue's original framing assumed, and worth a scoping conversation before starting rather than just diving in.

Not attempting the implementation tonight given that gap. Leaving the other open questions (2-4) unanswered too, since they're downstream of question 1 being resolved.

[dvystrcil]

Re-tested live against the current `fiction-writer` preset (now on `qwen3.5:122b`, with `fiction_writing_filter`'s reasoner still on `qwen3.6:35b` per its valve) — 2/2 clean runs, zero redundant tool calls:

• Run 1: model's own reasoning trace explicitly recognized the filter's result as authoritative — "The git_sync_filter has already intercepted and completed this /sync_down command before I saw the message... I should acknowledge... not re-run it."
• Run 2: response was the bare `sync_filter` result with no function calls at all.

The core problem this issue was raised for — redundant shell tool calls double-checking the filter's already-injected result — no longer reproduces. Whatever combination of PR #38's marker-tag approach plus subsequent prompt refinement in `fiction_writing_filter`'s `REASONER_PROMPTS` got this to reliably zero tool calls, without needing the LLM-bypass architecture originally proposed here.

On the proposed implementation itself: the 2026-08-02 investigation comment already found it isn't buildable as scoped — no short-circuit mechanism exists in this OWUI version's filter API (`process_filter_functions` hardcodes an empty `flags` return), so it would require patching OWUI itself, a materially bigger, unscoped lift.

The `<500ms` AC is now stale for an unrelated reason: `fiction_writing_filter`'s TRIAGE→REASONER pipeline runs unconditionally before `sync_filter`'s result is returned, adding 60-130s regardless of tool-call behavior — filed separately as `dvystrcil/open-webui#156`, since that's a real problem but a different one than what this issue is about.

Closing — the actual problem is resolved; the proposed fix path is a dead end that isn't worth pursuing given the problem it targeted no longer exists.

[dvystrcil]

Correction: the follow-up issue is dvystrcil/open-webui#173, not #156 as I mis-stated above (guessed the number before creating it).