Preserve model-declared EOS tokens in serve mode

[saskw2010]

What changed

• Preserve every EOS token explicitly declared by config.json or
  generation_config.json when the engine runs in SERVE mode.
• Continue filtering tokenizer-only special tokens, retaining the tool-call safety
  behavior introduced for [Bug]: Tool Calling Fail in serve API #401.
• Add a cross-platform regression test covering the SERVE path.

Root cause

GLM-5.2 declares three EOS IDs:

• 154820 — <|endoftext|>
• 154827 — <|user|>
• 154829 — <|observation|>

stops_arm_tok() initially loaded all three, but its SERVE branch subsequently
discarded every stop except the tokenizer's primary EOS. The Python CLI and
OpenAI-compatible server forward native output and do not own a streaming-safe role
boundary stop, so <|user|> was emitted as assistant content and generation continued
until NGEN.

User impact

Before:

GLM-5.2 is running.<|user|>...

After:

GLM-5.2 is running.

This affects private coli chat and the shared native path used by
/v1/chat/completions, including streaming.

Validation

• Windows MinGW regression test: tests/test_stops.exe passes.
• The test confirms that SERVE retains all three model-declared EOS tokens while
  filtering tokenizer-only <|assistant|> and <sop>.
• Windows GLM-5.2 real-model smoke test through /v1/chat/completions:
  • HTTP 200
  • output exactly GLM-5.2 is running.
  • 9 completion tokens with max_tokens: 32
  • finish_reason: stop
  • no leaked role token

The model files were unchanged; the test used the existing validated 144-shard
checkpoint.

[ZacharyZcR]

This is the right fix for the default, and it composes cleanly with the COLI_SERVE_ALL_STOPS escape hatch that just merged (#572) — worth stating how the two fit together, since we wrote #572 and diagnosed the underlying issue (#549).

The distinction is exactly the one #401 wanted and never drew. Three populations, not two:

• tok_eos — always kept.
• Model-declared EOS (config.json / generation_config.json → c->stop_ids): <|user|> 154827, <|observation|> 154829. The model says these end a turn; discarding them is the v1.1.1 role-token leak. This PR keeps them.
• Tokenizer-only specials (<sop>, <|assistant|>, …): kept out for tool-call safety. This PR still filters them.

The old serve branch collapsed the middle group into "discard," which is precisely the <|user|>-leaks-into-output / never-terminates bug from #549. Gating on "did the model declare it as EOS" is the correct predicate, and your regression test (config [100,101,102] → 100/101/102 kept, 103/104 filtered) pins it.

Relationship to #572 (already on dev): they're complementary, not competing.

• Preserve model-declared EOS tokens in serve mode #584 makes the default correct — a standard GLM-5.2 container terminates in serve mode with no env needed. That's the fix for @weber-software's Output is not terminating with int4 gs64 #549 case: rebuild and it just works, no COLI_SERVE_ALL_STOPS required.
• serve: COLI_SERVE_ALL_STOPS re-arms the full stop set for non-tool-call use (#549) #572's COLI_SERVE_ALL_STOPS=1 stays useful as the escape hatch for the residual case Preserve model-declared EOS tokens in serve mode #584 can't cover: a third-party container whose config.json omits a role token it should stop on. Preserve model-declared EOS tokens in serve mode #584 keys off what the config declares; if the config is incomplete, the env flag re-arms the full tokenizer set as a manual override.

So the layering ends up: correct by default (#584), with a "stop on everything" lever for non-conforming containers (#572). No conflict — #584 just means the lever is rarely needed.

One thing worth a line in the code comment: #584 trusts config.json/generation_config.json to declare the EOS set. That's the right source of truth, but it's worth noting explicitly that a container with an incomplete eos_token_id is where COLI_SERVE_ALL_STOPS earns its keep — otherwise a future reader sees two overlapping stop-filtering mechanisms and wonders which is authoritative.

+1. Cleaner root-cause fix than the opt-in flag.