Repository navigation
Conversation
jayleaton
force-pushed
the
stack/07-server
branch
2 times, most recently
from
October 11, 2026 14:42
c2e4f62 to
4828dfd
Compare
3 of 8 tasks
jayleaton
changed the title
3 of 8 tasks
jayleaton
force-pushed
the
stack/07-server
branch
from
October 12, 2026 09:15
4828dfd to
bb53ff4
Compare
jayleaton
changed the title
This was referenced Oct 14, 2026
Open
Owner
|
Hello @jayleaton, Thank you for the drain, the SSE keepalive, and the stop tests. The family render and reader are a second chat path beside the template. renderIds returns fam.render (prompt.zig:31). generate installs fam.reader (chat.zig:271). commitFamily parses the reply (chat_generation.zig:346). That path also skips the template's prompt-cache rewind (prompt.zig:114). Please keep drain and the keepalive on the one server path. Put a family's encoding in that family. Replacing the template for every family would need prompt ids, reply tokens, and a resumed turn equal to a fresh prompt on Nemotron (CUDA and Metal), GLM, Flash Next, and the 27B. A push that leaves the shared server on the template is the update we need. The family hook lands with #702. Thank you again. |
…alls them. graph_cache.Cache captures one executable graph per caller-named key and replays it when the key comes back; ranks agree on every capture, eviction and fallback through an Agree hook, and a memory floor turns misses eager. arena.Arena sub-allocates DeviceBuffer.alloc from a few large chunks (VMM or plain cuMemAlloc) through memory.setHook; borrowed spans and pointers it does not own are freed as before.
…; CUresult names fall back to a table when the driver cannot answer. cuFuncGetName is optional (CUDA 12.3+), so an older driver still opens. During the exit's teardown cuGetErrorName fails too, which printed CUDA_ERROR_DEINITIALIZED as "CUDA_ERROR_UNKNOWN_CODE (4)"; code 4 now says the process is exiting.
Use CUDA device attribute 110 for GPUDirect RDMA with VMM support. Attribute 126 reports the memory sync domain count and can incorrectly enable the RDMA allocation flag. Add a regression for the external ABI contract.
…r stays under the max, a max of 0 is eager - A capture whose agreement throws, or whose insert runs out of memory, no longer leaks its executable: the map's slot is reserved before the capture, and the executable is owned here until the insert, which cannot fail. - A host map that cannot grow counts as no room in the existing room vote, so every rank declines that miss together. - `min_keep` is clamped to `max`, so under the memory floor the cap never rises above the configured max. A max of 0 runs every step eagerly, with no collective. - A cap lowered to 0 under the floor goes back to max once every rank has room, instead of staying eager for good. - Hits make no collective: with `Agree` set, keys, settings and fingerprint changes must already match on every rank. The docs now say so: the fingerprint changes at the same step on every rank, and `fingerprintOf` is for one rank. Co-authored-by: Jonathan Regan-Beasley <197533392+jregan-beasley-smc@users.noreply.github.com>
The arena's `live` (small ranges) and `large` (own chunks) maps drop an entry on every free and gain one on every allocation for the life of the process. std's tombstones never clear, so in time they take every free slot, and each allocation, free and miss then probes the whole map. Each map now counts its removals and is rehashed in place once they reach a tenth of its capacity. Test: 20,000 random allocations and frees (small ranges and large chunks) keep a tenth of each map free at every step; without the rehash it fails.
…amily that installs it, without a rebuild Cache.init applies the operator's value through Settings.operator, so a deployment that measures graphs slower can turn them off at serve time. The value is read once a cache, and is set alike on every rank like the settings themselves: no agreement is added, and hits still make no collective.
jayleaton
force-pushed
the
stack/07-server
branch
from
October 16, 2026 01:51
bb53ff4 to
372ac94
Compare
Contributor
Author
|
Thanks, @ashhart. Updated to
Host gate at |
…d RoCE collectives, plan link, fail-fast. zig/src/fabric/tp brings a rank up from TF_TP_WORLD, TF_TP_RANK, TF_TP_MASTER and TF_TP_PORT, agrees the settings digest over every rank, and gives a family all-gather, all-reduce and broadcast over NCCL (TF_COMM_BACKEND=nccl, the default), a pinned host mailbox for ranks on one host, or RoCE verbs between two hosts (TF_TP_ROCE_*), with an NCCL fallback when RoCE setup fails. Rank 0 sends each round's plan to the others over its own link (TF_TP_PLAN_*). A failure or a vanished rank ends every rank shortly after (TF_TP_FAILFAST, TF_TP_FAILFAST_GRACE_S). No family uses it yet. zig build test-tp runs the host tests; zig build tf-tp-test builds the two-GPU runner (-Dtp-nvcc= or -Dtp-fatbin= for the mailbox kernel). The RoCE all-gather adapts b12x's RoCEnante protocol.
… the stop word with no fail-fast exit; a rank killed mid-drain still fails the other fast.
… /24 the peer also has The automatic pick took the first N active RoCE ports in device-list order and never checked that the peer can reach them. On a host with two neighbours, one picked port can sit on a subnet the peer does not have, and the queue pair's RTR step then fails, which reports RoCE unavailable. Every rank now sends its usable ports' IPv4 addresses over the bootstrap. Each rank then keeps the ports whose /24 the peer also lists, ordered by subnet and each /24 no more often than both sides have it, so both ranks pick the same number. With no shared /24 the old first-N order is kept, with a warning to set TF_TP_ROCE_HCA. The exchange runs on every rank, including ranks that set TF_TP_ROCE_HCA, so the bootstrap stays in step.
…FNAME and TF_TP_MASTER are needed, with a two-host example
… kernel backends up front; run_hosts.sh passes HCA picks - `tf-tp-test kernel` reports the embedded kernel and exits 1 without one. `suite` and `failfast` on the mailbox or RoCE backend stop at once with a message that names `-Dtp-nvcc`, instead of failing later on a missing kernel. - `run_hosts.sh` stops before any step when the kernel is missing. It exports TF_TP_ROCE_HCA and NCCL_IB_HCA (derived from TF_TP_ROCE_HCA when unset), warns when a host has more than one RDMA device and neither is set, and records all three variables in host.txt.
… with another region layout are refused roce_guard.stripes computes each HCA's share of a post (its send stripe, the peer's receive range and its flag word) and refuses with error.OutOfRegion any range past its slot, its flag area or the registered region, before post_send. The verbs wire and the shm wire both post through it. The shm wire moves to roce_shm.zig and can stripe like the verbs wire (Options.shm_stripes), with an optional tap that sees every post for tests. The connection record carries the slot and registered length, so a peer with another layout is refused at connect; its magic is bumped. Tests: stripes tile the slot over two HCAs and refuse a post past the slot, a third slot, an unpadded or empty post, no HCA and a short region. On fake verbs a post past its slot reaches no verb, and a full slot stays inside the region. Over the shm wire striped as two HCAs, 2,000 variable-length gathers stay inside slot and region, bit for bit.
…would ask for one A post of about 9.6 MB faulted a DGX Spark's HCA. roce_guard.max_write (1 MiB, a constant) is checked in stripes() before the slot bounds, so the verbs wire, the shm wire and the proxy refuse a larger post with error.WriteTooLarge before any verb, whatever the slot size. Roce.init refuses a slot past it. Config.parse fails boot with a clear message when TF_TP_MAILBOX_MAX_KB would make the RoCE slot pass 1 MiB; the value is never clamped. Tests: a 9.57 MB and a 1 MiB + 16 post are refused in a 10 MiB slot and reach no verb, a 1 MiB post goes out; boot refuses TF_TP_MAILBOX_MAX_KB 1028, 2048 and 16384 and keeps 1024.
…sized ops, at most 2 MiB in flight Default off. allGatherV strides past the slot, up to TF_TP_ROCE_CHUNKED_KB, go RoCE as ceil(n / slot) launches of the tp_roce kernel: op c moves at most a slot of each shard from c x slot (RoceArgs.chunk), and an op empty on both ranks rings no doorbell and leaves the epoch. Every RDMA write keeps the 1 MiB cap. The kernel poisons the transport rather than stage past a slot, and its four failure-record blocks fold into fail_record (same stores and fences). VerbsWire.post waits for completions while the bytes posted and not completed over both HCAs would pass max_inflight (2 MiB, a constant in the post path); a wait that sees no completion for 10 s fails the proxy. Boot refuses TF_TP_ROCE_CHUNKED_KB when the kernel image lacks the tp_roce_chunk_abi marker. The setting is part of the digest both ranks agree on. Tests: chunked strides from 300 KB to a 128 MiB union posted back to back on fake verbs whose completions lag keep at most 2 MiB in flight and no write past 1 MiB, and reach the cap; the wire's own in-flight count and peak match the fake HCAs'. Over the shm wire, 2,000 chunked variable-length gathers of up to four slots arrive bit for bit.
family.zig is an optional vtable a family installs in place of the chat template: prompt rendering, thinking and effort fields, a streaming reply reader that parses its own call markup, and the turn it remembers. Without a family the server runs as before. chat.zig's reply collection moves to chat_generation.zig and openai.zig's run to openai_run.zig (pure moves), so neither file passes 600 lines with the hooks in.
…region; the region is DONTFORK and NOHUGEPAGE A local protection error can arrive while the IOMMU invalidates its mappings lazily. On such an error the old path left the other HCA's QP ready to send (still taking the peer's writes), hung up the fate channel without telling the peer to stop, and freed the region at once: unregistered from CUDA with no GPU sync, then QP destroy, MR deregistration and munmap. A device write after the unmap could then reach a reused page through a stale IOTLB entry. - Roce.fail and Roce.abort (the fail-fast hook) move every QP to the error state: no send goes out and no peer write lands. - Roce.deinit quiesces first: QPs to error, every posted request reaped (flushed or not), cuCtxSynchronize. A failed transport, or one that does not quiesce, keeps its region registered and mapped; the process exit tears it down. - Session.close on a failed transport goes fail-fast before closing, so the peer aborts and halts its HCAs first. - The region is MADV_DONTFORK and MADV_NOHUGEPAGE before ibv_reg_mr; a refusal fails the start. - A QP whose actual max_inline_data is under 4 B is refused (the flag post uses lkey 0 on a stack word). - One log line a rank: region base and length, each HCA's lkey and rkey, the peer's region and rkeys. Tests: on fake verbs Roce.abort halts both QPs; quiesce halts both before the first flush and reaps every posted write; an aborted transport reports itself failed, and over the shm wire the aborted rank keeps its region while the other frees it; smaps shows dc and nh on a prepared region; the inline bound refuses 3 B and takes 4.
…pt loop ends, and again if it exits with connections open.
…path on real HCAs, only on an idle link - `tf-tp-test failfast V transport [N [HOLD_S]]`: N clean 4 KiB sums, the ranks meet over NCCL, each waits for its link to go idle (every HCA's posts completed), both agree over NCCL, then the victim takes the proxy's failure path (`Roce.inject`, which is `Roce.fail`) and closes its session: both ranks halt their QPs and exit 70, and the victim's region stays registered. When either rank's link is not idle there is no inject, a clean stop, REFUSED and rc 2. - Its host-only forms (TF_TP_FAILFAST=0, or a hold) are refused unless TF_TP_ALLOW_HOST_ONLY=1 and no RDMA wire is in use (the shm wire, or no RoCE), so they never run on an HCA. - `run_transport.sh MASTER PEER_SSH OUT`, driven from rank 0's host (rank 1 over ssh): prechecks on both hosts (no GPU process, memlock unlimited), then T0 (suite), T1 and T3 (each rank as the victim). After each step it reads both hosts' kernel log since the step and every HCA *err* counter; a hit, a counter that grew, an unexpected rc or a failure line prints STOP: and exits 1. `run_transport.sh t2 OUT` runs the host-only form on one host. - The `halted` and `quiesced` lines carry each QP's state read back with ibv_query_qp (an optional symbol), each HCA's posts and completions, the completions the drain reaped and how many of them were flushes. - `Roce.allGatherV` refuses a stride past the slot (one op; the route never sends one). - The README states the RoCE limits and how to run the check. Tests: on fake verbs the link reads busy with two posts out and idle after the drain, and the report line names both QPs in the error state with every completion a flush.
…ost halt Opened.halt came with the closed ashhart#662 and is absent from this stack, so the native build failed to compile at main.zig.
The CUDA root keeps main's ck and tp_watch test imports beside the graph cache and arena. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-fabric build.zig keeps main's kernel-contract and video steps beside the TP steps; nccl.zig keeps ashhart#642's send/recv with the TP abort and async-error symbols. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main's imprint and System One POST routes are counted and refused with 503 while a stop drains, like the other engine routes. The engine writes its kept states (main's engine.drain) before the drain stop closes it; the recall record closes after the engine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main builds zig/src/server/json.zig as the `json` module, so a relative import puts the file in two modules and `zig build test` / `test-server-cpu` fail to compile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jayleaton
added a commit
to jayleaton/TensorFold
that referenced
this pull request
Oct 16, 2026
… into stack/12a-serving-host ashhart#663 now keeps drain and keepalive on the template path, and the review moved the family hook here (ashhart#702). This layer adds that hook back: family.zig, chat_generation.zig and openai_run.zig, on top of current main's server: - Recall, /v1/imprint, System One, vision input (normalizeWith, api.Image, cache_key), Stale refusals, token_sha256, and reply_text.Shown all keep main's versions. A family's own prepared images are `family_images`, kept apart from main's `images`. - A family render now gets the prompt cache's rewind too: rewindLen probes through renderIds. That is the gap the ashhart#663 review named. No served-family gate has run on it yet. - zig/build/server.zig: `addTo` also takes the tokenizer and lanes, for `dsv41_serve`; cortheon_fuzz, hip and xpu pass them. test-cuda-host keeps main's goldens, tokenizer, GLM and tool-parse tests, and also compiles tensorfold-native's root. - Every server file imports the named `json` module (the json.zig-in-two-modules failure ashhart#663 hit). - DeepSeek.load writes a missing tokenizer's diagnostic into the caller's problem arena instead of gpa. A new test in deepseek_test.zig leaks with the old allocation and passes with the fix. - --recall with DeepSeek-V4.1 is refused at startup: Recall keys its record on the chat template's tokenizer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jayleaton
changed the title
4 of 8 tasks
Owner
|
Thank you, @jayleaton, keeping drain and keepalives on the existing template path closes the alternate-render concern, and the close-once owner fits the released engine interface. We checked af0bc53. The duplicate family route and old lane implementation are gone; prompt.zig matches main, and generation is counted before the drain check. We can extract this server increment onto the current shared server fixes, preserving admission before SSE and the health/auth behavior from #912. We will run the drain race, disconnect and silent-stream checks on that composition, then the native served shutdown cases. Your host receipts remain useful without merging the entire inherited stack over the release. Posted by Ash's AI Assistant Synthena |
newNode and place reserve the maps, the slot lists and the free lists before they change anything. A failed insertion no longer consumes a free-list ID or leaves an appended slot undefined (deinit read that slot's flag). The free lists keep room for every slot, so remove and prune never allocate or drop an ID. A failed insert or insertFrom prunes the path nodes it made. A failed setRam in insert drops the entry instead of leaving it indexed, and setRam returns the token blocks it took. prefix_fail_test.zig fails each allocation of a RAM insert and of a disk insert into an empty index in turn. Each time, the index must keep its shape and slot accounting, the same insert must then succeed, and deinit must leak nothing. The tests run in the Linux and the portable (macOS) session roots. Spark receipt: lane job doug-637-prefix-01 on dgx-01 (aarch64, Zig 0.17.0), 2026-10-17T02:08:30Z, VERDICT PASS, candidate tree 1322aee. zig fmt --check clean; lean 517/517 against merge-base 46a08c1, 0 new; zig build 16/16; test test-kernel-contract test-golden fuzz-cortheon 99/99 steps, 1499/1507 (8 skipped); test-sessions 3/3. With only the new tests on 9266f3f, both fail at the node-slot accounting check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… free allocLocked reserves room in `live` before it takes a range. A failed insertion used to pop a freed range and then lose it: the range was in neither the free list nor the live map. Both small-path inserts now cannot fail. A failure leaves the free list, the cursor and the chunks as they were, and the retry gets the same range. The new test spends the map's spare room so that the reuse's insertion must grow it, fails that allocation, and checks range, cursor and chunk conservation, the retry, and a clean deinit with every chunk released. Not in this change: tying memory.setHook / currentHook installation to the family that owns the arena. Jonathan Regan-Beasley's capture-ownership fix (5826f89) is unchanged. Spark receipt: lane job doug-635-arena-01 on dgx-02 (aarch64, Zig 0.17.0), 2026-10-17T02:10:06Z, VERDICT PASS, candidate tree 663ff7b. zig fmt --check clean; lean 517/517 against merge-base 46a08c1, 0 new; zig build 12/12; test test-kernel-contract test-golden fuzz-cortheon 94/94 steps, 1430/1437 (7 skipped); test-cuda-host 39/39. With only the new test on abcad7d, it fails (the freed range left its free list). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ck/03-tp-fabric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x fixes) into stack/04-sessions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fixes) into stack/07-server Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 854b57c) Reported-by: Bizuayeu <ashhart#636 (comment)> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 602a0fd) Reported-by: Bizuayeu <ashhart#636 (comment)> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 569577a) Reported-by: Christian Schröder <ashhart#636 (comment)> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 8f75a6a) Reported-by: Christian Schröder <ashhart#636 (comment)> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd HCA verbs-error naming) into stack/04-sessions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…HCA verbs-error naming) into stack/07-server Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts, both sides kept: - zig/src/cuda/memory.zig: main's allocation limit (reserveAllocation/finishAllocation) now wraps the arena hook path. A hooked allocation reserves, tries the hook, falls back to cuMemAlloc and is counted by finishAllocation like any other. - zig/src/cuda/root.zig: main's memory_receipt, setAllocationLimit and new kernel modules beside graph_cache and arena. - RUNBOOK.md: main's GLM heat-band paragraph, then the TF_GRAPH_CACHE paragraph. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r.check does launched() named the failing kernel for every non-success code, so a launch that main's tests answer with CUDA_ERROR_NOT_READY (qwen_cuda_test, experts3_host_test) reached errorName on a fake driver and crashed. It now returns error.NotReady first, like Driver.check. The ranks test reads the monotonic clock through std.Io, so it also compiles on Windows hosts (zig-windows). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…3-tp-fabric Conflicts, both sides kept: - THIRD_PARTY_NOTICES.md: two b12x RoCEnante entries, each naming its own files. Main's two-rank CUDA RoCE gather (zig/src/cuda/roce.zig, zig/kernels/cuda/roce_gather.cu), then the TP fabric's RoCE path (zig/src/fabric/tp/mailbox.cu, roce.zig, roce_verbs.zig). - build.zig: main's sys and checkpoint_config wiring beside tp_build; tp_build.add runs before both walks, so the TP modules (which compile zig/src/cuda/root.zig, now importing sys) get the sys import. Auto-merge fix: - zig/src/cuda/nccl.zig: main added ncclCommAbort and ncclCommGetAsyncError to Api too; one copy kept (main's). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflict, both sides kept: - build.zig: sessions_build.add beside main's sys and checkpoint_config wiring, declared before both walks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts, both sides kept: - zig/src/server/openai.zig: main's deferred stream opening (Sink.begin, unsent before the stream opens) with the SSE keepalive; chat.generate calls begin before its token loop, so a keepalive only follows an opened stream. - zig/src/server/routes.zig: main's /v1/audio/speech joins the counted dispatch as is_speech, so a draining server refuses it with 503 + Retry-After like the other generation routes. - zig/src/server/status_routes.zig: /health answers "draining" first, without asking the engine, then main's engine health check (503 "unavailable"). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
|
Thanks @jayleaton for #663. 1.1.1 drains requests on SIGTERM or SIGINT, returns 503 with Retry-After for new engine work, and drains kept states before closing the engine. TF_DRAIN_S defaults to 20 seconds; a second signal cuts the wait short. Silent streams receive keepalives before their first token, with TF_SSE_KEEPALIVE_S controlling the interval. Posted by Ash's AI Assistant Synthena |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
server: graceful drain on SIGTERM / SIGINT and pre-token SSE keepalives, on the existing chat-template server path. Depends on #637. This cumulative PR targets
main; this layer's net change over #637 (stack/04-sessions) touches onlyzig/src/server/(13 files).The family render / reader / commit hook is not in this PR: per review it moves to #702.
prompt.zigis identical to #637's, sorenderIdsand its prompt-cache rewind stay on the template.Merge with current
main(001ac2e5, merging #637 at9266f3f1), resolved inserver/{main,routes,serve,server}.zig:routes.zig: main's/v1/imprint/{resume,load}and System One POSTs pass through the same drain gate as the other engine routes.serve.zig: main'sengine.drain()runs before the drain stop closes the engine; main's Recall startup line is kept.main.zig: the Recall record closes after the engine (defer order);forgetFailfrom main is kept.server.zig: both config fields,sse_keepalive_sand main'sidentity.af0bc537fixes the merged head's compile failure inzig build test(CI run 38927762698 on001ac2e5):Commits in this layer:
e65583ca,9a486bbd,467504a1,23bf7d88,a5b8b357,2b3cccf2,93677846(removes the family hook again),368a0d38(@cwschroeder, the Linux NVMe lane-test guard),7a6b32f6, the merge001ac2e5, andaf0bc537.Receipt
af0bc5378687c188a2461e080db4b9db72ec9972, Linux x86_64, Zig 0.17.0. No GPU or checkpoint.GitHub checks at
af0bc537: zig, fmt, desktop, mobile.mainhas since moved to0a591781; its delta from46a08c18isdocs/community-benchmarks.mdonly, which this branch does not touch, and merging it changes no other file.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.Merge Danger
Door: two-way. Reverting restores #637's server without a data migration.
Blast Radius: server. Every family's HTTP server: stop handling (503 + Retry-After while draining, the drain deadline, the engine's disk drain before close) and SSE keepalive bytes before a stream's first token.