Repository navigation
Conversation
The use execution plan now carries a structured collection: which side's path is scored, which path and frozen steps are probed, and the scorer endpoint for each implementation. The decoder accepts the old empty collection and refuses non-empty legacy fields, whose values carried no scorer. Collection is validated against the contract before any process starts: requirements must have their score or probe entry, step lists must match the contract, scorer adapters must match the declared policies, and HF probe plans are refused with the named no-probe limitation. Score and probe calls reuse Score.prepare/bind/admit and Worker.run, retain their own transcripts, and are bound to the admitted inference they score. use run emits score/probe flags in runs.json, retains partial records on failure, and reports call counts, wall time and record/log sizes in cost.json. Premise prediction now includes score and probe obligations, negative scored and KL budgets are rejected, and the walkthrough documents the new collection shape. Tests cover collection gaps, conflicting probe steps, HF refusal, score/probe premises and negative budgets. Signed-off-by: bohaotu <bohaotu@pony.ai>
Collection records now retain the source log digest, target policy, score log digest and elapsed time, and plan.json records every allocated score/probe call with its scorer endpoint. cost.json reports per-call seconds and the peak memory reported by score/probe resource records, marking memory unmeasured when no worker measurement exists. A probe-set now rejects the old single-probe path and steps options instead of silently ignoring them. Collection tests cover missing scorers, extra score paths and duplicate probe paths. Signed-off-by: bohaotu <bohaotu@pony.ai>
|
I found five issues on the current branch:
I reproduced the first four and traced the fifth to the worker measurements. The current tests miss these cases. |
Transcript.file now writes complete records with their newline but writes the incomplete final fragment without adding one, preserving the worker's bytes. Probe-set endpoint lookup is scoped to the source path as well as the target, so collecting both probe paths no longer produces null endpoints. Identical probe steps on repeated KL requirements are no longer treated as conflicting, and probe-path may accompany probe-set so KL relations can select their source path. Cost reporting now names the actual device allocator metric and scope instead of calling it process RSS. Tests cover the transcript fragment and identical KL steps. Signed-off-by: bohaotu <bohaotu@pony.ai>
There was a problem hiding this comment.
Thanks for the update. I checked the new commit against the five points from my last comment.
Fixed:
- The file transcript now keeps the trailing fragment byte for byte, and the new test covers it.
- Probe endpoints are now looked up by both source path and target, so collecting both probe paths no longer produces
nullendpoints. - Repeated KL requirements with identical steps are accepted, and the new test covers it.
--probe-pathcan now accompany--probe-set, so a KL comparison over a probe set can run.
Still open:
- The help text now says "KL requires --probe-path or --probe-set".
selectedRelationstill requires--probe-pathfor both KL relations, so--probe-seton its own is refused. The--probe-pathdescription also says it selects the source path for the probes, but with--probe-setit only selects the path of the KL claim. Something like "KL requires --probe-path; the probes come either from reference-probe-/candidate-probe- at --probe-steps or from --probe-set" would match the code. memory_scopeis a fixed label ("mlx device allocator", "torch.cuda worker device allocator"), not the scope the measurement was recorded under. The records already carry it. Each vLLM worker entry has its ownscope(permit to completion, with loading and serialization excluded) along with host, pid and device. The MLX reading covers only thecross_scorestage. Sopeak_device_allocator_bytesis the largest reading taken over one probe's scoring interval, with loading excluded; it is not the peak of the run, which is how the name reads. Please carry the recorded scope or stage, and the call and device of each reading, intomemory_measurements, and either describe the top-level figure that way or drop it. Also, only full-vocabulary probes carry measurements; a plain score's output admits none. So "A score or probe worker that reports a resource measurement" in the walkthrough, andscore-probe-device-allocator, should both say probe.- The PR description still says
cost.jsonreports peak memory. Please update it to describe what is now reported.
The title and description still mark this as WIP, and no real use run followed by use admit has collected a score and a probe. Before the WIP marker comes off, I'd like to see one such run on MLX or vLLM, to show that the collected records are admitted end to end. I'll take another look after that.
KL now derives its source path from a probe set with exactly one path, and the help text describes when --probe-path is required. Probe memory readings retain their recorded stage, scope, call, path, target, host, pid and device; the top-level figure is named as a max over probe intervals rather than run RSS, and plain scores no longer claim to contribute memory measurements. The walkthrough now documents the probe-only scope and the named device allocator metric. Signed-off-by: bohaotu <bohaotu@pony.ai>
|
Thanks. I checked the new commit against the three open points. The KL path selection is resolved. A probe set with one path now selects the KL path, a set with two paths asks for The memory readings still differ from the records in a few places, and the first one comes from a mistake in my last comment:
The build and tests pass on my side, and CI is green. The WIP status and the request from my last comment stand: one real |
Status
Draft — not ready for review. This is the first phase of #80 (
use runcollection). We do not have an MLX host, and no real vLLM/MLXuse runfollowed byuse admithas collected a score and a probe yet, so the WIP marker and draft state remain until that end-to-end run is available.Why
invar use runpreviously refused numerical requirements that need scored paths or full-vocabulary probes. Those records had to be collected manually withinvar compare numerical/invar score, so a contract run and a training history could not reach the same admission path.Refs #80.
What
collection: scored path sides, probe paths with frozen steps, and scorer endpoints.Score.prepare/Score.bind/Score.Worker.run, retaining its own transcript.runs.json; support multiple probe paths with--probe-set.use prepare.Memory reporting
cost.jsonreports probe memory readings with their call, path, target, metric, recorded scope, stage, host, pid, device and bytes.peak_probe_device_allocator_bytesis the largest reading over probe scoring intervals, not the peak of the whole run or process RSS. A plain score produces no such measurement; when no probe measurement exists the memory fields arenullandmemory_statussays it was unmeasured.Validation
cabal test: PASS.fourmolu,hlint,test/types.sh ghc,cabal check: clean.Not ready / remaining
use runscored-path/probe E2E yet (no MLX host locally).