Phase 8 scheduling contract

AGENTS.md

@@ -10,7 +10,7 @@ Keep this `AGENTS.md` up to date whenever development workflows, architecture, s
 
 ## Current state
 
-Phases 1 through 7 are implemented. The real Gemma executor proof demonstrated exact checkpoint continuation. The logical context store owns shared token branches and transactional mappings. The physical manager owns tier capacity, representations, transfers, bindings, prepared transitions, eviction, observability, and committed device block tables. The server adds durable opaque external context IDs, atomic state recovery, model lifecycle APIs, bounded transition-cost scheduling, shared incremental streaming, canonical usage, authentication policy, OpenAI completion/chat adapters, and checked OpenAPI. Phase 5 adds transactional llama.cpp sequence mappings, reference-only activation, graph-reuse and byte-copy metrics, and a real-GPU staged-versus-mapped proof. Phase 6 adds the persistent mapped live executor, bounded lifecycle controls, and its real-GPU acceptance report. Phase 7 adds executor-reported operating points, capacity-admitted multi-model residency, transactional immutable model epochs, active-reference draining, pressure-driven idle LRU eviction, bounded local mapped-context spill/restore, and residency observability.
+Phases 1 through 8 are implemented. The real Gemma executor proof demonstrated exact checkpoint continuation. The logical context store owns shared token branches and transactional mappings. The physical manager owns tier capacity, representations, transfers, bindings, prepared transitions, eviction, observability, and committed device block tables. The server adds durable opaque external context IDs, atomic state recovery, model lifecycle APIs, bounded transition-cost scheduling, shared incremental streaming, canonical usage, authentication policy, OpenAI completion/chat adapters, and checked OpenAPI. Phase 5 adds transactional llama.cpp sequence mappings, reference-only activation, graph-reuse and byte-copy metrics, and a real-GPU staged-versus-mapped proof. Phase 6 integrates one persistent mapped Gemma executor with bounded admission, request-owned incremental generation, cancellation/deadline propagation, graceful shutdown/restart, and a real-GPU acceptance matrix. Phase 7 adds capacity-admitted multi-model residency, transactional epoch lifecycle, active-reference draining, idle LRU pressure eviction, bounded native-state spill/restore, and real-GPU lifecycle evidence. Phase 8 adds resumable execution sessions, priority-aware deficit round-robin with monotonic age promotion, trusted scheduling metadata, bounded non-blocking scheduler diagnostics, and a versioned real-GPU mixed-workload proof.
 
 The repository currently contains:
 
@@ -25,7 +25,7 @@ The repository currently contains:
 - `executor`: upstream and patch metadata;
 - `tools`: fetch, verification, integration-report, and coverage helpers.
 
-The server dynamically admits and reuses multiple model epochs within configured device, host, and storage budgets. Each resident model currently owns one native execution slot, so requests for the same model serialize at that slot while distinct resident models can execute independently. HTTP transport diagnostics default off and support privacy-safe and fully unredacted levels through `--http-debug` or `CUSCO_HTTP_DEBUG`; full mode exposes headers, query values, credentials, and body content. Phase 8 workload scheduling and operational hardening is next; this diagnostic slice alone does not implement that phase. Broader compatibility and production hardening remain later work and must not be represented as implemented.
+The server dynamically admits and reuses multiple model epochs within configured device, host, and storage budgets. Each resident model currently owns one native execution slot. The Phase 8 scheduler interleaves request-owned prefill and decode quanta with per-principal/class fairness within that slot, while distinct resident models can execute independently. Suspended sessions retain their model reference but release slot occupancy between native quanta; cancellation, deadline, and shutdown callbacks remain registered through each native ownership fence. HTTP transport diagnostics default off and support privacy-safe and fully unredacted levels through `--http-debug` or `CUSCO_HTTP_DEBUG`; full mode exposes headers, query values, credentials, and body content. Phase 9 compatibility, configuration, persistence, and production packaging is next. Broader model compatibility and later optimization remain prospective and must not be represented as implemented.
 
 The current unprefixed `/v1` completion and chat routes are minimal adapters, not the complete Phase 9 OpenAI compatibility profile. They do not yet preserve the full chat, tool, and stream-option semantics or emit OpenAI-native streaming chunks; unsupported compatibility input must not be represented as supported.
 

README.md

@@ -6,19 +6,21 @@ The project separates responsibilities deliberately: Rust will manage logical co
 
 ## Current state
 
-Cusco has completed Phases 1 through 7. The executor proof established exact
+Cusco has completed Phases 1 through 8. The executor proof established exact
 checkpoint continuation for a hybrid/recurrent Gemma model, and the Rust layers
 now provide durable logical contexts, capacity-accounted physical state,
 transactional mapped activation, an authenticated HTTP API, immutable model
-epochs, bounded live inference, and dynamic model residency.
-
-Phase 7 replaces the one-model process with a capacity-admitted residency
-scheduler. Executor-reported operating points account model weights, context
-capacity, and device/host placement; model loads, epoch reloads, retirement,
-and unload use transactional publication and active-reference draining.
-Pressure selects idle LRU victims, inactive mapped contexts can spill through
-the native sequence-state ABI and restore exactly, and `/native/status`
-reports configured budgets, resident epochs, and lifecycle/tier metrics.
+epochs, bounded live inference, dynamic model residency, and resumable
+priority-aware workload scheduling.
+
+Phase 8 schedules bounded prefill and one-token decode quanta with
+priority-aware deficit round robin, per-principal fairness, FIFO ordering, and
+monotonic age promotion. Request-owned execution sessions preserve unpublished
+successor state across quanta, while bounded asynchronous scheduler diagnostics
+attribute decisions without backpressuring inference. The checked-in versioned
+mixed workload gates fairness, starvation, cancellation, deadlines, capacity
+recovery, diagnostic loss, and relative first-event and per-quantum latency on
+the declared real model and GPU.
 
 ## Run the real-model inference integration test
 
@@ -86,6 +88,21 @@ inference. Machine-readable GPU, epoch, resident-set, capacity, lifecycle, and
 latency evidence is written to `results/phase7-server.json`.
 
 
+## Run the Phase 8 scheduler report
+
+With the validation GGUF and NVIDIA runtime available, run:
+
+```sh
+CUSCO_GPU_DEVICE_ID=0 tools/phase8-report.sh
+```
+
+The workflow runs the GPU-less 80%-per-file coverage gate and the versioned
+real-GPU workload in `config/phase8-workload.json`. It records the isolated
+baseline, mixed interactive/standard/batch results, cancellation and deadline
+injections, post-workload capacity recovery, scheduler policy and counters,
+fully drained attributed decision records, latency/starvation thresholds, build
+and workload provenance, and selected GPU in `results/phase8-server.json`.
+
 ## Run the production Compose service
 
 Place the initial model at `data/models/gemma-4-e2b-it.gguf`, then provide an
@@ -142,10 +159,10 @@ implicitly fetch models.
 
 ## Future goals
 
-Development now proceeds from dynamic residency toward:
+Development now proceeds from measured workload scheduling toward:
 
-- workload scheduling and operational hardening;
-- broader protocol and model compatibility, recovery, and deployment behavior;
+- compatibility, persistence, daemon configuration, and production packaging;
+- broader model compatibility and later execution-policy optimization;
 - optional semantic context compaction once the underlying state system is proven reliable.
 
 Each stage is intended to remain gated by correctness and measurable capacity results. The full design and phased acceptance criteria are documented in `docs/outline.md`.

compose.test.yaml

@@ -82,6 +82,30 @@ services:
       - cargo-target:/work/target
     deploy: { resources: { reservations: { devices: [{ driver: nvidia, device_ids: ["${CUSCO_GPU_DEVICE_ID:-1}"], capabilities: [gpu] }] } } }
 
+  phase8-proof:
+    build: *test-build
+    command:
+      - sh
+      - -c
+      - >-
+        umask 000 && chmod a+rwx /results &&
+        nvidia-smi --query-gpu=uuid,name,compute_cap,driver_version
+        --format=csv,noheader > /results/phase8-gpu.csv &&
+        cargo run -p cusco -- scheduler-proof
+        /models/gemma-4-e2b-it.gguf
+        --workload /work/config/phase8-workload.json
+        --output /results/phase8-server.json
+    environment:
+      CUSCO_REQUESTED_HOST_GPU: "${CUSCO_GPU_DEVICE_ID:-1}"
+      LD_LIBRARY_PATH: "/opt/llama-build/bin"
+    volumes:
+      - "${CUSCO_MODEL_DIR:-./models}:/models:ro"
+      - "${CUSCO_RESULT_DIR:-./results}:/results"
+      - cargo-registry:/root/.cargo/registry
+      - cargo-git:/root/.cargo/git
+      - cargo-target:/work/target
+    deploy: { resources: { reservations: { devices: [{ driver: nvidia, device_ids: ["${CUSCO_GPU_DEVICE_ID:-1}"], capabilities: [gpu] }] } } }
+
 volumes:
   cargo-registry:
   cargo-git:

config/phase8-workload.json

@@ -0,0 +1,114 @@
+{
+  "version": 2,
+  "name": "gemma-4-e2b-it-mixed-scheduler-v2",
+  "model_family": "gemma-4-e2b-it",
+  "policy": {
+    "version": 1,
+    "interactive_weight": 4,
+    "standard_weight": 2,
+    "batch_weight": 1,
+    "promotion_rounds": 8,
+    "deficit_refill": 1,
+    "prefill_tokens": 32,
+    "diagnostic_capacity": 2048
+  },
+  "baseline": {
+    "id": "isolated-standard",
+    "principal": "baseline",
+    "class": "standard",
+    "prompt": "Summarize why transactional state publication matters. ",
+    "prompt_repetitions": 4,
+    "max_tokens": 6,
+    "arrival_delay_ms": 0,
+    "expected": "complete"
+  },
+  "mixed": [
+    {
+      "id": "interactive-a-1",
+      "principal": "tenant-a",
+      "class": "interactive",
+      "prompt": "Give one concise scheduling invariant. ",
+      "prompt_repetitions": 1,
+      "max_tokens": 6,
+      "arrival_delay_ms": 0,
+      "expected": "complete"
+    },
+    {
+      "id": "interactive-b-1",
+      "principal": "tenant-b",
+      "class": "interactive",
+      "prompt": "Name one cancellation safety rule. ",
+      "prompt_repetitions": 1,
+      "max_tokens": 6,
+      "arrival_delay_ms": 2,
+      "expected": "complete"
+    },
+    {
+      "id": "standard-a-1",
+      "principal": "tenant-a",
+      "class": "standard",
+      "prompt": "Explain bounded fairness for an inference scheduler. ",
+      "prompt_repetitions": 4,
+      "max_tokens": 8,
+      "arrival_delay_ms": 0,
+      "expected": "complete"
+    },
+    {
+      "id": "standard-c-cancel",
+      "principal": "tenant-c",
+      "class": "standard",
+      "prompt": "Generate a response that will be cancelled after its first token. ",
+      "prompt_repetitions": 3,
+      "max_tokens": 8,
+      "arrival_delay_ms": 1,
+      "expected": "cancel"
+    },
+    {
+      "id": "standard-d-deadline",
+      "principal": "tenant-d",
+      "class": "standard",
+      "prompt": "Generate a response whose active deadline expires after its first token. ",
+      "prompt_repetitions": 3,
+      "max_tokens": 8,
+      "arrival_delay_ms": 1,
+      "expected": "deadline"
+    },
+    {
+      "id": "batch-a-1",
+      "principal": "tenant-a",
+      "class": "batch",
+      "prompt": "Describe transactional model-state ownership and safe publication. ",
+      "prompt_repetitions": 12,
+      "max_tokens": 8,
+      "arrival_delay_ms": 0,
+      "expected": "complete"
+    },
+    {
+      "id": "batch-b-1",
+      "principal": "tenant-b",
+      "class": "batch",
+      "prompt": "Describe checkpoint restoration, spill accounting, and capacity recovery. ",
+      "prompt_repetitions": 12,
+      "max_tokens": 8,
+      "arrival_delay_ms": 0,
+      "expected": "complete"
+    },
+    {
+      "id": "batch-e-1",
+      "principal": "tenant-e",
+      "class": "batch",
+      "prompt": "Explain why scheduler priority and model residency must remain separate decisions. ",
+      "prompt_repetitions": 10,
+      "max_tokens": 8,
+      "arrival_delay_ms": 3,
+      "expected": "complete"
+    }
+  ],
+  "thresholds": {
+    "max_queue_age_rounds": 128,
+    "max_first_event_baseline_multiplier": 20,
+    "max_first_event_additive_ms": 5000,
+    "max_quantum_baseline_multiplier": 20,
+    "max_quantum_additive_ms": 1000
+  }
+}

crates/cli/Cargo.toml

@@ -10,5 +10,6 @@ clap.workspace = true
 cusco-executor = { path = "../executor" }
 cusco-model-registry = { path = "../model-registry" }
 cusco-server = { path = "../server" }
+serde.workspace = true
 serde_json.workspace = true
 tokio.workspace = true