Skip to content

Stage M6 — Static decode plan on Apple *(old 20, Apple half)* #97

Description

@thanos

Stage M6 — Static decode plan on Apple (old 20, Apple half)

Goal: make the Apple hot decode path boring.

  • After warm-up: weights resident, KV preallocated to context bound,
    scratch arena fixed, pipelines compiled and cached, zero heap
    allocation per token
    — asserted by instrumentation, not claimed.
  • Bounded in-flight work; predictable encode/submit structure per token;
    ICB/replay decision from M3 finalized here.
  • Memory report: weights / KV / scratch / peak RSS, per model and
    context length.

Gate: an allocation-tracking assertion passes over a long generation;
per-token latency variance is characterized (p50/p95/p99).
Tutorial: docs/tutorials/20-static-decode-plan.md.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions