From 8b4f7d90eac4537fae00db6ed4fa35b4b249cde9 Mon Sep 17 00:00:00 2001 From: sohams25 <86193836+sohams25@users.noreply.github.com> Date: Wed, 22 Jul 2026 19:25:56 +0530 Subject: [PATCH] docs: consolidate public documentation and finalize PyPI guidance Delete release-process reports and the marketing draft directory; move the security review to docs/SECURITY_MODEL.md with process prose removed; point the README install at the live PyPI release with a pypi version badge; drop local-tooling ignore entries from the tracked .gitignore. No code, version, or packaging changes. --- .gitignore | 3 - CHANGELOG.md | 2 +- PUBLIC_RELEASE_REPORT.md | 98 -------- README.md | 12 +- RELEASE_NOTES_DRAFT.md | 133 ----------- RELEASE_READINESS_REPORT.md | 232 ------------------- docs/PYPI_RELEASE_CHECKLIST.md | 1 - SECURITY_REVIEW.md => docs/SECURITY_MODEL.md | 20 +- docs/marketing/earlyon-reddit.md | 101 -------- 9 files changed, 14 insertions(+), 588 deletions(-) delete mode 100644 PUBLIC_RELEASE_REPORT.md delete mode 100644 RELEASE_NOTES_DRAFT.md delete mode 100644 RELEASE_READINESS_REPORT.md rename SECURITY_REVIEW.md => docs/SECURITY_MODEL.md (78%) delete mode 100644 docs/marketing/earlyon-reddit.md diff --git a/.gitignore b/.gitignore index 43b77b1..75f3f3e 100644 --- a/.gitignore +++ b/.gitignore @@ -49,6 +49,3 @@ raw_earlyon.md logs/ docs/*.pth -# local agent config, never ship -CLAUDE.md -.claude/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ce3442..47884bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -61,7 +61,7 @@ format follows [keep a changelog](https://keepachangelog.com/en/1.1.0/). v1→v2 migration; a post-install wheel smoke script (`scripts/smoke_test.py`); a bounded seeded CUDA evidence runner (`scripts/evidence_run.py`); `MANIFEST.in` so the sdist stops shipping a - partial test tree; `SECURITY_REVIEW.md` and `docs/MIGRATION.md`. + partial test tree; `docs/SECURITY_MODEL.md` and `docs/MIGRATION.md`. - **staged deployment** (`earlyon.staged`): a documented protocol (ordered stages emitting continuation features + exit logits, routing between stages) with a verified reference splitter for Sequential backbones — diff --git a/PUBLIC_RELEASE_REPORT.md b/PUBLIC_RELEASE_REPORT.md deleted file mode 100644 index 5f3cfdd..0000000 --- a/PUBLIC_RELEASE_REPORT.md +++ /dev/null @@ -1,98 +0,0 @@ -# Public release report — earlyon v0.3.0 - -Date: 2026-07-22. - -## Status - -**Released on GitHub; PyPI publication blocked on one external step.** -The PR is merged, `v0.3.0` is tagged and has a GitHub Release, the published -docs preserve every negative result and hardware limitation, and a fresh -environment installs and smokes cleanly from the public tag. The -tag-triggered PyPI publish failed because **no PyPI trusted publisher (or -even project) exists for `earlyon`** — an account-level, 2FA-gated setup -that cannot be done from this environment (see Blocker). - -## Facts - -| | | -|---|---| -| Repository | https://github.com/sohams25/earlyon (public, owner `sohams25`) | -| PR | https://github.com/sohams25/earlyon/pull/5 — MERGED | -| Merge commit | `19e8072ad4d63386ebf87a6773b78514c559f851` (merge commit; 16-commit history preserved, nothing squashed or force-pushed) | -| Tag | `v0.3.0` → annotated `a18ffbf`, points at the merge commit | -| GitHub Release | https://github.com/sohams25/earlyon/releases/tag/v0.3.0 (from the finalized release notes) | -| PyPI | **Not published** — workflow run 29919523337: `invalid-publisher` (no trusted publisher configured; the v0.1.0 release job failed the same way in May 2026, so `earlyon` has never been on PyPI) | - -## Public install verification (fresh venv, outside the repo) - -`pip install git+https://github.com/sohams25/earlyon@v0.3.0` (public tag; -the PyPI path is unavailable per the blocker): - -- import + `__version__ == "0.3.0"` — PASS -- built-in model creation, training forward, routed inference — PASS -- deterministic calibration, **v2 checkpoint round-trip, v1 migration** — PASS -- staged reference execution == eager — PASS -- benchmark smoke — PASS -- installed CLI (`earlyon --version`, wrap/calibrate/benchmark/export help) — PASS - -No CIFAR retraining was performed during verification. - -## CI / tests / coverage - -- PR CI: `test (3.10)`, `test (3.11)`, `test (3.12)` all PASS (lint + black - + isort + mypy strict + pytest with coverage inside each job). -- Local at the merged commit: 270 tests pass (264 CPU + 6 CUDA on this - host), coverage 95%, mypy strict clean, ruff/black/isort clean. -- Post-merge: build + `twine check` PASSED on artifacts from `19e8072`; - wheel smoke (incl. v1 migration) PASS. - -## Release-only changes - -One commit: `d13d240` `chore(release): prepare v0.3.0` — version 0.3.0 in -`pyproject.toml` / `earlyon/__init__.py` / README badge + citation, -CHANGELOG dated, release notes finalized (limitations, migration, -bounded-evidence wording), PR body header finalized. No release-only code -fixes were needed; CI passed on the first run. - -## Honesty checks on the published pages (verified post-publish) - -The rendered v0.3.0 README/release notes state: the 1.10× throughput figure -is one seeded bounded run; the early-exit **median latency is worse** than -the full backbone (tails better); the static MobileNetV2 baseline is -competitive; noise-input 2.05× is labelled a best-case bound only; compute -fractions are estimates, not latency/energy; all-exits ONNX is not -conditional execution; staged execution has a narrow supported contract; -the Jetson procedure is documented but **unmeasured**; TensorRT is not -claimed; no universal speedup claim exists. - -## Blocker (one, external) - -**PyPI trusted publisher setup requires an interactive PyPI account login -(password + 2FA), which is unavailable here.** Exact remediation, once, by -the account owner: - -1. Log in to pypi.org → *Publishing* → *Add a pending publisher* with - exactly: project `earlyon`, owner `sohams25`, repository `earlyon`, - workflow `release.yml`, environment `pypi` (these match the OIDC claims - the failed run printed). -2. Re-run the existing failed job — no re-tagging needed: - `gh run rerun 29919523337 -R sohams25/earlyon` - (artifacts were already built from the tagged commit; only the publish - job re-executes). -3. Verify: https://pypi.org/project/earlyon/ shows 0.3.0, then - `python -m venv /tmp/e && /tmp/e/bin/pip install earlyon==0.3.0` and run - `scripts/smoke_test.py` from outside the repo. - -## Remaining unmeasured hardware evidence - -Jetson (tegrastats energy, staged TensorRT) — procedure documented in -`docs/STAGED_DEPLOYMENT.md`; no numbers exist and none are claimed. - -## Maintenance recommendation - -Freeze broad feature work. Focus on: user reports against 0.3.0, -compatibility fixes (new torch/torchvision releases), completing the PyPI -publisher setup above, and producing real Jetson evidence before any -hardware claim. The v0.3 legacy benchmark table in the README should be -retired only after a full-length `scripts/run_benchmarks.py` run on target -hardware. diff --git a/README.md b/README.md index 84716f5..d04ec06 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ python coverage mypy - version + pypi license

@@ -46,16 +46,16 @@ result.confidence # softmax confidence where it answered ## Install -PyPI publication is pending trusted-publisher configuration. The v0.3.0 -GitHub tag is the current installable release: +```bash +pip install earlyon==0.3.0 +``` + +Or from the GitHub tag: ```bash pip install git+https://github.com/sohams25/earlyon@v0.3.0 ``` -(Once PyPI publication completes, `pip install earlyon==0.3.0` will work; -until then it does not.) - From source for development: ```bash diff --git a/RELEASE_NOTES_DRAFT.md b/RELEASE_NOTES_DRAFT.md deleted file mode 100644 index 33e5b65..0000000 --- a/RELEASE_NOTES_DRAFT.md +++ /dev/null @@ -1,133 +0,0 @@ -# earlyon v0.3.0 — release notes - -(This file is the canonical text for the v0.3.0 GitHub Release.) - -earlyon 0.3.0 is a correctness release for the early-exit toolkit. It makes -the calibration methodology defensible (per-head temperatures, explicit exit -enablement, strict data-split discipline), the measurements fair (one -benchmark core, identical samples and boundaries for every compared model), -and the claims honest (estimated FLOPs labelled as estimates, legacy numbers -quarantined, "production-ready" retired in favor of "deployment-oriented"). - -## Install - -```bash -pip install earlyon==0.3.0 -``` - -Python 3.10+, torch 2.0+. Existing checkpoints load with automatic -migration — see **Migration** below. - -## Methodological corrections - -- **One temperature per head.** Temperature scaling used to be fit once from - the final classifier's logits and reused at every exit, though each exit - head is a differently miscalibrated classifier. 0.3 fits and applies a - temperature per head (`config.temperatures`), with per-fit convergence - status; a regression test proves a single global temperature misroutes - where per-head fitting does not. -- **Disabled exits cannot fire.** Disabling via threshold sentinels - (confidence `1.0` / entropy `0.0`) was numerically unsound: a - float32-saturated softmax met the criterion. Enablement is now an explicit - boolean per exit (`config.enabled_exits`), honored by the confidence and - entropy policies on the single-sample, batched and staged paths. -- **Staged calibration pipeline.** Collect logits once → fit per-head - temperatures → greedy threshold grid search on the cache → evaluate; each - stage separately testable, the simulator pinned bit-exact against the real - router, threshold search never sees test labels, never-firing exits stay - disabled, empty loaders fail fast. The method is named what it is - (`greedy-coordinate-grid`) — no global-optimality claim. -- **Centralized validation.** `EarlyExitConfig.validate()` raises actionable - errors for invalid config (a negative temperature used to be silently - clamped to `1e-6`, producing an artificially sharp softmax). - -## Fair benchmarking - -`benchmark_models` measures every compared model — full backbone, early-exit -wrapper, a static smaller baseline, a quantised variant — on the **exact -same preloaded sample sequence** with identical warmup, boundary -(model-only vs end-to-end, labelled) and synchronization, reporting -p50/p95/p99, throughput, exit distribution and accuracy for labelled -loaders. Pre-0.3 benchmark records are quarantined as `legacy_v0_2` in -`docs/benchmarks.json`: they fed compared models different inputs and are -not comparable. - -## Honest compute terminology - -`computation_used` is renamed `estimated_backbone_flops_fraction` (read -alias kept). It is a static estimate — it excludes exit-head cost and all -routing overhead, and it is neither a latency nor an energy measurement. -The estimator (`FlopsEstimate`) reports its method and reliability, detects -reused modules (degrading to a warned low-confidence fallback), and runs -lazily so large-model construction is fast. - -## Staged execution - -`earlyon.staged` defines a staged-deployment contract (ordered stages -emitting continuation features + exit logits, routing applied between -stages) with a verified reference splitter for `nn.Sequential` backbones. -Staged execution provably matches eager routing and *structurally* skips -later stages — unlike the all-exits ONNX export -(`export_all_exits_to_onnx`), which always computes every exit and is a -portability artifact, not conditional execution. The supported contract is -deliberately narrow; unsupported graphs are rejected. See -`docs/STAGED_DEPLOYMENT.md`. - -## Checkpoints and migration - -Checkpoints now carry `format_version: 2` (exit-point identity, routing -policy, thresholds, enablement, per-head temperatures, library version) and -are cross-checked against the rebuilt factory on load. Files written by -earlyon ≤ 0.2 migrate automatically and deterministically — scalar -temperature broadcast per head, legacy sentinels → explicit disables — with -a warning; the migration is pinned against a genuine v0.2-written fixture. -`custom_ee` models reload via `load_wrapper(path, factory=...)`. -Details and code-level changes: `docs/MIGRATION.md`. - -## Also fixed - -- `custom_ee`: example args/kwargs for inspection, device inference (no more - CPU probes of CUDA models), per-layer feature extractors for - tuple/dict/odd-rank outputs, and dry-run validation that every exit layer - runs exactly once in the listed order. -- Jetson monitoring: `TegrastatsMonitor` is restartable (a restarted monitor - previously collected nothing), missing telemetry parses to `None` instead - of invented zeros, and `profile_with_energy` distinguishes instantaneous - power, window-average power and trapezoid-integrated energy. - -## Real CUDA evidence (one seeded, bounded run — not a universal claim) - -A deliberately small validation run (CIFAR-10 at 224px, ResNet-18 EE -fine-tuned 2+2 epochs from ImageNet weights, seed 42, disjoint -train/temperature/calibration/test splits, RTX 4050 Laptop GPU; full data in -`docs/evidence/`): - -- early-exit ResNet-18: 92.92% test accuracy vs 94.07% full backbone; - **1.10× throughput** with **worse median latency** (1.43 vs 1.33 ms) and - better tail latency (p95/p99) — routing overhead is real on the ~70% of - images that ran the full network; -- a static MobileNetV2 baseline trained the same way was competitive - (92.67%, 0.99×) — smaller static models remain an essential comparison; -- a noise-input 2.05× figure exists only as a labelled best-case bound, not - a practical speed claim. - -Early exit is **not always faster**; the fair runner exists so you can see -when it isn't. - -## Limitations - -- Dynamic routing is primarily eager PyTorch execution (hooks + host sync); - `torch.compile` is refused with a clear error. -- All-exits ONNX export computes every exit — no conditional execution. -- Staged reference execution supports Sequential backbones only. -- The Jetson/tegrastats procedure is documented but **unmeasured** — no - Jetson numbers exist; TensorRT deployment is not claimed. -- Estimated FLOPs fractions are estimates, not measured latency or energy. -- `forward(mode="inference")` is batch-1; batched routing is conservative. - -## Verification behind this release - -270 tests (incl. CUDA suite on a CUDA host), 95% coverage, mypy strict, -ruff/black/isort clean; wheel + sdist twine-clean and smoked from a clean -environment (`scripts/smoke_test.py`); security review in -`SECURITY_REVIEW.md`; full audit trail in `RELEASE_READINESS_REPORT.md`. diff --git a/RELEASE_READINESS_REPORT.md b/RELEASE_READINESS_REPORT.md deleted file mode 100644 index 2ee5393..0000000 --- a/RELEASE_READINESS_REPORT.md +++ /dev/null @@ -1,232 +0,0 @@ -# Release readiness report — earlyon 0.3 release candidate - -Date: 2026-07-22. - -## 1. Executive status - -**COMPLETE.** Every hardening claim from the handoff was independently -re-verified (and, where under-tested, adversarially tested); one packaging -defect and one evidence-runner defect were found and fixed; the package -builds, twine-checks, installs into a clean environment and passes an -out-of-repo smoke; v1 checkpoint migration is proven against a genuine -prior-version file; a bounded, seeded CUDA evidence run produced real -CIFAR-10 numbers with the negatives preserved. No remote state was touched; -no Jetson/TensorRT claims exist because no such hardware exists here. - -## 2. Branches / commits - -- Starting point: the v0.3 hardening branch @ `db3832e` (8 hardening - commits on `main` @ `a5f8414`), working tree clean, untouched. -- Work branch: the release-readiness branch, created from the hardening - HEAD (hardening commits preserved, not rebased/squashed); merged to - `main` via PR #5 and since deleted. -- Final HEAD: the last commit listed in §16. - -## 3. Environment - -Python 3.13.9 · torch 2.10.0+cu128 · torchvision 0.25.0+cu128 · CUDA 12.8 · -NVIDIA RTX 4050 Laptop GPU (6 GB) · onnx 1.21.0 / onnxruntime 1.25.1 · -fvcore 0.1.5 · Linux 6.8. **Absent:** tegrastats, TensorRT, Jetson (all -verified absent; nothing was claimed from them). - -## 4. Claims independently verified - -All of the handoff's headline claims reproduced from scratch: - -- 261 tests passing at baseline (now 270), 95% coverage (badge said 96% — - corrected), strict mypy clean, ruff/black/isort clean for the CI scope. -- **Per-head temperatures (A):** independent per-head fit with convergence - status; deterministic head mapping; routing applies each head's own T on - the single-sample, batched (newly pinned) and staged paths; persisted in - v2 checkpoints; scalar v1 migrates deterministically; invalid values - raise / fall back safely. New decisive test: a single global temperature - demonstrably misroutes (accuracy 1.0 → <0.5) where per-head does not. -- **Explicit enablement (B):** disabled exits cannot fire at confidence - exactly 1.0 or entropy exactly 0.0 (both policies, both paths); all- - disabled routing reaches the final head; no sentinel thresholds remain; - migration preserves state. -- **Staged calibration (C):** collect/fit/search/evaluate are separable and - contracted; simulator pinned bit-exact against the real router; no test - leakage (split discipline documented and followed by the evidence run); - search is deterministic (newly pinned); edge cases covered: empty loader - raises, single-class data works, no-feasible-threshold → disabled, - NaN-emitting head safely disabled (new), NaN logits rejected in fitting, - non-converged fits fall back to 1.0 with status. -- **Benchmark fairness (D):** identical preloaded samples/order/warmup/ - sync/boundary proven by spy-module tests; batch-1 validated; model-only - vs end-to-end labelled; negative speedups preserved (ratio, no clamp); - full backbone + early-exit + static baseline compared in the evidence run. -- **Compute estimates (E):** `estimated_backbone_flops_fraction` naming, - `FlopsEstimate` provenance, exit-head exclusion disclosed, reused-module - detection → warned low-confidence fallback, lazy + cached analysis - (construction fast). Cache invalidation limitation documented (§17). -- **Checkpoint v2 (F):** format/library version, exit-point identity - cross-check, routing/enablement/temperatures/loss metadata, deterministic - v1 migration, future-version rejection, malformed-file rejection, - `weights_only=True` trust model documented in SECURITY_REVIEW.md. Newly - verified against a **genuine** prior-version checkpoint written by the - actual historical `save_wrapper` (committed fixture, ~1.2 MB, provenance - documented). -- **custom_ee (G):** CPU + CUDA (GPU test passes on this host), kwargs, - tuple and dict outputs via feature extractors (dict newly pinned), - missing/reused/out-of-order exits fail loudly; claims match tested scope - (single-tensor runtime contract stated in the docstring/README). -- **Jetson (H):** restartability, double-start no-op, thread/subprocess/pipe - cleanup, missing-tool degradation, conservative parsing (None ≠ 0), - trapezoidal integration with correct units, instantaneous vs average vs - integrated energy kept distinct — all covered by hardware-free tests. No - non-Jetson output is presented as Jetson evidence anywhere. -- **Staged deployment (I):** newly proven that later stages are *never - executed* when an exit fires (counting-module test); staged == eager - predictions/exit choices across policies, temperatures, enablement; - unsupported graphs rejected; all-exits ONNX vs conditional execution - distinction explicit in API name and docs. - -## 5. Defects discovered (beyond the handoff) - -1. **sdist shipped a partial test tree** (test modules without the fixtures - package) — broken if anyone ran tests from the sdist. -2. **The v1-migration fixture wasn't actually tracked** (`*.pth` gitignore), - so the committed migration test would fail on a fresh clone. -3. **Evidence runner OOM'd on 6 GB GPUs** (MobileNetV2 at batch 128 with the - wrapper still resident). -4. Lint/format debt in `scripts/` and `examples/` (outside CI's checked - scope): unused import, E402s, ambiguous lambda names, unformatted files. -5. README coverage badge overstated coverage (96% vs measured 95%). - -## 6. Fixes made - -(1) `MANIFEST.in` prunes tests from the sdist. (2) fixture force-added with -a provenance README. (3) baseline batch 32 + off-GPU staging between phases. -(4) fixed and brought into the format gate. (5) badge corrected. All -conservative; no behavior changes to library code were needed — the audit -found no functional defect in the hardening work itself. - -## 7. Tests, coverage, static checks (final) - -- `pytest` (full, incl. 6 CUDA tests): **270 passed**, ~13 s. -- Coverage: **95%** (`--cov=earlyon`). -- `mypy earlyon` (strict): clean, 28 files. -- `ruff check earlyon tests scripts examples`: clean. -- `black --check` / `isort --check-only` over the same scope: clean. - -## 8. Packaging - -`python -m build` → `earlyon-0.2.0-py3-none-any.whl` (73 KB) + -`earlyon-0.2.0.tar.gz`; both `twine check` PASSED; `pip check` clean. -Wheel installed into a fresh virtualenv and smoked **from outside the -repo** (`scripts/smoke_test.py`): import/version, model build, training + -routed inference, deterministic calibration, v2 round-trip, v1 migration, -staged==eager, benchmark smoke — all PASS; CLI entry point and all -subcommand `--help`s work from the installed console script. Caveat: the -clean venv used `--system-site-packages` for the torch stack (a from-scratch -torch download was out of scope); earlyon itself and its metadata were -installed with `--no-deps` from the wheel. - -## 9. v1 migration result - -PASS — against both hand-built payloads and the genuine historical fixture: -scalar temperature 1.6 broadcast to all four heads, sentinel threshold 1.0 -→ `enabled_exits=[True, False, True]`, warning emitted, model routes, -re-save produces a silent v2 round-trip. - -## 10. CUDA evidence - -Run completed (seed 42, 2+2+2 epochs, 10.4 min compute; full data in -`docs/evidence/cuda_evidence.json`, interpretation in -`docs/evidence/CUDA_EVIDENCE.md`): - -- ResNet-18 backbone 94.07% test acc, 1.33 ms p50, 712 ips. -- Early-exit ResNet-18 92.92% (−1.15% vs the 1% budget calibrated on the - calibration split — generalization gap reported), est. FLOPs fraction - 0.88, **1.10× throughput but worse median latency** (1.43 ms) with better - tails; 30.5% of images exited early. -- Static MobileNetV2 92.67%, 0.99× — competitive, as the docs predict at - tiny training budgets. -- Noise-input 2.05× recorded separately as a best-case bound. Peak CUDA - memory 350 MB during the timed comparison. No result was tuned on test - data; no negative was suppressed. - -## 11. Jetson / TensorRT status - -No hardware, no runtime, therefore **no claims**: monitor lifecycle, parser -and energy integration are verified hardware-free; the exact device -procedure is documented (`docs/STAGED_DEPLOYMENT.md` §Jetson, README -Reproducibility). Restartability on a real Jetson and any TensorRT staged -run remain external work. - -## 12. Security / privacy - -`SECURITY_REVIEW.md` created. Findings: no secrets/keys/tokens; no private -paths or employer identifiers; datasets and build artifacts gitignored; only -committed binary is the documented 1.2 MB untrained migration fixture; -`torch.load(weights_only=True)` + structural validation on every load path; -single subprocess (tegrastats) with fixed argv and cleanup; no eval/exec/ -shell=True; wheel/sdist contain no tests, data or weights. - -## 13. Public API changes (this branch) - -None. All library-code changes on this branch are tests, scripts, packaging -and docs. (API changes from the hardening branch are summarized in -`docs/MIGRATION.md` and `RELEASE_NOTES_DRAFT.md`.) - -## 14. Documentation changes - -README (architecture diagram, evidence-run results table with negative- -result interpretation, checkpoint-migration + reproducibility sections, -corrected badge), `docs/MIGRATION.md`, `SECURITY_REVIEW.md`, -`PR_BODY_DRAFT.md` (draft since superseded by PR #5 and removed), -`docs/evidence/CUDA_EVIDENCE.md`, updated -`RELEASE_NOTES_DRAFT.md` and `CHANGELOG.md`, `tests/fixtures/README.md`. - -## 15. Files changed (this branch, 17 files) - -- tests: `tests/test_release_audit.py` (new, 10 tests), - `tests/fixtures/v1_cifar_resnet20.pth` + `README.md` (new) -- scripts: `smoke_test.py` (new), `evidence_run.py` (new), - `run_benchmarks.py` / `re_evaluate.py` / examples (lint/format only) -- packaging: `MANIFEST.in` (new) -- docs: README, CHANGELOG, RELEASE_NOTES_DRAFT, MIGRATION, SECURITY_REVIEW, - PR_BODY_DRAFT (since removed), docs/evidence/* (this report) - -## 16. Local commits (release-readiness work on top of the hardening branch) - -1. `5d821aa` chore: bring scripts and examples up to the lint/format gate -2. `de8ce1c` test: independent release audit — adversarial verification -3. `1a0d90f` feat: packaging validation — sdist pruning, wheel smoke, v1 fixture, evidence runner -4. `e381db2` docs: security review, migration guide, PR draft, README architecture -5. `5ce0427` docs: coverage badge fix, changelog, release-notes audit section -6. `e38b073` feat: bounded CUDA evidence run — real seeded CIFAR-10 results -7. (final) docs: release readiness report - -## 17. Remaining external work - -- Jetson: run `earlyon profile` + the staged TensorRT procedure on a real - device before making any Jetson claim. -- Optional: full-length `scripts/run_benchmarks.py` (hours) to refresh - `docs/benchmarks.json` `runs` and retire the legacy README table. -- Torchvision-factory stage maps for `earlyon.staged` (protocol + acceptance - test exist). -- Known limitation (documented): `wrapper.flops_estimate` caches per - instance and does not invalidate if the backbone is structurally mutated - after first inference. -- Version bump to 0.3.0 + changelog dating at actual release time. - -## 18. Release commands (since executed — kept for the record) - -The steps below were subsequently carried out: the branch was merged via -PR #5, `v0.3.0` was tagged and the GitHub Release published. Only the PyPI -upload remains (blocked on trusted-publisher setup; see -`PUBLIC_RELEASE_REPORT.md`). - -```bash -# push + PR (done: PR #5, merged) -gh pr create --base main \ - --title "Release v0.3.0: defensible calibration, fair benchmarks, and staged inference" - -# after merge: version bump, tag, publish -# edit pyproject.toml + earlyon/__init__.py -> 0.3.0; date the CHANGELOG -git tag -a v0.3.0 -m "earlyon 0.3.0" -git push origin v0.3.0 -python -m build && python -m twine check dist/* && python -m twine upload dist/* -``` diff --git a/docs/PYPI_RELEASE_CHECKLIST.md b/docs/PYPI_RELEASE_CHECKLIST.md index 87e5c38..01d9fbd 100644 --- a/docs/PYPI_RELEASE_CHECKLIST.md +++ b/docs/PYPI_RELEASE_CHECKLIST.md @@ -58,7 +58,6 @@ manual, human decision — never automated. - [ ] Import + CLI smoke test from that venv (same commands as §3) - [ ] Create the GitHub release from the tag, pasting the CHANGELOG section: `gh release create vX.Y.Z --notes-file <(sed -n '/X.Y.Z/,/^## /p' CHANGELOG.md)` -- [ ] Announce (see `docs/marketing/`) ## If something went wrong diff --git a/SECURITY_REVIEW.md b/docs/SECURITY_MODEL.md similarity index 78% rename from SECURITY_REVIEW.md rename to docs/SECURITY_MODEL.md index 96a8cc3..10c2e9f 100644 --- a/SECURITY_REVIEW.md +++ b/docs/SECURITY_MODEL.md @@ -1,8 +1,7 @@ -# Security review — earlyon 0.3 release candidate +# Security model -Date: 2026-07-22. Scope: the `earlyon` package, scripts, tests, docs and -packaged artifacts on the v0.3.0 release-readiness branch (merged to -`main` via PR #5). +Scope: the `earlyon` package, scripts, tests, docs and packaged artifacts +as of v0.3.0. This is a project self-review, not an external audit. ## Checkpoint trust assumptions @@ -57,17 +56,12 @@ packaged artifacts on the v0.3.0 release-readiness branch (merged to - Tests use `tmp_path`/`TemporaryDirectory`; no world-writable fixed paths. - No archive extraction is performed by earlyon itself. -## Scan results (this review) +## Packaging boundaries -- No credentials, API keys, tokens or private keys in the tree. -- No private/internal paths, employer identifiers, or non-public material. - The author contact in `pyproject.toml` is deliberate public metadata. -- Coverage output, datasets, build artifacts (`htmlcov/`, `data/`, `dist/`, - `.coverage`) are gitignored and untracked. - The wheel contains only the `earlyon` package; the sdist prunes `tests/` - (`MANIFEST.in`), so the binary fixture ships in neither artifact. + (`MANIFEST.in`), so the binary test fixture ships in neither artifact. ## Reporting -Vulnerabilities: follow `SECURITY.md` (private disclosure; do not open -public issues for security reports). +Vulnerabilities: follow [`SECURITY.md`](../SECURITY.md) (private +disclosure; do not open public issues for security reports). diff --git a/docs/marketing/earlyon-reddit.md b/docs/marketing/earlyon-reddit.md deleted file mode 100644 index 0afb28e..0000000 --- a/docs/marketing/earlyon-reddit.md +++ /dev/null @@ -1,101 +0,0 @@ -> NOTE (v0.3.0): this is archived v0.2-era launch copy. The numbers below -> predate the v0.3 fair benchmark runner and are superseded by -> docs/evidence/CUDA_EVIDENCE.md; do not reuse them. - -# r/MachineLearning launch draft - -Flair: [P] (Project). Post as text, not a link post; r/ML buries bare links. -Best window: Tue–Thu, 14:00–16:00 UTC. - ---- - -## Title - -[P] earlyon: skip the last ResNet layers when the network is already -confident. On CIFAR-10, 58% of images never reach layer4. - -## Body - -A deep network runs every layer for every image, even the obvious ones. A -frontal close-up of a car and a blurry half-occluded bird both pay the full -forward pass. - -Early-exit networks fix this: attach small classifier heads partway through -the backbone, and let confident predictions leave early. The research is -mature. BranchyNet was 2016, and a 2024 ACM survey covers eight years of -follow-ups showing 1.3–2.5× compute savings at single-sample edge inference. -But every paper ships custom code for its one architecture. There was no tool -you could `pip install`, so I built one: - -``` -pip install earlyon -``` - -```python -from earlyon.models import resnet50_ee - -model = resnet50_ee(num_classes=10, pretrained=True).eval() -result = model(x, mode="inference") - -result.exit_taken # which head fired (-1 = full network) -result.estimated_backbone_flops_fraction # estimated fraction of backbone FLOPs (not measured latency) -result.confidence # how sure it was when it left -``` - -Numbers (CIFAR-10, trained on a laptop RTX 4050, thresholds calibrated to a -1% accuracy budget): - -| Model | Test Acc | Baseline | Avg FLOPs used | % exited early | -|---|---:|---:|---:|---:| -| ResNet18 | 94.42% | 96.32% | 89.88% | 35.3% | -| ResNet50 | 95.88% | 97.60% | 81.42% | 58.2% | -| MobileNetV2 | 93.31% | 95.08% | 93.90% | 8.5% | - -The headline stat: with thresholds calibrated for a ~1% accuracy budget, -58% of CIFAR-10 images exit ResNet50 before layer4, cutting ~19% of FLOPs on -the real test set. I'm only publishing FLOPs measured on real test images. -Wall-clock depends on your hardware and input mix, and I don't have a Jetson -number I trust enough to print yet. The repo ships a Jetson profiler (power + -thermals via `tegrastats`) so you can measure yours. - -Things I tried to get right: - -- The tool reports average FLOPs actually skipped on real test images, not - the theoretical best case. It also tells you when early exit doesn't help. - I left the MobileNetV2 row in even though it loses: that backbone is - already cheap, so only 8.5% of images exit early. -- No backbone surgery. Exit heads attach via forward hooks, so your existing - weights load unchanged. `custom_ee` wraps any `nn.Module`, including ViT - (token-pooled exits after encoder blocks). -- Training stays your training. Two-stage by default: train the backbone with - your existing recipe (or start from any pretrained checkpoint), then freeze - it and train the tiny heads for a few epochs. Joint training ships too if - you want peak accuracy. -- Thresholds come from a stated budget, not hand-tuning. Give an accuracy - budget ("within 1% of baseline") or a compute budget ("80% of FLOPs on - average") and a greedy sweep finds per-exit thresholds, with optional - temperature scaling first because raw softmax confidence is miscalibrated. - -Limitations, since r/ML will ask: per-sample routing is batch-1 (there's a -batched mode where the whole batch exits together); `torch.compile` can't -trace the routing control flow (compile the raw backbone instead); ONNX -export is static-graph only, all exits computed and routing left to the -caller. Results above are CIFAR-10; ImageNet-scale benchmarks are the next -milestone. - -Code, benchmarks, and the design-decisions doc: -https://github.com/sohams25/earlyon - -Happy to answer questions about the calibration method, the hook + sentinel -routing mechanism, or edge deployment. - ---- - -## First comment (post immediately, from the same account) - -Design decisions write-up (why two-stage training, why greedy calibration, why -temperature scaling, why entropy routing ships alongside confidence): -https://github.com/sohams25/earlyon/blob/main/docs/DESIGN_DECISIONS.md - -Raw benchmark JSON, reproducible with `scripts/run_benchmarks.py`: -https://github.com/sohams25/earlyon/blob/main/docs/benchmarks.json