This repository was archived by the owner on Oct 10, 2026. It is now read-only.
Repository navigation
docs(007): publish audit — keep seed-morph78 weights local-only #107
Merged
scrimshawlife-ctrl
merged 3 commits into
main
from
cursor/hyperlexical-publish-audit-41af
Sep 24, 2026
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
a21e4ce
docs(007): publish audit - keep seed-morph78 weights local-only
scrimshawlife-ctrl 9bf59d0
docs(007): publish audit - keep seed-morph78 weights local-only
scrimshawlife-ctrl 9ac1437
docs(007): publish audit - keep seed-morph78 weights local-only
scrimshawlife-ctrl File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| # Publish audit — `seed-morph78` weights (2026-09-24) | ||
|
|
||
| **Question:** should the `seed-morph78` weights (`hyperlex-structure-149m`) be published to the Hugging Face Hub now? | ||
|
|
||
| **Recommendation: DO NOT PUBLISH YET.** Keep weights local-only. Two blockers are about correctness, not paperwork: the promotion metric is contaminated, and the weights cannot be rebuilt from the canonical repo. Evidence: `receipts/publish-audit-20260924/`. | ||
|
|
||
| This audit does not revert `name_gate`, BEST, or any receipt. Hub stays an operator decision (C11). | ||
|
|
||
| ## Blockers | ||
|
|
||
| ### B1 — Promotion metric is train-contaminated (OBSERVED) | ||
|
|
||
| The soft_ceiling "broad OBSERVED val" eval (`eval_broad_observed.py`, off-git on Spark) scores every OBSERVED unbind row with `split=val`, but the force-train file adds val rows to training. | ||
|
|
||
| | Surface | n | morph78 | morph65 | | ||
| |---|--:|--:|--:| | ||
| | Broad val used for PROMOTE_BEST | 256 | 0.9883 | 0.8867 | | ||
| | … of which in `force_train_morph78_expanded.jsonl` | 193 | — | — | | ||
| | **Leak-free val** (not force-trained, text not in train split) | **63** | **1.0** | **1.0** | | ||
|
|
||
| The entire promotion margin comes from rows morph78 trained on. On leak-free val the two pins tie. PROMOTE_BEST shows better fit, not better generalization. (`clean-val-scores.json`, `audit_clean_val.py`.) | ||
|
|
||
| Related prior evidence: `receipts/2026-09-14-dual-scheme-holdout.md` scored structure-exact **0.0 (n=12)** on held-out structure bases while val was 0.625. The **test split (440 SoT rows) has never been scored** for any climb pin. | ||
|
|
||
| ### B2 — Weights are not reproducible from the canonical repo (OBSERVED) | ||
|
|
||
| - morph78 trained from Spark `~/Hyperlex` at `78c4d83` (branch `cursor/morph51-reject-escalate-963d`), **39 commits behind `main`**. | ||
| - That checkout has **uncommitted edits** to `export.py`, `loop.py`, `unbind_recipe.py` (799-line diff, including data-filtering rules not on `main`): `spark-uncommitted-edits.diff`. | ||
| - Settle / force-expand / finish / broad-eval logic lives in **86 scripts under `~/hlx`, outside git** (e.g. `promote_launch_morph78_val_settle.py`, `eval_broad_observed.py`). | ||
| - `train-receipt.json` records `data_sha256` but no code commit or patch hash. | ||
|
|
||
| A Hub card claiming these metrics could not cite code that produces them. | ||
|
|
||
| ## Other gaps | ||
|
|
||
| | # | Gap | State | | ||
| |---|---|---| | ||
| | G1 | Data rights | SoT 4,297 rows: `operator-local` 3,580, `operator-attested` 647, CC BY-SA (Wiktionary) 14, crawl 7. No dataset statement or redistribution basis for a model trained on crawled web slang; README already warns web text is not automatically redistributable. | | ||
| | G2 | License conflict | Personal repo MIT · org repo Zero State Proprietary · card frontmatter apache-2.0 (trunk's license). Weights licence and owner (Zero State vs personal) undecided. | | ||
| | G3 | Calibration | Classify returns `none` at confidence ≈1.0 on val-settled phrases (`receipts/20260924-infer-model-morph78.md`). E1 / E3 NOT_COMPUTABLE. | | ||
| | G4 | Size ceiling | 150,546,889 params (trunk 149,014,272 + heads) vs A1 "T1 ≤ 150M". Operator reading pending. | | ||
| | G5 | Loadability | Custom heads; no `trust_remote_code` modeling file or packaged loader on the card. Hub users could not load it without this repo. | | ||
| | G6 | Card evidence | Card metrics are val (contaminated) and train-val; no test table, no data statement. | | ||
|
|
||
| ## What is already up to practice | ||
|
|
||
| safetensors weights · no chat template or generation head · Brier null / not a forecast · restricted inputs drop surface · dual-use gate rows 1–12 hold and the card has out-of-scope text (row 9) · local-only trunk, no Hub fetch · receipts for every promote/reject · public claim is Restricted (CLAIM-HLX-NAME-004). | ||
|
|
||
| ## Path to a publishable release (in order) | ||
|
|
||
| 1. **Canonicalize training code.** Commit the Spark checkout edits and the `~/hlx` settle/finish/eval scripts into `scripts/shadow/` on `main`. Record `code_commit` in train receipts. | ||
| 2. **Fix the gate eval.** Broad val must exclude rows present in the force-train file (and any text in train). Re-run soft_ceiling decide for morph78 vs morph65 on the corrected surface. | ||
| 3. **Spend the test split once.** Score morph78 (and morph65) on `split=test` with canonical code. That number goes on the card. | ||
| 4. **Decide data + licence.** Dataset statement; which rows may train a public model (drop or isolate CC BY-SA if share-alike is a concern); weights licence and owner. | ||
| 5. **Calibration + A1.** Either calibrate classify or omit lineage confidence from the card; record the A1 reading. | ||
| 6. **Package.** Loader (`trust_remote_code` file or pip extra), card with test metrics and data statement. | ||
| 7. **Then** a private/gated Hub repo first; public only on a separate sentence. | ||
|
|
||
| ## Name gate note | ||
|
|
||
| `name_gate` rests on trained E2 PASS vs the Spec 004 probe, which this audit does not contradict. The PROMOTE_BEST decision over morph65 is not supported on leak-free val; whether to keep morph78 as BEST or treat morph65/morph78 as tied is an operator call. |
24 changes: 24 additions & 0 deletions
24
specs/007-hyperlexical-model/receipts/publish-audit-20260924/audit_clean_val.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| import json, os, sys | ||
| from pathlib import Path | ||
| sys.path.insert(0, "/home/morpheus/Hyperlex/scripts/shadow") | ||
| import torch | ||
| from transformers import AutoModel, AutoTokenizer | ||
| from hyperlexical.eval_forward import _weight_parts, _load_maps, _load_filler_head, apply_encoder_trainable, score_unbind_exact | ||
| from torch import nn | ||
| TRUNK = Path("/home/morpheus/.hyperlex/models/trunks/ModernBERT-base") | ||
| rows = json.loads(Path("/home/morpheus/hlx/audit-clean-val.json").read_text()) | ||
| out = {} | ||
| for name in ("morph78", "morph65"): | ||
| M = Path(f"/home/morpheus/.hyperlex/models/hyperlex-encoder-modernbert-base-seed-{name}") | ||
| dev = torch.device("cuda" if torch.cuda.is_available() else "cpu") | ||
| tok = AutoTokenizer.from_pretrained(str(TRUNK), local_files_only=True) | ||
| enc = AutoModel.from_pretrained(str(TRUNK), local_files_only=True) | ||
| fs, et, hb = _weight_parts(M / "model.safetensors", torch) | ||
| apply_encoder_trainable(enc, et) | ||
| maps = _load_maps(M, hb) | ||
| fh = _load_filler_head(nn, 768, fs, maps) | ||
| enc.to(dev); fh.to(dev) | ||
| s = score_unbind_exact(enc, fh, tok, maps, rows, dev) | ||
| out[name] = {"unbind_exact": s.get("unbind_exact"), "n": s.get("n_unbind_eval")} | ||
| print("AUDIT_CLEAN", json.dumps(out)) | ||
| Path("/home/morpheus/hlx/audit-clean-val-scores.json").write_text(json.dumps(out, indent=2)) | ||
10 changes: 10 additions & 0 deletions
10
specs/007-hyperlexical-model/receipts/publish-audit-20260924/clean-val-scores.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| { | ||
| "morph78": { | ||
| "unbind_exact": 1.0, | ||
| "n": 63 | ||
| }, | ||
| "morph65": { | ||
| "unbind_exact": 1.0, | ||
| "n": 63 | ||
| } | ||
| } |
131 changes: 131 additions & 0 deletions
131
specs/007-hyperlexical-model/receipts/publish-audit-20260924/eval_broad_observed.offgit.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| #!/usr/bin/env python3 | ||
| """Score a Hyperlex encoder on broad OBSERVED unbind val (no force shrink). | ||
|
|
||
| Used by soft_ceiling_tiebreak when force-fair == 1.0: promote requires | ||
| candidate broad OBSERVED exact > morph65 baseline 0.889763779527559 n=254. | ||
|
|
||
| Env: | ||
| HLX_FAIR_MODEL model dir (default morph65 BEST) | ||
| HLX_BROAD_OUT output JSON path | ||
| HLX_BROAD_PRIV optional private copy dir | ||
| """ | ||
| from __future__ import annotations | ||
|
|
||
| import json | ||
| import os | ||
| import sys | ||
| from datetime import datetime, timezone | ||
| from pathlib import Path | ||
|
|
||
| ROOT = Path("/home/morpheus/Hyperlex") | ||
| sys.path.insert(0, str(ROOT / "scripts/shadow")) | ||
|
|
||
| import torch | ||
| from torch import nn | ||
| from transformers import AutoModel, AutoTokenizer | ||
| from hyperlexical.export import export_dataset, repo_root | ||
| from hyperlexical.layout import HIDDEN | ||
| from hyperlexical.eval_forward import ( | ||
| apply_encoder_trainable, | ||
| score_unbind_exact, | ||
| _weight_parts, | ||
| _load_maps, | ||
| _load_filler_head, | ||
| ) | ||
|
|
||
| TRUNK = Path( | ||
| os.environ.get( | ||
| "HYPERLEX_TRUNK_DIR", | ||
| str(Path.home() / ".hyperlex/models/trunks/ModernBERT-base"), | ||
| ) | ||
| ) | ||
| MODEL = Path( | ||
| os.environ.get( | ||
| "HLX_FAIR_MODEL", | ||
| str(Path.home() / ".hyperlex/models/hyperlex-encoder-modernbert-base-seed-morph65"), | ||
| ) | ||
| ) | ||
| OUT = Path( | ||
| os.environ.get( | ||
| "HLX_BROAD_OUT", | ||
| "/home/morpheus/hlx/broad-eval-observed.json", | ||
| ) | ||
| ) | ||
| PRIV = Path(os.environ["HLX_BROAD_PRIV"]) if os.environ.get("HLX_BROAD_PRIV") else None | ||
|
|
||
| # Authorize-time pin (historical). Live soft_ceiling compare uses PRIOR eval. | ||
| PIN_EXACT = 0.889763779527559 | ||
| PIN_N = 254 | ||
|
|
||
|
|
||
| def _model_root(model: Path) -> Path: | ||
| if (model / "model.safetensors").is_file(): | ||
| return model | ||
| if (model / "best" / "model.safetensors").is_file(): | ||
| return model / "best" | ||
| return model | ||
|
|
||
|
|
||
| def main() -> int: | ||
| os.environ.pop("HYPERLEX_UNBIND_FORCE_TRAIN_PATH", None) | ||
| bundle = export_dataset(repo_root(), include_live=True) | ||
| rows = [r for r in bundle["rows"] if r.get("task") == "unbind"] | ||
| val = [r for r in rows if r.get("split") == "val"] | ||
| val_obs = [r for r in val if str(r.get("class") or "").upper() == "OBSERVED"] | ||
|
|
||
| device = torch.device("cuda" if torch.cuda.is_available() else "cpu") | ||
| root = _model_root(MODEL) | ||
| weight_path = root / "model.safetensors" | ||
| assert weight_path.is_file(), weight_path | ||
| maps_dir = MODEL if (MODEL / "atom_map.json").is_file() else root | ||
|
|
||
| tok = AutoTokenizer.from_pretrained(str(TRUNK), local_files_only=True) | ||
| enc = AutoModel.from_pretrained(str(TRUNK), local_files_only=True) | ||
| filler_state, encoder_tensors, heads_blob = _weight_parts(weight_path, torch) | ||
| apply_encoder_trainable(enc, encoder_tensors) | ||
| maps = _load_maps(maps_dir, heads_blob) | ||
| hidden = int(getattr(enc.config, "hidden_size", HIDDEN)) | ||
| filler_head = _load_filler_head(nn, hidden, filler_state, maps) | ||
| enc.to(device) | ||
| filler_head.to(device) | ||
|
|
||
| scored = score_unbind_exact(enc, filler_head, tok, maps, val_obs, device) | ||
| exact = float(scored.get("unbind_exact") or 0) | ||
| n = int(scored.get("n_unbind_eval") or scored.get("n") or 0) | ||
|
|
||
| out = { | ||
| "schema": "hyperlex.broad_observed_eval.v0.1", | ||
| "as_of": datetime.now(timezone.utc).isoformat(), | ||
| "gate": "soft_ceiling_tiebreak", | ||
| "model": str(MODEL), | ||
| "force": None, | ||
| "unbind_exact": exact, | ||
| "n_scored": n, | ||
| "scored": scored, | ||
| "authorize_pin": { | ||
| "seed": "seed-morph65", | ||
| "unbind_exact": PIN_EXACT, | ||
| "n": PIN_N, | ||
| "note": "Historical at gate authorize; live SoT may differ", | ||
| }, | ||
| "note": ( | ||
| "Broad OBSERVED val without apply_unbind_force_train. " | ||
| "soft_ceiling: compare candidate vs PRIOR on this same live surface." | ||
| ), | ||
| } | ||
| OUT.parent.mkdir(parents=True, exist_ok=True) | ||
| OUT.write_text(json.dumps(out, indent=2) + "\n") | ||
| if PRIV is not None: | ||
| PRIV.mkdir(parents=True, exist_ok=True) | ||
| (PRIV / OUT.name).write_text(OUT.read_text()) | ||
| print( | ||
| json.dumps( | ||
| {k: out[k] for k in ["unbind_exact", "n_scored", "model"]}, | ||
| indent=2, | ||
| ) | ||
| ) | ||
| return 0 | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| raise SystemExit(main()) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Even with the local checkpoints and dependencies installed, this audit cannot be reproduced from the committed evidence because
audit-clean-val.jsonis read from an absolute off-repo path, while neither that file nor code that derives its 63 rows is included anywhere in the repository. Consequently, reviewers cannot verify the row exclusions, the claimed 193/256 overlap, or the reported 1.0 tie that supports the central B1 conclusion; commit the evaluation rows or deterministically construct them from committed inputs.Useful? React with 👍 / 👎.