Skip to content
This repository was archived by the owner on Oct 10, 2026. It is now read-only.

docs(007): publish audit — keep seed-morph78 weights local-only - #107

Merged
scrimshawlife-ctrl merged 3 commits into
mainfrom
cursor/hyperlexical-publish-audit-41af
Sep 24, 2026
Merged

scrimshawlife-ctrl merged 3 commits into
mainfrom
cursor/hyperlexical-publish-audit-41af

Conversation

@scrimshawlife-ctrl

Copy link
Copy Markdown
Owner

Recommendation

Do not publish the seed-morph78 weights yet. Full audit: specs/007-hyperlexical-model/PUBLISH-AUDIT-20260924.md.

Blockers (OBSERVED, evidence in receipts/publish-audit-20260924/)

  1. Promotion metric is train-contaminated. 193 of the 256 soft_ceiling broad-val rows are in force_train_morph78_expanded.jsonl. On the 63 leak-free val rows, morph78 and morph65 both score 1.0, so the whole 0.9883 vs 0.8867 margin comes from trained-on rows. The 440-row test split has never been scored.
  2. Not reproducible from main. Training ran from Spark ~/Hyperlex at 78c4d83 (39 behind main) with a 799-line uncommitted diff in export.py / loop.py / unbind_recipe.py, plus 86 settle/finish/eval scripts in ~/hlx outside git. The train receipt has no code commit.

Other gaps: data rights for crawled slang + 14 CC BY-SA rows; license conflict (MIT / Proprietary / apache-2.0 card); classify overconfidence; A1 150M reading; no Hub loader; card has no test metrics.

The audit lists a 7-step fix order, ending with a private/gated Hub repo first.

Not changed

name_gate, BEST pin, receipts, code. Read-only eval on Spark (no training). Evidence copies of off-git scripts are included for review.

Validation

pytest 747 passed; mkdocs strict PASS; evidence scanned for secrets (none).

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9ac1437d1d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

from hyperlexical.eval_forward import _weight_parts, _load_maps, _load_filler_head, apply_encoder_trainable, score_unbind_exact
from torch import nn
TRUNK = Path("/home/morpheus/.hyperlex/models/trunks/ModernBERT-base")
rows = json.loads(Path("/home/morpheus/hlx/audit-clean-val.json").read_text())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include the leak-free evaluation input

Even with the local checkpoints and dependencies installed, this audit cannot be reproduced from the committed evidence because audit-clean-val.json is read from an absolute off-repo path, while neither that file nor code that derives its 63 rows is included anywhere in the repository. Consequently, reviewers cannot verify the row exclusions, the claimed 193/256 overlap, or the reported 1.0 tie that supports the central B1 conclusion; commit the evaluation rows or deterministically construct them from committed inputs.

Useful? React with 👍 / 👎.

@scrimshawlife-ctrl
scrimshawlife-ctrl merged commit 15ba1ef into main Sep 24, 2026
5 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant