feat: Procrustes feature-alignment / drift checker (framework 11 / 13.3) - #85
Open
mdheller wants to merge 1 commit into
Open
feat: Procrustes feature-alignment / drift checker (framework 11 / 13.3)#85mdheller wants to merge 1 commit into
mdheller wants to merge 1 commit into
Conversation
Closes the gap where cognition-decision-trace.v1 (PR #82) declares the alignment_drift_exceeded approval trigger but nothing computes the drift. Adds scripts/check-alignment.py: given two feature spaces over a shared trunk output basis (e.g. a new model's SAE dictionary F_A vs the previous model's F_B), it solves the orthogonal Procrustes alignment Q* = argmin ||F_A - F_B Q||_F via the orthogonal polar factor of M = F_B^T F_A (equivalently U V^T of M = U S V^T), then reports the aligned Frobenius residual as the drift metric. Both matrices are normalized to unit Frobenius norm first so the residual is scale-free. Because drift is judged after alignment, a rotated-but-equivalent space registers as aligned. The linear algebra is pure stdlib (Jacobi eigendecomposition of the small symmetric Gram matrix) - no third-party dependency added. Emits a feature-alignment-drift.v1 record. This is distinct from the existing alignment-check.v1 (a governance-invariant structural check, no room for Procrustes/residual data), so it is a dedicated schema. Classification per framework 13.3: stable if residual < 0.05 and no critical feature drifted; drifted if residual < 0.20 or <= 1 critical feature drifted; broken otherwise. decision = aligned iff stable, else drift; the schema binds the two so a drift record cannot claim alignment. Each record carries a SHA-256 (FIPS 180-4) replay_seal over the canonical inputs and verdict. Teeth both ways (make feature-alignment-ci, folded into make validate and make lawful-learning-ci; schema teeth in make lawful-learning-schema-ci): - near-identical space -> stable/aligned (residual 0.0052); - rotated-but-equivalent space (40 deg rotation) -> aligned after Procrustes (residual 0.0), proving the alignment works, not raw distance; - genuinely drifted space -> broken/drift, promotion blocked (residual 0.274, 3 critical features drifted); - malformed inputs (shape mismatch, non-finite) -> rejected; - schema negatives: decision/classification mismatch, weak replay seal. Runtime threshold calibration against live gemma-2-9b-it SAE activations and approval-gate wiring are deferred follow-ups; fixtures are synthetic. Refs #83, #82, #76.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #83. Refs #82 (cognition-decision-trace, merged), #76.
The gap
cognition-decision-trace.v1(PR #82) declares thealignment_drift_exceededapproval trigger, but nothing computed the drift. This adds the checker behind it — the §13.3 feature-geometry alignment/drift gate that complements the §13.2 decision trace.What ships
scripts/check-alignment.py— solves the orthogonal Procrustes alignmentQ* = argmin ||F_A − F_B Q||_Fas the orthogonal polar factor ofM = F_B^T F_A(equivalentlyU VᵀofM = U Σ Vᵀ), then reports the aligned Frobenius residual as the drift metric. Both matrices are normalized to unit Frobenius norm first (scale-free residual). Drift is judged after alignment, so a rotated-but-equivalent space registers as aligned. Linear algebra is pure stdlib (Jacobi eigendecomposition of the small symmetric Gram matrix) — no third-party dependency added (there is no numpy/scipy in this repo).schemas/lawful-learning/feature-alignment-drift.v1.json— the emitted record. This is a new, dedicated schema: the existingalignment-check.v1is a governance-invariant structural check with no room for Procrustes/residual data. Schema teeth binddecisiontoclassification(onlystable→aligned) and require a SHA-256 (FIPS 180-4)replay_seal.stableifresidual < 0.05and no critical feature drifted;driftedifresidual < 0.20or ≤1 critical feature drifted;brokenotherwise. Checker exits non-zero once classification exceeds--max-class(defaultdrifted), blocking promotion onbroken.make feature-alignment-ci(folded intomake validateandmake lawful-learning-ci) plus schema teeth inmake lawful-learning-schema-ci.Teeth (verified locally, all green)
stable/ aligned (residual 0.0052)broken/ drift, promotion blocked (residual 0.274, 3 critical drifted)make feature-alignment-ci,make lawful-learning-schema-ci,make lawful-learning-ciall exit 0.Deferred (follow-up @mdheller)
Runtime calibration of the 0.05 / 0.20 thresholds against live
gemma-2-9b-itSAE activations, and wiring the emitted record into the cognition loop's approval gate. Fixtures here are synthetic.