Add min angle graph 2#848
Open
klei22 wants to merge 5 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds an optional validation-time export for the LM-head “minimum-angle” nearest-neighbor graph, including a CSV/JSON export format, training-time integration via new CLI flags, and a local Plotly-based HTML viewer plus smoke-test/demo helpers.
Changes:
- Add a blockwise (non-materializing) min-angle graph computation + CSV/JSON exporter utility.
- Wire the exporter into
train.pyto run after validation when enabled via newtrain_args.pyflags. - Document usage and add an HTML viewer, an exploration config, and a demo script for running/inspecting exports.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| utils/min_angle_graph_export.py | Implements blockwise cosine/angle nearest-neighbor computation and writes CSV + JSON metadata. |
| train.py | Adds export hook after validation and a small stdlib-preload guard against module shadowing. |
| train_args.py | Introduces CLI flags controlling export directory, cadence, block size, device, and label. |
| README.md | Documents how to enable exports and how to use the Plotly viewer. |
| explorations/min_angle_graph_export.yaml | Adds a smoke-test experiment config for exercising exports on small runs. |
| demos/min_angle_graph_export_demo.sh | Provides a runnable demo script to generate exports and point to the viewer. |
| analysis/min_angle_graph_plotly_viewer.html | Adds a local HTML viewer to step through/export snapshots and visualize distributions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+24
to
+26
| def compute_min_angle_graph(weight, block_size=2048, compute_device="auto"): | ||
| """Compute each row vector's closest non-self row by signed angular distance.""" | ||
| compute_device = resolve_min_angle_graph_device(weight, compute_device) |
Comment on lines
+156
to
+159
| "compute_device": graph["compute_device"], | ||
| "has_token_text_escaped": escaped_token_texts is not None, | ||
| "angle_definition": "signed 0-180 degrees, closest non-self token by maximum cosine", | ||
| } |
Comment on lines
+87
to
+89
| training_group.add_argument('--export_min_angle_graph_each_eval', default=False, | ||
| action=argparse.BooleanOptionalAction, | ||
| help='Export the LM-head minimum-angle graph after every validation loss.') |
Comment on lines
+266
to
+268
| row/column blocks through the selected compute device, excludes each token's | ||
| self-distance, and records the closest non-self token by signed `0°–180°` | ||
| angular distance. The full `vocab_size × vocab_size` angle matrix is never |
Comment on lines
+1957
to
+1962
| if self.args.export_min_angle_graph_each_eval: | ||
| if live: | ||
| live.stop() | ||
| self.export_min_angle_graph(losses) | ||
| if live: | ||
| live.start() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.