feat: affixiate Python source into character-first gonols - #38
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e46d9273c5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| raw_tokens: list[tuple[int, int, str, str, str]] = [] | ||
| zero_width: list[str] = [] | ||
| unresolved: list[str] = [] | ||
| stream = tokenize.generate_tokens(io.StringIO(source).readline) |
There was a problem hiding this comment.
When a valid Python file uses classic-Mac \r line endings, passing the unnormalized source directly to generate_tokens bypasses universal-newline handling. For example, CPython executes b"x=1\ry=2\r", but affixiate_python_bytes raises tokenizer position outside source: (1, 9) instead of producing a receipt. Normalize the tokenizer witness while mapping its positions back to the exact original source so valid \r files remain admissible.
Useful? React with 👍 / 👎.
| or root is not receipt.gonols[-1] | ||
| or root.span.start != 0 | ||
| or root.span.end != len(source) | ||
| or root.scale not in {"module", "source"} |
There was a problem hiding this comment.
When an existing invalid-source receipt is reserialized with standing="implemented-candidate" and a regenerated receipt digest, this check accepts its scale="source" / python.source.hmmm root. The verification CLI consequently reports the receipt as verified with success standing even though its graph records a grammar or tokenizer failure; replay should require implemented candidates to have module roots and hmmm receipts to have source-hmmm roots.
Useful? React with 👍 / 👎.
| if sys.version_info[:2] != (3, 12): | ||
| raise PythonGonolConstructionError( | ||
| f"{LANGUAGE_PROFILE} requires a Python 3.12 recognition witness; got {sys.version_info.major}.{sys.version_info.minor}" | ||
| ) |
There was a problem hiding this comment.
On any CPython 3.12 patch—or even PyPy reporting 3.12—this major/minor-only check passes, although BASE.json pins python/cpython v3.12.14 at a specific commit and the workspace describes CPython as the pinned witness. Different tokenizer/parser implementations can therefore emit receipts under the same constructor and language-profile identity; validate the implementation and exact patch, and pin the workflow accordingly.
AGENTS.md reference: AGENTS.md:L11-L12
Useful? React with 👍 / 👎.
| for child in ast.iter_child_nodes(node): | ||
| child_span = node_span(child) | ||
| if child_span is not None: |
There was a problem hiding this comment.
For a syntactically valid expression containing roughly 1,000 left-associative terms such as x+x+..., CPython compiles the source successfully, but this recursive node_span() call exceeds the default recursion limit and affixiate_python_source raises a raw RecursionError instead of producing a receipt. Such generated files can be only a few kilobytes, so traverse the AST iteratively or otherwise handle deep valid trees without aborting construction.
Useful? React with 👍 / 👎.
| tree = ast.parse( | ||
| source, | ||
| filename=source_id, | ||
| mode="exec", | ||
| type_comments=True, |
There was a problem hiding this comment.
ast.parse accepts trees that CPython subsequently rejects during compilation, so inputs such as module-level return 1, await x, or a misplaced from __future__ import are emitted with standing="implemented-candidate" and a module root instead of being retained as hmmm. These are not valid Python 3.12 file inputs; perform non-executing compiler validation before assigning successful standing.
AGENTS.md reference: AGENTS.md:L16-L19
Useful? React with 👍 / 👎.
1 participant
Summary
_recognitionwitness logichmmmletter-floor receipts on replayEvidence
hmmm