From 359870ad68ab2b930534abe8200960255157e80f Mon Sep 17 00:00:00 2001 From: Erin Spencer Date: Sat, 12 Sep 2026 16:35:47 -0700 Subject: [PATCH 01/10] feat: affixiate Python source into gonols --- .github/workflows/python-gonol.yml | 39 + AGENTS.md | 6 + README.md | 12 + STACK_MANIFEST.md | 9 +- research/python-gonol/BASE.json | 26 + research/python-gonol/README.md | 82 ++ research/python-gonol/WORK_GRAPH.json | 49 + .../docs/PYTHON_AFFIXIATION_BOUNDARY.md | 163 +++ .../python-gonol/python_gonol/__init__.py | 64 + .../python-gonol/python_gonol/__main__.py | 102 ++ .../python-gonol/python_gonol/affixiation.py | 1120 +++++++++++++++++ research/python-gonol/python_gonol/model.py | 277 ++++ .../python-gonol/tests/test_affixiation.py | 213 ++++ .../tests/test_python312_surface.py | 175 +++ stack-manifest.json | 12 +- 15 files changed, 2347 insertions(+), 2 deletions(-) create mode 100644 .github/workflows/python-gonol.yml create mode 100644 research/python-gonol/BASE.json create mode 100644 research/python-gonol/README.md create mode 100644 research/python-gonol/WORK_GRAPH.json create mode 100644 research/python-gonol/docs/PYTHON_AFFIXIATION_BOUNDARY.md create mode 100644 research/python-gonol/python_gonol/__init__.py create mode 100644 research/python-gonol/python_gonol/__main__.py create mode 100644 research/python-gonol/python_gonol/affixiation.py create mode 100644 research/python-gonol/python_gonol/model.py create mode 100644 research/python-gonol/tests/test_affixiation.py create mode 100644 research/python-gonol/tests/test_python312_surface.py diff --git a/.github/workflows/python-gonol.yml b/.github/workflows/python-gonol.yml new file mode 100644 index 0000000..95e52f9 --- /dev/null +++ b/.github/workflows/python-gonol.yml @@ -0,0 +1,39 @@ +name: python-gonol + +on: + pull_request: + paths: + - "research/python-gonol/**" + - "libs/ucns/**" + - "README.md" + - "AGENTS.md" + - "STACK_MANIFEST.md" + - "stack-manifest.json" + - ".github/workflows/python-gonol.yml" + push: + branches: [main] + paths: + - "research/python-gonol/**" + - "libs/ucns/**" + - "README.md" + - "AGENTS.md" + - "STACK_MANIFEST.md" + - "stack-manifest.json" + - ".github/workflows/python-gonol.yml" + +permissions: + contents: read + +jobs: + contracts: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + - uses: actions/setup-python@v6 + with: + python-version: "3.12" + - name: Install pytest + run: python -m pip install pytest + - name: Python Gonol construction tests + working-directory: research/python-gonol + run: python -m pytest -q tests diff --git a/AGENTS.md b/AGENTS.md index 3108062..7819333 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,6 +13,10 @@ projects may later graduate into their own repositories. - `research/english-gonol/` is a distinct stack-local English lexical/gonol construction component. UCNS owns consumed geometry; EDCM may evaluate outputs but does not define the English Gonol construction. +- `research/python-gonol/` is a distinct stack-local Python source/gonol construction + component. It owns Python source admission and bottom-up affixiation only; METAPAT owns + affixiation semantics, UCNS owns consumed geometry, and parser objects are witnesses + rather than gonol identities. - root-level emerging projects such as `ahbg/` may be close to external repo-hood; root placement does not transfer authority from their inputs. - `STACK_MANIFEST.md` and `stack-manifest.json` own stack-level participant provenance. @@ -105,6 +109,8 @@ coherence; it does not replace workspace behavioral tests. - Organization aggregate and website-projection derivation specs are not yet registered. - English Gonol Construction has distinct stack-local authority but has not yet gained an independent repository/release authority boundary. +- Python Gonol Construction has distinct stack-local authority but has not yet gained an + independent repository/release authority boundary; exact UCNS affixiation geometry is unresolved. - The complete root `skill-lib/` snapshot refresh remains separate because the current provenance-bound fresh-making doctrine is newer than the local generator snapshot. - Project graduation automation remains unimplemented. diff --git a/README.md b/README.md index f9f525f..a887beb 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,7 @@ stack/ │ ├── metapat/ # current METAPAT research + BASE.json │ ├── ucns/ # current UCNS research + BASE.json │ ├── english-gonol/ # English lexical/gonol construction; distinct from EDCM +│ ├── python-gonol/ # Python 3.12 source affixiation from letters upward │ ├── edcm/ # current EDCM measurement research + BASE.json │ ├── pcea/ # current PCEA research + BASE.json │ ├── ptcna/ # current PTCNA research + BASE.json @@ -77,6 +78,13 @@ English Gonol Construction is a separated stack-local component at English text-domain construction candidate. EDCM may evaluate those outputs but does not define the construction. +Python Gonol Construction is a separate stack-local component at +`research/python-gonol/`. It admits every exact Python source occurrence as a letter +gonol, then affixiates lexical forms, delimiters, grammar constructions, and the module. +METAPAT owns affixiation semantics; UCNS owns optional consumed geometry; Python Gonol +owns Python source construction. Tokens and AST nodes are recognition witnesses, never +gonol substitutes. + ### Change stack structure Any change that alters a participant, pin, authority, relation, research workspace, @@ -103,6 +111,8 @@ stack-local implementation. English Gonol Construction is currently a distinct stack-local research component, separated from EDCM but not independently graduated. +Python Gonol Construction is likewise stack-local and ungraduated; its Python 3.12 +constructor is an implemented candidate, not stack or language canon. EPAC and psychsocio metafauna are currently in this pre-graduation state. EPAC is further along: it has an independent extracted repository, but extraction is not graduation, so its forge research remains here until EPAC completes its release, @@ -177,6 +187,8 @@ merge. is not yet implemented. - English Gonol Construction has distinct stack-local authority but has not yet gained an independent repository/release authority boundary. +- Python Gonol Construction has distinct stack-local authority but has not yet gained an + independent repository/release authority boundary; UCNS affixiation geometry remains unresolved. - Actual VM PostgreSQL/service-account/storage state and the independent backup device remain deployment observations until inspected on the VM. - A GitHub-hosted executor remains optional and unimplemented; VM-local execution is the diff --git a/STACK_MANIFEST.md b/STACK_MANIFEST.md index 80b03e8..0401dd6 100644 --- a/STACK_MANIFEST.md +++ b/STACK_MANIFEST.md @@ -7,9 +7,10 @@ Provenance and authority-boundary record for `The-Interdependency/stack`. - PCEA canonical refresh UTC: `2026-08-31T07:49:28Z` at `91ffa8c7249dfb810ca64a0bbc500481c0bd12a9` - EPAC extraction reconciliation UTC: `2026-09-05` at `d8868858b2e455381ce670797bdbe47189bdc496` - English Gonol separation reconciliation UTC: `2026-09-12` at `030022948fb7c749961ae65743a4448c4bb6cbbe` +- Python Gonol construction baseline UTC: `2026-09-12` at `0e8384bbb60e4c2189016a212bdd0030d04aed7d` - Stack-manifest schema: `the-interdependency.stack-manifest` version `1.1.0` - Work-graph digest (SHA-256 over canonical `repositories` + `research_participants` + `boundaries` JSON): - `9ab3b3f75a32f5f73b5df68419148181fc632593babe4ec6adf4269d4f35badb` + `482e9a4f70c18ad4888d1fab32e44a6ab550fe7486fabd2f1253876c266212eb` - Machine-readable copy: [`stack-manifest.json`](stack-manifest.json) ## Directory contract @@ -52,6 +53,7 @@ release identity. | Workspace | Participant | Exact commit | Relation | Canonical release | |---|---|---|---|---| | `research/english-gonol/` | `The-Interdependency/stack` | `030022948fb7c749961ae65743a4448c4bb6cbbe` | stack-local English lexical/gonol construction separated from EDCM; consumes UCNS geometry; EDCM may evaluate outputs but does not define construction | no | +| `research/python-gonol/` | `The-Interdependency/stack` | `0e8384bbb60e4c2189016a212bdd0030d04aed7d` | stack-local bottom-up Python 3.12 source gonol construction; applies METAPAT affixiation semantics, consumes optional UCNS geometry, and transfers no language authority to UCNS or EDCM | no | | `research/ucns/` | `The-Interdependency/ucns` | `1975fe70cf4e0826a8020c2da3047569e277af64` | explicit source base for integrated stack-local UCNS research; does not refresh or replace the manifest-pinned `libs/ucns` canonical view | no | | `research/from-photons-to-macroverse/` | `The-Interdependency/stack` | `77ef8c7fb0ff75a524181655ee9f9641372768f7` | target composition forge baseline at audit start | no | | `research/from-photons-to-macroverse/` | `The-Interdependency/skill-lib` | `61eb3b14db440e6ee9b7bf8de3b646dbfd00fb32` | audit, domain-claim, work-graph, and hmmm doctrine | no | @@ -139,9 +141,14 @@ immutable release, downstream stack reconsumption, and authority-transition rece English Gonol Construction is earlier in that lifecycle: it is a distinct stack-local research component, not EDCM and not an independent canonical release. +Python Gonol Construction is also stack-local research. Its Python 3.12 source +constructor has no independent repository/release authority, and its successful replay +transfers no semantic, geometric, measurement, or language-canon status. + ## hmmm - UCNS has no `LICENSE` file at pinned commit `828c0b8`. - EPAC clean install, license, stable release, downstream reconsumption, and authority-transition receipt remain incomplete; `libs/epac/` stays unpopulated until graduation. - English Gonol Construction remains stack-local research; independent repository/release authority has not been established. +- Python Gonol Construction remains stack-local research; independent repository/release authority and exact UCNS affixiation geometry remain unresolved. - `skill-lib/` remains a special operational snapshot at stack root rather than following the ordinary `libs/` + `research/` pair. diff --git a/research/python-gonol/BASE.json b/research/python-gonol/BASE.json new file mode 100644 index 0000000..c3fc132 --- /dev/null +++ b/research/python-gonol/BASE.json @@ -0,0 +1,26 @@ +{ + "schema": "the-interdependency.stack-research-base", + "version": "1.0.0", + "project": "python-gonol", + "source_repository": "The-Interdependency/stack", + "source_commit": "0e8384bbb60e4c2189016a212bdd0030d04aed7d", + "metapat_prerequisite": { + "repository": "The-Interdependency/metapat", + "commit": "510e0171f4ecc6d1889e66bb66a8734c49c3b1fa", + "relation": "METAPAT owns affixiation semantics; Python Gonol applies them without root impact" + }, + "ucns_prerequisite": { + "repository": "The-Interdependency/ucns", + "commit": "31bb761e49307b04a7c7dd7c7c2059ea35fdc089", + "relation": "UCNS owns geometry and Public Gonol positions; Python Gonol owns no UCNS operation" + }, + "python_language_profile": { + "repository": "python/cpython", + "tag": "v3.12.14", + "commit": "2d7d957248038635334b62b27f650220213d8e5d", + "profile": "Python 3.12 file input" + }, + "canon_path": null, + "authority": "stack-local Python source gonol construction", + "standing": "stack-local-research" +} diff --git a/research/python-gonol/README.md b/research/python-gonol/README.md new file mode 100644 index 0000000..42714a0 --- /dev/null +++ b/research/python-gonol/README.md @@ -0,0 +1,82 @@ +# Python Gonol Construction + +Stack-local research implementation for affixiating Python source into gonols +from its lowest admitted units upward. + +## Construction + +```text +exact decoded source occurrences + -> letter gonols + -> Python lexical-form gonols + -> matched delimiter gonols + -> recursive Python grammar-construction gonols + -> module gonol +``` + +Every source occurrence remains independently addressable. Every larger gonol +contains an identity-bearing relation whose ordered, role-bearing members refer +only to already-closed gonols. A parent consumes a child's atomic identity; the +receipt registry keeps the complete child recoverable. + +`letter` is the name of the source-floor construction, not a Unicode alphabetic +classification. Python program text is read as Unicode code points, so spaces, +newlines, digits, operators, and delimiters each begin as their own letter gonol +occurrence too. Nothing is normalized, deduplicated, trimmed, or silently +discarded. + +## Authority boundary + +```text +METAPAT affixiation semantics + -> UCNS geometry and Public Gonol positions + -> Python Gonol Python source admission and construction + -> later consumers no authority transfer +``` + +This workspace uses CPython 3.12 `tokenize` and `ast` only as recognition +witnesses after the letter floor has closed. Token, AST, compiler, and code +objects never become gonols and never replace the source-built relation graph. +The source bytes, every decoded occurrence, every closed construction, and all +constitutive relations remain visible in the receipt. + +Standing: **implemented stack-local candidate; not canon and not an independent +release**. + +## Usage guidance + +Run from this directory with Python 3.12: + +```bash +python -m python_gonol path/to/source.py --out source.gonol.json --pretty +python -m python_gonol --verify source.gonol.json +python -m pytest -q tests +``` + +Use the bytes entrypoint for files so an encoding declaration and the exact +original bytes remain bound: + +```python +from pathlib import Path +from python_gonol import affixiate_python_bytes, replay_python_affixiation + +path = Path("example.py") +receipt = affixiate_python_bytes(path.read_bytes(), source_id=path.as_posix()) +replay_python_affixiation(receipt) +``` + +For invalid or unfinished source, the constructor retains all admitted lower +closures, closes a `python.source.hmmm` root, records the exact tokenizer, +delimiter, or grammar boundary, and the CLI exits `2`. This makes a missing end +parenthesis visible without throwing away the construction completed beneath it. + +See [`docs/PYTHON_AFFIXIATION_BOUNDARY.md`](docs/PYTHON_AFFIXIATION_BOUNDARY.md) +for the full contract and [`WORK_GRAPH.json`](WORK_GRAPH.json) for exact inputs. + +## hmmm + +- exact UCNS geometric operation of Public Gonol function positions; +- exact UCNS Möbius-carrier affixiation/coupling law; +- Python language profiles after Python 3.12 file input; +- streaming/checkpointed receipt materialization for unusually large source + trees. diff --git a/research/python-gonol/WORK_GRAPH.json b/research/python-gonol/WORK_GRAPH.json new file mode 100644 index 0000000..561f30d --- /dev/null +++ b/research/python-gonol/WORK_GRAPH.json @@ -0,0 +1,49 @@ +{ + "schema": "the-interdependency.stack-manifest", + "version": "1.0.0", + "work_graph_sha256": "f677e63e96d1d27db543391a0999ed5f89ad1e3cb82656771137f3022cef4660", + "repositories": [ + { + "repository": "The-Interdependency/stack", + "commit": "0e8384bbb60e4c2189016a212bdd0030d04aed7d", + "authority": "stack-local Python Gonol implementation owner", + "relation": "construction workspace and starting identity" + }, + { + "repository": "The-Interdependency/metapat", + "commit": "510e0171f4ecc6d1889e66bb66a8734c49c3b1fa", + "authority": "affixiation semantic authority", + "relation": "defines identity-preserving recursive affixiation meaning" + }, + { + "repository": "The-Interdependency/ucns", + "commit": "31bb761e49307b04a7c7dd7c7c2059ea35fdc089", + "authority": "geometry and Public Gonol position authority", + "relation": "optional explicit geometry observation; no semantic or language authority transfer" + }, + { + "repository": "The-Interdependency/skill-lib", + "commit": "c3fe833cdd3a35c5dc12b0678e5489c9b6a18741", + "authority": "organization-wide construction and evidence discipline", + "relation": "gonol-build, work-graph, module, test, and hmmm discipline" + }, + { + "repository": "python/cpython", + "commit": "2d7d957248038635334b62b27f650220213d8e5d", + "authority": "Python 3.12.14 language and CPython recognition witness", + "relation": "exact Python language profile source at tag v3.12.14" + } + ], + "boundaries": { + "authority_transfer": false, + "proof_status_transfer": false, + "measurement_status_transfer": false, + "semantic_mapping": "external-provenance", + "agent_scope": "cross-repository-work-graph", + "hmmm": [ + "exact UCNS Public Gonol function operations remain unresolved", + "exact UCNS Mobius-carrier affixiation/coupling law remains unresolved", + "Python language profiles after 3.12 remain unresolved" + ] + } +} diff --git a/research/python-gonol/docs/PYTHON_AFFIXIATION_BOUNDARY.md b/research/python-gonol/docs/PYTHON_AFFIXIATION_BOUNDARY.md new file mode 100644 index 0000000..eab7fe8 --- /dev/null +++ b/research/python-gonol/docs/PYTHON_AFFIXIATION_BOUNDARY.md @@ -0,0 +1,163 @@ +# Python source affixiation boundary + +**Status:** implemented stack-local Python 3.12 file-input candidate. +**Scope:** determine and implement how Python source affixiates into gonols from +the lowest admitted source units upward. +**Root impact:** none. + +## Governing relation + +Python Gonol Construction applies METAPAT's affixiation meaning: already-bounded +participants remain individually addressable, with identity and provenance +preserved, while their declared relation may close as a higher-scale +object-whole. A closed whole may participate recursively without erasing its +constituents. + +UCNS owns any exact geometric realization. Python Gonol Construction owns the +Python source profile, admission, lexical relations, delimiter closure, grammar +relations, and receipts. Tokens, AST nodes, compiler objects, and metadata own +none of the construction. + +## Source floor: letters + +Python 3.12 reads program text as Unicode code points. This construction admits +each decoded Unicode scalar occurrence in exact source order and closes it as one +letter gonol. + +`letter` here means the irreducible admitted source occurrence. It includes an +alphabetic character, digit, space, tab, newline, quote, operator, delimiter, or +other exact source scalar without introducing UCNS letter/punctuation subclasses. +The original file bytes, detected Python encoding, decoded-source digest, +occurrence index, line/column span, scalar value, source identity, and optional +observed Public Gonol position remain receipt-bound. + +Repeated equal values have distinct occurrence addresses. No normalization, +identifier NFKC replacement, case folding, trimming, gap deletion, or +deduplication changes the admitted source. + +## Lexical affixiation + +After every letter closes, Python's lexical witness identifies nonempty lexical +forms. Each lexical form closes solely from the exact ordered letter gonols in +its source span. Source gaps that Python uses to separate forms also close as +`python.lexical.INTERTOKEN`; this preserves whitespace the tokenizer does not +emit as a token. + +The lexical relation, exact form class, source length, role order, and CPython +3.12 witness provenance live inside the lexical gonol. The runtime `TokenInfo` +does not. + +Zero-width `INDENT`/`DEDENT` witness events have no source occurrence from which +to form a separate gonol. They therefore remain identity-bearing properties of +the consuming module/source relation. The actual indentation characters remain +letter and lexical gonols. + +## Larger construction affixiation + +Matched `()`, `[]`, and `{}` relations close from lexical gonols and already +closed nested delimiter gonols. Opener, content order, nested multiplicity, and +closer belong inside the delimiter gonol. Missing or mismatched closure becomes +`hmmm`. + +After those closures, the CPython 3.12 grammar witness supplies accepted grammar +relations and child roles. A grammar construction closes over: + +1. already-closed child grammar gonols; +2. already-closed delimiter gonols intersecting its concrete source relation; +3. remaining already-closed lexical forms in its exact source span; and +4. identity-bearing relation properties such as operator or grammar-field roles. + +Construction proceeds postorder. Thus every member identity in a parent receipt +was closed earlier. Overlapping relations—such as a call's argument role and +the parentheses enclosing its arguments—may both reference the same canonical +source occurrences; they do not duplicate those occurrences. + +Some CPython witness records have no independent source span, including an +empty `arguments` record and `TypeIgnore`. Such a record cannot honestly close +as a separate gonol. Its relation is retained as an intrinsic property of the +source-built parent that contains the exact delimiter or comment gonols. The +witness object is then discarded. Decorated function and class spans are +extended to the exact leading `@` lexical gonol so the decorator relation does +not lose its constitutive marker. + +The module gonol closes last and covers every source occurrence, including +comments, blank space, redundant parentheses, and other concrete source material +that abstract grammar nodes omit. + +## Recognition is not substitution + +CPython's tokenizer and AST are bounded recognition witnesses for the Python +3.12 language profile. They determine where accepted lexical and grammar +relations apply. They are discarded after each relation is translated into +source-occurrence membership, closed-gonol references, order, roles, and +provenance. + +The following never serve as a gonol identity or participant: + +- token numbers or `TokenInfo` instances; +- AST instances or AST dumps; +- compiled code objects; +- evaluated literal values; +- symbol-table/compiler objects; +- whole-source hashes standing in for visible construction. + +SHA-256 identifies a complete visible gonol or receipt payload. The visible +payload—not its hash—is the construction. + +## Failure and replay + +Tokenizer, delimiter, or grammar failure does not erase completed work. The +constructor closes a `python.source.hmmm` root over every lower construction it +can honestly retain, records the exact unresolved finding, and sets receipt +standing to `hmmm`. + +Replay checks: + +- exact original bytes and encoding; +- exact source reconstruction from ordered letter gonols; +- one contiguous letter floor with distinct occurrence addresses; +- every member references an earlier closed gonol by exact address and identity; +- every larger gonol's exact span equals the union of its atomic closed-child + spans, without flattening descendant leaves back into the parent; +- every gonol identity and the final receipt digest recompute; and +- the root covers every source occurrence. + +Replay proves deterministic construction integrity only. It does not execute +the source or prove behavior, equivalence, safety, semantic quality, geometry, +measurement validity, or canon. + +## Python profile + +The implemented profile is CPython 3.12 file input. It covers any exact source +accepted by `ast.parse(..., mode="exec", type_comments=True, +feature_version=(3, 12))`, including Python 3.12 type statements/type parameters, +pattern matching, exception groups, asynchronous constructs, comprehensions, +formatted strings, comments within formatted replacement fields, Unicode +identifiers, explicit/implicit line joining, and all ordinary expression and +statement forms. + +The committed broad-surface fixture exercises these relations but does not claim +that one finite fixture enumerates all possible programs. Generic traversal, +rather than a hand-selected statement dispatcher, is what keeps the constructor +open over the complete accepted 3.12 grammar. + +## Usage guidance + +```bash +cd research/python-gonol +python -m pytest -q tests +python -m python_gonol ../../some-file.py --out /tmp/some-file.gonol.json +python -m python_gonol --verify /tmp/some-file.gonol.json +``` + +Prefer `affixiate_python_bytes()` for files. Use +`affixiate_python_source()` only when exact original file bytes are unavailable +or irrelevant to the declared source profile. + +## hmmm + +- exact UCNS function operation for each Public Gonol position; +- exact UCNS Möbius-carrier affixiation/coupling geometry; +- whether and how Python 3.13+ language profiles share or revise this constructor; +- streaming or checkpointed materialization for source large enough that a full + in-memory visible receipt would exceed available resources. diff --git a/research/python-gonol/python_gonol/__init__.py b/research/python-gonol/python_gonol/__init__.py new file mode 100644 index 0000000..2bc7b30 --- /dev/null +++ b/research/python-gonol/python_gonol/__init__.py @@ -0,0 +1,64 @@ +"""Public surface for stack-local Python Gonol Construction. + +Usage guidance: call ``affixiate_python_bytes`` for a file or +``affixiate_python_source`` for already-decoded text, then persist or compare +the returned receipt. Call ``replay_python_affixiation`` before consuming a +receipt across a boundary. +""" + +# === MODULE_BUILD === +# id: python_gonol_public_surface +# module_name: python_gonol +# module_kind: adapter +# summary: exposes the bounded Python 3.12 bottom-up affixiation constructor and immutable receipt types +# owner: Python Gonol Construction (stack-local research) +# public_surface: affixiate_python_source, affixiate_python_bytes, replay_python_affixiation, reconstruct_source, PythonAffixiationReceipt +# internal_surface: none +# auth_boundary: none +# storage_boundary: none +# network_boundary: none +# user_data_boundary: caller-owned source only +# admin_only: false +# tests: tests.test_affixiation, tests.test_python312_surface +# rollout: explicit import only +# rollback: remove package exports with the workspace +# requires: python_gonol_affixiation, python_gonol_model +# since: 2026-09-12 +# unresolved: independent package and release authority remain hmmm +# === END MODULE_BUILD === + +from .affixiation import ( + CONSTRUCTOR_ID, + CONSTRUCTOR_VERSION, + LANGUAGE_PROFILE, + PINNED_PUBLIC_GONOL_SHA256, + PythonGonolConstructionError, + affixiate_python_bytes, + affixiate_python_source, + reconstruct_source, + replay_python_affixiation, +) +from .model import ( + AffixiationRelation, + ClosedGonol, + PythonAffixiationReceipt, + RelationMember, + SourceSpan, +) + +__all__ = [ + "AffixiationRelation", + "CONSTRUCTOR_ID", + "CONSTRUCTOR_VERSION", + "ClosedGonol", + "LANGUAGE_PROFILE", + "PINNED_PUBLIC_GONOL_SHA256", + "PythonAffixiationReceipt", + "PythonGonolConstructionError", + "RelationMember", + "SourceSpan", + "affixiate_python_bytes", + "affixiate_python_source", + "reconstruct_source", + "replay_python_affixiation", +] diff --git a/research/python-gonol/python_gonol/__main__.py b/research/python-gonol/python_gonol/__main__.py new file mode 100644 index 0000000..ec40bbb --- /dev/null +++ b/research/python-gonol/python_gonol/__main__.py @@ -0,0 +1,102 @@ +"""Command line entry point for Python Gonol Construction. + +Usage guidance:: + + python -m python_gonol source.py --out source.gonol.json + python -m python_gonol --verify source.gonol.json + +The construction command exits 2 when source syntax remains ``hmmm``. The +receipt is still written so its admitted letter and lexical closures survive. +""" + +# === MODULE_BUILD === +# id: python_gonol_cli +# module_name: python_gonol.__main__ +# module_kind: adapter +# summary: provides file-to-receipt construction and receipt verification commands +# owner: Python Gonol Construction (stack-local research) +# public_surface: python -m python_gonol +# internal_surface: main +# auth_boundary: none +# storage_boundary: reads source or receipt files and writes an explicitly named receipt file or stdout +# network_boundary: none +# user_data_boundary: read and write at caller-selected paths +# admin_only: false +# tests: tests.test_affixiation +# rollout: explicit command only +# rollback: remove the CLI while retaining the importable constructor +# requires: python_gonol_affixiation, python_gonol_model +# since: 2026-09-12 +# unresolved: streaming receipts for very large sources remain hmmm +# === END MODULE_BUILD === + +# === BOUNDARIES === +# id: python_gonol_cli_file_boundary +# summary: reads one caller-selected local source or receipt and writes only the explicit output path +# auth_boundary: none +# storage_boundary: write +# network_boundary: none +# user_data_boundary: read +# admin_only: false +# pii: possible +# secrets: read +# owner: caller +# === END BOUNDARIES === + +from __future__ import annotations + +import argparse +from pathlib import Path +import sys + +from .affixiation import ( + affixiate_python_bytes, + reconstruct_source, + replay_python_affixiation, +) +from .model import PythonAffixiationReceipt + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description="Affixiate Python 3.12 source into gonols from letters upward." + ) + parser.add_argument("source", nargs="?", help="Python source file") + parser.add_argument("--out", help="receipt path; omit for stdout") + parser.add_argument("--verify", metavar="RECEIPT", help="verify an existing receipt") + parser.add_argument("--pretty", action="store_true", help="pretty-print JSON") + return parser + + +def main(argv: list[str] | None = None) -> int: + args = _parser().parse_args(argv) + if args.verify: + if args.source or args.out: + raise SystemExit("--verify does not accept source or --out") + receipt = PythonAffixiationReceipt.from_json( + Path(args.verify).read_text(encoding="utf-8") + ) + replay_python_affixiation(receipt) + print( + f"verified {receipt.receipt_digest} source={receipt.source_id} standing={receipt.standing}" + ) + return 0 + if not args.source: + raise SystemExit("source is required unless --verify is used") + path = Path(args.source) + receipt = affixiate_python_bytes(path.read_bytes(), source_id=path.as_posix()) + rendered = receipt.to_json(pretty=args.pretty) + if args.out: + Path(args.out).write_text(rendered, encoding="utf-8") + else: + sys.stdout.write(rendered) + if receipt.standing == "hmmm": + for item in receipt.hmmm: + if item.startswith(("tokenizer:", "grammar:", "unmatched", "delimiter mismatch")): + print(f"hmmm: {item}", file=sys.stderr) + return 2 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/research/python-gonol/python_gonol/affixiation.py b/research/python-gonol/python_gonol/affixiation.py new file mode 100644 index 0000000..d812fe1 --- /dev/null +++ b/research/python-gonol/python_gonol/affixiation.py @@ -0,0 +1,1120 @@ +"""Construct Python 3.12 gonols from exact source occurrences, bottom up. + +Usage guidance +-------------- +Use :func:`affixiate_python_bytes` for files so the encoding declaration, BOM, +and exact original bytes remain receipt-bound. Use +:func:`affixiate_python_source` for an already-decoded source string:: + + receipt = affixiate_python_source("answer = f'{40 + 2}'\n", source_id="demo.py") + assert receipt.standing == "implemented-candidate" + assert replay_python_affixiation(receipt).receipt_digest == receipt.receipt_digest + +The constructor always starts with one letter gonol per decoded Unicode scalar +occurrence. Here ``letter`` names the admitted source floor, including spaces, +newlines, punctuation, and digits; it is not an alphabetic subclass. Tokenizer +and AST values are recognition witnesses only. No token, AST node, code object, +or compiler object is stored as a gonol or used in place of source participants. +""" + +# === MODULE_BUILD === +# id: python_gonol_affixiation +# module_name: python_gonol.affixiation +# module_kind: engine +# summary: affixiates exact Python 3.12 source from letter occurrences through lexical, delimiter, and recursive grammar gonols without substituting parser objects for construction +# owner: Python Gonol Construction (stack-local research) +# public_surface: affixiate_python_source, affixiate_python_bytes, replay_python_affixiation, reconstruct_source, PythonGonolConstructionError +# internal_surface: _SourceIndex, _Registry, _lexical_floor, _delimiter_gonols, _grammar_root +# auth_boundary: Python Gonol Construction owns source admission and Python relation construction; METAPAT owns affixiation semantics; UCNS owns geometry +# storage_boundary: none; caller-owned bytes and receipts remain in memory +# network_boundary: none +# user_data_boundary: reads caller-supplied source into an explicit caller-owned receipt and transmits nothing +# admin_only: false +# tests: tests.test_affixiation, tests.test_python312_surface +# rollout: explicit Python 3.12 stack-local candidate +# rollback: remove the python-gonol workspace before downstream binding +# requires: Python 3.12 standard library; optional explicit UCNS Public Gonol authority +# since: 2026-09-12 +# unresolved: exact UCNS affixiation geometry and later Python language profiles remain hmmm +# === END MODULE_BUILD === + +# === CAPABILITIES === +# id: python_source_bottom_up_affixiation +# summary: constructs a lossless addressable gonol registry from every decoded Python source occurrence upward +# exposes: python_gonol.affixiation.affixiate_python_source, python_gonol.affixiation.affixiate_python_bytes +# inputs: exact source text or bytes, source_id, optional explicit UCNS geometry authority +# outputs: PythonAffixiationReceipt +# boundaries: auth:none, storage:none, network:none, user_data:caller-owned receipt +# owner: Python Gonol Construction (stack-local research) +# === END CAPABILITIES === + +# === DOCS === +# id: python_gonol_affixiation_boundary_docs +# summary: explains the bottom-up construction order, parser-witness boundary, replay contract, and unresolved geometry +# audience: developer +# source: docs/PYTHON_AFFIXIATION_BOUNDARY.md +# covers: affixiate_python_source, affixiate_python_bytes, replay_python_affixiation +# status: current +# === END DOCS === + +# === BOUNDARIES === +# id: python_gonol_affixiation_runtime_boundary +# summary: reads caller-owned Python source in memory, performs no execution or network access, and optionally observes an explicitly supplied matching UCNS carrier +# auth_boundary: none +# storage_boundary: none +# network_boundary: none +# user_data_boundary: read +# admin_only: false +# pii: possible +# secrets: none +# owner: Python Gonol Construction (stack-local research) +# === END BOUNDARIES === + +# === CONTRACTS === +# id: every_python_source_occurrence_closes_first +# given: any admitted decoded Python source string, including repeated whitespace or punctuation +# then: exactly one individually addressable letter gonol closes for every Unicode scalar occurrence in exact source order +# class: construction +# +# id: python_lexical_forms_affixiate_letters +# given: the Python 3.12 lexical witness recognizes a nonempty form or an inter-token source gap +# then: one lexical gonol closes over the exact ordered letter gonols covering that form without normalization or omission +# class: construction +# +# id: python_constructions_affixiate_closed_gonols +# given: delimiter and Python grammar relations are recognized after the lexical floor closes +# then: every larger construction references already-closed gonols atomically and carries its constitutive relation, order, roles, multiplicity, source span, and provenance inside its identity +# class: construction +# +# id: python_affixiation_is_lossless_and_replayable +# given: a completed Python affixiation receipt +# then: exact decoded source reconstructs from letter gonols and every gonol plus the receipt digest verifies deterministically +# class: replay +# +# id: unresolved_python_source_remains_hmmm +# given: source has an unmatched delimiter, tokenizer failure, or Python 3.12 grammar error +# then: admitted lower gonols remain preserved beneath a source root whose standing and exact unresolved boundary are hmmm +# class: safety + +# id: parser_objects_never_become_gonols +# given: tokenizer and AST recognition witnesses are used +# then: receipt gonols contain only source-built relation records and closed-gonol references, never TokenInfo, AST, code, or compiler objects +# class: boundary +# +# id: python_gonol_geometry_binding_fails_closed +# given: an explicit UCNS Public Gonol authority is supplied +# then: exact carrier positions are observed only when the carrier and declared digest match the pinned identity; function operations remain hmmm +# class: boundary +# === END CONTRACTS === + +from __future__ import annotations + +import ast +import base64 +from dataclasses import dataclass, replace +from hashlib import sha256 +import io +import json +import keyword +import platform +import sys +import token as token_module +import tokenize +from typing import Any, Iterable, Mapping, Sequence + +from .model import ( + AffixiationRelation, + ClosedGonol, + PythonAffixiationReceipt, + RelationMember, + SourceSpan, + canonical_json_bytes, +) + + +SCHEMA = "the-interdependency.python-gonol-affixiation" +SCHEMA_VERSION = "1.0.0" +CONSTRUCTOR_ID = "python-gonol.affixiation" +CONSTRUCTOR_VERSION = "0.1.0" +LANGUAGE_PROFILE = "python-3.12-file-input" +PINNED_PUBLIC_GONOL_SHA256 = ( + "55d10c84529a4d7bc7714786357e977b68d9df2ac3f73d20e229580b552c2ef5" +) +STANDING = "implemented-candidate" +SELECTION_EFFECT = "none" +RELATION_AUTHORITY = ( + "METAPAT affixiation semantics; Python 3.12 source relation witnessed by CPython; " + "UCNS geometry not implied" +) + +NONCLAIMS = ( + "not an AST, token stream, code object, or compiler-object representation", + "not execution or behavioral equivalence of the source", + "not selected UCNS geometry or a UCNS affixiation/coupling law", + "not EDCM measurement validity", + "not canon or independent release authority", +) + +BASE_HMMM = ( + "exact UCNS geometric operation of Public Gonol function positions", + "exact UCNS Mobius-carrier affixiation/coupling law", + "Python language profiles after Python 3.12 file input", +) + + +class PythonGonolConstructionError(RuntimeError): + """Raised when a receipt or explicit authority fails closed.""" + + +@dataclass(frozen=True, slots=True) +class _LexicalWitness: + start: int + end: int + token_name: str + exact_name: str + spelling: str + gonol: ClosedGonol + + +@dataclass(frozen=True, slots=True) +class _MemberCandidate: + role: str + gonol: ClosedGonol + priority: int + + +class _SourceIndex: + def __init__(self, source: str) -> None: + self.source = source + starts = [0] + for index, character in enumerate(source): + if character == "\n": + starts.append(index + 1) + self.line_starts = tuple(starts) + + def token_offset(self, position: tuple[int, int]) -> int: + row, column = position + if row < 1: + raise PythonGonolConstructionError(f"invalid tokenizer row: {row}") + if row > len(self.line_starts): + if row == len(self.line_starts) + 1 and column == 0: + return len(self.source) + raise PythonGonolConstructionError(f"tokenizer position outside source: {position!r}") + offset = self.line_starts[row - 1] + column + if not 0 <= offset <= len(self.source): + raise PythonGonolConstructionError(f"tokenizer position outside source: {position!r}") + return offset + + def ast_offset(self, row: int, utf8_column: int) -> int: + if row < 1 or row > len(self.line_starts): + raise PythonGonolConstructionError( + f"AST position outside decoded source: {(row, utf8_column)!r}" + ) + start = self.line_starts[row - 1] + end = self.line_starts[row] if row < len(self.line_starts) else len(self.source) + line = self.source[start:end] + byte_count = 0 + for char_column, character in enumerate(line): + if byte_count == utf8_column: + return start + char_column + byte_count += len(character.encode("utf-8")) + if byte_count > utf8_column: + break + if byte_count == utf8_column: + return end + raise PythonGonolConstructionError( + f"AST UTF-8 column does not align to a source scalar: {(row, utf8_column)!r}" + ) + + def line_column(self, offset: int) -> tuple[int, int]: + if not 0 <= offset <= len(self.source): + raise PythonGonolConstructionError(f"source offset outside source: {offset}") + low = 0 + high = len(self.line_starts) + while low + 1 < high: + middle = (low + high) // 2 + if self.line_starts[middle] <= offset: + low = middle + else: + high = middle + return (low + 1, offset - self.line_starts[low]) + + def span(self, start: int, end: int) -> SourceSpan: + if not 0 <= start <= end <= len(self.source): + raise PythonGonolConstructionError(f"invalid source span: {(start, end)!r}") + start_line, start_column = self.line_column(start) + end_line, end_column = self.line_column(end) + return SourceSpan(start, end, start_line, start_column, end_line, end_column) + + +class _Registry: + def __init__(self, source_id: str, source_index: _SourceIndex) -> None: + self.source_id = source_id + self.source_index = source_index + self.values: list[ClosedGonol] = [] + self.by_id: dict[str, ClosedGonol] = {} + self.by_address: dict[str, ClosedGonol] = {} + + def close( + self, + *, + address: str, + scale: str, + start: int, + end: int, + relation_kind: str, + candidates: Sequence[_MemberCandidate] = (), + properties: Sequence[tuple[str, str]] = (), + provenance: Sequence[tuple[str, str]] = (), + hmmm: Sequence[str] = (), + ) -> ClosedGonol: + if address in self.by_address: + raise PythonGonolConstructionError(f"duplicate gonol address: {address}") + members: list[RelationMember] = [] + for ordinal, candidate in enumerate(candidates): + child = self.by_id.get(candidate.gonol.gonol_id) + if child is None or child.address != candidate.gonol.address: + raise PythonGonolConstructionError( + "larger construction may reference only an already-closed gonol" + ) + members.append( + RelationMember( + ordinal=ordinal, + role=candidate.role, + gonol_id=child.gonol_id, + address=child.address, + ) + ) + relation = AffixiationRelation( + kind=relation_kind, + members=tuple(members), + properties=tuple((str(key), str(value)) for key, value in properties), + authority=RELATION_AUTHORITY, + ) + provisional = ClosedGonol( + address=address, + scale=scale, + span=self.source_index.span(start, end), + relation=relation, + provenance=( + ("constructor", f"{CONSTRUCTOR_ID}/{CONSTRUCTOR_VERSION}"), + ("language_profile", LANGUAGE_PROFILE), + ("source_id", self.source_id), + ) + + tuple((str(key), str(value)) for key, value in provenance), + hmmm=tuple(str(item) for item in hmmm), + gonol_id="", + ) + gonol_id = sha256(canonical_json_bytes(provisional.identity_payload())).hexdigest() + value = replace(provisional, gonol_id=gonol_id) + if gonol_id in self.by_id: + raise PythonGonolConstructionError( + "closed gonol identity collision; occurrence address failed to distinguish values" + ) + self.values.append(value) + self.by_id[gonol_id] = value + self.by_address[address] = value + return value + + +def _require_source(source: str, source_id: str) -> None: + if not isinstance(source, str): + raise TypeError("source must be an exact decoded Unicode string") + if not isinstance(source_id, str) or not source_id: + raise TypeError("source_id must be exact non-empty text") + for field, value in (("source", source), ("source_id", source_id)): + for character in value: + if 0xD800 <= ord(character) <= 0xDFFF: + raise PythonGonolConstructionError(f"{field} contains a surrogate code point") + + +def _public_gonol_positions(authority: Any | None) -> tuple[dict[str, int] | None, str]: + if authority is None: + return None, "not-supplied" + if isinstance(authority, Mapping): + carrier_value = authority.get("PUBLIC_GONOL_157") + declared_digest = authority.get("PUBLIC_GONOL_SHA256") + else: + carrier_value = getattr(authority, "PUBLIC_GONOL_157", None) + declared_digest = getattr(authority, "PUBLIC_GONOL_SHA256", None) + if not isinstance(carrier_value, Sequence) or isinstance(carrier_value, (str, bytes)): + raise PythonGonolConstructionError( + "explicit UCNS authority must expose PUBLIC_GONOL_157" + ) + carrier = tuple(carrier_value) + if len(carrier) != 157 or len(set(carrier)) != 157: + raise PythonGonolConstructionError( + "explicit UCNS Public Gonol carrier must contain 157 unique positions" + ) + if any(not isinstance(item, str) or len(item) != 1 for item in carrier): + raise PythonGonolConstructionError( + "explicit UCNS Public Gonol positions must be one Unicode scalar each" + ) + computed = sha256( + json.dumps(carrier, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + ).hexdigest() + if declared_digest != computed or computed != PINNED_PUBLIC_GONOL_SHA256: + raise PythonGonolConstructionError( + "explicit UCNS Public Gonol authority digest does not match the pinned carrier" + ) + name = ( + str(authority.get("authority_name", "mapping")) + if isinstance(authority, Mapping) + else str(getattr(authority, "__name__", authority.__class__.__name__)) + ) + return {glyph: index for index, glyph in enumerate(carrier)}, name + + +def _letter_floor( + source: str, + source_id: str, + source_index: _SourceIndex, + registry: _Registry, + positions: dict[str, int] | None, + geometry_name: str, +) -> tuple[ClosedGonol, ...]: + letters: list[ClosedGonol] = [] + for index, character in enumerate(source): + address = f"{source_id}#letter:{index}" + if positions is None: + position = "hmmm:not-supplied" + else: + found = positions.get(character) + position = "hmmm:not-on-pinned-carrier" if found is None else str(found) + letters.append( + registry.close( + address=address, + scale="letter", + start=index, + end=index + 1, + relation_kind="python.source.letter-occurrence", + properties=( + ("unicode_scalar", character), + ("code_point", f"U+{ord(character):04X}"), + ("occurrence", str(index)), + ("public_gonol_position", position), + ("public_gonol_function", "hmmm"), + ), + provenance=(("geometry_authority", geometry_name),), + ) + ) + return tuple(letters) + + +def _token_relation(token_type: int, spelling: str) -> tuple[str, str]: + token_name = token_module.tok_name[token_type] + if token_type == token_module.NAME and keyword.iskeyword(spelling): + return "KEYWORD", token_name + if token_type == token_module.OP: + return token_module.tok_name.get(tokenize.EXACT_TOKEN_TYPES.get(spelling, token_type), "OP"), token_name + return token_name, token_name + + +def _lexical_floor( + source: str, + source_id: str, + source_index: _SourceIndex, + registry: _Registry, + letters: Sequence[ClosedGonol], +) -> tuple[tuple[_LexicalWitness, ...], tuple[str, ...], tuple[str, ...]]: + raw_tokens: list[tuple[int, int, str, str, str]] = [] + zero_width: list[str] = [] + unresolved: list[str] = [] + stream = tokenize.generate_tokens(io.StringIO(source).readline) + while True: + try: + item = next(stream) + except StopIteration: + break + except (tokenize.TokenError, IndentationError, SyntaxError) as exc: + detail = exc.args[0] if exc.args else exc.__class__.__name__ + location = exc.args[1] if len(exc.args) > 1 else None + unresolved.append(f"tokenizer: {detail}; location={location!r}") + break + exact_name, token_name = _token_relation(item.type, item.string) + if item.string == "": + if token_name not in {"ENDMARKER", "ENCODING"}: + zero_width.append( + f"{token_name}@{source_index.token_offset(item.start)}" + ) + continue + start = source_index.token_offset(item.start) + end = source_index.token_offset(item.end) + if start == end: + if token_name not in {"ENDMARKER", "ENCODING"}: + zero_width.append(f"{token_name}@{start}") + continue + spelling = source[start:end] + if spelling != item.string: + unresolved.append( + f"tokenizer spelling mismatch at {start}:{end}; witness={item.string!r} source={spelling!r}" + ) + raw_tokens.append((start, end, token_name, exact_name, spelling)) + + raw_tokens.sort(key=lambda value: (value[0], value[1])) + previous = 0 + complete: list[tuple[int, int, str, str, str]] = [] + for start, end, token_name, exact_name, spelling in raw_tokens: + if start < previous: + unresolved.append(f"overlapping lexical witnesses at decoded offset {start}") + continue + if start > previous: + complete.append((previous, start, "INTERTOKEN", "INTERTOKEN", source[previous:start])) + complete.append((start, end, token_name, exact_name, spelling)) + previous = end + if previous < len(source): + complete.append((previous, len(source), "INTERTOKEN", "INTERTOKEN", source[previous:])) + + witnesses: list[_LexicalWitness] = [] + for lexical_index, (start, end, token_name, exact_name, spelling) in enumerate(complete): + candidates = tuple( + _MemberCandidate(role=f"letter[{offset - start}]", gonol=letters[offset], priority=3) + for offset in range(start, end) + ) + address = f"{source_id}#lexical:{lexical_index}:{start}-{end}" + gonol = registry.close( + address=address, + scale="lexical-form", + start=start, + end=end, + relation_kind=f"python.lexical.{exact_name}", + candidates=candidates, + properties=( + ("tokenizer_class", token_name), + ("exact_form", exact_name), + ("source_length", str(end - start)), + ), + provenance=(("recognition_witness", "stdlib.tokenize/python-3.12"),), + ) + witnesses.append(_LexicalWitness(start, end, token_name, exact_name, spelling, gonol)) + return tuple(witnesses), tuple(zero_width), tuple(unresolved) + + +def _member_sort_key(candidate: _MemberCandidate) -> tuple[int, int, int, str, str]: + span = candidate.gonol.span + return (span.start, candidate.priority, -span.end, candidate.role, candidate.gonol.address) + + +def _maximal_enclosures( + enclosures: Sequence[ClosedGonol], + start: int, + end: int, + *, + excluded: Sequence[tuple[int, int]] = (), +) -> tuple[ClosedGonol, ...]: + eligible = [ + item + for item in enclosures + if start <= item.span.start + and item.span.end <= end + and not any(left <= item.span.start and item.span.end <= right for left, right in excluded) + ] + selected: list[ClosedGonol] = [] + for item in sorted(eligible, key=lambda value: (value.span.start, -value.span.end)): + if any( + parent.span.start <= item.span.start and item.span.end <= parent.span.end + for parent in selected + ): + continue + selected.append(item) + return tuple(selected) + + +def _delimiter_gonols( + source_id: str, + registry: _Registry, + letters: Sequence[ClosedGonol], + lexicals: Sequence[_LexicalWitness], +) -> tuple[tuple[ClosedGonol, ...], tuple[str, ...]]: + openers = {"(": ")", "[": "]", "{": "}"} + names = {"(": "parentheses", "[": "brackets", "{": "braces"} + stack: list[_LexicalWitness] = [] + pairs: list[tuple[_LexicalWitness, _LexicalWitness]] = [] + unresolved: list[str] = [] + for item in lexicals: + if item.spelling in openers and item.exact_name in {"LPAR", "LSQB", "LBRACE"}: + stack.append(item) + elif item.spelling in {")", "]", "}"} and item.exact_name in {"RPAR", "RSQB", "RBRACE"}: + if not stack: + unresolved.append(f"unmatched closing delimiter {item.spelling!r} at {item.start}") + continue + opener = stack.pop() + if openers[opener.spelling] != item.spelling: + unresolved.append( + f"delimiter mismatch {opener.spelling!r}@{opener.start} with {item.spelling!r}@{item.start}" + ) + continue + pairs.append((opener, item)) + for opener in stack: + unresolved.append(f"unmatched opening delimiter {opener.spelling!r} at {opener.start}") + + built: list[ClosedGonol] = [] + for pair_index, (opener, closer) in enumerate( + sorted(pairs, key=lambda pair: (pair[1].end - pair[0].start, pair[0].start)) + ): + start, end = opener.start, closer.end + nested = _maximal_enclosures(built, start, end) + nested_ranges = tuple((item.span.start, item.span.end) for item in nested) + candidates: list[_MemberCandidate] = [] + for item in lexicals: + if not (start <= item.start and item.end <= end): + continue + if any(left <= item.start and item.end <= right for left, right in nested_ranges): + continue + role = "opener" if item is opener else "closer" if item is closer else "content" + candidates.append(_MemberCandidate(role, item.gonol, 2)) + candidates.extend(_MemberCandidate("nested", item, 1) for item in nested) + candidates.sort(key=_member_sort_key) + counter = 0 + normalized: list[_MemberCandidate] = [] + for candidate in candidates: + if candidate.role in {"content", "nested"}: + normalized.append(replace(candidate, role=f"content[{counter}]")) + counter += 1 + else: + normalized.append(candidate) + built.append( + registry.close( + address=f"{source_id}#delimiter:{pair_index}:{start}-{end}", + scale="delimiter-construction", + start=start, + end=end, + relation_kind=f"python.delimiter.{names[opener.spelling]}", + candidates=normalized, + properties=( + ("opener", opener.spelling), + ("closer", closer.spelling), + ("closure", "matched"), + ), + provenance=(("recognition_witness", "python-3.12 delimiter stack"),), + hmmm=("exact UCNS delimiter relation geometry",), + ) + ) + return tuple(built), tuple(unresolved) + + +def _primitive_properties(node: ast.AST) -> tuple[tuple[str, str], ...]: + properties: list[tuple[str, str]] = [] + skipped_values = { + ("Constant", "value"), + ("MatchSingleton", "value"), + } + for field, value in ast.iter_fields(node): + if field == "ctx" or (node.__class__.__name__, field) in skipped_values: + continue + if isinstance(value, ast.AST): + if isinstance(value, (ast.operator, ast.unaryop, ast.boolop, ast.cmpop)): + properties.append((f"grammar_field.{field}", value.__class__.__name__)) + continue + if isinstance(value, list): + if value and all(isinstance(item, str) for item in value): + properties.append((f"grammar_field.{field}", json.dumps(value, ensure_ascii=False))) + continue + if value is None: + continue + if isinstance(value, (str, int, bool)): + properties.append((f"grammar_field.{field}", json.dumps(value, ensure_ascii=False))) + return tuple(properties) + + +def _spanless_relation_property(field: str, node: ast.AST) -> tuple[str, str]: + """Keep a witnessed child relation inside its source-built parent. + + A few CPython grammar records, notably an empty ``arguments`` value and + ``TypeIgnore``, have no independent source span. They cannot honestly + close as gonols of their own. Their relation is therefore recorded on the + parent that already owns the exact surface gonols; the parser record itself + is discarded. + """ + + descriptor: dict[str, Any] = {"grammar_construct": node.__class__.__name__} + for name, value in ast.iter_fields(node): + if isinstance(value, (str, int, bool)) or value is None: + descriptor[name] = value + elif isinstance(value, list) and not any(isinstance(item, ast.AST) for item in value): + descriptor[name] = value + return ( + f"grammar_field.{field}.spanless_relation", + json.dumps(descriptor, ensure_ascii=False, sort_keys=True, separators=(",", ":")), + ) + + +def _grammar_root( + source: str, + source_id: str, + source_index: _SourceIndex, + registry: _Registry, + letters: Sequence[ClosedGonol], + lexicals: Sequence[_LexicalWitness], + enclosures: Sequence[ClosedGonol], + zero_width: Sequence[str], +) -> tuple[ClosedGonol | None, tuple[str, ...]]: + try: + tree = ast.parse( + source, + filename=source_id, + mode="exec", + type_comments=True, + feature_version=(3, 12), + ) + except (SyntaxError, ValueError, TypeError, MemoryError) as exc: + if isinstance(exc, SyntaxError): + detail = f"grammar: {exc.msg}; line={exc.lineno!r}; offset={exc.offset!r}" + else: + detail = f"grammar: {exc.__class__.__name__}: {exc}" + return None, (detail,) + + node_order = {id(node): index for index, node in enumerate(ast.walk(tree))} + span_memo: dict[int, tuple[int, int] | None] = {} + + def node_span(node: ast.AST) -> tuple[int, int] | None: + key = id(node) + if key in span_memo: + return span_memo[key] + spans: list[tuple[int, int]] = [] + if all(hasattr(node, name) for name in ("lineno", "col_offset", "end_lineno", "end_col_offset")): + end_line = getattr(node, "end_lineno", None) + end_column = getattr(node, "end_col_offset", None) + if end_line is not None and end_column is not None: + spans.append( + ( + source_index.ast_offset(int(node.lineno), int(node.col_offset)), + source_index.ast_offset(int(end_line), int(end_column)), + ) + ) + for child in ast.iter_child_nodes(node): + child_span = node_span(child) + if child_span is not None: + spans.append(child_span) + if isinstance(node, ast.Module): + result: tuple[int, int] | None = (0, len(source)) + elif spans: + result = (min(item[0] for item in spans), max(item[1] for item in spans)) + else: + result = None + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)) and node.decorator_list: + first_decorator = min(node.decorator_list, key=lambda item: (item.lineno, item.col_offset)) + decorator_start = source_index.ast_offset( + int(first_decorator.lineno), int(first_decorator.col_offset) + ) + marker = max( + ( + item + for item in lexicals + if item.spelling == "@" + and item.exact_name == "AT" + and item.end <= decorator_start + and source_index.line_column(item.start)[0] == int(first_decorator.lineno) + ), + key=lambda item: item.start, + default=None, + ) + if marker is not None and result is not None: + result = (min(marker.start, result[0]), result[1]) + span_memo[key] = result + return result + + node_span(tree) + built: dict[int, ClosedGonol] = {} + + def construct(node: ast.AST) -> ClosedGonol | None: + key = id(node) + if key in built: + return built[key] + span = node_span(node) + if span is None: + return None + start, end = span + child_candidates: list[_MemberCandidate] = [] + child_ranges: list[tuple[int, int]] = [] + spanless_properties: list[tuple[str, str]] = [] + for field, value in ast.iter_fields(node): + if isinstance(value, ast.AST): + child = construct(value) + if child is not None: + child_candidates.append(_MemberCandidate(field, child, 0)) + child_ranges.append((child.span.start, child.span.end)) + elif not isinstance( + value, + (ast.operator, ast.unaryop, ast.boolop, ast.cmpop, ast.expr_context), + ): + spanless_properties.append(_spanless_relation_property(field, value)) + elif isinstance(value, list): + for position, item in enumerate(value): + if not isinstance(item, ast.AST): + continue + child = construct(item) + if child is not None: + child_candidates.append(_MemberCandidate(f"{field}[{position}]", child, 0)) + child_ranges.append((child.span.start, child.span.end)) + elif not isinstance( + item, + (ast.operator, ast.unaryop, ast.boolop, ast.cmpop, ast.expr_context), + ): + spanless_properties.append( + _spanless_relation_property(f"{field}[{position}]", item) + ) + + selected_enclosures = _maximal_enclosures( + enclosures, + start, + end, + excluded=child_ranges, + ) + enclosure_ranges = tuple((item.span.start, item.span.end) for item in selected_enclosures) + surface_candidates: list[_MemberCandidate] = [ + _MemberCandidate("surface", item, 1) for item in selected_enclosures + ] + for lexical in lexicals: + if not (start <= lexical.start and lexical.end <= end): + continue + if any(left <= lexical.start and lexical.end <= right for left, right in child_ranges): + continue + if any(left <= lexical.start and lexical.end <= right for left, right in enclosure_ranges): + continue + surface_candidates.append(_MemberCandidate("surface", lexical.gonol, 2)) + + candidates = child_candidates + surface_candidates + candidates.sort(key=_member_sort_key) + surface_index = 0 + normalized: list[_MemberCandidate] = [] + for candidate in candidates: + if candidate.role == "surface": + normalized.append(replace(candidate, role=f"surface[{surface_index}]")) + surface_index += 1 + else: + normalized.append(candidate) + properties = [ + ("grammar_profile", LANGUAGE_PROFILE), + ("grammar_construct", node.__class__.__name__), + ] + properties.extend(_primitive_properties(node)) + properties.extend(spanless_properties) + if isinstance(node, ast.Module): + properties.extend( + (f"zero_width_witness[{index}]", item) + for index, item in enumerate(zero_width) + ) + value = registry.close( + address=( + f"{source_id}#grammar:{node_order[key]}:{node.__class__.__name__}:{start}-{end}" + ), + scale="module" if isinstance(node, ast.Module) else "python-construction", + start=start, + end=end, + relation_kind=f"python.grammar.{node.__class__.__name__}", + candidates=normalized, + properties=properties, + provenance=(("recognition_witness", "stdlib.ast/python-3.12"),), + hmmm=("exact UCNS geometry for this Python relation",), + ) + built[key] = value + return value + + return construct(tree), () + + +def _source_root_hmmm( + source_id: str, + registry: _Registry, + letters: Sequence[ClosedGonol], + lexicals: Sequence[_LexicalWitness], + enclosures: Sequence[ClosedGonol], + unresolved: Sequence[str], + zero_width: Sequence[str], +) -> ClosedGonol: + selected = _maximal_enclosures(enclosures, 0, len(letters)) + selected_ranges = tuple((item.span.start, item.span.end) for item in selected) + candidates: list[_MemberCandidate] = [ + _MemberCandidate("closed-delimiter", item, 1) for item in selected + ] + for item in lexicals: + if any(left <= item.start and item.end <= right for left, right in selected_ranges): + continue + candidates.append(_MemberCandidate("lexical-form", item.gonol, 2)) + candidates.sort(key=_member_sort_key) + properties = [("standing", "hmmm")] + properties.extend((f"unresolved[{index}]", item) for index, item in enumerate(unresolved)) + properties.extend( + (f"zero_width_witness[{index}]", item) for index, item in enumerate(zero_width) + ) + return registry.close( + address=f"{source_id}#source:hmmm", + scale="source", + start=0, + end=len(letters), + relation_kind="python.source.hmmm", + candidates=candidates, + properties=properties, + provenance=(("recognition_witness", "partial Python 3.12 recognition"),), + hmmm=unresolved, + ) + + +def _receipt_digest(receipt: PythonAffixiationReceipt) -> str: + return sha256(canonical_json_bytes(receipt.payload())).hexdigest() + + +def _construct( + source: str, + *, + source_id: str, + source_bytes: bytes, + encoding: str, + geometry_authority: Any | None, +) -> PythonAffixiationReceipt: + _require_source(source, source_id) + if sys.version_info[:2] != (3, 12): + raise PythonGonolConstructionError( + f"{LANGUAGE_PROFILE} requires a Python 3.12 recognition witness; got {sys.version_info.major}.{sys.version_info.minor}" + ) + source_index = _SourceIndex(source) + registry = _Registry(source_id, source_index) + positions, geometry_name = _public_gonol_positions(geometry_authority) + letters = _letter_floor( + source, + source_id, + source_index, + registry, + positions, + geometry_name, + ) + lexicals, zero_width, lexical_hmmm = _lexical_floor( + source, + source_id, + source_index, + registry, + letters, + ) + enclosures, delimiter_hmmm = _delimiter_gonols( + source_id, + registry, + letters, + lexicals, + ) + grammar_root, grammar_hmmm = _grammar_root( + source, + source_id, + source_index, + registry, + letters, + lexicals, + enclosures, + zero_width, + ) + syntax_hmmm = lexical_hmmm + delimiter_hmmm + grammar_hmmm + if grammar_root is None or syntax_hmmm: + root = _source_root_hmmm( + source_id, + registry, + letters, + lexicals, + enclosures, + syntax_hmmm or ("grammar root was not constructed",), + zero_width, + ) + standing = "hmmm" + else: + root = grammar_root + standing = STANDING + + hmmm = list(BASE_HMMM) + if geometry_authority is None: + hmmm.append("UCNS Public Gonol geometry authority was not supplied") + if positions is not None and any(character not in positions for character in source): + hmmm.append("one or more admitted source scalars have no position on the pinned Public Gonol carrier") + hmmm.extend(syntax_hmmm) + runtime = f"{platform.python_implementation()}-{platform.python_version()}" + provisional = PythonAffixiationReceipt( + schema=SCHEMA, + version=SCHEMA_VERSION, + constructor_id=CONSTRUCTOR_ID, + constructor_version=CONSTRUCTOR_VERSION, + language_profile=LANGUAGE_PROFILE, + source_id=source_id, + encoding=encoding, + source_bytes_base64=base64.b64encode(source_bytes).decode("ascii"), + source_bytes_sha256=sha256(source_bytes).hexdigest(), + decoded_source_sha256=sha256(source.encode("utf-8")).hexdigest(), + recognition_witness=runtime, + standing=standing, + selection_effect=SELECTION_EFFECT, + root_gonol_id=root.gonol_id, + gonols=tuple(registry.values), + nonclaims=NONCLAIMS, + hmmm=tuple(hmmm), + receipt_digest="", + ) + return replace(provisional, receipt_digest=_receipt_digest(provisional)) + + +def affixiate_python_source( + source: str, + *, + source_id: str, + geometry_authority: Any | None = None, +) -> PythonAffixiationReceipt: + """Affixiate exact decoded Python 3.12 file-input source from letters upward.""" + + if not isinstance(source, str): + raise TypeError("source must be an exact decoded Unicode string") + return _construct( + source, + source_id=source_id, + source_bytes=source.encode("utf-8"), + encoding="utf-8", + geometry_authority=geometry_authority, + ) + + +def affixiate_python_bytes( + source_bytes: bytes, + *, + source_id: str, + geometry_authority: Any | None = None, +) -> PythonAffixiationReceipt: + """Detect Python's declared encoding, preserve exact bytes, and affixiate source.""" + + if not isinstance(source_bytes, bytes): + raise TypeError("source_bytes must be exact bytes") + try: + encoding, _lines = tokenize.detect_encoding(io.BytesIO(source_bytes).readline) + source = source_bytes.decode(encoding) + except (SyntaxError, UnicodeDecodeError, LookupError) as exc: + raise PythonGonolConstructionError(f"Python source decoding failed: {exc}") from exc + return _construct( + source, + source_id=source_id, + source_bytes=source_bytes, + encoding=encoding, + geometry_authority=geometry_authority, + ) + + +def _property(gonol: ClosedGonol, name: str) -> str: + values = [value for key, value in gonol.relation.properties if key == name] + if len(values) != 1: + raise PythonGonolConstructionError( + f"gonol {gonol.address} must carry exactly one {name!r} property" + ) + return values[0] + + +def reconstruct_source(receipt: PythonAffixiationReceipt) -> str: + """Reconstruct decoded source solely from ordered letter gonols.""" + + letters = sorted( + (gonol for gonol in receipt.gonols if gonol.scale == "letter"), + key=lambda gonol: gonol.span.start, + ) + return "".join(_property(gonol, "unicode_scalar") for gonol in letters) + + +def replay_python_affixiation(receipt: PythonAffixiationReceipt) -> PythonAffixiationReceipt: + """Fail closed unless every visible gonol and receipt identity remains valid.""" + + if not isinstance(receipt, PythonAffixiationReceipt): + raise TypeError("receipt must be a PythonAffixiationReceipt") + expected = (SCHEMA, SCHEMA_VERSION, CONSTRUCTOR_ID, CONSTRUCTOR_VERSION, LANGUAGE_PROFILE) + actual = ( + receipt.schema, + receipt.version, + receipt.constructor_id, + receipt.constructor_version, + receipt.language_profile, + ) + if actual != expected: + raise PythonGonolConstructionError("receipt schema, constructor, or language profile mismatch") + try: + raw = base64.b64decode(receipt.source_bytes_base64, validate=True) + except ValueError as exc: + raise PythonGonolConstructionError("receipt source bytes are not valid base64") from exc + if sha256(raw).hexdigest() != receipt.source_bytes_sha256: + raise PythonGonolConstructionError("receipt source byte digest mismatch") + source = reconstruct_source(receipt) + if sha256(source.encode("utf-8")).hexdigest() != receipt.decoded_source_sha256: + raise PythonGonolConstructionError("decoded source digest mismatch") + try: + decoded = raw.decode(receipt.encoding) + except (UnicodeDecodeError, LookupError) as exc: + raise PythonGonolConstructionError("receipt source bytes no longer decode as declared") from exc + if decoded != source: + raise PythonGonolConstructionError("receipt bytes and letter gonols reconstruct different source") + + source_index = _SourceIndex(source) + known: dict[str, ClosedGonol] = {} + known_addresses: set[str] = set() + expected_letter_start = 0 + for gonol in receipt.gonols: + if gonol.gonol_id in known: + raise PythonGonolConstructionError("duplicate gonol identity in receipt") + if gonol.address in known_addresses: + raise PythonGonolConstructionError("duplicate gonol address in receipt") + expected_id = sha256(canonical_json_bytes(gonol.identity_payload())).hexdigest() + if gonol.gonol_id != expected_id: + raise PythonGonolConstructionError(f"gonol identity mismatch: {gonol.address}") + if gonol.span != source_index.span(gonol.span.start, gonol.span.end): + raise PythonGonolConstructionError(f"source coordinates drifted: {gonol.address}") + if tuple(member.ordinal for member in gonol.relation.members) != tuple( + range(len(gonol.relation.members)) + ): + raise PythonGonolConstructionError(f"member order drifted: {gonol.address}") + member_spans: list[tuple[int, int]] = [] + for member in gonol.relation.members: + child = known.get(member.gonol_id) + if child is None or child.address != member.address: + raise PythonGonolConstructionError( + f"gonol references a child that was not already closed: {gonol.address}" + ) + member_spans.append((child.span.start, child.span.end)) + if gonol.scale == "letter": + if gonol.span.start != expected_letter_start or gonol.span.end != expected_letter_start + 1: + raise PythonGonolConstructionError("letter floor is not contiguous and ordered") + scalar = _property(gonol, "unicode_scalar") + if len(scalar) != 1 or scalar != source[gonol.span.start : gonol.span.end]: + raise PythonGonolConstructionError("letter gonol does not match its source occurrence") + expected_letter_start += 1 + else: + merged: list[list[int]] = [] + for start, end in sorted(member_spans): + if not merged or start > merged[-1][1]: + merged.append([start, end]) + elif end > merged[-1][1]: + merged[-1][1] = end + expected = [] if gonol.span.start == gonol.span.end else [[gonol.span.start, gonol.span.end]] + if merged != expected: + raise PythonGonolConstructionError( + f"gonol span differs from its atomic closed participants: {gonol.address}" + ) + known[gonol.gonol_id] = gonol + known_addresses.add(gonol.address) + if expected_letter_start != len(source): + raise PythonGonolConstructionError("letter floor does not cover complete decoded source") + root = known.get(receipt.root_gonol_id) + if ( + root is None + or not receipt.gonols + or root is not receipt.gonols[-1] + or root.span.start != 0 + or root.span.end != len(source) + or root.scale not in {"module", "source"} + ): + raise PythonGonolConstructionError("receipt root does not cover every source occurrence") + if receipt.nonclaims != NONCLAIMS or not all(item in receipt.hmmm for item in BASE_HMMM): + raise PythonGonolConstructionError("receipt nonclaim or hmmm boundary mismatch") + if receipt.receipt_digest != _receipt_digest(replace(receipt, receipt_digest="")): + raise PythonGonolConstructionError("receipt digest mismatch") + return receipt + + +__all__ = [ + "CONSTRUCTOR_ID", + "CONSTRUCTOR_VERSION", + "LANGUAGE_PROFILE", + "PINNED_PUBLIC_GONOL_SHA256", + "PythonGonolConstructionError", + "affixiate_python_bytes", + "affixiate_python_source", + "reconstruct_source", + "replay_python_affixiation", +] diff --git a/research/python-gonol/python_gonol/model.py b/research/python-gonol/python_gonol/model.py new file mode 100644 index 0000000..2b61ae3 --- /dev/null +++ b/research/python-gonol/python_gonol/model.py @@ -0,0 +1,277 @@ +"""Immutable records for bottom-up Python source gonol affixiation. + +Usage guidance +-------------- +Callers normally receive these records from :func:`affixiate_python_source` or +:func:`affixiate_python_bytes`. A parent gonol contains only identity-bearing +references to already-closed children. The receipt registry keeps every child +recoverable without reopening it during construction. +""" + +# === MODULE_BUILD === +# id: python_gonol_model +# module_name: python_gonol.model +# module_kind: schema +# summary: defines immutable source spans, intrinsic affixiation relations, closed gonols, and deterministic Python affixiation receipts +# owner: Python Gonol Construction (stack-local research) +# public_surface: SourceSpan, RelationMember, AffixiationRelation, ClosedGonol, PythonAffixiationReceipt, canonical_json_bytes +# internal_surface: none +# auth_boundary: none +# storage_boundary: immutable caller-owned values only +# network_boundary: none +# user_data_boundary: caller-supplied source remains inside the receipt +# admin_only: false +# tests: tests.test_affixiation +# rollout: imported by the explicit Python 3.12 candidate constructor +# rollback: remove the python-gonol research workspace before any downstream consumer binds its schema +# requires: none +# since: 2026-09-12 +# unresolved: independent release schema and migration policy remain hmmm +# === END MODULE_BUILD === + +# === CONTRACTS === +# id: python_gonol_parent_references_closed_children +# given: a non-letter gonol is present in a receipt +# then: every constitutive member names an already-closed gonol by exact address and identity while the child remains independently recoverable +# class: construction +# +# id: python_gonol_receipt_is_canonical_json +# given: the same visible receipt payload is serialized repeatedly +# then: sorted compact UTF-8 JSON bytes and the resulting SHA-256 identity are byte-identical +# class: replay +# === END CONTRACTS === + +from __future__ import annotations + +from dataclasses import dataclass +import json +from typing import Any, Mapping + + +def canonical_json_bytes(value: Mapping[str, Any]) -> bytes: + """Return deterministic UTF-8 JSON bytes for identities and receipts.""" + + return json.dumps( + value, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ).encode("utf-8") + + +@dataclass(frozen=True, slots=True) +class SourceSpan: + """Half-open decoded-source span with human-readable line coordinates.""" + + start: int + end: int + start_line: int + start_column: int + end_line: int + end_column: int + + def to_dict(self) -> dict[str, int]: + return { + "start": self.start, + "end": self.end, + "start_line": self.start_line, + "start_column": self.start_column, + "end_line": self.end_line, + "end_column": self.end_column, + } + + @classmethod + def from_dict(cls, value: Mapping[str, Any]) -> "SourceSpan": + return cls( + start=int(value["start"]), + end=int(value["end"]), + start_line=int(value["start_line"]), + start_column=int(value["start_column"]), + end_line=int(value["end_line"]), + end_column=int(value["end_column"]), + ) + + +@dataclass(frozen=True, slots=True) +class RelationMember: + """One ordered, role-bearing reference to an already-closed gonol.""" + + ordinal: int + role: str + gonol_id: str + address: str + + def to_dict(self) -> dict[str, Any]: + return { + "ordinal": self.ordinal, + "role": self.role, + "gonol_id": self.gonol_id, + "address": self.address, + } + + @classmethod + def from_dict(cls, value: Mapping[str, Any]) -> "RelationMember": + return cls( + ordinal=int(value["ordinal"]), + role=str(value["role"]), + gonol_id=str(value["gonol_id"]), + address=str(value["address"]), + ) + + +@dataclass(frozen=True, slots=True) +class AffixiationRelation: + """The identity-bearing relation that makes one construction a whole.""" + + kind: str + members: tuple[RelationMember, ...] + properties: tuple[tuple[str, str], ...] + authority: str + + def to_dict(self) -> dict[str, Any]: + return { + "kind": self.kind, + "members": [member.to_dict() for member in self.members], + "properties": [list(pair) for pair in self.properties], + "authority": self.authority, + } + + @classmethod + def from_dict(cls, value: Mapping[str, Any]) -> "AffixiationRelation": + return cls( + kind=str(value["kind"]), + members=tuple(RelationMember.from_dict(item) for item in value["members"]), + properties=tuple((str(key), str(item)) for key, item in value["properties"]), + authority=str(value["authority"]), + ) + + +@dataclass(frozen=True, slots=True) +class ClosedGonol: + """One closed gonol; parents consume its identity, never flattened descendants.""" + + address: str + scale: str + span: SourceSpan + relation: AffixiationRelation + provenance: tuple[tuple[str, str], ...] + hmmm: tuple[str, ...] + gonol_id: str + + def identity_payload(self) -> dict[str, Any]: + return { + "address": self.address, + "scale": self.scale, + "span": self.span.to_dict(), + "relation": self.relation.to_dict(), + "provenance": [list(pair) for pair in self.provenance], + "hmmm": list(self.hmmm), + } + + def to_dict(self) -> dict[str, Any]: + return {**self.identity_payload(), "gonol_id": self.gonol_id} + + @classmethod + def from_dict(cls, value: Mapping[str, Any]) -> "ClosedGonol": + return cls( + address=str(value["address"]), + scale=str(value["scale"]), + span=SourceSpan.from_dict(value["span"]), + relation=AffixiationRelation.from_dict(value["relation"]), + provenance=tuple((str(key), str(item)) for key, item in value["provenance"]), + hmmm=tuple(str(item) for item in value["hmmm"]), + gonol_id=str(value["gonol_id"]), + ) + + +@dataclass(frozen=True, slots=True) +class PythonAffixiationReceipt: + """Complete visible registry and provenance for one source construction.""" + + schema: str + version: str + constructor_id: str + constructor_version: str + language_profile: str + source_id: str + encoding: str + source_bytes_base64: str + source_bytes_sha256: str + decoded_source_sha256: str + recognition_witness: str + standing: str + selection_effect: str + root_gonol_id: str + gonols: tuple[ClosedGonol, ...] + nonclaims: tuple[str, ...] + hmmm: tuple[str, ...] + receipt_digest: str + + def payload(self) -> dict[str, Any]: + return { + "schema": self.schema, + "version": self.version, + "constructor_id": self.constructor_id, + "constructor_version": self.constructor_version, + "language_profile": self.language_profile, + "source_id": self.source_id, + "encoding": self.encoding, + "source_bytes_base64": self.source_bytes_base64, + "source_bytes_sha256": self.source_bytes_sha256, + "decoded_source_sha256": self.decoded_source_sha256, + "recognition_witness": self.recognition_witness, + "standing": self.standing, + "selection_effect": self.selection_effect, + "root_gonol_id": self.root_gonol_id, + "gonols": [gonol.to_dict() for gonol in self.gonols], + "nonclaims": list(self.nonclaims), + "hmmm": list(self.hmmm), + } + + def to_dict(self) -> dict[str, Any]: + return {**self.payload(), "receipt_digest": self.receipt_digest} + + def to_json(self, *, pretty: bool = False) -> str: + if pretty: + return json.dumps(self.to_dict(), ensure_ascii=False, sort_keys=True, indent=2) + "\n" + return canonical_json_bytes(self.to_dict()).decode("utf-8") + "\n" + + @classmethod + def from_dict(cls, value: Mapping[str, Any]) -> "PythonAffixiationReceipt": + return cls( + schema=str(value["schema"]), + version=str(value["version"]), + constructor_id=str(value["constructor_id"]), + constructor_version=str(value["constructor_version"]), + language_profile=str(value["language_profile"]), + source_id=str(value["source_id"]), + encoding=str(value["encoding"]), + source_bytes_base64=str(value["source_bytes_base64"]), + source_bytes_sha256=str(value["source_bytes_sha256"]), + decoded_source_sha256=str(value["decoded_source_sha256"]), + recognition_witness=str(value["recognition_witness"]), + standing=str(value["standing"]), + selection_effect=str(value["selection_effect"]), + root_gonol_id=str(value["root_gonol_id"]), + gonols=tuple(ClosedGonol.from_dict(item) for item in value["gonols"]), + nonclaims=tuple(str(item) for item in value["nonclaims"]), + hmmm=tuple(str(item) for item in value["hmmm"]), + receipt_digest=str(value["receipt_digest"]), + ) + + @classmethod + def from_json(cls, source: str) -> "PythonAffixiationReceipt": + value = json.loads(source) + if not isinstance(value, dict): + raise TypeError("receipt JSON must contain one object") + return cls.from_dict(value) + + +__all__ = [ + "AffixiationRelation", + "ClosedGonol", + "PythonAffixiationReceipt", + "RelationMember", + "SourceSpan", + "canonical_json_bytes", +] diff --git a/research/python-gonol/tests/test_affixiation.py b/research/python-gonol/tests/test_affixiation.py new file mode 100644 index 0000000..927e9b7 --- /dev/null +++ b/research/python-gonol/tests/test_affixiation.py @@ -0,0 +1,213 @@ +# === CHECKS === +# id: every_python_source_occurrence_closes_first_check +# proves: every_python_source_occurrence_closes_first +# call: self::test_letter_floor_preserves_identity_order_multiplicity_and_provenance +# timeout: 30 +# mutates: none +# cleanup: none +# +# id: python_lexical_forms_affixiate_letters_check +# proves: python_lexical_forms_affixiate_letters +# call: self::test_lexical_floor_is_an_exact_partition_of_closed_letters +# timeout: 30 +# mutates: none +# cleanup: none +# +# id: python_constructions_affixiate_closed_gonols_check +# proves: python_constructions_affixiate_closed_gonols, python_gonol_parent_references_closed_children +# call: self::test_larger_constructions_reference_prior_closed_gonols_and_own_relations +# timeout: 30 +# mutates: none +# cleanup: none +# +# id: python_affixiation_is_lossless_and_replayable_check +# proves: python_affixiation_is_lossless_and_replayable, python_gonol_receipt_is_canonical_json +# call: self::test_receipt_roundtrip_reconstructs_exact_source_and_replays +# timeout: 30 +# mutates: none +# cleanup: none +# +# id: unresolved_python_source_remains_hmmm_check +# proves: unresolved_python_source_remains_hmmm +# call: self::test_missing_parenthesis_preserves_lower_closures_and_records_hmmm +# timeout: 30 +# mutates: none +# cleanup: none +# +# id: parser_objects_never_become_gonols_check +# proves: parser_objects_never_become_gonols +# call: self::test_receipt_contains_source_relations_not_parser_objects +# timeout: 30 +# mutates: none +# cleanup: none +# +# id: python_gonol_geometry_binding_fails_closed_check +# proves: python_gonol_geometry_binding_fails_closed +# call: self::test_exact_ucns_carrier_is_observed_and_drift_fails_closed +# timeout: 30 +# mutates: none +# cleanup: none +# === END CHECKS === + +from __future__ import annotations + +from collections import Counter +from dataclasses import replace +import importlib.util +import json +from pathlib import Path +import subprocess +import sys + +import pytest + +from python_gonol import ( + PythonAffixiationReceipt, + PythonGonolConstructionError, + affixiate_python_bytes, + affixiate_python_source, + reconstruct_source, + replay_python_affixiation, +) + + +def _letters(receipt: PythonAffixiationReceipt): + return tuple(gonol for gonol in receipt.gonols if gonol.scale == "letter") + + +def test_letter_floor_preserves_identity_order_multiplicity_and_provenance() -> None: + source = "π = (a + a)\n# a\n" + receipt = affixiate_python_source(source, source_id="fixture/repeated.py") + letters = _letters(receipt) + assert len(letters) == len(source) + assert [item.span.start for item in letters] == list(range(len(source))) + assert len({item.address for item in letters}) == len(source) + assert len({item.gonol_id for item in letters}) == len(source) + repeated = [item for item in letters if dict(item.relation.properties)["unicode_scalar"] == "a"] + assert len(repeated) == 3 + assert len({item.address for item in repeated}) == 3 + assert all(("source_id", "fixture/repeated.py") in item.provenance for item in letters) + + +def test_lexical_floor_is_an_exact_partition_of_closed_letters() -> None: + source = "value = f'{name!r:>{width}}' # keep both spaces\n" + receipt = affixiate_python_source(source, source_id="fixture/lexical.py") + lexical = [gonol for gonol in receipt.gonols if gonol.scale == "lexical-form"] + counts = Counter(member.address for gonol in lexical for member in gonol.relation.members) + assert counts == Counter(item.address for item in _letters(receipt)) + earlier = {gonol.gonol_id: index for index, gonol in enumerate(receipt.gonols)} + for gonol in lexical: + assert gonol.relation.members + assert all(receipt.gonols[earlier[member.gonol_id]].scale == "letter" for member in gonol.relation.members) + kinds = {gonol.relation.kind for gonol in lexical} + assert "python.lexical.FSTRING_START" in kinds + assert "python.lexical.COMMENT" in kinds + assert "python.lexical.INTERTOKEN" in kinds + + +def test_larger_constructions_reference_prior_closed_gonols_and_own_relations() -> None: + source = "answer = (((left + right)))\n" + receipt = affixiate_python_source(source, source_id="fixture/nesting.py") + position = {gonol.gonol_id: index for index, gonol in enumerate(receipt.gonols)} + delimiters = [gonol for gonol in receipt.gonols if gonol.scale == "delimiter-construction"] + assert len(delimiters) == 3 + assert all(gonol.relation.kind == "python.delimiter.parentheses" for gonol in delimiters) + assert receipt.gonols[-1].scale == "module" + assert receipt.gonols[-1].relation.kind == "python.grammar.Module" + assert any(gonol.relation.kind == "python.grammar.BinOp" for gonol in receipt.gonols) + for parent_index, gonol in enumerate(receipt.gonols): + for member in gonol.relation.members: + assert position[member.gonol_id] < parent_index + assert receipt.gonols[position[member.gonol_id]].address == member.address + + +def test_receipt_roundtrip_reconstructs_exact_source_and_replays() -> None: + raw = b"# coding: latin-1\nname = 'caf\xe9'\r\n" + receipt = affixiate_python_bytes(raw, source_id="fixture/latin1.py") + assert receipt.source_bytes_sha256 + assert receipt.encoding == "iso-8859-1" + assert reconstruct_source(receipt) == "# coding: latin-1\nname = 'caf\N{LATIN SMALL LETTER E WITH ACUTE}'\r\n" + roundtrip = PythonAffixiationReceipt.from_json(receipt.to_json()) + assert replay_python_affixiation(roundtrip).receipt_digest == receipt.receipt_digest + assert receipt.to_json() == roundtrip.to_json() + + root = roundtrip.gonols[-1] + tampered_root = replace(root, relation=replace(root.relation, kind="python.grammar.Expression")) + tampered = replace(roundtrip, gonols=roundtrip.gonols[:-1] + (tampered_root,)) + with pytest.raises(PythonGonolConstructionError, match="gonol identity mismatch"): + replay_python_affixiation(tampered) + + +def test_missing_parenthesis_preserves_lower_closures_and_records_hmmm(tmp_path: Path) -> None: + source = "result = call(1, 2\n" + receipt = affixiate_python_source(source, source_id="fixture/missing.py") + assert receipt.standing == "hmmm" + assert receipt.gonols[-1].relation.kind == "python.source.hmmm" + assert reconstruct_source(receipt) == source + assert len(_letters(receipt)) == len(source) + assert any("unmatched opening delimiter" in item for item in receipt.hmmm) + assert any(item.startswith("grammar:") for item in receipt.hmmm) + replay_python_affixiation(receipt) + + source_path = tmp_path / "missing.py" + receipt_path = tmp_path / "missing.gonol.json" + source_path.write_text(source, encoding="utf-8") + run = subprocess.run( + [sys.executable, "-m", "python_gonol", str(source_path), "--out", str(receipt_path)], + cwd=Path(__file__).resolve().parents[1], + check=False, + capture_output=True, + text=True, + ) + assert run.returncode == 2 + assert receipt_path.is_file() + assert "hmmm:" in run.stderr + + +def test_receipt_contains_source_relations_not_parser_objects() -> None: + receipt = affixiate_python_source( + "def f(x: int = 1) -> int:\n return x + 1\n", + source_id="fixture/no-substitution.py", + ) + encoded = json.dumps(receipt.to_dict(), ensure_ascii=False) + assert "TokenInfo" not in encoded + assert "<_ast." not in encoded + assert "occurrence_addresses" not in encoded + assert all(gonol.scale in {"letter", "lexical-form", "delimiter-construction", "python-construction", "module"} for gonol in receipt.gonols) + assert all(type(gonol).__module__ == "python_gonol.model" for gonol in receipt.gonols) + replay_python_affixiation(receipt) + + +def test_exact_ucns_carrier_is_observed_and_drift_fails_closed() -> None: + path = Path(__file__).resolve().parents[3] / "libs" / "ucns" / "src" / "ucns" / "public_gonol.py" + spec = importlib.util.spec_from_file_location("pinned_ucns_public_gonol", path) + assert spec is not None and spec.loader is not None + authority = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = authority + try: + spec.loader.exec_module(authority) + finally: + sys.modules.pop(spec.name, None) + + receipt = affixiate_python_source( + "a + 1", + source_id="fixture/ucns.py", + geometry_authority=authority, + ) + for letter in _letters(receipt): + properties = dict(letter.relation.properties) + assert properties["public_gonol_position"].isdigit() + assert properties["public_gonol_function"] == "hmmm" + assert "UCNS Public Gonol geometry authority was not supplied" not in receipt.hmmm + replay_python_affixiation(receipt) + + drifted = { + "PUBLIC_GONOL_157": authority.PUBLIC_GONOL_157, + "PUBLIC_GONOL_SHA256": "0" * 64, + } + with pytest.raises(PythonGonolConstructionError, match="digest"): + affixiate_python_source( + "a", + source_id="fixture/drift.py", + geometry_authority=drifted, + ) diff --git a/research/python-gonol/tests/test_python312_surface.py b/research/python-gonol/tests/test_python312_surface.py new file mode 100644 index 0000000..ca43b70 --- /dev/null +++ b/research/python-gonol/tests/test_python312_surface.py @@ -0,0 +1,175 @@ +# === CHECKS === +# id: python312_full_surface_affixiates_check +# proves: python_constructions_affixiate_closed_gonols, python_affixiation_is_lossless_and_replayable +# call: self::test_python312_surface_affixiates_from_letters_through_module +# timeout: 30 +# mutates: none +# cleanup: none +# +# id: spanless_python_relations_remain_intrinsic_check +# proves: python_constructions_affixiate_closed_gonols, parser_objects_never_become_gonols +# call: self::test_spanless_grammar_relations_remain_inside_the_source_parent +# timeout: 30 +# mutates: none +# cleanup: none +# === END CHECKS === + +from __future__ import annotations + +from python_gonol import ( + affixiate_python_source, + reconstruct_source, + replay_python_affixiation, +) + + +PYTHON_312_SURFACE = '''\ +from __future__ import annotations +import os as operating_system +from .package import value as renamed + +type Pair[T] = tuple[T, T] + +@decorate(flag=True) +class Example[T](Base, metaclass=Meta): + class_attr: int = 1 + + def method(self, x: int = 0, /, *args: str, flag=True, **kwargs) -> int: + global module_name + local: int + local = (x := x + 1) + assert local >= 0, "nonnegative" + if flag and local: + values = [item * 2 for item in range(local) if item % 2] + elif not flag or local is None: + values = {key: value for key, value in enumerate(args)} + else: + values = {*(1, 2), *range(3)} + for item in values: + if item == 2: + continue + break + while local > 0: + local -= 1 + try: + with open("path") as handle, lock: + data = handle.read() + except OSError as exc: + raise RuntimeError("failed") from exc + else: + pass + finally: + del local + match data: + case {"x": [first, *rest]} if first > 0: + point = first + case Example(value=point): + point = point + case 1 | 2: + point = 0 + case _: + point = -1 + return lambda y: y if y else None + + async def stream(self, source): + async with source as opened: + async for value in opened: + yield f"{value=!r:>{width}}" + await source.close() + + +def nested(): + captured = 1 + def inner(): + nonlocal captured + captured += 1 + return captured + return inner + + +def generator(): + yield from (number for number in range(3)) + + +def grouped_errors(): + try: + raise ExceptionGroup("group", [ValueError()]) + except* ValueError as errors: + pass + + +raw = rb"bytes\\x00" +text = r"raw" "joined" +ellipsis_value = ... +mapping = {"slice": values[1:10:2], "call": callable(*args, **kwargs)} +comparisons = 0 < value <= 10 != other +bits = (~value & mask) | (value ^ mask) << 2 >> 1 +matrix = left @ right +power = base ** exponent // divisor / ratio +''' + + +def test_python312_surface_affixiates_from_letters_through_module() -> None: + receipt = affixiate_python_source(PYTHON_312_SURFACE, source_id="fixtures/python312.py") + assert receipt.standing == "implemented-candidate" + assert reconstruct_source(receipt) == PYTHON_312_SURFACE + replay_python_affixiation(receipt) + relations = {gonol.relation.kind for gonol in receipt.gonols} + expected = { + "python.grammar.Module", + "python.grammar.TypeAlias", + "python.grammar.TypeVar", + "python.grammar.ClassDef", + "python.grammar.FunctionDef", + "python.grammar.AsyncFunctionDef", + "python.grammar.AsyncFor", + "python.grammar.AsyncWith", + "python.grammar.Await", + "python.grammar.Yield", + "python.grammar.YieldFrom", + "python.grammar.Try", + "python.grammar.TryStar", + "python.grammar.Match", + "python.grammar.MatchMapping", + "python.grammar.MatchSequence", + "python.grammar.MatchClass", + "python.grammar.MatchOr", + "python.grammar.NamedExpr", + "python.grammar.JoinedStr", + "python.grammar.FormattedValue", + "python.grammar.ListComp", + "python.grammar.DictComp", + "python.grammar.Set", + "python.grammar.GeneratorExp", + "python.grammar.comprehension", + "python.grammar.Lambda", + "python.grammar.arguments", + "python.grammar.keyword", + "python.grammar.alias", + } + assert expected <= relations + assert any(gonol.relation.kind == "python.delimiter.parentheses" for gonol in receipt.gonols) + assert any(gonol.relation.kind == "python.delimiter.brackets" for gonol in receipt.gonols) + assert any(gonol.relation.kind == "python.delimiter.braces" for gonol in receipt.gonols) + decorated_class = next( + gonol for gonol in receipt.gonols if gonol.relation.kind == "python.grammar.ClassDef" + ) + assert decorated_class.span.start == PYTHON_312_SURFACE.index("@decorate") + + +def test_spanless_grammar_relations_remain_inside_the_source_parent() -> None: + source = "# type: ignore[index]\ndef empty():\n pass\n" + receipt = affixiate_python_source(source, source_id="fixtures/spanless-relations.py") + module = receipt.gonols[-1] + function = next( + gonol for gonol in receipt.gonols if gonol.relation.kind == "python.grammar.FunctionDef" + ) + module_properties = dict(module.relation.properties) + function_properties = dict(function.relation.properties) + assert '"grammar_construct":"TypeIgnore"' in module_properties[ + "grammar_field.type_ignores[0].spanless_relation" + ] + assert '"grammar_construct":"arguments"' in function_properties[ + "grammar_field.args.spanless_relation" + ] + replay_python_affixiation(receipt) diff --git a/stack-manifest.json b/stack-manifest.json index e205464..197962f 100644 --- a/stack-manifest.json +++ b/stack-manifest.json @@ -1,7 +1,7 @@ { "schema": "the-interdependency.stack-manifest", "version": "1.1.0", - "work_graph_sha256": "9ab3b3f75a32f5f73b5df68419148181fc632593babe4ec6adf4269d4f35badb", + "work_graph_sha256": "482e9a4f70c18ad4888d1fab32e44a6ab550fe7486fabd2f1253876c266212eb", "repositories": [ { "repository": "The-Interdependency/skill-lib", @@ -55,6 +55,7 @@ "hmmm": [ "ucns has no LICENSE file at snapshot commit 828c0b8", "epac exists independently at d8868858b2e455381ce670797bdbe47189bdc496, but clean install, license, stable release, downstream reconsumption, and authority-transition receipt remain incomplete; libs/epac/ stays unpopulated until graduation", + "Python Gonol Construction remains stack-local research without independent repository or release authority", "skill-lib remains a special operational snapshot at stack root rather than following the libs/research pair" ] }, @@ -131,6 +132,15 @@ "canonical_release": false, "authority_transfer": false }, + { + "workspace": "research/python-gonol/", + "participant_id": "python-gonol", + "repository": "The-Interdependency/stack", + "commit": "0e8384bbb60e4c2189016a212bdd0030d04aed7d", + "relation": "stack-local bottom-up Python 3.12 source gonol construction; applies METAPAT affixiation semantics, consumes optional UCNS geometry, and transfers no language authority to UCNS or EDCM", + "canonical_release": false, + "authority_transfer": false + }, { "workspace": "research/ucns/", "participant_id": "ucns-source-base", From 4320f0b3932a9e74415c8b77eaf0c9f8de3b3688 Mon Sep 17 00:00:00 2001 From: Erin Spencer Date: Sat, 12 Sep 2026 16:47:51 -0700 Subject: [PATCH 02/10] refactor(python-gonol): preserve parser engine as private recognition witness --- .../python-gonol/python_gonol/_recognition.py | 1120 +++++++++++++++++ 1 file changed, 1120 insertions(+) create mode 100644 research/python-gonol/python_gonol/_recognition.py diff --git a/research/python-gonol/python_gonol/_recognition.py b/research/python-gonol/python_gonol/_recognition.py new file mode 100644 index 0000000..d812fe1 --- /dev/null +++ b/research/python-gonol/python_gonol/_recognition.py @@ -0,0 +1,1120 @@ +"""Construct Python 3.12 gonols from exact source occurrences, bottom up. + +Usage guidance +-------------- +Use :func:`affixiate_python_bytes` for files so the encoding declaration, BOM, +and exact original bytes remain receipt-bound. Use +:func:`affixiate_python_source` for an already-decoded source string:: + + receipt = affixiate_python_source("answer = f'{40 + 2}'\n", source_id="demo.py") + assert receipt.standing == "implemented-candidate" + assert replay_python_affixiation(receipt).receipt_digest == receipt.receipt_digest + +The constructor always starts with one letter gonol per decoded Unicode scalar +occurrence. Here ``letter`` names the admitted source floor, including spaces, +newlines, punctuation, and digits; it is not an alphabetic subclass. Tokenizer +and AST values are recognition witnesses only. No token, AST node, code object, +or compiler object is stored as a gonol or used in place of source participants. +""" + +# === MODULE_BUILD === +# id: python_gonol_affixiation +# module_name: python_gonol.affixiation +# module_kind: engine +# summary: affixiates exact Python 3.12 source from letter occurrences through lexical, delimiter, and recursive grammar gonols without substituting parser objects for construction +# owner: Python Gonol Construction (stack-local research) +# public_surface: affixiate_python_source, affixiate_python_bytes, replay_python_affixiation, reconstruct_source, PythonGonolConstructionError +# internal_surface: _SourceIndex, _Registry, _lexical_floor, _delimiter_gonols, _grammar_root +# auth_boundary: Python Gonol Construction owns source admission and Python relation construction; METAPAT owns affixiation semantics; UCNS owns geometry +# storage_boundary: none; caller-owned bytes and receipts remain in memory +# network_boundary: none +# user_data_boundary: reads caller-supplied source into an explicit caller-owned receipt and transmits nothing +# admin_only: false +# tests: tests.test_affixiation, tests.test_python312_surface +# rollout: explicit Python 3.12 stack-local candidate +# rollback: remove the python-gonol workspace before downstream binding +# requires: Python 3.12 standard library; optional explicit UCNS Public Gonol authority +# since: 2026-09-12 +# unresolved: exact UCNS affixiation geometry and later Python language profiles remain hmmm +# === END MODULE_BUILD === + +# === CAPABILITIES === +# id: python_source_bottom_up_affixiation +# summary: constructs a lossless addressable gonol registry from every decoded Python source occurrence upward +# exposes: python_gonol.affixiation.affixiate_python_source, python_gonol.affixiation.affixiate_python_bytes +# inputs: exact source text or bytes, source_id, optional explicit UCNS geometry authority +# outputs: PythonAffixiationReceipt +# boundaries: auth:none, storage:none, network:none, user_data:caller-owned receipt +# owner: Python Gonol Construction (stack-local research) +# === END CAPABILITIES === + +# === DOCS === +# id: python_gonol_affixiation_boundary_docs +# summary: explains the bottom-up construction order, parser-witness boundary, replay contract, and unresolved geometry +# audience: developer +# source: docs/PYTHON_AFFIXIATION_BOUNDARY.md +# covers: affixiate_python_source, affixiate_python_bytes, replay_python_affixiation +# status: current +# === END DOCS === + +# === BOUNDARIES === +# id: python_gonol_affixiation_runtime_boundary +# summary: reads caller-owned Python source in memory, performs no execution or network access, and optionally observes an explicitly supplied matching UCNS carrier +# auth_boundary: none +# storage_boundary: none +# network_boundary: none +# user_data_boundary: read +# admin_only: false +# pii: possible +# secrets: none +# owner: Python Gonol Construction (stack-local research) +# === END BOUNDARIES === + +# === CONTRACTS === +# id: every_python_source_occurrence_closes_first +# given: any admitted decoded Python source string, including repeated whitespace or punctuation +# then: exactly one individually addressable letter gonol closes for every Unicode scalar occurrence in exact source order +# class: construction +# +# id: python_lexical_forms_affixiate_letters +# given: the Python 3.12 lexical witness recognizes a nonempty form or an inter-token source gap +# then: one lexical gonol closes over the exact ordered letter gonols covering that form without normalization or omission +# class: construction +# +# id: python_constructions_affixiate_closed_gonols +# given: delimiter and Python grammar relations are recognized after the lexical floor closes +# then: every larger construction references already-closed gonols atomically and carries its constitutive relation, order, roles, multiplicity, source span, and provenance inside its identity +# class: construction +# +# id: python_affixiation_is_lossless_and_replayable +# given: a completed Python affixiation receipt +# then: exact decoded source reconstructs from letter gonols and every gonol plus the receipt digest verifies deterministically +# class: replay +# +# id: unresolved_python_source_remains_hmmm +# given: source has an unmatched delimiter, tokenizer failure, or Python 3.12 grammar error +# then: admitted lower gonols remain preserved beneath a source root whose standing and exact unresolved boundary are hmmm +# class: safety + +# id: parser_objects_never_become_gonols +# given: tokenizer and AST recognition witnesses are used +# then: receipt gonols contain only source-built relation records and closed-gonol references, never TokenInfo, AST, code, or compiler objects +# class: boundary +# +# id: python_gonol_geometry_binding_fails_closed +# given: an explicit UCNS Public Gonol authority is supplied +# then: exact carrier positions are observed only when the carrier and declared digest match the pinned identity; function operations remain hmmm +# class: boundary +# === END CONTRACTS === + +from __future__ import annotations + +import ast +import base64 +from dataclasses import dataclass, replace +from hashlib import sha256 +import io +import json +import keyword +import platform +import sys +import token as token_module +import tokenize +from typing import Any, Iterable, Mapping, Sequence + +from .model import ( + AffixiationRelation, + ClosedGonol, + PythonAffixiationReceipt, + RelationMember, + SourceSpan, + canonical_json_bytes, +) + + +SCHEMA = "the-interdependency.python-gonol-affixiation" +SCHEMA_VERSION = "1.0.0" +CONSTRUCTOR_ID = "python-gonol.affixiation" +CONSTRUCTOR_VERSION = "0.1.0" +LANGUAGE_PROFILE = "python-3.12-file-input" +PINNED_PUBLIC_GONOL_SHA256 = ( + "55d10c84529a4d7bc7714786357e977b68d9df2ac3f73d20e229580b552c2ef5" +) +STANDING = "implemented-candidate" +SELECTION_EFFECT = "none" +RELATION_AUTHORITY = ( + "METAPAT affixiation semantics; Python 3.12 source relation witnessed by CPython; " + "UCNS geometry not implied" +) + +NONCLAIMS = ( + "not an AST, token stream, code object, or compiler-object representation", + "not execution or behavioral equivalence of the source", + "not selected UCNS geometry or a UCNS affixiation/coupling law", + "not EDCM measurement validity", + "not canon or independent release authority", +) + +BASE_HMMM = ( + "exact UCNS geometric operation of Public Gonol function positions", + "exact UCNS Mobius-carrier affixiation/coupling law", + "Python language profiles after Python 3.12 file input", +) + + +class PythonGonolConstructionError(RuntimeError): + """Raised when a receipt or explicit authority fails closed.""" + + +@dataclass(frozen=True, slots=True) +class _LexicalWitness: + start: int + end: int + token_name: str + exact_name: str + spelling: str + gonol: ClosedGonol + + +@dataclass(frozen=True, slots=True) +class _MemberCandidate: + role: str + gonol: ClosedGonol + priority: int + + +class _SourceIndex: + def __init__(self, source: str) -> None: + self.source = source + starts = [0] + for index, character in enumerate(source): + if character == "\n": + starts.append(index + 1) + self.line_starts = tuple(starts) + + def token_offset(self, position: tuple[int, int]) -> int: + row, column = position + if row < 1: + raise PythonGonolConstructionError(f"invalid tokenizer row: {row}") + if row > len(self.line_starts): + if row == len(self.line_starts) + 1 and column == 0: + return len(self.source) + raise PythonGonolConstructionError(f"tokenizer position outside source: {position!r}") + offset = self.line_starts[row - 1] + column + if not 0 <= offset <= len(self.source): + raise PythonGonolConstructionError(f"tokenizer position outside source: {position!r}") + return offset + + def ast_offset(self, row: int, utf8_column: int) -> int: + if row < 1 or row > len(self.line_starts): + raise PythonGonolConstructionError( + f"AST position outside decoded source: {(row, utf8_column)!r}" + ) + start = self.line_starts[row - 1] + end = self.line_starts[row] if row < len(self.line_starts) else len(self.source) + line = self.source[start:end] + byte_count = 0 + for char_column, character in enumerate(line): + if byte_count == utf8_column: + return start + char_column + byte_count += len(character.encode("utf-8")) + if byte_count > utf8_column: + break + if byte_count == utf8_column: + return end + raise PythonGonolConstructionError( + f"AST UTF-8 column does not align to a source scalar: {(row, utf8_column)!r}" + ) + + def line_column(self, offset: int) -> tuple[int, int]: + if not 0 <= offset <= len(self.source): + raise PythonGonolConstructionError(f"source offset outside source: {offset}") + low = 0 + high = len(self.line_starts) + while low + 1 < high: + middle = (low + high) // 2 + if self.line_starts[middle] <= offset: + low = middle + else: + high = middle + return (low + 1, offset - self.line_starts[low]) + + def span(self, start: int, end: int) -> SourceSpan: + if not 0 <= start <= end <= len(self.source): + raise PythonGonolConstructionError(f"invalid source span: {(start, end)!r}") + start_line, start_column = self.line_column(start) + end_line, end_column = self.line_column(end) + return SourceSpan(start, end, start_line, start_column, end_line, end_column) + + +class _Registry: + def __init__(self, source_id: str, source_index: _SourceIndex) -> None: + self.source_id = source_id + self.source_index = source_index + self.values: list[ClosedGonol] = [] + self.by_id: dict[str, ClosedGonol] = {} + self.by_address: dict[str, ClosedGonol] = {} + + def close( + self, + *, + address: str, + scale: str, + start: int, + end: int, + relation_kind: str, + candidates: Sequence[_MemberCandidate] = (), + properties: Sequence[tuple[str, str]] = (), + provenance: Sequence[tuple[str, str]] = (), + hmmm: Sequence[str] = (), + ) -> ClosedGonol: + if address in self.by_address: + raise PythonGonolConstructionError(f"duplicate gonol address: {address}") + members: list[RelationMember] = [] + for ordinal, candidate in enumerate(candidates): + child = self.by_id.get(candidate.gonol.gonol_id) + if child is None or child.address != candidate.gonol.address: + raise PythonGonolConstructionError( + "larger construction may reference only an already-closed gonol" + ) + members.append( + RelationMember( + ordinal=ordinal, + role=candidate.role, + gonol_id=child.gonol_id, + address=child.address, + ) + ) + relation = AffixiationRelation( + kind=relation_kind, + members=tuple(members), + properties=tuple((str(key), str(value)) for key, value in properties), + authority=RELATION_AUTHORITY, + ) + provisional = ClosedGonol( + address=address, + scale=scale, + span=self.source_index.span(start, end), + relation=relation, + provenance=( + ("constructor", f"{CONSTRUCTOR_ID}/{CONSTRUCTOR_VERSION}"), + ("language_profile", LANGUAGE_PROFILE), + ("source_id", self.source_id), + ) + + tuple((str(key), str(value)) for key, value in provenance), + hmmm=tuple(str(item) for item in hmmm), + gonol_id="", + ) + gonol_id = sha256(canonical_json_bytes(provisional.identity_payload())).hexdigest() + value = replace(provisional, gonol_id=gonol_id) + if gonol_id in self.by_id: + raise PythonGonolConstructionError( + "closed gonol identity collision; occurrence address failed to distinguish values" + ) + self.values.append(value) + self.by_id[gonol_id] = value + self.by_address[address] = value + return value + + +def _require_source(source: str, source_id: str) -> None: + if not isinstance(source, str): + raise TypeError("source must be an exact decoded Unicode string") + if not isinstance(source_id, str) or not source_id: + raise TypeError("source_id must be exact non-empty text") + for field, value in (("source", source), ("source_id", source_id)): + for character in value: + if 0xD800 <= ord(character) <= 0xDFFF: + raise PythonGonolConstructionError(f"{field} contains a surrogate code point") + + +def _public_gonol_positions(authority: Any | None) -> tuple[dict[str, int] | None, str]: + if authority is None: + return None, "not-supplied" + if isinstance(authority, Mapping): + carrier_value = authority.get("PUBLIC_GONOL_157") + declared_digest = authority.get("PUBLIC_GONOL_SHA256") + else: + carrier_value = getattr(authority, "PUBLIC_GONOL_157", None) + declared_digest = getattr(authority, "PUBLIC_GONOL_SHA256", None) + if not isinstance(carrier_value, Sequence) or isinstance(carrier_value, (str, bytes)): + raise PythonGonolConstructionError( + "explicit UCNS authority must expose PUBLIC_GONOL_157" + ) + carrier = tuple(carrier_value) + if len(carrier) != 157 or len(set(carrier)) != 157: + raise PythonGonolConstructionError( + "explicit UCNS Public Gonol carrier must contain 157 unique positions" + ) + if any(not isinstance(item, str) or len(item) != 1 for item in carrier): + raise PythonGonolConstructionError( + "explicit UCNS Public Gonol positions must be one Unicode scalar each" + ) + computed = sha256( + json.dumps(carrier, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + ).hexdigest() + if declared_digest != computed or computed != PINNED_PUBLIC_GONOL_SHA256: + raise PythonGonolConstructionError( + "explicit UCNS Public Gonol authority digest does not match the pinned carrier" + ) + name = ( + str(authority.get("authority_name", "mapping")) + if isinstance(authority, Mapping) + else str(getattr(authority, "__name__", authority.__class__.__name__)) + ) + return {glyph: index for index, glyph in enumerate(carrier)}, name + + +def _letter_floor( + source: str, + source_id: str, + source_index: _SourceIndex, + registry: _Registry, + positions: dict[str, int] | None, + geometry_name: str, +) -> tuple[ClosedGonol, ...]: + letters: list[ClosedGonol] = [] + for index, character in enumerate(source): + address = f"{source_id}#letter:{index}" + if positions is None: + position = "hmmm:not-supplied" + else: + found = positions.get(character) + position = "hmmm:not-on-pinned-carrier" if found is None else str(found) + letters.append( + registry.close( + address=address, + scale="letter", + start=index, + end=index + 1, + relation_kind="python.source.letter-occurrence", + properties=( + ("unicode_scalar", character), + ("code_point", f"U+{ord(character):04X}"), + ("occurrence", str(index)), + ("public_gonol_position", position), + ("public_gonol_function", "hmmm"), + ), + provenance=(("geometry_authority", geometry_name),), + ) + ) + return tuple(letters) + + +def _token_relation(token_type: int, spelling: str) -> tuple[str, str]: + token_name = token_module.tok_name[token_type] + if token_type == token_module.NAME and keyword.iskeyword(spelling): + return "KEYWORD", token_name + if token_type == token_module.OP: + return token_module.tok_name.get(tokenize.EXACT_TOKEN_TYPES.get(spelling, token_type), "OP"), token_name + return token_name, token_name + + +def _lexical_floor( + source: str, + source_id: str, + source_index: _SourceIndex, + registry: _Registry, + letters: Sequence[ClosedGonol], +) -> tuple[tuple[_LexicalWitness, ...], tuple[str, ...], tuple[str, ...]]: + raw_tokens: list[tuple[int, int, str, str, str]] = [] + zero_width: list[str] = [] + unresolved: list[str] = [] + stream = tokenize.generate_tokens(io.StringIO(source).readline) + while True: + try: + item = next(stream) + except StopIteration: + break + except (tokenize.TokenError, IndentationError, SyntaxError) as exc: + detail = exc.args[0] if exc.args else exc.__class__.__name__ + location = exc.args[1] if len(exc.args) > 1 else None + unresolved.append(f"tokenizer: {detail}; location={location!r}") + break + exact_name, token_name = _token_relation(item.type, item.string) + if item.string == "": + if token_name not in {"ENDMARKER", "ENCODING"}: + zero_width.append( + f"{token_name}@{source_index.token_offset(item.start)}" + ) + continue + start = source_index.token_offset(item.start) + end = source_index.token_offset(item.end) + if start == end: + if token_name not in {"ENDMARKER", "ENCODING"}: + zero_width.append(f"{token_name}@{start}") + continue + spelling = source[start:end] + if spelling != item.string: + unresolved.append( + f"tokenizer spelling mismatch at {start}:{end}; witness={item.string!r} source={spelling!r}" + ) + raw_tokens.append((start, end, token_name, exact_name, spelling)) + + raw_tokens.sort(key=lambda value: (value[0], value[1])) + previous = 0 + complete: list[tuple[int, int, str, str, str]] = [] + for start, end, token_name, exact_name, spelling in raw_tokens: + if start < previous: + unresolved.append(f"overlapping lexical witnesses at decoded offset {start}") + continue + if start > previous: + complete.append((previous, start, "INTERTOKEN", "INTERTOKEN", source[previous:start])) + complete.append((start, end, token_name, exact_name, spelling)) + previous = end + if previous < len(source): + complete.append((previous, len(source), "INTERTOKEN", "INTERTOKEN", source[previous:])) + + witnesses: list[_LexicalWitness] = [] + for lexical_index, (start, end, token_name, exact_name, spelling) in enumerate(complete): + candidates = tuple( + _MemberCandidate(role=f"letter[{offset - start}]", gonol=letters[offset], priority=3) + for offset in range(start, end) + ) + address = f"{source_id}#lexical:{lexical_index}:{start}-{end}" + gonol = registry.close( + address=address, + scale="lexical-form", + start=start, + end=end, + relation_kind=f"python.lexical.{exact_name}", + candidates=candidates, + properties=( + ("tokenizer_class", token_name), + ("exact_form", exact_name), + ("source_length", str(end - start)), + ), + provenance=(("recognition_witness", "stdlib.tokenize/python-3.12"),), + ) + witnesses.append(_LexicalWitness(start, end, token_name, exact_name, spelling, gonol)) + return tuple(witnesses), tuple(zero_width), tuple(unresolved) + + +def _member_sort_key(candidate: _MemberCandidate) -> tuple[int, int, int, str, str]: + span = candidate.gonol.span + return (span.start, candidate.priority, -span.end, candidate.role, candidate.gonol.address) + + +def _maximal_enclosures( + enclosures: Sequence[ClosedGonol], + start: int, + end: int, + *, + excluded: Sequence[tuple[int, int]] = (), +) -> tuple[ClosedGonol, ...]: + eligible = [ + item + for item in enclosures + if start <= item.span.start + and item.span.end <= end + and not any(left <= item.span.start and item.span.end <= right for left, right in excluded) + ] + selected: list[ClosedGonol] = [] + for item in sorted(eligible, key=lambda value: (value.span.start, -value.span.end)): + if any( + parent.span.start <= item.span.start and item.span.end <= parent.span.end + for parent in selected + ): + continue + selected.append(item) + return tuple(selected) + + +def _delimiter_gonols( + source_id: str, + registry: _Registry, + letters: Sequence[ClosedGonol], + lexicals: Sequence[_LexicalWitness], +) -> tuple[tuple[ClosedGonol, ...], tuple[str, ...]]: + openers = {"(": ")", "[": "]", "{": "}"} + names = {"(": "parentheses", "[": "brackets", "{": "braces"} + stack: list[_LexicalWitness] = [] + pairs: list[tuple[_LexicalWitness, _LexicalWitness]] = [] + unresolved: list[str] = [] + for item in lexicals: + if item.spelling in openers and item.exact_name in {"LPAR", "LSQB", "LBRACE"}: + stack.append(item) + elif item.spelling in {")", "]", "}"} and item.exact_name in {"RPAR", "RSQB", "RBRACE"}: + if not stack: + unresolved.append(f"unmatched closing delimiter {item.spelling!r} at {item.start}") + continue + opener = stack.pop() + if openers[opener.spelling] != item.spelling: + unresolved.append( + f"delimiter mismatch {opener.spelling!r}@{opener.start} with {item.spelling!r}@{item.start}" + ) + continue + pairs.append((opener, item)) + for opener in stack: + unresolved.append(f"unmatched opening delimiter {opener.spelling!r} at {opener.start}") + + built: list[ClosedGonol] = [] + for pair_index, (opener, closer) in enumerate( + sorted(pairs, key=lambda pair: (pair[1].end - pair[0].start, pair[0].start)) + ): + start, end = opener.start, closer.end + nested = _maximal_enclosures(built, start, end) + nested_ranges = tuple((item.span.start, item.span.end) for item in nested) + candidates: list[_MemberCandidate] = [] + for item in lexicals: + if not (start <= item.start and item.end <= end): + continue + if any(left <= item.start and item.end <= right for left, right in nested_ranges): + continue + role = "opener" if item is opener else "closer" if item is closer else "content" + candidates.append(_MemberCandidate(role, item.gonol, 2)) + candidates.extend(_MemberCandidate("nested", item, 1) for item in nested) + candidates.sort(key=_member_sort_key) + counter = 0 + normalized: list[_MemberCandidate] = [] + for candidate in candidates: + if candidate.role in {"content", "nested"}: + normalized.append(replace(candidate, role=f"content[{counter}]")) + counter += 1 + else: + normalized.append(candidate) + built.append( + registry.close( + address=f"{source_id}#delimiter:{pair_index}:{start}-{end}", + scale="delimiter-construction", + start=start, + end=end, + relation_kind=f"python.delimiter.{names[opener.spelling]}", + candidates=normalized, + properties=( + ("opener", opener.spelling), + ("closer", closer.spelling), + ("closure", "matched"), + ), + provenance=(("recognition_witness", "python-3.12 delimiter stack"),), + hmmm=("exact UCNS delimiter relation geometry",), + ) + ) + return tuple(built), tuple(unresolved) + + +def _primitive_properties(node: ast.AST) -> tuple[tuple[str, str], ...]: + properties: list[tuple[str, str]] = [] + skipped_values = { + ("Constant", "value"), + ("MatchSingleton", "value"), + } + for field, value in ast.iter_fields(node): + if field == "ctx" or (node.__class__.__name__, field) in skipped_values: + continue + if isinstance(value, ast.AST): + if isinstance(value, (ast.operator, ast.unaryop, ast.boolop, ast.cmpop)): + properties.append((f"grammar_field.{field}", value.__class__.__name__)) + continue + if isinstance(value, list): + if value and all(isinstance(item, str) for item in value): + properties.append((f"grammar_field.{field}", json.dumps(value, ensure_ascii=False))) + continue + if value is None: + continue + if isinstance(value, (str, int, bool)): + properties.append((f"grammar_field.{field}", json.dumps(value, ensure_ascii=False))) + return tuple(properties) + + +def _spanless_relation_property(field: str, node: ast.AST) -> tuple[str, str]: + """Keep a witnessed child relation inside its source-built parent. + + A few CPython grammar records, notably an empty ``arguments`` value and + ``TypeIgnore``, have no independent source span. They cannot honestly + close as gonols of their own. Their relation is therefore recorded on the + parent that already owns the exact surface gonols; the parser record itself + is discarded. + """ + + descriptor: dict[str, Any] = {"grammar_construct": node.__class__.__name__} + for name, value in ast.iter_fields(node): + if isinstance(value, (str, int, bool)) or value is None: + descriptor[name] = value + elif isinstance(value, list) and not any(isinstance(item, ast.AST) for item in value): + descriptor[name] = value + return ( + f"grammar_field.{field}.spanless_relation", + json.dumps(descriptor, ensure_ascii=False, sort_keys=True, separators=(",", ":")), + ) + + +def _grammar_root( + source: str, + source_id: str, + source_index: _SourceIndex, + registry: _Registry, + letters: Sequence[ClosedGonol], + lexicals: Sequence[_LexicalWitness], + enclosures: Sequence[ClosedGonol], + zero_width: Sequence[str], +) -> tuple[ClosedGonol | None, tuple[str, ...]]: + try: + tree = ast.parse( + source, + filename=source_id, + mode="exec", + type_comments=True, + feature_version=(3, 12), + ) + except (SyntaxError, ValueError, TypeError, MemoryError) as exc: + if isinstance(exc, SyntaxError): + detail = f"grammar: {exc.msg}; line={exc.lineno!r}; offset={exc.offset!r}" + else: + detail = f"grammar: {exc.__class__.__name__}: {exc}" + return None, (detail,) + + node_order = {id(node): index for index, node in enumerate(ast.walk(tree))} + span_memo: dict[int, tuple[int, int] | None] = {} + + def node_span(node: ast.AST) -> tuple[int, int] | None: + key = id(node) + if key in span_memo: + return span_memo[key] + spans: list[tuple[int, int]] = [] + if all(hasattr(node, name) for name in ("lineno", "col_offset", "end_lineno", "end_col_offset")): + end_line = getattr(node, "end_lineno", None) + end_column = getattr(node, "end_col_offset", None) + if end_line is not None and end_column is not None: + spans.append( + ( + source_index.ast_offset(int(node.lineno), int(node.col_offset)), + source_index.ast_offset(int(end_line), int(end_column)), + ) + ) + for child in ast.iter_child_nodes(node): + child_span = node_span(child) + if child_span is not None: + spans.append(child_span) + if isinstance(node, ast.Module): + result: tuple[int, int] | None = (0, len(source)) + elif spans: + result = (min(item[0] for item in spans), max(item[1] for item in spans)) + else: + result = None + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)) and node.decorator_list: + first_decorator = min(node.decorator_list, key=lambda item: (item.lineno, item.col_offset)) + decorator_start = source_index.ast_offset( + int(first_decorator.lineno), int(first_decorator.col_offset) + ) + marker = max( + ( + item + for item in lexicals + if item.spelling == "@" + and item.exact_name == "AT" + and item.end <= decorator_start + and source_index.line_column(item.start)[0] == int(first_decorator.lineno) + ), + key=lambda item: item.start, + default=None, + ) + if marker is not None and result is not None: + result = (min(marker.start, result[0]), result[1]) + span_memo[key] = result + return result + + node_span(tree) + built: dict[int, ClosedGonol] = {} + + def construct(node: ast.AST) -> ClosedGonol | None: + key = id(node) + if key in built: + return built[key] + span = node_span(node) + if span is None: + return None + start, end = span + child_candidates: list[_MemberCandidate] = [] + child_ranges: list[tuple[int, int]] = [] + spanless_properties: list[tuple[str, str]] = [] + for field, value in ast.iter_fields(node): + if isinstance(value, ast.AST): + child = construct(value) + if child is not None: + child_candidates.append(_MemberCandidate(field, child, 0)) + child_ranges.append((child.span.start, child.span.end)) + elif not isinstance( + value, + (ast.operator, ast.unaryop, ast.boolop, ast.cmpop, ast.expr_context), + ): + spanless_properties.append(_spanless_relation_property(field, value)) + elif isinstance(value, list): + for position, item in enumerate(value): + if not isinstance(item, ast.AST): + continue + child = construct(item) + if child is not None: + child_candidates.append(_MemberCandidate(f"{field}[{position}]", child, 0)) + child_ranges.append((child.span.start, child.span.end)) + elif not isinstance( + item, + (ast.operator, ast.unaryop, ast.boolop, ast.cmpop, ast.expr_context), + ): + spanless_properties.append( + _spanless_relation_property(f"{field}[{position}]", item) + ) + + selected_enclosures = _maximal_enclosures( + enclosures, + start, + end, + excluded=child_ranges, + ) + enclosure_ranges = tuple((item.span.start, item.span.end) for item in selected_enclosures) + surface_candidates: list[_MemberCandidate] = [ + _MemberCandidate("surface", item, 1) for item in selected_enclosures + ] + for lexical in lexicals: + if not (start <= lexical.start and lexical.end <= end): + continue + if any(left <= lexical.start and lexical.end <= right for left, right in child_ranges): + continue + if any(left <= lexical.start and lexical.end <= right for left, right in enclosure_ranges): + continue + surface_candidates.append(_MemberCandidate("surface", lexical.gonol, 2)) + + candidates = child_candidates + surface_candidates + candidates.sort(key=_member_sort_key) + surface_index = 0 + normalized: list[_MemberCandidate] = [] + for candidate in candidates: + if candidate.role == "surface": + normalized.append(replace(candidate, role=f"surface[{surface_index}]")) + surface_index += 1 + else: + normalized.append(candidate) + properties = [ + ("grammar_profile", LANGUAGE_PROFILE), + ("grammar_construct", node.__class__.__name__), + ] + properties.extend(_primitive_properties(node)) + properties.extend(spanless_properties) + if isinstance(node, ast.Module): + properties.extend( + (f"zero_width_witness[{index}]", item) + for index, item in enumerate(zero_width) + ) + value = registry.close( + address=( + f"{source_id}#grammar:{node_order[key]}:{node.__class__.__name__}:{start}-{end}" + ), + scale="module" if isinstance(node, ast.Module) else "python-construction", + start=start, + end=end, + relation_kind=f"python.grammar.{node.__class__.__name__}", + candidates=normalized, + properties=properties, + provenance=(("recognition_witness", "stdlib.ast/python-3.12"),), + hmmm=("exact UCNS geometry for this Python relation",), + ) + built[key] = value + return value + + return construct(tree), () + + +def _source_root_hmmm( + source_id: str, + registry: _Registry, + letters: Sequence[ClosedGonol], + lexicals: Sequence[_LexicalWitness], + enclosures: Sequence[ClosedGonol], + unresolved: Sequence[str], + zero_width: Sequence[str], +) -> ClosedGonol: + selected = _maximal_enclosures(enclosures, 0, len(letters)) + selected_ranges = tuple((item.span.start, item.span.end) for item in selected) + candidates: list[_MemberCandidate] = [ + _MemberCandidate("closed-delimiter", item, 1) for item in selected + ] + for item in lexicals: + if any(left <= item.start and item.end <= right for left, right in selected_ranges): + continue + candidates.append(_MemberCandidate("lexical-form", item.gonol, 2)) + candidates.sort(key=_member_sort_key) + properties = [("standing", "hmmm")] + properties.extend((f"unresolved[{index}]", item) for index, item in enumerate(unresolved)) + properties.extend( + (f"zero_width_witness[{index}]", item) for index, item in enumerate(zero_width) + ) + return registry.close( + address=f"{source_id}#source:hmmm", + scale="source", + start=0, + end=len(letters), + relation_kind="python.source.hmmm", + candidates=candidates, + properties=properties, + provenance=(("recognition_witness", "partial Python 3.12 recognition"),), + hmmm=unresolved, + ) + + +def _receipt_digest(receipt: PythonAffixiationReceipt) -> str: + return sha256(canonical_json_bytes(receipt.payload())).hexdigest() + + +def _construct( + source: str, + *, + source_id: str, + source_bytes: bytes, + encoding: str, + geometry_authority: Any | None, +) -> PythonAffixiationReceipt: + _require_source(source, source_id) + if sys.version_info[:2] != (3, 12): + raise PythonGonolConstructionError( + f"{LANGUAGE_PROFILE} requires a Python 3.12 recognition witness; got {sys.version_info.major}.{sys.version_info.minor}" + ) + source_index = _SourceIndex(source) + registry = _Registry(source_id, source_index) + positions, geometry_name = _public_gonol_positions(geometry_authority) + letters = _letter_floor( + source, + source_id, + source_index, + registry, + positions, + geometry_name, + ) + lexicals, zero_width, lexical_hmmm = _lexical_floor( + source, + source_id, + source_index, + registry, + letters, + ) + enclosures, delimiter_hmmm = _delimiter_gonols( + source_id, + registry, + letters, + lexicals, + ) + grammar_root, grammar_hmmm = _grammar_root( + source, + source_id, + source_index, + registry, + letters, + lexicals, + enclosures, + zero_width, + ) + syntax_hmmm = lexical_hmmm + delimiter_hmmm + grammar_hmmm + if grammar_root is None or syntax_hmmm: + root = _source_root_hmmm( + source_id, + registry, + letters, + lexicals, + enclosures, + syntax_hmmm or ("grammar root was not constructed",), + zero_width, + ) + standing = "hmmm" + else: + root = grammar_root + standing = STANDING + + hmmm = list(BASE_HMMM) + if geometry_authority is None: + hmmm.append("UCNS Public Gonol geometry authority was not supplied") + if positions is not None and any(character not in positions for character in source): + hmmm.append("one or more admitted source scalars have no position on the pinned Public Gonol carrier") + hmmm.extend(syntax_hmmm) + runtime = f"{platform.python_implementation()}-{platform.python_version()}" + provisional = PythonAffixiationReceipt( + schema=SCHEMA, + version=SCHEMA_VERSION, + constructor_id=CONSTRUCTOR_ID, + constructor_version=CONSTRUCTOR_VERSION, + language_profile=LANGUAGE_PROFILE, + source_id=source_id, + encoding=encoding, + source_bytes_base64=base64.b64encode(source_bytes).decode("ascii"), + source_bytes_sha256=sha256(source_bytes).hexdigest(), + decoded_source_sha256=sha256(source.encode("utf-8")).hexdigest(), + recognition_witness=runtime, + standing=standing, + selection_effect=SELECTION_EFFECT, + root_gonol_id=root.gonol_id, + gonols=tuple(registry.values), + nonclaims=NONCLAIMS, + hmmm=tuple(hmmm), + receipt_digest="", + ) + return replace(provisional, receipt_digest=_receipt_digest(provisional)) + + +def affixiate_python_source( + source: str, + *, + source_id: str, + geometry_authority: Any | None = None, +) -> PythonAffixiationReceipt: + """Affixiate exact decoded Python 3.12 file-input source from letters upward.""" + + if not isinstance(source, str): + raise TypeError("source must be an exact decoded Unicode string") + return _construct( + source, + source_id=source_id, + source_bytes=source.encode("utf-8"), + encoding="utf-8", + geometry_authority=geometry_authority, + ) + + +def affixiate_python_bytes( + source_bytes: bytes, + *, + source_id: str, + geometry_authority: Any | None = None, +) -> PythonAffixiationReceipt: + """Detect Python's declared encoding, preserve exact bytes, and affixiate source.""" + + if not isinstance(source_bytes, bytes): + raise TypeError("source_bytes must be exact bytes") + try: + encoding, _lines = tokenize.detect_encoding(io.BytesIO(source_bytes).readline) + source = source_bytes.decode(encoding) + except (SyntaxError, UnicodeDecodeError, LookupError) as exc: + raise PythonGonolConstructionError(f"Python source decoding failed: {exc}") from exc + return _construct( + source, + source_id=source_id, + source_bytes=source_bytes, + encoding=encoding, + geometry_authority=geometry_authority, + ) + + +def _property(gonol: ClosedGonol, name: str) -> str: + values = [value for key, value in gonol.relation.properties if key == name] + if len(values) != 1: + raise PythonGonolConstructionError( + f"gonol {gonol.address} must carry exactly one {name!r} property" + ) + return values[0] + + +def reconstruct_source(receipt: PythonAffixiationReceipt) -> str: + """Reconstruct decoded source solely from ordered letter gonols.""" + + letters = sorted( + (gonol for gonol in receipt.gonols if gonol.scale == "letter"), + key=lambda gonol: gonol.span.start, + ) + return "".join(_property(gonol, "unicode_scalar") for gonol in letters) + + +def replay_python_affixiation(receipt: PythonAffixiationReceipt) -> PythonAffixiationReceipt: + """Fail closed unless every visible gonol and receipt identity remains valid.""" + + if not isinstance(receipt, PythonAffixiationReceipt): + raise TypeError("receipt must be a PythonAffixiationReceipt") + expected = (SCHEMA, SCHEMA_VERSION, CONSTRUCTOR_ID, CONSTRUCTOR_VERSION, LANGUAGE_PROFILE) + actual = ( + receipt.schema, + receipt.version, + receipt.constructor_id, + receipt.constructor_version, + receipt.language_profile, + ) + if actual != expected: + raise PythonGonolConstructionError("receipt schema, constructor, or language profile mismatch") + try: + raw = base64.b64decode(receipt.source_bytes_base64, validate=True) + except ValueError as exc: + raise PythonGonolConstructionError("receipt source bytes are not valid base64") from exc + if sha256(raw).hexdigest() != receipt.source_bytes_sha256: + raise PythonGonolConstructionError("receipt source byte digest mismatch") + source = reconstruct_source(receipt) + if sha256(source.encode("utf-8")).hexdigest() != receipt.decoded_source_sha256: + raise PythonGonolConstructionError("decoded source digest mismatch") + try: + decoded = raw.decode(receipt.encoding) + except (UnicodeDecodeError, LookupError) as exc: + raise PythonGonolConstructionError("receipt source bytes no longer decode as declared") from exc + if decoded != source: + raise PythonGonolConstructionError("receipt bytes and letter gonols reconstruct different source") + + source_index = _SourceIndex(source) + known: dict[str, ClosedGonol] = {} + known_addresses: set[str] = set() + expected_letter_start = 0 + for gonol in receipt.gonols: + if gonol.gonol_id in known: + raise PythonGonolConstructionError("duplicate gonol identity in receipt") + if gonol.address in known_addresses: + raise PythonGonolConstructionError("duplicate gonol address in receipt") + expected_id = sha256(canonical_json_bytes(gonol.identity_payload())).hexdigest() + if gonol.gonol_id != expected_id: + raise PythonGonolConstructionError(f"gonol identity mismatch: {gonol.address}") + if gonol.span != source_index.span(gonol.span.start, gonol.span.end): + raise PythonGonolConstructionError(f"source coordinates drifted: {gonol.address}") + if tuple(member.ordinal for member in gonol.relation.members) != tuple( + range(len(gonol.relation.members)) + ): + raise PythonGonolConstructionError(f"member order drifted: {gonol.address}") + member_spans: list[tuple[int, int]] = [] + for member in gonol.relation.members: + child = known.get(member.gonol_id) + if child is None or child.address != member.address: + raise PythonGonolConstructionError( + f"gonol references a child that was not already closed: {gonol.address}" + ) + member_spans.append((child.span.start, child.span.end)) + if gonol.scale == "letter": + if gonol.span.start != expected_letter_start or gonol.span.end != expected_letter_start + 1: + raise PythonGonolConstructionError("letter floor is not contiguous and ordered") + scalar = _property(gonol, "unicode_scalar") + if len(scalar) != 1 or scalar != source[gonol.span.start : gonol.span.end]: + raise PythonGonolConstructionError("letter gonol does not match its source occurrence") + expected_letter_start += 1 + else: + merged: list[list[int]] = [] + for start, end in sorted(member_spans): + if not merged or start > merged[-1][1]: + merged.append([start, end]) + elif end > merged[-1][1]: + merged[-1][1] = end + expected = [] if gonol.span.start == gonol.span.end else [[gonol.span.start, gonol.span.end]] + if merged != expected: + raise PythonGonolConstructionError( + f"gonol span differs from its atomic closed participants: {gonol.address}" + ) + known[gonol.gonol_id] = gonol + known_addresses.add(gonol.address) + if expected_letter_start != len(source): + raise PythonGonolConstructionError("letter floor does not cover complete decoded source") + root = known.get(receipt.root_gonol_id) + if ( + root is None + or not receipt.gonols + or root is not receipt.gonols[-1] + or root.span.start != 0 + or root.span.end != len(source) + or root.scale not in {"module", "source"} + ): + raise PythonGonolConstructionError("receipt root does not cover every source occurrence") + if receipt.nonclaims != NONCLAIMS or not all(item in receipt.hmmm for item in BASE_HMMM): + raise PythonGonolConstructionError("receipt nonclaim or hmmm boundary mismatch") + if receipt.receipt_digest != _receipt_digest(replace(receipt, receipt_digest="")): + raise PythonGonolConstructionError("receipt digest mismatch") + return receipt + + +__all__ = [ + "CONSTRUCTOR_ID", + "CONSTRUCTOR_VERSION", + "LANGUAGE_PROFILE", + "PINNED_PUBLIC_GONOL_SHA256", + "PythonGonolConstructionError", + "affixiate_python_bytes", + "affixiate_python_source", + "reconstruct_source", + "replay_python_affixiation", +] From 829d90c3e3e14fd84d07dd6e59767743843b1c8a Mon Sep 17 00:00:00 2001 From: Erin Spencer Date: Sat, 12 Sep 2026 16:48:31 -0700 Subject: [PATCH 03/10] fix(python-gonol): make character definitions the public source floor --- .../python-gonol/python_gonol/affixiation.py | 1208 +++++------------ 1 file changed, 303 insertions(+), 905 deletions(-) diff --git a/research/python-gonol/python_gonol/affixiation.py b/research/python-gonol/python_gonol/affixiation.py index d812fe1..3265ac2 100644 --- a/research/python-gonol/python_gonol/affixiation.py +++ b/research/python-gonol/python_gonol/affixiation.py @@ -1,30 +1,26 @@ -"""Construct Python 3.12 gonols from exact source occurrences, bottom up. +"""Construct Python 3.12 gonols from exact source characters upward. Usage guidance -------------- -Use :func:`affixiate_python_bytes` for files so the encoding declaration, BOM, -and exact original bytes remain receipt-bound. Use -:func:`affixiate_python_source` for an already-decoded source string:: - - receipt = affixiate_python_source("answer = f'{40 + 2}'\n", source_id="demo.py") - assert receipt.standing == "implemented-candidate" - assert replay_python_affixiation(receipt).receipt_digest == receipt.receipt_digest - -The constructor always starts with one letter gonol per decoded Unicode scalar -occurrence. Here ``letter`` names the admitted source floor, including spaces, -newlines, punctuation, and digits; it is not an alphabetic subclass. Tokenizer -and AST values are recognition witnesses only. No token, AST node, code object, -or compiler object is stored as a gonol or used in place of source participants. +Use :func:`affixiate_python_bytes` for source files so encoding and exact bytes +remain receipt-bound. Use :func:`affixiate_python_source` for already-decoded +text. The public constructor closes one occurrence-specific character gonol per +Unicode scalar, closes that character's applicable definition gonols, and only +then permits the closed characters to participate atomically in lexical and +larger Python constructions. + +CPython ``tokenize`` and ``ast`` remain recognition witnesses only. They never +become gonols and never replace source-built participants. """ # === MODULE_BUILD === # id: python_gonol_affixiation # module_name: python_gonol.affixiation # module_kind: engine -# summary: affixiates exact Python 3.12 source from letter occurrences through lexical, delimiter, and recursive grammar gonols without substituting parser objects for construction +# summary: affixiates exact Python 3.12 source from character occurrences and their definition-spaces through lexical, delimiter, and recursive grammar gonols # owner: Python Gonol Construction (stack-local research) -# public_surface: affixiate_python_source, affixiate_python_bytes, replay_python_affixiation, reconstruct_source, PythonGonolConstructionError -# internal_surface: _SourceIndex, _Registry, _lexical_floor, _delimiter_gonols, _grammar_root +# public_surface: affixiate_python_source, affixiate_python_bytes, replay_python_affixiation, reconstruct_source, grammar_witness_inventory, PythonGonolConstructionError +# internal_surface: _upgrade_recognition_receipt, _character_definitions # auth_boundary: Python Gonol Construction owns source admission and Python relation construction; METAPAT owns affixiation semantics; UCNS owns geometry # storage_boundary: none; caller-owned bytes and receipts remain in memory # network_boundary: none @@ -33,916 +29,272 @@ # tests: tests.test_affixiation, tests.test_python312_surface # rollout: explicit Python 3.12 stack-local candidate # rollback: remove the python-gonol workspace before downstream binding -# requires: Python 3.12 standard library; optional explicit UCNS Public Gonol authority +# requires: Python 3.12 standard library; python_gonol._recognition; optional explicit UCNS Public Gonol authority # since: 2026-09-12 -# unresolved: exact UCNS affixiation geometry and later Python language profiles remain hmmm +# unresolved: exact UCNS affixiation geometry and exhaustive CPython grammar-corpus parity remain hmmm # === END MODULE_BUILD === -# === CAPABILITIES === -# id: python_source_bottom_up_affixiation -# summary: constructs a lossless addressable gonol registry from every decoded Python source occurrence upward -# exposes: python_gonol.affixiation.affixiate_python_source, python_gonol.affixiation.affixiate_python_bytes -# inputs: exact source text or bytes, source_id, optional explicit UCNS geometry authority -# outputs: PythonAffixiationReceipt -# boundaries: auth:none, storage:none, network:none, user_data:caller-owned receipt -# owner: Python Gonol Construction (stack-local research) -# === END CAPABILITIES === - -# === DOCS === -# id: python_gonol_affixiation_boundary_docs -# summary: explains the bottom-up construction order, parser-witness boundary, replay contract, and unresolved geometry -# audience: developer -# source: docs/PYTHON_AFFIXIATION_BOUNDARY.md -# covers: affixiate_python_source, affixiate_python_bytes, replay_python_affixiation -# status: current -# === END DOCS === - -# === BOUNDARIES === -# id: python_gonol_affixiation_runtime_boundary -# summary: reads caller-owned Python source in memory, performs no execution or network access, and optionally observes an explicitly supplied matching UCNS carrier -# auth_boundary: none -# storage_boundary: none -# network_boundary: none -# user_data_boundary: read -# admin_only: false -# pii: possible -# secrets: none -# owner: Python Gonol Construction (stack-local research) -# === END BOUNDARIES === - # === CONTRACTS === -# id: every_python_source_occurrence_closes_first -# given: any admitted decoded Python source string, including repeated whitespace or punctuation -# then: exactly one individually addressable letter gonol closes for every Unicode scalar occurrence in exact source order +# id: every_python_source_character_closes_first +# given: any admitted decoded Python source string +# then: exactly one independently addressable character gonol closes for every Unicode scalar occurrence in exact source order +# class: construction +# +# id: python_character_definitions_share_character_origin +# given: a source character occurrence has applicable Unicode or Python lexical-profile definitions +# then: each definition closes as its own gonol over that already-closed character origin before lexical construction begins # class: construction # -# id: python_lexical_forms_affixiate_letters -# given: the Python 3.12 lexical witness recognizes a nonempty form or an inter-token source gap -# then: one lexical gonol closes over the exact ordered letter gonols covering that form without normalization or omission +# id: python_lexical_forms_affixiate_characters +# given: the Python 3.12 lexical witness recognizes a nonempty form or inter-token source gap +# then: one lexical gonol closes over the exact ordered character gonols covering that form without normalization or omission # class: construction # # id: python_constructions_affixiate_closed_gonols -# given: delimiter and Python grammar relations are recognized after the lexical floor closes +# given: delimiter and Python grammar relations are recognized after the character and lexical floors close # then: every larger construction references already-closed gonols atomically and carries its constitutive relation, order, roles, multiplicity, source span, and provenance inside its identity # class: construction # # id: python_affixiation_is_lossless_and_replayable # given: a completed Python affixiation receipt -# then: exact decoded source reconstructs from letter gonols and every gonol plus the receipt digest verifies deterministically +# then: exact decoded source reconstructs from character gonols and every gonol plus the receipt digest verifies deterministically # class: replay # -# id: unresolved_python_source_remains_hmmm -# given: source has an unmatched delimiter, tokenizer failure, or Python 3.12 grammar error -# then: admitted lower gonols remain preserved beneath a source root whose standing and exact unresolved boundary are hmmm -# class: safety - # id: parser_objects_never_become_gonols # given: tokenizer and AST recognition witnesses are used # then: receipt gonols contain only source-built relation records and closed-gonol references, never TokenInfo, AST, code, or compiler objects # class: boundary -# -# id: python_gonol_geometry_binding_fails_closed -# given: an explicit UCNS Public Gonol authority is supplied -# then: exact carrier positions are observed only when the carrier and declared digest match the pinned identity; function operations remain hmmm -# class: boundary # === END CONTRACTS === from __future__ import annotations import ast import base64 -from dataclasses import dataclass, replace +import binascii +from collections import Counter +from dataclasses import replace from hashlib import sha256 -import io -import json -import keyword -import platform -import sys import token as token_module import tokenize -from typing import Any, Iterable, Mapping, Sequence +import unicodedata +from typing import Any, Iterable +from . import _recognition from .model import ( AffixiationRelation, ClosedGonol, PythonAffixiationReceipt, RelationMember, - SourceSpan, canonical_json_bytes, ) - -SCHEMA = "the-interdependency.python-gonol-affixiation" -SCHEMA_VERSION = "1.0.0" -CONSTRUCTOR_ID = "python-gonol.affixiation" -CONSTRUCTOR_VERSION = "0.1.0" -LANGUAGE_PROFILE = "python-3.12-file-input" -PINNED_PUBLIC_GONOL_SHA256 = ( - "55d10c84529a4d7bc7714786357e977b68d9df2ac3f73d20e229580b552c2ef5" -) -STANDING = "implemented-candidate" -SELECTION_EFFECT = "none" -RELATION_AUTHORITY = ( - "METAPAT affixiation semantics; Python 3.12 source relation witnessed by CPython; " - "UCNS geometry not implied" -) - -NONCLAIMS = ( - "not an AST, token stream, code object, or compiler-object representation", - "not execution or behavioral equivalence of the source", - "not selected UCNS geometry or a UCNS affixiation/coupling law", - "not EDCM measurement validity", - "not canon or independent release authority", -) - -BASE_HMMM = ( - "exact UCNS geometric operation of Public Gonol function positions", - "exact UCNS Mobius-carrier affixiation/coupling law", - "Python language profiles after Python 3.12 file input", -) - - -class PythonGonolConstructionError(RuntimeError): - """Raised when a receipt or explicit authority fails closed.""" - - -@dataclass(frozen=True, slots=True) -class _LexicalWitness: - start: int - end: int - token_name: str - exact_name: str - spelling: str - gonol: ClosedGonol - - -@dataclass(frozen=True, slots=True) -class _MemberCandidate: - role: str - gonol: ClosedGonol - priority: int - - -class _SourceIndex: - def __init__(self, source: str) -> None: - self.source = source - starts = [0] - for index, character in enumerate(source): - if character == "\n": - starts.append(index + 1) - self.line_starts = tuple(starts) - - def token_offset(self, position: tuple[int, int]) -> int: - row, column = position - if row < 1: - raise PythonGonolConstructionError(f"invalid tokenizer row: {row}") - if row > len(self.line_starts): - if row == len(self.line_starts) + 1 and column == 0: - return len(self.source) - raise PythonGonolConstructionError(f"tokenizer position outside source: {position!r}") - offset = self.line_starts[row - 1] + column - if not 0 <= offset <= len(self.source): - raise PythonGonolConstructionError(f"tokenizer position outside source: {position!r}") - return offset - - def ast_offset(self, row: int, utf8_column: int) -> int: - if row < 1 or row > len(self.line_starts): - raise PythonGonolConstructionError( - f"AST position outside decoded source: {(row, utf8_column)!r}" - ) - start = self.line_starts[row - 1] - end = self.line_starts[row] if row < len(self.line_starts) else len(self.source) - line = self.source[start:end] - byte_count = 0 - for char_column, character in enumerate(line): - if byte_count == utf8_column: - return start + char_column - byte_count += len(character.encode("utf-8")) - if byte_count > utf8_column: - break - if byte_count == utf8_column: - return end - raise PythonGonolConstructionError( - f"AST UTF-8 column does not align to a source scalar: {(row, utf8_column)!r}" - ) - - def line_column(self, offset: int) -> tuple[int, int]: - if not 0 <= offset <= len(self.source): - raise PythonGonolConstructionError(f"source offset outside source: {offset}") - low = 0 - high = len(self.line_starts) - while low + 1 < high: - middle = (low + high) // 2 - if self.line_starts[middle] <= offset: - low = middle - else: - high = middle - return (low + 1, offset - self.line_starts[low]) - - def span(self, start: int, end: int) -> SourceSpan: - if not 0 <= start <= end <= len(self.source): - raise PythonGonolConstructionError(f"invalid source span: {(start, end)!r}") - start_line, start_column = self.line_column(start) - end_line, end_column = self.line_column(end) - return SourceSpan(start, end, start_line, start_column, end_line, end_column) - - -class _Registry: - def __init__(self, source_id: str, source_index: _SourceIndex) -> None: - self.source_id = source_id - self.source_index = source_index - self.values: list[ClosedGonol] = [] - self.by_id: dict[str, ClosedGonol] = {} - self.by_address: dict[str, ClosedGonol] = {} - - def close( - self, - *, - address: str, - scale: str, - start: int, - end: int, - relation_kind: str, - candidates: Sequence[_MemberCandidate] = (), - properties: Sequence[tuple[str, str]] = (), - provenance: Sequence[tuple[str, str]] = (), - hmmm: Sequence[str] = (), - ) -> ClosedGonol: - if address in self.by_address: - raise PythonGonolConstructionError(f"duplicate gonol address: {address}") - members: list[RelationMember] = [] - for ordinal, candidate in enumerate(candidates): - child = self.by_id.get(candidate.gonol.gonol_id) - if child is None or child.address != candidate.gonol.address: - raise PythonGonolConstructionError( - "larger construction may reference only an already-closed gonol" - ) - members.append( - RelationMember( - ordinal=ordinal, - role=candidate.role, - gonol_id=child.gonol_id, - address=child.address, - ) - ) - relation = AffixiationRelation( - kind=relation_kind, - members=tuple(members), - properties=tuple((str(key), str(value)) for key, value in properties), - authority=RELATION_AUTHORITY, - ) - provisional = ClosedGonol( - address=address, - scale=scale, - span=self.source_index.span(start, end), - relation=relation, - provenance=( - ("constructor", f"{CONSTRUCTOR_ID}/{CONSTRUCTOR_VERSION}"), - ("language_profile", LANGUAGE_PROFILE), - ("source_id", self.source_id), - ) - + tuple((str(key), str(value)) for key, value in provenance), - hmmm=tuple(str(item) for item in hmmm), - gonol_id="", - ) - gonol_id = sha256(canonical_json_bytes(provisional.identity_payload())).hexdigest() - value = replace(provisional, gonol_id=gonol_id) - if gonol_id in self.by_id: - raise PythonGonolConstructionError( - "closed gonol identity collision; occurrence address failed to distinguish values" - ) - self.values.append(value) - self.by_id[gonol_id] = value - self.by_address[address] = value - return value - - -def _require_source(source: str, source_id: str) -> None: - if not isinstance(source, str): - raise TypeError("source must be an exact decoded Unicode string") - if not isinstance(source_id, str) or not source_id: - raise TypeError("source_id must be exact non-empty text") - for field, value in (("source", source), ("source_id", source_id)): - for character in value: - if 0xD800 <= ord(character) <= 0xDFFF: - raise PythonGonolConstructionError(f"{field} contains a surrogate code point") - - -def _public_gonol_positions(authority: Any | None) -> tuple[dict[str, int] | None, str]: - if authority is None: - return None, "not-supplied" - if isinstance(authority, Mapping): - carrier_value = authority.get("PUBLIC_GONOL_157") - declared_digest = authority.get("PUBLIC_GONOL_SHA256") - else: - carrier_value = getattr(authority, "PUBLIC_GONOL_157", None) - declared_digest = getattr(authority, "PUBLIC_GONOL_SHA256", None) - if not isinstance(carrier_value, Sequence) or isinstance(carrier_value, (str, bytes)): - raise PythonGonolConstructionError( - "explicit UCNS authority must expose PUBLIC_GONOL_157" - ) - carrier = tuple(carrier_value) - if len(carrier) != 157 or len(set(carrier)) != 157: - raise PythonGonolConstructionError( - "explicit UCNS Public Gonol carrier must contain 157 unique positions" - ) - if any(not isinstance(item, str) or len(item) != 1 for item in carrier): - raise PythonGonolConstructionError( - "explicit UCNS Public Gonol positions must be one Unicode scalar each" - ) - computed = sha256( - json.dumps(carrier, ensure_ascii=False, separators=(",", ":")).encode("utf-8") - ).hexdigest() - if declared_digest != computed or computed != PINNED_PUBLIC_GONOL_SHA256: - raise PythonGonolConstructionError( - "explicit UCNS Public Gonol authority digest does not match the pinned carrier" - ) - name = ( - str(authority.get("authority_name", "mapping")) - if isinstance(authority, Mapping) - else str(getattr(authority, "__name__", authority.__class__.__name__)) - ) - return {glyph: index for index, glyph in enumerate(carrier)}, name - - -def _letter_floor( - source: str, - source_id: str, - source_index: _SourceIndex, - registry: _Registry, - positions: dict[str, int] | None, - geometry_name: str, -) -> tuple[ClosedGonol, ...]: - letters: list[ClosedGonol] = [] - for index, character in enumerate(source): - address = f"{source_id}#letter:{index}" - if positions is None: - position = "hmmm:not-supplied" +SCHEMA = _recognition.SCHEMA +SCHEMA_VERSION = "1.1.0" +CONSTRUCTOR_ID = _recognition.CONSTRUCTOR_ID +CONSTRUCTOR_VERSION = "0.2.0" +LANGUAGE_PROFILE = _recognition.LANGUAGE_PROFILE +PINNED_PUBLIC_GONOL_SHA256 = _recognition.PINNED_PUBLIC_GONOL_SHA256 +STANDING = _recognition.STANDING +SELECTION_EFFECT = _recognition.SELECTION_EFFECT +RELATION_AUTHORITY = _recognition.RELATION_AUTHORITY +NONCLAIMS = _recognition.NONCLAIMS +BASE_HMMM = _recognition.BASE_HMMM +PythonGonolConstructionError = _recognition.PythonGonolConstructionError + + +def _identity(gonol: ClosedGonol) -> str: + return sha256(canonical_json_bytes(gonol.identity_payload())).hexdigest() + + +def _rewrite_provenance(provenance: Iterable[tuple[str, str]]) -> tuple[tuple[str, str], ...]: + rewritten: list[tuple[str, str]] = [] + saw_constructor = False + for key, value in provenance: + if key == "constructor": + rewritten.append((key, f"{CONSTRUCTOR_ID}/{CONSTRUCTOR_VERSION}")) + saw_constructor = True else: - found = positions.get(character) - position = "hmmm:not-on-pinned-carrier" if found is None else str(found) - letters.append( - registry.close( - address=address, - scale="letter", - start=index, - end=index + 1, - relation_kind="python.source.letter-occurrence", - properties=( - ("unicode_scalar", character), - ("code_point", f"U+{ord(character):04X}"), - ("occurrence", str(index)), - ("public_gonol_position", position), - ("public_gonol_function", "hmmm"), - ), - provenance=(("geometry_authority", geometry_name),), - ) - ) - return tuple(letters) - - -def _token_relation(token_type: int, spelling: str) -> tuple[str, str]: - token_name = token_module.tok_name[token_type] - if token_type == token_module.NAME and keyword.iskeyword(spelling): - return "KEYWORD", token_name - if token_type == token_module.OP: - return token_module.tok_name.get(tokenize.EXACT_TOKEN_TYPES.get(spelling, token_type), "OP"), token_name - return token_name, token_name - - -def _lexical_floor( - source: str, - source_id: str, - source_index: _SourceIndex, - registry: _Registry, - letters: Sequence[ClosedGonol], -) -> tuple[tuple[_LexicalWitness, ...], tuple[str, ...], tuple[str, ...]]: - raw_tokens: list[tuple[int, int, str, str, str]] = [] - zero_width: list[str] = [] - unresolved: list[str] = [] - stream = tokenize.generate_tokens(io.StringIO(source).readline) - while True: - try: - item = next(stream) - except StopIteration: - break - except (tokenize.TokenError, IndentationError, SyntaxError) as exc: - detail = exc.args[0] if exc.args else exc.__class__.__name__ - location = exc.args[1] if len(exc.args) > 1 else None - unresolved.append(f"tokenizer: {detail}; location={location!r}") - break - exact_name, token_name = _token_relation(item.type, item.string) - if item.string == "": - if token_name not in {"ENDMARKER", "ENCODING"}: - zero_width.append( - f"{token_name}@{source_index.token_offset(item.start)}" - ) - continue - start = source_index.token_offset(item.start) - end = source_index.token_offset(item.end) - if start == end: - if token_name not in {"ENDMARKER", "ENCODING"}: - zero_width.append(f"{token_name}@{start}") - continue - spelling = source[start:end] - if spelling != item.string: - unresolved.append( - f"tokenizer spelling mismatch at {start}:{end}; witness={item.string!r} source={spelling!r}" - ) - raw_tokens.append((start, end, token_name, exact_name, spelling)) - - raw_tokens.sort(key=lambda value: (value[0], value[1])) - previous = 0 - complete: list[tuple[int, int, str, str, str]] = [] - for start, end, token_name, exact_name, spelling in raw_tokens: - if start < previous: - unresolved.append(f"overlapping lexical witnesses at decoded offset {start}") - continue - if start > previous: - complete.append((previous, start, "INTERTOKEN", "INTERTOKEN", source[previous:start])) - complete.append((start, end, token_name, exact_name, spelling)) - previous = end - if previous < len(source): - complete.append((previous, len(source), "INTERTOKEN", "INTERTOKEN", source[previous:])) - - witnesses: list[_LexicalWitness] = [] - for lexical_index, (start, end, token_name, exact_name, spelling) in enumerate(complete): - candidates = tuple( - _MemberCandidate(role=f"letter[{offset - start}]", gonol=letters[offset], priority=3) - for offset in range(start, end) - ) - address = f"{source_id}#lexical:{lexical_index}:{start}-{end}" - gonol = registry.close( - address=address, - scale="lexical-form", - start=start, - end=end, - relation_kind=f"python.lexical.{exact_name}", - candidates=candidates, - properties=( - ("tokenizer_class", token_name), - ("exact_form", exact_name), - ("source_length", str(end - start)), - ), - provenance=(("recognition_witness", "stdlib.tokenize/python-3.12"),), - ) - witnesses.append(_LexicalWitness(start, end, token_name, exact_name, spelling, gonol)) - return tuple(witnesses), tuple(zero_width), tuple(unresolved) - - -def _member_sort_key(candidate: _MemberCandidate) -> tuple[int, int, int, str, str]: - span = candidate.gonol.span - return (span.start, candidate.priority, -span.end, candidate.role, candidate.gonol.address) - - -def _maximal_enclosures( - enclosures: Sequence[ClosedGonol], - start: int, - end: int, - *, - excluded: Sequence[tuple[int, int]] = (), -) -> tuple[ClosedGonol, ...]: - eligible = [ - item - for item in enclosures - if start <= item.span.start - and item.span.end <= end - and not any(left <= item.span.start and item.span.end <= right for left, right in excluded) - ] - selected: list[ClosedGonol] = [] - for item in sorted(eligible, key=lambda value: (value.span.start, -value.span.end)): - if any( - parent.span.start <= item.span.start and item.span.end <= parent.span.end - for parent in selected - ): - continue - selected.append(item) - return tuple(selected) - - -def _delimiter_gonols( - source_id: str, - registry: _Registry, - letters: Sequence[ClosedGonol], - lexicals: Sequence[_LexicalWitness], -) -> tuple[tuple[ClosedGonol, ...], tuple[str, ...]]: - openers = {"(": ")", "[": "]", "{": "}"} - names = {"(": "parentheses", "[": "brackets", "{": "braces"} - stack: list[_LexicalWitness] = [] - pairs: list[tuple[_LexicalWitness, _LexicalWitness]] = [] - unresolved: list[str] = [] - for item in lexicals: - if item.spelling in openers and item.exact_name in {"LPAR", "LSQB", "LBRACE"}: - stack.append(item) - elif item.spelling in {")", "]", "}"} and item.exact_name in {"RPAR", "RSQB", "RBRACE"}: - if not stack: - unresolved.append(f"unmatched closing delimiter {item.spelling!r} at {item.start}") - continue - opener = stack.pop() - if openers[opener.spelling] != item.spelling: - unresolved.append( - f"delimiter mismatch {opener.spelling!r}@{opener.start} with {item.spelling!r}@{item.start}" - ) - continue - pairs.append((opener, item)) - for opener in stack: - unresolved.append(f"unmatched opening delimiter {opener.spelling!r} at {opener.start}") - - built: list[ClosedGonol] = [] - for pair_index, (opener, closer) in enumerate( - sorted(pairs, key=lambda pair: (pair[1].end - pair[0].start, pair[0].start)) - ): - start, end = opener.start, closer.end - nested = _maximal_enclosures(built, start, end) - nested_ranges = tuple((item.span.start, item.span.end) for item in nested) - candidates: list[_MemberCandidate] = [] - for item in lexicals: - if not (start <= item.start and item.end <= end): - continue - if any(left <= item.start and item.end <= right for left, right in nested_ranges): - continue - role = "opener" if item is opener else "closer" if item is closer else "content" - candidates.append(_MemberCandidate(role, item.gonol, 2)) - candidates.extend(_MemberCandidate("nested", item, 1) for item in nested) - candidates.sort(key=_member_sort_key) - counter = 0 - normalized: list[_MemberCandidate] = [] - for candidate in candidates: - if candidate.role in {"content", "nested"}: - normalized.append(replace(candidate, role=f"content[{counter}]")) - counter += 1 - else: - normalized.append(candidate) - built.append( - registry.close( - address=f"{source_id}#delimiter:{pair_index}:{start}-{end}", - scale="delimiter-construction", - start=start, - end=end, - relation_kind=f"python.delimiter.{names[opener.spelling]}", - candidates=normalized, - properties=( - ("opener", opener.spelling), - ("closer", closer.spelling), - ("closure", "matched"), - ), - provenance=(("recognition_witness", "python-3.12 delimiter stack"),), - hmmm=("exact UCNS delimiter relation geometry",), - ) - ) - return tuple(built), tuple(unresolved) + rewritten.append((key, value)) + if not saw_constructor: + rewritten.insert(0, ("constructor", f"{CONSTRUCTOR_ID}/{CONSTRUCTOR_VERSION}")) + return tuple(rewritten) -def _primitive_properties(node: ast.AST) -> tuple[tuple[str, str], ...]: - properties: list[tuple[str, str]] = [] - skipped_values = { - ("Constant", "value"), - ("MatchSingleton", "value"), - } - for field, value in ast.iter_fields(node): - if field == "ctx" or (node.__class__.__name__, field) in skipped_values: - continue - if isinstance(value, ast.AST): - if isinstance(value, (ast.operator, ast.unaryop, ast.boolop, ast.cmpop)): - properties.append((f"grammar_field.{field}", value.__class__.__name__)) - continue - if isinstance(value, list): - if value and all(isinstance(item, str) for item in value): - properties.append((f"grammar_field.{field}", json.dumps(value, ensure_ascii=False))) - continue - if value is None: - continue - if isinstance(value, (str, int, bool)): - properties.append((f"grammar_field.{field}", json.dumps(value, ensure_ascii=False))) - return tuple(properties) - +def _character_definitions(character: str) -> tuple[tuple[str, str], ...]: + """Return deterministic definitions applicable to one source character. -def _spanless_relation_property(field: str, node: ast.AST) -> tuple[str, str]: - """Keep a witnessed child relation inside its source-built parent. - - A few CPython grammar records, notably an empty ``arguments`` value and - ``TypeIgnore``, have no independent source span. They cannot honestly - close as gonols of their own. Their relation is therefore recorded on the - parent that already owns the exact surface gonols; the parser record itself - is discarded. + These are definition-space facts, not the contextual lexical role of this + occurrence. A character may therefore have several definitions at once. """ - descriptor: dict[str, Any] = {"grammar_construct": node.__class__.__name__} - for name, value in ast.iter_fields(node): - if isinstance(value, (str, int, bool)) or value is None: - descriptor[name] = value - elif isinstance(value, list) and not any(isinstance(item, ast.AST) for item in value): - descriptor[name] = value - return ( - f"grammar_field.{field}.spanless_relation", - json.dumps(descriptor, ensure_ascii=False, sort_keys=True, separators=(",", ":")), + values: set[tuple[str, str]] = { + ("unicode-category", unicodedata.category(character)), + } + unicode_name = unicodedata.name(character, "") + if unicode_name: + values.add(("unicode-name", unicode_name)) + + if character.isidentifier(): + values.add(("python-identifier", "start")) + if ("A" + character).isidentifier(): + values.add(("python-identifier", "continue")) + if character.isdecimal(): + values.add(("python-numeric", "decimal-digit")) + if character in " \t\f": + values.add(("python-layout", "horizontal-whitespace")) + if character in "\r\n": + values.add(("python-layout", "line-break")) + if character in {"'", '"'}: + values.add(("python-delimiter", "string-quote-candidate")) + if character == "#": + values.add(("python-delimiter", "comment-introducer")) + if character == "\\": + values.add(("python-layout", "explicit-line-join-candidate")) + + for spelling, token_type in tokenize.EXACT_TOKEN_TYPES.items(): + if len(spelling) == 1 and spelling == character: + values.add(("python-exact-token", token_module.tok_name[token_type])) + + return tuple(sorted(values)) + + +def _character_from_recognition(old: ClosedGonol) -> ClosedGonol: + relation = replace(old.relation, kind="python.source.character-occurrence", members=()) + prefix, marker, suffix = old.address.rpartition("#letter:") + if not marker: + raise PythonGonolConstructionError(f"recognition letter address is malformed: {old.address}") + provisional = replace( + old, + address=prefix + "#character:" + suffix, + scale="character", + relation=relation, + provenance=_rewrite_provenance(old.provenance), + gonol_id="", ) + return replace(provisional, gonol_id=_identity(provisional)) -def _grammar_root( - source: str, +def _definition_gonol( + *, source_id: str, - source_index: _SourceIndex, - registry: _Registry, - letters: Sequence[ClosedGonol], - lexicals: Sequence[_LexicalWitness], - enclosures: Sequence[ClosedGonol], - zero_width: Sequence[str], -) -> tuple[ClosedGonol | None, tuple[str, ...]]: - try: - tree = ast.parse( - source, - filename=source_id, - mode="exec", - type_comments=True, - feature_version=(3, 12), - ) - except (SyntaxError, ValueError, TypeError, MemoryError) as exc: - if isinstance(exc, SyntaxError): - detail = f"grammar: {exc.msg}; line={exc.lineno!r}; offset={exc.offset!r}" - else: - detail = f"grammar: {exc.__class__.__name__}: {exc}" - return None, (detail,) + character: ClosedGonol, + character_index: int, + definition_index: int, + kind: str, + value: str, +) -> ClosedGonol: + relation = AffixiationRelation( + kind="python.character.definition", + members=( + RelationMember( + ordinal=0, + role="origin", + gonol_id=character.gonol_id, + address=character.address, + ), + ), + properties=( + ("definition_kind", kind), + ("definition_value", value), + ), + authority=RELATION_AUTHORITY, + ) + provisional = ClosedGonol( + address=f"{source_id}#character-definition:{character_index}:{definition_index}", + scale="character-definition", + span=character.span, + relation=relation, + provenance=( + ("constructor", f"{CONSTRUCTOR_ID}/{CONSTRUCTOR_VERSION}"), + ("language_profile", LANGUAGE_PROFILE), + ("source_id", source_id), + ("definition_authority", f"Python 3.12 lexical profile + Unicode {unicodedata.unidata_version}"), + ), + hmmm=(), + gonol_id="", + ) + return replace(provisional, gonol_id=_identity(provisional)) - node_order = {id(node): index for index, node in enumerate(ast.walk(tree))} - span_memo: dict[int, tuple[int, int] | None] = {} - def node_span(node: ast.AST) -> tuple[int, int] | None: - key = id(node) - if key in span_memo: - return span_memo[key] - spans: list[tuple[int, int]] = [] - if all(hasattr(node, name) for name in ("lineno", "col_offset", "end_lineno", "end_col_offset")): - end_line = getattr(node, "end_lineno", None) - end_column = getattr(node, "end_col_offset", None) - if end_line is not None and end_column is not None: - spans.append( - ( - source_index.ast_offset(int(node.lineno), int(node.col_offset)), - source_index.ast_offset(int(end_line), int(end_column)), - ) - ) - for child in ast.iter_child_nodes(node): - child_span = node_span(child) - if child_span is not None: - spans.append(child_span) - if isinstance(node, ast.Module): - result: tuple[int, int] | None = (0, len(source)) - elif spans: - result = (min(item[0] for item in spans), max(item[1] for item in spans)) - else: - result = None - if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)) and node.decorator_list: - first_decorator = min(node.decorator_list, key=lambda item: (item.lineno, item.col_offset)) - decorator_start = source_index.ast_offset( - int(first_decorator.lineno), int(first_decorator.col_offset) - ) - marker = max( - ( - item - for item in lexicals - if item.spelling == "@" - and item.exact_name == "AT" - and item.end <= decorator_start - and source_index.line_column(item.start)[0] == int(first_decorator.lineno) - ), - key=lambda item: item.start, - default=None, +def _remap_gonol(old: ClosedGonol, mapped: dict[str, ClosedGonol]) -> ClosedGonol: + members: list[RelationMember] = [] + for member in old.relation.members: + child = mapped.get(member.gonol_id) + if child is None: + raise PythonGonolConstructionError( + f"recognition witness referenced an unclosed child: {old.address}" ) - if marker is not None and result is not None: - result = (min(marker.start, result[0]), result[1]) - span_memo[key] = result - return result - - node_span(tree) - built: dict[int, ClosedGonol] = {} - - def construct(node: ast.AST) -> ClosedGonol | None: - key = id(node) - if key in built: - return built[key] - span = node_span(node) - if span is None: - return None - start, end = span - child_candidates: list[_MemberCandidate] = [] - child_ranges: list[tuple[int, int]] = [] - spanless_properties: list[tuple[str, str]] = [] - for field, value in ast.iter_fields(node): - if isinstance(value, ast.AST): - child = construct(value) - if child is not None: - child_candidates.append(_MemberCandidate(field, child, 0)) - child_ranges.append((child.span.start, child.span.end)) - elif not isinstance( - value, - (ast.operator, ast.unaryop, ast.boolop, ast.cmpop, ast.expr_context), - ): - spanless_properties.append(_spanless_relation_property(field, value)) - elif isinstance(value, list): - for position, item in enumerate(value): - if not isinstance(item, ast.AST): - continue - child = construct(item) - if child is not None: - child_candidates.append(_MemberCandidate(f"{field}[{position}]", child, 0)) - child_ranges.append((child.span.start, child.span.end)) - elif not isinstance( - item, - (ast.operator, ast.unaryop, ast.boolop, ast.cmpop, ast.expr_context), - ): - spanless_properties.append( - _spanless_relation_property(f"{field}[{position}]", item) - ) - - selected_enclosures = _maximal_enclosures( - enclosures, - start, - end, - excluded=child_ranges, - ) - enclosure_ranges = tuple((item.span.start, item.span.end) for item in selected_enclosures) - surface_candidates: list[_MemberCandidate] = [ - _MemberCandidate("surface", item, 1) for item in selected_enclosures - ] - for lexical in lexicals: - if not (start <= lexical.start and lexical.end <= end): - continue - if any(left <= lexical.start and lexical.end <= right for left, right in child_ranges): - continue - if any(left <= lexical.start and lexical.end <= right for left, right in enclosure_ranges): - continue - surface_candidates.append(_MemberCandidate("surface", lexical.gonol, 2)) - - candidates = child_candidates + surface_candidates - candidates.sort(key=_member_sort_key) - surface_index = 0 - normalized: list[_MemberCandidate] = [] - for candidate in candidates: - if candidate.role == "surface": - normalized.append(replace(candidate, role=f"surface[{surface_index}]")) - surface_index += 1 - else: - normalized.append(candidate) - properties = [ - ("grammar_profile", LANGUAGE_PROFILE), - ("grammar_construct", node.__class__.__name__), - ] - properties.extend(_primitive_properties(node)) - properties.extend(spanless_properties) - if isinstance(node, ast.Module): - properties.extend( - (f"zero_width_witness[{index}]", item) - for index, item in enumerate(zero_width) + role = member.role + if role.startswith("letter["): + role = "character[" + role[len("letter[") :] + members.append( + RelationMember( + ordinal=member.ordinal, + role=role, + gonol_id=child.gonol_id, + address=child.address, ) - value = registry.close( - address=( - f"{source_id}#grammar:{node_order[key]}:{node.__class__.__name__}:{start}-{end}" - ), - scale="module" if isinstance(node, ast.Module) else "python-construction", - start=start, - end=end, - relation_kind=f"python.grammar.{node.__class__.__name__}", - candidates=normalized, - properties=properties, - provenance=(("recognition_witness", "stdlib.ast/python-3.12"),), - hmmm=("exact UCNS geometry for this Python relation",), ) - built[key] = value - return value - - return construct(tree), () - - -def _source_root_hmmm( - source_id: str, - registry: _Registry, - letters: Sequence[ClosedGonol], - lexicals: Sequence[_LexicalWitness], - enclosures: Sequence[ClosedGonol], - unresolved: Sequence[str], - zero_width: Sequence[str], -) -> ClosedGonol: - selected = _maximal_enclosures(enclosures, 0, len(letters)) - selected_ranges = tuple((item.span.start, item.span.end) for item in selected) - candidates: list[_MemberCandidate] = [ - _MemberCandidate("closed-delimiter", item, 1) for item in selected - ] - for item in lexicals: - if any(left <= item.start and item.end <= right for left, right in selected_ranges): - continue - candidates.append(_MemberCandidate("lexical-form", item.gonol, 2)) - candidates.sort(key=_member_sort_key) - properties = [("standing", "hmmm")] - properties.extend((f"unresolved[{index}]", item) for index, item in enumerate(unresolved)) - properties.extend( - (f"zero_width_witness[{index}]", item) for index, item in enumerate(zero_width) - ) - return registry.close( - address=f"{source_id}#source:hmmm", - scale="source", - start=0, - end=len(letters), - relation_kind="python.source.hmmm", - candidates=candidates, - properties=properties, - provenance=(("recognition_witness", "partial Python 3.12 recognition"),), - hmmm=unresolved, + provisional = replace( + old, + relation=replace(old.relation, members=tuple(members)), + provenance=_rewrite_provenance(old.provenance), + gonol_id="", ) + return replace(provisional, gonol_id=_identity(provisional)) def _receipt_digest(receipt: PythonAffixiationReceipt) -> str: return sha256(canonical_json_bytes(receipt.payload())).hexdigest() -def _construct( - source: str, - *, - source_id: str, - source_bytes: bytes, - encoding: str, - geometry_authority: Any | None, +def _upgrade_recognition_receipt( + recognized: PythonAffixiationReceipt, ) -> PythonAffixiationReceipt: - _require_source(source, source_id) - if sys.version_info[:2] != (3, 12): - raise PythonGonolConstructionError( - f"{LANGUAGE_PROFILE} requires a Python 3.12 recognition witness; got {sys.version_info.major}.{sys.version_info.minor}" - ) - source_index = _SourceIndex(source) - registry = _Registry(source_id, source_index) - positions, geometry_name = _public_gonol_positions(geometry_authority) - letters = _letter_floor( - source, - source_id, - source_index, - registry, - positions, - geometry_name, - ) - lexicals, zero_width, lexical_hmmm = _lexical_floor( - source, - source_id, - source_index, - registry, - letters, - ) - enclosures, delimiter_hmmm = _delimiter_gonols( - source_id, - registry, - letters, - lexicals, - ) - grammar_root, grammar_hmmm = _grammar_root( - source, - source_id, - source_index, - registry, - letters, - lexicals, - enclosures, - zero_width, - ) - syntax_hmmm = lexical_hmmm + delimiter_hmmm + grammar_hmmm - if grammar_root is None or syntax_hmmm: - root = _source_root_hmmm( - source_id, - registry, - letters, - lexicals, - enclosures, - syntax_hmmm or ("grammar root was not constructed",), - zero_width, - ) - standing = "hmmm" - else: - root = grammar_root - standing = STANDING + """Close the public character/definition construction from a private witness plan.""" + + mapped: dict[str, ClosedGonol] = {} + values: list[ClosedGonol] = [] + character_index = 0 + + for old in recognized.gonols: + if old.scale == "letter": + character = _character_from_recognition(old) + mapped[old.gonol_id] = character + values.append(character) + scalar = dict(character.relation.properties).get("unicode_scalar") + if scalar is None or len(scalar) != 1: + raise PythonGonolConstructionError( + f"recognition character has no exact Unicode scalar: {old.address}" + ) + for definition_index, (kind, value) in enumerate(_character_definitions(scalar)): + values.append( + _definition_gonol( + source_id=recognized.source_id, + character=character, + character_index=character_index, + definition_index=definition_index, + kind=kind, + value=value, + ) + ) + character_index += 1 + continue - hmmm = list(BASE_HMMM) - if geometry_authority is None: - hmmm.append("UCNS Public Gonol geometry authority was not supplied") - if positions is not None and any(character not in positions for character in source): - hmmm.append("one or more admitted source scalars have no position on the pinned Public Gonol carrier") - hmmm.extend(syntax_hmmm) - runtime = f"{platform.python_implementation()}-{platform.python_version()}" - provisional = PythonAffixiationReceipt( - schema=SCHEMA, + value = _remap_gonol(old, mapped) + mapped[old.gonol_id] = value + values.append(value) + + root = mapped.get(recognized.root_gonol_id) + if root is None: + raise PythonGonolConstructionError("recognition root was not remapped") + + provisional = replace( + recognized, version=SCHEMA_VERSION, - constructor_id=CONSTRUCTOR_ID, constructor_version=CONSTRUCTOR_VERSION, - language_profile=LANGUAGE_PROFILE, - source_id=source_id, - encoding=encoding, - source_bytes_base64=base64.b64encode(source_bytes).decode("ascii"), - source_bytes_sha256=sha256(source_bytes).hexdigest(), - decoded_source_sha256=sha256(source.encode("utf-8")).hexdigest(), - recognition_witness=runtime, - standing=standing, - selection_effect=SELECTION_EFFECT, root_gonol_id=root.gonol_id, - gonols=tuple(registry.values), - nonclaims=NONCLAIMS, - hmmm=tuple(hmmm), + gonols=tuple(values), receipt_digest="", ) return replace(provisional, receipt_digest=_receipt_digest(provisional)) @@ -954,17 +306,14 @@ def affixiate_python_source( source_id: str, geometry_authority: Any | None = None, ) -> PythonAffixiationReceipt: - """Affixiate exact decoded Python 3.12 file-input source from letters upward.""" + """Affixiate exact decoded Python 3.12 file-input source from characters upward.""" - if not isinstance(source, str): - raise TypeError("source must be an exact decoded Unicode string") - return _construct( + recognized = _recognition.affixiate_python_source( source, source_id=source_id, - source_bytes=source.encode("utf-8"), - encoding="utf-8", geometry_authority=geometry_authority, ) + return _upgrade_recognition_receipt(recognized) def affixiate_python_bytes( @@ -975,20 +324,12 @@ def affixiate_python_bytes( ) -> PythonAffixiationReceipt: """Detect Python's declared encoding, preserve exact bytes, and affixiate source.""" - if not isinstance(source_bytes, bytes): - raise TypeError("source_bytes must be exact bytes") - try: - encoding, _lines = tokenize.detect_encoding(io.BytesIO(source_bytes).readline) - source = source_bytes.decode(encoding) - except (SyntaxError, UnicodeDecodeError, LookupError) as exc: - raise PythonGonolConstructionError(f"Python source decoding failed: {exc}") from exc - return _construct( - source, + recognized = _recognition.affixiate_python_bytes( + source_bytes, source_id=source_id, - source_bytes=source_bytes, - encoding=encoding, geometry_authority=geometry_authority, ) + return _upgrade_recognition_receipt(recognized) def _property(gonol: ClosedGonol, name: str) -> str: @@ -1001,17 +342,37 @@ def _property(gonol: ClosedGonol, name: str) -> str: def reconstruct_source(receipt: PythonAffixiationReceipt) -> str: - """Reconstruct decoded source solely from ordered letter gonols.""" + """Reconstruct decoded source solely from ordered character gonols.""" - letters = sorted( - (gonol for gonol in receipt.gonols if gonol.scale == "letter"), + characters = sorted( + (gonol for gonol in receipt.gonols if gonol.scale == "character"), key=lambda gonol: gonol.span.start, ) - return "".join(_property(gonol, "unicode_scalar") for gonol in letters) + return "".join(_property(gonol, "unicode_scalar") for gonol in characters) + + +def grammar_witness_inventory() -> tuple[str, ...]: + """Return the public CPython AST witness vocabulary for the pinned runtime profile. + + The constructor itself does not dispatch on this list; ``_recognition`` walks + arbitrary AST fields recursively. This inventory exists so runtime/profile + drift is visible instead of silently narrowing the claimed witness surface. + """ + + return tuple( + sorted( + name + for name, value in vars(ast).items() + if isinstance(value, type) + and issubclass(value, ast.AST) + and value is not ast.AST + and not name.startswith("_") + ) + ) def replay_python_affixiation(receipt: PythonAffixiationReceipt) -> PythonAffixiationReceipt: - """Fail closed unless every visible gonol and receipt identity remains valid.""" + """Fail closed unless every public character-first construction remains valid.""" if not isinstance(receipt, PythonAffixiationReceipt): raise TypeError("receipt must be a PythonAffixiationReceipt") @@ -1025,12 +386,14 @@ def replay_python_affixiation(receipt: PythonAffixiationReceipt) -> PythonAffixi ) if actual != expected: raise PythonGonolConstructionError("receipt schema, constructor, or language profile mismatch") + try: raw = base64.b64decode(receipt.source_bytes_base64, validate=True) - except ValueError as exc: + except (ValueError, binascii.Error) as exc: raise PythonGonolConstructionError("receipt source bytes are not valid base64") from exc if sha256(raw).hexdigest() != receipt.source_bytes_sha256: raise PythonGonolConstructionError("receipt source byte digest mismatch") + source = reconstruct_source(receipt) if sha256(source.encode("utf-8")).hexdigest() != receipt.decoded_source_sha256: raise PythonGonolConstructionError("decoded source digest mismatch") @@ -1039,18 +402,23 @@ def replay_python_affixiation(receipt: PythonAffixiationReceipt) -> PythonAffixi except (UnicodeDecodeError, LookupError) as exc: raise PythonGonolConstructionError("receipt source bytes no longer decode as declared") from exc if decoded != source: - raise PythonGonolConstructionError("receipt bytes and letter gonols reconstruct different source") + raise PythonGonolConstructionError("receipt bytes and character gonols reconstruct different source") - source_index = _SourceIndex(source) + source_index = _recognition._SourceIndex(source) known: dict[str, ClosedGonol] = {} known_addresses: set[str] = set() - expected_letter_start = 0 + character_definitions: Counter[str] = Counter() + lexical_characters: Counter[str] = Counter() + expected_character_start = 0 + for gonol in receipt.gonols: if gonol.gonol_id in known: raise PythonGonolConstructionError("duplicate gonol identity in receipt") if gonol.address in known_addresses: raise PythonGonolConstructionError("duplicate gonol address in receipt") - expected_id = sha256(canonical_json_bytes(gonol.identity_payload())).hexdigest() + if gonol.scale == "letter" or "#letter:" in gonol.address: + raise PythonGonolConstructionError("deprecated letter-floor gonol present") + expected_id = _identity(replace(gonol, gonol_id="")) if gonol.gonol_id != expected_id: raise PythonGonolConstructionError(f"gonol identity mismatch: {gonol.address}") if gonol.span != source_index.span(gonol.span.start, gonol.span.end): @@ -1059,37 +427,66 @@ def replay_python_affixiation(receipt: PythonAffixiationReceipt) -> PythonAffixi range(len(gonol.relation.members)) ): raise PythonGonolConstructionError(f"member order drifted: {gonol.address}") - member_spans: list[tuple[int, int]] = [] + + member_children: list[ClosedGonol] = [] for member in gonol.relation.members: child = known.get(member.gonol_id) if child is None or child.address != member.address: raise PythonGonolConstructionError( f"gonol references a child that was not already closed: {gonol.address}" ) - member_spans.append((child.span.start, child.span.end)) - if gonol.scale == "letter": - if gonol.span.start != expected_letter_start or gonol.span.end != expected_letter_start + 1: - raise PythonGonolConstructionError("letter floor is not contiguous and ordered") + member_children.append(child) + + if gonol.scale == "character": + if gonol.relation.members: + raise PythonGonolConstructionError("character gonol cannot contain prior participants") + if gonol.span.start != expected_character_start or gonol.span.end != expected_character_start + 1: + raise PythonGonolConstructionError("character floor is not contiguous and ordered") scalar = _property(gonol, "unicode_scalar") if len(scalar) != 1 or scalar != source[gonol.span.start : gonol.span.end]: - raise PythonGonolConstructionError("letter gonol does not match its source occurrence") - expected_letter_start += 1 + raise PythonGonolConstructionError("character gonol does not match its source occurrence") + expected_character_start += 1 + elif gonol.scale == "character-definition": + if len(member_children) != 1 or member_children[0].scale != "character": + raise PythonGonolConstructionError("character definition must have one closed character origin") + if member_children[0].span != gonol.span: + raise PythonGonolConstructionError("character definition span differs from its origin") + _property(gonol, "definition_kind") + _property(gonol, "definition_value") + character_definitions[member_children[0].address] += 1 else: merged: list[list[int]] = [] - for start, end in sorted(member_spans): - if not merged or start > merged[-1][1]: - merged.append([start, end]) - elif end > merged[-1][1]: - merged[-1][1] = end - expected = [] if gonol.span.start == gonol.span.end else [[gonol.span.start, gonol.span.end]] - if merged != expected: + for child in sorted(member_children, key=lambda item: (item.span.start, item.span.end)): + if not merged or child.span.start > merged[-1][1]: + merged.append([child.span.start, child.span.end]) + elif child.span.end > merged[-1][1]: + merged[-1][1] = child.span.end + expected_span = [] if gonol.span.start == gonol.span.end else [[gonol.span.start, gonol.span.end]] + if merged != expected_span: raise PythonGonolConstructionError( f"gonol span differs from its atomic closed participants: {gonol.address}" ) + if gonol.scale == "lexical-form": + for member, child in zip(gonol.relation.members, member_children, strict=True): + if child.scale != "character" or not member.role.startswith("character["): + raise PythonGonolConstructionError( + "lexical form must affixiate exact closed character occurrences" + ) + lexical_characters[child.address] += 1 + known[gonol.gonol_id] = gonol known_addresses.add(gonol.address) - if expected_letter_start != len(source): - raise PythonGonolConstructionError("letter floor does not cover complete decoded source") + + if expected_character_start != len(source): + raise PythonGonolConstructionError("character floor does not cover complete decoded source") + character_addresses = { + gonol.address for gonol in receipt.gonols if gonol.scale == "character" + } + if set(character_definitions) != character_addresses: + raise PythonGonolConstructionError("every source character must have a definition-space") + if lexical_characters != Counter({address: 1 for address in character_addresses}): + raise PythonGonolConstructionError("lexical floor is not an exact partition of source characters") + root = known.get(receipt.root_gonol_id) if ( root is None @@ -1115,6 +512,7 @@ def replay_python_affixiation(receipt: PythonAffixiationReceipt) -> PythonAffixi "PythonGonolConstructionError", "affixiate_python_bytes", "affixiate_python_source", + "grammar_witness_inventory", "reconstruct_source", "replay_python_affixiation", ] From 4c5333ce6d2aa8aa32a83936aa70bcc5fd356c0a Mon Sep 17 00:00:00 2001 From: Erin Spencer Date: Sat, 12 Sep 2026 16:48:41 -0700 Subject: [PATCH 04/10] fix(python-gonol): expose character-first constructor --- research/python-gonol/python_gonol/__init__.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/research/python-gonol/python_gonol/__init__.py b/research/python-gonol/python_gonol/__init__.py index 2bc7b30..1241941 100644 --- a/research/python-gonol/python_gonol/__init__.py +++ b/research/python-gonol/python_gonol/__init__.py @@ -1,18 +1,18 @@ """Public surface for stack-local Python Gonol Construction. Usage guidance: call ``affixiate_python_bytes`` for a file or -``affixiate_python_source`` for already-decoded text, then persist or compare -the returned receipt. Call ``replay_python_affixiation`` before consuming a -receipt across a boundary. +``affixiate_python_source`` for already-decoded text, persist the returned +receipt, and call ``replay_python_affixiation`` before consuming a receipt +across a boundary. """ # === MODULE_BUILD === # id: python_gonol_public_surface # module_name: python_gonol # module_kind: adapter -# summary: exposes the bounded Python 3.12 bottom-up affixiation constructor and immutable receipt types +# summary: exposes the bounded Python 3.12 character-first affixiation constructor and immutable receipt types # owner: Python Gonol Construction (stack-local research) -# public_surface: affixiate_python_source, affixiate_python_bytes, replay_python_affixiation, reconstruct_source, PythonAffixiationReceipt +# public_surface: affixiate_python_source, affixiate_python_bytes, replay_python_affixiation, reconstruct_source, grammar_witness_inventory, PythonAffixiationReceipt # internal_surface: none # auth_boundary: none # storage_boundary: none @@ -35,6 +35,7 @@ PythonGonolConstructionError, affixiate_python_bytes, affixiate_python_source, + grammar_witness_inventory, reconstruct_source, replay_python_affixiation, ) @@ -59,6 +60,7 @@ "SourceSpan", "affixiate_python_bytes", "affixiate_python_source", + "grammar_witness_inventory", "reconstruct_source", "replay_python_affixiation", ] From 582a882d037f77c421e957cc438f3acc6c2f9107 Mon Sep 17 00:00:00 2001 From: Erin Spencer Date: Sat, 12 Sep 2026 16:48:51 -0700 Subject: [PATCH 05/10] fix(python-gonol): update CLI to character-first receipts --- research/python-gonol/python_gonol/__main__.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/research/python-gonol/python_gonol/__main__.py b/research/python-gonol/python_gonol/__main__.py index ec40bbb..a679f86 100644 --- a/research/python-gonol/python_gonol/__main__.py +++ b/research/python-gonol/python_gonol/__main__.py @@ -5,15 +5,16 @@ python -m python_gonol source.py --out source.gonol.json python -m python_gonol --verify source.gonol.json -The construction command exits 2 when source syntax remains ``hmmm``. The -receipt is still written so its admitted letter and lexical closures survive. +The construction command exits 2 when source syntax remains ``hmmm``. The +receipt is still written so admitted character, definition, and lexical +closures survive. """ # === MODULE_BUILD === # id: python_gonol_cli # module_name: python_gonol.__main__ # module_kind: adapter -# summary: provides file-to-receipt construction and receipt verification commands +# summary: provides file-to-receipt character-first construction and receipt verification commands # owner: Python Gonol Construction (stack-local research) # public_surface: python -m python_gonol # internal_surface: main @@ -59,7 +60,7 @@ def _parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( - description="Affixiate Python 3.12 source into gonols from letters upward." + description="Affixiate Python 3.12 source into gonols from characters upward." ) parser.add_argument("source", nargs="?", help="Python source file") parser.add_argument("--out", help="receipt path; omit for stdout") From 4d18c4fe615b7b3bc45149e673d0841a5a93f0e3 Mon Sep 17 00:00:00 2001 From: Erin Spencer Date: Sat, 12 Sep 2026 16:49:16 -0700 Subject: [PATCH 06/10] test(python-gonol): enforce character definition floor --- .../python-gonol/tests/test_affixiation.py | 110 ++++++++++++++---- 1 file changed, 86 insertions(+), 24 deletions(-) diff --git a/research/python-gonol/tests/test_affixiation.py b/research/python-gonol/tests/test_affixiation.py index 927e9b7..2cb3397 100644 --- a/research/python-gonol/tests/test_affixiation.py +++ b/research/python-gonol/tests/test_affixiation.py @@ -1,14 +1,21 @@ # === CHECKS === -# id: every_python_source_occurrence_closes_first_check -# proves: every_python_source_occurrence_closes_first -# call: self::test_letter_floor_preserves_identity_order_multiplicity_and_provenance +# id: every_python_source_character_closes_first_check +# proves: every_python_source_character_closes_first +# call: self::test_character_floor_preserves_identity_order_multiplicity_and_provenance # timeout: 30 # mutates: none # cleanup: none # -# id: python_lexical_forms_affixiate_letters_check -# proves: python_lexical_forms_affixiate_letters -# call: self::test_lexical_floor_is_an_exact_partition_of_closed_letters +# id: python_character_definitions_share_character_origin_check +# proves: python_character_definitions_share_character_origin +# call: self::test_character_definition_space_closes_before_lexical_forms +# timeout: 30 +# mutates: none +# cleanup: none +# +# id: python_lexical_forms_affixiate_characters_check +# proves: python_lexical_forms_affixiate_characters +# call: self::test_lexical_floor_is_an_exact_partition_of_closed_characters # timeout: 30 # mutates: none # cleanup: none @@ -66,39 +73,77 @@ PythonGonolConstructionError, affixiate_python_bytes, affixiate_python_source, + grammar_witness_inventory, reconstruct_source, replay_python_affixiation, ) -def _letters(receipt: PythonAffixiationReceipt): - return tuple(gonol for gonol in receipt.gonols if gonol.scale == "letter") +def _characters(receipt: PythonAffixiationReceipt): + return tuple(gonol for gonol in receipt.gonols if gonol.scale == "character") -def test_letter_floor_preserves_identity_order_multiplicity_and_provenance() -> None: +def _definitions(receipt: PythonAffixiationReceipt): + return tuple(gonol for gonol in receipt.gonols if gonol.scale == "character-definition") + + +def test_character_floor_preserves_identity_order_multiplicity_and_provenance() -> None: source = "π = (a + a)\n# a\n" receipt = affixiate_python_source(source, source_id="fixture/repeated.py") - letters = _letters(receipt) - assert len(letters) == len(source) - assert [item.span.start for item in letters] == list(range(len(source))) - assert len({item.address for item in letters}) == len(source) - assert len({item.gonol_id for item in letters}) == len(source) - repeated = [item for item in letters if dict(item.relation.properties)["unicode_scalar"] == "a"] + characters = _characters(receipt) + assert len(characters) == len(source) + assert [item.span.start for item in characters] == list(range(len(source))) + assert len({item.address for item in characters}) == len(source) + assert len({item.gonol_id for item in characters}) == len(source) + repeated = [item for item in characters if dict(item.relation.properties)["unicode_scalar"] == "a"] assert len(repeated) == 3 assert len({item.address for item in repeated}) == 3 - assert all(("source_id", "fixture/repeated.py") in item.provenance for item in letters) + assert all("#character:" in item.address for item in characters) + assert all(("source_id", "fixture/repeated.py") in item.provenance for item in characters) + assert not any(gonol.scale == "letter" for gonol in receipt.gonols) + + +def test_character_definition_space_closes_before_lexical_forms() -> None: + source = "x-y # note\n" + receipt = affixiate_python_source(source, source_id="fixture/definitions.py") + positions = {gonol.gonol_id: index for index, gonol in enumerate(receipt.gonols)} + definitions = _definitions(receipt) + by_origin: Counter[str] = Counter() + for definition in definitions: + assert definition.relation.kind == "python.character.definition" + assert len(definition.relation.members) == 1 + origin = definition.relation.members[0] + assert origin.role == "origin" + assert receipt.gonols[positions[origin.gonol_id]].scale == "character" + assert positions[origin.gonol_id] < positions[definition.gonol_id] + by_origin[origin.address] += 1 + assert set(by_origin) == {item.address for item in _characters(receipt)} + + minus = next(item for item in _characters(receipt) if dict(item.relation.properties)["unicode_scalar"] == "-") + minus_defs = { + (dict(item.relation.properties)["definition_kind"], dict(item.relation.properties)["definition_value"]) + for item in definitions + if item.relation.members[0].gonol_id == minus.gonol_id + } + assert ("python-exact-token", "MINUS") in minus_defs + + first_lexical = min( + positions[item.gonol_id] for item in receipt.gonols if item.scale == "lexical-form" + ) + assert all(positions[item.gonol_id] < first_lexical for item in definitions) -def test_lexical_floor_is_an_exact_partition_of_closed_letters() -> None: +def test_lexical_floor_is_an_exact_partition_of_closed_characters() -> None: source = "value = f'{name!r:>{width}}' # keep both spaces\n" receipt = affixiate_python_source(source, source_id="fixture/lexical.py") lexical = [gonol for gonol in receipt.gonols if gonol.scale == "lexical-form"] counts = Counter(member.address for gonol in lexical for member in gonol.relation.members) - assert counts == Counter(item.address for item in _letters(receipt)) - earlier = {gonol.gonol_id: index for index, gonol in enumerate(receipt.gonols)} + assert counts == Counter(item.address for item in _characters(receipt)) + position = {gonol.gonol_id: index for index, gonol in enumerate(receipt.gonols)} for gonol in lexical: assert gonol.relation.members - assert all(receipt.gonols[earlier[member.gonol_id]].scale == "letter" for member in gonol.relation.members) + assert all(receipt.gonols[position[member.gonol_id]].scale == "character" for member in gonol.relation.members) + assert all(member.role.startswith("character[") for member in gonol.relation.members) kinds = {gonol.relation.kind for gonol in lexical} assert "python.lexical.FSTRING_START" in kinds assert "python.lexical.COMMENT" in kinds @@ -144,7 +189,8 @@ def test_missing_parenthesis_preserves_lower_closures_and_records_hmmm(tmp_path: assert receipt.standing == "hmmm" assert receipt.gonols[-1].relation.kind == "python.source.hmmm" assert reconstruct_source(receipt) == source - assert len(_letters(receipt)) == len(source) + assert len(_characters(receipt)) == len(source) + assert len(_definitions(receipt)) >= len(source) assert any("unmatched opening delimiter" in item for item in receipt.hmmm) assert any(item.startswith("grammar:") for item in receipt.hmmm) replay_python_affixiation(receipt) @@ -173,11 +219,27 @@ def test_receipt_contains_source_relations_not_parser_objects() -> None: assert "TokenInfo" not in encoded assert "<_ast." not in encoded assert "occurrence_addresses" not in encoded - assert all(gonol.scale in {"letter", "lexical-form", "delimiter-construction", "python-construction", "module"} for gonol in receipt.gonols) + assert all( + gonol.scale in { + "character", + "character-definition", + "lexical-form", + "delimiter-construction", + "python-construction", + "module", + } + for gonol in receipt.gonols + ) assert all(type(gonol).__module__ == "python_gonol.model" for gonol in receipt.gonols) replay_python_affixiation(receipt) +def test_runtime_ast_witness_inventory_is_explicit_without_becoming_construction() -> None: + inventory = grammar_witness_inventory() + assert {"Module", "FunctionDef", "TypeAlias", "Match", "TryStar", "FormattedValue"} <= set(inventory) + assert "AST" not in inventory + + def test_exact_ucns_carrier_is_observed_and_drift_fails_closed() -> None: path = Path(__file__).resolve().parents[3] / "libs" / "ucns" / "src" / "ucns" / "public_gonol.py" spec = importlib.util.spec_from_file_location("pinned_ucns_public_gonol", path) @@ -194,8 +256,8 @@ def test_exact_ucns_carrier_is_observed_and_drift_fails_closed() -> None: source_id="fixture/ucns.py", geometry_authority=authority, ) - for letter in _letters(receipt): - properties = dict(letter.relation.properties) + for character in _characters(receipt): + properties = dict(character.relation.properties) assert properties["public_gonol_position"].isdigit() assert properties["public_gonol_function"] == "hmmm" assert "UCNS Public Gonol geometry authority was not supplied" not in receipt.hmmm From 3f9b643de72f497b5d29177d0ea84551af1caaec Mon Sep 17 00:00:00 2001 From: Erin Spencer Date: Sat, 12 Sep 2026 16:49:27 -0700 Subject: [PATCH 07/10] docs(python-gonol): define character-first construction --- research/python-gonol/README.md | 49 ++++++++++++--------------------- 1 file changed, 17 insertions(+), 32 deletions(-) diff --git a/research/python-gonol/README.md b/research/python-gonol/README.md index 42714a0..83ac04d 100644 --- a/research/python-gonol/README.md +++ b/research/python-gonol/README.md @@ -1,47 +1,37 @@ # Python Gonol Construction -Stack-local research implementation for affixiating Python source into gonols -from its lowest admitted units upward. +Stack-local research implementation for affixiating Python source into gonols from its lowest admitted source units upward. ## Construction ```text exact decoded source occurrences - -> letter gonols - -> Python lexical-form gonols + -> character gonols + -> character-definition gonols sharing each character origin + -> Python lexical-form gonols from closed character occurrences -> matched delimiter gonols -> recursive Python grammar-construction gonols -> module gonol ``` -Every source occurrence remains independently addressable. Every larger gonol -contains an identity-bearing relation whose ordered, role-bearing members refer -only to already-closed gonols. A parent consumes a child's atomic identity; the -receipt registry keeps the complete child recoverable. +Every source occurrence remains independently addressable. Every character may close multiple definition gonols without changing the character's identity: Unicode category/name, Python identifier eligibility, exact-token membership, layout role, quote/comment introducer role, and other profile-backed definitions may coexist. Lexical forms consume the already-closed character occurrences atomically; larger constructions consume already-closed lexical or construction gonols atomically. -`letter` is the name of the source-floor construction, not a Unicode alphabetic -classification. Python program text is read as Unicode code points, so spaces, -newlines, digits, operators, and delimiters each begin as their own letter gonol -occurrence too. Nothing is normalized, deduplicated, trimmed, or silently -discarded. +`character` means one exact decoded Unicode scalar occurrence. Spaces, newlines, digits, operators, delimiters, letters, and punctuation are all character gonols. Nothing is normalized, deduplicated, trimmed, or silently discarded. -## Authority boundary +## Recognition boundary ```text METAPAT affixiation semantics -> UCNS geometry and Public Gonol positions -> Python Gonol Python source admission and construction - -> later consumers no authority transfer + -> CPython pinned recognition witness only ``` -This workspace uses CPython 3.12 `tokenize` and `ast` only as recognition -witnesses after the letter floor has closed. Token, AST, compiler, and code -objects never become gonols and never replace the source-built relation graph. -The source bytes, every decoded occurrence, every closed construction, and all -constitutive relations remain visible in the receipt. +The private `_recognition` module uses CPython 3.12 `tokenize` and `ast` after the source occurrence floor has been admitted. Its former public "letter" vocabulary is deprecated and removed from the public receipt: it is now only an internal recognition plan. Token, AST, compiler, and code objects never become gonols and never replace the source-built relation graph. -Standing: **implemented stack-local candidate; not canon and not an independent -release**. +Any valid Python 3.12 file-input source accepted by the pinned recognition witness is traversed recursively without a grammar-node whitelist. The broad Python 3.12 surface fixture remains a regression witness; exhaustive parity against CPython's complete grammar/test corpus remains `hmmm` rather than being claimed from that fixture alone. + +Standing: **implemented stack-local candidate; not canon and not an independent release**. ## Usage guidance @@ -53,8 +43,7 @@ python -m python_gonol --verify source.gonol.json python -m pytest -q tests ``` -Use the bytes entrypoint for files so an encoding declaration and the exact -original bytes remain bound: +Use the bytes entry point for files so an encoding declaration and exact original bytes remain bound: ```python from pathlib import Path @@ -65,18 +54,14 @@ receipt = affixiate_python_bytes(path.read_bytes(), source_id=path.as_posix()) replay_python_affixiation(receipt) ``` -For invalid or unfinished source, the constructor retains all admitted lower -closures, closes a `python.source.hmmm` root, records the exact tokenizer, -delimiter, or grammar boundary, and the CLI exits `2`. This makes a missing end -parenthesis visible without throwing away the construction completed beneath it. +For invalid or unfinished source, the constructor retains every admitted character, character-definition, lexical, and matched-delimiter closure available beneath the failure, closes a `python.source.hmmm` root, records the exact tokenizer/delimiter/grammar boundary, and the CLI exits `2`. A missing end parenthesis therefore remains visible without discarding prior construction. -See [`docs/PYTHON_AFFIXIATION_BOUNDARY.md`](docs/PYTHON_AFFIXIATION_BOUNDARY.md) -for the full contract and [`WORK_GRAPH.json`](WORK_GRAPH.json) for exact inputs. +See [`docs/PYTHON_AFFIXIATION_BOUNDARY.md`](docs/PYTHON_AFFIXIATION_BOUNDARY.md) for the contract and [`WORK_GRAPH.json`](WORK_GRAPH.json) for exact inputs. ## hmmm - exact UCNS geometric operation of Public Gonol function positions; - exact UCNS Möbius-carrier affixiation/coupling law; +- exhaustive parity replay against the complete CPython 3.12 grammar/test corpus; - Python language profiles after Python 3.12 file input; -- streaming/checkpointed receipt materialization for unusually large source - trees. +- streaming/checkpointed receipt materialization for unusually large source trees. From 0bb1e0322f05234caf8e21fe5442f3a7d5498da6 Mon Sep 17 00:00:00 2001 From: Erin Spencer Date: Sat, 12 Sep 2026 16:49:41 -0700 Subject: [PATCH 08/10] docs(python-gonol): replace letter floor with character definitions --- .../docs/PYTHON_AFFIXIATION_BOUNDARY.md | 254 +++++++----------- 1 file changed, 103 insertions(+), 151 deletions(-) diff --git a/research/python-gonol/docs/PYTHON_AFFIXIATION_BOUNDARY.md b/research/python-gonol/docs/PYTHON_AFFIXIATION_BOUNDARY.md index eab7fe8..ec274b1 100644 --- a/research/python-gonol/docs/PYTHON_AFFIXIATION_BOUNDARY.md +++ b/research/python-gonol/docs/PYTHON_AFFIXIATION_BOUNDARY.md @@ -1,163 +1,115 @@ -# Python source affixiation boundary - -**Status:** implemented stack-local Python 3.12 file-input candidate. -**Scope:** determine and implement how Python source affixiates into gonols from -the lowest admitted source units upward. -**Root impact:** none. - -## Governing relation - -Python Gonol Construction applies METAPAT's affixiation meaning: already-bounded -participants remain individually addressable, with identity and provenance -preserved, while their declared relation may close as a higher-scale -object-whole. A closed whole may participate recursively without erasing its -constituents. +# Python affixiation boundary -UCNS owns any exact geometric realization. Python Gonol Construction owns the -Python source profile, admission, lexical relations, delimiter closure, grammar -relations, and receipts. Tokens, AST nodes, compiler objects, and metadata own -none of the construction. - -## Source floor: letters - -Python 3.12 reads program text as Unicode code points. This construction admits -each decoded Unicode scalar occurrence in exact source order and closes it as one -letter gonol. - -`letter` here means the irreducible admitted source occurrence. It includes an -alphabetic character, digit, space, tab, newline, quote, operator, delimiter, or -other exact source scalar without introducing UCNS letter/punctuation subclasses. -The original file bytes, detected Python encoding, decoded-source digest, -occurrence index, line/column span, scalar value, source identity, and optional -observed Public Gonol position remain receipt-bound. - -Repeated equal values have distinct occurrence addresses. No normalization, -identifier NFKC replacement, case folding, trimming, gap deletion, or -deduplication changes the admitted source. - -## Lexical affixiation - -After every letter closes, Python's lexical witness identifies nonempty lexical -forms. Each lexical form closes solely from the exact ordered letter gonols in -its source span. Source gaps that Python uses to separate forms also close as -`python.lexical.INTERTOKEN`; this preserves whitespace the tokenizer does not -emit as a token. - -The lexical relation, exact form class, source length, role order, and CPython -3.12 witness provenance live inside the lexical gonol. The runtime `TokenInfo` -does not. - -Zero-width `INDENT`/`DEDENT` witness events have no source occurrence from which -to form a separate gonol. They therefore remain identity-bearing properties of -the consuming module/source relation. The actual indentation characters remain -letter and lexical gonols. - -## Larger construction affixiation - -Matched `()`, `[]`, and `{}` relations close from lexical gonols and already -closed nested delimiter gonols. Opener, content order, nested multiplicity, and -closer belong inside the delimiter gonol. Missing or mismatched closure becomes -`hmmm`. - -After those closures, the CPython 3.12 grammar witness supplies accepted grammar -relations and child roles. A grammar construction closes over: - -1. already-closed child grammar gonols; -2. already-closed delimiter gonols intersecting its concrete source relation; -3. remaining already-closed lexical forms in its exact source span; and -4. identity-bearing relation properties such as operator or grammar-field roles. - -Construction proceeds postorder. Thus every member identity in a parent receipt -was closed earlier. Overlapping relations—such as a call's argument role and -the parentheses enclosing its arguments—may both reference the same canonical -source occurrences; they do not duplicate those occurrences. - -Some CPython witness records have no independent source span, including an -empty `arguments` record and `TypeIgnore`. Such a record cannot honestly close -as a separate gonol. Its relation is retained as an intrinsic property of the -source-built parent that contains the exact delimiter or comment gonols. The -witness object is then discarded. Decorated function and class spans are -extended to the exact leading `@` lexical gonol so the decorator relation does -not lose its constitutive marker. - -The module gonol closes last and covers every source occurrence, including -comments, blank space, redundant parentheses, and other concrete source material -that abstract grammar nodes omit. - -## Recognition is not substitution - -CPython's tokenizer and AST are bounded recognition witnesses for the Python -3.12 language profile. They determine where accepted lexical and grammar -relations apply. They are discarded after each relation is translated into -source-occurrence membership, closed-gonol references, order, roles, and -provenance. - -The following never serve as a gonol identity or participant: - -- token numbers or `TokenInfo` instances; -- AST instances or AST dumps; -- compiled code objects; -- evaluated literal values; -- symbol-table/compiler objects; -- whole-source hashes standing in for visible construction. - -SHA-256 identifies a complete visible gonol or receipt payload. The visible -payload—not its hash—is the construction. - -## Failure and replay - -Tokenizer, delimiter, or grammar failure does not erase completed work. The -constructor closes a `python.source.hmmm` root over every lower construction it -can honestly retain, records the exact unresolved finding, and sets receipt -standing to `hmmm`. - -Replay checks: - -- exact original bytes and encoding; -- exact source reconstruction from ordered letter gonols; -- one contiguous letter floor with distinct occurrence addresses; -- every member references an earlier closed gonol by exact address and identity; -- every larger gonol's exact span equals the union of its atomic closed-child - spans, without flattening descendant leaves back into the parent; -- every gonol identity and the final receipt digest recompute; and -- the root covers every source occurrence. - -Replay proves deterministic construction integrity only. It does not execute -the source or prove behavior, equivalence, safety, semantic quality, geometry, -measurement validity, or canon. - -## Python profile - -The implemented profile is CPython 3.12 file input. It covers any exact source -accepted by `ast.parse(..., mode="exec", type_comments=True, -feature_version=(3, 12))`, including Python 3.12 type statements/type parameters, -pattern matching, exception groups, asynchronous constructs, comprehensions, -formatted strings, comments within formatted replacement fields, Unicode -identifiers, explicit/implicit line joining, and all ordinary expression and -statement forms. - -The committed broad-surface fixture exercises these relations but does not claim -that one finite fixture enumerates all possible programs. Generic traversal, -rather than a hand-selected statement dispatcher, is what keeps the constructor -open over the complete accepted 3.12 grammar. +Status: **stack-local implemented candidate**. This document defines construction and evidence boundaries; it does not promote Python Gonol Construction to canon. + +## Construction contract + +Python source is admitted from the bottom up: + +```text +source bytes + -> exact decoded source + -> occurrence-specific character gonols + -> character-definition gonols + -> lexical-form gonols + -> delimiter constructions + -> recursive grammar constructions + -> module +``` + +A parent may reference only an already-closed child. Closing a larger gonol never erases the child's identity, order, multiplicity, source span, relation, or provenance. + +### Character floor + +Every decoded Unicode scalar occurrence closes exactly once as `scale="character"`. "Character" is a source-floor term, not an alphabetic category. Newlines, spaces, punctuation, digits, quote marks, operators, and letters are admitted identically as occurrences. + +Each occurrence then receives one or more `scale="character-definition"` gonols. Every definition has exactly one `origin` member: the already-closed character occurrence. Definitions are deliberately non-exclusive. Examples include: + +- `unicode-category = Pd` and `python-exact-token = MINUS` for `-`; +- `python-identifier = start` and `python-identifier = continue` for an identifier-capable character; +- `python-layout = line-break` for a newline; +- `python-delimiter = comment-introducer` for `#`. + +Definitions describe capabilities or source-profile facts. They do not pre-decide the contextual lexical role of an occurrence. + +### Lexical forms + +CPython 3.12 `tokenize` is a recognition witness after character closure. A lexical gonol is built from the exact ordered character gonols spanning the recognized surface form. Inter-token gaps are explicit lexical gonols too, so whitespace/comments/source gaps do not disappear. + +Tokenizer objects never enter the receipt. + +### Delimiters and grammar + +Matched `()`, `[]`, and `{}` relations close from already-closed lexical/enclosed gonols. CPython 3.12 `ast.parse(..., mode="exec", feature_version=(3, 12))` then supplies the grammar witness. The recursive constructor walks arbitrary AST fields rather than dispatching through a hand-maintained Python syntax whitelist. AST nodes themselves are discarded after their source relation has been projected onto closed source-built gonols. + +Spanless parser relations that cannot honestly close as independently sourced gonols remain intrinsic relation properties on the source-owning parent rather than being invented as source objects. + +## Private recognition module + +`python_gonol._recognition` is the original source-recognition implementation. Its historical `letter` scale is no longer a public construction. The current public constructor consumes that module only as an internal plan and rematerializes the receipt as: + +```text +character -> definitions -> lexical -> larger constructions +``` + +`replay_python_affixiation()` rejects any public receipt containing the deprecated `letter` scale or `#letter:` address. + +This removes the deprecated contract while preserving the proven tokenizer/AST witnessing logic. + +## Failure boundary + +Unfinished or invalid Python remains constructible below the unresolved point. Tokenizer failure, delimiter mismatch, or grammar failure produces a `python.source.hmmm` root over the largest honest lower closures. Exact unresolved details are retained in the receipt. + +No error recovery is allowed to invent the missing source. + +## Geometry boundary + +UCNS owns geometry. When an explicit UCNS Public Gonol authority with the pinned digest is supplied, character occurrences record their observed Public Gonol positions. When it is absent, geometry remains `hmmm`. Python Gonol Construction does not invent a UCNS function operation or Möbius coupling law. + +## Replay + +Replay verifies, at minimum: + +1. receipt schema/profile/constructor identity; +2. exact source bytes and decoded-source digests; +3. contiguous occurrence-specific character coverage; +4. at least one definition-space gonol for every character occurrence; +5. definitions reference one already-closed character origin; +6. lexical forms are an exact partition of the closed character occurrences; +7. every larger relation references only already-closed children; +8. every non-character construction span equals the union of its atomic participants; +9. the root covers the complete source; and +10. every gonol identity plus the receipt digest reproduces deterministically. + +Replay proves this construction is internally reproducible. It does not establish semantic quality, runtime equivalence, measurement validity, or canon. + +## Python 3.12 surface evidence + +The constructor is generic over the AST witness rather than an AST-node whitelist, and the regression fixture exercises modern Python 3.12 constructs including type aliases/generics, decorators, positional-only and variadic arguments, async constructs, comprehensions, assignment expressions, pattern matching, exception groups, f-strings, lambdas, slicing, calls, and the operator families. + +That fixture is evidence, not an exhaustive grammar proof. Complete parity replay against CPython's full grammar/test corpus remains `hmmm` until run as its own declared experiment. ## Usage guidance ```bash cd research/python-gonol python -m pytest -q tests -python -m python_gonol ../../some-file.py --out /tmp/some-file.gonol.json -python -m python_gonol --verify /tmp/some-file.gonol.json +python -m python_gonol example.py --out example.gonol.json --pretty +python -m python_gonol --verify example.gonol.json ``` -Prefer `affixiate_python_bytes()` for files. Use -`affixiate_python_source()` only when exact original file bytes are unavailable -or irrelevant to the declared source profile. +Programmatic use: + +```python +from python_gonol import affixiate_python_source, replay_python_affixiation + +receipt = affixiate_python_source("answer = 40 + 2\n", source_id="example.py") +assert replay_python_affixiation(receipt).receipt_digest == receipt.receipt_digest +``` ## hmmm -- exact UCNS function operation for each Public Gonol position; -- exact UCNS Möbius-carrier affixiation/coupling geometry; -- whether and how Python 3.13+ language profiles share or revise this constructor; -- streaming or checkpointed materialization for source large enough that a full - in-memory visible receipt would exceed available resources. +- exact UCNS relation geometry for Python constructions; +- full CPython 3.12 grammar/test-corpus parity replay; +- language profiles after Python 3.12; +- large-source streaming/checkpoint policy. From c538f49c83be42dfa91917b322b30ac577d48667 Mon Sep 17 00:00:00 2001 From: Erin Spencer Date: Sat, 12 Sep 2026 16:50:00 -0700 Subject: [PATCH 09/10] docs(stack): describe Python character-definition floor --- README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index a887beb..b0959bc 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ stack/ │ ├── metapat/ # current METAPAT research + BASE.json │ ├── ucns/ # current UCNS research + BASE.json │ ├── english-gonol/ # English lexical/gonol construction; distinct from EDCM -│ ├── python-gonol/ # Python 3.12 source affixiation from letters upward +│ ├── python-gonol/ # Python 3.12 source affixiation from characters upward │ ├── edcm/ # current EDCM measurement research + BASE.json │ ├── pcea/ # current PCEA research + BASE.json │ ├── ptcna/ # current PTCNA research + BASE.json @@ -79,11 +79,12 @@ English text-domain construction candidate. EDCM may evaluate those outputs but not define the construction. Python Gonol Construction is a separate stack-local component at -`research/python-gonol/`. It admits every exact Python source occurrence as a letter -gonol, then affixiates lexical forms, delimiters, grammar constructions, and the module. -METAPAT owns affixiation semantics; UCNS owns optional consumed geometry; Python Gonol -owns Python source construction. Tokens and AST nodes are recognition witnesses, never -gonol substitutes. +`research/python-gonol/`. It admits every exact Python source occurrence as a character +gonol, closes the occurrence's applicable character-definition gonols, then affixiates +lexical forms, delimiters, grammar constructions, and the module from already-closed +participants. METAPAT owns affixiation semantics; UCNS owns optional consumed geometry; +Python Gonol owns Python source construction. Tokens and AST nodes are recognition +witnesses, never gonol substitutes. ### Change stack structure From e46d9273c52b4e820b1e879c06dfc0588cb8511f Mon Sep 17 00:00:00 2001 From: Erin Spencer Date: Sat, 12 Sep 2026 16:50:09 -0700 Subject: [PATCH 10/10] ci(python-gonol): gate structural consistency before merge --- .github/workflows/python-gonol.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/python-gonol.yml b/.github/workflows/python-gonol.yml index 95e52f9..89fa21a 100644 --- a/.github/workflows/python-gonol.yml +++ b/.github/workflows/python-gonol.yml @@ -34,6 +34,8 @@ jobs: python-version: "3.12" - name: Install pytest run: python -m pip install pytest + - name: Stack structural consistency + run: python tools/check_stack_consistency.py - name: Python Gonol construction tests working-directory: research/python-gonol run: python -m pytest -q tests