Skip to content

Commit d24b05b

Browse files
feat(english-gonol): add relational density scales, determinable not stored
- Add on-demand streaming scales through the constructed relations: word_to_words_in_definitions (origin word -> target word counts in its definitions) and characters_in_word_to_characters_in_definitions (origin word -> character counts in its definitions via word expansion plus whitespace scalars) - Relational scales are determinable at any time and recorded with stored=false; never materialized into the density table - relational_receipt streams one scale and returns a compact row count plus sha256 digest without storing rows - Semantic valuation now lists all six scales as determinable - Full-corpus on-demand receipts (not stored): word_to_words_in_definitions: 1,611,237 rows, sha 0e17c96945494e03 characters_in_word_to_characters_in_definitions: 2,509,227 rows, sha 1a7543f5bae9dce0 - Regenerated stored evidence (v2.1.0), receipt 6f54b5fc379a2ae7188603accf5d0ff9ae78e90452bcc5438a3815c80cdeb1a5
1 parent 513c2d3 commit d24b05b

4 files changed

Lines changed: 290 additions & 27 deletions

File tree

research/english-gonol/english_gonol/density_run.py

Lines changed: 168 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
# module_kind: measurement
55
# summary: exact per-scalar occurrence counts through every constructed occurrence relation of the verified v2 construct, recorded as semantic-valuation inputs at every scale
66
# owner: Erin Spencer
7-
# public_surface: SCHEMA, VERSION, DensityError, DensityResult, build_density, run
7+
# public_surface: SCHEMA, VERSION, DensityError, DensityResult, build_density, run, iter_word_to_words_in_definitions, iter_characters_in_word_to_characters_in_definitions, relational_receipt
88
# internal_surface: read-only construct inspection, exact Fraction arithmetic, canonical receipt serialization
99
# auth_boundary: measures the already-constructed English Gonol v2 database; does not retokenize, normalize, or re-admit corpus text; supplies no geometry
1010
# storage_boundary: one caller-selected out directory with density.json and density.md
@@ -56,6 +56,18 @@
5656
# class: doctrine
5757
# since: 2026-09-15
5858
#
59+
# id: density_relational_scales_are_determinable_not_stored
60+
# given: word-to-words-in-definitions and characters-in-word-to-characters-in-definitions scales
61+
# then: each is streamed on demand from the construct and recorded as determinable with stored=false, never materialized into the density table
62+
# class: correctness
63+
# since: 2026-09-15
64+
#
65+
# id: density_relational_receipt_is_compact
66+
# given: one relational scale
67+
# then: streaming its rows yields a compact row count and sha256 digest without storing the rows
68+
# class: correctness
69+
# since: 2026-09-15
70+
#
5971
# id: density_stays_outside_the_construct
6072
# given: a density result
6173
# then: it is a separate measurement table that modifies no construct table and invents no geometry
@@ -103,10 +115,10 @@
103115
import json
104116
from pathlib import Path
105117
import sqlite3
106-
from typing import Any
118+
from typing import Any, Iterator
107119

108120
SCHEMA = "english-gonol.corpus-native-density"
109-
VERSION = "2.0.0"
121+
VERSION = "2.1.0"
110122
CONSTRUCT_SCHEMA = "english-gonol.full-construct"
111123

112124
_HMMM = (
@@ -116,6 +128,18 @@
116128

117129
_SCALES = ("character", "word", "definition", "semantic")
118130

131+
_RELATIONAL_SCALES: dict[str, str] = {
132+
"word_to_words_in_definitions": (
133+
"for each origin word, exact counts of word identities appearing in "
134+
"its definitions through definition_components word references"
135+
),
136+
"characters_in_word_to_characters_in_definitions": (
137+
"for each origin word, exact counts of character identities appearing "
138+
"in its definitions through definition_components (word references "
139+
"expanded through word_characters plus whitespace character references)"
140+
),
141+
}
142+
119143

120144
class DensityError(ValueError):
121145
"""Raised when the density measurement fails closed."""
@@ -129,6 +153,7 @@ class DensityResult:
129153
builder: dict[str, Any]
130154
semantic_valuation: dict[str, Any]
131155
scales: dict[str, dict[str, Any]]
156+
relational_scales: dict[str, dict[str, Any]]
132157
hmmm: str
133158
receipt_sha256: str
134159

@@ -140,6 +165,7 @@ def as_dict(self) -> dict[str, Any]:
140165
"builder": self.builder,
141166
"semantic_valuation": self.semantic_valuation,
142167
"scales": self.scales,
168+
"relational_scales": self.relational_scales,
143169
"hmmm": self.hmmm,
144170
"receipt_sha256": self.receipt_sha256,
145171
}
@@ -303,6 +329,114 @@ def _scale_record(
303329
"""
304330

305331

332+
def _open_construct(construct_db: Path) -> sqlite3.Connection:
333+
return sqlite3.connect(f"file:{construct_db}?mode=ro", uri=True)
334+
335+
336+
def iter_word_to_words_in_definitions(
337+
construct_db: Path,
338+
) -> Iterator[dict[str, Any]]:
339+
"""Stream exact word-to-word definition counts on demand.
340+
341+
One row per ``(origin word, target word)`` pair: the exact count of
342+
target word identities inside the origin word's definitions, taken only
343+
from ``definitions`` + ``definition_components`` word references.
344+
"""
345+
346+
connection = _open_construct(construct_db)
347+
try:
348+
cursor = connection.execute(
349+
"""
350+
SELECT w.id AS word_id, w.surface AS surface,
351+
dc.word_id AS target_word_id, COUNT(*) AS count
352+
FROM words AS w
353+
JOIN definitions AS d ON d.origin_word_id = w.id
354+
JOIN definition_components AS dc
355+
ON dc.definition_id = d.id AND dc.word_id IS NOT NULL
356+
GROUP BY w.id, dc.word_id
357+
ORDER BY w.id, dc.word_id
358+
"""
359+
)
360+
for word_id, surface, target_word_id, count in cursor:
361+
yield {
362+
"word_id": word_id,
363+
"surface": surface,
364+
"target_word_id": target_word_id,
365+
"count": count,
366+
}
367+
finally:
368+
connection.close()
369+
370+
371+
def iter_characters_in_word_to_characters_in_definitions(
372+
construct_db: Path,
373+
) -> Iterator[dict[str, Any]]:
374+
"""Stream exact character-to-character definition counts on demand.
375+
376+
One row per ``(origin word, character scalar)`` pair: the exact count of
377+
character identities inside the origin word's definitions, taken only
378+
from ``definition_components`` — word references expanded through
379+
``word_characters`` and whitespace character references taken directly.
380+
"""
381+
382+
connection = _open_construct(construct_db)
383+
try:
384+
cursor = connection.execute(
385+
"""
386+
SELECT x.wid AS word_id, c.scalar AS scalar, SUM(x.cnt) AS count
387+
FROM (
388+
SELECT w.id AS wid, wc.character_id AS cid, COUNT(*) AS cnt
389+
FROM words AS w
390+
JOIN definitions AS d ON d.origin_word_id = w.id
391+
JOIN definition_components AS dc ON dc.definition_id = d.id
392+
JOIN word_characters AS wc ON wc.word_id = dc.word_id
393+
WHERE dc.word_id IS NOT NULL
394+
GROUP BY w.id, wc.character_id
395+
UNION ALL
396+
SELECT w.id AS wid, dc.character_id AS cid, COUNT(*) AS cnt
397+
FROM words AS w
398+
JOIN definitions AS d ON d.origin_word_id = w.id
399+
JOIN definition_components AS dc ON dc.definition_id = d.id
400+
WHERE dc.character_id IS NOT NULL
401+
GROUP BY w.id, dc.character_id
402+
) AS x
403+
JOIN characters AS c ON c.id = x.cid
404+
GROUP BY x.wid, c.scalar
405+
ORDER BY x.wid, c.id
406+
"""
407+
)
408+
for word_id, scalar, count in cursor:
409+
yield {"word_id": word_id, "scalar": scalar, "count": count}
410+
finally:
411+
connection.close()
412+
413+
414+
def relational_receipt(construct_db: Path, scale: str) -> dict[str, Any]:
415+
"""Stream one relational scale and return a compact digest, without storing it."""
416+
417+
if scale not in _RELATIONAL_SCALES:
418+
raise DensityError(f"unknown relational scale: {scale}")
419+
if scale == "word_to_words_in_definitions":
420+
iterator = iter_word_to_words_in_definitions(construct_db)
421+
else:
422+
iterator = iter_characters_in_word_to_characters_in_definitions(construct_db)
423+
424+
digest = sha256()
425+
rows = 0
426+
for row in iterator:
427+
digest.update(
428+
json.dumps(row, sort_keys=True, separators=(",", ":")).encode("utf-8")
429+
)
430+
rows += 1
431+
return {
432+
"scale": scale,
433+
"description": _RELATIONAL_SCALES[scale],
434+
"stored": False,
435+
"rows": rows,
436+
"sha256": digest.hexdigest(),
437+
}
438+
439+
306440
def build_density(construct_db: Path, construct_manifest: Path) -> DensityResult:
307441
"""Measure per-scale letter density through the constructed relations only."""
308442

@@ -351,12 +485,26 @@ def build_density(construct_db: Path, construct_manifest: Path) -> DensityResult
351485
),
352486
}
353487

488+
relational_scales = {
489+
name: {
490+
"description": description,
491+
"stored": False,
492+
"determinable": True,
493+
"iterator": (
494+
"iter_word_to_words_in_definitions"
495+
if name == "word_to_words_in_definitions"
496+
else "iter_characters_in_word_to_characters_in_definitions"
497+
),
498+
}
499+
for name, description in _RELATIONAL_SCALES.items()
500+
}
501+
354502
semantic_valuation = {
355503
"counts_are": (
356504
"exact per-scalar occurrence counts through the constructed "
357505
"occurrence relations at every scale"
358506
),
359-
"determinable_at_scales": list(_SCALES),
507+
"determinable_at_scales": list(_SCALES) + list(_RELATIONAL_SCALES),
360508
"scale_totals": {
361509
"character": character_total,
362510
"word": word_total,
@@ -381,6 +529,7 @@ def build_density(construct_db: Path, construct_manifest: Path) -> DensityResult
381529
},
382530
"semantic_valuation": semantic_valuation,
383531
"scales": scales,
532+
"relational_scales": relational_scales,
384533
"hmmm": _HMMM,
385534
}
386535
receipt = sha256(
@@ -394,6 +543,7 @@ def build_density(construct_db: Path, construct_manifest: Path) -> DensityResult
394543
builder=payload["builder"],
395544
semantic_valuation=semantic_valuation,
396545
scales=scales,
546+
relational_scales=relational_scales,
397547
hmmm=_HMMM,
398548
receipt_sha256=receipt,
399549
)
@@ -429,6 +579,20 @@ def _render_markdown(result: DensityResult) -> str:
429579
]
430580
for scale in _SCALES:
431581
lines.append(f"| {scale} | {result.scales[scale]['total']} |")
582+
lines.extend(
583+
[
584+
"",
585+
"## Relational scales (determinable, not stored)",
586+
"",
587+
]
588+
)
589+
for name, record in result.relational_scales.items():
590+
lines.extend(
591+
[
592+
f"- {name}: {record['description']}",
593+
f" stored: {record['stored']}, determinable: {record['determinable']}",
594+
]
595+
)
432596
for scale in _SCALES:
433597
record = result.scales[scale]
434598
lines.extend(

research/english-gonol/experiments/density-v0/density.json

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

research/english-gonol/experiments/density-v0/density.md

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,12 @@ semantic-valuation inputs, determinable at any and every scale.
1111
- construct version: 2.0.0
1212
- construct receipt: 12277b4959c0c72b7af12097b8a77bf91866bbf669e7f4ac07b6a5f1426ebb57
1313
- construct database sha256: af609bbba504f95e349f3c1a30aa42923acc8e48c1e67bb481521dbf7e49162b
14-
- density receipt: ba0abf4a3043a6487d49042d0504432e3d4ba10788a67dad90e845f46c022f6a
14+
- density receipt: 6f54b5fc379a2ae7188603accf5d0ff9ae78e90452bcc5438a3815c80cdeb1a5
1515

1616
## Semantic valuation
1717

1818
- counts: exact per-scalar occurrence counts through the constructed occurrence relations at every scale
19-
- determinable at scales: character, word, definition, semantic
19+
- determinable at scales: character, word, definition, semantic, word_to_words_in_definitions, characters_in_word_to_characters_in_definitions
2020
- valuation weight: None
2121

2222
## Scale totals
@@ -28,6 +28,13 @@ semantic-valuation inputs, determinable at any and every scale.
2828
| definition | 11085231 |
2929
| semantic | 7924690 |
3030

31+
## Relational scales (determinable, not stored)
32+
33+
- word_to_words_in_definitions: for each origin word, exact counts of word identities appearing in its definitions through definition_components word references
34+
stored: False, determinable: True
35+
- characters_in_word_to_characters_in_definitions: for each origin word, exact counts of character identities appearing in its definitions through definition_components (word references expanded through word_characters plus whitespace character references)
36+
stored: False, determinable: True
37+
3138
## Character scale counts
3239

3340
Relation: characters identity: each admitted scalar exists once

0 commit comments

Comments
 (0)