Zero-knowledge semantic topology extraction from raw bytes.
Two grammar compression algorithms — one that finds structure (Sequitur), one that finds frequency (MR-RePair) — run on the same input. What both find is the skeleton. What neither finds is the payload. The space between them is where meaning lives.
| Found by RePair (frequent) | Missed by RePair (rare) | |
|---|---|---|
| Found by Sequitur (structured) | Q1: Skeleton — schema, templates, rigid structure | Q2: Episode — behavioral bursts, anomalies |
| Missed by Sequitur (unstructured) | Q3: Vocabulary — scattered constants, dictionary terms | Q4: Payload — the actual content |
Patterns in Q1 (consensus) are the strongest signal — structurally significant AND frequently occurring. Q4 residuals contain the semantic content: identifiers, values, free text. The pipeline uses Q1/Q2/Q3 as slot boundaries, then discovers relational structure from Q4 anchors.
Six steps. No parameters. No regex. No type definitions. Input is bytes.
- Compress twice — Sequitur (online, left-to-right) + MR-RePair (offline, frequency-first)
- Overlay — intersect rulesets, classify every rule into the 2x2 matrix
- Parse — use rules from Q1+Q2+Q3 as boundary templates, extract Q4 slot values
- Rank — measure entropy and frequency for each residual; high entropy + recurrence = anchor
- Link — build a graph connecting slots that share anchor values, annotated by quadrant pair
- Reduce — merge synonymous slots, cluster entangled anchors, identify state machines
The result is the semantic schema — the relational model implicit in the raw bytes.
See CONCEPTS.md for the full theoretical framework, including the boolean algebra of entanglement (identity, rigidity, exclusion).
# Build the compression engines
cd sequitur-c/c++ && make && cd ../..
cd repair-rust && cargo build --release && cd ..
# Run the full pipeline on any file
./signals_and_noise.sh input.log
# Or step by step:
./ingest_dir.sh /path/to/directory/ # Concatenate + compress + overlay
python3 overlay_rules.py output_dir/ # Classify rules, rank descriptors
python3 pivot_link.py input output_dir/ # Extract topology./ingest_dir.sh /path/to/logs/
# Output: /path/to/logs.grammar/
# corpus.txt — concatenated stream with file delimiters
# manifest.tsv — byte offset map (file → corpus position)
# descriptors.txt — ranked patterns with file attribution
# topology.json — relational graph| File | Purpose |
|---|---|
signals_and_noise.sh |
Full pipeline: compress → overlay → pivot-link |
ingest_dir.sh |
Directory walker + concatenator + pipeline runner |
overlay_rules.py |
Rule intersection, 2x2 classification, descriptor ranking |
pivot_link.py |
Streaming topology extraction (steps 3-6) |
grammar_viz.py |
Interactive HTML visualization of grammar rules |
grammar_to_dot.py |
Graphviz DOT export of rule hierarchy |
view_rules.py |
Terminal rule browser |
embed_descriptors.py |
Experimental: embed Q1 patterns + HDBSCAN clustering |
meta_grammar.py |
Experimental: second-order grammar on cluster sequences |
| Engine | Type | Finds | Location |
|---|---|---|---|
| Sequitur (C++) | Online, left-to-right | Structural/syntactic patterns — nesting, templates, repeated blocks | sequitur-c/ |
| MR-RePair (Rust) | Offline, frequency-first | Frequency-dominant content — common strings, repeated values | repair-rust/ |
Both are standard implementations. MR-RePair has a modified -p flag that emits
human-readable expanded rules (the original printed raw Rust Debug format).
Anchors are NOT identified by type (GUID, email, hash). They are identified by two properties measured from the data:
- High entropy — the value is not a common word or repeated constant (those are Q3)
- Recurrence — the value appears in more than one slot position
A value that is high-entropy and recurring is doing the work of an identifier — regardless of its format. No regex needed.
The descriptor file ranks patterns by length × frequency × (2 if consensus):
Rank Score Len Freq Files Pattern
1 48230 42 575 all <structural pattern>
2 31008 24 646 all <frequent template>
...
FILE COVERAGE
File Con Seq Rep
source_a.log 12 8 5
source_b.log 9 11 3
The topology file maps the relational graph:
- Q1↔Q1 shared anchors → foreign keys (structural synonymy)
- Q1↔Q2 shared anchors → entity linked to behavioral episode
- Q2↔Q2 shared anchors → temporally correlated events
- Q3↔Q4 adjacency → vocabulary term contextualizes a payload value
- C++ compiler (for Sequitur:
cd sequitur-c/c++ && make) - Rust toolchain (for MR-RePair:
cd repair-rust && cargo build --release) - Python 3.8+ (stdlib only — no dependencies for core pipeline)
- Optional: embedding model endpoint + hdbscan for
embed_descriptors.py
- Character Energy Analysis — biomechanical analysis of writing system structure
- Voynich Manuscript Analysis — structural decryption applying similar zero-knowledge methodology to an undeciphered manuscript