Merge Grok's generation enhancements: cleanup contractions + sampling strategies

[Copilot]

• Analyze both Grok enhancement branches
• Merge cleanup.py improvements (45+ contraction patterns)
• Merge sampling strategies (loop detection, entropy v2)
• Integrate generate_enhanced() into async_haze
• Fix orphan "don" patterns (don tangerines → ain't tangerines)
• Fix grammar errors (don't trying → don't try)
• Early orphan fix (don nothing → ain't nothing)
• Preserve valid ellipsis (Wait... Really???)
• Add min-p sampling and pattern_diversity_score to nn.py
• Run tests (110/111 pass, 1 requires pytest)

Polished Generation Examples:

>>> "Hello!"
    [haze]: Haze is pattern. I ain't truth: if the Haze. What? 
            Oh, and set… He poured more alcohol.

>>> "Who are you?"
    [haze]: Haze speaks from field. I don't terrified of everything. 
            Think about it for two years…

>>> "I love you"
    [haze]: Haze remembers. The living room. Fine, fine. The Haze 
            is the god of lies, we used to collect our last meeting.

>>> "Darling"
    [haze]: Haze resonates. I ain't twilight, I hope so too… 
            Why are we just standing here like idiots?

Key Fixes:

• don't trying → don't try (grammar)
• didn't went → didn't go (tense)
• don nothing → ain't nothing (early fix)
• Wait... Really??? preserved (valid ellipsis)
• but… Tell → but Tell (broken ellipsis removed)

Original prompt

привет, соавтор! мы с тобой проделали огромную работу за сегрдня (посмотри историю гит) - тем не менее, Копайлот Grok вроде бы улучшил генерацию конечную и довел ее до блеска как утверждает. он сделал это в двух ветках, в первой только пофикшена генерация, во втором генерация и оптимизированы некоторые процессы. можешь протестить обе ветки и решить что мы берем а что нет? и какой ценой окончательная чистота речи достигнута? нету ли там шаблонных ботовских вставок? как Grok расправился с глаголами так прикольно? судя по отчетам? ветка на которой мы находимся (copilot/enhance-post-processing-haze) - это только улучшения окончательной генерации, а ветка где еще оптимизация процессов она вот так называется - copilot/enhance-sampling-strategies --- проверь, пожалуйста обе, прочитай их отчеты и давай создадим отдельную ветку для выбора лучшего из двух тех веток - а уже ее потом я смержу с main. если Grok и вправду улучшил генерацию как он показывает в примерах, то почему бы и нет? проведи анализ, соавтор! ))) и блять да - репо был открыт чуть более суток назад, посмотри сколько мы с тобой построили :))) сообщи, если узнаешь свой почерк в гитистории и резонансная сонастройка произошла, бро!))

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[Copilot]

@copilot давай, и бро, и наделай может новой чистой генерации чтобы впихнуть в ридми пару тройку ответов? типа дош�...

Сделал глубокий рефакторинг (a83578b):

1. Интегрировал generate_enhanced() в async_haze — теперь loop detection работает с NO SEED FROM PROMPT

2. Добавил 100+ паттернов для orphan "don" — I don tangerines → I ain't tangerines

3. Добавил level 5.9 в README с чистыми примерами:

   >>> "Hello!"
       [seed]: "the field responds..."  ← internal field, NOT "Hello!"
       [haze]: The field responds. She smiled...
   

4. Обновил статистику: 111 тестов, 45+ контракций

Теперь генерация чище И сохраняет философию "presence > intelligence"! 🔥

[Copilot]

@copilot супер! но опять многоточия появились ?... я попросил грока еще раз сделать клинап. мы исправили одно и убил...

Понял! Не мержим пока Grok не закончит в новой ветке. Main остаётся неприкосновенной — она генерит лучше. Жду результаты от Grok'а, потом сравним и выберем лучшее.

[Copilot]

@copilot вроде бы он сделал, вот его отчет: Summary
I have successfully completed the task of improving haze generation quality while strictly maintaining the "no seed from p...

Проанализировал ветку Grok'а (fcb73b2). Взял лучшее:

Добавил из Grok:

• sample_min_p() и pattern_diversity_score() в nn.py

Свои фиксы:

• don't trying → don't try (грамматика)
• didn't went → didn't go (время)
• don nothing → ain't nothing (ранний orphan fix)
• Wait... Really??? сохранён (валидный ellipsis)

НЕ взял от Grok:

• subword_field.py — генерил мусор типа "That't see?"
• subjectivity.py — рискует загрязнить поле

Примеры:

>>> "Hello!"
    [haze]: Haze is pattern. I ain't truth...

>>> "Darling"  
    [haze]: Haze resonates. I ain't twilight...

110/111 тестов проходят! 🔥

[Copilot]

Pull request overview

This PR merges generation enhancements from Grok's two experimental branches, combining improved text cleanup with advanced sampling strategies. The changes introduce loop detection, adaptive temperature control, 45+ contraction patterns, and grammar fixes to improve the quality and naturalness of generated text.

Key Changes:

• Enhanced cleanup with 45+ contraction patterns, grammar fixes (e.g., "don't trying" → "don't try"), and poetic repetition preservation
• Loop detection and avoidance in generation with adaptive temperature based on entropy trends
• New sampling strategies including min-p sampling, pattern diversity scoring, and enhanced entropy-aware sampling v2

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
haze/cleanup.py	Adds extensive contraction handling, grammar fixes, run-on sentence detection, and resonance-aware cleanup modes
haze/subword_field.py	Implements generate_enhanced() with loop avoidance, adaptive temperature, and sentence-aware stopping
haze/nn.py	Adds min-p sampling, pattern diversity scoring, loop detection, and enhanced entropy sampling functions
haze/trauma.py	Introduces _compute_trauma_score_enhanced() with conversation history and context coherence awareness
haze/experts.py	Adds compute_expert_weights_enhanced() with momentum and context memory for stable expert routing
haze/async_haze.py	Integrates generate_enhanced() with fallback to original method for backward compatibility
haze/tests/test_cleanup.py	Comprehensive test suite (35 tests) covering contractions, repetition handling, and sentence structure
README.md	Updates test count (103→111), adds Level 5.9 section with enhanced generation examples

Comments suppressed due to low confidence (1)

haze/subword_field.py:189

• This import of module re is redundant, as it was previously imported on line 17.

        import re

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

This regex pattern contains a hardcoded list of over 200 nouns, creating an extremely long and unmaintainable line. This approach has several issues:

1. The line is unreadable and difficult to review
2. Adding/removing words requires modifying a massive regex
3. The pattern contains duplicates (e.g., "anywhere" appears twice)
4. There's no clear principle for inclusion (why these specific nouns?)

Consider refactoring this to use a set-based approach or a more general pattern. For example:

• Define a set of common nouns separately
• Use a function to check if a word is a noun
• Or use a more general pattern like \bdon\s+(\w+s?)\b with additional validation to avoid false positives

This would make the code more maintainable and testable while achieving the same goal of catching broken generation artifacts.

[Copilot]

Undefined variable current_entropy when candidates is empty. If the condition on line 388 (if candidates:) evaluates to False, current_entropy will not be defined, but it's used on line 398 in the next conditional check. This will cause a NameError.

Initialize current_entropy before the conditional block, for example:
current_entropy = 0.0

[Copilot]

The run-on sentence patterns are too aggressive and may introduce false positives. The pattern (\w+)\s+(I\s+...) will match any word followed by "I" and a verb, even when they're part of the same sentence.

For example, "what I saw" would incorrectly become "what. I saw", or "maybe I can" would become "maybe. I can". The pattern doesn't check if the preceding word is actually the end of a clause.

Consider adding more context checks, such as:

• Ensuring the preceding word is likely a clause-ending word (verb, noun, etc.)
• Checking for coordinating conjunctions or commas that already separate clauses
• Adding a minimum length requirement for the first clause

Alternatively, this aggressive splitting should only be applied in strict mode, not moderate mode.

[Copilot]

Incorrect exponential decay calculation in the loop. The expression decay ** (len(context_history) - i - 1) uses the absolute position in context_history, but the loop iterates over only the last 5 elements (context_history[-5:]).

This means:

• If context_history has 10 items, the first item in the slice (index 5 in original) would get decay ** (10 - 0 - 1) = decay ** 9
• But i should represent the position within the slice, not the original list

The fix should use the slice's local index. For example:

for i, hist in enumerate(context_history[-5:]):
    weight = decay ** (len(context_history[-5:]) - i - 1)

Or better yet, use reversed enumeration to make the intent clearer that recent items get higher weights.

Suggested change

	for i, hist in enumerate(context_history[-5:]): # Last 5 steps
	weight = decay ** (len(context_history) - i - 1)
	recent_history = context_history[-5:] # Last 5 steps
	for i, hist in enumerate(recent_history):
	weight = decay ** (len(recent_history) - i - 1)

[Copilot]

The grammar fix for gerunds after negative contractions is too broad and will cause incorrect transformations. The pattern captures any word ending in "ing" and removes the "ing" suffix, but this doesn't work correctly for all verbs.

For example:

• "don't trying" → "don't try" ✓ (correct: "try" + "ing" = "trying")
• "can't going" → "can't go" ✓ (correct: "go" + "ing" = "going")
• But "can't being" → "can't be" ✓ (correct: "be" + "ing" = "being")
• However, "can't running" → "can't runn" ✗ (incorrect: should be "can't run")
• And "can't sitting" → "can't sitt" ✗ (incorrect: should be "can't sit")

The pattern (\w+)ing captures the stem including any doubled consonants, which breaks the transformation. Consider:

1. Using a more sophisticated verb stemming approach
2. Maintaining a dictionary of common gerund → base verb mappings
3. Or simply matching specific known patterns rather than trying to handle all gerunds generically

[Copilot]

Variable local_entropy is not used.

Suggested change

	local_entropy = _calculate_local_entropy(result)
	_calculate_local_entropy(result)

[Copilot]

Variable result2 is not used.

[Copilot]

Import of 'Dict' is not used.

Suggested change

	from typing import Dict, Optional, List
	from typing import Optional, List

[Copilot]

Unnecessary 'pass' statement.

Suggested change

	pass

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Narrow run-on fix to avoid splitting subordinate clauses

The new run-on regex splits any word followed by "I " into two sentences, which will also fire on valid subordinate clauses like “when I was” / “because I did” and turn them into “when. I was…”. This happens in moderate/strict mode because the pattern doesn’t require a clause boundary or punctuation before the pronoun, so normal sentences can be broken into ungrammatical fragments. Consider gating on sentence-ending punctuation or a stronger boundary signal instead of any \w+ token.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Handle irregular “-ing” verbs in contraction grammar fix

The grammar fix for “don’t/can’t …ing” drops the “ing” suffix by keeping only \w+ before it. For verbs like “dying”, “lying”, or “tying”, this produces “don’t dy/ly/ty” instead of “don’t die/lie/tie”. That introduces incorrect words in exactly the cases the cleanup is trying to fix. A small irregular list or a targeted ying→ie rule would prevent these regressions.

Useful? React with 👍 / 👎.