Skip to content

Fresh redeploy hardening: lint baseline, test floor + CI, #236, #180 - #273

Open
overcuriousity wants to merge 10 commits into
mainfrom
chore/redeploy-hardening
Open

overcuriousity wants to merge 10 commits into
mainfrom
chore/redeploy-hardening

Conversation

@overcuriousity

Copy link
Copy Markdown
Owner

Implements ROADMAP Tier 1 and Tier 2 — the work needed to redeploy fresh and run in production. Spec and plan are committed under docs/superpowers/.

Deliberately out of scope: #248/#247 (memory confidence, episodic/semantic split), the eval harness and published benchmark, FTS5 search, session export/import, Phase 3 god-file splits, and the mechanical refactors #204–#210.

What landed

Area Result
Lint baseline 581 findings → 0; 48 files reformatted; 4 rule ignores documented with rationale
CI New .github/workflows/ci.yml: ruff check + ruff format --check + pytest, all blocking, on push and PR
Tests 29 → 124 (95 new)
Dead deps asyncpg and aiosqlite removed
#236 Nightly skill-documentation condensation removed
#180 Zero-config local embedding fallback

Commit-by-commit

The two style: commits are mechanical and land first so every later diff stays readable. Review them separately from the rest.

  1. style: ruff auto-fixes + formatter (no --unsafe-fixes)
  2. style: remaining findings + documented ignores
  3. test: convergence validators + credential redaction, plus the CI workflow
  4. test: tool-call rescue parsers + NL translator helpers
  5. chore: dead deps + stale docs
  6. fix: Remove skill documentation condensation from dreaming cycle #236
  7. feat: local ONNX embedding provider
  8. feat: Local embedding fallback for zero-config memory resilience #180 wiring

Test coverage added

Only synchronous, IO-free surfaces — no mocks, no pytest-asyncio, no network.

  • All 8 Convergence Protocol stage-2 validators, each with true-positive and false-positive cases. The false positives are the point: an over-firing validator blocks legitimate agent actions.
  • All 6 tool-call rescue parser paths plus rescue_tool_calls end to end, including the case that ordinary prose must never be rescued into a phantom tool call.
  • Credential redaction, config secret extraction, NL detection, JSON control-character repair.

The async orchestrators (run_convergence_protocol, process_tool_call) remain untested — they need a stubbed backend pool, which is follow-up work.

#236 — remove skill condensation

prompt_assembler injects only the skills table of contents, so condensing documentation reduced no prompt size while lossily LLM-rewriting every skill nightly. Losses compound across cycles, and because skill_store embeds summary + documentation, condensation also flattens the embedding signal.

Note the ordering hazard: DREAM_SKILLS_CONDENSATION_PROMPT.txt was listed in prompt_loader.REQUIRED_FILES, where a missing file is a startup failure. Deleting the template without that registry edit would brick startup. Verified with prompt_loader.validate_all().

#180 — zero-config local embeddings

A fresh deployment with no embeddings endpoint previously failed at startup.

  • New infra/local_embeddings.py: lazily loaded ONNX all-MiniLM-L6-v2 (384-dim), cached in data/.embedding_cache/, fetched by URL rather than adding a huggingface_hub dependency.
  • Wired at the single choke point llm_utils.embed(), so all 15 _embed() call sites benefit from one edit.
  • onnxruntime + tokenizers ship as an optional extra (uv sync --extra local-embeddings), keeping the required dependency count flat.
  • Scope is zero-config only. A configured endpoint stays authoritative and is never silently replaced — writing 384-dim vectors into a 1536-dim store would corrupt the index.
  • Dimension guard: LocalVectorBackend now records the provider and dimension that built the store and refuses to open under a mismatched provider, instead of failing as an opaque numpy shape error mid-query. Pre-existing stores adopt their dimension from a stored vector on first open.

Verified end to end: with no endpoint configured, embed() downloads the model and returns 384-dim unit-norm vectors.

Reviewer notes

  • sqlalchemy was kept deliberately. It is not imported directly; workers/scheduler_thread.py:28 uses APScheduler's optional SQLAlchemy jobstore, which APScheduler does not install itself. Removing it breaks scheduling at runtime, not at import.
  • Scoped --unsafe-fixes use. Commit 2 applies ruff's own fixes for a narrow, behaviour-preserving rule selection (SIM102/103/108/117/118, C401, C408, F401, B905) rather than 25 hand edits; every hunk was reviewed individually. zip() sites take strict=False to preserve today's truncating behaviour rather than change it silently. No SIM105 rewrites — the broad except Exception handlers keep the agent loop alive and are ignored with a written rationale.
  • Local build caveat (not a code issue). On Fedora, uv sync fails to build python-olm: CMake 4 rejects its cmake_minimum_required, and system Python 3.14 lacks dev headers. Use CMAKE_POLICY_VERSION_MINIMUM=3.5 uv sync --python 3.12. GitHub runners ship CMake 3.x, so CI is unaffected today.

Verification

ruff check .              All checks passed!
ruff format --check .     86 files already formatted
pytest -q                 124 passed
import wintermute.main    ok

Closes #236
Closes #180

🤖 Generated with Claude Code

overcuriousity and others added 10 commits August 6, 2026 12:46
Covers ROADMAP Tier 1 and Tier 2: lint baseline, test floor plus CI,
dead dependency removal, issue #236 (remove skill condensation from
dreaming), and issue #180 (zero-config local embedding fallback).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six tasks with per-step verification: ruff baseline, blocking CI plus
tests for the convergence validators, redaction, tool-call rescue and NL
translator helpers, dead dependency removal, issue #236, and issue #180.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Mechanical only: ruff check --fix (no --unsafe-fixes) plus ruff format.
565 lint findings fixed, 48 files reformatted. No behavior change; the
test suite (29 passed) and module imports were verified before and after.

Also reformats the Python snippets inside the implementation plan, since
ruff format normalises fenced python blocks in Markdown.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hand-fixed: raise ... from exc in two except blocks (llm_thread,
thread_config), an unused local in onboarding, an unused typing.Optional
import, and two E402 sources — a mid-file `import uuid` in database.py and
a module constant sitting between import blocks in matrix_thread.py.

Applied ruff's own fixes for a narrow, behaviour-preserving rule selection
(SIM102/103/108/117/118, C401, C408, F401, B905); every hunk was reviewed
individually. zip() sites take strict=False to preserve today's truncating
behaviour rather than change it silently.

Ignored with rationale in pyproject.toml: SIM105 (deliberate broad handlers
keep the agent loop alive) and SIM102/SIM108/SIM117 (every remaining
occurrence is one ruff declines to auto-fix because collapsing would drop a
comment or spill a ternary across lines).

ruff check, ruff format --check, and pytest (29 passed) are all clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds blocking CI (ruff check, ruff format --check, pytest) on push and PR.
Covers all eight stage-2 validators with true-positive and false-positive
cases, plus redaction and config secret extraction. 40 new tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Parametrised coverage of all six rescue parser paths plus rescue_tool_calls
end to end, including the negative case that prose must never be rescued
into a phantom tool call. Adds NL detection and JSON control-character
repair tests. 48 new tests; suite is now 117.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Neither package is imported anywhere in wintermute/. sqlalchemy stays:
scheduler_thread.py uses APScheduler's SQLAlchemyJobStore, which APScheduler
itself does not install, so the explicit dependency is load-bearing.

CLAUDE.md: correct the module table to the core/infra/tools/workers/
interfaces package layout, note that tools.py is now a package, drop the
"no test suite" claim, and document the pytest and ruff commands.

ROADMAP.md: check off the Phase 0 items this batch completed and record
that WAL mode was already implemented.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
prompt_assembler injects only the skills table of contents, so condensing
documentation reduced no prompt size while lossily LLM-rewriting every
skill each night — compounding across cycles and flattening the embedding
signal, since skill_store embeds summary + documentation.

Removes the condensation loop, its prompt template, and the template's
entry in prompt_loader.REQUIRED_FILES (a missing required file is a startup
failure, so the registry edit is not optional). Stale retirement and
dedup/merge are unchanged.

Verified: prompt_loader.validate_all() passes, dreaming imports, 117 tests.

Closes #236

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
all-MiniLM-L6-v2 (384-dim) behind an optional local-embeddings extra,
cached under data/.embedding_cache/. Fetched by URL rather than via
huggingface_hub to keep the dependency budget flat. Not wired in yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A fresh deployment with no embeddings endpoint no longer fails at startup.
llm_utils.embed() routes to the local ONNX provider when no endpoint is
configured, which covers all 15 _embed() call sites at once. A configured
endpoint stays authoritative and is never silently replaced.

Adds a dimension guard to LocalVectorBackend: the store records the
provider and vector dimension that built it and refuses to open under a
provider with a different dimension, rather than degrading similarity
search silently. Pre-existing stores adopt their dimension from a stored
vector on first open.

skill_store gets the same relaxed startup check.

Verified end to end: with no endpoint configured, embed() downloads the
model and returns 384-dim unit-norm vectors.

Closes #180

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 6, 2026 11:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Comment thread .github/workflows/ci.yml
Comment on lines +10 to +32
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Install uv
uses: astral-sh/setup-uv@v5
with:
enable-cache: true

- name: Set up Python
run: uv python install 3.12

- name: Install dependencies
run: uv sync --all-groups

- name: Lint
run: uv run ruff check .

- name: Format check
run: uv run ruff format --check .

- name: Test
run: uv run pytest -q
CREDENTIALS_FILE.write_text(
json.dumps(creds, indent=2), encoding="utf-8"
)
CREDENTIALS_FILE.write_text(json.dumps(creds, indent=2), encoding="utf-8")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove skill documentation condensation from dreaming cycle Local embedding fallback for zero-config memory resilience

3 participants