Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 13 additions & 9 deletions .claude/skills/add-feature/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,19 +67,23 @@ if data is None: # Can this actually be None here?

## InsightWeaver Patterns

### Async Operations
### Source adapters are async; everything after ingestion is not
```python
async def fetch_and_process():
data = await collector.fetch() # Await external calls
return process(data) # Sync processing is fine
class MyAdapter: # src/sources/base.py describes the contract
async def fetch(self, since): # raise SourceUnavailable; never return [] for an error
...
```

### Claude API Integration
### The one model call (2026-09-22, backlog task 029)
```python
# Always handle markdown-wrapped JSON
response = await client.analyze(prompt)
# Parser handles: ```json {...} ``` and raw JSON
data = parse_claude_response(response)
from src.llm.claude_client import ClaudeClient, ModelCallFailed

client = ClaudeClient("triage") # key from the OS keychain; every request audited
response = client.analyze(system, user, effort="medium", output_schema=SCHEMA)
# response.text is JSON constrained by SCHEMA; validate it with pydantic anyway.
# ModelCallFailed.outcome is "answered", "rejected" or "unavailable"; only an
# answered failure is a fact about the pair. src/evidence/claude_adjudicator.py
# is the only caller and should stay the only one.
```

### Database Operations
Expand Down
96 changes: 96 additions & 0 deletions .claude/skills/onboard/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
---
name: onboard
description: Interview the operator for the decisions they are carrying and the claims they want watched, transcribe the answers into position.yaml and watches.yaml in the private directory, validate them, and run `insightweaver watch sync`. Use on first setup, when a decision is added or closed, or when the operator asks to review the Position.
allowed-tools: Read, Write, Bash(insightweaver:*), Bash(ls:*), Bash(cat:*), Bash(diff:*), Bash(mkdir:*), Bash(python:*)
---

# Onboard

You are transcribing, not authoring. Every decision, deadline, stake, claim, belief, trigger
word, expiry and staleness window in the files you write is something the operator said in this
conversation and heard read back. If an answer is missing, ask; if it is vague, ask for the
concrete form; never fill a field yourself. This is invariant 6 of `docs/CONCEPTS.md`, the system
never authors its own watches, applied to the one place where it is easiest to break: a drafting
interview is one helpful suggestion away from being a generator.

The files are private. They name real decisions and real deadlines. They live outside the
repository at `POSITION_PATH` and `WATCHES_PATH` (default `~/.config/insightweaver/`), ideally
under git in the operator's private repo, because a review's value is the diff. This
repository's `.gitignore` refuses `position.yaml` and `watches.yaml` at the root and under
`config/`. Do not write them anywhere under the checkout.

## 1. Find what exists

```bash
insightweaver watch list
ls -la ~/.config/insightweaver/ 2>/dev/null
echo "POSITION_PATH=$POSITION_PATH WATCHES_PATH=$WATCHES_PATH"
```

If a Position exists, read it and start from it: show each decision and ask whether it is still
live, closed, or changed. If none exists, say so and start the interview. Read
`config/position.example.yaml` and `config/watches.example.yaml` once for the exact field names
and the trigger rules; the loaders reject any field that is missing or malformed, and default
nothing.

## 2. Decisions

A decision is something the operator will do or not do by a date. Ask, one at a time, and stop
when the operator says there are no more:

1. What is the decision? (`name`: one sentence they could act on.)
2. By when? (`deadline`: a date. A decision with no date is an interest, and the loader refuses
it; do not invent one and do not accept "soon".)
3. What is at stake if it goes the wrong way? (`stake`: their words, one or two sentences.)
4. A short key for it (`key`: a slug such as `renew-authorization`; propose one built from
their words and let them confirm).

Read the list back. Then set `reviewed` to today's date and `version: 1`. If the file
runs past two pages the loader warns; that is drift from stakes into interests, so ask which
decisions are really dated before writing more.

## 3. Watches

For each decision, ask: what would you want to know before that date that would change what you
do? Each answer that is a claim about the world becomes one watch. Ask the fields in this order
and write down the answers verbatim:

1. `claim`: the thing that is or is not true, phrased so that afterwards they could say which.
If the answer is a topic ("FedRAMP changes"), ask for the claim inside it.
2. `belief`: their probability now, 0.0 to 1.0. Do not suggest a number.
3. `so_what.because`: what changes for the decision if the claim resolves.
4. `triggers`: the words, names and sources that would appear in a document that bears on the
claim. Explain the rule once: a trigger is a list of clauses; within a clause every populated
field must match and any listed value will do; across clauses any one firing is enough;
`terms` and `entities` match whole words, an all-caps term matches case-sensitively, `sources`
must equal a registered source name exactly (case aside): the `name` fields in
`config/feeds/*.json`, or `Federal Register - Documents API`. The loader does not check
source names, so check them yourself against those files before writing; a misspelt name is
a clause that never fires and no command will say so. Ask for the actual words. A trigger
that cannot be compiled never fires, so do not accept a sentence.
5. `expires`: when the question stops mattering, usually on or before the decision's deadline.
6. `staleness_alert_days`: after how many days of nothing routed the silence should be reported.

A watch serves exactly one decision (`so_what.decision` is that decision's key). Give each an
`id` slug built from the claim and confirmed by the operator.

## 4. Write, validate, sync

Write both files with the exact structure of the example files. If a file already exists, show
`diff` against the new content and ask before replacing it. Then:

```bash
insightweaver watch sync
insightweaver watch list
```

`watch sync` validates both files together and refuses the whole set if any field is missing or
malformed (it does not check source names; see step 3); fix what it names and run it again. Do not work around a refusal by changing what the operator said
without asking them. When it succeeds, tell the operator what was added, updated or retired, and
that `insightweaver run` is the morning command.

## What this skill does not do

It does not rank decisions, suggest beliefs, propose claims the operator did not state, or write
a watch for a decision the operator did not connect it to. It does not run `ingest`, `route` or
`adjudicate`. It does not touch the database except through `watch sync`.
196 changes: 93 additions & 103 deletions .claude/skills/review-tests/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,125 +15,115 @@ Review tests to ensure they describe behaviors, not implementations.
### Good Test Characteristics

1. **Describes behavior, not methods**
- Test name: `test_citation_map_overrides_claude_response` (behavior)
- Not: `test_build_synthesis_task_with_citations_returns_tuple` (implementation)

2. **Decoupled from implementation details**
- Test the contract, not the internals
- If refactoring breaks tests but not behavior, tests are too coupled

3. **Follows Arrange-Act-Assert structure**
```python
# Arrange: Set up preconditions
articles = [{"id": 1, "title": "Test"}]

# Act: Execute the behavior
result = await synthesizer.synthesize()

# Assert: Verify outcomes
assert result["status"] == "success"
```

4. **Tests at boundaries, not internals**
- Public API boundaries
- Integration points (database, external APIs)
- Error boundaries

5. **Meaningful assertions**
- Assert on outcomes users care about
- Not: `assert mock.called` (verifies plumbing)
- Yes: `assert result["citation_map"]["1"]["title"] == "Expected Title"` (verifies value)
- Test name: `test_a_second_run_asks_nothing` (behavior: idempotency)
- Not: `test_pending_pairs_calls_query_filter` (implementation)

2. **Can actually fail**
- A test earns its place only if a plausible bug turns it red.
- `tests/routing/test_regression.py` plants exactly 20 whole-word matches
among 1000 observations and 100 look-alikes, then asserts on the
identity of the routed set, not just a count -- a routing predicate
that is too loose bills real model calls, so the test is pinned with a
number and was verified to fail (61 routed) when the word-boundary
anchors were emptied out.
- `tests/brief/test_select.py` inserts rows out of the order the brief
must print them (`test_clusters_and_citations_are_ordered_by_date_before_hash`
even gives the earlier observation the *larger* hash in both pairs), so
a missing or wrong `ORDER BY` fails the test instead of passing by
accident.

3. **No tautologies**
- Don't assert that a mock returned what you told it to return.
- Assert on what the code under test *did* with that input: what it
wrote, what it computed, what it left alone.
- `tests/position/test_ledger.py::test_only_sync_and_the_two_commands_write_belief_or_resolution`
is the extreme version of this: instead of trusting that every call
site remembers to go through `record_belief`/`resolve_watch`, it scans
every file under `src/` for the patterns that would write to the
ledger and asserts the only matches are the three allowed writers, by
name.

4. **Mocks only at the boundary**
- Mock the external boundary (the model API, the network), never an
internal method of the unit under test.
- `tests/evidence/test_adjudicate.py`'s `FakeClient` stands in for the
Claude API client itself -- the one seam that would otherwise cost
money and require a key -- and returns scripted `ModelResponse` or
raises a scripted `ModelCallFailed`. Everything downstream (routing,
the adjudicator, the ledger writes) is the real code.
- `tests/sources/test_runner.py`'s `FakeAdapter` plays the same role for
a source: it returns a scripted list of items, an empty list, or
raises `SourceUnavailable`, and the runner, the silence watchdog and
the database writes are all real.
- `tests/brief/test_end_to_end.py` goes one boundary further: it patches
`get_db` to yield one shared test session and patches the HTTP layer
under each adapter, then runs the real CLI commands (`watch sync`,
`ingest`, `route`, `adjudicate`, `brief`) end to end and compares the
rendered bytes against a committed golden file.

## Red Flags

- Test names that mirror method names exactly
- Assertions that only check mock call counts
- Tests that break when implementation changes but behavior doesn't
- Heavy use of `mock.assert_called_with()` on internal methods
- Tests that require knowing private method signatures
- Excessive mocking of the unit under test

## InsightWeaver-Specific Patterns

### Async Tests
```python
@pytest.mark.asyncio
async def test_synthesis_produces_valid_output(self, mock_curator, mock_client):
# Arrange
mock_curator.curate.return_value = {"articles": sample_articles}
mock_client.analyze.return_value = valid_response_json

# Act
result = await synthesizer.synthesize_with_trust_verification()

# Assert on behavior
assert result["status"] == "success"
assert "citation_map" in result["synthesis_data"]["metadata"]
```

### Mocking External Services
Mock at the boundary (Claude API, database), not internal methods:
```python
# Good: Mock the external boundary
mock_client.analyze_with_context = AsyncMock(return_value=response)

# Bad: Mock internal helper methods
mock_synthesizer._parse_synthesis_response = MagicMock()
```

### Testing Error Handling
Test that errors produce correct outcomes, not that specific exceptions flow:
```python
# Good: Behavior when API fails
async def test_synthesis_handles_api_failure_gracefully():
mock_client.analyze.side_effect = Exception("API Error")
result = await synthesizer.synthesize()
assert result["status"] == "error" # Behavior

# Bad: Testing exception plumbing
async def test_api_exception_propagates():
mock_client.analyze.side_effect = APIError()
with pytest.raises(APIError): # Implementation detail
await synthesizer.synthesize()
```
- Test names that mirror method or function names exactly
- Assertions that only check a mock's call count (`mock.assert_called_once()`
with nothing about what it produced)
- Tests that would still pass if the feature were deleted
- A test that asserts a value it just told a stub to return
- Mocking an internal function of the module under test rather than the
true external boundary (an LLM client, the network, the database engine)
- A clock read from the wall in a test that asserts on dates or ordering
(see `frozen_clock` in `tests/brief/test_end_to_end.py`, which replaces
the `datetime` class `src.utils.utcnow` reads)

## Review Checklist

When reviewing tests, ask:

- [ ] Does the test name describe a behavior or outcome?
- [ ] Would this test break if I refactored without changing behavior?
- [ ] Am I testing what users/callers care about?
- [ ] Are assertions on outcomes rather than call patterns?
- [ ] Is mocking limited to true external boundaries?
- [ ] Can I understand what the code does just from reading tests?
When reviewing a test, ask:

- [ ] Does the test name describe a behavior or outcome, not a method call?
- [ ] Is there a concrete bug that would turn this test red? (If you can't
name one, the test may be a tautology.)
- [ ] Are assertions on outcomes -- what got written, computed, or
rendered -- rather than on call patterns?
- [ ] Is mocking limited to a true external boundary (model client,
network, wall clock), with everything else real?
- [ ] Would this test survive a refactor that kept behavior the same but
changed internals (renamed a private function, changed a return
type)?
- [ ] If the test is about ordering or a threshold, is it exercised with
data that would expose a wrong sort or an off-by-one (both sides of
the boundary, not just one)?

## Questions to Ask

1. "If I renamed a private method, would this test break?"
2. "Does this test tell me what the feature does?"
3. "Am I testing the contract or the implementation?"
4. "Would a user of this code care about what I'm asserting?"
1. "If I renamed a private function, would this test break?"
2. "What real defect would this test catch?"
3. "Am I testing the contract (what the caller observes) or the internals?"
4. "Is every mock here standing in for the network, the model, or the
clock -- or did I mock something inside the unit under test?"

## Example Transformation

**Before (implementation-coupled):**
**Before (tautological, mock-verification only):**
```python
def test_build_synthesis_task_with_citations_returns_tuple():
result = synthesizer._build_synthesis_task_with_citations(articles, 2)
assert isinstance(result, tuple)
assert len(result) == 2
def test_run_calls_the_client():
client = FakeClient(_verdict())
run(corpus, ClaudeAdjudicator(client))
assert client.calls == 1
```

**After (behavior-focused):**
**After (behavior-focused, from `tests/evidence/test_adjudicate.py`):**
```python
def test_synthesis_includes_citation_map_matching_input_articles():
"""Citation map in output should reference the exact articles provided"""
articles = [{"id": 1, "title": "Article One", "source": "Source A"}]
def test_every_pair_gets_a_ledger_row_and_evidence_gets_an_evidence_row(self, corpus):
client = FakeClient(_verdict(), _verdict(is_evidence=False, direction="none"))

result = await synthesizer.synthesize_with_trust_verification()
result = run(corpus, ClaudeAdjudicator(client))

citation_map = result["synthesis_data"]["metadata"]["citation_map"]
assert citation_map["1"]["title"] == "Article One"
assert citation_map["1"]["source"] == "Source A"
assert (result.asked, result.evidence, result.none, result.failed) == (2, 1, 1, 0)
ledger = corpus.query(Adjudication).order_by(Adjudication.watch_id).all()
assert [(a.watch_id, a.outcome, a.audit_id) for a in ledger] == [
("conmon-scope-expands", "evidence", "audit-1"),
("hiring-market-tightens", "none", "audit-2"),
]
```

See `examples.md` for the full good/bad contrast pairs drawn from the
current suite.
Loading
Loading