Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,10 +84,7 @@ or in a sibling package. This repo extends `@earendil-works/pi-coding-agent`
(pi-mono), so re-implementing upstream features creates maintenance debt
and divergence.

Use the checklist below **during the Explore/Research/Plan phases** of your
orchestration workflow. It does not replace your main orchestration
pipeline; it replaces the normal Explore/Research/Plan content for
feature-work sessions.
Use the checklist below when orchestrating a feature-work session. It does not replace your main orchestration delegation pipeline; it replaces the normal Explore/Research/Plan content for feature-work sessions.

### Frame the capability

Expand Down
23 changes: 4 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Kimchi operates in one of two modes:

Use `ctrl+p` to cycle through models. The last entry in the cycle is `multi-model`. You can also open the `/model` picker and select a specific model or `multi-model` from the list.

In single-model mode the orchestration system prompt (environment, tools, research rules, guidelines, phase tagging) stays active, but task classification and delegation are disabled. The subagent tool remains available if you explicitly ask the agent to delegate.
In single-model mode the orchestration system prompt (environment, tools, research rules, guidelines, role routing) stays active, but task classification and delegation are disabled. The subagent tool remains available if you explicitly ask the agent to delegate.

### Model roles

Expand Down Expand Up @@ -82,7 +82,7 @@ Defaults are hardcoded in `DEFAULT_MODEL_ROLES`. Roles accept any `provider/mode

#### How delegation works

The orchestrator receives explicit per-phase directives generated from the role configuration. For each pipeline phase (plan, build, review, explore, research), the system prompt tells the orchestrator exactly what to do:
The orchestrator receives explicit per-role directives generated from the role configuration. For each role (planner, builder, reviewer, explorer, researcher), the system prompt tells the orchestrator exactly what to do:

- **Roles it owns** (its model ID appears in the role pool): "DO perform this work yourself."
- **Roles it does not own**: "DO NOT perform this work yourself. Delegate to Agent(type: X, model: Y)." — with the concrete model IDs from the pool.
Expand Down Expand Up @@ -125,20 +125,6 @@ Metadata can also be managed interactively via `/multi-model` → "Edit model me

Kimchi omits Pi's estimated `max_completion_tokens` and `max_tokens` fields from requests to managed Kimchi providers. The gateway determines how much output fits using the model's actual tokenizer; output-budget enforcement belongs at the gateway rather than in Pi's approximate client-side context calculation.

### Phase tracking

Kimchi tags every LLM request with a `phase:{name}` label for usage analytics and cost attribution. The orchestrator sets the phase as work progresses and it is displayed in the status line.

| Phase | Description |
|-------|-------------|
| `explore` | Navigating the codebase, reading files to understand structure |
| `plan` | Designing, breaking down tasks, writing specs |
| `build` | Writing, modifying, or refactoring code |
| `review` | Analyzing output, verifying correctness |
| `research` | Investigating documentation, researching issues |

Subagents inherit the current phase from the orchestrator but cannot change it.

## Tags

Kimchi supports tagging LLM requests for usage tracking and cost attribution. Tags are included with every request and displayed in the status line, grouped by key with color coding.
Expand Down Expand Up @@ -166,10 +152,9 @@ export KIMCHI_TAGS="team:backend,project:api"

### Auto-tags

Two tags are added automatically to every request and do not count toward the 10 tag limit:
One tag is added to every request and does not count toward the 10 tag limit:

- `model:{model_id}` -- the model handling the request
- `phase:{phase}` -- the current work phase

### Persistence

Expand Down Expand Up @@ -571,7 +556,7 @@ The `benchmark/` directory contains tools for smoke-testing kimchi sessions and

- **Manual benchmarks** (`benchmark/manual/`) -- run predefined tasks against different models and compare results. See `benchmark/manual/README.md`.
- **Terminal-bench-2** (`benchmark/terminal-bench-2/`) -- run the [terminal-bench](https://www.harborframework.com/) suite (89 tasks) against kimchi inside Docker containers. See `benchmark/terminal-bench-2/README.md`.
- **Session audit** (`benchmark/audit-session/`) -- audit a completed session for phase discipline, code quality, architecture, testing, model alignment, and cost efficiency. See `benchmark/audit-session/README.md`.
- **Session audit** (`benchmark/audit-session/`) -- audit a completed session for delegation discipline, code quality, architecture, testing, role-model alignment, and cost efficiency. See `benchmark/audit-session/README.md`.

## Release

Expand Down
26 changes: 13 additions & 13 deletions benchmark/audit-session/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Session Phase Audit
# Session Delegation Audit

Audits a completed kimchi harness session, analyzing phase discipline, code quality, model alignment, and cost efficiency. Produces a graded report written to `.kimchi/audits/`.
Audits a completed kimchi harness session, analyzing delegation discipline, code quality, role-model alignment, and cost efficiency. Produces a graded report written to `.kimchi/audits/`.

## Files

Expand Down Expand Up @@ -55,22 +55,22 @@ The agent grades the session across 6 dimensions:

| Dimension | Weight | What it checks |
|-----------|--------|----------------|
| Phase Discipline | 15% | Logical phase ordering, timely transitions, phase-work alignment |
| Delegation Discipline | 15% | Right persona at the right time, timely delegations, minimal self-performed work |
| Architecture | 20% | Design decision timing, module boundaries, project conventions |
| Code Quality | 20% | Lint results, naming, duplication, over-engineering |
| Testing | 20% | Coverage, negative paths, test organization patterns |
| Phase-Model Alignment | 10% | Expensive models for complex phases, cheap models for routine work |
| Cost Efficiency | 15% | Per-phase cost breakdown, counterfactual analysis |
| Role-Model Alignment | 10% | Expensive models for complex work, cheap models for routine execution |
| Cost Efficiency | 15% | Per-segment cost breakdown, counterfactual analysis |

## Output

The audit report is written to `.kimchi/audits/{sessionId}-{task}-{runner}-{model}-AUDIT.md` in the project directory. It includes:

- Summary grade table
- Phase timeline with duration, model, turn count, and cost per phase
- Delegation timeline with duration, model, turn count, and cost per segment
- Detailed findings per dimension
- Tool usage breakdown by phase
- Cost counterfactuals (opus-only, phase-optimized)
- Tool usage breakdown by segment
- Cost counterfactuals (opus-only, role-optimized)
- Top 3 actionable improvements

## Example: benchmark a complex task then audit it
Expand Down Expand Up @@ -122,7 +122,7 @@ The audit agent writes its findings to:
.kimchi/audits/session-20260511-124841-complex-AUDIT.md
```

The report contains phase-by-phase cost breakdown, grade summary, and actionable improvements. Use it to decide whether to adjust model assignments, phase transitions, or task decomposition for future sessions.
The report contains segment-by-segment cost breakdown, grade summary, and actionable improvements. Use it to decide whether to adjust model assignments, delegation decisions, or task decomposition for future sessions.

## How it works

Expand All @@ -137,14 +137,14 @@ The audit evaluates 13 dimensions of session quality. Sections 1–6 are graded;

| # | Metric | Description |
|---|--------|-------------|
| 1 | Phase Discipline | Logical phase ordering, timely transitions, phase-work alignment |
| 1 | Delegation Discipline | Right persona at the right time, timely delegations, minimal self-performed work |
| 2 | Architecture & Design | Decision timing, module boundaries, project conventions |
| 3 | Code Quality | Lint results, naming, duplication, over-engineering |
| 4 | Testing Strategy | Coverage, negative paths, test organization patterns |
| 5 | Phase-Model Alignment | Expensive models for complex phases, cheap for routine work |
| 6 | Cost Efficiency | Per-phase cost breakdown, counterfactual analysis |
| 5 | Role-Model Alignment | Expensive models for complex work, cheap for routine execution |
| 6 | Cost Efficiency | Per-segment cost breakdown, counterfactual analysis |
| 7 | Per-Turn Model Attribution | Active model per turn, provider, switch boundaries |
| 8 | Routing Decision Rationale | What triggered each model switch (phase, tool call, explicit text) |
| 8 | Routing Decision Rationale | What triggered each model switch (delegation, tool call, explicit text) |
| 9 | Switching Latency | ms between model_change and first assistant message per model |
| 10 | Subagent Lifecycle | Loops, budget usage, context completeness per subagent invocation |
| 11 | Token Consumption per Task Class | Tokens/cost per class (explore, plan, build, review, research, orchestration) |
Expand Down
Loading
Loading