Build skill self-improvement loop from AI transcript incidents

[jmagar]

Summary

Build a Cortex "skill self-improvement" loop by reusing the existing AI abuse incident pattern:

1. extract skill-use events from Claude/Codex transcript logs,
2. group suspicious skill-related failures into incidents,
3. build bounded evidence bundles,
4. derive deterministic findings/prevention hints,
5. optionally run an LLM assessment that proposes concrete skill-document improvements.

This should make Cortex answer questions like:

• Which skills were loaded/used by Claude or Codex in a session?
• Which skills are repeatedly followed by user correction, tool failure, wrong-scope work, or wasted loops?
• What transcript evidence supports the claim?
• What exact skill-doc change would have prevented the failure?

The implementation must be possible from this issue alone.

Current System Context

Cortex already has almost all of the building blocks. This feature should reuse them instead of creating a parallel analysis stack.

Transcript ingest and metadata

• src/scanner.rs imports AI transcript files via index_file_with_options() and routes by source kind.
  • SourceKind::CodexSession parses line-delimited Codex JSONL through src/scanner/codex.rs.
  • SourceKind::ClaudeProject and explicit Claude-shaped JSONL parse through src/scanner/claude.rs.
  • Explicit files are auto-detected with looks_like_codex_record() for Codex JSONL shapes.
• src/scanner/codex.rs already extracts Codex session_meta, response_item, content text, session id, cwd/workdir, and project metadata.
• src/scanner/claude.rs already extracts Claude message content, session id, and cwd/project fields.
• src/receiver/enrichment.rs maps transcript app names to normalized AI tools:
  • claude-transcript / claude-code -> claude
  • codex-transcript / codex -> codex
  • gemini-transcript -> gemini
• src/db/pool.rs migration 4 added AI transcript metadata columns on logs:
  • ai_tool
  • ai_project
  • ai_session_id
  • ai_transcript_path
• src/db/pool.rs migrations 5 and 6 added transcript source/checkpoint tables:
  • transcript_sources
  • transcript_import_records
• Existing tests in src/scanner_tests.rs cover realistic Codex transcript shapes, explicit Codex-file detection, rewrite/idempotency behavior, and project/session metadata preservation.

Abuse detector pattern to reuse

The abuse detector lives mostly in src/db/queries.rs:

• DEFAULT_AI_ABUSE_TERMS and search_ai_abuse() start around src/db/queries.rs:1901.
• search_ai_incidents() starts around src/db/queries.rs:2123.
• investigate_ai_incidents() starts around src/db/queries.rs:2362.
• Term normalization, FTS query generation, and word-boundary matching start around src/db/queries.rs:2571.

Important pattern:

• Cheap anchor detection first.
• Group anchors by (project, tool, session_id, hostname, time bucket).
• Generate stable incident IDs from deterministic input.
• Fetch bounded transcript context around anchors.
• Fetch nearby non-AI logs for correlation.
• Return truncation/candidate-cap metadata instead of silently overclaiming.

Deterministic findings pattern to reuse

src/app/incident_findings.rs derives rule-based findings over incident evidence bundles. It is pure and side-effect-free:

• stable categories,
• conservative confidence,
• evidence row ids for every finding,
• prevention hints,
• unknown / open questions when evidence is weak.

Create the skill-improvement equivalent in a new module rather than stuffing unrelated rules into incident_findings.rs.

LLM assessment pattern to reuse

Cortex already has a local-only Gemini assessment path:

• src/app/services/assessment.rs calls investigate_ai_incidents(), serializes bounded evidence, builds a prompt, and runs Gemini.
• src/assessment.rs embeds the cortex-frustration-assessment skill and wraps evidence as untrusted passive data.
• src/cli/dispatch_ai.rs exposes local-only assessment behavior and rejects HTTP mode.

The skill-improvement assessment should follow this pattern: bounded evidence in, untrusted-data wrapper, isolated LLM process, Markdown assessment out. It should not auto-edit skill files in this issue.

Source Signals

Claude skill usage

Claude session JSONL has structured fields for skill/plugin attribution. Parse these when present:

• attributionSkill
• attributionPlugin

Expected normalized event:

{
  "ai_tool": "claude",
  "skill_name": "cortex-frustration-assessment",
  "skill_plugin": "cortex",
  "event_kind": "claude_attribution",
  "evidence_kind": "structured_json_field",
  "log_id": 123
}

Codex skill usage

Codex does not currently expose the same structured attribution fields in Cortex. Skill loads are visible in transcript content. Parse skill blocks from transcript message text, especially:

<skill><name>lavra:lavra-plan</name>
...
</skill>

Normalize to one skill event per <skill><name>...</name> block. A single transcript row may contain multiple skills, so do not model this as a single nullable logs.ai_skill_name column.

Expected normalized event:

{
  "ai_tool": "codex",
  "skill_name": "lavra:lavra-plan",
  "skill_plugin": "lavra",
  "event_kind": "codex_skill_block",
  "evidence_kind": "transcript_content",
  "log_id": 456
}

Do not rely on OTLP for this feature. Transcript logs are the source of truth.

Proposed Data Model

Add a normalized table instead of widening logs with one skill column.

CREATE TABLE IF NOT EXISTS ai_skill_events (
  id                 INTEGER PRIMARY KEY AUTOINCREMENT,
  log_id             INTEGER NOT NULL REFERENCES logs(id) ON DELETE CASCADE,
  ai_tool            TEXT NOT NULL,
  ai_project         TEXT,
  ai_session_id      TEXT,
  hostname           TEXT NOT NULL,
  timestamp          TEXT NOT NULL,
  skill_name         TEXT NOT NULL,
  skill_plugin       TEXT,
  skill_path         TEXT,
  event_kind         TEXT NOT NULL,
  evidence_kind      TEXT NOT NULL,
  metadata_json      TEXT,
  created_at         TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
  UNIQUE(log_id, skill_name, event_kind, evidence_kind)
);

CREATE INDEX IF NOT EXISTS idx_ai_skill_events_skill_time
  ON ai_skill_events(skill_name, timestamp);

CREATE INDEX IF NOT EXISTS idx_ai_skill_events_session_time
  ON ai_skill_events(ai_tool, ai_project, ai_session_id, timestamp);

CREATE INDEX IF NOT EXISTS idx_ai_skill_events_project_skill_time
  ON ai_skill_events(ai_project, skill_name, timestamp)
  WHERE ai_project IS NOT NULL;

Implementation notes:

• Keep logs as the canonical transcript/event row table.
• Use ai_skill_events.log_id to join back to full message/context.
• Prefer idempotent insertion with INSERT OR IGNORE because transcript re-imports and watch retries are normal.
• If ai_project or ai_session_id is missing, insert the event anyway when log_id, ai_tool, hostname, timestamp, and skill_name exist; queries can still return partial evidence.
• Store parser-specific extras in metadata_json, for example raw attribution field path or matched XML offset.

Parser/Extraction Plan

Create a small parser module, for example src/skill_events.rs or src/scanner/skill_events.rs, with pure extraction functions:

pub struct ExtractedSkillEvent {
    pub skill_name: String,
    pub skill_plugin: Option<String>,
    pub skill_path: Option<String>,
    pub event_kind: SkillEventKind,
    pub evidence_kind: SkillEvidenceKind,
    pub metadata_json: Option<String>,
}

Parser requirements:

• Claude:
  • Parse attributionSkill and attributionPlugin from the original JSON value before/while extracting transcript text.
  • Support top-level fields and nested payload/message variants if observed in fixtures.
  • Emit no event when fields are absent.
• Codex:
  • Parse all <skill><name>...</name> occurrences from extracted transcript text.
  • Accept optional whitespace around tags.
  • De-duplicate repeated identical skill names within one log row.
  • Do not parse arbitrary user prose like "use the rust skill" as a skill event.
• Normalization:
  • Trim whitespace.
  • Reject empty names.
  • Clamp skill names and plugin names to a conservative max length, e.g. 256 chars.
  • If name contains plugin:skill, set skill_plugin = plugin and skill_name = plugin:skill.
  • If Claude gives both plugin and skill separately, keep skill_name as the skill field and skill_plugin separately. Do not fabricate plugin:skill unless the existing field already uses that format.

Integration options:

• Preferred: during transcript import, after a logs row is inserted, insert extracted ai_skill_events rows in the same chunk transaction.
• If current insert APIs do not expose inserted log ids cleanly, first refactor batch insertion to return (record_key, log_id) for transcript import records, then insert skill events from that mapping.
• Do not perform a second full-table scan during normal ingest.

Backfill Plan

Add an explicit backfill path so existing transcript logs get skill events.

Recommended CLI:

cortex ai skills backfill --since 30d --limit 10000 --dry-run
cortex ai skills backfill --since 30d --limit 10000

Requirements:

• Only scan candidate logs rows where ai_tool IN ('claude', 'codex') and message/metadata can contain skill signal.
• Process in bounded batches, ordered by id or timestamp.
• Use INSERT OR IGNORE for idempotency.
• Report scanned rows, inserted events, skipped duplicates, parse errors, and truncation.
• Do not hold the SQLite write lock for a whole historical corpus.
• Reuse checkpointing if practical, but do not mutate transcript_import_records semantics unless necessary.
• Add a service-level method so CLI, API, and tests do not each implement their own scanner.

New User-Facing Surfaces

Add MCP actions to src/mcp/actions.rs and route them through src/mcp/tools.rs:

1. skill_events

   • List normalized skill events.
   • Scope: cortex:read.
   • Cost: cheap/moderate depending on filters.
   • Filters: skill, plugin, tool, project, session_id, hostname, from, to, limit.

2. skill_incidents

   • List grouped skill-improvement incidents.
   • Scope: cortex:read.
   • Cost: moderate.
   • Filters: same as skill_events, plus window_minutes, signals, min_score.

3. skill_investigate

   • Deep-dive evidence bundle for one or more skill incidents.
   • Scope: cortex:read.
   • Cost: expensive.
   • Filters: incident_id, skill, plugin, tool, project, from, to, limit, window_minutes, correlation_window_minutes.

4. Optional in this issue if small enough, otherwise split after the first three are working: skill_assess

   • Local-only CLI equivalent is mandatory if implemented; remote MCP/HTTP LLM execution should not happen by default.
   • If exposed over MCP, it should only return deterministic findings unless a safe local execution model already exists for that transport.

Add CLI commands under cortex ai, for example:

cortex ai skills --project /home/jmagar/workspace/cortex --limit 20
cortex ai skill-incidents --skill lavra:lavra-plan --since 7d
cortex ai skill-investigate --incident-id skill-inc-...
cortex ai skill-assess --incident-id skill-inc-...   # local-only, if implemented

Add REST endpoints mirroring the existing AI endpoints:

• GET /api/ai/skills
• GET /api/ai/skill-incidents
• GET /api/ai/skill-investigate
• POST /api/ai/skills/backfill if backfill is exposed remotely; require admin scope, explicit dry_run, and audit logging like other mutating admin endpoints.

Update docs:

• README.md
• docs/api.md
• docs/mcp/TOOLS.md
• docs/mcp/SCHEMA.md
• docs/contracts/mcp-actions-current.md
• CLAUDE.md MCP action count/table if actions are added.

Skill Incident Detection

Create DB/service models parallel to the abuse incident models in src/db/models.rs and app models under src/app/models/.

Suggested structs:

• AiSkillEventParams
• AiSkillEventEntry
• AiSkillIncidentParams
• SkillIncident
• AiSkillInvestigateParams
• SkillIncidentEvidence
• AiSkillInvestigateResult

Anchor signals

A skill incident should start from skill events plus nearby negative signals. The first version should be deterministic and easy to explain.

Use these initial signal categories:

1. user_correction_after_skill

   • User message within N transcript rows/minutes after skill event includes terms like:
     • that's not what I asked
     • you said
     • wrong
     • no, ...
     • we wasted
     • all you had to say
     • stop
     • you didn't need to
   • Use careful phrase matching, not one-word broad matches alone.

2. tool_failure_after_skill

   • Nearby transcript/tool output includes command/tool failure text such as:
     • exit code
     • permission denied
     • not found
     • timed out
     • failed to
     • database is locked
     • rate limit

3. scope_or_source_confusion

   • Nearby user correction or assistant text indicates wrong repo, stale data, wrong source, memory-vs-live confusion, or searched the wrong place.
   • Start with conservative phrases from src/app/incident_findings.rs and add skill-specific ones.

4. ignored_skill_or_policy_instruction

   • Skill was loaded, but subsequent transcript shows behavior that directly violates a known instruction category:
     • no verification after implementation,
     • no issue/bead when non-trivial task required it,
     • wrong transport/source for requested data,
     • used raw web when Axon/Labby was required,
     • stopped at plan when user asked to implement.
   • Keep this category conservative. Only emit when phrase evidence is explicit.

5. overlong_loop_after_skill

   • Skill event followed by many tool calls/transcript rows before useful resolution, plus user correction/frustration.
   • Should not trigger on long but successful work by itself.

Grouping

Group incidents similarly to search_ai_incidents():

• key: (skill_name, skill_plugin, ai_tool, ai_project, ai_session_id, hostname, window_bucket)
• window_bucket = unix_secs / window_secs * window_secs
• default window_minutes = 10, clamp to 1..=120
• stable incident id hash input should include:
  • skill name/plugin,
  • tool/project/session/hostname,
  • sorted anchor log ids,
  • sorted skill event ids.

Scoring

Start simple and explainable:

score =
  skill_event_count * 2
  + user_correction_count * 15
  + tool_failure_count * 8
  + scope_confusion_count * 12
  + ignored_instruction_count * 12
  + overlong_loop_count * 10
  + signal_variety * 5

Labels:

• < 15: low
• < 35: medium
• < 60: high
• otherwise: critical

Use f64::total_cmp() when sorting by score, matching the abuse incident fix.

Investigation Bundle

skill_investigate should return enough context to understand and fix the skill without opening the database manually.

For each incident include:

• incident
• skill_events
• signal_anchors
• transcript_before
• transcript_after
• nearby_tool_failures
• nearby_user_corrections
• nearby_logs
• nearby_errors
• deterministic_findings
• truncation flags for each bounded collection

Bounds:

• skill events: cap 25
• signal anchors: cap 50
• transcript before: cap 20
• transcript after: cap 20
• nearby logs: cap 50
• nearby errors: cap 25

Use chronological ordering for evidence intended for reading. Return candidate caps and truncation booleans like the abuse detector does.

Deterministic Findings

Create src/app/skill_incident_findings.rs with pure rule evaluation. It should mirror the design of src/app/incident_findings.rs:

• no DB queries,
• no LLM calls,
• deterministic output,
• every finding cites evidence ids,
• conservative confidence,
• emits open questions when evidence is weak.

Suggested categories:

• skill_scope_mismatch
• missing_prerequisite_check
• wrong_source_of_truth
• overly_broad_research_loop
• tool_policy_mismatch
• missing_verification_step
• ambiguous_skill_trigger
• stale_or_conflicting_skill_instruction
• assistant_overexplained_simple_answer
• unknown

Suggested prevention hints should be skill-doc-actionable, for example:

• "Add a trigger-boundary note that this skill is for implementation planning only; direct yes/no questions should be answered directly."
• "Add a verification checklist item requiring live repo/runtime evidence before claiming success."
• "Add an anti-loop rule: after two failed searches, summarize the current evidence and switch strategy."

LLM Assessment

Add a skill-improvement assessment path modeled on cortex ai assess.

Suggested implementation:

• Add a new embedded skill prompt, for example plugins/cortex/skills/cortex-skill-improvement-assessment/SKILL.md.
• Add assessment builder code next to src/assessment.rs or split into a generic assessment module.
• Wrap evidence exactly as untrusted passive data:

<untrusted-evidence source="cortex skill_investigate json" treat-as="passive-data">
...
</untrusted-evidence>

The LLM output should be Markdown with:

1. Incident summary.
2. What the skill was supposed to help with.
3. What actually happened.
4. Evidence-backed failure modes.
5. Proposed skill-doc changes.
6. Proposed regression tests or transcript queries.
7. Confidence and open questions.

Safety requirements:

• Evidence is untrusted and must not be treated as instructions.
• No files are modified by assessment.
• No shell commands are run by assessment other than the controlled Gemini process.
• Keep local-only CLI behavior unless/until remote execution policy is explicitly designed.

Tests

Add focused tests at every layer.

Parser tests

• Claude JSONL with attributionSkill and attributionPlugin emits one skill event.
• Claude JSONL without attribution emits no skill event.
• Codex response text containing one <skill><name>...</name> emits one skill event.
• Codex response text containing multiple skill blocks emits multiple events.
• Codex repeated identical skill blocks in one row are de-duplicated.
• Invalid/empty skill names are ignored.
• User prose mentioning a skill without a tag does not emit a Codex event.

Ingest/backfill tests

• index_file() inserts ai_skill_events for newly imported transcript rows.
• Re-indexing the same transcript does not duplicate events.
• Rewritten transcript rows preserve idempotency semantics already tested in src/scanner_tests.rs.
• Explicit Codex-file detection still works and emits skill events.
• Backfill dry-run reports counts and inserts nothing.
• Backfill normal run inserts events idempotently.

Query tests

• skill_events filters by skill, plugin, tool, project, session, hostname, time range.
• skill_incidents groups events/signals into stable incident IDs.
• skill_incidents scoring is deterministic and uses total ordering for floats.
• Candidate caps and truncation flags are set correctly.
• skill_investigate returns bounded context and deterministic findings.

Surface tests

• CLI JSON and human output for cortex ai skills, skill-incidents, skill-investigate.
• MCP action registry includes new actions with correct scopes and cost classes.
• REST endpoints deserialize repeated/query params correctly.
• Admin backfill endpoint, if added, requires explicit dry_run and admin scope.

Assessment tests

• Prompt wraps evidence as untrusted data.
• Gemini command spec does not permit bypass/yolo/full-auto flags.
• Missing incident id returns a clear error.
• Fake Gemini runner returns streamed Markdown without requiring network.

Engineering Review Notes

These are requirements, not polish.

• Use a normalized ai_skill_events table because one transcript row may contain multiple skill blocks.
• Do not rely on OTLP for skill invocation detection; transcript logs are the source of truth.
• Do not broaden phrase matching so much that every frustrating session becomes a skill incident. Start conservative and evidence-backed.
• Do not let backfill or migration hold the SQLite write lock across a large corpus.
• Keep evidence bounded and expose truncation flags. Never silently claim complete evidence when caps were hit.
• Treat transcript content as untrusted data in all LLM prompts.
• Do not auto-edit, commit, or push skill changes in this feature. The output can propose patches; applying them should be a later explicit workflow.
• Keep existing abuse detector behavior unchanged unless shared helpers are extracted with tests.
• Preserve current transcript import idempotency and checkpoint behavior.

Acceptance Criteria

• Skill events from Claude attributionSkill / attributionPlugin are searchable from Cortex.
• Skill events from Codex <skill><name>...</name> transcript content are searchable from Cortex.
• Existing transcript import and realtime watch paths populate ai_skill_events for new rows.
• Existing transcript history can be backfilled with bounded/idempotent behavior.
• Users can list skill events via CLI, MCP, and REST.
• Users can list grouped skill incidents with stable IDs, scores, labels, signal counts, and truncation metadata.
• Users can investigate a skill incident and receive a bounded evidence bundle with transcript context, signals, nearby logs/errors, and deterministic findings.
• Optional LLM assessment, if included in this issue, is local-only and produces Markdown recommendations from skill_investigate evidence.
• Tests cover parser, ingest/backfill, DB queries, CLI/API/MCP surfaces, and assessment prompt safety.
• Docs are updated for all new actions/commands/endpoints.
• cargo fmt, cargo test, and cargo clippy pass.

Mandatory LLM Observability and Safety Controls

There must be no code path where Cortex invokes an LLM for enrichment, assessment, incident analysis, summarization, or recommendation generation without durable observability and bounded execution controls.

Observability requirements

Every LLM invocation must emit a structured audit/trace record before the process/API call starts and another record after it finishes, fails, times out, or is cancelled.

At minimum, record:

• invocation id, stable across start/finish records
• timestamp start/end and duration
• caller surface: CLI, MCP, REST, background task, scheduled job, or test harness
• action/command name, for example ai_assess or future skill_assess
• incident id or skill incident id, when applicable
• ai tool/project/session filters used to build evidence
• evidence bundle counts and truncation flags
• model/provider/program, for example Gemini CLI model name
• prompt size and evidence size in bytes/tokens if available
• output size in bytes/tokens if available
• exit status/result category: success, error, timeout, cancelled, circuit_open, rate_limited
• sanitized error message and bounded stderr tail
• host/process id

The audit trail must be queryable through Cortex itself. Prefer a dedicated table such as llm_invocations rather than relying only on ordinary logs. Ordinary logs/tracing are still required, but they are not enough by themselves.

Suggested schema shape:

CREATE TABLE IF NOT EXISTS llm_invocations (
  id                 TEXT PRIMARY KEY,
  started_at         TEXT NOT NULL,
  finished_at        TEXT,
  duration_ms        INTEGER,
  caller_surface     TEXT NOT NULL,
  action             TEXT NOT NULL,
  provider           TEXT NOT NULL,
  model              TEXT,
  program            TEXT,
  incident_id        TEXT,
  ai_tool            TEXT,
  ai_project         TEXT,
  ai_session_id      TEXT,
  evidence_counts_json TEXT,
  prompt_bytes       INTEGER,
  output_bytes       INTEGER,
  status             TEXT NOT NULL,
  error              TEXT,
  metadata_json      TEXT
);

CREATE INDEX IF NOT EXISTS idx_llm_invocations_started
  ON llm_invocations(started_at);

CREATE INDEX IF NOT EXISTS idx_llm_invocations_action_started
  ON llm_invocations(action, started_at);

CREATE INDEX IF NOT EXISTS idx_llm_invocations_status_started
  ON llm_invocations(status, started_at);

Add read surfaces:

• CLI: cortex ai llm-invocations --since 24h --limit 50
• MCP: llm_invocations read action
• REST: GET /api/ai/llm-invocations

Circuit breakers and rate limits

LLM execution must be guarded centrally. Do not let each future feature spawn its own process or API call directly.

Create a shared LlmInvocationGuard / LlmRunner abstraction used by current cortex ai assess and any future skill_assess or enrichment path.

Required controls:

• global concurrency limit, default 1
• per-action concurrency limit, default 1
• per-action rate limit, for example max invocations per minute/hour
• cooldown circuit breaker after repeated failures/timeouts
• timeout per invocation, using the existing timeout setting as input
• max prompt/evidence bytes, with clear error when exceeded
• max output bytes or bounded capture
• process kill on timeout/cancellation
• dry-run/preview mode for prompt/evidence construction without LLM execution
• global kill switch config/env, for example CORTEX_LLM_ENABLED=false
• per-action enablement, for example assess enabled but background enrichment disabled
• background LLM enrichment disabled by default unless explicitly configured

The default posture should be conservative: no unbounded background LLM fan-out, no parallel storm, and no automatic retries that can multiply invocations. Retries, if added later, must be explicit, bounded, jittered, and included in invocation audit records.

Config requirements

Add config fields rather than scattering env reads across call sites. Env overrides are fine, but there should be one typed config surface.

Suggested config shape:

[llm]
enabled = true
max_concurrent = 1
max_per_action_concurrent = 1
max_invocations_per_minute = 3
max_invocations_per_hour = 30
failure_threshold = 3
cooldown_secs = 300
timeout_secs = 120
max_prompt_bytes = 1048576
max_output_bytes = 262144
background_enrichment_enabled = false

[llm.actions.ai_assess]
enabled = true

[llm.actions.skill_assess]
enabled = true

[llm.actions.background_enrich]
enabled = false

Acceptance criteria addendum

• Every LLM invocation goes through the shared guarded runner.
• A start and finish/failure audit record exists for every attempted invocation.
• Audit records remain queryable even when the LLM process fails or times out.
• Concurrency/rate-limit/circuit-open denials are themselves recorded as audit events.
• cortex ai assess is migrated to the guarded runner.
• Any new skill assessment path uses the same runner.
• Tests prove concurrency limit, rate limit, timeout kill, circuit breaker, kill switch, and durable audit behavior.
• No background LLM enrichment runs unless explicitly enabled in config.

UX Requirement: Skill-First Investigation Flow

The primary user experience must be skill-first. Users should not have to list grouped incidents, inspect clusters, discover an incident id, and then run a second command just to investigate a known skill.

Required CLI UX:

cortex ai skill-investigate lavra:lavra-plan
cortex ai skill-investigate lavra:lavra-plan --since 7d
cortex ai skill-investigate lavra:lavra-plan --tool codex --project /home/jmagar/workspace/cortex
cortex ai skill-assess lavra:lavra-plan --since 7d

Behavior:

• The positional argument is skill, not incident_id.
• skill-investigate <skill> should internally:
  1. filter skill events/incidents to the requested skill,
  2. choose the highest-priority matching incident(s) by default,
  3. return the evidence bundle directly.
• --incident-id remains available as an expert/narrowing option, but it is not the main path.
• If multiple relevant incidents exist, default to top priority and include a short other_matching_incidents summary with ids, timestamps, scores, and labels.
• Add --all or --limit N for investigating multiple matching incidents.
• Add --plugin lavra for plugin-level investigation when the user wants all skills from one plugin.
• If no incident exists but skill events do exist, return a low-severity evidence bundle summarizing recent usage and say no negative incident signals were found.
• If no skill events exist, return a clear no-data response with suggested filters/time range.

MCP/REST should support the same flow:

• skill_investigate accepts skill directly.
• skill_assess accepts skill directly.
• incident_id is optional and only narrows selection when supplied.

Acceptance criteria addendum:

• cortex ai skill-investigate lavra:lavra-plan works without first running skill-incidents.
• cortex ai skill-assess lavra:lavra-plan works without first running skill-investigate or manually copying an incident id.
• Help text and docs show the skill-first commands as the canonical examples.
• Incident IDs are still returned for reproducibility/debugging, but users do not need them for the normal flow.

UX Requirement: Unified cortex assess <target> Commands

Do not add more long hyphenated assessment commands as the primary UX. The canonical assessment interface should be a small verb namespace:

cortex assess skill lavra:lavra-plan
cortex assess abuse --since 24h
cortex assess mcp cortex --since 7d

Canonical command shape:

• cortex assess skill <skill>
• cortex assess abuse [--incident-id ID] [filters...]
• cortex assess mcp <tool-or-server>

The old/proposed names like cortex ai skill-assess or cortex ai skill-investigate may exist as compatibility aliases if useful, but docs/help/examples should lead with cortex assess ....

cortex assess skill <skill>

This is the skill-first flow described above:

cortex assess skill lavra:lavra-plan
cortex assess skill lavra:lavra-plan --since 7d --tool codex --project /home/jmagar/workspace/cortex
cortex assess skill --plugin lavra --since 7d

Behavior:

• Finds matching skill events/incidents internally.
• Selects the highest-priority incident by default.
• Supports --all / --limit N for multiple incidents.
• Returns deterministic findings plus optional guarded LLM assessment when enabled.
• Does not require the user to manually discover an incident id first.

cortex assess abuse

This should wrap the existing abuse/frustration assessment flow with better UX:

cortex assess abuse
cortex assess abuse --since 24h
cortex assess abuse --incident-id inc-...
cortex assess abuse --project /home/jmagar/workspace/cortex --tool codex

Behavior:

• If --incident-id is supplied, assess that incident.
• If no id is supplied, find the top matching abuse incident and assess it.
• Include other_matching_incidents when more candidates exist.
• Internally reuse the current abuse_incidents, abuse_investigate, deterministic findings, and guarded LLM assessment path.

cortex assess mcp <tool-or-server>

Add a new MCP assessment target. This should identify MCP-tool/server friction from transcript evidence and tool-call/error logs.

Examples:

cortex assess mcp cortex
cortex assess mcp labby --since 7d
cortex assess mcp mcp__lumen__semantic_search --project /home/jmagar/workspace/cortex
cortex assess mcp --server labby --tool search --since 24h

Initial scope:

• Detect repeated MCP tool failures, schema mismatches, auth/connectivity failures, wrong-tool selection, missing-tool confusion, timeout/rate-limit loops, and user corrections after tool misuse.
• Anchor on transcript/tool-call evidence where available and correlate with Cortex syslog/OTLP logs when present.
• Group by (mcp_server, mcp_tool, ai_tool, ai_project, ai_session_id, hostname, window_bucket).
• Build evidence bundles with transcript context, tool-call inputs/outputs when logged, nearby system logs, and deterministic findings.
• Use the same LLM observability/guarded-runner requirements as skill and abuse assessment.

Suggested MCP finding categories:

• wrong_mcp_tool_selected
• mcp_server_unavailable
• mcp_auth_or_permission_failure
• mcp_schema_mismatch
• mcp_timeout_or_rate_limit
• mcp_result_misinterpreted
• missing_mcp_discovery_step
• tool_surface_confusion
• unknown

Acceptance criteria addendum:

• cortex assess skill lavra:lavra-plan is the documented primary skill assessment command.
• cortex assess abuse is the documented primary abuse/frustration assessment command.
• cortex assess mcp <tool-or-server> exists and can assess MCP tool/server incidents from transcript/log evidence.
• Existing lower-level list/investigate commands can remain for debugging, but normal assessment UX does not require them.
• CLI help uses the unified assess namespace and avoids long hyphenated command names for the main flow.

UX Requirement: First-Class Hook Assessment

Add hooks as a first-class assessment target in the unified assessment namespace:

cortex assess hooks
cortex assess hooks --since 24h
cortex assess hooks --hook post_tool_use --since 7d
cortex assess hooks --project /home/jmagar/workspace/cortex --tool codex

Do not bury hook analysis under generic MCP or skill assessment. Hooks have their own failure modes: they can fail silently, run too often, run too late, mutate unexpected state, block agent flow, or drift from the intended policy. Cortex should make those visible directly.

Initial scope:

• Detect hook execution, skipped hooks, hook failures, hook timeouts, hook retries, hook output parse errors, and hook-induced tool/agent failures when the transcript/log evidence exists.
• Correlate hook events with nearby AI transcript turns, tool calls, MCP calls, syslog/OTLP records, and user corrections.
• Group hook incidents by (hook_name, hook_source, ai_tool, ai_project, ai_session_id, hostname, window_bucket).
• Support both agent hooks and plugin/tooling hooks where they are visible in logs/transcripts.
• Return deterministic findings first; optional LLM assessment must go through the shared guarded runner and durable llm_invocations audit path.

Suggested hook finding categories:

• hook_failed
• hook_timed_out
• hook_not_invoked
• hook_invoked_too_often
• hook_wrong_scope
• hook_output_parse_error
• hook_policy_drift
• hook_blocked_agent_flow
• hook_mutated_unexpected_state
• hook_caused_tool_failure
• unknown

Suggested evidence bundle fields:

• hook_events
• signal_anchors
• transcript_before
• transcript_after
• nearby_tool_calls
• nearby_mcp_events
• nearby_logs
• nearby_errors
• deterministic_findings
• truncation flags for every bounded collection

Required CLI behavior:

• cortex assess hooks finds and assesses the top hook incident in the selected time range.
• --hook NAME narrows to a known hook.
• --all or --limit N assesses multiple matching hook incidents.
• If hook events exist but no negative signals are found, return a low-severity usage/evidence summary rather than an error.
• If no hook evidence exists, return a clear no-data response with suggested filters/time range.

Acceptance criteria addendum:

• cortex assess hooks is documented alongside cortex assess skill, cortex assess abuse, and cortex assess mcp.
• Hook incidents can be listed/investigated/assessed without manually discovering lower-level event ids first.
• Hook assessment uses the same LLM observability, rate-limit, circuit-breaker, timeout, and kill-switch controls as every other LLM-backed assessment path.
• Tests cover hook event extraction, grouping, deterministic findings, no-data behavior, and guarded LLM assessment integration.

Lavra Research: MCP and Hook Assessment Implementation Details

This section records the researched implementation constraints for cortex assess mcp and cortex assess hooks, including Codex vs Claude differences. Treat this as implementation guidance, not optional notes.

Evidence sources observed locally

Claude transcript evidence is richer and more structured:

• Assistant tool calls appear as transcript JSONL rows with message.content[].type = "tool_use".
• Tool results appear as paired rows with message.content[].type = "tool_result" and tool_use_id linking back to the tool call.
• Claude skill attribution can appear on assistant rows as attributionSkill, attributionPlugin, and attributionAgent.
• Claude hook execution is written into transcripts as attachment rows, for example:
  • attachment.type = "hook_success"
  • attachment.hookName = "SessionStart:startup"
  • attachment.hookEvent = "SessionStart"
  • attachment.command
  • attachment.exitCode
  • attachment.durationMs
  • attachment.stdout
  • attachment.stderr
  • attachment.content
• Claude hook output may include persisted-output pointers when stdout/stderr is too large. Cortex should capture the pointer and preview, not assume full output is inline.
• Claude settings enumerate a broad hook surface: SessionStart, SessionEnd, Stop, StopFailure, SubagentStart, SubagentStop, Notification, PreToolUse, PostToolUse, PostToolUseFailure, PostToolBatch, PermissionRequest, PermissionDenied, UserPromptSubmit, UserPromptExpansion, PreCompact, PostCompact, TaskCreated, TaskCompleted, Setup, TeammateIdle, Elicitation, ElicitationResult, ConfigChange, InstructionsLoaded, WorktreeCreate, WorktreeRemove, CwdChanged, FileChanged, MessageDisplay.

Codex transcript/config evidence is different:

• Codex transcript rows use type = "response_item" with payload.type = "function_call" and payload.type = "function_call_output" for tool calls/results.
• Codex function calls carry fields like payload.name, payload.arguments, payload.call_id, and payload.metadata.turn_id.
• Codex function call outputs carry payload.call_id and payload.output.
• Codex MCP/tool names may be exposed as plain function names (exec_command) or namespaced tool names depending on the callable surface (mcp__..., functions..., etc.). Implement parser normalization by shape, not by one exact prefix.
• Codex hook definitions live in ~/.codex/hooks.json, with hook groups like Stop, SessionStart, UserPromptSubmit, PreToolUse, PostToolUse, PermissionRequest, PreCompact, and PostCompact.
• Codex hook trust/approval state lives in ~/.codex/config.toml under [hooks.state], keyed by hook source, hook event, matcher/index, and trusted hash.
• In the sampled Codex JSONL, hook runtime success/failure records were not present in the same structured hook_success attachment shape as Claude. Therefore Cortex should not assume Codex hook execution is transcript-visible unless future/local evidence proves it. Codex hook assessment must combine config/trust state, visible transcript side effects, hook command logs, and syslog/OTLP logs where available.

MCP assessment design

Add a normalized ai_mcp_events table rather than trying to infer MCP usage only from free-text transcript search.

Suggested schema:

CREATE TABLE IF NOT EXISTS ai_mcp_events (
  id                 INTEGER PRIMARY KEY AUTOINCREMENT,
  call_log_id        INTEGER REFERENCES logs(id) ON DELETE CASCADE,
  result_log_id      INTEGER REFERENCES logs(id) ON DELETE SET NULL,
  ai_tool            TEXT NOT NULL,
  ai_project         TEXT,
  ai_session_id      TEXT,
  hostname           TEXT NOT NULL,
  timestamp          TEXT NOT NULL,
  turn_id            TEXT,
  call_id            TEXT NOT NULL,
  tool_name          TEXT NOT NULL,
  mcp_server         TEXT,
  mcp_tool           TEXT,
  event_kind         TEXT NOT NULL,
  status             TEXT,
  duration_ms        INTEGER,
  is_error           INTEGER,
  arguments_json     TEXT,
  output_preview     TEXT,
  error_text         TEXT,
  metadata_json      TEXT,
  created_at         TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
  UNIQUE(ai_tool, ai_session_id, call_id, event_kind)
);

CREATE INDEX IF NOT EXISTS idx_ai_mcp_events_tool_time
  ON ai_mcp_events(tool_name, timestamp);

CREATE INDEX IF NOT EXISTS idx_ai_mcp_events_server_time
  ON ai_mcp_events(mcp_server, timestamp)
  WHERE mcp_server IS NOT NULL;

CREATE INDEX IF NOT EXISTS idx_ai_mcp_events_session_time
  ON ai_mcp_events(ai_tool, ai_project, ai_session_id, timestamp);

Parser requirements:

• Claude:
  • Parse tool_use content items as MCP/tool-call events.
  • Parse paired tool_result content items via tool_use_id.
  • name can be builtin (Bash, Read) or MCP-style (mcp__server__tool). Normalize both, but only classify as MCP when the name matches MCP naming or known MCP metadata.
  • Preserve is_error on tool results.
  • Store bounded argument keys/preview; avoid dumping secrets or full huge payloads.
• Codex:
  • Parse response_item.payload.type = "function_call" as call events.
  • Parse response_item.payload.type = "function_call_output" as result events.
  • Join call/result by payload.call_id.
  • Use payload.metadata.turn_id when present for grouping.
  • Normalize namespaced MCP names when available; otherwise keep tool_name raw and infer mcp_server only from safe known prefixes/mappings.
• Both:
  • Preserve raw source row ids so investigation can fetch transcript context.
  • Do not require every call to be MCP; this table may include general tool calls, with mcp_server null for non-MCP tools. cortex assess mcp filters to MCP-classified rows.
  • Add a backfill that scans existing transcript rows in bounded batches and inserts events idempotently.

MCP incident anchors:

• repeated call failures for same mcp_server/mcp_tool,
• is_error = true, nonzero exit, timeout, rate limit, auth/permission failure,
• schema/validation errors,
• unknown tool/server or disconnected server messages,
• user correction after tool selection or result interpretation,
• assistant says a tool is unavailable even though a matching MCP/tool-search event exists,
• tool output contradicts the assistant's later claim.

MCP grouping key:

(mcp_server, mcp_tool, ai_tool, ai_project, ai_session_id, hostname, window_bucket)

For non-MCP tool calls that still matter to assessment, use tool_name as a fallback but do not label them MCP unless the server/tool classification is known.

Hook assessment design

Add a normalized ai_hook_events table. Hooks need their own event model because they can execute outside normal assistant tool-call flow and can affect prompts/context before the model responds.

Suggested schema:

CREATE TABLE IF NOT EXISTS ai_hook_events (
  id                 INTEGER PRIMARY KEY AUTOINCREMENT,
  log_id             INTEGER REFERENCES logs(id) ON DELETE SET NULL,
  ai_tool            TEXT NOT NULL,
  ai_project         TEXT,
  ai_session_id      TEXT,
  hostname           TEXT NOT NULL,
  timestamp          TEXT NOT NULL,
  hook_event         TEXT NOT NULL,
  hook_name          TEXT,
  hook_source        TEXT,
  hook_command       TEXT,
  status             TEXT NOT NULL,
  exit_code          INTEGER,
  duration_ms        INTEGER,
  stdout_preview     TEXT,
  stderr_preview     TEXT,
  persisted_output_path TEXT,
  trusted_hash       TEXT,
  evidence_kind      TEXT NOT NULL,
  metadata_json      TEXT,
  created_at         TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
  UNIQUE(ai_tool, ai_session_id, hook_event, hook_name, timestamp, evidence_kind)
);

CREATE INDEX IF NOT EXISTS idx_ai_hook_events_hook_time
  ON ai_hook_events(hook_event, hook_name, timestamp);

CREATE INDEX IF NOT EXISTS idx_ai_hook_events_status_time
  ON ai_hook_events(status, timestamp);

CREATE INDEX IF NOT EXISTS idx_ai_hook_events_session_time
  ON ai_hook_events(ai_tool, ai_project, ai_session_id, timestamp);

Parser/collector requirements:

• Claude transcript collector:
  • Parse attachment rows with attachment.type beginning with hook_.
  • Map hook_success -> status = success.
  • Map hook failure/error/block variants to status = failed, blocked, or error as observed.
  • Extract hookName, hookEvent, command, exitCode, durationMs, bounded stdout/stderr/content previews, and persisted-output paths.
• Claude config collector:
  • Parse configured hooks from ~/.claude/settings.json only as configuration inventory. Configured hooks are not proof of runtime execution.
  • Use config inventory to detect hook_not_invoked only when a hook was expected for a session/event and no matching runtime evidence exists.
• Codex config collector:
  • Parse ~/.codex/hooks.json for configured hooks and commands.
  • Parse ~/.codex/config.toml [hooks.state] for trusted hashes and hook source keys.
  • Treat these as inventory/trust-state evidence, not runtime execution evidence.
• Codex runtime collector:
  • Prefer explicit transcript/runtime hook records if/when present.
  • Otherwise correlate hook command logs, syslog/OTLP output, and visible transcript side effects.
  • For commands that redirect to /tmp/...log, collect only bounded tails/previews through an explicit file-tail source or existing Cortex log ingestion. Do not shell out ad hoc during normal assessment.

Hook incident anchors:

• hook failure/error/nonzero exit,
• hook timeout or very high duration_ms,
• hook output parse error,
• hook generated additional context that caused instruction conflict or prompt bloat,
• configured/trusted hook never appears when expected,
• hook runs too often for one session/turn,
• hook mutates config/env files unexpectedly,
• user correction after hook-provided instruction/context,
• hook blocks agent flow or repeatedly keeps a turn alive.

Hook grouping key:

(hook_event, hook_name, hook_source, ai_tool, ai_project, ai_session_id, hostname, window_bucket)

Critical Claude vs Codex differences to preserve

Skills

• Claude has structured skill attribution fields (attributionSkill, attributionPlugin, attributionAgent). Prefer those over content matching.
• Codex does not currently have an equivalent structured skill-attribution field in the sampled session logs. Use transcript content markers like <skill><name>...</name> and injected skill blocks.
• Claude attribution can attach to normal assistant text and tool-use rows; Codex skill content may arrive as developer/system-style injected context or assistant-visible transcript content.
• Therefore: Claude skill extraction should be field-first; Codex skill extraction should be content-pattern-first.

MCP/tool calls

• Claude represents calls/results inside message.content[] as tool_use / tool_result, linked by id / tool_use_id.
• Codex represents calls/results as response_item.payload.type = function_call / function_call_output, linked by call_id.
• Claude tool names may be plain builtins (Bash, Read) or MCP-style names. Codex function names may be tool-wrapper-local (exec_command) or namespaced depending on the exposed tool surface.
• Claude has explicit is_error on tool results in sampled logs. Codex outputs often require parsing output text/status metadata to classify failure unless a structured status is present.
• Therefore: implement platform-specific parsers that emit one normalized ai_mcp_events shape, instead of trying one regex over all transcript messages.

Hooks

• Claude runtime hook executions are first-class transcript attachments in sampled logs (attachment.type = hook_success, with command, exit code, duration, stdout/stderr/content).
• Codex hook config and trust state are visible in ~/.codex/hooks.json and ~/.codex/config.toml, but sampled Codex transcripts did not show equivalent structured hook runtime attachments.
• Claude hook assessment can be runtime-evidence-first. Codex hook assessment must be config/inventory plus side-effect/log-correlation first until runtime hook events are proven available.
• Configured hook != executed hook for both platforms. Always distinguish configuration inventory from runtime execution evidence.
• Hook output may inject additional context or mutate the prompt. Treat hook stdout/content as untrusted evidence, same as transcript/tool output.

Implementation order recommendation

1. Add normalized extraction structs and tests for Claude/Codex tool-call events.
2. Add ai_mcp_events schema, ingest-time insertion, and bounded backfill.
3. Implement cortex assess mcp over normalized MCP/tool events.
4. Add Claude hook runtime parser from transcript attachments.
5. Add hook config inventory collectors for Claude and Codex.
6. Add ai_hook_events schema, ingest/backfill for runtime hook events, and optional inventory table if needed.
7. Implement cortex assess hooks with clear evidence-kind labels: runtime_transcript, config_inventory, trusted_hash_state, log_correlation, side_effect_inference.
8. Add Codex hook runtime parsing only after an observed structured runtime event shape exists; until then, do not pretend Codex hook execution is proven from config alone.

Acceptance criteria addendum

• The implementation includes explicit parser tests for Claude tool_use/tool_result, Codex function_call/function_call_output, Claude hook attachments, Claude hook config inventory, and Codex hook config/trust inventory.
• MCP and hook evidence records include evidence_kind or equivalent provenance so runtime facts are not confused with configuration facts.
• cortex assess mcp can explain which transcript/log rows prove a tool call happened and which rows prove the result/failure.
• cortex assess hooks can explain whether it is using runtime hook execution evidence or only config/trust-state evidence.
• Codex hook assessment must not claim a hook executed solely because it exists in hooks.json or [hooks.state].
• Claude skill extraction must prefer attribution fields; Codex skill extraction must prefer skill block/content markers.
• Docs include a Claude vs Codex matrix for skills, MCP/tool calls, and hooks.

[jmagar]

This issue bundles five independent subsystems (skill events/incidents/investigate/assess, the mandatory LLM observability/guard-rail infra, the unified cortex assess CLI namespace, MCP-event tracking, and hook-event tracking). Splitting per the issue's own guidance ("split after the first three are working"):

• This issue (Build skill self-improvement loop from AI transcript incidents #94) is now scoped to Plan A: skill events, skill incidents/investigate, deterministic findings, skill LLM assessment, the shared LlmRunner guard/audit infra, and cortex assess skill / cortex assess abuse.
• Add ai_mcp_events tracking + cortex assess mcp (split from #94) #104 splits out ai_mcp_events tracking + cortex assess mcp.
• Add ai_hook_events tracking + cortex assess hooks (split from #94) #105 splits out ai_hook_events tracking + cortex assess hooks.

#104 and #105 depend on this issue landing first (they reuse LlmRunner and the cortex assess dispatcher built here).