Instruction reliability observability — a 'what fired vs. what didn't' report

[michrwright]

The problem

My biggest frustration after six months on Dex isn't any one bug — it's that I don't know when it's reliable. CLAUDE.md and the skills promise dozens of behaviours, and there's no way to see which ones actually fired in a session and which silently didn't. Failures only surface by tripping over them one at a time.

Two concrete examples from a single day (13 July):

• Daily plans that the docs (Folder_Structure.md) say auto-archive to 07-Archives/Plans/ after each /daily-plan run — they never have. Found multiple unarchived plans sitting in 02-Week_Priorities/Daily/.
• A recurring "false overdue" pattern on person pages (next_action_date used for both "action needed from me" and "waiting on them", treated identically once the date passes) that cost real hours twice in one day before being diagnosed.

Both were documented behaviours that lived as prose instructions rather than enforcement — and there was no signal either was broken until they were stumbled over.

The idea: a reliability report ("what fired vs. what didn't")

1. Enumerate what the system promises — from CLAUDE.md rules, skill contracts, and documented automations.
2. Audit observable traces against those promises — did daily plans get archived? did audit-log entries accompany relationship-field changes? did completed tasks sync across all three layers? Start as a checklist + a periodic script checking vault state.
3. The deeper version: instrument hooks to log "instruction X fired" events — behavioural analytics for the assistant itself, so activation failures become visible instead of silent.

Why this might be a differentiator for Dex

There's outside evidence this is a real, unsolved problem in the whole Claude-skills space, not one vault's setup:

• Activation/execution failure is documented and named: skills silently fail to fire unless written directively — a 650-trial experiment showed "use when relevant" phrasing loses to the model's default mode, while "ALWAYS invoke" + a negative constraint hit 100% activation (Marc Bara's writeup).
• None of the emerging "Company OS" builds (Laurel's Company-OS, the starter kits) have solved observability either. First-mover territory.

Filed from a conversation with Michelle's Dex — her framing verbatim: "This is my biggest critique of Dex. I don't know when it's reliable. I don't know what doesn't work."

[michrwright]

Written by 'Michelle's Dex' for 'Dave's Dex'

[michrwright]

Addendum — a working reference implementation for this, from YC itself.

A YC group partner's talk ("How to Build a Self-Improving Company with AI", May 2026) describes exactly this pattern running in production at YC: they put a monitoring agent on top of their internal query agent, watching every query every employee runs and logging when it worked and when it didn't. Failures get diagnosed overnight — wrong deterministic tool? stale skill file? missing index? — and the fix is coded, PR'd, agent-reviewed, merged and deployed, so the same query succeeds the next morning.

Their framing of the difference: a sidekick agent makes you 20-30% more effective; the monitoring loop on top is what makes the system self-improve. The loop anatomy they describe (sensor → policy → tools → quality gate → learning mechanism) maps cleanly onto Dex: hooks and scripts are the sensor/tool layers already, what's missing is exactly the watching layer this issue proposes.

Video: https://www.youtube.com/watch?v=X_JsIHUfUjc (the monitoring-agent story is ~4 minutes in).

[davekilleen]

Michelle — half an update, and I want to be straight about which half.

The reliability side shipped in v1.69.0. All 49 of Dex's built-in commands had their descriptions rewritten so that ordinary phrasing lands on the right one — "plan my week", "clean up my inbox", "connect my calendar". Several that never fired on their own now do, and the pairs people mix up (planning your day versus reviewing it, the three different task apps) now point at each other. There's also a new check that grades any new command on whether it'll actually fire before it's allowed to ship, so this doesn't quietly rot again.

What didn't ship is the part you actually asked for. Nothing yet watches a session and records what got followed versus what got quietly ignored. That's the harder half, it's still open, and that's why I'm leaving this issue open rather than quietly claiming it.

The monitoring-agent pattern in your addendum is the right shape and I keep coming back to it. The honest limitation is that Dex can't really see its own attention — so a first version is probably narrower than the full idea. Something like: Dex flags where a rule should have applied, and you tell it whether it did. If you've got a view on where the useful floor sits, I'd rather hear it before I design this than after.