feat(plugin-utils): extract data-storage skill into new plugin (from #157)

.claude-plugin/marketplace.json

@@ -53,6 +53,29 @@
         "self-poll"
       ]
     },
+    {
+      "name": "plugin-utils",
+      "description": "Skills for plugin and tool development: data storage conventions, file format selection, sync patterns, and storage architecture for Claude Code plugins and MCP servers.",
+      "version": "0.1.1",
+      "author": {
+        "name": "Nathan Heaps"
+      },
+      "source": "./plugins/claude-code/plugin-utils",
+      "category": "utility",
+      "tags": [
+        "utility",
+        "skill"
+      ],
+      "keywords": [
+        "plugin-development",
+        "data-storage",
+        "yaml",
+        "file-formats",
+        "mcp",
+        "hooks",
+        "claude-code"
+      ]
+    },
     {
       "name": "reddit",
       "description": "Fetch Reddit posts, comments, and subreddit listings via Reddit's public JSON API. Returns markdown-formatted content suitable for LLM consumption. No authentication required.",

plugins/claude-code/plugin-utils/.claude-plugin/plugin.json

@@ -0,0 +1,21 @@
+{
+  "name": "plugin-utils",
+  "version": "0.1.1",
+  "description": "Skills for plugin and tool development: data storage conventions, file format selection, sync patterns, and storage architecture for Claude Code plugins and MCP servers.",
+  "author": {
+    "name": "Nathan Heaps",
+    "email": "nsheaps@gmail.com",
+    "url": "https://github.com/nsheaps"
+  },
+  "homepage": "https://github.com/nsheaps/agents/tree/main/plugins/claude-code/plugin-utils",
+  "repository": "https://github.com/nsheaps/agents",
+  "keywords": [
+    "plugin-development",
+    "data-storage",
+    "yaml",
+    "file-formats",
+    "mcp",
+    "hooks",
+    "claude-code"
+  ]
+}

plugins/claude-code/plugin-utils/README.md

@@ -0,0 +1,31 @@
+# plugin-utils
+
+Skills for Claude Code plugin and tool development.
+
+## Skills
+
+### `data-storage-using-the-filesystem`
+
+Covers conventions for persistent data storage in Claude Code plugins:
+
+- File format selection (YAML > JSON5 > Markdown-with-frontmatter > JSON)
+- Metadata storage (YAML frontmatter, comment-block frontmatter)
+- File sizing and splitting
+- YAML pros/cons and copy-swap concurrency pattern
+- YAML → JSONL conversion at RPC/streaming boundaries
+- Known path discipline (`$CLAUDE_PROJECT_DIR/.claude/`)
+- Sync model: fetch-before-edit, push-after-write, daemon/MCP preferred
+- Conflict detection and resolution
+
+**Triggers:** "how to store data in a plugin", "YAML vs JSON", "keep task files in sync",
+"streaming storage", "frontmatter metadata", "concurrent writes"
+
+## Installation
+
+This plugin is part of the `nsheaps/agents` repository. Enable it in your Claude Code settings:
+
+```json
+{
+  "enabledPlugins": ["plugin-utils@agents-local"]
+}
+```

plugins/claude-code/plugin-utils/skills/data-storage-using-the-filesystem/SKILL.md

@@ -0,0 +1,297 @@
+---
+name: Data Storage Using the Filesystem
+description: >
+  This skill should be used when the user asks about "how to store data in a plugin",
+  "what file format to use for plugin storage", "YAML vs JSON in a plugin",
+  "how to keep task files in sync", "streaming storage", "frontmatter metadata",
+  "data storage for hooks", "writing files in MCP servers", "concurrent writes in plugin storage",
+  "file format selection", "storing plugin state", or "persistent data in Claude Code plugins".
+version: 0.1.0
+---
+
+# Data Storage Using the Filesystem
+
+## Purpose
+
+Defines the conventions for how Claude Code plugins, hooks, MCP servers, and agents store persistent
+data on the filesystem. Covers file format selection, metadata conventions, path discipline,
+concurrency, sync, and streaming boundaries.
+
+---
+
+## Storage Format Preference Order
+
+**YAML > JSON5 > Markdown-with-frontmatter > JSON**
+
+Only deviate from this order when a specific constraint forces it (e.g., a wire protocol requires
+JSON, or a tool only parses TOML).
+
+| Format                    | Use When                                                                          |
+| ------------------------- | --------------------------------------------------------------------------------- |
+| YAML                      | Default for structured data: task files, config, plugin state                     |
+| JSON5                     | Config files where comments are needed but tooling only supports JSON-like syntax |
+| Markdown-with-frontmatter | Docs, plans, specs — when prose + queryable metadata both needed                  |
+| JSON                      | Wire protocols, MCP stdio, external APIs that require it                          |
+| TOON/TRON                 | Streaming tool output where structured types are needed (narrow case)             |
+| XML / SQLite              | Special cases only — structured queries across many records, or existing schema   |
+
+---
+
+## File Sizing
+
+Keep files small. Prefer many small files over a few large ones.
+
+**Why:**
+
+- **Grep / embeddings**: Large files produce low-precision hits; small files are O(1) to scan
+- **Context window**: Agents load files into context — large files bloat the window
+- **Write contention**: A single large file is a contention point for concurrent writers
+- **Diffs**: Small files produce readable, reviewable diffs
+
+Rule of thumb: if a file exceeds ~300 lines, split it. Store each logical record in its own file
+(e.g., one task per file, one spec per file). Use an INDEX file or directory structure for discovery.
+
+---
+
+## Metadata Storage
+
+### For Markdown files
+
+Store metadata as YAML frontmatter at the top of the file:
+
+```markdown
+---
+id: "42"
+status: in_progress
+created_at: 2026-05-23T10:00:00Z
+---
+
+Free-form prose content here.
+```
+
+### For non-Markdown files (scripts, config, YAML data)
+
+Use a comment-block frontmatter at the file head, after any shebang:
+
+```yaml
+# ---
+# schema_version: 1
+# created_by: task-utils
+# ---
+id: "42"
+status: in_progress
+```
+
+### Schema discipline
+
+- **Define the schema explicitly.** Extra fields are not queryable unless declared.
+- Use `yq` or `jq` for structured queries against stored data.
+- Document the schema in the plugin's `docs/` or a `references/` file.
+- Prefer flat structures (fewer nesting levels) for grep-ability.
+
+---
+
+## YAML: Pros, Cons, and Conventions
+
+### Pros
+
+- Comments (`#`) — documents intent inline without breaking parsers
+- Multi-document support (`---` separator) — stream multiple records per file
+- References and anchors (`&anchor` / `*alias`) — reduce repetition in config
+- JSON superset — valid JSON is valid YAML
+- Human-readable diffs — block scalars and newlines are preserved
+- Structured types: strings, numbers, booleans, nulls, sequences, mappings — no ambiguity
+
+### Cons
+
+- **Token cost** — YAML is verbose; large YAML files consume more context tokens than compact JSON
+- **No atomic in-place edit** — YAML must be rewritten in full; use copy-swap for safe updates
+  (write to `<file>.tmp`, then `mv <file>.tmp <file>`)
+- **JSONL conversion required** for RPC formats\*\* — MCP stdio protocol, HTTP streaming, and other
+  line-delimited transports need JSONL, not YAML; convert at the boundary
+
+### YAML style in task/data files
+
+- Use block style (`key: value\n`) not flow style (`{key: value}`)
+- Multi-line strings: prefer `|` (literal) or `|-` (literal, strip trailing newline)
+- String IDs: always quote numeric-looking IDs (`id: "42"`, not `id: 42`)
+- Status values: never quote (`status: in_progress`, not `status: "in_progress"`)
+
+---
+
+## YAML → JSONL at Streaming / RPC Boundaries
+
+When outputting to a line-delimited protocol (MCP stdio, HTTP streaming, shell pipelines):
+
+1. **Convert YAML → JSON** at the output boundary using `yq` or a serialization layer
+2. **One JSON object per line** (JSONL / NDJSON format)
+3. Never write YAML to an RPC wire protocol
+
+**YAML list → JSONL mapping:**
+
+```yaml
+# YAML (storage)
+- id: "1"
+  status: completed
+- id: "2"
+  status: in_progress
+```
+
+```jsonl
+{"id":"1","status":"completed"}
+{"id":"2","status":"in_progress"}
+```
+
+**Tradeoff:** Per-row streaming (JSONL) loses formatted-document output (comments, structure).
+Use YAML on disk; produce JSONL only at the wire boundary.
+
+For streaming tool output where structured types with type metadata are needed, consider
+TOON/TRON format (see `references/streaming-formats.md`).
+
+---
+
+## Markdown with Frontmatter
+
+Use when both prose content and queryable metadata are needed:
+
+- **Plans, specs, docs** — human-readable body + machine-queryable metadata
+- **Task files with rich descriptions** — status, ID, dates in frontmatter; notes in body
+
+```markdown
+---
+id: "42"
+subject: Fix the authentication bug
+status: in_progress
+created_at: 2026-05-23T10:00:00Z
+---
+
+## Work Log
+
+2026-05-23T10:00:00Z — Started investigation.
+```
+
+Query frontmatter with `yq` (treats the YAML block as a document):
+
+```bash
+yq -y '.status' task-42.yaml   # Returns "in_progress"
+```
+
+---
+
+## Known Paths and Path Discipline
+
+**Always store data in known paths**, built from well-known environment variables:
+
+| Variable                             | Use For                                     |
+| ------------------------------------ | ------------------------------------------- |
+| `$CLAUDE_PROJECT_DIR/.claude/tasks/` | Task files (flat, one per ID)               |
+| `$CLAUDE_PROJECT_DIR/.claude/logs/`  | Hook and daemon log files                   |
+| `$CLAUDE_PROJECT_DIR/.claude/tmp/`   | Disposable intermediate files               |
+| `$CLAUDE_PLUGIN_ROOT/data/`          | Plugin-level persistent state               |
+| `$CLAUDE_PLUGIN_DATA/`               | Plugin data dir (if supported by runtime)   |
+| `$HOME/.claude/tasks/<session_id>/`  | Legacy built-in Task tool per-session store |
+
+**Never use random temp paths** (`/tmp/<uuid>/`) for persistent data. Temp paths are invisible to
+git, invisible to other agents, and are deleted on reboot.
+
+**Commit persistent data.** Task files, plugin state, and index files belong in git. This enables:
+
+- Cross-agent visibility (other agents can pull and see state)
+- History and auditability
+- Recovery after session restart
+
+---
+
+## Concurrency: Copy-Swap Pattern
+
+YAML files cannot be atomically edited in place. Use copy-swap for safe concurrent writes:
+
+```bash
+# Write to a temp file, then atomically replace
+yq -y '. + {"status": "completed"}' "$task_file" > "${task_file}.tmp"
+mv "${task_file}.tmp" "$task_file"
+```
+
+For higher contention (multiple agents writing the same directory), prefer a daemon or MCP server
+that serializes writes (see Sync Model section below).
+
+---
+
+## Sync Model
+
+### Default: fetch-before-edit, push-after-write
+
+1. Before editing any task file: `git pull --ff-only` from the repo root
+2. Write the file (copy-swap pattern)
+3. Immediately after write: `git add <file> && git commit -m "chore(tasks): ..." && git push`
+
+This keeps the flat store visible to all agents. If two agents race, one will get a non-fast-forward
+error on push; the losing agent must pull, re-apply, and push again.
+
+### Preferred: daemon or MCP server
+
+A daemon that owns the store directory (writes-on-change, syncs in the background) is more robust
+than per-agent git operations:
+
+- The daemon serializes writes, eliminating race conditions
+- Agents call the daemon's API (tool call or HTTP) instead of touching files directly
+- The daemon handles git operations transparently
+
+The task-utils MCP server (`plugins/claude-code/task-utils/mcp/`) is an example of this pattern.
+
+### Conflict resolution
+
+When a merge conflict occurs in a task file:
+
+- Do NOT auto-resolve by picking "ours" or "theirs"
+- Emit a structured error with both versions
+- A designated conflict-resolver sub-agent (baked into the plugin) reconciles them
+- The conflict-resolver is never the writing agent — separation of concerns
+
+### Alternative: network filesystem
+
+For high-concurrency teams, consider a network filesystem (NFS, FUSE-mounted object store) instead
+of synced local storage. Agents read/write directly; the filesystem handles consistency.
+
+---
+
+## Programmatic Updates (Agent Last Resort)
+
+Agents should rarely touch storage files directly. Prefer:
+
+1. **Hooks** — fire on tool events, write side-effects automatically
+2. **Daemon/MCP** — agent calls a tool; daemon writes the file
+3. **Background poller** — daemon watches for changes, writes on interval
+
+If an agent must write directly:
+
+- Only after configurable-retry exhaustion (e.g., daemon unavailable)
+- On failure, emit a detailed error message so the agent knows whether escalation will help
+- Never silently swallow write errors
+
+---
+
+## Storage Plugins and External APIs
+
+Plugins can expose storage as an API:
+
+- Provide a `jq`-like query interface where possible: `task_query({filter: ".status == 'in_progress'"})`
+- Storage plugins wrap external APIs (S3, GCS, Notion, etc.) transparently
+- Agents use the same tool interface regardless of backend
+- The plugin handles serialization format differences at the boundary
+
+---
+
+## Additional Resources
+
+### Reference Files
+
+For detailed information on specific topics, consult:
+
+- **`references/storage-formats.md`** — Detailed YAML vs JSON5 vs Markdown-frontmatter vs JSON comparison, with examples
+- **`references/sync-and-concurrency.md`** — Copy-swap pattern, daemon/MCP architecture, conflict resolution strategies
+- **`references/streaming-formats.md`** — TOON/TRON, JSONL, per-row vs formatted-document tradeoffs
+
+### Related Plugins
+
+- **`plugins/claude-code/task-utils/`** — Reference implementation: YAML flat store, MCP server, git auto-commit hooks