From b781acfbf211f8185b7759a4d2b5825d5b7ddde9 Mon Sep 17 00:00:00 2001 From: Lakshya77089 Date: Tue, 16 Jun 2026 22:52:42 +0530 Subject: [PATCH 1/2] docs(benchmarks): note ponytail's cost is task-shaped in agentic runs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The headline benchmark is single-shot completions; a Notes bullet generalized 'the cost gap widens further in ponytail's favor' to real sessions. Independent multi-turn Cursor-SDK A/B (#121) shows that on large completion-forced tasks ponytail ON can raise tool calls and tokens (more reading before writing) while shrinking output — net cost either way; the win is clearest on snowball-prone/blocked tasks. Scope the single-shot claim and add a 'saves vs. costs' section, plus the skillCount-is-availability-not-usage clarification. --- benchmarks/README.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/benchmarks/README.md b/benchmarks/README.md index 426d4e6..f420c76 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -79,5 +79,14 @@ Running the benchmark requires **Python 3**, **pandas**, and **Node.js** (18+). ## Notes - Caveman is a prose-compression skill (it leaves code "normal"), so it lands between baseline and ponytail on code size and wins mainly on prose tokens. -- Cost reflects single-shot calls that re-send the skill every time. In real sessions the skill is injected once and prompt-cached, so the cost gap widens further in ponytail's favor. +- Cost reflects single-shot calls that re-send the skill every time. In real sessions the skill is injected once and prompt-cached, so on tasks shaped like these the cost gap widens further in ponytail's favor. - These are everyday tasks. For production-grade specs, where an unconstrained agent bloats much harder, see the writeups in `results/`. + +## Where ponytail saves vs. costs + +These numbers are single-shot completions. In multi-turn agentic runs the result is task-shaped, not universal: + +- **Snowball-prone or blocked tasks** (agent keeps adding, installs a dep, scaffolds "for later"): ponytail's restraint cuts the runaway work — this is where the cost win is largest. +- **Large completion-forced tasks** (a full draft the agent must finish): ponytail's "understand before you write" discipline can add reading/exploration up front, so it may raise tool calls and tokens while shrinking the written output. Net cost can go either way. + +An independent Cursor-SDK A/B measuring this (isolated worktrees, toggling only the rule file) saw ponytail ON correlate with more tool calls and higher estimated cost but leaner drafts on completion-forced tasks, with per-model exceptions: [RicardoCostaGit/ponytail-benchmark-from-cursor](https://github.com/RicardoCostaGit/ponytail-benchmark-from-cursor) (#121). Note also that an SDK's startup `skillCount` is the count of skills *available* in the workspace, not skills the model read — only a `read` of a `SKILL.md` is usage. From 1054075e5e0bcf083771878dd3026c05c8bb6bad Mon Sep 17 00:00:00 2001 From: Lakshya77089 Date: Thu, 18 Jun 2026 13:59:07 +0530 Subject: [PATCH 2/2] fix: handle stdin error in ponytail-mode-tracker to avoid uncaught crash (#147) Register a process.stdin 'error' handler so an abnormal stdin close (broken pipe, parent crash) exits cleanly instead of throwing an uncaught exception. process.exit(0) preserves the hook contract: always exit 0, never block session start. --- hooks/ponytail-mode-tracker.js | 2 ++ 1 file changed, 2 insertions(+) diff --git a/hooks/ponytail-mode-tracker.js b/hooks/ponytail-mode-tracker.js index d4fda46..59db3b1 100644 --- a/hooks/ponytail-mode-tracker.js +++ b/hooks/ponytail-mode-tracker.js @@ -6,6 +6,8 @@ const { getDefaultMode } = require('./ponytail-config'); const { clearMode, setMode, writeHookOutput } = require('./ponytail-runtime'); let input = ''; +// Exit cleanly if stdin errors (broken pipe, parent crash) — never block session start +process.stdin.on('error', () => { process.exit(0); }); process.stdin.on('data', chunk => { input += chunk; }); process.stdin.on('end', () => { try {