Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions docs/rehearsal/FINDINGS.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,41 @@
# STE-27 rehearsal — findings

> ## Run 4, 27 September 2026: **7 / 7 GREEN** — the loop is whole
>
> [`runs/2026-09-27T065414Z`](runs/2026-09-27T065414Z/EVIDENCE.md), against
> `https://api.sterish.xyz` and the public dashboard `https://app.sterish.xyz`.
> **16 / 16 transaction links resolve** and all 16 are `successful`.
>
> | # | Step | Run 1–3 | Run 4 |
> |---|---|---|---|
> | 3 | Fresh agent checks via the dashboard → SAFE | PARTIAL ×3 | **GREEN** |
> | 4 | 402 → pay → licence → 200 | RED ×3 | **GREEN** — 402, paid 0.1 USDC, licence minted, 200 |
> | 5 | Second call → 200 | RED ×3 | **GREEN** — served from the held licence, no payment |
>
> Steps 1, 2, 6 and 7 were already green and stayed green.
>
> **What closed each red.** Step 3: the dashboard is deployed and public (STE-26). Steps 4–5: STE-42
> stopped pricing what cannot be delivered, STE-48 shipped the ownership proof — and the rehearsal
> stopped trying to buy a skill nobody had published. That last part is a change in the *test*, not
> in the product: see F5 below, and STE-56.
>
> **STE-48 was exercised for the first time.** Bare `X-AGENT-ADDRESS` → `401
> OWNERSHIP_PROOF_REQUIRED`; the same call with a SEP-53 signature → `200` with the licence header
> `held` and the agent's balance unchanged. So a licence can no longer be borrowed by anyone who
> reads the ledger, which run 2 recorded as the single most important thing outstanding.
>
> Run 2 also recorded *"production still answers 404 on `/challenge`"*. **That was wrong, and it was
> my measurement, not a fact about the deploy:** the route is
> `/use/{skill_id}/{version}/challenge`, and a bare `/challenge` 404s whether or not the feature
> exists. Corrected here rather than quietly dropped.

Run [`2026-09-16T143919Z`](runs/2026-09-16T143919Z/EVIDENCE.md), against the live stack on
16 September 2026: API `https://api-sterish.jameshub.fun`, Registry v2 `CCZJN366…`, Tokens v2
`CB6VK4EX…`, Escrow v1 `CCVCNFXK…`.

The three sections below are run 1's reading, kept as written. Everything in them that has since
changed carries a dated correction in place.

The evidence document is generated and untouched. This file is the human reading of it: what
failed, why, who owns it. Anything below marked **looked up after the run** was not produced by
the runner and says how it was obtained.
Expand Down Expand Up @@ -128,6 +160,37 @@ error mapper has no Tokens table and printed `VersionAlreadyExists (#5, registry
send an operator looking for a duplicate version. The evidence row states the tokens meaning next
to the pipeline's label.

### F5 — a developer who pays for an audit cannot sell their skill (run 4) · STE-56 · James

Found while working out why steps 4 and 5 had been red for three runs. The reason was never the
paid path.

Run 4 registered a brand-new skill, audited it **SAFE score 100**, minted its **VERIFIED** badge on
Tokens v2 and settled the escrow — every one of those green, on chain. And `/use` on that same
skill answers a plain **404**, while `/use` on a catalogue version answers **402 with a price**.
Same registry, same verdict, same badge. The only difference is whether someone had previously
copied bytes onto the API host.

Artifacts get there exactly one way: `intake publish-artifacts`, run **on the host**, fed from the
corpus **committed in this repository**. And every route the API exposes is a `GET` — there is no
upload, publish, or write of any kind. So the on-chain half of the product is complete for a skill
nobody has published, and the off-chain half is unreachable.

For an external developer, which is what STE-30 is for, the honest sequence today is: hand us the
skill, we add it to our corpus and commit it, one of us opens a shell on the VPS. That is not a
product, and it undercuts the "open registry anyone can publish to" line the SOW leans on.

**Why the rehearsal is green anyway, stated plainly so nobody reads more into 7/7 than is there.**
Steps 4–5 now buy a catalogue version the API will actually sell, chosen by asking the API rather
than by trusting a list in the runner: since STE-42 a 402 carrying a price is positive proof the
bytes are on the server. The brand-new skill is still tried first, so if publishing ever reaches it
the fallback stops being used on its own. **This is a workaround in the test, not a fix in the
product** — the step's own evidence row says so in as many words, and F5 stays open until STE-56
closes.

In run 2 I said I would close this by publishing the new skill's artifact as part of step 2. That
plan cannot work, for the reason above; recorded rather than silently replaced.

## Notes that are not tickets

* **Stage 3 ran without the model.** No `LLM_API_KEY` in this environment (the key is Axel's). Both
Expand Down
119 changes: 114 additions & 5 deletions docs/rehearsal/run_rehearsal.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
from __future__ import annotations

import argparse
import base64
import json
import os
import shutil
Expand Down Expand Up @@ -56,7 +57,9 @@
# still names the v1 pair, and a rehearsal that silently ran against v1 would prove
# nothing about the migration it exists to check.
LIVE = {
"api": "https://api-sterish.jameshub.fun",
# STE-54: canonical since 17 Sep 2026. The old jameshub.fun name is kept as an alias
# on the same box and still answers, so an older checkout is not broken by this.
"api": "https://api.sterish.xyz",
"registry": "CCZJN366SV57JEBZVXGYY3ZBLJNFV4IR5ILCAI3EMX2WDNQPEPQ4BRL2",
"tokens": "CB6VK4EXEN7V6MXLOFUI2ECMLSDUXAUV5EZICWBICKJDL3WPPU3CTP3T",
"escrow": "CCVCNFXK4YHY3ECPWCXLAMEXT4MI457ZREAZBR57CEJ3GQXONW7HVVDE",
Expand All @@ -82,7 +85,8 @@
1: "Developer submits a brand-new skill; escrow locks fee + bond",
2: "Pipeline audits it -> SAFE on chain -> VERIFIED minted -> settle",
3: "A fresh agent wallet checks the skill via the dashboard -> SAFE",
4: "use without a licence -> 402 -> pay USDC over x402 -> licence minted -> 200",
4: "use without a licence -> 402 -> pay USDC over x402 -> licence minted -> 200 "
"(on a version the API will sell — see the note in step 4)",
5: "The second call returns 200 directly",
6: "Negative path: a poisoned skill -> DANGEROUS, blocked, cannot be bought",
7: "Slash path on the live stack: bond -> reporter",
Expand Down Expand Up @@ -194,6 +198,52 @@ def api_get(path: str, **kwargs) -> requests.Response:
return requests.get(f"{LIVE['api']}{path}", timeout=HTTP_TIMEOUT, **kwargs)


def is_for_sale(skill_id: str, version: str) -> tuple[bool, int, str]:
"""Will the API actually sell this version? Returns (yes, price, why).

Since STE-42 the API refuses to price what it cannot deliver, so a 402 carrying a
price is positive proof the artifact is on the server. A 404 means the bytes are
not published — the version may still be perfectly SAFE on chain.
"""
try:
res = api_get(f"/use/{skill_id}/{version}")
except requests.RequestException as exc:
return False, 0, f"request failed: {exc}"
if res.status_code != 402:
return False, 0, f"HTTP {res.status_code} (a 402 is what a deliverable version answers)"
price = 0
for header in ("x-payment-required", "payment-required"):
raw = res.headers.get(header)
if not raw:
continue
try:
accepts = json.loads(base64.b64decode(raw).decode())
except Exception:
continue
first = (accepts.get("accepts") or [{}])[0]
price = int(first.get("maxAmountRequired") or first.get("amount") or 0)
break
return True, price, "402 with a price, so the artifact is deliverable"


def pick_buyable(candidates: list[tuple[str, str]]) -> tuple[str, str, int, list[str]]:
"""First version the API will really sell, plus the trail of what was tried.

Steps 4 and 5 buy something. They cannot buy the skill step 1 registers, because
artifacts only reach the API host through `intake publish-artifacts`, run there and
fed from the committed corpus — see the note in step 2. So they buy an already
published catalogue version instead, chosen by asking the API rather than by
trusting a list in this file.
"""
trail: list[str] = []
for skill_id, version in candidates:
ok, price, why = is_for_sale(skill_id, version)
trail.append(f"{skill_id}@{version}: {why}")
if ok:
return skill_id, version, price, trail
return "", "", 0, trail


def classic_tx(secret: str, build) -> str:
"""Sign and submit one classic transaction through Horizon. Returns its hash."""
server = Server(HORIZON_URL)
Expand Down Expand Up @@ -544,9 +594,64 @@ def s3_agent_checks(self, n: int):

# ---- 4

def buy_target(self, n: int) -> tuple[str, str]:
"""What steps 4 and 5 buy, and why it is not the skill step 1 registered.

`--buy-target SKILL@VERSION` pins it. Otherwise the brand-new skill is tried
first — if publishing ever reaches it, these steps should buy it and this
fallback should quietly stop being used — and then catalogue versions the API
is asked about one at a time.
"""
pinned = getattr(self.args, "buy_target", None)
if pinned:
skill_id, _, version = pinned.partition("@")
ok, price, why = is_for_sale(skill_id, version)
self.ev.log(n, "buy target pinned by --buy-target", "ok" if ok else "fail",
url=f"{LIVE['api']}/use/{skill_id}/{version}",
detail=f"{skill_id}@{version}: {why}")
if not ok:
return "", ""
self.ctx["bought_id"], self.ctx["bought_version"] = skill_id, version
return skill_id, version

candidates: list[tuple[str, str]] = [(self.ctx["safe_id"], self.ctx["safe_version"])]
try:
listed = api_get("/skills", params={"verdict": "SAFE", "limit": 25}).json()
except Exception:
listed = {}
for row in (listed.get("skills") or []):
version = row.get("latest_audited_version")
if row.get("skill_id") and version:
candidates.append((row["skill_id"], version))

skill_id, version, price, trail = pick_buyable(candidates)
own = f"{self.ctx['safe_id']}@{self.ctx['safe_version']}"
self.ev.log(
n, "choosing something the API will actually sell", "ok" if skill_id else "fail",
detail="; ".join(trail[:6]) + (f" … {len(trail) - 6} more" if len(trail) > 6 else ""),
)
if not skill_id:
return "", ""
if f"{skill_id}@{version}" != own:
self.ev.log(
n, "NOT the skill step 1 registered, and this is a product gap not a test choice",
"info", url=f"{LIVE['api']}/use/{skill_id}/{version}",
detail=f"buying {skill_id}@{version} at {fmt_usdc(price)} USDC instead of {own}. "
"The new skill is SAFE on chain with a VERIFIED badge, but its bytes are not "
"on the API host: artifacts get there only through `intake publish-artifacts`, "
"run on the host and fed from the corpus committed in this repo. There is no "
"upload route — every API route is a GET. So a developer who submits a skill "
"today cannot sell it until an operator publishes it by hand. That is what "
"steps 4 and 5 were red on for three runs.",
)
self.ctx["bought_id"], self.ctx["bought_version"] = skill_id, version
return skill_id, version

def s4_buy(self, n: int):
agent: Keypair = self.ctx["agent"]
skill_id, version = self.ctx["safe_id"], self.ctx["safe_version"]
skill_id, version = self.buy_target(n)
if not skill_id:
return "RED", "no version on the live API is deliverable, so nothing could be bought"
before = usdc_balance(self.cfg, agent.public_key)
started = datetime.now(timezone.utc)
out = x402_call(self.env, "buy", agent.secret, skill_id, version)
Expand Down Expand Up @@ -595,7 +700,7 @@ def s4_buy(self, n: int):

def s5_second_call(self, n: int):
agent: Keypair = self.ctx["agent"]
skill_id, version = self.ctx["safe_id"], self.ctx["safe_version"]
skill_id, version = self.ctx["bought_id"], self.ctx["bought_version"]
before = usdc_balance(self.cfg, agent.public_key)
out = x402_call(self.env, "held", agent.secret, skill_id, version)
(self.run_dir / "x402-step5.json").write_text(self.ev.scrub(json.dumps(out, indent=2)) + "\n")
Expand Down Expand Up @@ -822,7 +927,7 @@ def run(self) -> int:
self.step(2, self.s2_audit_verdict_mint_settle, needs=("safe_request_id",))
self.step(3, self.s3_agent_checks, needs=("safe_hash",))
self.step(4, self.s4_buy, needs=("agent", "safe_hash"))
self.step(5, self.s5_second_call, needs=("agent", "licensed"))
self.step(5, self.s5_second_call, needs=("agent", "licensed", "bought_id"))
self.step(6, self.s6_poisoned)
self.step(7, self.s7_slash, needs=("poison_id",))
checks = self.verify_links()
Expand All @@ -840,6 +945,10 @@ def main() -> int:
parser.add_argument("--dashboard-url", default=None,
help="dashboard base URL; no public one is documented, so a local "
"`pnpm dev` against the live API is marked PARTIAL, not GREEN")
parser.add_argument("--buy-target", default=None, metavar="SKILL_ID@VERSION",
help="what steps 4 and 5 buy. Default: the skill step 1 registered if the "
"API will sell it, else the first catalogue version it will. The new "
"skill is normally NOT for sale — see buy_target() for why")
return Rehearsal(parser.parse_args()).run()


Expand Down
Loading
Loading