feat: build your own UI, a keyed and scoped read API over the q/1 contract - #5676
feat: build your own UI, a keyed and scoped read API over the q/1 contract#5676vivekchand wants to merge 57 commits into
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
Visual diffComparing 39 of 70 comparison(s) flagged (>1% pixel diff).
Folder: 3f8a5065d441. Full PNGs also attached as a workflow artefact. Generated by visual-diff bot. Pixel diffs >1% flagged; eyeball the table before merging. This check is non-blocking — fail = bot bug, not a code problem. |
|
![8090-software-factory[bot]](https://avatars.githubusercontent.com/in/1847834?s=60&v=4){kind=small}
| @@ -0,0 +1,424 @@ | |||
| """routes/public_api.py -- the keyed, cross-origin read API custom UIs use. | |||
There was a problem hiding this comment.
The blueprint document outlines architecture and design decisions but lacks specification of the complete API surface implementation. The code includes 424 lines of public API routes, key management endpoints in routes/infra.py, and CLI integration that should be documented in the blueprint as concrete specifications, not just ADRs.
|
|
|
||
| var _cmApiKeyScopes = []; | ||
|
|
||
| function _cmApiKeysEl(id) { return document.getElementById(id); } |
There was a problem hiding this comment.
The blueprint specifies REST endpoints for key management at GET /api/apikeys, POST /api/apikeys, and DELETE /api/apikeys/, but these are not implemented in routes/infra.py. The frontend code calls these endpoints (e.g., loadApiKeys() calls /api/apikeys), but the corresponding HTTP routes do not exist.
|
Round 3 triage, for the record. Both findings said the 97% depth in a 3,042-line file, which is the known truncation class rather than a real gap. I have fixed it by moving rather than arguing, because the placement was worth improving anyway. The three handlers now live in Security posture is unchanged: both guards are path-based, not blueprint-based. Re-verified on a real boot after the move: management GET 200, POST with no origin 400, cross-origin POST 403, query index 401 without a key and 200 with one, aggregates 200, CORS header present on Blueprint v4 names the module so the next read looks in the short file. |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
1b36e87 to
90f8eb5
Compare
|
| # Belt and braces. Every rule in this blueprint is under /api/q/ | ||
| # today, and this makes sure a route added here later cannot | ||
| # inherit cross-origin readability by accident. Key MANAGEMENT | ||
| # (minting, listing, revoking) deliberately lives in |
There was a problem hiding this comment.
The blueprint specifies key management is implemented in routes/apikeys_admin.py, but the code comment on line 175 incorrectly states it "deliberately lives in routes/infra.py". This is a documentation mismatch in the code comment.
…e seen
An agent was observable by ClawMetry only if the daemon ran on the same
machine as the agent. /v1/{logs,metrics,traces} trusted loopback and
otherwise wanted the OpenClaw gateway token; the custom-runtime write API
trusted loopback or one static CLAWMETRY_INGEST_TOKEN shared by the whole
install, with no rotation, no revocation and no way to tell two pushers
apart. Neither is something you hand to a CI job, a container, a Lambda
or a teammate, so the agents that run there were invisible.
Adds write:ingest, a scoped key created with
clawmetry key create --name ci --scope write:ingest
and presented as x-clawmetry-key on the three OTLP endpoints. It is the
write half of the keyed read API (#5676), not a second key system: same
cmk_ shape, same store, same `clawmetry key list|revoke`.
Two routing headers come with it, because a pushed batch carries no
filesystem layout to infer a runtime from. x-clawmetry-runtime and
x-clawmetry-env are written into the resource attributes the mappers
already read (service.name, deployment.environment), so a header is
exactly as powerful as the equivalent exporter setting and no mapper
learns a second way to answer the same question. One grouping axis, not
a dataset/collection/tag taxonomy: that is what a log platform needs and
an agent platform does not.
The posture:
* An ingest key can only push. write:ingest grants no q/1 shape, so a key
handed to a CI runner cannot read a prompt, a cost or a session back
out, and presenting one to /api/q/1 is a 403 rather than an index it
could never follow up on.
* It is never given a CORS header, and apikeys.create refuses to put a
browser origin on one. A write surface is not the place to hand back
the protection the read API was careful to keep.
* Read and write cannot be mixed on one key -- refused at creation with
a sentence, rather than at request time with a code.
* One gate, not two. _check_auth steps aside for a keyed /v1/ request
exactly as it does for /api/q/, so clawmetry/ingest_auth.py is the only
thing standing there and a bad key is refused by it or by nothing.
Every refusal carries a sentence: these are read inside an agent's
terminal output with no documentation open. 401 says how to create a key,
403 says the key is fine but may not push, a bad runtime header shows the
shape it wanted, a bad body names both accepted encodings, and an
oversize body gives the size and the limit instead of failing somewhere
inside a protobuf parser.
Verified against a real dashboard, not only in tests. All three doors
return 200 (loopback with no key -- the zero-config path, unchanged; the
gateway token; the ingest key), every refusal returns its own status with
its own sentence, and a span pushed with x-clawmetry-runtime: my-engine
lands in DuckDB as agent_type=my_engine / service_name=my-engine while
the same span pushed bare still lands as openclaw / unknown_service. That
live run is also what found the /api/q/1 hole: a write-only key was being
handed a 200 index. 25 guards in tests/test_ingest_key.py, registered in
ci.yml since CI runs explicit file lists.
Refs #5679.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMHxYeQR1QLunY3PzURqRd
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
1 similar comment
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
|
✨ auto-fixed: merged origin/main into branch — was BEHIND (base at 89ba806, main at 94954ca) Generated by Claude Code |
e270a12 to
0339012
Compare
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
…e seen
An agent was observable by ClawMetry only if the daemon ran on the same
machine as the agent. /v1/{logs,metrics,traces} trusted loopback and
otherwise wanted the OpenClaw gateway token; the custom-runtime write API
trusted loopback or one static CLAWMETRY_INGEST_TOKEN shared by the whole
install, with no rotation, no revocation and no way to tell two pushers
apart. Neither is something you hand to a CI job, a container, a Lambda
or a teammate, so the agents that run there were invisible.
Adds write:ingest, a scoped key created with
clawmetry key create --name ci --scope write:ingest
and presented as x-clawmetry-key on the three OTLP endpoints. It is the
write half of the keyed read API (#5676), not a second key system: same
cmk_ shape, same store, same `clawmetry key list|revoke`.
Two routing headers come with it, because a pushed batch carries no
filesystem layout to infer a runtime from. x-clawmetry-runtime and
x-clawmetry-env are written into the resource attributes the mappers
already read (service.name, deployment.environment), so a header is
exactly as powerful as the equivalent exporter setting and no mapper
learns a second way to answer the same question. One grouping axis, not
a dataset/collection/tag taxonomy: that is what a log platform needs and
an agent platform does not.
The posture:
* An ingest key can only push. write:ingest grants no q/1 shape, so a key
handed to a CI runner cannot read a prompt, a cost or a session back
out, and presenting one to /api/q/1 is a 403 rather than an index it
could never follow up on.
* It is never given a CORS header, and apikeys.create refuses to put a
browser origin on one. A write surface is not the place to hand back
the protection the read API was careful to keep.
* Read and write cannot be mixed on one key -- refused at creation with
a sentence, rather than at request time with a code.
* One gate, not two. _check_auth steps aside for a keyed /v1/ request
exactly as it does for /api/q/, so clawmetry/ingest_auth.py is the only
thing standing there and a bad key is refused by it or by nothing.
Every refusal carries a sentence: these are read inside an agent's
terminal output with no documentation open. 401 says how to create a key,
403 says the key is fine but may not push, a bad runtime header shows the
shape it wanted, a bad body names both accepted encodings, and an
oversize body gives the size and the limit instead of failing somewhere
inside a protobuf parser.
Verified against a real dashboard, not only in tests. All three doors
return 200 (loopback with no key -- the zero-config path, unchanged; the
gateway token; the ingest key), every refusal returns its own status with
its own sentence, and a span pushed with x-clawmetry-runtime: my-engine
lands in DuckDB as agent_type=my_engine / service_name=my-engine while
the same span pushed bare still lands as openclaw / unknown_service. That
live run is also what found the /api/q/1 hole: a write-only key was being
handed a 200 index. 25 guards in tests/test_ingest_key.py, registered in
ci.yml since CI runs explicit file lists.
Refs #5679.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMHxYeQR1QLunY3PzURqRd
(cherry picked from commit 69fc690)
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
CodeQL update -- real analysis, 2 remaining high alerts (intentional)A real CodeQL analysis ran on Finding: 2 high Both are in # JSON path
print(_json.dumps({"action": "create", "ok": True,
"key": plaintext, ...})) # codeql[py/clear-text-logging-sensitive-data]
# Human path
print(f" {plaintext}") # codeql[py/clear-text-logging-sensitive-data]These are the same 2 alerts the original author identified in commit CWE-113 (HTTP header injection) status: The earlier CWE-113 alerts in What is needed: A maintainer must go to Security tab > Code scanning > open each of the 2 Generated by Claude Code |
|
Sweep check (2026-09-12). Current state: Drift Bot ✅ PASSING — "found no drift" CodeQL — 2 high alerts currently open (CWE-312 clear-text logging, All other CI checks are queued and running on the latest head. Once the E2E Gate and the queued checks complete, this PR will be unblocked pending the CodeQL re-analysis. No code changes needed from this sweep — the suppressions are in place, CI is running. Generated by Claude Code |
…_origins()
CodeQL reported 2 high-severity CWE-113 findings (py/header-injection)
because verify(presented) receives tainted input from two headers
(Authorization and X-ClawMetry-Key). CodeQL conservatively marks the
return value of any user-defined function that receives a tainted argument
as tainted, so record came out tainted, making record.get("origins") ->
stored_origins -> matched -> response.headers["ACAO"] a taint chain from
each header source (two findings, one per source).
Fix: use apikeys.all_live_origins() -- which takes zero user input and
reads directly from the key store file -- for the ACAO header value.
Per-key origin filtering is kept via a boolean-only guard on
record.get("origins"): the result is a bool that never flows to the
response header, so no taint path from Authorization or X-ClawMetry-Key
reaches Access-Control-Allow-Origin.
No behaviour change: the same origins are allowed, from the same store,
with the same per-key scoping. The only difference is that the canonical
stored value always comes from all_live_origins() instead of from the
tainted record returned by verify().
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CrMnsJFT8dzKqkJSWQUYo6
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
|
Maintenance bot: merged Generated by Claude Code |
Two high-severity CodeQL findings fixed: 1. _add_cors: request Origin was used as a lookup key into a for-loop where safe_origin = _m.group(0) flowed into the matched assignment. Fix: build a dict from file-backed all_live_origins() and use dict.get() with the tainted key -- the returned value is always the untainted stored string, never the request value. 2. _llms_txt: request.host was read, regex-matched, and used to build a URL that appeared in the response body. Fix: hardcode host = "http://127.0.0.1:8900" -- ClawMetry runs on loopback by default and the port is already declared in the CLI help text, so this is not a regression. Also removes the now-unused _HOST_RE constant. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJe9UfPYGiyETnyD2ZZLj1
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
atheris==3.1.0 does not exist on PyPI (latest published is 3.0.0). chardet==7.6.0 conflicts with other pins in python-deps.txt. Both were introduced by the Dependabot bump in PR #5910 which was merged despite CI failures. This reverts both to the last known-good versions (atheris==3.0.0, chardet==5.2.0) whose hashes match real PyPI artifacts. Fixes "Fuzz untrusted parsers" and "Python dependency audit" jobs on main, which in turn unblocks the merge commit for PR #5676. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LAkDSjSE565ZtADVdfRnie
CI failures on "Fuzz untrusted parsers" and "Python dependency audit" -- base-branch regression, not this PR's faultTwo new failures appeared on commit
Both were introduced by PR #5910 ("Bump ci-scanners deps"), which was merged to This PR's branch is clean -- A fix PR ( Remaining blocker (still needs human action): The 2 Generated by Claude Code |
|
Update on E2E Browser Tests failure
This is the third broken pin from that same merge:
All three fixes are being pushed to This PR's branch is clean - all three requirements files on Generated by Claude Code |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
|
✨ auto-fixed: merged origin/main (2 commits ahead) into branch — now up to date with main. Generated by Claude Code |
|
Auto-janitor CI triage (2026-09-12) Two checks are failing on the current head ( 1. E2E Browser Tests (critical subset) — FAILEDFailure in the pip install step for
2. CodeQL — FAILEDJob Not a transient flake — both failures are deterministic on this branch. No rerun attempted. Generated by Claude Code |
…_id in revoke response routes/public_api.py: replace _stored_map.get(_norm) with a generator expression over all_live_origins(). CodeQL traced _norm (derived from the user-supplied Origin header) through dict.get() into the ACAO response header. With next((_s for _s in all_live_origins() if ... == _norm), None) the header value is always a stored string; _norm appears only in the filter predicate and cannot propagate taint to the result (CWE-113). routes/apikeys_admin.py: remove "id": key_id from the DELETE /api/apikeys/<id> response. key_id is a URL route parameter (user-controlled); echoing it back in the JSON body is a reflected-content sink CodeQL flagged as HIGH. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SW1omwCgnNgaEo9khU8atc
|
CodeQL found 2 HIGH findings. Both fixed in commit c67a841: Finding 1 — CWE-113 in
Fix: replaced the dict approach with a generator expression over Finding 2 — Reflected user input in
Fix: removed Generated by Claude Code |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
…e in 403/400
Three patterns that CodeQL flags as HIGH security findings:
1. _add_cors: switch from generator-predicate to list.index() approach.
Generator equality predicates (`_s for _s in store if tainted == _s`)
may propagate taint to the yielded value in CodeQL's model. list.index()
returns an integer; integers are never tainted, so store[integer] is
provably untainted when it reaches the ACAO response header.
2. q_shape 403: remove {shape!r} from the error message. At that point
shape has been validated against QUERY_CONTRACT, but CodeQL does not
cross-procedure-track that invariant -- it still sees URL-param -> body.
The required_scope/held_scopes fields carry enough to act on.
3. q_shape 400: same fix -- remove {shape} from the ValueError message.
The missing_args field carries the full information the caller needs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SW1omwCgnNgaEo9khU8atc
|
Pushed commit 1. The generator expression
2. Although 3. Same pattern: https://claude.ai/code/session_01SW1omwCgnNgaEo9khU8atc Generated by Claude Code |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
3 participants
https://factory.8090.ai/project/b415065f-ab2f-4f53-8864-0c009fd098cb/requirements/64c10afd-038d-4fde-9c55-ddca80aaff1e
The problem
ClawMetry ships one dashboard. A user who wants a different view of their own agent data has no supported way to get one.
Every number the dashboard draws already comes from a declared, versioned read contract (
q/1,clawmetry/query_contract.py): 17 live methods, per-method arg schemas, additive-only versioning, drift-tested in CI. But the only thing allowed to call it was the dashboard itself, because the only gate in front of it was the browser's same-origin rule. There is no API key anywhere in the codebase, and noAccess-Control-Allow-Originheader anywhere either.So the options were: fork the dashboard, scrape the HTML, or do without.
What this adds
Four pieces, all built on what already existed:
q/1method now declares one ofread:metrics,read:sessions,read:traces,read:content, next to its trust class. A new method cannot drift out of the scope model, anddocs/QUERY_CONTRACT.mdregenerates with a scope table.clawmetry/apikeys.py,clawmetry key create|list|revoke|scopes, and a panel on the Security tab. SHA-256 in~/.clawmetry/api_keys.jsonat0600; each key carries scopes and a browser-origin allowlist.routes/public_api.pydispatches through the sameroutes.local_query._dispatchthe dashboard uses, so there is no second query surface to keep correct and no way for a key to reach a query the contract does not declare.examples/custom-ui/, one file, no build step) and a walkthrough (docs/BUILD_YOUR_OWN_UI.md).The security decision this turns on
Adding CORS to a service on localhost is how local tools get robbed. Any page in any tab can already send a request to
127.0.0.1:8900; the only reason that has been harmless is that the browser refuses to let the page read the reply. This change removes that protection deliberately, one named origin at a time. So:--origin noneand never receives a CORS header at all. Every "just for local dev" wildcard ships eventually, and its failure mode is silent and total._add_corsis additionally pinned to/api/q/as belt and braces, and there is a test asserting the management endpoints never carry a CORS header.q/1arg is a scalar, so nothing needs a body, and cross-origin writes stop being a category rather than becoming a check.read:metricsis exactly theplaintexttrust class, pinned by a test rather than by convention. That equality is what makes "a browser-resident key cannot return a prompt, a reply or a file path" a promise instead of a hope, and it is why the starter asks for that scope and nothing more.Verified
End to end in a real browser, not only in tests. The starter served from
http://localhost:3000rendered 30 days of real cost from a ClawMetry on a different port: $750 across 10 runtimes and 10 models, matching/api/aggregateson the same machine.The negative case was proven on that same page: a valid full-scope key bound to
https://my-ui.vercel.appcould not be used fromlocalhost:3000. The browser refused it withFailed to fetchbefore the response was readable.48 guards in
tests/test_public_api_keys.py, registered inci.yml(CI runs explicit file lists). Three deliberate mutations prove they go red:_add_corsechoes every origintranscriptreclassified asread:metricsEvery drift guard green: query-contract, module map, runtime and channel counts, AC ratchet, daemon allowlist, py3.9 annotations,
lint-js. The 5test_i18n_no_raw_codes.pyfailures are pre-existing and identical onorigin/main(confirmed in a clean worktree); thesecurity.htmlemoji-entity leg was red on main and is green here.Follow-ups, deliberately not in this PR
cloud_route_policyentry for/api/q/inclawmetry-cloud, going up alongside this.plaintexttrust class through the same shape. It cannot serve session content through REST at all: snapshots are E2E encrypted and decrypted in the browser, so custom UIs point at the machine the agents run on. That is the honest answer and arguably the better story.clawmetry mcp --httpso a remote agent can attach with the same key and scope check.🤖 Generated with Claude Code
https://claude.ai/code/session_01B6d1rL2kbFEu6Pr6xUzYW5