diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b6be3787cf..1b8a3afa75 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -314,14 +314,6 @@ jobs: - name: Required-checks list has one source of truth run: python3 -m pytest tests/test_c6_required_checks_single_source.py -q - # Regression guard for PR #4553: the C6 push path must exit 0, never 1. - # Before the fix apply_required_status_checks.py called sys.exit(1) on - # every push, turning main red whenever branch protection was not yet - # configured -- hiding real failures behind a permanent red badge. - # Named here because this job runs FILE LISTS. - - name: C6 push path is non-blocking (must exit 0 on push) - run: python3 -m pytest tests/test_c6_push_non_blocking.py -q - # Every merge-blocking check must have a path to green WITHOUT privileged # secrets. A gated check that requires one leaves forks, outside # contributors, and everyone with an open PR at credential-rotation time @@ -762,6 +754,10 @@ jobs: tests/test_otel_export_sessions_shape.py \ tests/test_query_contract_drift.py \ tests/test_public_api_keys.py \ + tests/test_ingest_key.py \ + tests/test_ingest_contract_drift.py \ + tests/test_setup_prompt.py \ + tests/test_ingest_status.py \ tests/test_query_contract_goldens.py \ tests/test_local_store_concurrent_flush_1590.py \ tests/test_duckdb_invalidated_recovery.py \ diff --git a/CHANGELOG.md b/CHANGELOG.md index d54df26900..dd3e1be6f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -334,6 +334,39 @@ - **Cloud is unaffected by construction:** the hosted container has no discovery file, so no round trip is attempted and nothing is stamped. - **Verified:** 17 tests in `tests/test_store_unreachable_is_not_empty.py`, named in `ci.yml`, each proven to fail against the unfixed code before being trusted. Closes #5534. +### Added: the first-run gate now says whether data is actually arriving (2026-09-08) +- **Why:** a user who installs ClawMetry and sees an empty dashboard cannot tell "nothing is running" from "it is broken", and that question is what kills setup funnels. We have the number: 285 launches produced 13 choices in 14 days on the old gate. The gate already named the runtimes it detected; it never said whether any of them had produced a single event. +- **What:** `GET /api/onboarding/ingest-status` answers it from real data — total events, the runtimes actually sending, and the OTLP receiver's own state — and the gate renders it as a live strip that polls while it is open. When nothing has arrived the strip says so **and what to do about it**, including `clawmetry setup-prompt` for an agent that runs somewhere else. It never blocks: this is a confirmation, not a step, in a flow whose whole selling point is having no steps. +- **Two clocks, kept apart on purpose.** The durable count comes from DuckDB and survives a restart; the OTLP receiver's counters are in-process and empty on restart, so they are reported separately and named `has_data_this_process`. Folding them together would tell a working install it was broken every time the dashboard restarted. +- **A runtime we merely know about is not a runtime that is sending.** Rows with no sessions and no tokens are dropped rather than listed, because listing them would answer "is anything arriving?" with a yes they have not earned — the same shape as a tab that renders empty and calls it success. And the rollup is one row per runtime *per day*, so a runtime sending for a week arrived seven times; the endpoint collapses to one row each. +- **Read through `local_query._dispatch`, never from raw files**, so it answers identically on a laptop and in a cloud container with no `~/.openclaw`. Memoised for 2s because it is polled: the first call is ~200 ms against a real store, the rest are ~0.2 ms. +- **Verified in both states against a running dashboard:** the strip renders inside the card in the real page, an empty store returns `connected: false` with an actionable `next_step`, and the populated store on this machine returns 25,349 events across 9 runtimes. 9 guards in `tests/test_ingest_status.py`, three mutation-proven — reporting an idle runtime as a source, dropping the memo, and never stopping the poll each turn one red. The last one exists because a dismissed modal that keeps fetching is the same defect as the Home widget that fetched every sub-agent into a hidden element. +- **Refs** #5680. + +### Added: a setup prompt you hand to your agent (2026-09-08) +- **Why:** ClawMetry's users delegate work to coding agents by definition — that is what the product observes — yet the only setup paths shipped were "run the installer" and "read a doc", neither aimed at the thing the user actually drives. `clawmetry setup-prompt ` and `GET /api/setup-prompt` print a prompt written for the agent, for the case auto-detection cannot cover: an agent in CI, a container, a serverless function, or on somebody else's laptop. +- **Generated, not written.** Every endpoint, header, content type and cap comes from `clawmetry/ingest_contract.py`, the same declaration the server validates against. A hand-written prompt drifts the first time a header is renamed, and a drifted setup prompt is worse than none: the agent writes the wrong header *confidently* and the request fails where nobody is looking. +- **Half of it is negative space**, which is the useful half. Coding agents reliably mis-substitute secrets, "correct" a content type that was already right, and invent config keys that look plausible. So the prompt says the key goes in one header and nowhere else; that the placeholder is a placeholder and the real key must be asked for, not invented; that both encodings are already accepted and need no fixing; that a key not in the prompt does not exist; and that this is observability — it watches, it does not change what runs. It ends by making the agent verify and report the real event count, because an agent that checks its own work fails loudly instead of silently. +- **Two guards worth keeping.** The first reads *backwards*: every `x-clawmetry-*` token in the prompt must be a header the contract declares. Checking only that the right headers appear was too weak — proven by mutation, where swapping the config block's header for `x-clawmetry-apikey` left every other assertion green because the correct name still appeared in the prose. The second is general: **every registered subcommand must also be in `cli.py`'s `_subcmds` allowlist**, because a parser without an entry there falls through to the dashboard's argparse and dies with "invalid choice", which reads like the command was never written. `setup-prompt` did exactly that when first added — the same two-list trap CLAUDE.md documents for runtimes — and asserting only that `setup-prompt` is present would not have prevented the next one. +- **The seam holds:** this module names no runtime and hardcodes no vendor value, pinned by a test. Where a runtime has registered an OTel profile (paid runtimes register theirs from clawmetry-pro), its label and `clawmetry instrument` support are read at render time; a free install gets the generic OTLP prompt, which works. +- **Refs** #5681. + +### Added: the ingest contract is declared once, and the reference is generated from it (2026-09-08) +- **Why:** four things describe what ClawMetry accepts — the server that validates requests, `docs/INGEST.md`, the per-runtime setup prompts, and the public reference on the landing site — and none of them shared a source. Four hand-maintained descriptions of one contract is four chances to drift, and the drift is worst in the prompts: a prompt that teaches an agent a header we do not accept is worse than shipping no prompt at all, because the agent writes it confidently and the failure is silent. The landing side now enforces this from the other direction too — drift-bot fails a public claim the repo denies. +- **What:** `clawmetry/ingest_contract.py` declares the surfaces, headers, encodings, caps, response codes, auth modes and GenAI attributes as data. `clawmetry/ingest_auth.py` imports its constants from there rather than keeping its own, so the thing the server enforces and the thing we publish cannot diverge. `scripts/gen_ingest_doc.py` renders `docs/INGEST.md` from it with a `--check` mode, on the same pattern as `gen_query_contract_doc.py` on the read side. +- **The guard that earns the file.** Declaring "we read `gen_ai.usage.cache_read.input_tokens`" is cheap; the claim is only worth printing if something checks it. The test asserts every attribute declared read is actually named in the mapper **and** that every attribute declared unread really is — both directions, because someone wiring one up should have to move it in the same change, so the published reference is never behind the code either. Writing that check is what surfaced #5685: we were advertising a convention we did not implement, and cached tokens were being priced as free. +- **The doc says what we do NOT accept**, and that section is pinned by a test. There is no endpoint for syslog, CEF, GELF or raw text, and there is not going to be: ClawMetry's inputs are typed on arrival, and a parser layer would exist only to accept data this product has nothing to say about. A reference that only says what works is not one anybody can plan against. +- **Verified by mutation:** claiming a made-up attribute is read turns 2 guards red; claiming a genuinely-read one is unread turns 3 red. 26 guards in `tests/test_ingest_contract_drift.py`, registered in `ci.yml`. +- **Refs** #5682. + +### Added: an ingest key, so an agent that is not on this machine can be observed (2026-09-08) +- **Why:** until now an agent was observable by ClawMetry only if the daemon ran on the same machine as the agent. `/v1/{logs,metrics,traces}` trusted loopback and otherwise wanted the OpenClaw gateway token; the custom-runtime write API trusted loopback or one static `CLAWMETRY_INGEST_TOKEN` shared by the whole install — no rotation, no revocation, no way to tell two pushers apart. Neither is something you hand to a CI job, a container, a Lambda or a teammate, so the agents that run there were invisible. That is one capability, not a catalogue, and it is what stands between the product and "every agent, anywhere it runs". +- **What:** `write:ingest`, a scoped key created with `clawmetry key create --name ci --scope write:ingest` and presented as `x-clawmetry-key: cmk_...` on the three OTLP endpoints. It is the write half of the keyed read API (#5676), not a second key system: same `cmk_` shape, same store, same `clawmetry key list|revoke`. Plus two routing headers — `x-clawmetry-runtime` and `x-clawmetry-env` — because a pushed batch carries no filesystem layout to infer a runtime from. They are written into the resource attributes the mappers already read (`service.name`, `deployment.environment`), so a header is exactly as powerful as the equivalent exporter setting and no mapper learns a second way to answer the same question. Deliberately one grouping axis, not a dataset/collection/tag taxonomy: that is what a log platform needs and an agent platform does not. +- **The posture, stated plainly.** An ingest key **can only push**: `write:ingest` grants no `q/1` shape, so a key handed to a CI runner cannot read a prompt, a cost or a session back out, and presenting one to `/api/q/1` is a `403` rather than an index it could never follow up on. It is **never given a CORS header** and `apikeys.create` refuses to put a browser origin on one, so a page cannot hold one usefully — a write surface is not the place to hand back the protection the read API was careful to keep. Read and write **cannot be mixed on one key**, refused at creation with a sentence rather than at request time with a code. And there is **one gate, not two**: `_check_auth` steps aside for a keyed `/v1/` request exactly as it does for `/api/q/`, so `clawmetry/ingest_auth.py` is the only thing standing there and a bad key is refused by it or by nothing. +- **Every refusal carries a sentence.** These are read inside an agent's terminal output with no documentation open, so `401` says how to create a key, `403` says the key is fine but may not push, `400` on a bad runtime header shows the shape it wanted, `400` on a bad body names both accepted encodings, and `413` gives the size and the limit instead of failing somewhere inside a protobuf parser. +- **Verified live, not only in tests.** Against a real dashboard on port 8917 with a gateway token set: all three doors return 200 (loopback with no key at all — the zero-config path, unchanged; gateway token; ingest key), and every refusal returns its own status with its own sentence. A span pushed with `x-clawmetry-runtime: my-engine` lands in DuckDB as `agent_type=my_engine, service_name=my-engine`, while the same span pushed without headers still lands as `openclaw`/`unknown_service` — the pre-existing behaviour, untouched. The live run is also what found the `/api/q/1` hole: a write-only key was being handed a 200 index, now a 403, pinned by a test. 25 guards in `tests/test_ingest_key.py`. +- **Refs** #5679. Part of phase 1 of the ingest plan (#5680, #5681, #5682, clawmetry-cloud#2343, #2344, clawmetry-pro#230). + ### Fixed: two functions named `_session_cwd`, and the later one silently replaced the other (2026-09-08) - **Why:** shipped in 0.12.837 and live through 0.12.839. `sync.py` already had `_session_cwd(row)`, a raw adapter/gateway dict read through a **twelve-alias** set (`cwd`, `workingDir`, `workingDirectory`, `workspace`, `workspaceRoot`, `project_dir`, `projectRoot`, `directory`, `folder`, ...). The workspace-scan change added a second function with the same name 2,000 lines below it, reading two keys. Python keeps the last definition and says nothing, so all THREE callers of the original quietly switched: gateway session shaping (`sync.py:4328`), the session row build (`4586`), and the FAMILY ingest cwd (`15293`). That last one writes `sessions.cwd`, which is the column `process_control` promotes to find a pid and the one the workspace scanner keys on, so a runtime that spells its directory `workingDir` or `directory` in metadata stopped persisting it. Nothing failed. The tests passed. Three wheels shipped. - **What:** the newer helper becomes `_session_row_cwd` (it reads a store ROW: the `cwd` column first, then metadata), and its metadata fallback now DELEGATES to `_session_cwd`, so both paths honour the same alias set. Both are strictly better than before the collision. A second shadowing found by the same walk is also removed: `start_log_streamer` was defined twice, the first an empty docstring-only stub. diff --git a/clawmetry/apikeys.py b/clawmetry/apikeys.py index 5582c3c0c7..cc7445e2e5 100644 --- a/clawmetry/apikeys.py +++ b/clawmetry/apikeys.py @@ -34,9 +34,14 @@ is created with ``--origin none`` and simply never gets a CORS header. * Scopes are least-revealing-first and ``read:content`` is never granted implicitly -- it has to be asked for by name. -* Keys are read-only. Nothing in this module can pause, stop or kill an - agent, and ``routes/public_api.py`` dispatches only ``q/1`` read - shapes, so this adds nothing to ClawMetry's control plane. +* Read scopes are read-only. ``routes/public_api.py`` dispatches only + ``q/1`` read shapes, so a browser-resident key adds nothing to + ClawMetry's control plane. +* There is exactly one write scope, ``write:ingest``, and it can only + push telemetry IN. It cannot read a single byte back, and nothing in + this module can pause, stop or kill an agent. An ingest key is + server-to-server: it is never granted a CORS header, so a page cannot + hold one usefully (see ``ingest_auth``). Storage ------- @@ -57,7 +62,11 @@ import time from typing import Any, Optional -from clawmetry.query_contract import SCOPE_CONTENT, SCOPE_DOC, SCOPES +from clawmetry.query_contract import ( + SCOPE_CONTENT, + SCOPE_DOC as _READ_SCOPE_DOC, + SCOPES as READ_SCOPES, +) # ── Shape of the thing ────────────────────────────────────────────────── @@ -77,6 +86,23 @@ #: script cannot grow the file without bound; it is not a paywall. MAX_KEYS = 50 +#: The one write scope. It lives here rather than in ``query_contract`` +#: on purpose: that module declares what can be READ, shape by shape, +#: and a scope with no shape behind it would be a lie in that table. +#: Ingest is the opposite direction and has no q/1 method at all. +SCOPE_INGEST = "write:ingest" + +#: Every scope a key may carry. Read scopes stay in their declared +#: least-revealing-first order; the write scope sorts last because it is +#: the one a reader should notice. +SCOPES: tuple = tuple(READ_SCOPES) + (SCOPE_INGEST,) + +SCOPE_DOC: dict = dict(_READ_SCOPE_DOC) +SCOPE_DOC[SCOPE_INGEST] = ( + "Push telemetry in: OTLP logs, metrics and traces, and run events. " + "Grants no read access of any kind." +) + #: Sentinel origin meaning "this key is not used from a browser". Stored #: as an empty origin list; kept as a word so the CLI can say it back. ORIGIN_NONE = "none" @@ -292,6 +318,25 @@ def create(name: str, scopes, origins, *, note: str = "") -> tuple: scope_list = normalise_scopes(scopes) origin_list = normalise_origins(origins) + # An ingest key is server-to-server and is never granted a CORS + # header, so browser origins on one would be dead configuration that + # reads like a permission. Refusing the mix also keeps a single key + # from being both "pasted into a web page" and "allowed to write", + # which is the combination worth not having. + if SCOPE_INGEST in scope_list: + if len(scope_list) > 1: + raise ApiKeyError( + "An ingest key does one job. Create it with write:ingest " + "alone, and mint a separate read key for anything that " + "needs to read data back." + ) + if origin_list: + raise ApiKeyError( + "An ingest key is used by a server, a container or a CI " + "job, never by a browser, so it takes no origin. Create " + "it with --origin none." + ) + doc = _read_store() live = [k for k in doc["keys"] if not k.get("revoked_at")] if len(live) >= MAX_KEYS: @@ -507,22 +552,41 @@ def granted_shapes(record: dict) -> set: def scope_catalogue() -> list: - """``[{scope, doc, methods, sensitive}]`` for the UI and the CLI help. + """``[{scope, kind, doc, methods, sensitive}]`` for the UI and CLI help. - Derived from the query contract, so a method added there shows up - here with no second list to update. + Read scopes are derived from the query contract, so a method added + there shows up here with no second list to update. + + ``write:ingest`` has no ``q/1`` method behind it and never will -- + it is the other direction. It carries ``methods: []`` with + ``kind: "write"``, so a caller can tell "this scope reads nothing" + apart from "this scope's method list failed to load". An empty list + with no explanation is the kind of thing that sends someone to the + source to find out whether the UI is broken. """ from clawmetry.query_contract import live_methods_by_scope - return [ - { + rows = [] + for s in SCOPES: + write = s == SCOPE_INGEST + rows.append({ "scope": s, + "kind": "write" if write else "read", "doc": SCOPE_DOC[s], - "methods": live_methods_by_scope(s), + "methods": [] if write else live_methods_by_scope(s), "sensitive": s == SCOPE_CONTENT, - } - for s in SCOPES - ] + }) + return rows + + +def allows_ingest(record: dict) -> bool: + """True when this key may push telemetry in. + + The single question the ingest path asks. Kept next to the scope it + checks so a future scope rename cannot leave a stale string literal + behind in a route module. + """ + return SCOPE_INGEST in (record.get("scopes") or []) def redact(presented: str) -> str: diff --git a/clawmetry/cli.py b/clawmetry/cli.py index 68bccbfd81..d48a22fbad 100644 --- a/clawmetry/cli.py +++ b/clawmetry/cli.py @@ -4872,7 +4872,8 @@ def _cmd_key(args) -> None: import time as _time from clawmetry import apikeys as _ak - from clawmetry.query_contract import SCOPE_CONTENT, SCOPE_DOC, SCOPE_METRICS + from clawmetry.apikeys import SCOPE_DOC, SCOPE_INGEST + from clawmetry.query_contract import SCOPE_CONTENT, SCOPE_METRICS action = getattr(args, "key_cmd", None) or "list" as_json = bool(getattr(args, "as_json", False)) @@ -4899,7 +4900,10 @@ def _fmt_age(ts): flag = " (sensitive)" if row["sensitive"] else "" print(f" {row['scope']}{flag}") print(f" {row['doc']}") - print(f" queries: {', '.join(row['methods'])}") + if row["kind"] == "write": + print(" queries: none. This scope only pushes data in.") + else: + print(f" queries: {', '.join(row['methods'])}") print("") print("Pick the narrowest scope that makes your UI work. A key that") print("only needs a cost chart should be read:metrics, so it cannot") @@ -4954,6 +4958,11 @@ def _fmt_age(ts): wants_no_origin = any( str(o).strip().lower() == _ak.ORIGIN_NONE for o in raw_origins ) + if SCOPE_INGEST in scopes and not raw_origins: + # An ingest key is server-to-server by definition, so asking + # which website may use it is a question with no answer. + wants_no_origin = True + raw_origins = [_ak.ORIGIN_NONE] if not raw_origins: print("A key needs to know which site may use it from a browser.") print("") @@ -4993,6 +5002,27 @@ def _fmt_age(ts): print(f" {plaintext}") # codeql[py/clear-text-logging-sensitive-data] print("") print(f"Name: {record['name']} (id {record['id']})") + if SCOPE_INGEST in record["scopes"]: + print(f"Grants: {', '.join(record['scopes'])}") + print(f" {SCOPE_INGEST}: {SCOPE_DOC[SCOPE_INGEST]}") + print("Origins: none. Ingest is server-to-server; this key is never") + print(" given a CORS header, so a web page cannot use it.") + print("") + print("Push a span from anywhere that can reach this machine:") + print("") + print(" curl -X POST http://localhost:8900/v1/traces \\") + print(f" -H 'x-clawmetry-key: {plaintext}' \\") + print(" -H 'x-clawmetry-runtime: my-engine' \\") + print(" -H 'x-clawmetry-env: production' \\") + print(" -H 'Content-Type: application/json' \\") + print(" --data-binary @spans.json") + print("") + print("OTLP protobuf and OTLP/JSON are both accepted, gzip too.") + print("The runtime and env headers are optional; without them the") + print("runtime is taken from the resource's service.name.") + print("") + print("Reference: docs/CUSTOM_RUNTIME_INGEST.md") + return print(f"Reads: {', '.join(record['scopes'])}") for s in record["scopes"]: print(f" {s}: {SCOPE_DOC[s]}") @@ -5025,6 +5055,44 @@ def _fmt_age(ts): raise SystemExit(1) +def _cmd_setup_prompt(args) -> None: + """`clawmetry setup-prompt [runtime]` -- the prompt you hand your agent. + + ClawMetry detects agents on this machine with no configuration. This + is for the other case: an agent in CI, a container, a serverless + function or on someone else's laptop, which has to push instead. + + The text is generated from the ingest contract, so it cannot tell an + agent to send a header the server does not read -- which is the + failure worth designing against, because an agent writes a wrong + header confidently and the request fails where nobody is looking. + """ + from clawmetry import setup_prompt as _sp + + runtime = (getattr(args, "runtime", "") or "").strip().lower() + if runtime and not _sp.VALID_RUNTIME.match(runtime): + print( + f"{runtime!r} is not a runtime name. Use a short name like " + "claude_code or my-engine: lower-case letters, digits, " + "underscore and dash, 40 characters at most." + ) + raise SystemExit(1) + + port = getattr(args, "port", None) or 8900 + endpoint = (getattr(args, "endpoint", "") or f"http://localhost:{port}").rstrip("/") + print(_sp.render(runtime, endpoint=endpoint)) + print("") + print("-" * 68) + print("Copy everything above into your coding agent.") + print("") + print("It needs a key. Create one, and paste it in place of the") + print("placeholder:") + print("") + print(" clawmetry key create --name ci --scope write:ingest") + print("") + print("Reference: docs/INGEST.md") + + def _cmd_reports(args) -> None: """Open the reports browser (refs #1005).""" import webbrowser @@ -8305,12 +8373,6 @@ def main() -> None: type=str, help="OpenClaw config directory (default: ~/.openclaw). Env: CLAWMETRY_OPENCLAW_DIR", ) - parser.add_argument( - "--sample", - action="store_true", - help="Open the dashboard on three synthetic sample sessions instead " - "of your own data (a separate store; your data is untouched)", - ) sub = parser.add_subparsers(dest="cmd") # onboard — first-time setup wizard (called by install.sh) @@ -8550,6 +8612,20 @@ def main() -> None: ) # reports — open the reports browser (refs #1005) + p_setup_prompt = sub.add_parser( + "setup-prompt", + help="Print the prompt that points an off-box agent at this ClawMetry", + ) + p_setup_prompt.add_argument( + "runtime", nargs="?", default="", + help="Runtime being pointed here (claude_code, my-engine, ...)", + ) + p_setup_prompt.add_argument( + "--endpoint", default="", + help="Where the agent should send data (default http://localhost:)", + ) + p_setup_prompt.add_argument("--port", type=int, default=8900) + p_reports = sub.add_parser( "reports", help="Open the reports browser (renders ~/.clawmetry/reports/*.md + DuckDB SQL)", @@ -9297,6 +9373,7 @@ def main() -> None: "reports", "eval", "key", + "setup-prompt", "mcp", "update", "uninstall", @@ -9353,32 +9430,6 @@ def main() -> None: parser.print_help() sys.exit(0) - # --sample: load synthetic sessions instead of the user's own, so a fresh - # install on a machine with no agent history is never an empty product. - # This has to happen BEFORE `from dashboard import ...` below, because - # local_store reads CLAWMETRY_LOCAL_STORE_PATH into its module-level - # DB_PATH at import time -- setting it afterwards would open the real - # store and then serve it under a "sample data" banner. - if "--sample" in sys.argv: - sys.argv = [a for a in sys.argv if a != "--sample"] - try: - from clawmetry import sample_data as _sample_data - _sample_path = _sample_data.enable_sample_mode() - _n_sessions, _n_events = _sample_data.ensure_built() - if _n_sessions: - print(f"Sample data built: {_n_sessions} sessions, " - f"{_n_events} events -> {_sample_path}") - else: - print(f"Sample data ready -> {_sample_path}") - print("This is synthetic data, not your machine. " - "Restart without --sample for your own agents.") - except Exception as _e: - # Never make --sample a way to fail to start. - print(f"Could not build sample data ({_e}); " - "starting on your real store instead.", file=sys.stderr) - os.environ.pop("CLAWMETRY_SAMPLE", None) - os.environ.pop("CLAWMETRY_LOCAL_STORE_PATH", None) - # Tag this process as the dashboard BEFORE importing dashboard, so every # get_store() in dashboard.py (module-level + handlers) is barred from the # DuckDB writer — only the sync daemon writes. Set before the import or a @@ -9443,6 +9494,8 @@ def main() -> None: _cmd_mcp(args) elif args.cmd == "key": _cmd_key(args) + elif args.cmd == "setup-prompt": + _cmd_setup_prompt(args) elif args.cmd == "update": _cmd_update(args) elif args.cmd == "uninstall": diff --git a/clawmetry/ingest_auth.py b/clawmetry/ingest_auth.py new file mode 100644 index 0000000000..fe2f2eb129 --- /dev/null +++ b/clawmetry/ingest_auth.py @@ -0,0 +1,217 @@ +"""clawmetry/ingest_auth.py -- the gate in front of the ingest surfaces. + +Why this exists +--------------- +Until now an agent could only be observed by ClawMetry if the daemon ran +on the same machine as the agent. ``/v1/{logs,metrics,traces}`` trusted +loopback and otherwise wanted the OpenClaw gateway token, and the custom +runtime write API trusted loopback or one static secret shared by the +whole install. Neither is something you can hand to a CI job, a +container, a Lambda or a teammate, so the agents that run there were +simply invisible. + +This module adds one more way in: a scoped, revocable ingest key +(``clawmetry key create --scope write:ingest``). It does not replace the +existing paths. Loopback still works with no key at all, and the gateway +token still works exactly as it did -- this is a door, not a relocation. + +The posture, stated plainly +--------------------------- +* **An ingest key can only push.** ``write:ingest`` grants no ``q/1`` + shape (``granted_shapes`` returns an empty set for it), so a leaked + ingest key cannot read a prompt, a cost or a session back out. +* **No CORS, ever.** Ingest is server-to-server and POST-only. This + module never emits an ``Access-Control-Allow-Origin`` header and + ``apikeys.create`` refuses to put a browser origin on an ingest key, + so a page cannot hold one usefully. That is deliberate: the whole + protection ``routes/public_api.py`` buys is the browser refusing to + let a page read a reply, and a write surface should not be the place + it gets handed back. +* **One gate, not two.** ``dashboard.py::_check_auth`` steps aside for a + request that presents a key, the same way it does for ``/api/q/``, so + the check here is the only one. Two gates on one path is how a request + ends up accepted by the wrong one. + +Routing +------- +A pushed event carries no filesystem layout to infer a runtime from, so +the pusher says which runtime it is and (optionally) which environment, +in headers. Both are resolved once per request and stamped on every +event in it. ``x-clawmetry-env`` is the single grouping axis above +runtime -- deliberately one axis and not a dataset/collection/tag +taxonomy, because that taxonomy is what a log platform needs and an +agent platform does not. +""" + +from __future__ import annotations + +import logging +import re +from typing import Any, Optional + +logger = logging.getLogger("clawmetry.ingest_auth") + +# The header names and the body cap are declared ONCE, in +# ``clawmetry/ingest_contract.py``, and re-exported here. The doc, the +# setup prompts and the landing reference are generated from that same +# declaration, so the thing the server enforces and the thing we tell +# people to send cannot drift apart -- which matters most for the setup +# prompts, where a wrong header name is a silent failure an agent will +# write confidently. +from clawmetry.ingest_contract import ( # noqa: F401 (re-exported) + HEADER_ENV, + HEADER_KEY, + HEADER_RUNTIME, + MAX_BODY_BYTES, +) + +#: A runtime name we have never heard of is allowed -- in-house engines +#: are a supported case -- but it still has to be a name, not a payload. +_RUNTIME_RE = re.compile(r"^[a-z0-9][a-z0-9_-]{0,39}$") +_ENV_RE = re.compile(r"^[a-z0-9][a-z0-9_.-]{0,63}$") + + +def _err(code: str, message: str, status: int) -> tuple: + """One error shape for every rejection. + + ``message`` is a sentence for a person. A bare error code with no + sentence is the thing a user should never be shown -- they are + usually looking at it inside an agent's terminal output, with no + documentation open. + """ + return {"error": code, "message": message}, status + + +def presented_key(headers) -> str: + """The raw key string from the request, or ``""``.""" + return (headers.get(HEADER_KEY) or "").strip() + + +def authenticate(headers) -> tuple: + """``(record, None)`` when a live ingest key is presented, else + ``(None, (body, status))``. + + Callers that reach this function have already been let past + ``_check_auth`` on the strength of the header being present, so a + bad key must be rejected here or it is not rejected anywhere. + """ + from clawmetry import apikeys + + presented = presented_key(headers) + if not presented: + return None, _err( + "unauthorized", + f"This endpoint needs an ingest key in the {HEADER_KEY} header. " + "Create one with: clawmetry key create --name ci " + "--scope write:ingest --origin none", + 401, + ) + record = apikeys.verify(presented) + if not record: + logger.warning("ingest: rejected key %s", apikeys.redact(presented)) + return None, _err( + "unauthorized", + "That ingest key is not valid on this ClawMetry. It may have " + "been revoked, or it may belong to a different install.", + 401, + ) + if not apikeys.allows_ingest(record): + return None, _err( + "forbidden", + f"Key {record.get('name') or record.get('id')} may read, but it " + "cannot push. Create a separate key with " + "--scope write:ingest --origin none.", + 403, + ) + apikeys.touch(record.get("id") or "") + return record, None + + +def resolve_runtime(headers) -> tuple: + """``(runtime, None)`` or ``(None, (body, status))``. + + ``None`` for the runtime means "the pusher did not say", which is + fine: the OTLP path already derives a runtime from the resource's + ``service.name``. The header, when present, wins -- an explicit + statement by the pusher beats a value inferred from a field that + exists for a different purpose. + """ + raw = (headers.get(HEADER_RUNTIME) or "").strip().lower() + if not raw: + return None, None + if not _RUNTIME_RE.match(raw): + return None, _err( + "bad_runtime", + f"{HEADER_RUNTIME} must be a short name like claude_code or " + "my-engine: lower-case letters, digits, underscore and dash, " + "40 characters at most.", + 400, + ) + return raw, None + + +def resolve_env(headers) -> tuple: + """``(env, None)`` or ``(None, (body, status))``. ``None`` is fine.""" + raw = (headers.get(HEADER_ENV) or "").strip().lower() + if not raw: + return None, None + if not _ENV_RE.match(raw): + return None, _err( + "bad_env", + f"{HEADER_ENV} must be a short label like production or " + "team-a.staging: lower-case letters, digits, dot, underscore " + "and dash, 64 characters at most.", + 400, + ) + return raw, None + + +def check_size(body: Any) -> Optional[tuple]: + """``None`` when the body fits, else ``(body, 413)``. + + Checked before decode: a 40 MB protobuf should be refused with a + sentence, not parsed and then rejected by whatever runs out of + patience first. + """ + try: + size = len(body or b"") + except TypeError: + return None + if size > MAX_BODY_BYTES: + mb = MAX_BODY_BYTES // (1024 * 1024) + return _err( + "payload_too_large", + f"That request is {size // (1024 * 1024)} MB and the limit is " + f"{mb} MB. Split the batch and send it in parts; compressing " + "with Content-Encoding: gzip also helps.", + 413, + ) + return None + + +def prologue(headers, body) -> tuple: + """Everything the ingest surfaces check, in one call. + + Returns ``(context, None)`` or ``(None, (body, status))``. The + context is ``{"key": record|None, "runtime": str|None, + "env": str|None}``; ``key`` is ``None`` for a request that got in on + loopback or the gateway token, which stays legal. + """ + err = check_size(body) + if err: + return None, err + + record = None + if presented_key(headers): + record, err = authenticate(headers) + if err: + return None, err + + runtime, err = resolve_runtime(headers) + if err: + return None, err + env, err = resolve_env(headers) + if err: + return None, err + + return {"key": record, "runtime": runtime, "env": env}, None diff --git a/clawmetry/ingest_contract.py b/clawmetry/ingest_contract.py index bf335db99c..8d953de712 100644 --- a/clawmetry/ingest_contract.py +++ b/clawmetry/ingest_contract.py @@ -1,210 +1,255 @@ -"""clawmetry/ingest_contract.py — the declared ingest/1 contract registry. +"""clawmetry/ingest_contract.py -- what ClawMetry accepts, declared once. + +Why this module exists +---------------------- +Four things describe the ingest surface, and until now none of them +shared a source: + +1. the server, which validates requests; +2. ``docs/INGEST.md``, which a person reads; +3. the per-runtime setup prompts, which an AGENT reads -- and where a + wrong header name is a silent failure, because the agent will + confidently write it; +4. the public reference on the landing site, where a claim the repo + denies now fails the truthfulness gate. + +Four hand-maintained descriptions of one contract is four chances to +drift, and the drift is worst in (3): a prompt that teaches an agent a +header we do not accept is worse than no prompt at all. + +So the contract is data. The server reads its constants from here, the +doc is generated from here, and CI fails when the committed doc and this +file disagree -- the same shape as ``query_contract.py`` and +``gen_query_contract_doc.py`` for the read side. + +What this is NOT +---------------- +Not a schema validator and not a parser registry. ClawMetry's inputs are +typed on arrival: OTLP has its own proto, and the run/event API takes a +declared shape. There is deliberately no "accepts any format" surface +here to describe, because accepting syslog, CEF or raw text would mean +accepting data this product has nothing to say about. +""" -Single source of truth for the two ingest surfaces ClawMetry exposes: +from __future__ import annotations -* **OTLP receiver** — ``/v1/metrics``, ``/v1/traces``, ``/v1/logs`` - (standard OpenTelemetry HTTP/JSON and HTTP/protobuf, plus gzip). -* **Run/event ingest API** — ``/api/v1/runs*`` - (structured run/step records from custom runtimes; Pro feature). +CONTRACT_VERSION = "ingest/1" -``docs/INGEST.md`` is generated from this module by -``scripts/gen_ingest_doc.py``; ``tests/test_ingest_doc_drift.py`` fails CI -when the committed doc drifts from the generator output. +# ── Headers ───────────────────────────────────────────────────────────── +# Lower-case on the wire. HTTP header lookup is case-insensitive, but +# every example we publish uses this spelling so a copy-paste, a grep and +# an agent's guess all agree. -Versioning: evolution inside ``ingest/1`` is additive only. Adding an -endpoint, a content-type, or an attribute is fine. Removing or renaming -one requires bumping to ``ingest/2``. -""" -from __future__ import annotations +HEADER_KEY = "x-clawmetry-key" +HEADER_RUNTIME = "x-clawmetry-runtime" +HEADER_ENV = "x-clawmetry-env" +HEADER_LEGACY_TOKEN = "X-ClawMetry-Token" -CONTRACT_VERSION = "ingest/1" +#: Body cap, enforced before decode. A 40 MB protobuf is refused with a +#: sentence rather than parsed until something runs out of patience. +MAX_BODY_BYTES = 10 * 1024 * 1024 -# ── OTLP receiver ──────────────────────────────────────────────────────────── +#: Events per run/event request. Already the write API's cap. +MAX_EVENTS_PER_BATCH = 1000 -# Endpoints served by ``routes/meta.py`` (bp_otel / bp_otlp_traces). -OTLP_ENDPOINTS: dict[str, dict] = { - "/v1/metrics": { - "methods": ["POST"], +HEADERS = ( + { + "name": HEADER_KEY, + "required": "for a caller that is not on this machine", + "doc": "An ingest key: clawmetry key create --name ci --scope write:ingest", + }, + { + "name": HEADER_RUNTIME, + "required": "no", "doc": ( - "OTLP metrics. Ingests ``gen_ai.client.token.usage`` counters " - "and ``gen_ai.client.operation.duration`` histograms into the " - "token/cost tiles." + "Which runtime is pushing -- claude_code, my-engine, anything " + "matching [a-z0-9][a-z0-9_-]{0,39}. An unrecognised name is " + "accepted: in-house engines are a supported case. Sets the " + "resource's service.name, which is what the runtime is derived " + "from. Without it the runtime comes from service.name as sent." ), - "json_support": True, - "protobuf_support": True, }, - "/v1/traces": { - "methods": ["POST"], + { + "name": HEADER_ENV, + "required": "no", "doc": ( - "OTLP traces. Ingests GenAI spans (LLM calls, tool calls, " - "sub-agent spawns) into the span tree and session timeline." + "Environment or project label -- production, team-a.staging. " + "Sets deployment.environment. This is the one grouping axis " + "above runtime, and deliberately the only one." ), - "json_support": True, - "protobuf_support": True, }, - "/v1/logs": { - "methods": ["POST"], + { + "name": "Content-Type", + "required": "yes, on OTLP", + "doc": "application/x-protobuf or application/json. See encodings.", + }, + { + "name": "Content-Encoding", + "required": "no", + "doc": "gzip, if you compressed the body.", + }, +) + +# ── Surfaces ──────────────────────────────────────────────────────────── + +SURFACES = ( + { + "path": "/v1/traces", + "method": "POST", + "accepts": "OTLP traces", "doc": ( - "OTLP logs. Ingests the agent event stream exported by " - "Claude Code, Codex, and other runtimes (cost/token/model " - "per log record) into the cost and usage tiles." + "Spans. The GenAI convention's model-call spans land here, and " + "this is the surface an OTel SDK or Collector already speaks." ), - "json_support": True, - "protobuf_support": True, - }, -} - -# Content-Type values accepted on all three OTLP endpoints. -# Source: ``dashboard.py::_otlp_decode``. -OTLP_CONTENT_TYPES: dict[str, str] = { - "application/x-protobuf": ( - "Binary protobuf encoding. Requires ``pip install clawmetry[otel]`` " - "(``opentelemetry-proto`` + ``protobuf``). Raises HTTP 501 when the " - "extra is absent." - ), - "application/json": ( - "OTLP/JSON encoding. Works on a plain ``pip install clawmetry`` with " - "no extras. ``ignore_unknown_fields`` is set, so forward-compat keys " - "from newer producers do not cause a 400." - ), -} - -# Content-Encoding values accepted on all OTLP endpoints. -# Source: ``dashboard.py::_gunzip_safe``. -OTLP_ENCODINGS: list[str] = ["identity", "gzip"] - -# Hard cap on the decompressed body size. -# Source: ``dashboard.py::_OTLP_MAX_DECOMPRESSED``. -OTLP_MAX_DECOMPRESSED_MB: int = 64 -OTLP_MAX_DECOMPRESSED_ENVVAR: str = "CLAWMETRY_OTLP_MAX_DECOMPRESSED_MB" - -# HTTP response codes returned by the OTLP receiver. -# Source: ``routes/meta.py::_otlp_receive``. -OTLP_RESPONSE_CODES: dict[int, str] = { - 200: "Accepted. Response body is ``{}`` (empty JSON object).", - 400: "Malformed or undecodable body. Response body contains ``{\"error\": \"\"}``.", - 429: "Budget limit exceeded; OTLP intake is paused. Response body contains ``{\"paused\": true}``.", - 501: ( - "Binary protobuf body received but ``opentelemetry-proto`` is not " - "installed. Install with ``pip install clawmetry[otel]`` or switch " - "to OTLP/JSON (``Content-Type: application/json``)." - ), -} - -# ── gen_ai.* attribute mapping ─────────────────────────────────────────────── - -# Span / log-record attributes read by ``dashboard.py::_otel_to_row``. -# Values are (description, fallback_attrs) tuples. -GEN_AI_ATTRS_READ: dict[str, tuple[str, list[str]]] = { - "gen_ai.request.model": ( - "Model name used for the request.", - ["gen_ai.response.model", "llm.model", "model"], - ), - "gen_ai.usage.input_tokens": ( - "Input token count.", - ["llm.usage.prompt_tokens", "input_tokens"], - ), - "gen_ai.usage.output_tokens": ( - "Output token count.", - ["llm.usage.completion_tokens", "output_tokens"], - ), - "gen_ai.usage.total_tokens": ( - "Total token count.", - ["llm.usage.total_tokens", "total_tokens"], - ), - "gen_ai.usage.cache_read.input_tokens": ( - "Cached input tokens read (Anthropic / OpenAI prompt caching).", - ["gen_ai.usage.cache_read_input_tokens", "cache_read_input_tokens"], - ), - "gen_ai.usage.cache_creation.input_tokens": ( - "Cache-write tokens (Anthropic prompt caching).", - ["gen_ai.usage.cache_creation_input_tokens", "cache_creation_input_tokens"], - ), - "gen_ai.usage.cost_usd": ( - "Pre-computed cost in USD. When absent, ClawMetry prices the tokens locally.", - ["llm.usage.cost", "cost_usd"], - ), - "gen_ai.provider.name": ( - "Provider string (e.g. ``anthropic``, ``openai``).", - ["gen_ai.system", "llm.provider", "provider"], - ), - "gen_ai.tool.name": ( - "Name of the tool called (on ``execute_tool`` spans).", - ["tool.name", "code.function"], - ), - "gen_ai.conversation.id": ( - "Session or conversation identifier.", - ["session.id", "openclaw.session_id", "session_id"], - ), - "gen_ai.agent.id": ( - "Agent identifier.", - ["agent.id", "openclaw.agent_id", "agent_id"], - ), - "gen_ai.input.messages": ( - "Input message list (current GenAI semconv).", - ["gen_ai.prompt"], - ), - "gen_ai.output.messages": ( - "Output message list (current GenAI semconv).", - ["gen_ai.completion"], - ), - "gen_ai.operation.name": ( - "Operation kind (``chat``, ``text_completion``, ``generate_content``). " - "Read to decide whether a span counts as a run: OpenLLMetry and " - "traceloop-sdk name LLM spans ``.chat`` / ``.completion`` " - "and tag the operation here, so without it a bring-your-own-agent " - "install records spans while the live Runs tile stays at zero.", - [], - ), -} - -# Attributes that appear in GenAI semconv but are NOT yet consumed. -# -# This list is published in docs/INGEST.md, so a name here is a promise to the -# reader that sending it changes nothing. ``gen_ai.operation.name`` was listed -# and was in fact read by ``dashboard.py::_process_otlp_traces`` to classify a -# span as a run -- caught by this module's own drift check (#5682). Verify -# against the code before adding a name, not against intent. -# -# ``gen_ai.agent.name`` is genuinely unread on the ingest path: ClawMetry's own -# exporter WRITES it (``clawmetry/otel_exporter.py``), and nothing reads it back. -GEN_AI_ATTRS_NOT_READ: list[str] = [ - "gen_ai.agent.name", + }, + { + "path": "/v1/metrics", + "method": "POST", + "accepts": "OTLP metrics", + "doc": "Counters and histograms.", + }, + { + "path": "/v1/logs", + "method": "POST", + "accepts": "OTLP logs", + "doc": ( + "Log records. Claude Code and Codex export their per-turn event " + "stream this way, with cost and tokens per record." + ), + }, + { + "path": "/api/v1/runs", + "method": "POST", + "accepts": "JSON", + "doc": "Open a run; returns run_id. For engines that would rather " + "push typed records than build OTLP.", + }, + { + "path": "/api/v1/runs//events", + "method": "POST", + "accepts": "JSON", + "doc": f"Append one event or a batch of up to {MAX_EVENTS_PER_BATCH}.", + }, + { + "path": "/api/v1/runs//end", + "method": "POST", + "accepts": "JSON", + "doc": "Mark the run ended. Optional.", + }, + { + "path": "/api/v1/runs/", + "method": "GET", + "accepts": "-", + "doc": "Read back: was the run persisted?", + }, + { + "path": "/api/v1/runtimes", + "method": "GET", + "accepts": "-", + "doc": "The runtimes ClawMetry knows about. Free, no key needed.", + }, +) + +# ── Encodings ─────────────────────────────────────────────────────────── + +CONTENT_TYPES = ( + ("application/x-protobuf", "OTLP protobuf. The default for the OTel " + "Collector and the SDKs, and smaller on the wire."), + ("application/json", "OTLP/JSON. Useful when a client cannot produce " + "protobuf -- a script, a serverless function, a " + "platform that only emits JSON."), +) + +CONTENT_ENCODINGS = ( + ("gzip", "Accepted on every surface."), +) + +# ── Responses ─────────────────────────────────────────────────────────── +# Every one of these carries a sentence in its body, not only a code. +# These are read inside an agent's terminal with no documentation open. + +RESPONSES = ( + (200, "Accepted."), + (400, "The body did not decode, or a routing header was malformed. The " + "message names what was expected."), + (401, "No key, or a key this ClawMetry does not know. It may have been " + "revoked, or belong to a different install."), + (403, "A valid key that lacks write:ingest. Different from 401 on " + "purpose: 'your key is wrong' and 'your key is fine and may not " + "do this' send you to different fixes."), + (413, f"Body over {MAX_BODY_BYTES // (1024 * 1024)} MB. Split the batch, " + "or compress with Content-Encoding: gzip."), + (429, "Intake is paused because a budget limit was exceeded."), + (501, "OTLP support is not installed on this ClawMetry: " + "pip install clawmetry[otel]"), +) + +# ── Authentication ────────────────────────────────────────────────────── + +AUTH_MODES = ( + { + "name": "Loopback", + "when": "The agent and ClawMetry are on the same machine.", + "how": "Nothing to configure. This is the zero-config path and it is " + "what most installs use.", + }, + { + "name": "Gateway token", + "when": "An exporter elsewhere on a trusted LAN.", + "how": "Authorization: Bearer $OPENCLAW_GATEWAY_TOKEN", + }, + { + "name": "Ingest key", + "when": "Anything that is not on this machine: CI, a container, a " + "serverless function, a hosted product, a teammate's laptop.", + "how": f"clawmetry key create --name ci --scope write:ingest, then " + f"{HEADER_KEY}: cmk_...", + }, +) + +# ── GenAI attributes ──────────────────────────────────────────────────── +# Saying what we do NOT read is what makes the rest of this credible, and +# it is checked against the source: tests/test_ingest_contract_drift.py +# asserts every name below appears (or does not appear) in the mapper. + +GENAI_READ = ( + ("gen_ai.operation.name", "Marks the span as a model call."), + ("gen_ai.request.model", "The model asked for."), + ("gen_ai.response.model", "The model actually served."), + ("gen_ai.provider.name", "Provider; gen_ai.system is read as the older name."), + ("gen_ai.usage.input_tokens", "Input tokens."), + ("gen_ai.usage.output_tokens", "Output tokens."), + ("gen_ai.usage.cache_read.input_tokens", + "Prompt-cache reads, current convention (dot before the noun)."), + ("gen_ai.usage.cache_read_input_tokens", + "Prompt-cache reads, earlier spelling. Both are read; a span carrying " + "both is counted once."), + ("gen_ai.usage.cache_creation.input_tokens", + "Prompt-cache writes, current convention."), + ("gen_ai.usage.cache_creation_input_tokens", + "Prompt-cache writes, earlier spelling."), + ("gen_ai.usage.cost_usd", "Cost, when the exporter states one. An " + "explicit cost always wins over a derived one."), + ("gen_ai.tool.name", "Tool name on execute_tool spans."), + ("gen_ai.conversation.id", "Session id."), + ("gen_ai.agent.id", "Agent id."), + ("gen_ai.input.messages", "Prompt content, when the exporter sends it."), + ("gen_ai.output.messages", "Response content, when the exporter sends it."), +) + +GENAI_NOT_READ = ( + ("gen_ai.usage.reasoning.output_tokens", + "Reasoning tokens. There is no column to put them in yet, so they are " + "dropped rather than mis-filed into output tokens."), + ("gen_ai.response.finish_reasons", + "Not yet read. Would let us spot truncation and unusual stops."), + ("gen_ai.response.time_to_first_chunk", + "Not yet read. Streaming latency per span."), +) + +__all__ = [ + "CONTRACT_VERSION", "HEADER_KEY", "HEADER_RUNTIME", "HEADER_ENV", + "HEADER_LEGACY_TOKEN", "MAX_BODY_BYTES", "MAX_EVENTS_PER_BATCH", + "HEADERS", "SURFACES", "CONTENT_TYPES", "CONTENT_ENCODINGS", + "RESPONSES", "AUTH_MODES", "GENAI_READ", "GENAI_NOT_READ", ] - -# ── Run / event ingest API ─────────────────────────────────────────────────── - -# Endpoints served by ``routes/runtime_ingest.py`` (bp_runtime_ingest). -# The stub OSS blueprint returns HTTP 402 on all Pro-gated write paths. -RUN_ENDPOINTS: dict[str, dict] = { - "GET /api/v1/runtimes": { - "tier": "free", - "doc": "List runtimes ClawMetry knows about. Same data the runtime switcher reads.", - }, - "POST /api/v1/runs": { - "tier": "pro", - "doc": "Open a run. Returns ``{ok, run_id, runtime}``.", - }, - "POST /api/v1/runs//events": { - "tier": "pro", - "doc": "Append one or many events to an open run.", - }, - "POST /api/v1/runs//end": { - "tier": "pro", - "doc": "Mark the run ended. Optional — the run also closes on inactivity.", - }, - "GET /api/v1/runs/": { - "tier": "pro", - "doc": "Read-back: confirm the run was persisted and return its metadata.", - }, -} - -# HTTP response codes for the run/event endpoints. -RUN_RESPONSE_CODES: dict[int, str] = { - 200: "Success (GET requests).", - 201: "Accepted (POST requests).", - 400: "Malformed request body.", - 401: "Token required but missing or incorrect.", - 402: "Pro plan required; OSS stub returns this on all write endpoints.", - 429: "Rate limit or budget pause.", -} diff --git a/clawmetry/setup_prompt.py b/clawmetry/setup_prompt.py new file mode 100644 index 0000000000..c092516841 --- /dev/null +++ b/clawmetry/setup_prompt.py @@ -0,0 +1,182 @@ +"""clawmetry/setup_prompt.py -- the copy-paste prompt you hand your agent. + +Why this exists +--------------- +ClawMetry's users delegate work to coding agents by definition -- that is +what the product observes. Yet the only setup paths shipped were "run the +installer" and "read a doc", neither of which is aimed at the thing the +user actually drives. + +So: a prompt per runtime, written for the agent, that gets telemetry +flowing in one attempt. + +Generated, not written +---------------------- +Every endpoint, header, content type and cap here comes from +``clawmetry.ingest_contract``, the same declaration the server validates +against. A hand-written prompt drifts the first time a header is renamed, +and a drifted setup prompt is worse than none: the agent writes the wrong +header confidently and the request fails somewhere the user cannot see. + +Half of it is negative space +---------------------------- +The instructive part of a good setup prompt is what it forbids. Coding +agents reliably: mis-substitute secrets (dropping characters, wrapping +them in quotes, or leaving the literal placeholder in place); "correct" a +content type that was already right; and invent config keys, env-var +names and endpoint paths that look plausible and do not exist. Each of +those gets a line, because an unstated constraint is one the agent will +violate helpfully. + +The seam +-------- +This module names no runtime and hardcodes no vendor value. Where a +runtime has registered an OTel profile (``clawmetry.otel_profiles``, which +paid runtimes populate from clawmetry-pro), its label and its +``clawmetry instrument`` support are read from that profile at render +time. A free install renders the generic OTLP prompt, which works. +""" + +from __future__ import annotations + +import re +from typing import Optional + +from clawmetry.ingest_contract import ( + CONTRACT_VERSION, + HEADER_ENV, + HEADER_KEY, + HEADER_RUNTIME, + MAX_BODY_BYTES, +) + +#: Shown where a real key is not available to substitute. Deliberately +#: ugly and obviously not a key, so an agent leaving it in place produces +#: a 401 with a clear message rather than something that looks plausible. +KEY_PLACEHOLDER = "PASTE_YOUR_CLAWMETRY_KEY_HERE" + +DEFAULT_LOCAL_ENDPOINT = "http://localhost:8900" + +#: Same shape the ingest gate accepts, so a runtime this renders a prompt +#: for is a runtime the server will take the header for. +VALID_RUNTIME = re.compile(r"^[a-z0-9][a-z0-9_-]{0,39}$") + + +def _profile(runtime: str): + """The registered OTel profile for ``runtime``, or None. Never raises.""" + try: + from clawmetry import otel_profiles + + return otel_profiles.by_runtime(runtime) + except Exception: + return None + + +def render( + runtime: str = "", + *, + endpoint: str = "", + key: Optional[str] = None, + include_verification: bool = True, +) -> str: + """The prompt text for ``runtime``. + + ``key`` is substituted when given; otherwise the placeholder is used + and the prompt tells the agent to ask for the real one rather than + invent it. + """ + rt = (runtime or "").strip().lower() + base = (endpoint or DEFAULT_LOCAL_ENDPOINT).rstrip("/") + secret = key or KEY_PLACEHOLDER + prof = _profile(rt) if rt else None + label = (getattr(prof, "label", "") or rt or "this agent") + mb = MAX_BODY_BYTES // (1024 * 1024) + + out: list = [] + a = out.append + + a(f"Set up ClawMetry observability for {label}.") + a("") + a("ClawMetry watches AI agents: what they did, what it cost, and") + a("whether they got stuck. It already detects agents running on the") + a("same machine as ClawMetry itself, with no configuration. This task") + a("is the other case -- pointing an agent that runs somewhere else at") + a("it.") + a("") + a(f"Contract version: {CONTRACT_VERSION}") + a("") + + a("## What to do") + a("") + a("Configure this agent's OpenTelemetry exporter to send OTLP over") + a("HTTP to ClawMetry:") + a("") + a(f" Endpoint: {base}/v1/traces (also /v1/logs and /v1/metrics)") + a(f" Header: {HEADER_KEY}: {secret}") + if rt: + a(f" Header: {HEADER_RUNTIME}: {rt}") + a(f" Header: {HEADER_ENV}: (optional)") + a("") + a("If the agent or its framework already exports OpenTelemetry, change") + a("its exporter endpoint and headers rather than adding a second") + a("exporter. If it does not, use the OpenTelemetry SDK for its") + a("language.") + a("") + + if prof is not None and getattr(prof, "instrumenter", None) is not None: + a("This runtime has a native exporter ClawMetry can configure for") + a("you. Try this first, and only fall back to manual setup if it") + a("reports that it cannot:") + a("") + a(f" clawmetry instrument {rt}") + a("") + + a("## Rules -- these are the ones that get this wrong") + a("") + a(f"1. The key goes in the `{HEADER_KEY}` header. Not in a query") + a(" parameter, not in `Authorization: Bearer`, not in the body.") + if key: + a("2. The key above is real. Copy it exactly -- do not re-wrap it in") + a(" quotes, do not shorten it, do not regenerate it.") + else: + a(f"2. `{KEY_PLACEHOLDER}` is a placeholder, not a key. Ask the") + a(" person who gave you this prompt for the real one. Do NOT") + a(" invent a key, and do not leave the placeholder in place.") + a("3. Both `application/x-protobuf` and `application/json` are") + a(" accepted, and gzip is accepted on either. If the exporter") + a(" already produces one of those, leave it alone -- there is") + a(" nothing to fix.") + a("4. Use only the header names, endpoints and settings written above.") + a(" If a config key you want is not in this prompt, it does not") + a(" exist. Do not guess one that looks plausible.") + a("5. Do not modify anything about how the agent itself behaves. This") + a(" is observability: it watches, it does not change what runs.") + a(f"6. Keep request bodies under {mb} MB. The exporter's default") + a(" batching handles this; do not raise a batch size to 'improve") + a(" throughput'.") + a("") + + if include_verification: + a("## Then verify, and report back") + a("") + a("Do not report success because the config was written. Confirm") + a("that data actually arrived:") + a("") + a(" 1. Make the agent do one small thing that calls a model.") + a(f" 2. curl -s {base}/api/otel-status") + a(" 3. Tell the user the counts you got back.") + a("") + a("If the counts are zero, the setup did not work. Say so plainly") + a("and report the exact error, rather than describing the") + a("configuration you wrote.") + a("") + a("Common causes, in the order worth checking:") + a(" - a 401 means the key is wrong or was not sent;") + a(" - a 403 means the key is real but cannot push -- it needs the") + a(" write:ingest scope;") + a(" - a 400 names what it could not decode;") + a(" - nothing at all usually means the exporter was configured") + a(" but the process was never restarted.") + a("") + + return "\n".join(out) diff --git a/clawmetry/static/js/app.js b/clawmetry/static/js/app.js index e231590327..636a823aa3 100644 --- a/clawmetry/static/js/app.js +++ b/clawmetry/static/js/app.js @@ -11550,9 +11550,18 @@ function _cmRenderApiKeyScopeChoices() { var warn = s.sensitive ? '
Keep this one server-side. Do not ship it in a page.
' : ''; + // A write scope pushes data in and reads nothing back, so it cannot + // be combined with a read scope on one key. Saying that next to the + // box is better than letting someone tick both and meet an error. + if (s.kind === 'write') { + warn = '
' + + 'On its own key. Used by a server, a container or a CI job — never by a web page.
'; + } return '