diff --git a/README.md b/README.md index c150708..20837d2 100644 --- a/README.md +++ b/README.md @@ -83,6 +83,10 @@ $ python -m srdcheck query mage-hand.use '{"kind": "attack"}' "rule_ids": ["mage-hand.cant-attack"], "adapter": "srd-5.2.1@0.2.1" } +$ python -m srdcheck query mage-hand.use '{"kind": "attack"}' --table-evaluation +# deterministic table.evaluation/1.0; status=findings, authority=self_attested +$ python -m srdcheck query save.check '{"modifier":2,"dc":10,"d20_result":12,"save_ability":"dex","saver_conditions":[]}' \ + --table-evaluation --table-context '{"session_id":"session-7","correlation_id":"discord-42"}' $ python -m srdcheck cite "Command" # verbatim SRD text + page for any registered name $ python -m srdcheck --schema # I/O contract for agents $ python -m srdcheck capabilities # engine, protocol, adapters, digests, tools @@ -93,6 +97,18 @@ Deterministic, offline, and no tokens. CI gates 100-verdict p95 latency below surface is young and growing slice by slice — the architecture (kernel + [adapters](docs/adapter-spec.md), spec at v0.9 RC) is the point. +`query` and `--pipe` accept `--table-evaluation` for portfolio hosts that need +the shared `table.evaluation/1.0` envelope. Legal/illegal verdicts become +complete checked-clean/finding results only within the named query scope; +missing facts, unsupported content, ambiguity, and DM discretion remain +non-clean. The subject carries a machine-readable `srdcheck-query-scope:*` +reference; optional `--table-context` JSON can add caller-owned `session_id`, +`entity_refs`, and a `correlation_id` (emitted as `correlation:*`). Rule and +citation evidence is self-contained and keyed by every finding evidence ref. +Every result is `self_attested`, including when the calling agent is the DM: +this adapter cannot grant table, encounter-state, action, or protected-host +authority. + ## For agents (MCP) srdcheck is an MCP server with zero dependencies — stdlib only. After `pip install`, the command is `srdcheck-mcp`; from a clone it's: @@ -117,7 +133,9 @@ boundary refusals do not invent provenance. An `illegal` verdict is a result, not an error; `cannot-adjudicate` is an honest refusal, not a failure. Tool descriptions and schemas come from loaded adapters, so new adapters extend the tool list without kernel changes. See also [`tool.json`](tool.json) for the CLI -surface. +surface. The opt-in `table_evaluation` MCP tool accepts `{query_type, params, +context?}` and emits the same shared envelope without changing native query-tool +responses. Every first-party `cannot-adjudicate` result carries stable recovery metadata under `data`: a reason code, recoverability, exact missing-input paths, and a @@ -167,7 +185,7 @@ Generated by `scripts/truth_scorecard.py` — regenerated and diff-checked in CI | truth | claim | status | evidence | |---|---|---|---| -| T1 | wrong verdicts | enforced in CI | 639 tests including gold suites ported from the Phase 0 eval; any wrong verdict fails the build | +| T1 | wrong verdicts | enforced in CI | 647 tests including gold suites ported from the Phase 0 eval; any wrong verdict fails the build | | T2 | no citation, no rule | enforced in CI | 90/90 rule atoms carry verbatim source quotes; applied-rule paths cite, while boundary refusals do not invent provenance | | T3 | advise, never overrule | structural | the API has no blocking or veto interface to wire; verdicts are advisory by construction | | T4 | one payload, two audiences | enforced in CI | every verdict carries machine fields plus a templated plain-English why; schema-tested | @@ -176,7 +194,7 @@ Generated by `scripts/truth_scorecard.py` — regenerated and diff-checked in CI | T7 | mechanism never knows the game | enforced in CI | kernel lint scans every kernel module for game vocabulary (it caught a real violation during development) | | T8 | honest boundaries | enforced in CI | refusal goldens: unknown content, unmodeled conditions, and genuinely ambiguous rules text all return exit 2; ambiguity cites its source, while registry and input boundaries may have no citation | | T9 | never a single number | enforced in CI | bench scorecard freshness test; per-category tables, no aggregate score exists anywhere in this repository — even the leaderboard ranks by wrong-count alone (T1) and shows the other failure modes unblended | -| T10 | stranger-agent bootstrap | enforced in CI | cold-start conformance test reaches a first verdict from tool.json/--schema/MCP alone (22 tools); live probe: a frontier model given only tool.json produced a correct first verdict in 1 attempt(s), 3.9s (2026-07-16) | +| T10 | stranger-agent bootstrap | enforced in CI | cold-start conformance test reaches a first verdict from tool.json/--schema/MCP alone (23 tools); live probe: a frontier model given only tool.json produced a correct first verdict in 1 attempt(s), 3.9s (2026-07-16) | | T11 | table speed | enforced in CI | p95 latency budget test: 100 verdicts must stay under 100 ms at p95; lower local observations are not a compatibility promise | | T12 | never sell what the model has | held in review | a strategy invariant: features pitched on knowledge parity are cut in review — enforced by humans and admitted as such | | T13 | the benchmark is a product | shipped | bench/ publishes 5 sets across 5 subjects with cited gold verdicts, incl. a per-horizon drift lane whose golds are engine-derived; a generated LEADERBOARD ranks any subject; cmd: driver + validate command let a third party submit a tamper-checked result | diff --git a/docs/capability-map.json b/docs/capability-map.json index 35aa98c..4195722 100644 --- a/docs/capability-map.json +++ b/docs/capability-map.json @@ -679,6 +679,28 @@ "rules": "0.1.0" }, "description": "Enumerate the legal moves for a player on a board (same encoding as ttt.move). Empty when it is not the player's turn or the game has ended." + }, + { + "adapter": "kernel", + "query_type": "table.evaluation", + "tool": "table_evaluation", + "capability": "Project one native rules query into table.evaluation/1.0 with machine-readable scope, caller references, and portable evidence.", + "checked_scope": [ + "deterministic projection of one named query", + "self-attested coverage and exact finding evidence", + "caller-owned session, entity, and correlation references" + ], + "unchecked_scope": [ + "global action or table legality", + "table, encounter-state, or execution authority", + "protected-host attestation" + ], + "evidence": [ + "tests/test_table_evaluation.py", + "tests/test_mcp.py::test_table_evaluation_tool_is_opt_in_scoped_and_joinable", + "scripts/cold_artifact_smoke.py" + ], + "description": "Opt-in shared-envelope transport; native verdict tools remain unchanged." } ], "targets": [ diff --git a/docs/capability-map.md b/docs/capability-map.md index 38309d5..41fb8f5 100644 --- a/docs/capability-map.md +++ b/docs/capability-map.md @@ -60,6 +60,7 @@ A `legal` result below means only that the named checked scope passed. | `feature_uses` | Return the modeled use-count formula for the closed feature set. | Divine Sense and Lay on Hands formulas; arithmetic when all blanks are supplied | other features; tables; whether a use remains in campaign state | `tests/test_spell_facts.py::test_divine_sense_blanks`; `tests/test_spell_facts.py::test_lay_on_hands_blanks` | | `ttt_move` | Check one move against the bundled tutorial ruleset. | board shape, turn order, cell occupancy, and terminal state | strategy; choosing a move | `tests/test_toy_adapter.py::test_move_legality` | | `ttt_options` | Enumerate legal cells under the bundled tutorial ruleset. | legal empty cells for the current player | strategy; ranking options | `tests/test_toy_adapter.py::test_enumerate_validate_consistency` | +| `table_evaluation` | Project one native rules query into table.evaluation/1.0 with machine-readable scope, caller references, and portable evidence. | deterministic projection of one named query; self-attested coverage and exact finding evidence; caller-owned session, entity, and correlation references | global action or table legality; table, encounter-state, or execution authority; protected-host attestation | `tests/test_table_evaluation.py`; `tests/test_mcp.py::test_table_evaluation_tool_is_opt_in_scoped_and_joinable`; `scripts/cold_artifact_smoke.py` | ## Target architecture — not shipped diff --git a/llms.txt b/llms.txt index 39ac668..5e48275 100644 --- a/llms.txt +++ b/llms.txt @@ -17,6 +17,14 @@ currently validate multiclass builds or enumerate every creature-specific move. The calling agent may itself be the authorized DM; its table rulings remain distinct from SRD-derived results. +For a suite result envelope, add `--table-evaluation` to `query` or `--pipe`. +The deterministic `table.evaluation/1.0` projection is always self_attested; +it preserves self-contained rule/citation evidence and never grants table +authority, including when the caller is an agent-DM. Its subject includes a +machine-readable named-query scope. Add `--table-context` JSON for caller-owned +session/entity/correlation references, or call the opt-in MCP tool +`table_evaluation` with `{query_type, params, context?}`. + On `cannot-adjudicate`, branch on the recovery fields in `data`, never on `why`: repair the request, provide the named missing facts, select an adapter, use another capability, resolve a table ruling, or stop. diff --git a/scripts/capability_map.py b/scripts/capability_map.py index 0225aa8..3d40ac5 100644 --- a/scripts/capability_map.py +++ b/scripts/capability_map.py @@ -36,6 +36,32 @@ def build(): } entry["description"] = adapter.query_meta[query_type]["description"] shipped.append(entry) + shipped.append({ + "adapter": "kernel", + "query_type": "table.evaluation", + "tool": "table_evaluation", + "capability": ( + "Project one native rules query into table.evaluation/1.0 with " + "machine-readable scope, caller references, and portable evidence."), + "checked_scope": [ + "deterministic projection of one named query", + "self-attested coverage and exact finding evidence", + "caller-owned session, entity, and correlation references", + ], + "unchecked_scope": [ + "global action or table legality", + "table, encounter-state, or execution authority", + "protected-host attestation", + ], + "evidence": [ + "tests/test_table_evaluation.py", + "tests/test_mcp.py::test_table_evaluation_tool_is_opt_in_scoped_and_joinable", + "scripts/cold_artifact_smoke.py", + ], + "description": ( + "Opt-in shared-envelope transport; native verdict tools remain " + "unchanged."), + }) return { "schema_version": CLAIMS_SCHEMA_VERSION, "generated_from": ( diff --git a/scripts/cold_artifact_smoke.py b/scripts/cold_artifact_smoke.py index 71aecb9..f120bf9 100644 --- a/scripts/cold_artifact_smoke.py +++ b/scripts/cold_artifact_smoke.py @@ -165,6 +165,16 @@ def smoke(artifact): )) assert query["exit_code"] == 0 and query["rule_ids"] + shared = json.loads(run( + [str(python), "-m", "srdcheck", "query", "mage-hand.use", + '{"kind":"attack"}', "--table-evaluation"], + outside, env=clean_env, allowed_returncodes=(1,), + )) + assert shared["schema_version"] == "table.evaluation/1.0" + assert shared["status"] == "findings" + assert shared["authority_status"] == "self_attested" + assert "mage-hand.cant-attack" in shared["findings"][0]["evidence_refs"] + capabilities = json.loads(run( [str(python), "-m", "srdcheck", "capabilities"], outside, env=clean_env, @@ -201,6 +211,11 @@ def smoke(artifact): {"jsonrpc": "2.0", "id": 4, "method": "tools/call", "params": {"name": "jurisdiction", "arguments": { "name": unicode_name}}}, + {"jsonrpc": "2.0", "id": 5, "method": "tools/call", + "params": {"name": "table_evaluation", "arguments": { + "query_type": "mage-hand.use", "params": {"kind": "attack"}, + "context": {"session_id": "artifact-session", + "correlation_id": "artifact-call"}}}}, ]) replies = [json.loads(line) for line in run( [str(python), "-m", "srdcheck.mcp"], outside, @@ -212,6 +227,11 @@ def smoke(artifact): assert by_id[3]["result"]["structuredContent"]["exit_code"] == 0 assert by_id[4]["result"]["structuredContent"]["exit_code"] == 2 assert unicode_name in by_id[4]["result"]["structuredContent"]["why"] + projected = by_id[5]["result"]["structuredContent"] + assert projected["status"] == "findings" + assert projected["authority_status"] == "self_attested" + assert projected["subject"]["session_id"] == "artifact-session" + assert "correlation:artifact-call" in projected["subject"]["entity_refs"] # --target places project.scripts launchers beneath the target itself. # Invoke them directly so Windows .exe wrappers and both entry-point diff --git a/scripts/gen_tool_json.py b/scripts/gen_tool_json.py index 3b69add..11229b3 100644 --- a/scripts/gen_tool_json.py +++ b/scripts/gen_tool_json.py @@ -42,6 +42,11 @@ def build(): "query ": ( "Adjudicate a structured proposal. Query types are supplied by " "the loaded adapters; see query_types below."), + "query --table-evaluation": ( + "Project the scoped verdict into deterministic, self-attested " + "table.evaluation/1.0. Add --table-context JSON for caller-owned " + "session/entity/correlation references. Table and execution " + "authority stay external."), "--pipe": "Read one {\"type\", \"params\"} JSON query from stdin.", "--schema": "Print JSON Schema for input and output.", "capabilities": ("Print engine, protocol, adapter versions/digests, " @@ -73,6 +78,12 @@ def build(): "contains stable reason and recovery fields. Schema and " "refusal contract identities are published by --schema, " "MCP outputSchema, and capabilities."), + "table_evaluation": ( + "query/--pipe --table-evaluation maps legal to checked_clean, " + "illegal to an exact-evidence finding, and structured refusals to " + "invalid, unsupported, or incomplete. Output is always " + "self_attested. Its named-query scope is machine-readable and " + "every finding evidence ref resolves in its effective policy."), "notes": ( "The 'why' field is templated explanatory prose, not a machine " "compatibility field. Verdicts are deterministic: same query, same " @@ -91,7 +102,9 @@ def build(): "Zero-dependency stdlib MCP server. Tool descriptions/schemas " "are supplied by loaded adapters (queries.json). Verdicts " "arrive as structuredContent; illegal and cannot-adjudicate " - "are results, not protocol errors."), + "are results, not protocol errors. The opt-in table_evaluation " + "tool projects one native query without changing native tool " + "response schemas."), }, "_generated_by": "scripts/gen_tool_json.py", } diff --git a/srdcheck/__init__.py b/srdcheck/__init__.py index 92c80d4..2f0ccce 100644 --- a/srdcheck/__init__.py +++ b/srdcheck/__init__.py @@ -42,3 +42,5 @@ def _detect_version(): CANNOT_ADJUDICATE, ILLEGAL, LEGAL, Citation, Verdict, cannot_adjudicate, illegal, legal, ) +from .table_evaluation import (TABLE_EVALUATION_SCHEMA_VERSION, + project_table_evaluation) # noqa: F401 diff --git a/srdcheck/access.py b/srdcheck/access.py index 08b53f0..d2dd0ce 100644 --- a/srdcheck/access.py +++ b/srdcheck/access.py @@ -73,7 +73,7 @@ def capabilities(): claims = json.loads(CLAIMS_PATH.read_text(encoding="utf-8")) refusal_contract = v.refusal_contract() adapters = [] - tool_names = {"jurisdiction"} + tool_names = {"jurisdiction", "table_evaluation"} for identifier in available_adapters(): root = ADAPTERS_DIR / identifier manifest = json.loads((root / "manifest.json").read_text()) diff --git a/srdcheck/cli.py b/srdcheck/cli.py index b1ac78c..88c334a 100644 --- a/srdcheck/cli.py +++ b/srdcheck/cli.py @@ -13,6 +13,7 @@ (3 = usage or internal error, never a verdict.) """ +import hashlib import json import pathlib import sys @@ -39,6 +40,11 @@ "additionalProperties": False, }, "output": VERDICT_OUTPUT_SCHEMA, + "table_evaluation": ( + "query/--pipe --table-evaluation emits deterministic self-attested " + "table.evaluation/1.0; optional --table-context JSON supplies " + "caller-owned session/entity/correlation references; table and " + "execution authority remain external"), "exit_codes": {"0": "legal", "1": "illegal", "2": "cannot-adjudicate", "3": "usage or internal error (not a verdict)"}, } @@ -49,13 +55,60 @@ def _engine(): return Engine(default_adapter_paths()) -def _emit(verdict): - print(json.dumps(verdict.as_dict(), indent=2)) +def _emit(verdict, table_evaluation=False, query_type=None, params=None, + table_context=None): + value = verdict.as_dict() + if table_evaluation: + from .table_evaluation import project_table_evaluation + value = project_table_evaluation( + value, query_type, params, context=table_context) + print(json.dumps(value, indent=2)) + return value["exit_code"] + print(json.dumps(value, indent=2)) return verdict.exit_code +def _invalid_json_evaluation(query_type, raw, message, table_context=None): + from .table_evaluation import project_table_evaluation + from .verdict import cannot_adjudicate + + digest = hashlib.sha256(raw.encode("utf-8", errors="replace")).hexdigest() + verdict = cannot_adjudicate( + message, reason_code="invalid-input", missing_inputs=["/params"]) + return project_table_evaluation( + verdict, query_type or "invalid-query", {"invalid_json_sha256": digest}, + context=table_context) + + def main(argv=None): args = list(sys.argv[1:] if argv is None else argv) + table_evaluation = "--table-evaluation" in args + if args.count("--table-evaluation") > 1: + print(json.dumps({"error": "--table-evaluation may be supplied once"})) + return 3 + if table_evaluation: + args.remove("--table-evaluation") + table_context = None + if args.count("--table-context") > 1: + print(json.dumps({"error": "--table-context may be supplied once"})) + return 3 + if "--table-context" in args: + index = args.index("--table-context") + if index + 1 >= len(args): + print(json.dumps({"error": "--table-context requires JSON"})) + return 3 + raw_context = args[index + 1] + del args[index:index + 2] + if not table_evaluation: + print(json.dumps({ + "error": "--table-context requires --table-evaluation" + })) + return 3 + try: + table_context = json.loads(raw_context) + except json.JSONDecodeError: + print(json.dumps({"error": "--table-context must be valid JSON"})) + return 3 if not args: print(__doc__) return 3 @@ -66,13 +119,35 @@ def main(argv=None): from .access import capabilities print(json.dumps(capabilities(), indent=2)) return 0 + if table_evaluation and args[0] not in {"query", "--pipe"}: + print(json.dumps({ + "error": "--table-evaluation is valid only with query or --pipe" + })) + return 3 try: if args[0] == "--pipe": - q = json.loads(sys.stdin.read()) + raw = sys.stdin.read() + try: + q = json.loads(raw) + except json.JSONDecodeError as exc: + if table_evaluation: + value = _invalid_json_evaluation( + "invalid-query", raw, "query JSON is invalid", + table_context) + print(json.dumps(value, indent=2)) + return value["exit_code"] + raise exc problems = schema_issues(q, SCHEMA["input"]) if problems: - return _emit(validation_refusal(problems)) - return _emit(_engine().query(q["type"], q.get("params", {}))) + q_type = (q.get("type", "invalid-query") + if isinstance(q, dict) else "invalid-query") + q_params = (q.get("params", {}) + if isinstance(q, dict) else {}) + return _emit(validation_refusal(problems), table_evaluation, + q_type, q_params, table_context) + return _emit(_engine().query(q["type"], q.get("params", {})), + table_evaluation, q["type"], q.get("params", {}), + table_context) if args[0] == "conformance" and len(args) == 2: from .conformance import check as _cf probs = _cf(args[1]) @@ -88,7 +163,18 @@ def main(argv=None): if args[0] == "jurisdiction" and len(args) == 2: return _emit(_engine().jurisdiction(args[1])) if args[0] == "query" and len(args) == 3: - return _emit(_engine().query(args[1], json.loads(args[2]))) + try: + params = json.loads(args[2]) + except json.JSONDecodeError as exc: + if table_evaluation: + value = _invalid_json_evaluation( + args[1], args[2], "params JSON is invalid", + table_context) + print(json.dumps(value, indent=2)) + return value["exit_code"] + raise exc + return _emit(_engine().query(args[1], params), table_evaluation, + args[1], params, table_context) if args[0] == "edition-check" and len(args) >= 2: from .access import edition_check name = args[1] diff --git a/srdcheck/mcp.py b/srdcheck/mcp.py index 72a8035..3256b21 100644 --- a/srdcheck/mcp.py +++ b/srdcheck/mcp.py @@ -15,6 +15,10 @@ from . import __version__ from .engine import Engine, NON_EMPTY_NAME_INPUT_SCHEMA from .schema import ValidationError, validate +from .table_evaluation import ( + CALLER_CONTEXT_SCHEMA, TABLE_EVALUATION_OUTPUT_SCHEMA, + project_table_evaluation, +) from .verdict import VERDICT_OUTPUT_SCHEMA PROTOCOL_VERSION = "2025-06-18" @@ -45,6 +49,26 @@ def negotiate_protocol(requested): "outputSchema": VERDICT_OUTPUT_SCHEMA, } +TABLE_EVALUATION_TOOL = { + "name": "table_evaluation", + "description": ( + "Adjudicate one named SRDCheck query and project its exact scoped " + "result into deterministic table.evaluation/1.0. Always self-attested, " + "including when the calling agent is the DM; table authority remains " + "with the caller or protected host."), + "inputSchema": { + "type": "object", + "additionalProperties": False, + "required": ["query_type", "params"], + "properties": { + "query_type": {"type": "string", "minLength": 1}, + "params": {"type": "object"}, + "context": CALLER_CONTEXT_SCHEMA, + }, + }, + "outputSchema": TABLE_EVALUATION_OUTPUT_SCHEMA, +} + def _published_input_schema(adapter, meta): """Expand adapter-local refs into self-contained MCP tool schemas. @@ -73,8 +97,9 @@ def _published_input_schema(adapter, meta): def build_tools(engine): - tools = [JURISDICTION_TOOL] - mapping = {"jurisdiction": "jurisdiction"} + tools = [JURISDICTION_TOOL, TABLE_EVALUATION_TOOL] + mapping = {"jurisdiction": "jurisdiction", + "table_evaluation": "table_evaluation"} for a in engine.adapters: for qt, meta in sorted(a.query_meta.items()): name = qt.replace(".", "_").replace("-", "_") @@ -188,12 +213,31 @@ def _call(self, mid, params): if not isinstance(args, dict): return self._error(mid, -32602, "invalid params: arguments must be an object") qt = self.mapping[name] + if qt == "table_evaluation": + try: + normalized = validate( + args, TABLE_EVALUATION_TOOL["inputSchema"]) + except ValidationError: + return self._result(mid, { + "content": [{"type": "text", + "text": "invalid table_evaluation arguments"}], + "isError": True, + }) try: - if qt == "jurisdiction": + if qt == "table_evaluation": + query_type = normalized["query_type"] + query_params = normalized["params"] + native = self.engine.query(query_type, query_params).as_dict() + validate(native, VERDICT_OUTPUT_SCHEMA) + vd = project_table_evaluation( + native, query_type, query_params, normalized.get("context")) + validate(vd, TABLE_EVALUATION_OUTPUT_SCHEMA) + elif qt == "jurisdiction": vd = self.engine.query("jurisdiction", args).as_dict() + validate(vd, VERDICT_OUTPUT_SCHEMA) else: vd = self.engine.query(qt, args).as_dict() - validate(vd, VERDICT_OUTPUT_SCHEMA) + validate(vd, VERDICT_OUTPUT_SCHEMA) except ValidationError: return self._result(mid, { "content": [{"type": "text", "text": "internal output validation failed"}], diff --git a/srdcheck/table_evaluation.py b/srdcheck/table_evaluation.py new file mode 100644 index 0000000..d037753 --- /dev/null +++ b/srdcheck/table_evaluation.py @@ -0,0 +1,276 @@ +"""Project one srdcheck verdict into ``table.evaluation/1.0``. + +The shared envelope remains self-attested and advisory. It does not turn a +rules verdict into table, encounter-state, or action-execution authority. +""" + +import hashlib +import json + +from . import __version__ +from .access import capabilities + + +TABLE_EVALUATION_SCHEMA_VERSION = "table.evaluation/1.0" + +CALLER_CONTEXT_SCHEMA = { + "type": "object", + "additionalProperties": False, + "properties": { + "session_id": {"type": "string", "minLength": 1, "maxLength": 256}, + "entity_refs": { + "type": "array", + "items": {"type": "string", "minLength": 1, "maxLength": 256}, + }, + "correlation_id": { + "type": "string", "minLength": 1, "maxLength": 244, + "description": ("Caller-owned correlation identity; projected as " + "subject.entity_refs entry correlation:."), + }, + }, +} + +# Self-contained MCP discovery schema. table-kit remains the canonical owner; +# this transport copy deliberately describes the full envelope shape without +# claiming host attestation or importing a runtime dependency. +TABLE_EVALUATION_OUTPUT_SCHEMA = { + "type": "object", + "additionalProperties": False, + "required": [ + "schema_version", "evaluation_id", "tool", "subject", "status", + "exit_code", "authority_status", "coverage", "cursor", "context", + "findings", "advisories", "warnings", "errors", + ], + "properties": { + "schema_version": {"type": "string", "enum": [TABLE_EVALUATION_SCHEMA_VERSION]}, + "evaluation_id": {"type": "string", "minLength": 1, "maxLength": 256}, + "tool": {"type": "object"}, + "subject": {"type": "object"}, + "status": {"type": "string", "enum": [ + "checked_clean", "checked_with_advisories", "findings", + "incomplete", "unsupported", "invalid", "internal_error", + ]}, + "exit_code": {"type": "integer", "enum": [0, 1, 2]}, + "authority_status": {"type": "string", "enum": ["self_attested"]}, + "coverage": {"type": "object"}, + "cursor": {"type": "object"}, + "context": {"type": "object"}, + "findings": {"type": "array"}, + "advisories": {"type": "array"}, + "warnings": {"type": "array"}, + "errors": {"type": "array"}, + }, +} + + +def _canonical_digest(value): + encoded = json.dumps(value, ensure_ascii=False, sort_keys=True, + separators=(",", ":"), allow_nan=False).encode("utf-8") + return "sha256:" + hashlib.sha256(encoded).hexdigest() + + +def _request(query_type, params): + return {"type": query_type, "params": params} + + +def _adapter_policy(adapter_id): + for adapter in capabilities()["adapters"]: + identity = "%s@%s" % (adapter["name"], adapter["version"]) + if identity == adapter_id: + digest = "sha256:" + adapter["digest"] + return identity, digest + identity = adapter_id or "srdcheck-adapter/unknown" + return identity, _canonical_digest({"adapter": identity}) + + +def _evidence(verdict): + refs = [] + catalog = {} + for rule_id in verdict.get("rule_ids") or []: + value = str(rule_id) + if value and len(value) <= 256 and value not in refs: + refs.append(value) + catalog[value] = {"kind": "rule_id", "rule_id": value} + for citation in verdict.get("citations") or []: + if isinstance(citation, dict): + value = "citation-" + _canonical_digest(citation).split(":", 1)[1][:40] + if value not in refs: + refs.append(value) + catalog[value] = {"kind": "citation", "citation": citation} + return refs, catalog + + +def _caller_context(context): + if context is None: + context = {} + if not isinstance(context, dict): + raise ValueError("table evaluation context must be an object") + unknown = sorted(set(context) - set(CALLER_CONTEXT_SCHEMA["properties"])) + if unknown: + raise ValueError("unknown table evaluation context field(s): %s" % + ", ".join(unknown)) + session_id = context.get("session_id") + if session_id is not None and (not isinstance(session_id, str) + or not 1 <= len(session_id) <= 256): + raise ValueError("context session_id must be a 1..256 character string") + supplied_refs = context.get("entity_refs", []) + if (not isinstance(supplied_refs, list) + or any(not isinstance(item, str) or not 1 <= len(item) <= 256 + for item in supplied_refs)): + raise ValueError("context entity_refs must contain 1..256 character strings") + correlation_id = context.get("correlation_id") + if correlation_id is not None and ( + not isinstance(correlation_id, str) + or not 1 <= len(correlation_id) <= 244): + raise ValueError("context correlation_id must be a 1..244 character string") + refs = [] + for value in supplied_refs + (["correlation:" + correlation_id] + if correlation_id is not None else []): + if value not in refs: + refs.append(value) + return session_id, refs + + +def _coverage(status, evaluator_id, errors): + complete = status in {"checked_clean", "checked_with_advisories", "findings"} + if complete: + compatible = eligible = evaluated = 1 + evaluator_status, skipped, reasons = "evaluated", 0, [] + elif status in {"unsupported", "invalid"}: + compatible = eligible = evaluated = skipped = 0 + evaluator_status, reasons = "not_applicable", [] + else: + compatible = eligible = 1 + evaluated, skipped = 0, 1 + evaluator_status, reasons = "skipped", list(errors) + return { + "complete": complete, "evidence_required": True, + "input": 1, "compatible": compatible, "eligible": eligible, + "evaluated": evaluated, "skipped": skipped, + "evaluators": [{ + "id": evaluator_id, "required": True, + "status": evaluator_status, "eligible": eligible, + "evaluated": evaluated, "skipped": skipped, + "skip_reasons": reasons, + }], + } + + +def _diagnostic(verdict, refs): + data = verdict.get("data") if isinstance(verdict.get("data"), dict) else {} + reason = data.get("reason_code") or "cannot-adjudicate" + result = { + "code": ("srdcheck." + str(reason).replace("-", "_"))[:256], + "message": str(verdict.get("why") or "srdcheck could not adjudicate"), + } + missing = data.get("missing_inputs") + if isinstance(missing, list) and missing and isinstance(missing[0], str): + result["pointer"] = missing[0] + if refs: + result["evidence_refs"] = refs + return result + + +def project_table_evaluation(verdict, query_type, params, context=None): + """Return a deterministic, self-attested shared evaluation envelope.""" + if not isinstance(verdict, dict): + verdict = verdict.as_dict() + evaluator_id = (query_type if isinstance(query_type, str) and query_type + else "invalid-query")[:256] + request = _request(query_type, params) + input_digest = _canonical_digest(request) + scope_name = query_type if isinstance(query_type, str) and query_type else "invalid-query" + scope_ref = "srdcheck-query-scope:" + scope_name + if len(scope_ref) > 256: + scope_ref = "srdcheck-query-scope:" + _canonical_digest(scope_name).split(":", 1)[1] + subject_id = "rules-query:" + scope_name + ":" + input_digest.split(":", 1)[1][:40] + if len(subject_id) > 256: + subject_id = "rules-query:" + input_digest.split(":", 1)[1] + session_id, entity_refs = _caller_context(context) + entity_refs = [scope_ref] + [ref for ref in entity_refs if ref != scope_ref] + policy_version, policy_digest = _adapter_policy(verdict.get("adapter")) + refs, evidence = _evidence(verdict) + native_exit = verdict.get("exit_code") + native_name = verdict.get("verdict") + errors = [] + findings = [] + + if native_exit == 0 and native_name == "legal": + status, exit_code, complete = "checked_clean", 0, True + elif native_exit == 1 and native_name == "illegal": + if refs: + status, exit_code, complete = "findings", 1, True + material = {"request": request, "verdict": verdict} + findings.append({ + "finding_id": "srdcheck-" + _canonical_digest( + material).split(":", 1)[1][:40], + "code": "srdcheck.illegal", + "severity": "finding", + "summary": str(verdict.get("why") or + "proposal conflicts with the checked rule scope"), + "evidence_refs": refs, + "effective_policy": { + "adapter": verdict.get("adapter"), + "query_type": query_type, + "rule_ids": list(verdict.get("rule_ids") or []), + "citations": list(verdict.get("citations") or []), + "evidence": evidence, + "scope": {"kind": "rules_query", "query_type": scope_name, + "input_digest": input_digest}, + "authority_boundary": "advisory rules verdict", + }, + "policy_version": policy_version, + "policy_digest": policy_digest, + }) + else: + status, exit_code, complete = "internal_error", 2, False + errors.append({ + "code": "table_evaluation.finding_evidence_missing", + "message": "an illegal verdict lacked rule or citation evidence", + }) + elif native_exit == 2 and native_name == "cannot-adjudicate": + reason = (verdict.get("data") or {}).get("reason_code") + if reason == "invalid-input": + status = "invalid" + elif reason in {"unsupported-content", "unmodeled-rule"}: + status = "unsupported" + else: + status = "incomplete" + exit_code, complete = 2, False + errors.append(_diagnostic(verdict, refs)) + else: + status, exit_code, complete = "internal_error", 2, False + errors.append({ + "code": "table_evaluation.native_verdict_invalid", + "message": "srdcheck returned an incoherent native verdict", + }) + + evaluation_material = { + "tool": "srdcheck", "version": __version__, "request": request, + "native_verdict": verdict, "session_id": session_id, + "entity_refs": entity_refs, + } + return { + "schema_version": TABLE_EVALUATION_SCHEMA_VERSION, + "evaluation_id": "srdcheck-" + _canonical_digest( + evaluation_material).split(":", 1)[1][:40], + "tool": {"name": "srdcheck", "version": __version__}, + "subject": {"kind": "rules_query", "id": subject_id, + "session_id": session_id, "entity_refs": entity_refs}, + "status": status, + "exit_code": exit_code, + "authority_status": "self_attested", + "coverage": _coverage(status, evaluator_id, errors), + "cursor": {"checked_through_event_id": None, + "gap_state": "none" if complete else "unknown", + "input_digest": input_digest}, + "context": {"roster_digest": None, + "policy_digest": policy_digest, + "config_digest": None, + "source_set_digest": policy_digest, + "session_descriptor_digest": None}, + "findings": findings, + "advisories": [], + "warnings": [], + "errors": errors, + } diff --git a/tests/test_mcp.py b/tests/test_mcp.py index 26466e6..46f72dc 100644 --- a/tests/test_mcp.py +++ b/tests/test_mcp.py @@ -45,7 +45,8 @@ def test_handshake_and_tool_list(): "method": "notifications/initialized"}) is None tools = s.handle(rpc("tools/list"))["result"]["tools"] names = {t["name"] for t in tools} - assert names == {"jurisdiction", "turn_plan", "turn_options", + assert names == {"jurisdiction", "table_evaluation", + "turn_plan", "turn_options", "reaction_available", "roll_compose", "attack_modifiers", "mage_hand_use", "spell_facts", "feature_uses", "event_apply", "creature_valid", "creature_stats", @@ -54,8 +55,14 @@ def test_handshake_and_tool_list(): "ttt_move", "ttt_options"} for t in tools: assert t["description"] and t["inputSchema"]["type"] == "object" - assert t["outputSchema"]["required"] == [ - "verdict", "exit_code", "why", "citations", "rule_ids", "adapter"] + if t["name"] == "table_evaluation": + assert t["outputSchema"]["properties"]["schema_version"]["enum"] == [ + "table.evaluation/1.0"] + assert t["outputSchema"]["properties"]["authority_status"]["enum"] == [ + "self_attested"] + else: + assert t["outputSchema"]["required"] == [ + "verdict", "exit_code", "why", "citations", "rule_ids", "adapter"] def test_tool_calls_return_verdicts(): @@ -84,6 +91,31 @@ def test_tool_calls_return_verdicts(): r = s.handle(rpc("tools/call", {"name": "nope", "arguments": {}})) assert r["result"]["isError"] + +def test_table_evaluation_tool_is_opt_in_scoped_and_joinable(): + s = ready_server() + response = s.handle(rpc("tools/call", { + "name": "table_evaluation", + "arguments": { + "query_type": "mage-hand.use", + "params": {"kind": "attack"}, + "context": {"session_id": "session-mcp", + "entity_refs": ["actor:wizard"], + "correlation_id": "discord-99"}, + }, + })) + result = response["result"]["structuredContent"] + assert response["result"]["isError"] is False + assert result["status"] == "findings" + assert result["authority_status"] == "self_attested" + assert result["subject"]["session_id"] == "session-mcp" + assert "srdcheck-query-scope:mage-hand.use" in \ + result["subject"]["entity_refs"] + assert "correlation:discord-99" in result["subject"]["entity_refs"] + finding = result["findings"][0] + assert set(finding["evidence_refs"]) == set( + finding["effective_policy"]["evidence"]) + r = s.handle(rpc("no/such/method")) assert r["error"]["code"] == -32601 @@ -194,7 +226,7 @@ def test_stdio_subprocess_end_to_end(): assert len(replies) == 3 by_id = {r["id"]: r for r in replies} assert by_id[1]["result"]["serverInfo"]["name"] == "srdcheck" - assert len(by_id[2]["result"]["tools"]) == 22 + assert len(by_id[2]["result"]["tools"]) == 23 sc = by_id[3]["result"]["structuredContent"] assert sc["data"]["roll"] == "straight" # the infiltration composition assert proc.returncode == 0 diff --git a/tests/test_table_evaluation.py b/tests/test_table_evaluation.py new file mode 100644 index 0000000..4beb284 --- /dev/null +++ b/tests/test_table_evaluation.py @@ -0,0 +1,146 @@ +import json + +from srdcheck import load_adapter, project_table_evaluation +from srdcheck.cli import main +from srdcheck.verdict import illegal + + +ADAPTER = load_adapter("srd-5.2.1") + + +def save_params(d20_result): + return {"modifier": 2, "dc": 10, "d20_result": d20_result, + "save_ability": "dex", "saver_conditions": []} + + +def test_legal_and_illegal_project_complete_scoped_results(): + legal_params = save_params(12) + legal = project_table_evaluation( + ADAPTER.query("save.check", legal_params), "save.check", legal_params) + assert legal["status"] == "checked_clean" + assert legal["exit_code"] == 0 + assert legal["coverage"]["complete"] is True + assert legal["authority_status"] == "self_attested" + assert "srdcheck-query-scope:save.check" in \ + legal["subject"]["entity_refs"] + + illegal_params = {"kind": "attack"} + result = project_table_evaluation( + ADAPTER.query("mage-hand.use", illegal_params), + "mage-hand.use", illegal_params) + assert result["status"] == "findings" + assert result["exit_code"] == 1 + assert "mage-hand.cant-attack" in result["findings"][0]["evidence_refs"] + assert result["findings"][0]["effective_policy"]["adapter"].startswith( + "srd-5.2.1@") + evidence = result["findings"][0]["effective_policy"]["evidence"] + assert set(evidence) == set(result["findings"][0]["evidence_refs"]) + assert any(item["kind"] == "citation" and item["citation"]["quote"] + for item in evidence.values()) + + +def test_refusal_reasons_map_without_false_clean(): + cases = [ + ("save.check", {"modifier": 2}, "incomplete", "srdcheck.missing_fact", + (1, 1, 0, 1)), + ("spell.facts", {"name": "Hexblade"}, "unsupported", + "srdcheck.unsupported_content", (0, 0, 0, 0)), + ("save.check", {"modifier": 2, "dc": 10, "bogus": 1}, "invalid", + "srdcheck.invalid_input", (0, 0, 0, 0)), + ] + for query_type, params, status, code, counts in cases: + result = project_table_evaluation( + ADAPTER.query(query_type, params), query_type, params) + assert result["status"] == status + assert result["exit_code"] == 2 + assert result["coverage"]["complete"] is False + assert result["findings"] == [] + assert result["errors"][0]["code"] == code + coverage = result["coverage"] + assert (coverage["compatible"], coverage["eligible"], + coverage["evaluated"], coverage["skipped"]) == counts + + +def test_projection_is_deterministic_and_input_bound(): + params = save_params(12) + verdict = ADAPTER.query("save.check", params) + first = project_table_evaluation(verdict, "save.check", params) + second = project_table_evaluation(verdict, "save.check", params) + assert first == second + assert first["evaluation_id"].startswith("srdcheck-") + assert first["cursor"]["input_digest"].startswith("sha256:") + changed = project_table_evaluation( + ADAPTER.query("save.check", save_params(13)), + "save.check", save_params(13)) + assert changed["evaluation_id"] != first["evaluation_id"] + + +def test_illegal_without_evidence_is_typed_internal_error(): + verdict = illegal("future illegal result", adapter="future@1.0") + result = project_table_evaluation(verdict, "future.query", {}) + assert result["status"] == "internal_error" + assert result["findings"] == [] + assert result["errors"][0]["code"] == \ + "table_evaluation.finding_evidence_missing" + + +def test_caller_context_is_joinable_but_never_grants_authority(): + params = save_params(12) + verdict = ADAPTER.query("save.check", params) + context = { + "session_id": "session-7", + "entity_refs": ["character:aria", "encounter:bridge"], + "correlation_id": "discord-message-42", + } + result = project_table_evaluation( + verdict, "save.check", params, context=context) + assert result["subject"]["session_id"] == "session-7" + assert result["subject"]["entity_refs"] == [ + "srdcheck-query-scope:save.check", "character:aria", + "encounter:bridge", "correlation:discord-message-42", + ] + assert result["authority_status"] == "self_attested" + assert result != project_table_evaluation( + verdict, "save.check", params, context={"session_id": "session-8"}) + + +def test_invalid_caller_context_is_rejected(): + params = save_params(12) + verdict = ADAPTER.query("save.check", params) + for context in ({"host_attested": True}, {"entity_refs": [""]}, []): + try: + project_table_evaluation( + verdict, "save.check", params, context=context) + except ValueError: + pass + else: + raise AssertionError("invalid caller context was accepted") + + +def test_cli_and_pipe_emit_shared_envelope(capsys, monkeypatch): + params = json.dumps({"kind": "attack"}) + assert main(["query", "mage-hand.use", params, + "--table-evaluation"]) == 1 + result = json.loads(capsys.readouterr().out) + assert result["schema_version"] == "table.evaluation/1.0" + assert result["status"] == "findings" + + context = json.dumps({"session_id": "session-cli", + "correlation_id": "turn-3"}) + assert main(["query", "mage-hand.use", params, "--table-evaluation", + "--table-context", context]) == 1 + contextual = json.loads(capsys.readouterr().out) + assert contextual["subject"]["session_id"] == "session-cli" + assert "correlation:turn-3" in contextual["subject"]["entity_refs"] + + monkeypatch.setattr("sys.stdin", __import__("io").StringIO("not-json")) + assert main(["--pipe", "--table-evaluation"]) == 2 + invalid = json.loads(capsys.readouterr().out) + assert invalid["status"] == "invalid" + assert invalid["errors"][0]["code"] == "srdcheck.invalid_input" + + monkeypatch.setattr("sys.stdin", __import__("io").StringIO("[]")) + assert main(["--pipe", "--table-evaluation"]) == 2 + wrong_shape = json.loads(capsys.readouterr().out) + assert wrong_shape["status"] == "invalid" + assert wrong_shape["coverage"]["evaluators"][0]["id"] == "invalid-query" diff --git a/tool.json b/tool.json index b963428..18e2ab1 100644 --- a/tool.json +++ b/tool.json @@ -23,6 +23,7 @@ "jurisdiction ": "Is this named entity known to the loaded rulesets? exit 0 = known content (all matching categories in data.categories); exit 2 = unknown/third-party content.", "cite ": "Verbatim, page-numbered source text for a named SRD heading. Provenance surface: returns the page's own text, never interpretation.", "query ": "Adjudicate a structured proposal. Query types are supplied by the loaded adapters; see query_types below.", + "query --table-evaluation": "Project the scoped verdict into deterministic, self-attested table.evaluation/1.0. Add --table-context JSON for caller-owned session/entity/correlation references. Table and execution authority stay external.", "--pipe": "Read one {\"type\", \"params\"} JSON query from stdin.", "--schema": "Print JSON Schema for input and output.", "capabilities": "Print engine, protocol, adapter versions/digests, query types, refusal recovery contract, and MCP tool names." @@ -52,6 +53,7 @@ ] }, "output": "JSON verdict object on stdout: {verdict, exit_code, why, citations[], rule_ids[], adapter, data?}. Exit-2 data contains stable reason and recovery fields. Schema and refusal contract identities are published by --schema, MCP outputSchema, and capabilities.", + "table_evaluation": "query/--pipe --table-evaluation maps legal to checked_clean, illegal to an exact-evidence finding, and structured refusals to invalid, unsupported, or incomplete. Output is always self_attested. Its named-query scope is machine-readable and every finding evidence ref resolves in its effective policy.", "notes": "The 'why' field is templated explanatory prose, not a machine compatibility field. Verdicts are deterministic: same query, same answer, every time. On cannot-adjudicate, branch on structured recovery data rather than prose; an authorized agent-DM may resolve a DM-authority table ruling directly. turn.options and turn.plan are consistency-tested against each other. Use `capabilities` for checked/unchecked scope, refusal mappings, and the exact release tuple.", "mcp": { "command": "python3 -m srdcheck.mcp", @@ -59,6 +61,7 @@ "protocolVersion": "2025-06-18", "tools": [ "jurisdiction", + "table_evaluation", "attack_modifiers", "check_make", "concentration_check", @@ -81,7 +84,7 @@ "ttt_move", "ttt_options" ], - "notes": "Zero-dependency stdlib MCP server. Tool descriptions/schemas are supplied by loaded adapters (queries.json). Verdicts arrive as structuredContent; illegal and cannot-adjudicate are results, not protocol errors." + "notes": "Zero-dependency stdlib MCP server. Tool descriptions/schemas are supplied by loaded adapters (queries.json). Verdicts arrive as structuredContent; illegal and cannot-adjudicate are results, not protocol errors. The opt-in table_evaluation tool projects one native query without changing native tool response schemas." }, "_generated_by": "scripts/gen_tool_json.py" }