Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,10 @@ $ python -m srdcheck query mage-hand.use '{"kind": "attack"}'
"rule_ids": ["mage-hand.cant-attack"],
"adapter": "srd-5.2.1@0.2.1"
}
$ python -m srdcheck query mage-hand.use '{"kind": "attack"}' --table-evaluation
# deterministic table.evaluation/1.0; status=findings, authority=self_attested
$ python -m srdcheck query save.check '{"modifier":2,"dc":10,"d20_result":12,"save_ability":"dex","saver_conditions":[]}' \
--table-evaluation --table-context '{"session_id":"session-7","correlation_id":"discord-42"}'
$ python -m srdcheck cite "Command" # verbatim SRD text + page for any registered name
$ python -m srdcheck --schema # I/O contract for agents
$ python -m srdcheck capabilities # engine, protocol, adapters, digests, tools
Expand All @@ -93,6 +97,18 @@ Deterministic, offline, and no tokens. CI gates 100-verdict p95 latency below
surface is young and growing slice by slice — the architecture (kernel +
[adapters](docs/adapter-spec.md), spec at v0.9 RC) is the point.

`query` and `--pipe` accept `--table-evaluation` for portfolio hosts that need
the shared `table.evaluation/1.0` envelope. Legal/illegal verdicts become
complete checked-clean/finding results only within the named query scope;
missing facts, unsupported content, ambiguity, and DM discretion remain
non-clean. The subject carries a machine-readable `srdcheck-query-scope:*`
reference; optional `--table-context` JSON can add caller-owned `session_id`,
`entity_refs`, and a `correlation_id` (emitted as `correlation:*`). Rule and
citation evidence is self-contained and keyed by every finding evidence ref.
Every result is `self_attested`, including when the calling agent is the DM:
this adapter cannot grant table, encounter-state, action, or protected-host
authority.

## For agents (MCP)

srdcheck is an MCP server with zero dependencies — stdlib only. After `pip install`, the command is `srdcheck-mcp`; from a clone it's:
Expand All @@ -117,7 +133,9 @@ boundary refusals do not invent provenance. An `illegal` verdict is a result,
not an error; `cannot-adjudicate` is an honest refusal, not a failure. Tool
descriptions and schemas come from loaded adapters, so new adapters extend the
tool list without kernel changes. See also [`tool.json`](tool.json) for the CLI
surface.
surface. The opt-in `table_evaluation` MCP tool accepts `{query_type, params,
context?}` and emits the same shared envelope without changing native query-tool
responses.

Every first-party `cannot-adjudicate` result carries stable recovery metadata
under `data`: a reason code, recoverability, exact missing-input paths, and a
Expand Down Expand Up @@ -167,7 +185,7 @@ Generated by `scripts/truth_scorecard.py` — regenerated and diff-checked in CI

| truth | claim | status | evidence |
|---|---|---|---|
| T1 | wrong verdicts | enforced in CI | 639 tests including gold suites ported from the Phase 0 eval; any wrong verdict fails the build |
| T1 | wrong verdicts | enforced in CI | 647 tests including gold suites ported from the Phase 0 eval; any wrong verdict fails the build |
| T2 | no citation, no rule | enforced in CI | 90/90 rule atoms carry verbatim source quotes; applied-rule paths cite, while boundary refusals do not invent provenance |
| T3 | advise, never overrule | structural | the API has no blocking or veto interface to wire; verdicts are advisory by construction |
| T4 | one payload, two audiences | enforced in CI | every verdict carries machine fields plus a templated plain-English why; schema-tested |
Expand All @@ -176,7 +194,7 @@ Generated by `scripts/truth_scorecard.py` — regenerated and diff-checked in CI
| T7 | mechanism never knows the game | enforced in CI | kernel lint scans every kernel module for game vocabulary (it caught a real violation during development) |
| T8 | honest boundaries | enforced in CI | refusal goldens: unknown content, unmodeled conditions, and genuinely ambiguous rules text all return exit 2; ambiguity cites its source, while registry and input boundaries may have no citation |
| T9 | never a single number | enforced in CI | bench scorecard freshness test; per-category tables, no aggregate score exists anywhere in this repository — even the leaderboard ranks by wrong-count alone (T1) and shows the other failure modes unblended |
| T10 | stranger-agent bootstrap | enforced in CI | cold-start conformance test reaches a first verdict from tool.json/--schema/MCP alone (22 tools); live probe: a frontier model given only tool.json produced a correct first verdict in 1 attempt(s), 3.9s (2026-07-16) |
| T10 | stranger-agent bootstrap | enforced in CI | cold-start conformance test reaches a first verdict from tool.json/--schema/MCP alone (23 tools); live probe: a frontier model given only tool.json produced a correct first verdict in 1 attempt(s), 3.9s (2026-07-16) |
| T11 | table speed | enforced in CI | p95 latency budget test: 100 verdicts must stay under 100 ms at p95; lower local observations are not a compatibility promise |
| T12 | never sell what the model has | held in review | a strategy invariant: features pitched on knowledge parity are cut in review — enforced by humans and admitted as such |
| T13 | the benchmark is a product | shipped | bench/ publishes 5 sets across 5 subjects with cited gold verdicts, incl. a per-horizon drift lane whose golds are engine-derived; a generated LEADERBOARD ranks any subject; cmd: driver + validate command let a third party submit a tamper-checked result |
Expand Down
22 changes: 22 additions & 0 deletions docs/capability-map.json
Original file line number Diff line number Diff line change
Expand Up @@ -679,6 +679,28 @@
"rules": "0.1.0"
},
"description": "Enumerate the legal moves for a player on a board (same encoding as ttt.move). Empty when it is not the player's turn or the game has ended."
},
{
"adapter": "kernel",
"query_type": "table.evaluation",
"tool": "table_evaluation",
"capability": "Project one native rules query into table.evaluation/1.0 with machine-readable scope, caller references, and portable evidence.",
"checked_scope": [
"deterministic projection of one named query",
"self-attested coverage and exact finding evidence",
"caller-owned session, entity, and correlation references"
],
"unchecked_scope": [
"global action or table legality",
"table, encounter-state, or execution authority",
"protected-host attestation"
],
"evidence": [
"tests/test_table_evaluation.py",
"tests/test_mcp.py::test_table_evaluation_tool_is_opt_in_scoped_and_joinable",
"scripts/cold_artifact_smoke.py"
],
"description": "Opt-in shared-envelope transport; native verdict tools remain unchanged."
}
],
"targets": [
Expand Down
1 change: 1 addition & 0 deletions docs/capability-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ A `legal` result below means only that the named checked scope passed.
| `feature_uses` | Return the modeled use-count formula for the closed feature set. | Divine Sense and Lay on Hands formulas; arithmetic when all blanks are supplied | other features; tables; whether a use remains in campaign state | `tests/test_spell_facts.py::test_divine_sense_blanks`; `tests/test_spell_facts.py::test_lay_on_hands_blanks` |
| `ttt_move` | Check one move against the bundled tutorial ruleset. | board shape, turn order, cell occupancy, and terminal state | strategy; choosing a move | `tests/test_toy_adapter.py::test_move_legality` |
| `ttt_options` | Enumerate legal cells under the bundled tutorial ruleset. | legal empty cells for the current player | strategy; ranking options | `tests/test_toy_adapter.py::test_enumerate_validate_consistency` |
| `table_evaluation` | Project one native rules query into table.evaluation/1.0 with machine-readable scope, caller references, and portable evidence. | deterministic projection of one named query; self-attested coverage and exact finding evidence; caller-owned session, entity, and correlation references | global action or table legality; table, encounter-state, or execution authority; protected-host attestation | `tests/test_table_evaluation.py`; `tests/test_mcp.py::test_table_evaluation_tool_is_opt_in_scoped_and_joinable`; `scripts/cold_artifact_smoke.py` |

## Target architecture — not shipped

Expand Down
8 changes: 8 additions & 0 deletions llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,14 @@ currently validate multiclass builds or enumerate every creature-specific move.
The calling agent may itself be the authorized DM; its table rulings remain
distinct from SRD-derived results.

For a suite result envelope, add `--table-evaluation` to `query` or `--pipe`.
The deterministic `table.evaluation/1.0` projection is always self_attested;
it preserves self-contained rule/citation evidence and never grants table
authority, including when the caller is an agent-DM. Its subject includes a
machine-readable named-query scope. Add `--table-context` JSON for caller-owned
session/entity/correlation references, or call the opt-in MCP tool
`table_evaluation` with `{query_type, params, context?}`.

On `cannot-adjudicate`, branch on the recovery fields in `data`, never on
`why`: repair the request, provide the named missing facts, select an adapter,
use another capability, resolve a table ruling, or stop.
Expand Down
26 changes: 26 additions & 0 deletions scripts/capability_map.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,32 @@ def build():
}
entry["description"] = adapter.query_meta[query_type]["description"]
shipped.append(entry)
shipped.append({
"adapter": "kernel",
"query_type": "table.evaluation",
"tool": "table_evaluation",
"capability": (
"Project one native rules query into table.evaluation/1.0 with "
"machine-readable scope, caller references, and portable evidence."),
"checked_scope": [
"deterministic projection of one named query",
"self-attested coverage and exact finding evidence",
"caller-owned session, entity, and correlation references",
],
"unchecked_scope": [
"global action or table legality",
"table, encounter-state, or execution authority",
"protected-host attestation",
],
"evidence": [
"tests/test_table_evaluation.py",
"tests/test_mcp.py::test_table_evaluation_tool_is_opt_in_scoped_and_joinable",
"scripts/cold_artifact_smoke.py",
],
"description": (
"Opt-in shared-envelope transport; native verdict tools remain "
"unchanged."),
})
return {
"schema_version": CLAIMS_SCHEMA_VERSION,
"generated_from": (
Expand Down
20 changes: 20 additions & 0 deletions scripts/cold_artifact_smoke.py
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,16 @@ def smoke(artifact):
))
assert query["exit_code"] == 0 and query["rule_ids"]

shared = json.loads(run(
[str(python), "-m", "srdcheck", "query", "mage-hand.use",
'{"kind":"attack"}', "--table-evaluation"],
outside, env=clean_env, allowed_returncodes=(1,),
))
assert shared["schema_version"] == "table.evaluation/1.0"
assert shared["status"] == "findings"
assert shared["authority_status"] == "self_attested"
assert "mage-hand.cant-attack" in shared["findings"][0]["evidence_refs"]

capabilities = json.loads(run(
[str(python), "-m", "srdcheck", "capabilities"],
outside, env=clean_env,
Expand Down Expand Up @@ -201,6 +211,11 @@ def smoke(artifact):
{"jsonrpc": "2.0", "id": 4, "method": "tools/call",
"params": {"name": "jurisdiction", "arguments": {
"name": unicode_name}}},
{"jsonrpc": "2.0", "id": 5, "method": "tools/call",
"params": {"name": "table_evaluation", "arguments": {
"query_type": "mage-hand.use", "params": {"kind": "attack"},
"context": {"session_id": "artifact-session",
"correlation_id": "artifact-call"}}}},
])
replies = [json.loads(line) for line in run(
[str(python), "-m", "srdcheck.mcp"], outside,
Expand All @@ -212,6 +227,11 @@ def smoke(artifact):
assert by_id[3]["result"]["structuredContent"]["exit_code"] == 0
assert by_id[4]["result"]["structuredContent"]["exit_code"] == 2
assert unicode_name in by_id[4]["result"]["structuredContent"]["why"]
projected = by_id[5]["result"]["structuredContent"]
assert projected["status"] == "findings"
assert projected["authority_status"] == "self_attested"
assert projected["subject"]["session_id"] == "artifact-session"
assert "correlation:artifact-call" in projected["subject"]["entity_refs"]

# --target places project.scripts launchers beneath the target itself.
# Invoke them directly so Windows .exe wrappers and both entry-point
Expand Down
15 changes: 14 additions & 1 deletion scripts/gen_tool_json.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ def build():
"query <type> <params-json>": (
"Adjudicate a structured proposal. Query types are supplied by "
"the loaded adapters; see query_types below."),
"query <type> <params-json> --table-evaluation": (
"Project the scoped verdict into deterministic, self-attested "
"table.evaluation/1.0. Add --table-context JSON for caller-owned "
"session/entity/correlation references. Table and execution "
"authority stay external."),
"--pipe": "Read one {\"type\", \"params\"} JSON query from stdin.",
"--schema": "Print JSON Schema for input and output.",
"capabilities": ("Print engine, protocol, adapter versions/digests, "
Expand Down Expand Up @@ -73,6 +78,12 @@ def build():
"contains stable reason and recovery fields. Schema and "
"refusal contract identities are published by --schema, "
"MCP outputSchema, and capabilities."),
"table_evaluation": (
"query/--pipe --table-evaluation maps legal to checked_clean, "
"illegal to an exact-evidence finding, and structured refusals to "
"invalid, unsupported, or incomplete. Output is always "
"self_attested. Its named-query scope is machine-readable and "
"every finding evidence ref resolves in its effective policy."),
"notes": (
"The 'why' field is templated explanatory prose, not a machine "
"compatibility field. Verdicts are deterministic: same query, same "
Expand All @@ -91,7 +102,9 @@ def build():
"Zero-dependency stdlib MCP server. Tool descriptions/schemas "
"are supplied by loaded adapters (queries.json). Verdicts "
"arrive as structuredContent; illegal and cannot-adjudicate "
"are results, not protocol errors."),
"are results, not protocol errors. The opt-in table_evaluation "
"tool projects one native query without changing native tool "
"response schemas."),
},
"_generated_by": "scripts/gen_tool_json.py",
}
Expand Down
2 changes: 2 additions & 0 deletions srdcheck/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,5 @@ def _detect_version():
CANNOT_ADJUDICATE, ILLEGAL, LEGAL,
Citation, Verdict, cannot_adjudicate, illegal, legal,
)
from .table_evaluation import (TABLE_EVALUATION_SCHEMA_VERSION,
project_table_evaluation) # noqa: F401
2 changes: 1 addition & 1 deletion srdcheck/access.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ def capabilities():
claims = json.loads(CLAIMS_PATH.read_text(encoding="utf-8"))
refusal_contract = v.refusal_contract()
adapters = []
tool_names = {"jurisdiction"}
tool_names = {"jurisdiction", "table_evaluation"}
for identifier in available_adapters():
root = ADAPTERS_DIR / identifier
manifest = json.loads((root / "manifest.json").read_text())
Expand Down
Loading
Loading