Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
213 changes: 177 additions & 36 deletions toolbox/mdcode/demo/semantic-model/skill/README.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,11 @@
#
# One of the judgments compares the call against a row in the database, which
# a judge can settle only if it can read the database. Running this demo
# therefore takes a judge that has been given one -- `--judge-reads-store` on
# the command line, or a `store` on the judge the agent hires.
# therefore takes a judge that has been given one -- a `store` on the judge the
# agent hires. Giving a judge your tables belongs to whoever embeds the
# runtime, so no kcmd flag offers it. `kcmd action-run` settles no guard at
# all -- it names them and writes anyway -- so running this demo from the
# command line reaches none of the rules below.
#
# What this costs: every gate is a model call, and a model can answer two
# identical calls differently. Don't copy the `$25` ceiling below -- a threshold
Expand Down
2 changes: 1 addition & 1 deletion toolbox/mdcode/docs/semantic-model/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ concepts the call changes — an executor in another system is an opaque handle,
so its blast radius is declared or it is unknown. Knowledge Catalog is the only
system an action is published to, and the only place it is governed; an action
that carries its own DML is also the one kind `kcmd` runs itself, with
`kcmd action run`. See
`kcmd action-run`. See
[Modeling write operations](actions.md).

A model can also state **constraints**: named invariants over the ontology that
Expand Down
310 changes: 146 additions & 164 deletions toolbox/mdcode/docs/semantic-model/actions.md

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions toolbox/mdcode/docs/semantic-model/fidelity.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,9 +114,9 @@ agree on every structural row and differ only where a Spanner target has no
13. **Constraints.** A constraint reaches Knowledge Catalog only, as one
`semantic-constraint` entry under the model entry, and `pull` reads it back.
Every other push target deploys nothing for it and warns once. Publishing is
all that push does with a constraint; what settles one is a run, where
`kcmd action run --judge` puts the guards an action names to a judge before
the write.
all that push does with a constraint; what settles one is a run, and the
runtime an application embeds is what puts each guard to a judge before the
write. `kcmd action-run` is not that runtime and checks no guard.
14. **Binding profiles.** A model may define several physical realizations, one
per binding profile. `--all-profiles` deploys a graph for every profile that
declares a deployment target, each to the backend its own target names; a
Expand Down
9 changes: 6 additions & 3 deletions toolbox/mdcode/docs/semantic-model/model_spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -537,9 +537,12 @@ reads the document ([§6](#6-the-extension-mechanism)).
Status: authored, validated, published, and settled at run time. Whatever
dispatches a call puts each guard to a language model before the transaction
opens and routes the verdict by `on_violation`; given no judge, the action is
refused rather than run past its rules. `kcmd action run --judge` does this
from a command line. Rules in
[Reference → Validation](reference.md#validation).
refused rather than run past its rules. No command line here does that:
`kcmd action-run` checks no guard, because who settles one belongs to whoever
dispatches the call in earnest. The [commerce
demo](../../demo/semantic-model/skill/README.md) shows what hiring a judge
takes, in recorded runs — nothing in this repository embeds one today.
Rules in [Reference → Validation](reference.md#validation).

A constraint says two things about a violation, under two separate keys.
**`on_violation`** is what a violation does to the write that tripped it:
Expand Down
108 changes: 80 additions & 28 deletions toolbox/mdcode/docs/semantic-model/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,35 +63,86 @@ scope you authored under. See [Pull](README.md#pull) for behavior.
| `--dry-run` | Reconstruct from the catalog and report what would be written, but write no files. |
| `--force-remove` | Replace a differently-named local model with the catalog's (see [Pull](README.md#pull)); without it, a pull that would leave the entry group holding two models fails. |

### action
### profiles

```bash
kcmd action list
kcmd action run <name> --arg <name>=<value> ...
kcmd profiles
```

`list` prints every action the models in the scope declare, with the store a run
would reach and the command line that runs each one. `run` executes one against
the Spanner or AlloyDB database the selected profile's deployment target names;
only a `sql` executor runs. A guard states its rule as a `judgment`, and is
settled by `--judge` before the transaction opens, and by a judge that can query
the model's tables when `--judge-reads-store` is passed as well. Without a judge
the action is refused rather than run unchecked. See
[Run it](actions.md#7-run-it).
Reports each binding profile the model declares: its deployment target, the
source each entity binds to, and what the profile cannot answer or cannot run.
Read-only — it merges and prunes each profile the way `push` does, but deploys
nothing and runs no live probe, so you can compare coverage before choosing one.

| Flag | Effect |
|------|--------|
| `--arg <name>=<value>` | Bind one action parameter. Repeat the flag for each one; the value is text, parsed against the parameter's declared ontology type. `run` only. |
| `--profile [name]` | Report only this profile. Naming one the model does not declare is an error, not an empty report. Defaults to every profile. |
| `--print-store` | Print only the store the profile deploys to, on one line and nothing else, for a script to read rather than parse out of the report: `project/instance/database` for a Spanner store, and the backend named ahead of the path for any other (`alloydb:project/region/cluster/instance/database`, `bigquery:project/dataset`). Errors when the scope holds more than one model, since those may name different databases. |

This is a read of the binding, never a choice of one. Nothing on any `kcmd`
command line names a store directly; `--profile` selects a binding and the
binding's deployment target decides where writes land. A script that creates,
seeds or drops that database asks for the name rather than repeating it:

```bash
IFS=/ read -r PROJECT INSTANCE DATABASE <<<"$(kcmd profiles --print-store)"
```

### action-list

```bash
kcmd action-list
```

Prints every action the models in the scope declare, with the store a run would
reach and the command line that runs each one. Opens no store and calls no
model. See [Run it](actions.md#7-run-it).

| Flag | Effect |
|------|--------|
| `--profile [name]` | Read the model under this binding profile. Its deployment target names the database the action runs against, so this is how you change stores. Defaults to `default_profile`, else the model's inline bindings. |

To read back the store a profile deploys to, ask the binding rather than the
listing: [`profiles --print-store`](#profiles).

### action-run

```bash
kcmd action-run <name> --arg <name>=<value> ...
```

Executes one action against the Spanner or AlloyDB database the selected
profile's deployment target names; only a `sql` executor runs.

**This command does not check the action's guards.** A guard states its rule as
a `judgment`, and settling one means putting it to a judge — which is a piece of
the runtime an application embeds, not of a command line for curating a model.
So the write happens and every rule the model states goes unenforced. The run
names the guards it passed over, before it opens the transaction, so a reader
watching one land sees what did not stand between them and it.

Running an action is not what `kcmd` is for — the command exists so that an
author can exercise a model they are curating, and find out whether the
statements do what they meant, without first standing up an agent. Settling the
guards takes something that embeds the runtime and hires a judge for it. Nothing
in this repository does today — the [commerce
demo](../../demo/semantic-model/skill/README.md) shows what it takes and keeps
recorded runs of it, but its own executor is this command, which settles none.

What this command still refuses is a model that is wrong about its own rules: a
guard naming a constraint the model does not declare, or one whose `judgment`
states nothing. Neither is a check standing down — they are the same two things
a push refuses, and no judge would have repaired either.

| Flag | Effect |
|------|--------|
| `--arg <name>=<value>` | Bind one action parameter. Repeat the flag for each one; the value is text, parsed against the parameter's declared ontology type. |
| `--profile [name]` | Read the model under this binding profile. Its deployment target names the database the action runs against, so this is how you change stores. Defaults to `default_profile`, else the model's inline bindings. |
| `--store` | Print only where a run would land, on one line and nothing else, for a script to read rather than parse back out of the listing: `project/instance/database` for a Spanner store, `bigquery:project/dataset` for a BigQuery one. Errors when the scope holds more than one model, since those may name different databases. `list` only. |
| `--judge [model]` | Settle each guard stated as a `judgment` by asking Gemini on Vertex AI, using the project and credentials `kcmd` already holds. Takes a model id, defaulting to `gemini-2.5-flash`. Without the flag, an action guarded by such a rule is refused rather than run unchecked, unless the rule declares `warn`, in which case the run commits and reports that the rule went unchecked. `run` only. |
| `--judge-location <region>` | Ask the judge in this Vertex AI region. The region is where the argument values are sent, so a project that has to keep them somewhere in particular names that region here. Defaults to `us-central1`. The environment's `compute/region` is deliberately not read, because a region chosen for Compute Engine is often one Vertex AI does not serve. `global` is accepted and reaches the host that serves it. `run` only. |
| `--judge-reads-store` | Let the judge query the model's own tables while it settles a rule, so a guard can compare the call against what is recorded rather than only against what the caller stated. The judge is shown the entities, tables and columns the selected profile binds, writes its own statement in that profile's dialect, and every statement it sends is printed. Each one is checked to be a single read beginning with `SELECT` or `WITH` and wrapped in a subquery, so that a data-modifying CTE cannot run; at most 20 rows come back and each value is clipped. Costs one model call more per guard, plus one for each round of reading. Says what a judge may do rather than hiring one, so pass `--judge` as well. Errors when the profile binds no table to read. `run` only. See [When the judge needs a fact](actions.md#when-the-judge-needs-a-fact). |

### agent
### agent-tools

```bash
kcmd agent tools
kcmd agent-tools
```

Prints what an agent would be handed for the models in the scope: one lookup
Expand All @@ -102,15 +153,15 @@ runs nothing.

A tool the runtime cannot call is listed and marked `[NOT RUNNABLE]` rather than
dropped, with the reason in its description, so a refusal is visible before any
agent exists. To see what a guard costs today, add a constraint to an action's
`guards` and run this again. What is offerable depends on what the caller holds,
so an action guarded by a `judgment` is marked unrunnable without `--judge` and
callable with it.
agent exists. A guard is not one of those reasons: who settles a rule belongs to
the application that embeds the runtime, and this command cannot know what that
will be, so withholding a guarded tool here would describe a caller rather than
the model. What it does print, in the tool's own description, is which rules the
agent's calls will be held to.

| Flag | Effect |
|------|--------|
| `--profile [name]` | Read the model under this binding profile. Defaults to `default_profile`, else the model's inline bindings. |
| `--judge [model]` | List what an agent holding a judge is offered: an action guarded by a rule stated in words is callable rather than `[NOT RUNNABLE]`. Takes a Gemini model id, defaulting to `gemini-2.5-flash`, on the same region rule as [`action run --judge`](#action). No model is called either way — a judge settles a rule when an action runs, and this listing runs none. |

A model whose profile names no Spanner database offers no tools, because calling
one needs a store. That model is reported as offering none and the rest of the
Expand All @@ -135,16 +186,17 @@ the caller typed, and a name the format does not allow fails before anything is
written.

Everything the binding decides lives in `SKILL.md`: the store, the executor
kinds, which actions this deployment cannot run and why, and the `kcmd action
run` line to try one with, all under one heading, plus the snippet for reading
kinds, which actions this deployment cannot run and why, and the `kcmd
action-run` line to try one with, all under one heading, plus the snippet for reading
the store directly. A reference page is the same bytes under any profile.

A guarded action is always described as runnable, because a rule stated in words
is settled by the runtime before the transaction opens and a guarded action only
ever runs against a runtime that has a judge. The command line printed for one
says `--judge`. There is no flag here to say otherwise: whether the caller of
`skills-generate` had a judge configured is a fact about that invocation, not
about the deployment the document is read against.
is `kcmd`'s, and `kcmd` is not that runtime: it settles no guard, and the
paragraph under the line says so. There is no flag here to say otherwise:
whether the caller of `skills-generate` had a judge configured is a fact about
that invocation, not about the deployment the document is read against.

| Flag | Effect |
|------|--------|
Expand Down
20 changes: 11 additions & 9 deletions toolbox/mdcode/docs/semantic-model/skills.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ at startup as a model with one.
## Generate one

Run it in a semantic-model scope, the same directory `kcmd push` and
`kcmd action run` work in:
`kcmd action-run` work in:

```bash
kcmd skills-generate --out skills
Expand Down Expand Up @@ -122,7 +122,7 @@ it, which lives in the model rather than in whoever wrote the agent:
```

The row names the action the way it was authored, because that's the string
`kcmd action run` takes and the string a refusal quotes back. The snake_case tool
`kcmd action-run` takes and the string a refusal quotes back. The snake_case tool
name a framework would register it under is on the reference page, stated once.

Then **Finding a record**, which exists because a skill of writes has a hole in
Expand Down Expand Up @@ -275,17 +275,19 @@ one. A rule stated in words is settled by asking a judge, and the runtime asks
it before the transaction opens — not the agent making the call. An agent that
judged its own call would be the constrained thing certifying itself, which is
no guard at all. So a guarded action only ever runs against a runtime that has a
judge, and that's the runtime every generated skill is written for: the command
line says `--judge`, and the paragraph under it says what the flag settles.
judge, and that's the runtime every generated skill is written for. The command
line in the skill is `kcmd`'s, and `kcmd` isn't that runtime: it settles no
guard, and the paragraph under the line says so, so an agent that tries the call
and watches it commit doesn't read that as the rules having held.
Whether you had a judge configured when you ran `skills-generate` is a fact
about that invocation, not about the deployment the document describes, so
there's no flag here to write the other kind of skill.

That section names the profile, the store, the executor kinds in play, and any
action that can't run here. It also carries a `kcmd action run` command line,
built by the same code that prints one under `kcmd action list` — so it arrives
with the flags this action's rules need and a typed placeholder per required
argument, and it's marked, in the skill itself, as the debugging path. `kcmd` is
action that can't run here. It also carries a `kcmd action-run` command line,
built by the same code that prints one under `kcmd action-list` — so it arrives
with a typed placeholder per required argument and nothing else, and it's
marked, in the skill itself, as the debugging path. `kcmd` is
a command line for inspecting a model, not the runtime an agent should call in
production; an agent that runs continuously should be handed these actions as
tools by its own framework, which reaches the same runtime.
Expand All @@ -309,7 +311,7 @@ tools by its own framework, which reaches the same runtime.
## See also

* [Modeling write operations](actions.md) — declaring the actions a skill
describes, and `kcmd agent tools`, which prints the same derivation instead of
describes, and `kcmd agent-tools`, which prints the same derivation instead of
writing it out
* [Binding profiles](profiles.md) — the profile the deployment-specific section
reads
17 changes: 9 additions & 8 deletions toolbox/mdcode/src/libts/gcp/gemini.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,12 @@ export const DEFAULT_JUDGE_MODEL = 'gemini-2.5-flash';


// Vertex serves models from a region, and not every region serves every model.
// A caller that knows better names one, through `--judge-location` or this
// option; everything else uses a region that serves Gemini. The region is also
// where the argument values are sent, so a project that has to keep them
// somewhere in particular names that region. What is deliberately NOT consulted
// is `gcloud config get-value compute/region`, which is whatever the user set
// for Compute Engine and is routinely somewhere Vertex is not -- `us`, say,
// A caller that knows better names one through this option; everything else
// uses a region that serves Gemini. The region is also where the argument
// values are sent, so a project that has to keep them somewhere in particular
// names that region. What is deliberately NOT consulted is `gcloud config
// get-value compute/region`, which is whatever the user set for Compute
// Engine and is routinely somewhere Vertex is not -- `us`, say,
// which is not a Vertex endpoint at all. Reading it would make a judge
// unreachable over an unrelated setting, and an unreachable judge refuses
// writes that are fine.
Expand Down Expand Up @@ -187,8 +187,9 @@ export class GeminiJudge extends ApiClient implements Judge {
// this judge will ever be asked, and rebuilding it per call would put the
// cost of describing the model on every guard.
this._system = systemInstruction(options.store);
// Read off which model this is, so `--judge` and `--judge gemini-2.5-flash`
// send the same request. A budget of 0 is a per-model limit. The model this
// Read off which model this is, so a caller taking the default and a
// caller naming that same model send the same request. A budget of 0 is a
// per-model limit. The model this
// file picked accepts it; gemini-2.5-pro rejects it outright with `The model
// does not support setting thinking_budget to 0`, and an unreachable judge
// refuses every guarded write. So every other model is sent no budget and
Expand Down
15 changes: 13 additions & 2 deletions toolbox/mdcode/src/libts/semantic/runtime/agent_tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,16 @@ export interface ActionToolOptions {
* refused mid-call.
*/
judge?: Judge;
/**
* Derive the tools as a caller that will not check the guards at all: the
* write happens and every rule the model states goes unenforced. For trying
* a model out where no judge is configured, which is otherwise a model whose
* every guarded action is unofferable.
*
* Passed to the call as well as to the derivation, for the reason `judge` is:
* a tool derived one way and called the other is advertised wrongly.
*/
skipGuards?: boolean;
}


Expand Down Expand Up @@ -177,8 +187,8 @@ function toolFor(action: Action, opts: ActionToolOptions): ActionTool {
// model, then the runtime having no store, which is the same sentence on
// every tool and says nothing about this one.
const model = opts.runtime.model;
const blocked =
whyRefusedWithoutRunning(model, action, handler, opts.judge) ??
const blocked = whyRefusedWithoutRunning(
model, action, handler, opts.judge, opts.skipGuards) ??
noStore(opts.runtime) ?? undefined;
const tool: ActionTool = {
name: snakeCase(action.name),
Expand All @@ -193,6 +203,7 @@ function toolFor(action: Action, opts: ActionToolOptions): ActionTool {
args,
handler,
judge: opts.judge,
skipGuards: opts.skipGuards,
});
return describeOutcome(outcome);
},
Expand Down
Loading
Loading