diff --git a/openspec/changes/export-import-prompts-and-assistants/.openspec.yaml b/openspec/changes/export-import-prompts-and-assistants/.openspec.yaml new file mode 100644 index 000000000..5fcc9ecae --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-10-13 diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md new file mode 100644 index 000000000..6dc34d265 --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -0,0 +1,216 @@ +# Design + +## Context + +`langstar prompt push` in `cli/src/commands/prompt.rs` builds one shape of prompt: a single human message, with an optional schema from [#402 structured output prompts](https://github.com/codekiln/langstar/issues/402). `langstar prompt pull` prints any manifest LangSmith returns, but `PromptClient` in `sdk/src/prompts.rs` never asks LangSmith for the model, so that manifest has none. `PromptClient` keeps the manifest as raw JSON and has a typed `StructuredPrompt` view that holds messages, the schema and the method. + +Langstar's assistant SDK lacks what an export needs: `Assistant` in `sdk/src/assistants.rs` has no `context` or `version` field, and the search in `AssistantClient` cannot filter by graph ID. The assistant commands themselves already work against the test deployment: `test_assistant_description_field` in `cli/tests/assistant_command_test.rs` creates and reads an assistant there in the Integration Tests CI job. + +Langstar reads one API key and one workspace at a time, from flags, environment variables or the config file. Langstar exits with status 1 when a command fails, and clap, the library that parses langstar's flags, exits with status 2 when it rejects a mistyped flag or a missing required one. + +## Goals / Non-Goals + +**Goals:** + +- Recreate a prompt or an assistant from its export file with nothing lost. A prompt keeps its manifest and model, its commit description and tags, and its repo details. An assistant keeps its name, description, graph ID, configuration, context and metadata. +- Make a repeated import of the same file safe: a second import of one file finds the copy the first one made and changes nothing until the user asks. +- Give users one way to check a copy: the same report shape and exit statuses for both resource types. + +**Non-Goals:** + +- Copy a prompt or assistant between environments in one command. +- Export or import every prompt in a workspace, or every assistant on a deployment, in one run. +- Import onto an Agent Server the control plane does not list, such as a local `langgraph dev` server. The assistant commands find a deployment with `--deployment `, which [`resolve_deployment_url` in cli/src/deployment_utils.rs:23](https://github.com/codekiln/langstar/blob/7850530475f49eb6374652855d6f3a04741899d3/cli/src/deployment_utils.rs#L23) looks up in the control plane's deployment list, and the destination record keys an assistant by that deployment's ID. +- Redesign `prompt create`, `get` and `update`, which [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728) covers. + +## Decisions + +### Export and import run at different times, each with its own environment's API key + +Export runs with the source environment's credentials and writes a file. Import runs later with the destination's credentials and reads it. Each run uses langstar's existing settings, so a user moving a prompt to another organization sets a different `LANGSMITH_API_KEY` for the import. Langstar needs no second set of credentials and no profiles. The file sits in Git between the two runs, so a reviewer sees the prompt or assistant change in a pull request. + +The LangSmith Go CLI moves a prompt through a file the same way, as `references/langsmith-cli.md` shows. Langstar's export asks for the model, and the Go CLI's pull does not. Langstar's import reads the file export writes, while the Go CLI's push sends whatever JSON it reads as the manifest, so its own pull output reaches LangSmith wrapped in an envelope. Langstar's import also requires `--workspace-id`, so langstar imports into the workspace the user typed on the command line, even when `LANGSMITH_WORKSPACE_ID` names another. + +### The export file keeps the manifest LangSmith returned + +Export stores the manifest as raw JSON. The typed `StructuredPrompt` struct cannot hold a model binding, placeholders, several message types or partial variables, so routing through it would drop them. Langstar reads the manifest through typed views only to fill in `requires`. + +One prompt export looks like this: + +```json +{ + "kind": "prompt", + "format_version": 1, + "source": { "workspace_id": "", "handle": "my-prompt", "commit_hash": "" }, + "repo": { "handle": "my-prompt", "description": "...", "readme": "...", "tags": ["support"], "is_public": false }, + "commit": { "description": "...", "manifest": { "lc": 1, "type": "constructor", "id": ["langsmith", "playground", "PromptPlayground"], "kwargs": { "first": {}, "last": {} } }, "model_config": {}, "model_provider": "...", "tags": ["prod"] }, + "requires": { "secrets": ["OPENAI_API_KEY"], "model_class": "ChatOpenAI" } +} +``` + +An assistant export has `kind: "assistant"`, a `source` with the deployment, assistant ID and version, an `assistant` object with `name`, `description`, `graph_id`, `config`, `context` and `metadata`, and `requires.prompts`. + +The file holds no export time, and langstar sorts its keys, the commit's tags, the repo's tags and the secret names under `requires.secrets`, so exporting an unchanged resource leaves Git with nothing to commit. + +Without `--output`, export prints the file to stdout and nothing else, so `langstar prompt export my-prompt > my-prompt.json` writes a file import can read. Export writes its progress messages to stderr in every output format. [`OutputFormatter::print_message` in cli/src/output.rs:173](https://github.com/codekiln/langstar/blob/e4ae746d51f1f3f9ccf3fc97787148c3b4df2bce/cli/src/output.rs#L173), which `info` calls, prints to stdout in table mode, so export does not use it for those messages. + +Import, compare and verify read a file only when its `format_version` is 1 and its `kind` matches the command: `prompt` for the prompt commands and `assistant` for the assistant commands. For any other file the command writes nothing, exits with status 1 and names the field and the value it found. When a user runs `prompt import` on an assistant export, or on a file a newer langstar wrote, langstar stops with that error before it changes anything in the destination. + +### Export refuses to write a file that holds a credential + +Before it writes anything, export runs a credential check over every string it would put in the `repo`, `commit` and `assistant` objects, because a secret pushed to Git stays in the repository's history after someone deletes the file. For a prompt, the check reads the manifest with each model setting, header, `model_kwargs` entry, message template and `metadata` value in it, `commit.model_config`, the commit description and the repo's description and readme. For an assistant it covers the name, description, configuration, context and metadata. The check skips a LangChain secret reference, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, because it holds only a secret's name, and skips the empty string. + +The check flags a string when the name of its key or the shape of its value marks it as a credential. + +- **By key name:** the check flags a string whose key, or the key of the array it sits in, holds one of the words `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or holds `key` or `keys` right after `api`, `access`, `private`, `secret`, `signing` or `account`. It splits a key name into words at `_`, `-`, `.`, spaces and each change from a lower-case to an upper-case letter, and ignores case. It flags `openai_api_key`, `OPENAI_API_KEY`, `x-api-key`, `Authorization`, `access_token`, `client_secret`, `db_password`, `privateKey` and `api_keys`, and passes `max_tokens` and `cache_key`. +- **By value shape**, under any key: the check flags a string that holds a bearer token (`Bearer ` followed by at least 16 token characters) or a `Basic ` value whose base64 payload decodes to printable text with a colon in it, the `user:password` shape, however short that payload is: `Basic dXNlcjpwYXNz` carries twelve base64 characters and holds `user:pass`. A `Basic ` value whose payload decodes to anything else, as `Basic instructions follow.` does, passes; a PEM private key (`-----BEGIN` and `PRIVATE KEY-----`); a URL with a password in it, `scheme://user:password@host` or `scheme://:password@host`, which covers database and queue connection strings; a key-value connection string with a non-empty `Password=`, `Pwd=`, `AccountKey=`, `SharedAccessKey=` or `SharedAccessSignature=`; a JSON Web Token, three base64url parts that start with `eyJ`; or a token with a provider's published prefix, which are `sk-` (OpenAI and Anthropic), `lsv2_` (LangSmith), `ghp_`, `gho_`, `ghs_` and `github_pat_` (GitHub), `xoxb-`, `xoxp-` and `xoxa-` (Slack), `AKIA` and `ASIA` followed by 16 upper-case letters or digits (AWS), `AIza` (Google) and `hf_` (Hugging Face), each followed by enough characters to rule out an ordinary word. + +When the check flags any string, export writes no file and nothing to stdout, lists the JSON Pointer path of each flagged string with the rule that flagged it, such as `key name` or `URL with a password`, leaves the string and every part of it out of the message, and exits with status 1. Export refuses rather than writing `` in place of the credential. A redacted file would import a prompt or assistant that cannot authenticate, and a reviewer reading the file in a pull request would not see that the source holds a credential. The refusal asks the user to move the credential into the workspace's secrets, which a prompt's model reads through a secret reference, or into the deployment's environment variables, which a graph reads. + +Some flagged strings are harmless, such as a message template that shows `Bearer ` followed by an example token. A user who has read such a string names its path with `--not-a-secret `, once for each path, and export writes the string and prints a warning to stderr naming the path. When a pointer names a string the check lets through, export writes nothing, names the pointer and exits with status 1, so a script that keeps naming a path after the string there has changed stops at the export. Every export runs the check. + +A credential under an ordinary key name with an ordinary-looking value, such as a random string under a key called `x`, passes the check and reaches the file. The export docs tell users to keep credentials in secrets and environment variables, and to read an export file before committing it. + +### Export asks LangSmith for the model + +Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest wraps the prompt and the chat model with its settings in one constructor, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). No field of the create-commit request holds the model, so the model reaches the destination inside the manifest, and export keeps `model_config` and `model_provider` in the file only for the reader. The LangSmith Python SDK's [`PromptCommit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/schemas.py#L1054) documents `model_provider` as the model's class name, such as `ChatOpenAI`. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer pushes that manifest to a second workspace and records whether LangSmith returns the same manifest, `model_config` and `model_provider`, and which outer `id` the live API returns. + +### Import records the IDs it created in a separate file next to the export + +Import writes the IDs of what it created to `.destinations.json`, beside the export file, keyed by destination workspace ID for prompts and by deployment ID for assistants. A separate file keeps the two apart: export never has to carry an earlier record forward when it overwrites its file, the file a reviewer reads in Git changes only when the source changes, and a fresh export never wipes out the record of earlier imports. Import replaces the record by writing the whole new record to a temporary file in the same directory and renaming it over the old one, so a langstar that stops in the middle of a write leaves the earlier record whole, pending assistant IDs included. + +Langstar supports one import of an export file at a time, because a person runs import by hand from a Git checkout. When two imports of the same file run at once, each reads the record before the other writes it, and the import that writes last drops the other's entries. Two first imports of one assistant into the same deployment can also create two assistants. [#828 make concurrent prompt and assistant imports into one deployment safe](https://github.com/codekiln/langstar/issues/828) covers making concurrent imports safe. + +A repeated prompt import finds the copy by its handle and uses the record to tell whether someone changed it. Import writes the new commit hash to the record before it reads the commit back, so a read-back that finds a difference still leaves the hash in the record. The next import compares the destination's latest commit with that hash, and reports a change only when someone else pushed a commit after langstar's. Langstar can stop after it pushes a commit and before it writes that commit's hash to the record. The next import finds the destination's latest commit matching the file, writes its hash to the record and reports the prompt unchanged. + +An assistant's ID differs on every deployment, so the record is the only way a repeated import finds the assistant it made. Import chooses the new assistant's ID itself and writes it to the record, marked pending, before it sends the create request with that `assistant_id` and `if_exists` set to `do_nothing`, which the Agent Server's `AssistantCreate` accepts. After the create succeeds, import marks the entry created. If langstar stops before then, the next import finds the pending entry and sends the same create again, and the Agent Server returns the assistant the first import made. Assistant compare and verify find the assistant through the same record, and take `--record ` as import does, so a record kept at another path still works. + +### Import changes an existing resource only with --update + +A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. + +A public destination repo is the one repo detail import cannot leave for later. When the destination repo is public and the file's `repo.is_public` is false, a push would publish the prompt's text, and reporting the difference afterwards cannot take it back. So with `--update`, import makes the repo private before it pushes anything; without `--update`, it pushes nothing, reports `/repo/is_public` and exits with status 3. This holds for an import that finds a repo someone else made and for one finishing after an earlier run stopped between creating the repo and pushing its first commit. + +Import treats the destination commit's tags as one more field to match. A tag in the file that the destination lacks, or that points at another commit there, is a difference, and so is a tag on the destination's matching commit that the file lacks. With `--update`, import creates or moves each of the file's tags to that commit and deletes each tag on it that the file lacks, through `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, so the read-back finds the same tags as the file. It adds no commit when the latest commit already matches. A tag on an older destination commit is not part of the comparison, so import leaves it alone. The tag endpoints take a commit's `id`, which differs from its hash. The commit read and the repo read return no `id`, so import looks the commit up in the commit list, `GET /api/v1/commits/{owner}/{repo}`, whose entries carry both the `id` and the hash. + +An import that stops after it creates the repo and before it pushes the commit leaves a prompt with no commit. The next import finds the prompt by its handle and, through the commit list, finds no commit. It then pushes the file's manifest as the first commit and creates the file's tags, as it does for a new prompt, without `--update`. The repo details it created hold the file's values already, and any that someone changed in between still need `--update`: without it, import pushes the commit and tags, leaves those repo details alone, lists them as differences and exits with status 3. + +With `--update`, import makes an existing assistant match the file and asks the Agent Server to remove what the file lacks. The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So import sends the file's name, graph ID and settings in full, and a null value for each metadata key the destination has and the file lacks. [Import sends an assistant's settings in one field](#import-sends-an-assistants-settings-in-one-field) says which of the two settings fields import sends and which it leaves alone. `AssistantPatch` documents no way to clear a description, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer sends a null and an empty description to the live server, along with a null metadata value and a replaced configuration and context, and records what the server keeps. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 3. The Agent Server saves each update as a new version of the assistant. + +Without `--update`, import lists the differences and exits with status 3. `--dry-run` reports the action import would take and writes nothing. + +### Import sends an assistant's settings in one field + +An assistant keeps its settings in `config.configurable`, the older field, and in `context`, which arrived with LangGraph 0.6. When langstar reads an assistant, the Agent Server returns the same settings in both, so an export file holds them twice. A newer Agent Server refuses to create an assistant that sets both, with status 400 and `Cannot specify both configurable and context. Prefer setting context alone.` An import that sent the file's fields as they stand would fail on that server. + +So import sends one field: + +- When the file has both and they hold the same values, import sends `config` with `configurable` and leaves `context` out. The server fills `context` from `configurable` itself, so the read-back matches the file and compare finds no difference. A trial run that copied assistants from one deployment to a newer one sent `config` this way, and every copy matched its file. +- When the file has only one of them, import sends that one. +- When the file has both and they differ, import cannot tell which one the user meant. It writes nothing, names `/assistant/config/configurable` and `/assistant/context` without their values, and exits with status 1, the way it rejects a file of another kind. + +Compare reads an assistant's settings once, because the two fields hold one value. It takes the file's settings from whichever field the file holds and the destination's from whichever the server returned, and reports a difference only when the values differ. A settings field one side carries and the other lacks is no difference, since the server fills the second field from the first. So a file that holds settings only in `config.configurable` imports into a server that answers with both, and compare reports nothing. + +`--update` sends the settings in one field the same way, and leaves the other field alone. An empty `context` sent beside a `config.configurable` would either be refused by a newer server or wipe out the settings the server had just filled in from `configurable`. Import clears `context` only for a file that holds settings in neither field. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer records whether an update that sends both is refused as a create is, and whether an update that sends `configurable` alone replaces `context` too. + +Export still writes both fields as the server returned them, so the file shows what the source held. + +### Compare reports differences by JSON path + +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Each difference names its JSON Pointer path and both values. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It reads an assistant's `config.configurable` and `context` as one settings value, as [Import sends an assistant's settings in one field](#import-sends-an-assistants-settings-in-one-field) says, and compares every other key under `config` as it stands. For a prompt, it also covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. + +Compare skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets out of those objects altogether — IDs, version numbers, timestamps, owner and like counts — and keeps the source environment's workspace, deployment and IDs in `source`, so none of them reads as a difference against a copy. + +The report goes to a terminal or a CI log, so it hides credentials before it prints. It prints `` in place of every string export's credential check flags, by either rule, whether the difference sits at that string or at an object holding it. A credential someone pasted into the destination prompt, the destination assistant or the file therefore stays off the screen while the user still sees where each difference is. A LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, prints unchanged, because it holds the secret's name and never its value. + +### `prompt verify` and `assistant verify` answer whether the copy can run + +A user who has just imported a prompt or an assistant wants one question answered: will it run here? A create request that came back 200 does not answer it. The copy runs only if the destination workspace holds the API keys the prompt's model reads, the destination deployment serves the graph the assistant names, and the destination workspace holds the prompts the assistant refers to. Verify asks the destination about each of those and names what is missing, so the user fixes it before a learner or a test hits the copy. + +Prompt verify starts with the workspace's secrets. A prompt's model needs an API key, and the manifest names the key rather than carrying its value, as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`; that is the shape the Python SDK's `_manifest_has_secrets` matches. Export walks the manifest for those references and writes each name into `requires.secrets`. Verify reads the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, the request `langstar secrets list` already sends) and prints the name of every one the workspace lacks. It prints names, never values. + +Prompt verify then looks for a saved model configuration for the prompt's model, and reports a missing one as a note. The prompt still runs without one, because its manifest carries the model and its settings. Export reads the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and writes it to `requires.model_class`. Verify then looks for a saved model configuration whose settings `id` ends in that class name (`GET /api/v1/playground-settings`, the request `langstar model-config list` sends). A prompt someone saved without a model has no chat model constructor, so export writes `requires.model_class` as null, and verify and import skip this check and still check the secrets. [Prompt verify reports a missing saved model configuration as a note, not a failure](#prompt-verify-reports-a-missing-saved-model-configuration-as-a-note-not-a-failure) gives the reason. + +Import and verify both read the file's manifest and work out `requires` from it again, then compare that with the `requires` the file carries. When the two disagree, they write nothing, name the field and exit with status 1. This catches a manifest someone edited in Git and left `requires` behind. + +Assistant verify asks the destination deployment whether it serves the graph the assistant names. The Agent Server has no call that lists a deployment's graphs, so verify searches the deployment's assistants for that graph ID (`POST /assistants/search` with `graph_id`). Every graph a deployment serves has a default assistant, so a deployment that serves the graph answers with at least that one, and an empty answer means the deployment does not serve it. Verify then names the graph ID as missing and the user picks a deployment that serves it. + +The Agent Server may refuse the search instead of answering it with nothing: its OpenAPI spec lists the deployment's graph IDs as the allowed values of `graph_id`, so an unknown graph ID may come back as a 422. Verify reads a 422 whose error names `graph_id` the same way it reads an empty answer, as a graph this deployment does not serve, and exits with status 1 on every other error. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer runs the search against the live server and records which of the two answers it gives. + +Assistant verify also checks each prompt the assistant refers to. `requires.prompts` holds names such as `support-system:prod`, and verify asks LangSmith for the commit each tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A name has the form `[/][:]`: with no owner, verify uses `-`, which LangSmith reads as the current workspace, and with no tag it asks for `latest`. A 404 means the workspace lacks that prompt or that tag, and verify prints the name. + +### Import runs verify's checks before it writes anything + +Import runs the checks `verify` runs, and it runs them before it creates anything. Prompt import stops when the destination workspace lacks a secret from `requires.secrets`. Assistant import stops when the deployment does not serve the file's graph, or when the workspace — the one from `--workspace-id`, or the configured one — lacks a prompt from `requires.prompts`. Either one names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failures. + +Checking first saves the user a cleanup. An import that creates a prompt whose API key the destination workspace lacks leaves behind a copy that cannot run, and the user has to find that copy and delete it by hand. Running the checks first costs only the order of the steps. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. + +A user therefore creates the destination workspace's secrets before importing a prompt, and imports the prompts before importing an assistant that refers to them. + +### Users name an assistant's prompt references with `--prompt-ref` + +An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find those references by itself, and guessing from strings that look like prompt handles would miss some and flag others. The user names each one with `assistant export --prompt-ref `. The assistant's configuration should name each prompt by handle and tag, such as `support-system:prod`, because the same prompt gets a different commit hash in each workspace. Import copies the commit tags to the destination workspace, so when the graph on the destination deployment asks LangSmith for `support-system:prod`, it gets the same prompt. Compare, verify and import look up each pointer from `requires.prompts` in the file's `assistant` object, and exit with status 1 when the value there differs from the prompt name recorded beside the pointer. This catches a configuration someone edited in Git without updating `requires.prompts`. + +### compare, verify and import exit with 3 when they find a difference + +Status 3 is new. Langstar exits today with 0 when a command succeeds, with 1 from `main` in `cli/src/main.rs` when a command returns an error, and with clap's 2, so 3 is the first status free to carry a meaning of its own. Status 0 means the resource matches and every check passed. Status 3 means langstar found a difference or a failed check. Status 1 means any other error. + +Status 2 stays clap's: `Cli::parse()` in `run` in `cli/src/main.rs` exits with 2 when a user mistypes a flag or leaves out a required one, and this change leaves that alone, so a script never reads a typo as a difference. `prompt import` declares `--workspace-id` optional, as the other `langstar prompt` subcommands do, and checks for the flag itself, so a user who leaves it out gets status 1 and an error naming `--workspace-id` where clap would have exited with 2. `diff` uses 1 for differences and 2 for errors; langstar keeps 1 for errors, so a script testing for failure sees the same status from every command. + +### An optional Python script checks that an imported structured prompt still returns structured data + +The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model and no schema, and no LangSmith SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. + +A mise task runs the script with `uv run --locked --script` against a committed script lockfile, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the workspace import wrote to: the import test passes it the workspace ID it gave `prompt import --workspace-id` and the handle it imported under, and the script builds its LangSmith `Client` with `workspace_id` set to that ID rather than reading `LANGSMITH_WORKSPACE_ID`. That is the existing test workspace today, and the destination workspace once [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16) adds one. + +The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. Running the prompt bills an OpenAI API key on every run, so the mise task skips the check and says so when `OPENAI_API_KEY` is unset. Every other test runs without a paid key, and a developer with a key of their own runs this one by hand. + +### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` + +[#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728) plans `prompt create`, `get` and `update` commands that edit a prompt's text, model or structured output one field at a time. Export and import read and write the whole prompt at once, through the same `PromptClient` calls in `sdk/src/prompts.rs` that those commands will use, so a field the SDK reads for export is there for `prompt get` too. + +### Developers commit the new SDK fields as `✨ feat(sdk)`, so the next release is a minor version + +Developers commit the SDK's new fields and calls as `✨ feat(sdk)`, with no `BREAKING CHANGE:` footer and no `!` after the type, and the Prepare Release workflow sets the version in `Cargo.toml`. The new fields go on `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`, in [export-import-prompts-and-assistants Task group 3 SDK reads and writes the fields export and import need](tasks.md#L21). The repository's release docs ask for this: + +- [docs/dev/git-scm-conventions.md:26](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L26) gives `✨ feat` a minor version bump, and [docs/dev/git-scm-conventions.md:28](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L28) keeps the major bump for `🚨 BREAKING CHANGE`. [docs/dev/ci-cd.md:209](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/ci-cd.md#L209) also gives a major bump to a commit with a `BREAKING CHANGE:` footer or a `!` before the colon. +- [docs/dev/procedures.md:116](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/procedures.md#L116) names changing function signatures and removing fields as breaking SDK changes. Adding a field or a call is neither. +- The Prepare Release workflow, [.github/workflows/prepare-release.yml:62](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/.github/workflows/prepare-release.yml#L62), picks the bump by running the bump-release skill's `analyze_commits.py` over the commits since the last tag, and writes the new version itself. git-cliff writes the changelog from the same commit messages, so nobody writes release notes or a version by hand. +- [release.toml:5](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/release.toml#L5) sets `publish = false`, which keeps `langstar-sdk` off crates.io, so the only crate known to build these structs is the CLI in this repository. +- [4a57ccb ✨ feat(assistant): add description field support from Agent Server API (#726)](https://github.com/codekiln/langstar/commit/4a57ccb6802e988a1210ba8a8b1434a9bf9a803e) added `description` to `Assistant`, `CreateAssistantRequest` and `UpdateAssistantRequest` with no footer, and langstar released it in v2.2.0, a minor version after v2.1.2. + +Developers leave the structs as they are, without `#[non_exhaustive]`, because adding that attribute would stop the CLI and the SDK's integration tests from building `Assistant`, `CommitRequest`, `CreateAssistantRequest` and `UpdateAssistantRequest` with struct literals, which [docs/dev/procedures.md:116](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/procedures.md#L116) counts as a breaking change. + +### Prompt import requires `--workspace-id`, even when `LANGSMITH_WORKSPACE_ID` is set + +Import writes, and `LANGSMITH_WORKSPACE_ID` is easy to leave pointing at the wrong workspace: `fnox.toml` and CI set it to the source workspace, so a fallback would put the copy back where it came from without a word. Requiring the flag puts the destination on the command line, where the user and the shell history can see it, as assistant import does with `--deployment`. + +### Prompt verify reports a missing saved model configuration as a note, not a failure + +The prompt's manifest carries the model and its settings, so the prompt runs without a saved model configuration, and the secret check is the one that finds a prompt that cannot authenticate. Failing on the note would make verify exit with 3 for a prompt that runs. + +### Developers add the second workspace and deployment before they archive the change + +The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for that move: assistant tests that create an assistant on a separate deployment, and an assistant whose prompts a developer imported into a separate workspace. When the source and the destination are one workspace and one deployment, an import that ignores `--workspace-id`, or that writes to the deployment it read from, passes every test anyway, because it writes where the test looks. The tests that need a second environment carry an `#[ignore]` naming the task that adds it: the assistant graph moves to `test_graph_alt` in [export-import-prompts-and-assistants Task 3.3 Add the assistant fields to the SDK](tasks.md#L29) and [export-import-prompts-and-assistants Task 10.2 Add assistant import --update](tasks.md#L158) need the destination deployment from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19), and the test in [export-import-prompts-and-assistants Task 9.2 Add assistant verify](tasks.md#L138) of a prompt that only the destination workspace holds needs the workspace from [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16). Developers merge [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16), [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18) and [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) before anyone archives the change. The pull request for each one removes the `#[ignore]` from the tests that waited for it, and the pull request that archives the change links a CI run in which those tests passed. If the control plane answers [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18) with a 409, this repository cannot hold a second deployment, and the developer reports that answer to codekiln, who decides what the assistant tests run against. + +Trying export and compare on a real set of assistants and prompts, with langstar's current commands and the SDK, showed that one workspace and one test deployment are enough for the rest of the work. Only the assistants needed to move; the prompts stayed in one shared workspace, and the assistants name them without an owner or a commit. Every failure the trial found shows up in an export, or in an import to any deployment. The trial needed no live model call either: comparing each structured prompt's schema and model settings finds a bad copy, and a live call tests the model, not the copy. + +## Risks / Trade-offs + +- [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. +- [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. +- [The SDK changes for export and import add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal stops compiling] → `release.toml` keeps `langstar-sdk` off crates.io, and the developer who adds a field updates every struct literal of that struct in the same pull request, as [docs/dev/procedures.md:116](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/procedures.md#L116) asks. [Developers commit the new SDK fields as `✨ feat(sdk)`, so the next release is a minor version](#developers-commit-the-new-sdk-fields-as--featsdk-so-the-next-release-is-a-minor-version) gives the release policy. +- [A prompt or an assistant may hold a credential, such as an API key, a token, a password, an authorization header, a connection string with a password or a private key, which export would copy into a file meant for Git] → Export runs the credential check from [Export refuses to write a file that holds a credential](#export-refuses-to-write-a-file-that-holds-a-credential) and, when it flags a string, writes nothing, names each path and exits with status 1. Only `--not-a-secret ` lets a flagged string through, one path at a time. +- [A credential under an ordinary key name with an ordinary-looking value, such as a random string under a key called `x`, passes the credential check] → The export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L67) and [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L125) list what the check catches and what it cannot, tell users to keep credentials in the workspace's secrets or the deployment's environment variables, and tell them to read an export file before committing it. +- [The check flags a string that holds no secret, such as an example token in a message template] → Export names the path and the rule, and the user passes `--not-a-secret` with that path after reading the string. +- [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. +- [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. +- [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer records the answer in this design. + +## Resolved Questions + +None. + +## Open Questions + +None. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md new file mode 100644 index 000000000..a1b2e5561 --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -0,0 +1,45 @@ +# Export, import, compare and verify prompts and assistants + +Issue: [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) + +## Why + +Moving an application to a new LangSmith workspace and a new deployment means recreating its prompts and its assistants there. Today a user does that by hand in the web interface, or with `langstar prompt push`, which builds a single-message prompt and drops the model, the system message and any placeholders. Nothing checks afterwards that the copy matches the original or that it can still run. + +A prompt belongs to a workspace and an assistant belongs to a deployment, but both need the same four operations: write the definition to a file that can live in Git, recreate it somewhere else, compare a copy with that file, and check that the copy has what it needs to run. + +## What Changes + +- `langstar prompt export ` writes one JSON file holding a saved prompt commit as LangSmith returns it with the model included: message templates, input variables, output schema, structured output method, model binding and commit description, plus the repo's description, readme, tags and visibility, the commit tags, and where it came from. +- `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt, updates its repo details and creates, moves or deletes its commit tags when the user passes `--update`. An import that stopped before it pushed the first commit finishes on the next run, and one that stopped before it created the commit tags finishes with `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. +- `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's class, such as `ChatOpenAI`. +- `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. +- `prompt export` and `assistant export` write no file when the prompt or assistant holds a credential. Before writing, export checks every string for a key name such as `api_key`, `token`, `password`, `secret` or `Authorization`, and for a value shaped like a bearer token, a private key, a URL or connection string with a password, a JSON Web Token or a provider's API key. Export names the path of each flagged string and exits with status 1. A user who has read a flagged string and found it harmless passes `--not-a-secret ` to write it. Compare, verify and import print `` in place of the same strings. +- Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export. When a user imports the same file again, langstar reads that file to find the assistant it created earlier instead of creating another one, and to tell whether someone pushed a commit to the prompt since the last import. +- Import runs the dependency checks of `verify` before it writes, and writes nothing when one fails. +- `compare`, `verify` and `import` share one exit status scheme, 0 for a match, 3 for a difference or a failed check and 1 for any other error, and print a JSON report with `-f json`. Status 2 stays clap's usage error. + +## Capabilities + +### New Capabilities + +- `prompt`: the `langstar prompt` command group. This change adds its export, import, compare and verify requirements; the existing prompt subcommands get their requirements when [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728) lands. + +### Modified Capabilities + +- `assistant`: adds export, import, compare and verify requirements to `langstar assistant`. + +## Impact + +- SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. Developers commit these fields as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer, so the Prepare Release workflow makes the next release a minor version. [design.md's decision on the SDK release](design.md#developers-commit-the-new-sdk-fields-as--featsdk-so-the-next-release-is-a-minor-version) links the release docs that ask for this. +- CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. +- Tests that copy between environments need a second workspace, a second deployment and one model provider key for the structured output check, and the structured output check skips without the key. Developers run the tests in the single existing workspace and test deployment while they build the commands, and in a second workspace and a second deployment before anyone archives the change, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for a copy between environments. Once the second workspace and deployment exist, langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. +- Docs: a new usage page on moving prompts and assistants between environments. + +## Citations + +- [My/Principle/Declarative over Imperative](https://github.com/codekiln/logseq-encode-garden/blob/main/pages/My___Principle___Declarative%20over%20Imperative.md) +- [My/Principle/Explicit is better than Implicit](https://github.com/codekiln/logseq-encode-garden/blob/main/pages/My___Principle___Explicit%20is%20better%20than%20Implicit.md) +- [My/Principle/Simplify/Create Uniform Interfaces](https://github.com/codekiln/logseq-encode-garden/blob/main/pages/My___Principle___Simplify___Create%20Uniform%20Interfaces.md) +- [My/Principle/Make the Right Thing Easy and the Wrong Thing Hard](https://github.com/codekiln/logseq-encode-garden/blob/main/pages/My___Principle___Make%20the%20Right%20Thing%20Easy%20and%20the%20Wrong%20Thing%20Hard.md) +- [My/Principle/Simplify/Prefer Standards and Defaults](https://github.com/codekiln/logseq-encode-garden/blob/main/pages/My___Principle___Simplify___Prefer%20Standards%20and%20Defaults.md) diff --git a/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md b/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md new file mode 100644 index 000000000..8e3c07697 --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md @@ -0,0 +1,13 @@ +# LangGraph Python SDK assistants client + +Read at [langgraph 8a77332d](https://github.com/langchain-ai/langgraph/tree/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8). + +[`create`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L314) posts `graph_id`, `config`, `context`, `metadata`, `assistant_id`, `if_exists` (`raise` or `do_nothing`), `name` and `description` to `/assistants`. + +[`update`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L385) patches `graph_id`, `config`, `context`, `metadata`, `name` and `description`, and the server merges `metadata` into the existing metadata. [`get_versions`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L656) lists the versions updates create. + +[`AssistantBase`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/schema.py#L246) returns `assistant_id`, `graph_id`, `config`, `context`, `created_at`, `metadata`, `version`, `name` and `description`. `context` sits beside `config.configurable`; newer graphs read it and older ones read `configurable`. + +There is no endpoint that lists graphs. [`search`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L528) filters by `graph_id`, and every graph has a default assistant with `metadata.created_by` set to `system`, as the [`get` docstring](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L81) shows. + +What langstar takes from it: the fields an assistant export carries, including `context`, and the graph check through search. diff --git a/openspec/changes/export-import-prompts-and-assistants/references/langsmith-cli.md b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-cli.md new file mode 100644 index 000000000..27873927b --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-cli.md @@ -0,0 +1,17 @@ +# LangSmith CLI (Go) + +Read at [langsmith-cli 26135038](https://github.com/langchain-ai/langsmith-cli/tree/26135038671155f05ddd725703de7bdd43c7aa9f). + +[`prompt pull`](https://github.com/langchain-ai/langsmith-cli/blob/26135038671155f05ddd725703de7bdd43c7aa9f/internal/cmd/prompt.go#L301) takes `OWNER/REPO[:COMMIT_OR_TAG]` and writes this envelope to stdout or `--output`: + +```json +{"owner": "...", "repo": "...", "commit_hash": "...", "description": "...", "manifest": {}} +``` + +It reads the commit with empty `CommitGetParams`, so the model is never requested. + +[`prompt push`](https://github.com/langchain-ai/langsmith-cli/blob/26135038671155f05ddd725703de7bdd43c7aa9f/internal/cmd/prompt.go#L359) reads the bare manifest from `--file` or stdin and sends it as `manifest`, with optional `--parent-commit` and `--description`. It does not create the repo and has no tag flag. A pull envelope fed to push is sent as the manifest itself. + +[`resolveClientOptions`](https://github.com/langchain-ai/langsmith-cli/blob/26135038671155f05ddd725703de7bdd43c7aa9f/internal/cmd/root.go#L161) picks one workspace per run from `--workspace`, `LANGSMITH_WORKSPACE_ID`, `LANGSMITH_TENANT_ID` or the profile, highest first. + +What langstar takes from it: a file between a read run and a write run, a `:tag` or `--commit` to choose the commit, and output to stdout or a file. diff --git a/openspec/changes/export-import-prompts-and-assistants/references/langsmith-go.md b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-go.md new file mode 100644 index 000000000..c8b7c4707 --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-go.md @@ -0,0 +1,13 @@ +# LangSmith Go SDK + +Read at [langsmith-go 59bb42c3](https://github.com/langchain-ai/langsmith-go/tree/59bb42c30052067fe488f9bb8370df9037c715ef). + +[`CommitGetParams`](https://github.com/langchain-ai/langsmith-go/blob/59bb42c30052067fe488f9bb8370df9037c715ef/commit.go#L337) takes `include`, a comma-separated list of `model` and `is_draft`, and marks the boolean `include_model` deprecated. + +[`CommitGetResponse`](https://github.com/langchain-ai/langsmith-go/blob/59bb42c30052067fe488f9bb8370df9037c715ef/commit.go#L262) returns `commit_hash`, `description`, `examples`, `is_draft`, `manifest`, `model_config` and `model_provider`. + +[`CommitNewParams`](https://github.com/langchain-ai/langsmith-go/blob/59bb42c30052067fe488f9bb8370df9037c715ef/commit.go#L324) sends `manifest`, `parent_commit`, `description` and `skip_webhooks`. There is no field for the model outside the manifest. + +[`RepoNewParams`](https://github.com/langchain-ai/langsmith-go/blob/59bb42c30052067fe488f9bb8370df9037c715ef/repo.go#L310) requires `repo_handle` and `is_public` and takes `description`, `readme` and `tags`. + +What langstar takes from it: `include=model` on export, and the repo fields an export carries. diff --git a/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md new file mode 100644 index 000000000..78e10390f --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md @@ -0,0 +1,13 @@ +# LangSmith Python SDK + +Read at [langsmith-sdk 47c553ca](https://github.com/langchain-ai/langsmith-sdk/tree/47c553caf08c1989e2be3ff1886da3eb801f709f). + +[`pull_prompt_commit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10107) adds `?include_model=true` when asked. The result, [`PromptCommit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/schemas.py#L1035), holds the manifest plus `model_config` and `model_provider`. + +With the model included, the manifest in the [integration fixture `prompt_with_model`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/tests/integration_tests/test_prompts.py#L48) is a constructor with the `id` `["langsmith", "playground", "PromptPlayground"]`, which LangChain loads as a `RunnableSequence`. Its `first` step is the `ChatPromptTemplate` or `StructuredPrompt` (with `schema_`), and its `last` step is a `RunnableBinding` around the chat model constructor and its settings. + +The chat model refers to its API key as a secret reference, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`. [`_manifest_has_secrets`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L434) finds these by matching objects whose keys are exactly `lc`, `type` and `id`. [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228) defaults `secrets_from_env` to `False`, and loading a manifest that holds a secret reference raises unless the caller passes a `secrets` map or sets `secrets_from_env=True`. + +[`create_commit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L9915) resolves a missing parent to the latest commit, posts `{parent_commit, manifest, description}`, then posts each tag to `/repos/{owner}/{repo}/tags`. [`push_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10309) creates the repo first when it does not exist. + +What langstar takes from it: the shape of a manifest with its model, how to find secret names, and the order of create repo, commit with parent, then tags. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md new file mode 100644 index 000000000..fca9a3528 --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -0,0 +1,443 @@ +## ADDED Requirements + +### Requirement: Export an assistant to a file + +`langstar assistant export --deployment ` SHALL write one JSON file holding the assistant's name, description, graph ID, configuration, context and metadata, and a `source` object naming the deployment, assistant ID and version. `--output ` SHALL choose the file; without it stdout SHALL hold only the file, and messages go to stderr. Exporting the same assistant version twice SHALL produce identical files. + +#### Scenario: Export an assistant + +- **WHEN** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** `support-bot.json` holds the assistant's name, graph ID, configuration, context and metadata +- **AND** `source.assistant_id` holds the source assistant's ID + +#### Scenario: Assistant not found + +- **WHEN** the deployment has no assistant with that ID +- **THEN** langstar exits with status 1 and an error naming the assistant ID + +#### Scenario: Export to stdout + +- **WHEN** a user runs `langstar assistant export --deployment my-agent` without `--output` +- **THEN** stdout holds only the export file's JSON document +- **AND** any progress message goes to stderr + +### Requirement: Refuse to export a graph's default assistant + +`langstar assistant export` SHALL refuse a graph's default assistant, which every deployment creates for each graph with the same ID. + +#### Scenario: Assistant is a graph's default assistant + +- **WHEN** the assistant is the default assistant the deployment created for its graph, with `created_by` set to `system` in its metadata +- **THEN** langstar writes no file, says that every deployment creates that assistant, and exits with status 1 + +### Requirement: Refuse to export an assistant that holds a credential + +Before it writes, `langstar assistant export` SHALL run the credential check over every string at any depth of the `assistant` object, skipping secret references and empty strings. When the check flags a string, export SHALL write no file and nothing to stdout, list the JSON Pointer path and rule of each flagged string, without the string or any part of it, and exit with status 1. Export SHALL NOT write a file with `` in place of a flagged string. + +#### Scenario: Configuration holds a literal API key + +- **WHEN** the assistant's `config.configurable.OPENAI_API_KEY` holds a string +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** langstar writes no file, names `/config/configurable/OPENAI_API_KEY` without the string and exits with status 1 + +#### Scenario: Configuration reads its key from a secret + +- **WHEN** the assistant's `config.configurable.openai_api_key` holds a secret reference to `OPENAI_API_KEY` +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** langstar writes the file with the secret reference and exits with status 0 + +### Requirement: Flag an assistant string by the name of its key + +The credential check SHALL flag a string when the key that holds it, or the array it is in, has a name whose words, split at `_`, `-`, `.`, spaces and lower-to-upper case changes and compared without regard to case, include `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or `key` or `keys` right after `api`, `access`, `private`, `secret`, `signing` or `account`. + +#### Scenario: Context holds an access token + +- **WHEN** the assistant's `context.github_access_token` holds a string +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** langstar writes no file, names `/context/github_access_token` without the string and exits with status 1 + +#### Scenario: Configuration holds a database password + +- **WHEN** the assistant's `config.configurable.db.password` holds a string +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** langstar writes no file, names `/config/configurable/db/password` without the string and exits with status 1 + +#### Scenario: Configuration sends an authorization header + +- **WHEN** the assistant's `config.configurable.tool_headers.Authorization` holds `Bearer ` and a token +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** langstar writes no file, names `/config/configurable/tool_headers/Authorization` without the string and exits with status 1 + +### Requirement: Flag an assistant string by its shape under any key + +The credential check SHALL flag a string, under any key, that holds a bearer token, a `Basic ` authorization value, a PEM private key, a URL with a password in its user info, a connection string with a non-empty `Password=`, `Pwd=`, `AccountKey=`, `SharedAccessKey=` or `SharedAccessSignature=`, a JSON Web Token, or a token that starts with a provider prefix the design lists: `sk-`, `lsv2_`, `ghp_`, `gho_`, `ghs_`, `github_pat_`, `xoxb-`, `xoxp-`, `xoxa-`, `AKIA`, `ASIA`, `AIza` or `hf_`. + +#### Scenario: Configuration holds a connection string with a password + +- **WHEN** the assistant's `config.configurable.checkpoint_uri` holds `postgresql://app:@db.example.com/agent` +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** langstar writes no file, names `/config/configurable/checkpoint_uri` as a URL with a password and exits with status 1 + +#### Scenario: Metadata holds a private key + +- **WHEN** the assistant's `metadata.deploy_note` holds a PEM block that begins `-----BEGIN OPENSSH PRIVATE KEY-----` +- **AND** a user runs `langstar assistant export --deployment my-agent` +- **THEN** stdout is empty, langstar names `/metadata/deploy_note` as a private key and exits with status 1 + +### Requirement: Write each flagged assistant string the user names with --not-a-secret + +`langstar assistant export` SHALL take `--not-a-secret ` any number of times. For each pointer at a string the credential check flags, export SHALL write that string and print a warning to stderr naming the pointer. A flagged string at any path not named SHALL still make export refuse. When a pointer names a string the check lets through, export SHALL write nothing, name the pointer and exit with status 1. Export SHALL run the check on every run. + +#### Scenario: User writes a token budget setting with --not-a-secret + +- **WHEN** the assistant's `config.configurable.token_budget_policy` holds the string `strict` +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json --not-a-secret /config/configurable/token_budget_policy` +- **THEN** langstar writes the file with `strict` at that path, prints a warning naming the path to stderr and exits with status 0 + +### Requirement: Record the prompts an assistant refers to + +`langstar assistant export` SHALL accept `--prompt-ref `, repeatable, naming a string in the assistant's configuration or context that holds a prompt name of the form `[/][:]`. A name without an owner SHALL mean the current workspace's prompt, and a name without a tag SHALL mean the prompt's latest commit. The file SHALL list each pointer and the prompt name it holds under `requires.prompts`. A pointer that resolves to no string SHALL be an error. + +#### Scenario: Assistant names its system prompt + +- **WHEN** a user runs `langstar assistant export --deployment my-agent --prompt-ref /config/configurable/system_prompt` +- **AND** that value is `support-system:prod` +- **THEN** `requires.prompts` holds the pointer and `support-system:prod` + +#### Scenario: Assistant names a prompt without a tag + +- **WHEN** a user runs `langstar assistant export --deployment my-agent --prompt-ref /config/configurable/system_prompt` +- **AND** that value is `support-system` +- **THEN** `requires.prompts` holds the pointer and `support-system` +- **AND** `langstar assistant verify` checks the latest commit of the workspace's `support-system` prompt + +#### Scenario: Pointer names nothing + +- **WHEN** the pointer resolves to no value in the assistant +- **THEN** langstar exits with status 1 and an error naming the pointer + +### Requirement: Read the owner `-` as the current workspace + +A prompt name under `requires.prompts` with the owner `-`, such as `-/support-system:prod`, SHALL mean the current workspace's prompt, as a name without an owner does. + +#### Scenario: Assistant names a private prompt with the owner `-` + +- **WHEN** a user runs `langstar assistant export --deployment my-agent --prompt-ref /config/configurable/system_prompt` +- **AND** that value is `-/support-system:prod` +- **THEN** `requires.prompts` holds the pointer and `-/support-system:prod` +- **AND** `langstar assistant verify` checks the `prod` tag of the workspace's `support-system` prompt + +### Requirement: Import an assistant into a deployment + +With no created assistant ID recorded for the deployment and no `--assistant-id`, `langstar assistant import --deployment ` SHALL choose a new assistant ID, write it as pending in the destination record, then create the assistant with that `assistant_id`, `if_exists: do_nothing` and the file's name, description, graph ID, settings and metadata, mark the entry created, and print the ID. A pending entry SHALL make the next import repeat the create. + +#### Scenario: First import + +- **WHEN** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **AND** the destination record has no entry for `my-agent-prod` +- **THEN** langstar creates the assistant and prints its new ID +- **AND** `support-bot.destinations.json` holds the ID of the `my-agent-prod` deployment, with the new assistant ID under it, marked created + +#### Scenario: Import stopped after the create and before marking the ID created + +- **WHEN** the destination record holds a pending assistant ID for `my-agent-prod` +- **AND** the deployment already has an assistant with that ID +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** langstar sends the create with that ID and `if_exists` set to `do_nothing`, so the deployment still holds one assistant from the file +- **AND** the destination record marks that ID created + +### Requirement: Send an assistant's settings in one field + +When the file's `assistant` object holds both `config.configurable` and `context` with the same values, `langstar assistant import` SHALL send `config` with `configurable` and SHALL NOT send `context`. When it holds only one of them, import SHALL send that one. When it holds both and they differ, import SHALL write nothing, leave the destination record unchanged, name `/assistant/config/configurable` and `/assistant/context` without their values, and exit with status 1. + +#### Scenario: Both fields hold the same settings + +- **WHEN** the file's `config.configurable` and `context` both hold `{"system_prompt": "support-system:prod"}` +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** the create request holds `config.configurable` and no `context` +- **AND** the read-back's `context` holds `{"system_prompt": "support-system:prod"}`, so import reports no difference + +#### Scenario: Only context holds the settings + +- **WHEN** the file's `context` holds `{"system_prompt": "support-system:prod"}` and its `config` has no `configurable` +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** the create request holds `context` and no `config.configurable` + +#### Scenario: Only config.configurable holds the settings + +- **WHEN** the file's `config.configurable` holds `{"system_prompt": "support-system:prod"}` and the file has no `context` +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** the create request holds `config.configurable` and no `context` + +#### Scenario: The two fields differ + +- **WHEN** the file's `config.configurable.system_prompt` is `support-system:prod` and its `context.system_prompt` is `support-system:staging` +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** langstar creates no assistant, leaves `support-bot.destinations.json` as it was, names `/assistant/config/configurable` and `/assistant/context` without their values and exits with status 1 + +### Requirement: Read an assistant's settings from whichever field holds them + +When the file holds settings in only one of `config.configurable` and `context`, `langstar assistant compare`, and the read-back import makes, SHALL compare the file's settings with the destination's settings from whichever field the destination holds, and SHALL NOT report the field the file lacks as a difference. + +#### Scenario: Import of a context-only file reports no difference + +- **WHEN** the file's `context` holds `{"system_prompt": "support-system:prod"}` and its `config` has no `configurable` +- **AND** the read-back's `config.configurable` holds `{"system_prompt": "support-system:prod"}` +- **THEN** import reports no difference and exits with status 0 + +#### Scenario: Import of a config-only file reports no difference + +- **WHEN** the file's `config.configurable` holds `{"system_prompt": "support-system:prod"}` and the file has no `context` +- **AND** the read-back's `context` holds `{"system_prompt": "support-system:prod"}` +- **THEN** import reports no difference and exits with status 0 + +#### Scenario: A second import of a config-only file creates nothing + +- **WHEN** the file's `config.configurable` holds `{"system_prompt": "support-system:prod"}` and the file has no `context` +- **AND** the destination record names the assistant an earlier import created, whose `config.configurable` and `context` both hold `{"system_prompt": "support-system:prod"}` +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** langstar creates no assistant, reports no difference and exits with status 0 + +#### Scenario: Compare of a config-only file against a destination holding both fields + +- **WHEN** the file's `config.configurable` holds `{"system_prompt": "support-system:prod"}` and the file has no `context` +- **AND** the destination assistant's `config.configurable` and `context` both hold `{"system_prompt": "support-system:prod"}` +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` +- **THEN** langstar reports no difference and exits with status 0 + +### Requirement: Keep the assistant destination record at the path the user names + +`langstar assistant import`, `assistant compare` and `assistant verify` SHALL read and write the destination record at `.destinations.json` next to the export file, unless `--record ` names another file. + +#### Scenario: Import writes the record to another path + +- **WHEN** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod --record records/prod.json` +- **THEN** langstar writes the new assistant ID to `records/prod.json` and writes no `support-bot.destinations.json` + +### Requirement: Replace the destination record in one step + +`langstar assistant import` SHALL write each new destination record to a temporary file in the record's directory and rename it over the old record, so that a langstar that stops during a write leaves the earlier record whole, with its pending assistant IDs. + +#### Scenario: Import stopped while marking the ID created + +- **WHEN** langstar stops while it writes the record that marks a new assistant ID created +- **THEN** the destination record still holds that ID marked pending +- **AND** the next `langstar assistant import support-bot.json --deployment my-agent-prod` sends the create with that ID and creates no second assistant + +### Requirement: Check the destination before importing an assistant + +Before it writes, `langstar assistant import` SHALL run the graph check and the prompt check of `assistant verify`, reading prompts from `--workspace-id` or the configured workspace. When a check fails it SHALL write nothing, name each missing graph and prompt, and exit with status 3. + +#### Scenario: Referenced prompt missing at import + +- **WHEN** the file lists `support-system:prod` under `requires.prompts` +- **AND** the workspace has no `support-system` prompt with a `prod` tag +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** langstar creates no assistant, reports `support-system:prod` as missing and exits with status 3 + +### Requirement: Find the earlier copy instead of creating another + +When the destination record holds a created assistant ID for the deployment, `langstar assistant import` SHALL compare the file with that assistant and create nothing. It SHALL exit with status 0 when they match, and list the differences and exit with status 3 when they differ, unless the user passes `--update`. When the recorded assistant no longer exists, it SHALL exit with status 1 and say so. + +#### Scenario: Import the same file again + +- **WHEN** a user imports `support-bot.json` into `my-agent-prod` a second time with no changes +- **THEN** langstar creates no assistant and reports that the assistant is unchanged + +#### Scenario: Recorded assistant was deleted + +- **WHEN** the recorded assistant ID no longer exists on the deployment +- **THEN** langstar exits with status 1 and names the missing assistant ID + +### Requirement: Update an existing assistant only on request + +`langstar assistant import --update` SHALL update the recorded assistant, or the assistant named by `--assistant-id `, with the file's name, description, graph ID, settings and metadata, sending the settings as "Send an assistant's settings in one field" requires. `--assistant-id` SHALL take precedence over the destination record. + +#### Scenario: Update the recorded assistant + +- **WHEN** the file's configuration differs from the recorded assistant +- **AND** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --update` +- **THEN** langstar updates that assistant and creates no other + +#### Scenario: Update an assistant chosen by ID + +- **WHEN** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --update --assistant-id ` +- **THEN** langstar updates that assistant and records its ID for `my-agent-prod` + +### Requirement: Compare with the assistant named by --assistant-id when --update is absent + +Without `--update`, `langstar assistant import --assistant-id ` SHALL compare the file with that assistant, as `assistant compare` does, and create and change nothing. It SHALL exit with status 0 when they match, and list the differences and exit with status 3 when they differ. + +#### Scenario: Named assistant differs from the file + +- **WHEN** the assistant `` has configuration that differs from the file +- **AND** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --assistant-id ` +- **THEN** langstar creates no assistant, changes none, lists the differences and exits with status 3 + +### Requirement: Import with --update asks the Agent Server to remove the metadata and description the file lacks + +`langstar assistant import --update` SHALL ask the Agent Server to remove each metadata key the destination assistant has and the file lacks, and to clear a description the file lacks, then read the assistant back. When such a key or field remains after the update, it SHALL list it as a difference and exit with status 3. + +#### Scenario: Remove a metadata key the file lacks + +- **WHEN** the recorded assistant's metadata is `{"kept": 1, "stale": 2}` +- **AND** the file's metadata is `{"kept": 1}` +- **AND** the Agent Server removes a metadata key that an update sends with a null value +- **AND** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --update` +- **THEN** the assistant's metadata read back from the deployment is `{"kept": 1}` +- **AND** langstar exits with status 0 + +#### Scenario: Deployment keeps a removed metadata key + +- **WHEN** the Agent Server keeps `stale` in the assistant's metadata after the update +- **THEN** langstar lists `/metadata/stale` as a difference and exits with status 3 + +### Requirement: Import with --update leaves the settings field it did not send alone + +`langstar assistant import --update` SHALL clear `context` only when the file holds settings in neither `config.configurable` nor `context`. + +#### Scenario: --update of a config-only file leaves context alone + +- **WHEN** the file's `config.configurable` holds `{"system_prompt": "support-system:prod"}` and the file has no `context` +- **AND** the recorded assistant's `context` holds `{"system_prompt": "support-system:staging"}` +- **AND** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --update` +- **THEN** the update request holds `config.configurable` and no `context` +- **AND** the assistant's settings read back from the deployment hold `support-system:prod` +- **AND** langstar exits with status 0 + +#### Scenario: Deployment keeps a description the file lacks + +- **WHEN** the recorded assistant has a description and the file has none +- **AND** the Agent Server keeps that description after the update +- **THEN** langstar lists `/description` as a difference and exits with status 3 + +### Requirement: Dry run an assistant import + +`langstar assistant import --dry-run` SHALL report whether it would create the assistant, update it, or leave it unchanged, list the differences, and change nothing on the deployment or in the destination record. + +#### Scenario: Dry run of a first import + +- **WHEN** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod --dry-run` +- **THEN** langstar reports that it would create the assistant and creates nothing + +### Requirement: Read the assistant back after import + +After writing, `langstar assistant import` SHALL record the assistant ID in the destination record, then read the assistant back from the deployment and compare it with the file. When they differ it SHALL list the differences and exit with status 3, and the record SHALL keep the ID so that the next import finds that assistant instead of creating another. + +#### Scenario: Deployment drops a configuration key + +- **WHEN** the assistant read back lacks a configuration key the file holds +- **THEN** langstar lists that key as a difference and exits with status 3 +- **AND** the destination record holds the new assistant ID, so importing the file again creates no assistant + +### Requirement: Compare an assistant with its export file + +`langstar assistant compare --deployment ` SHALL compare the file with the assistant recorded in the destination record (`.destinations.json` unless `--record ` names another), or the one named by `--assistant-id`, and list each difference by its JSON path with both values. It SHALL compare only the fields the export file holds, leaving out the assistant ID, version and timestamps the server sets. + +#### Scenario: Metadata changed on the deployment + +- **WHEN** someone changed a metadata value on the deployment's assistant +- **THEN** langstar lists that metadata path with both values and exits with status 3 + +#### Scenario: No assistant to compare + +- **WHEN** the destination record has no entry for `my-agent-prod` +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` without `--assistant-id` +- **THEN** langstar exits with status 1 and an error that names the deployment and suggests `--assistant-id` + +#### Scenario: Record kept at another path + +- **WHEN** a user imported `support-bot.json` with `--record records/prod.json` +- **AND** runs `langstar assistant compare support-bot.json --deployment my-agent-prod --record records/prod.json` +- **THEN** langstar compares the file with the assistant that import recorded in `records/prod.json` + +### Requirement: Verify that an assistant can run on a deployment + +`langstar assistant verify --deployment ` SHALL do what `compare` does and also check that the deployment serves the file's graph ID and that the workspace, from `--workspace-id` or the configured workspace, has each prompt under `requires.prompts`. It SHALL name each missing graph and prompt and exit with status 3 when any check fails. + +#### Scenario: Graph missing on the destination + +- **WHEN** the destination deployment serves no graph with the file's graph ID +- **THEN** langstar reports the missing graph ID and exits with status 3 + +#### Scenario: Referenced prompt missing + +- **WHEN** the file lists `support-system:prod` under `requires.prompts` +- **AND** the workspace has no `support-system` prompt with a `prod` tag +- **THEN** langstar reports `support-system:prod` as missing and exits with status 3 + +### Requirement: Reject an export file of another kind or format version + +`assistant import`, `assistant compare` and `assistant verify` SHALL read a file only when its `kind` is `assistant` and its `format_version` is 1. For any other file they SHALL write nothing, exit with status 1 and name the field and the value they found. + +#### Scenario: Prompt file given to an assistant command + +- **WHEN** a user runs `langstar assistant import my-prompt.json --deployment my-agent-prod` +- **AND** `my-prompt.json` holds `"kind": "prompt"` +- **THEN** langstar creates no assistant, exits with status 1 and names `kind` and the value `prompt` + +### Requirement: Reject a file whose prompt reference no longer matches its assistant + +`assistant import`, `assistant compare` and `assistant verify` SHALL resolve each pointer under `requires.prompts` against the file's `assistant` object when they read the file. When a pointer resolves to no string, or to a string other than the prompt name recorded with it, they SHALL write nothing, exit with status 1 and name the pointer and both values, with `` in place of each value the credential check flags, as the difference report does. + +#### Scenario: Configuration edited after export + +- **WHEN** `requires.prompts` records `/config/configurable/system_prompt` with `support-system:prod` +- **AND** the file's `assistant.config.configurable.system_prompt` is `support-system:staging` +- **AND** the user runs `langstar assistant verify support-bot.json --deployment my-agent-prod` +- **THEN** langstar exits with status 1 and names the pointer, `support-system:prod` and `support-system:staging` + +#### Scenario: Edited prompt reference holds an API key + +- **WHEN** `requires.prompts` records `/config/configurable/system_prompt` with `support-system:prod` +- **AND** the file's `assistant.config.configurable.system_prompt` holds `sk-` followed by a 40-character token +- **AND** the user runs `langstar assistant verify support-bot.json --deployment my-agent-prod` +- **THEN** langstar exits with status 1, names the pointer and `support-system:prod`, and prints `` in place of the string + +### Requirement: Assistant compare, verify and import use the prompt commands' exit statuses and JSON report + +`assistant compare`, `assistant verify` and `assistant import` SHALL use the exit statuses and the JSON report shape of `prompt compare`, `prompt verify` and `prompt import`. + +#### Scenario: Machine-readable verify + +- **WHEN** a user runs `langstar -f json assistant verify support-bot.json --deployment my-agent-prod` +- **THEN** stdout holds one JSON object with `status`, `differences` and `checks` + +### Requirement: Redact a credential when reporting how an assistant differs + +When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in text or JSON, langstar SHALL print `` in place of each string the credential check from "Refuse to export an assistant that holds a credential" flags. It SHALL redact the string when the difference is at the string or at an object that contains it, and whether the string came from the destination or from the file. + +#### Scenario: Destination configuration holds a literal API key + +- **WHEN** someone updated the destination assistant so `config.configurable.openai_api_key` holds a string +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` +- **THEN** langstar reports `/assistant/config/configurable/openai_api_key` with `` in place of the string and exits with status 3 + +#### Scenario: Destination configuration holds a literal API key under an upper-case name + +- **WHEN** someone updated the destination assistant so `config.configurable.OPENAI_API_KEY` holds a string +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` +- **THEN** langstar reports `/assistant/config/configurable/OPENAI_API_KEY` with `` in place of the string and exits with status 3 + +#### Scenario: Destination context holds a connection string with a password + +- **WHEN** someone updated the destination assistant so `context.checkpoint_uri` holds `postgresql://app:@db.example.com/agent` +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod -f json` +- **THEN** the text and the JSON report print `` at `/assistant/context/checkpoint_uri`, neither holds the password, and langstar exits with status 3 + +#### Scenario: Destination configuration has a new object that holds a password + +- **WHEN** the file's configuration has no `db` key +- **AND** someone updated the destination assistant so `config.configurable.db` holds `{"user": "app", "password": ""}` +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` +- **THEN** langstar reports `/assistant/config/configurable/db` with `user` unchanged and `` under `password`, and exits with status 3 + +### Requirement: Print a secret reference in a destination assistant's configuration unchanged + +When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, langstar SHALL print a LangChain secret reference unchanged under any key, so the report names the secret. A secret reference looks like `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`. + +#### Scenario: Destination configuration reads its key from a secret + +- **WHEN** someone updated the destination assistant so `config.configurable.openai_api_key` refers to the secret `OPENAI_API_KEY` +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` +- **THEN** langstar reports `/assistant/config/configurable/openai_api_key` with the secret reference and `OPENAI_API_KEY` in it, and exits with status 3 diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md new file mode 100644 index 000000000..ab919f310 --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -0,0 +1,472 @@ +## Purpose + +`langstar prompt` manages the prompts saved in a LangSmith workspace, including writing a prompt to a file and recreating it in another workspace. + +## ADDED Requirements + +### Requirement: Export a prompt commit to a file + +`langstar prompt export ` SHALL write one JSON file holding the repo details (handle, description, readme, tags, visibility), the commit's description, its manifest as LangSmith returns it with the model included, its model configuration and provider, the tags that point at it, and a `source` object naming the workspace, handle and commit hash. `--output ` SHALL choose the file; without it stdout SHALL hold only the file, and messages go to stderr. + +#### Scenario: Export the latest commit + +- **WHEN** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes `my-prompt.json` with the manifest of the latest commit, including the model binding +- **AND** `source.commit_hash` holds that commit's hash +- **AND** `commit.description` holds that commit's description + +#### Scenario: Export a structured prompt + +- **WHEN** the prompt is a structured prompt with a nested JSON Schema and the `function_calling` method +- **THEN** the exported manifest holds the schema and the method unchanged + +#### Scenario: Prompt not found + +- **WHEN** no prompt in the workspace has the handle +- **THEN** langstar exits with status 1 and an error naming the handle + +#### Scenario: Export a private prompt + +- **WHEN** the workspace's `my-prompt` is private +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` or `langstar prompt export -/my-prompt --output my-prompt.json` +- **THEN** langstar reads the commit as `-/my-prompt` and writes `my-prompt.json` + +#### Scenario: Export to stdout + +- **WHEN** a user runs `langstar prompt export my-prompt` without `--output` +- **THEN** stdout holds only the export file's JSON document +- **AND** any progress message goes to stderr + +### Requirement: Export a chosen commit or tag + +`langstar prompt export` SHALL accept `--commit ` and export that commit. Without the flag it SHALL export the latest commit. + +#### Scenario: Export a tagged commit + +- **WHEN** a user runs `langstar prompt export my-prompt --commit prod --output my-prompt.json` +- **THEN** the file holds the commit the `prod` tag points at + +### Requirement: Write the same file for the same commit + +Exporting the same commit twice SHALL produce identical files when nobody changed the repo details or the tags that point at the commit in between, since those change without a new commit. The file SHALL hold no export time, and langstar SHALL write object keys in a fixed order and sort `commit.tags`, `repo.tags` and `requires.secrets` by name. Compare SHALL sort the destination's tags the same way before comparing. + +#### Scenario: Export twice + +- **WHEN** a user exports the same commit twice to two files +- **AND** nobody changed the repo details or the commit's tags between the two exports +- **THEN** the two files are identical + +### Requirement: List the credentials a prompt needs without their values + +The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_class`, the class name that ends the chat model constructor's `id`, such as `ChatOpenAI`. It SHALL hold no value of a secret the manifest refers to. Import and verify SHALL read both again from the file's manifest, and when either differs from `requires.secrets` or `requires.model_class`, write nothing, name the field and exit with status 1. + +#### Scenario: Model reads an API key secret + +- **WHEN** the prompt's model refers to the secret `OPENAI_API_KEY` +- **THEN** `requires.secrets` holds `OPENAI_API_KEY` +- **AND** no value of that secret appears in the file + +#### Scenario: Manifest edited without its requires list + +- **WHEN** someone edits the file's manifest so its model refers to the secret `ANTHROPIC_API_KEY`, and leaves `requires.secrets` holding only `OPENAI_API_KEY` +- **AND** a user runs `langstar prompt import my-prompt.json --workspace-id ` +- **THEN** langstar creates no prompt, names `requires.secrets` and exits with status 1 + +### Requirement: Refuse to export a prompt that holds a credential + +Before it writes, `langstar prompt export` SHALL run the credential check over every string in the `repo` and `commit` objects it would write, skipping LangChain secret references and empty strings. When the check flags a string, export SHALL write no file and nothing to stdout, list the JSON Pointer path of each flagged string and the rule that flagged it, without the string or any part of it, and exit with status 1. Export SHALL NOT write a file with `` in place of a flagged string. + +#### Scenario: Model holds a literal API key + +- **WHEN** the prompt's `ChatOpenAI` constructor holds a string under `openai_api_key` +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes no file, names the path of `openai_api_key` without its value and exits with status 1 + +#### Scenario: Model holds a literal API key under an upper-case name + +- **WHEN** the prompt's `ChatOpenAI` constructor holds a string under `OPENAI_API_KEY` +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes no file, names the path of `OPENAI_API_KEY` without its value and exits with status 1 + +#### Scenario: Readme holds a private key + +- **WHEN** the repo's readme holds a PEM block that begins `-----BEGIN PRIVATE KEY-----` +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes no file, names `/repo/readme` and exits with status 1 + +#### Scenario: Model reads its key from a secret + +- **WHEN** the prompt's chat model refers to the secret `OPENAI_API_KEY` under `openai_api_key` +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes the file with the secret reference and exits with status 0 + +### Requirement: Flag a prompt string by the name of its key + +The credential check SHALL flag a string when the key that holds it, or the array it is in, has a name whose words, split at `_`, `-`, `.`, spaces and lower-to-upper case changes and compared without regard to case, include `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or `key` or `keys` right after `api`, `access`, `private`, `secret`, `signing` or `account`. + +#### Scenario: Model sends an authorization header + +- **WHEN** the prompt's chat model holds `default_headers` with `Authorization` set to `Bearer ` and a token +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes no file, names `/commit/manifest/kwargs/last/kwargs/default_headers/Authorization` without its value and exits with status 1 + +#### Scenario: Model sets a token limit + +- **WHEN** the prompt's chat model holds the number 1024 under `max_tokens` +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes the file with `max_tokens` unchanged and exits with status 0 + +### Requirement: Flag a prompt string by its shape under any key + +The credential check SHALL flag a string, under any key, that holds a bearer token, a `Basic ` authorization value, a PEM private key, a URL with a password in its user info, a connection string with a non-empty `Password=`, `Pwd=`, `AccountKey=`, `SharedAccessKey=` or `SharedAccessSignature=`, a JSON Web Token, or a token that starts with a provider prefix the design lists: `sk-`, `lsv2_`, `ghp_`, `gho_`, `ghs_`, `github_pat_`, `xoxb-`, `xoxp-`, `xoxa-`, `AKIA`, `ASIA`, `AIza` or `hf_`. + +#### Scenario: Model setting holds a token under a neutral name + +- **WHEN** the prompt's chat model holds `sk-` followed by a 40-character token under `model_kwargs.extra` +- **AND** a user runs `langstar prompt export my-prompt` +- **THEN** stdout is empty, langstar names the path of `extra` with the rule that flagged it and exits with status 1 + +#### Scenario: Prompt metadata holds a connection string with a password + +- **WHEN** the manifest's `metadata` holds `postgresql://app:@db.example.com/prod` under `source_db` +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes no file, names the path of `source_db` as a URL with a password and exits with status 1 + +### Requirement: Write each flagged prompt string the user names with --not-a-secret + +`langstar prompt export` SHALL take `--not-a-secret ` any number of times. For each pointer at a string the credential check flags, export SHALL write that string and print a warning to stderr naming the pointer. A flagged string at any path not named SHALL still make export refuse. When a pointer names a string the check lets through, export SHALL write nothing, name the pointer and exit with status 1. Export SHALL run the check on every run. + +#### Scenario: Message template shows an example token + +- **WHEN** a message template holds `Authorization: Bearer ` followed by an example token +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json --not-a-secret ` +- **THEN** langstar writes the file with the template unchanged, prints a warning naming the path to stderr and exits with status 0 + +#### Scenario: User names an unflagged string with --not-a-secret + +- **WHEN** a user runs `langstar prompt export my-prompt --output my-prompt.json --not-a-secret /repo/description` +- **AND** the check flags nothing at `/repo/description` +- **THEN** langstar writes no file, names `/repo/description` and exits with status 1 + +### Requirement: Export, import and verify a prompt without a model + +When the manifest holds no chat model constructor, `langstar prompt export` SHALL write `requires.model_class` as null. Import and verify SHALL accept null when the manifest they read again holds no chat model, skip the saved model configuration check, report no note for it, and still run the secret check. + +#### Scenario: Prompt saved without a model + +- **WHEN** a user exports a prompt saved without a model and imports the file with `langstar prompt import my-prompt.json --workspace-id ` +- **THEN** the file holds null under `requires.model_class` +- **AND** langstar creates the prompt without a saved model configuration check and exits with status 0 +- **AND** `langstar prompt verify my-prompt.json` against the destination reports no saved model configuration note and exits with status 0 + +### Requirement: Import requires --workspace-id + +`langstar prompt import ` SHALL require `--workspace-id`. It SHALL refuse to run when the flag is missing, even when `LANGSMITH_WORKSPACE_ID` is set, and exit with status 1 and an error naming the flag. + +#### Scenario: Missing destination workspace + +- **WHEN** a user runs `langstar prompt import my-prompt.json` without `--workspace-id` +- **AND** `LANGSMITH_WORKSPACE_ID` is set +- **THEN** langstar exits with status 1 and an error naming `--workspace-id` + +### Requirement: Check the destination before importing a prompt + +Before it writes, `langstar prompt import` SHALL run the secret check and the saved model configuration check of `prompt verify` against the destination workspace. When the workspace lacks a secret in `requires.secrets` it SHALL write nothing, name each missing secret and exit with status 3. A missing saved model configuration SHALL be a note, as in `prompt verify`. + +#### Scenario: Secret missing at import + +- **WHEN** the file lists `OPENAI_API_KEY` under `requires.secrets` +- **AND** the destination workspace has no secret named `OPENAI_API_KEY` +- **AND** a user runs `langstar prompt import my-prompt.json --workspace-id ` +- **THEN** langstar creates no prompt, reports `OPENAI_API_KEY` as missing and exits with status 3 + +### Requirement: Import a prompt that does not exist yet + +When the destination workspace has no prompt with the file's handle, `langstar prompt import` SHALL create the prompt with the file's repo details, push the file's manifest and commit description as its first commit, and create the file's commit tags on that commit. + +#### Scenario: First import + +- **WHEN** a user runs `langstar prompt import my-prompt.json --workspace-id ` +- **AND** the destination has no prompt named `my-prompt` +- **THEN** langstar creates `my-prompt` in the destination with the file's manifest, model and tags +- **AND** prints the new commit hash + +### Requirement: Finish an import that stopped before its first commit + +When the destination workspace has a prompt with the file's handle and no commit, `langstar prompt import` SHALL push the file's manifest and commit description as the first commit and create the file's commit tags, without `--update`. Differing repo details SHALL still need `--update`: without it, import SHALL push the commit and tags, leave them unchanged, list them as differences and exit with status 3. "Keep a private prompt out of a public destination repo" covers a public one. + +#### Scenario: Earlier import created the repo and stopped + +- **WHEN** an earlier import created `my-prompt` in the destination and stopped before it pushed the commit +- **AND** the user runs `langstar prompt import my-prompt.json --workspace-id ` again +- **THEN** langstar pushes the file's manifest as the prompt's first commit and creates the file's tags +- **AND** exits with status 0 + +### Requirement: Keep a private prompt out of a public destination repo + +When the destination repo is public and the file's `repo.is_public` is false, `langstar prompt import` without `--update` SHALL push no commit and no tag, report `/repo/is_public` and exit with status 3. With `--update` it SHALL make the repo private before it pushes the commit. + +#### Scenario: Public destination repo and no --update + +- **WHEN** the destination holds a repo named `my-prompt` that is public and has no commit +- **AND** the file's `repo.is_public` is false +- **AND** a user runs `langstar prompt import my-prompt.json --workspace-id ` +- **THEN** langstar pushes no commit and no tag, reports `/repo/is_public` and exits with status 3 + +#### Scenario: Public destination repo with --update + +- **WHEN** the same user runs the same import with `--update` +- **THEN** langstar sets the repo's visibility to private before it pushes the commit +- **AND** the prompt holds the file's manifest and the destination repo is private + +### Requirement: Change an existing prompt only with --update + +When the destination prompt exists and its latest commit differs from the file, `langstar prompt import` SHALL change nothing and exit with status 3, listing the differences, unless the user passes `--update`. With `--update` it SHALL push the file's manifest and commit description as a new commit whose parent is the destination's latest commit, and move the file's tags to it. + +#### Scenario: Destination differs and --update is missing + +- **WHEN** the destination prompt's latest commit differs from the file +- **AND** the user runs `langstar prompt import` without `--update` +- **THEN** langstar pushes nothing, lists the differences and exits with status 3 + +#### Scenario: Update an existing prompt + +- **WHEN** the user runs the same import with `--update` +- **THEN** langstar adds a commit to the destination prompt holding the file's manifest + + +#### Scenario: Destination already matches + +- **WHEN** the destination prompt's latest commit and its commit tags match the file +- **THEN** langstar pushes nothing, reports that the prompt is unchanged and exits with status 0 + +### Requirement: Update the repo details only with --update + +When the destination repo's description, readme, tags or visibility differ from the file, `langstar prompt import` SHALL treat them as differences. With `--update` it SHALL set those fields to the file's values. When only those fields differ and the latest commit matches the file, it SHALL update the repo and add no commit. + +#### Scenario: Only the repo readme differs + +- **WHEN** the destination prompt's latest commit matches the file +- **AND** the destination repo's readme differs from the file's readme +- **AND** the user runs `langstar prompt import my-prompt.json --workspace-id --update` +- **THEN** langstar sets the destination repo's readme to the file's readme and adds no commit +- **AND** exits with status 0 + +### Requirement: Move commit tags only with --update + +`prompt compare` and `prompt import` SHALL list as a difference a file tag the destination lacks or points at another commit, and a tag on the destination's matching commit that the file lacks; import SHALL then exit with status 3. With `--update`, import SHALL create or move each file tag to that commit, delete each tag on it the file lacks, and add no commit when the latest commit already matches. + +#### Scenario: Destination tag points at an older commit + +- **WHEN** the destination prompt's latest commit matches the file +- **AND** the destination's `prod` tag points at an earlier commit +- **AND** the user runs `langstar prompt import my-prompt.json --workspace-id --update` +- **THEN** langstar moves `prod` to the latest commit and adds no commit +- **AND** exits with status 0 + +#### Scenario: Destination commit has a tag the file lacks + +- **WHEN** the destination prompt's latest commit matches the file +- **AND** that commit carries a `staging` tag that the file's `commit.tags` lacks +- **AND** the user runs `langstar prompt import my-prompt.json --workspace-id --update` +- **THEN** langstar deletes the destination's `staging` tag and adds no commit +- **AND** exits with status 0 + +### Requirement: Record the destination commit + +After import writes a commit, langstar SHALL record the destination workspace, handle and commit hash in a destination record file, `.destinations.json` next to the export unless `--record ` names another, before it reads the commit back. Each import SHALL add or replace only the entry for its own destination workspace. + +#### Scenario: Import into two workspaces + +- **WHEN** a user imports `my-prompt.json` into a staging workspace and then into a production workspace +- **THEN** `my-prompt.destinations.json` holds one entry for each workspace with its own commit hash +- **AND** `my-prompt.json` is unchanged + +### Requirement: Replace the prompt destination record in one step + +`langstar prompt import` SHALL write each new destination record to a temporary file in the record's directory and rename it over the old record, so that a langstar that stops during a write leaves the earlier record whole. + +#### Scenario: Import stopped while writing the record + +- **WHEN** `my-prompt.destinations.json` holds commit hashes for a staging and a production workspace +- **AND** langstar stops while it writes the record after an import into the production workspace +- **THEN** `my-prompt.destinations.json` still holds both workspaces' entries as they were before that import + +### Requirement: Warn when the destination changed since the last import + +When the destination record holds a commit for the destination workspace, and the destination's latest commit is not that commit and differs from the file, `langstar prompt import` SHALL say that someone changed the prompt after the last import, and list the differences. + +#### Scenario: Someone edited the destination prompt + +- **WHEN** a user edited the destination prompt in the web interface after the last import +- **AND** runs `langstar prompt import my-prompt.json --workspace-id ` +- **THEN** langstar reports that the destination changed since the last recorded import and exits with status 3 + +### Requirement: Record a matching commit that an earlier import pushed + +When the destination's latest commit matches the file, `langstar prompt import` without `--dry-run` SHALL write that commit's hash to the destination record under the destination workspace. When the commit tags match the file too, it SHALL report the prompt unchanged and exit with status 0. + +#### Scenario: Earlier import stopped before it wrote the record + +- **WHEN** an earlier import pushed a commit and stopped before it wrote that commit's hash to `my-prompt.destinations.json` +- **AND** the user runs `langstar prompt import my-prompt.json --workspace-id ` again +- **THEN** langstar reports that the prompt is unchanged, writes the latest commit's hash to the record and exits with status 0 + +#### Scenario: First import stopped before it wrote the record + +- **WHEN** the first import into a workspace pushed the prompt and stopped before it wrote `my-prompt.destinations.json` +- **AND** the user runs `langstar prompt import my-prompt.json --workspace-id ` again +- **THEN** langstar reports that the prompt is unchanged, writes the latest commit's hash to the record under that workspace and exits with status 0 + +### Requirement: Dry run an import + +`langstar prompt import --dry-run` SHALL report whether it would create the prompt, add a commit, or leave it unchanged, list the differences, and change nothing in the workspace or the destination record. + +#### Scenario: Dry run against a new workspace + +- **WHEN** a user runs `langstar prompt import my-prompt.json --workspace-id --dry-run` +- **AND** the destination has no prompt named `my-prompt` +- **THEN** langstar reports that it would create the prompt and creates nothing + +### Requirement: Read the prompt back after import + +After writing, `langstar prompt import` SHALL record the commit hash in the destination record, then read the destination commit back with the model included and compare it with the file as `langstar prompt compare` does. When they differ it SHALL list the differences and exit with status 3, and the record SHALL keep the hash, so that the next import reports a change only when someone pushed a commit after the one langstar wrote. + +#### Scenario: LangSmith drops a field + +- **WHEN** the destination commit read back after import lacks a field the file holds +- **THEN** langstar lists that field as a difference and exits with status 3 +- **AND** the destination record holds the new commit hash + +### Requirement: Redact a credential when reporting how a prompt differs + +When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in text or JSON, langstar SHALL print `` in place of each string the credential check from "Refuse to export a prompt that holds a credential" flags. It SHALL redact the string when the difference is at the string or at an object that contains it, and whether the string came from the destination or from the file. + +#### Scenario: Destination model holds a literal API key + +- **WHEN** someone saved the destination prompt with an `openai_api_key` written as a string +- **AND** a user runs `langstar prompt compare my-prompt.json` +- **THEN** langstar reports the setting's path with `` in place of the string and exits with status 3 + +#### Scenario: Destination adds a model with a literal API key + +- **WHEN** the file holds a prompt saved without a model +- **AND** someone saved the destination prompt with a model whose `openai_api_key` is written as a string +- **AND** a user runs `langstar prompt compare my-prompt.json` +- **THEN** langstar reports the model's path with `` in place of the string and exits with status 3 + +#### Scenario: Destination model sends a bearer token in a header + +- **WHEN** someone saved the destination prompt with a model whose `default_headers.Authorization` holds `Bearer ` and a token +- **AND** a user runs `langstar prompt compare my-prompt.json -f json` +- **THEN** the text and the JSON report print `` at that header's path, neither holds the token, and langstar exits with status 3 + +#### Scenario: Destination model holds a token under a neutral name + +- **WHEN** someone saved the destination prompt with a model whose `model_kwargs.extra` holds a string that starts with `lsv2_` +- **AND** a user runs `langstar prompt compare my-prompt.json` +- **THEN** langstar reports that path with `` in place of the string and exits with status 3 + +### Requirement: Print a secret reference in a destination prompt's model unchanged + +When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, langstar SHALL print a LangChain secret reference unchanged under any key, so the report names the secret. A secret reference looks like `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`. + +#### Scenario: Destination adds a model that reads its key from a secret + +- **WHEN** the file holds a prompt saved without a model +- **AND** someone saved the destination prompt with a model whose `openai_api_key` refers to the secret `OPENAI_API_KEY` +- **AND** a user runs `langstar prompt compare my-prompt.json` +- **THEN** langstar reports the model's path with the secret reference and `OPENAI_API_KEY` in it, and exits with status 3 + +### Requirement: Compare a prompt with its export file + +`langstar prompt compare ` SHALL read the prompt with the file's handle from the workspace, with its model and repo details, and list each difference by its JSON path with both values. `--commit ` SHALL choose the commit, and without it the latest. It SHALL compare every field in the file's `repo` and `commit`, including every manifest key and its `id` arrays, except `commit.model_config` and `commit.model_provider`, which import cannot set. + +#### Scenario: Prompt matches + +- **WHEN** the workspace's prompt matches the file +- **THEN** langstar reports no differences and exits with status 0 + +#### Scenario: Model temperature changed + +- **WHEN** someone changed the model temperature in the workspace's prompt +- **THEN** langstar lists the temperature's JSON path with both values and exits with status 3 + +#### Scenario: Model class changed + +- **WHEN** someone changed the workspace's prompt to use `ChatAnthropic` where the file's manifest names `ChatOpenAI` +- **THEN** langstar lists the JSON path of the model's `id` array with both values and exits with status 3 + +#### Scenario: Destination fills the model configuration differently + +- **WHEN** the workspace's prompt has the file's manifest and a `model_config` or `model_provider` that differs from the file's +- **THEN** langstar reports no differences and exits with status 0 + +#### Scenario: Repo readme changed + +- **WHEN** someone changed the readme of the workspace's prompt +- **THEN** langstar lists the readme's JSON path with both values and exits with status 3 + +#### Scenario: Prompt missing from the workspace + +- **WHEN** a user runs `langstar prompt compare my-prompt.json` +- **AND** the workspace has no prompt named `my-prompt` +- **THEN** langstar exits with status 1 and an error naming the handle + +### Requirement: Verify that a prompt can run in a workspace + +`langstar prompt verify ` SHALL do what `compare` does and also check that the workspace has every secret in `requires.secrets`. It SHALL name each missing secret and exit with status 3 when any secret is missing. When `requires.model_class` is not null, it SHALL also look for a saved model configuration whose settings `id` ends in that class name, and report a missing one as a note that leaves the exit status unchanged, because the prompt runs with the model its manifest carries. + +#### Scenario: Secret missing in the destination + +- **WHEN** the destination workspace has no secret named `OPENAI_API_KEY` +- **AND** the file lists `OPENAI_API_KEY` under `requires.secrets` +- **THEN** langstar reports `OPENAI_API_KEY` as missing and exits with status 3 + +#### Scenario: No saved model configuration for the model class + +- **WHEN** the file's `requires.model_class` is `ChatOpenAI` +- **AND** no saved model configuration in the destination workspace has a settings `id` ending in `ChatOpenAI` +- **AND** every secret in `requires.secrets` exists and the prompt matches the file +- **THEN** langstar notes that no saved model configuration matches `ChatOpenAI` and exits with status 0 + +### Requirement: Reject an export file of another kind or format version + +`prompt import`, `prompt compare` and `prompt verify` SHALL read a file only when its `kind` is `prompt` and its `format_version` is 1. For any other file they SHALL write nothing, exit with status 1 and name the field and the value they found. + +#### Scenario: File from a newer format + +- **WHEN** a user runs `langstar prompt compare my-prompt.json` +- **AND** `my-prompt.json` holds `"format_version": 2` +- **THEN** langstar exits with status 1 and an error naming `format_version` and the value 2 + +#### Scenario: Assistant file given to a prompt command + +- **WHEN** a user runs `langstar prompt import support-bot.json --workspace-id ` +- **AND** `support-bot.json` holds `"kind": "assistant"` +- **THEN** langstar creates no prompt, exits with status 1 and names `kind` and the value `assistant` + +### Requirement: Exit with 0 for a match, 3 for a difference and 1 for an error + +`prompt compare`, `prompt verify` and `prompt import` SHALL exit with status 0 when the prompt matches and every check passes, 3 when they find a difference or a failed check, and 1 on any other error. A usage error that clap rejects, such as a mistyped flag, SHALL keep clap's status 2. With `-f json` each SHALL print one JSON object with `status`, `differences` and `checks`. + +#### Scenario: Machine-readable compare + +- **WHEN** a user runs `langstar -f json prompt compare my-prompt.json` +- **THEN** stdout holds one JSON object with `status`, `differences` and `checks` +- **AND** progress messages go to stderr + +#### Scenario: Mistyped flag + +- **WHEN** a user runs `langstar prompt compare my-prompt.json --wrkspace-id ` +- **THEN** langstar prints clap's usage error and exits with status 2 + +### Requirement: Report the import's action and destination ID in the JSON report + +With `-f json`, the object that `prompt import` and `assistant import` print SHALL also hold `action`, one of `create`, `update` and `unchanged`, and the destination's commit hash or assistant ID. With `--dry-run`, `action` SHALL name the action import would take, and the ID SHALL be null when import would create the resource. + +#### Scenario: Machine-readable first import + +- **WHEN** a user runs `langstar -f json prompt import my-prompt.json --workspace-id ` +- **AND** the destination has no prompt named `my-prompt` +- **THEN** stdout holds one JSON object whose `action` is `create` and that holds the new commit hash diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md new file mode 100644 index 000000000..3bd241687 --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -0,0 +1,180 @@ +# Tasks + +Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.3 and 2.4 merge in that order: the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.2 can merge at any time. Tasks 2.1, 2.3 and 2.4 add the second workspace and deployment, and developers merge them before anyone archives the change, as `design.md`'s decision on test environments says. Developers can build and merge every other group before them. Until they merge, a task below that names the destination workspace means the existing test workspace from `LANGSMITH_WORKSPACE_ID`, into which each test imports a prompt under a fresh handle, and a task that names the destination deployment means the existing test deployment. The tests that need a second environment carry `#[ignore = "needs task 2.1"]` or `#[ignore = "needs task 2.4"]`, and the pull request for that task removes the attribute: the graph moves in 3.3 and 10.2 need `test_graph_alt` on the destination deployment from 2.4, and the prompt in 9.2 that exists only in the destination workspace needs the workspace from 2.1. The structured output check in 7.3 skips without an OpenAI key. + +## 1. Check the API round trip before building on it + +- [ ] 1.1 Rerun the jq query for `prompt-endpoints.json` in `reference/api-specs/langsmith/FRAGMENTS.md`, so the extract picks up the `include` parameter of the commit read that its 2025-11-29 copy lacks, and extract the workspace secrets endpoints (`/api/v1/workspaces/current/secrets`) from `reference/openapi/langchain/langsmith/openapi.json` into a new file in `reference/api-specs/langsmith/` with its jq query in `FRAGMENTS.md`; verify by rerunning each query and getting the same file +- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording what the live API returns. Verify that every claim in the report names the request that produced it. Without a second workspace, this task pushes to the existing test workspace under a fresh handle, and it runs a prompt with a model only when an OpenAI key is set. + - For a structured prompt saved with a model, read with `include=model`: the manifest, its outer `id`, the `model_config` and `model_provider` beside it, and whether `model_provider` matches the class name export writes to `requires.model_class`. + - Push that manifest to a second workspace: whether the read-back returns the same manifest, `model_config` and `model_provider`, and what pushing an identical manifest again returns. + - On the Agent Server: what `POST /assistants/search` returns for a graph ID the deployment does not serve, what it returns for `context`, whether an assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, and whether an update can clear the description. Also whether a create and an update that send both `config.configurable` and `context` are refused, and whether an update that sends `configurable` alone replaces `context` too. + - What `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model. + +## 2. Test environments + +- [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task adds the model provider key only once CI has one. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. +- [ ] 2.2 Replace the bare `#[ignore]` on `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs`, whose reason says a deployment built from GitHub has no `custom_url`, with `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, the guard `test_assistant_description_field` beside it carries. The control plane sets `source_config.custom_url` on a deployment built from GitHub, as `SourceConfig.custom_url` in `reference/openapi/langchain/control-plane/openapi.json` says. Verify by running the test against the test deployment. +- [ ] 2.3 Add a second `langgraph.json` under `tests/fixtures/` that registers the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so the update tests in 3.3 and 10.2 can move an assistant from one graph to the other. To verify, deploy the new file while the source test deployment is live, which shows whether the control plane accepts a second deployment from this repository: run `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file. A 409 "A deployment already exists for this agent environment" means it does not, and the developer reports that to codekiln before anyone starts task 2.4. Otherwise run `langstar assistant list --deployment `, find one assistant with graph ID `test_graph` and one with `test_graph_alt`, then delete the deployment. +- [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that deploys the `langgraph.json` from 2.3 under the name prefix `dest-integration-test`. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, so this prefix keeps the source fixture, whose prefix is `pr-integration-test`, from picking up the destination deployment. `find_reusable_deployment` then falls back to any live deployment built from the same repository and graph config file, so the separate file from 2.3 keeps the destination fixture from picking up the source deployment. The control plane answers a second deployment of one agent environment with 409 "A deployment already exists for this agent environment". No document says which fields make an agent environment, and the verification in 2.3 shows whether a second graph config file in the same repository makes a new one. `.github/workflows/cleanup-test-deployments.yml` runs every four hours and deletes each deployment whose name contains `integration-test` once it is four hours old or older, so the destination deployment is deleted and rebuilt on the same schedule as the source deployment. Verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment with `langstar assistant list` + +## 3. SDK reads and writes the fields export and import need + +Each task in this group adds public fields to SDK structs and updates every struct literal of those structs in this repository, so the workspace still compiles. The structs are `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. Commit these tasks as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer and leave the version in `Cargo.toml` alone, as [Developers commit the new SDK fields as `✨ feat(sdk)`, so the next release is a minor version](design.md#developers-commit-the-new-sdk-fields-as--featsdk-so-the-next-release-is-a-minor-version) explains; the Prepare Release workflow picks the version. + +Each integration test in this group carries `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, as the tests in `sdk/tests/structured_prompts_integration_test.rs` do, so that the SDK Integration Tests job in `.github/workflows/ci.yml` and `mise run check` both run it. + +- [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with a `mockito` test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model +- [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls to list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which `prompt import --update` from 7.2 and the end-to-end test in 10.3 use; verify with `mockito` tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it +- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with `mockito` tests of the request bodies, an integration test that creates an assistant with context on the existing test deployment, reads it back with its version and finds it by graph ID, and a second integration test that creates an assistant on the destination deployment from 2.4, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. The second test carries `#[ignore = "needs task 2.4"]`, because the existing test deployment serves only `test_graph`. +- [ ] 3.4 Add `description` to `CommitRequest`, and `readme`, the repo's `tags` and `tenant_id` to `Prompt`, which `PromptClient::get` returns and export reads the repo details and `source.workspace_id` from, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with `mockito` tests of the request bodies and of a `PromptClient::get` response that holds `readme`, `tags` and `tenant_id`, and an integration test that pushes a commit with a description and reads it back through the commit read from 3.1, then updates the repo's description, readme, tags and visibility and reads each back through `PromptClient::get` + +## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report + +- [ ] 4.1 Add the export file types, with sorted keys, sorted `commit.tags`, `repo.tags` and `requires.secrets`, no export time and a reader that accepts only `format_version` 1 and the expected `kind`, and the destination record file, which langstar replaces by writing a temporary file in the same directory and renaming it over the old one, to a new module in `cli/src/`. Verify with these unit tests: + - Tests that build equivalent nested values with keys inserted in different orders, serialize each, and assert identical text with keys sorted at every level, and a test that `commit.tags`, `repo.tags` and `requires.secrets` given in different orders serialize to the same text. + - A test that updating one destination leaves the others unchanged. + - A test that a record write that fails before the rename leaves the earlier record file unchanged, its pending entries included. + - A test that reading a file whose `format_version` is 2, or whose `kind` names the other resource, returns an error naming the field and its value. +- [ ] 4.2 Add the JSON comparison, which compares every key under the `repo`, `commit` and `assistant` objects except `commit.model_config` and `commit.model_provider`, reads `assistant.config.configurable` and `assistant.context` as one settings value, as [Import sends an assistant's settings in one field](design.md#import-sends-an-assistants-settings-in-one-field) says, and reports differences by JSON Pointer. Verify with these tests: + - Unit tests on nested objects, arrays, missing keys, and keys holding `/` or `~`, which the report escapes as RFC 6901 requires, so `a/b` appears as `/a~1b` and `a~b` as `/a~0b`. + - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. + - A test that a destination whose `commit.model_config` and `commit.model_provider` differ from the file's, while the manifest matches, is reported as a match. + - Tests of the settings alias: a file with `assistant.config.configurable` alone against a destination holding the same values in both fields, and a file with `assistant.context` alone against the same destination, each reported as a match; a file with `config.configurable` alone against a destination whose `context` holds other values, reported as one difference; and a test that every other key under `assistant.config` is compared as it stands, so a changed `config.recursion_limit` is reported at its path. +- [ ] 4.3 Add the shared result report, which the compare, verify and import commands for both prompts and assistants print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr. Before it prints either side of a difference, the report puts `` in place of every string the credential check from 4.4 flags, by either rule, and prints a secret reference unchanged under any key. It does this both when the difference is at the string and when it is at an object that contains it, for strings from the destination and from the file. Leave clap's status 2 for usage errors. Verify with these tests: + - Unit tests of the report and of the exit status each result gives. + - Redaction tests of a difference at `openai_api_key` where the destination holds a string; at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`; at `/assistant/config/configurable/openai_api_key` and at `/assistant/config/configurable/OPENAI_API_KEY` where the destination holds a string; at a model's `default_headers/Authorization` holding `Bearer ` and a token; at a model's `model_kwargs/extra` holding a string that starts with `lsv2_`; at `/assistant/context/checkpoint_uri` holding a `postgresql://` URL with a password; and at `/assistant/config/configurable/db` where the destination adds `{"user": "app", "password": ""}`. Each asserts that the text and the `-f json` report print `` at the flagged string, that neither holds the string or any part of it, and, for the `db` object, that `user` prints unchanged. + - Tests of a difference at `/commit/manifest/kwargs/last` where the destination's model refers to the secret `OPENAI_API_KEY`, and at `/assistant/config/configurable/openai_api_key` where the destination refers to that secret, asserting that the text and the `-f json` report both print the secret reference with `OPENAI_API_KEY` in it, and that neither prints ``. +- [ ] 4.4 Add the secret reference walk that lists secret names from a manifest, and the credential check that export, compare, verify and import share. The check walks every string under a JSON value, skips secret references and empty strings, and returns the JSON Pointer path of each flagged string with the rule that flagged it, the key-name rule or one of the value shapes in [Export refuses to write a file that holds a credential](design.md#export-refuses-to-write-a-file-that-holds-a-credential), and leaves the string out. Keep the word list and the value shapes in one module with a doc comment that names each. Verify with these tests: + - Unit tests of the secret reference walk on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, and on a manifest with no secret reference. + - Key-name tests that flag a string under `openai_api_key`, `OPENAI_API_KEY`, `x-api-key`, `Authorization`, `Cookie`, `access_token`, `refresh_token`, `client_secret`, `db_password`, `privateKey`, `signingKey` and `credentials`, and under an array whose key is `api_keys`, and that pass a string under `max_tokens`, `cache_key`, `author` or `keywords`, and a number or boolean under a flagged name. + - Value-shape tests under a neutral key such as `note` that flag `Bearer ` followed by a 20-character token, `Basic ` followed by 20 base64 characters, `Basic dXNlcjpwYXNz`, whose twelve base64 characters decode to `user:pass`, a PEM block for each of `PRIVATE KEY`, `RSA PRIVATE KEY` and `OPENSSH PRIVATE KEY`, `postgresql://app:@host/db`, `redis://:@host:6379`, `Server=db;User Id=app;Password=`, an Azure string with `AccountKey=`, a JSON Web Token, and a token with each prefix the design lists. Each asserts the rule named in the result. + - Tests that pass ordinary text: `Basic instructions follow.`, `Bearer in mind`, `https://example.com/path`, `postgresql://host/db` with no password, `sk-` on its own, and `-----BEGIN PUBLIC KEY-----`. + - Tests that skip a secret reference under `openai_api_key` and an empty string under `password`, and that escape a flagged key holding `/` or `~` in the path as RFC 6901 requires. + - A test that the error message for each flagged string holds the path and the rule, and does not hold the string or any eight-character run of it. + +## 5. prompt export + +- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output`, `--not-a-secret` and the scoping flags, and run the credential check from 4.4 over the `repo` and `commit` objects before writing. Verify with these tests: + - Integration tests that create four prompts through the SDK (a `json_schema` prompt with no model, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit. The file for the prompt with a model lists `OPENAI_API_KEY` under `requires.secrets` and `ChatOpenAI` under `requires.model_class`, and holds no secret value. The file for the `json_schema` prompt holds an empty `requires.secrets` and null under `requires.model_class`. + - An integration test that exports, with `--commit prod`, a prompt whose `prod` tag points at an earlier commit, and asserts that `source.commit_hash` and the manifest are that earlier commit's. + - An error test that exports a handle the workspace has no prompt for, and asserts that it exits with status 1, names the handle and writes no file. + - An integration test that exports a private prompt once as `my-prompt` and once as `-/my-prompt`, without `--`, and asserts that both files are identical. + - A CLI test that runs `prompt export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.commit_hash` matches the SDK's read and that any progress message is on stderr. + - An integration test that gives one commit two tags, exports it twice, and asserts that the two files are identical. + - Unit tests in `cli/src/commands/prompt.rs` that serve, through a `mockito` server and `LangchainClient::with_base_urls`, one commit for each export refusal scenario in `specs/prompt/spec.md`: a `ChatOpenAI` constructor with a string under `openai_api_key`, and another under `OPENAI_API_KEY`; `default_headers.Authorization` holding `Bearer ` and a token; `model_kwargs.extra` holding a string that starts with `sk-`; manifest `metadata.source_db` holding a `postgresql://` URL with a password; and a repo readme holding a PEM private key. Each calls the export code with `--output my-prompt.json`, and once without `--output`, and asserts that it exits with status 1, names the path and the rule without the string, writes no file and prints nothing to stdout. + - A unit test that serves a commit whose model refers to the secret `OPENAI_API_KEY` under `openai_api_key`, and asserts that export writes the file with the secret reference. + - Unit tests of `--not-a-secret`: a message template holding `Authorization: Bearer ` and an example token, exported with `--not-a-secret` at the template's path, writes the file with the template unchanged and prints a warning naming the path to stderr; the same commit with a second flagged string at another path still exits with status 1 and names only that path; and `--not-a-secret /repo/description`, where nothing is flagged, exits with status 1, names the pointer and writes no file. + - A unit test in `cli/src/commands/prompt.rs` that serves a prompt through a `mockito` server and `LangchainClient::with_base_urls`, calls the export code with no workspace ID set, and asserts that `source.workspace_id` holds the repo's `tenant_id`. +- [ ] 5.2 Document `prompt export` in `docs/usage/` and the README. The page lists the key names and value shapes the credential check flags, says that export then names each path and writes no file, shows `--not-a-secret` for a flagged string that holds no secret, and says that a credential under an ordinary key name with an ordinary-looking value, such as a random string under a key called `x`, passes the check and reaches the file. It tells users to keep credentials in the workspace's secrets, and to read an export file before they commit it. Verify that each documented command runs as written. + +## 6. prompt compare and verify + +- [ ] 6.1 Add `langstar prompt compare`. Verify with these tests: + - An integration test that exports a prompt, changes its model temperature through the SDK, and asserts the reported path, both values and exit status 3. + - Integration tests that a changed commit description and a changed repo readme are each reported the same way. + - An integration test that exports a prompt, adds a tag to the exported commit through the SDK, and asserts that compare reports `/commit/tags` with both values and exits with status 3. + - An integration test that exports a tagged commit, pushes a newer commit to the prompt through the SDK, runs `prompt compare --commit `, and asserts that it compares the tagged commit and exits with status 0. + - A CLI test that `langstar -f json prompt compare` prints one JSON object with `status`, `differences` and `checks` to stdout and its progress to stderr. + - A CLI test that `prompt compare` with a mistyped flag exits with status 2. + - A CLI test that `prompt compare` of a file holding `"format_version": 2` exits with status 1 and names `format_version` and the value 2. + - A CLI test that `prompt compare` of a file whose handle the workspace lacks exits with status 1 and names the handle. +- [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, which matches `requires.model_class` against the class name at the end of each configuration's settings `id`. Verify with these tests: + - An integration test against a workspace that lacks a secret the file lists, asserting that verify names the secret and exits with status 3, and one against a workspace that has every listed secret, asserting that verify exits with status 0. + - An integration test that exports a prompt, changes its model temperature through the SDK, runs verify, and asserts that it reports the path and both values as `prompt compare` does and exits with status 3. + - A test that a missing saved model configuration is reported as a note and leaves the exit status at 0. + - A test that verify of a prompt without a model, whose `requires.model_class` is null, reports no saved model configuration note and exits with status 0. + - A test that a file whose manifest refers to a secret its `requires.secrets` leaves out exits with status 1 and names `requires.secrets`. +- [ ] 6.3 Document compare and verify; verify that each documented command runs as written + +## 7. prompt import + +- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back. When the destination has a prompt with the file's handle and no commit, import pushes the first commit and creates the tags, as for a new prompt. Verify with these tests: + - Integration tests that export each prompt shape from 5.1, the prompt without a model among them, from the test workspace, import it into the destination workspace, and compare the source and destination through the SDK with the model included: the manifest, the commit description, and the repo's description, readme, tags and visibility. + - An integration test that imports a file whose `requires.secrets` names a secret the destination workspace lacks, and asserts that it creates no prompt, names the secret and exits with status 3. + - An integration test that creates a repo with the file's handle and repo details and no commit in the destination workspace, as an import that stopped after creating the repo leaves it, then imports the file without `--update`, and asserts that the prompt holds one commit with the file's manifest and the file's tags, and that import exits with status 0. + - A CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set, and asserts that it exits with status 1, names the missing flag and creates no prompt. + - A CLI test that `langstar -f json prompt import` of a new prompt prints one JSON object whose `action` is `create` and that holds the new commit hash, and that the same command with `--dry-run` prints `create` and a null commit hash. + - An integration test that imports a file whose repo readme differs from the destination's, into a destination that holds the repo and no commit, without `--update`, and asserts that import pushes the commit and tags, leaves the readme unchanged, reports `/repo/readme` and exits with status 3. + - An integration test that creates a public repo with the file's handle and no commit in the destination workspace, then imports a file whose `repo.is_public` is false without `--update`, and asserts that the repo still holds no commit, that import reports `/repo/is_public` and exits with status 3, and that the repo is still the only thing there to find. + - A CLI test that runs `prompt import` on an assistant export, and asserts that it exits with status 1, names `kind` and the value `assistant`, and creates no prompt. + - CLI tests that import a file whose manifest refers to `ANTHROPIC_API_KEY` while `requires.secrets` holds only `OPENAI_API_KEY`, and a file whose `requires.model_class` names a class other than the one at the end of the manifest's chat model `id`, and assert that each creates no prompt, names the field and exits with status 1. +- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record. Verify with these tests: + - An integration test that a repeated import creates no commit. + - An integration test that a changed destination is refused without `--update`. + - An integration test that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3. + - An integration test that pushes the file's manifest to the destination, leaves an older commit hash in the destination record, imports the file, and asserts that import exits with status 0, prints no change warning and leaves the latest commit's hash in the record. + - An integration test that pushes the file's manifest and tags to the destination, imports the file with no destination record, and asserts that import exits with status 0 and writes the latest commit's hash to the record under the destination workspace. + - An integration test that `--update` adds one commit with the right parent and the file's commit description. + - An integration test that pushes the file's manifest to the destination without the file's tags, as an import that stopped before creating its tags leaves it, runs import with `--update`, and asserts that it creates the missing tags on that commit and adds no commit. + - An integration test that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit. + - An integration test that `--update` against a public destination repo, from a file whose `repo.is_public` is false, makes the repo private before it pushes, asserted by the order of the two requests and by reading the repo back private with the file's commit. + - An integration test that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, and that `--dry-run` lists the move. + - An integration test that `--update` against a destination whose latest commit matches the file and carries a `staging` tag the file lacks deletes `staging`, adds no commit and exits with status 0, and that `--dry-run` lists the delete. + - An integration test that `--dry-run` writes nothing. + - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit read back that lacks a field the file holds, calls the import code, and asserts that the result names that field's JSON Pointer path and gives exit status 3, and that the destination record holds the new commit hash. +- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --locked --script`. The script names `langsmith`, `langchain-openai` and `jsonschema` as inline script dependencies, each pinned to an exact version. Commit the lockfile `uv lock --script` writes next to the script, so `--locked` fixes every transitive dependency too and a new upstream release cannot change the CI result. The script validates each result with `jsonschema` against the prompt's schema. The script takes the workspace ID and the handle as arguments, and the import integration test passes the same workspace ID it passed to `prompt import --workspace-id` and the handle it imported under. The script builds its `Client` with `workspace_id` set to that argument rather than reading `LANGSMITH_WORKSPACE_ID`, so it reads the workspace import wrote to, before and after task 2.1 adds a second one. It then pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each with that workspace's ID, under `mise run check` and in that CI job. When `OPENAI_API_KEY` is unset, the task prints that it skipped the check and exits with status 0; verify that too, by running it without the key. +- [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written + +## 8. assistant export + +- [ ] 8.1 Add `langstar assistant export` with `--output`, `--prompt-ref` and `--not-a-secret`, and run the credential check from 4.4 over the `assistant` object before writing. Verify with these tests: + - An integration test that creates an assistant with configuration, context and metadata through the SDK and asserts every field of the exported file. + - An integration test that exports, with `--prompt-ref /config/configurable/system_prompt`, an assistant whose value there is `support-system:prod`, and asserts that the file holds that pointer and `support-system:prod` under `requires.prompts`. + - An integration test that exports the same assistant version twice and asserts that the two files are identical. + - An error test for a pointer that names nothing. + - An error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file. + - An integration test that exporting a graph's default assistant on the test deployment exits with status 1 and writes no file. + - Unit tests in `cli/src/commands/assistant.rs` that serve, through a `mockito` server and `LangchainClient::with_base_urls`, one assistant for each export refusal scenario in `specs/assistant/spec.md`: `config.configurable.OPENAI_API_KEY`, `context.anthropic_api_key` and `metadata.openai_api_key` holding a string; `context.github_access_token` holding a string; `config.configurable.db.password` holding a string; `config.configurable.checkpoint_uri` holding a `postgresql://` URL with a password; `config.configurable.tool_headers.Authorization` holding `Bearer ` and a token; and `metadata.deploy_note` holding an OpenSSH private key. Each calls the export code with `--output support-bot.json`, and once without `--output`, and asserts that it exits with status 1, names the JSON Pointer path and the rule without the string, writes no file and prints nothing to stdout. + - A unit test that serves an assistant whose `config.configurable.openai_api_key` holds a secret reference, and asserts that export writes the file with the reference. + - A unit test that exports an assistant whose `config.configurable.token_budget_policy` holds `strict` with `--not-a-secret /config/configurable/token_budget_policy`, and asserts that the file holds `strict` there and stderr names the path, and one without the flag that asserts status 1. + - A CLI test that runs `assistant export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.assistant_id` is the exported assistant's ID and that any progress message is on stderr. +- [ ] 8.2 Document assistant export. The page links the prompt export page's list of what the credential check flags, says that export then names each path and writes no file, shows `--not-a-secret`, and says that a credential under an ordinary key name with an ordinary-looking value passes the check and reaches the file. It tells users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before they commit it. Verify that each documented command runs as written. + +## 9. assistant compare and verify + +- [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, `--record`, the lookup of the recorded assistant in the destination record from 4.1, and the check, shared by assistant compare, verify and import, that each pointer under `requires.prompts` still resolves in the file's `assistant` object to the prompt name recorded with it. Verify with these tests: + - An integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3. + - Unit tests that compare reads the assistant ID from a destination record written by the 4.1 types, once at the default path and once at a path passed with `--record`. + - An integration test that `assistant compare` with no destination record entry and no `--assistant-id` exits with status 1 and names the deployment. + - A unit test that a file whose configuration names another prompt at a pointer under `requires.prompts` exits with status 1 and names the pointer and both values. Another unit test puts `sk-` followed by a 40-character token at that pointer and asserts that the error prints `` in place of the string. +- [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--record`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`. Verify with these tests: + - An integration test that runs verify with `--assistant-id` against an exported assistant and exits with status 0. + - An integration test that exports an assistant, changes its metadata through the SDK, runs verify with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3. + - An integration test of a file whose graph ID the deployment lacks, which reports the graph ID and exits with status 3. + - Unit tests in `cli/src/commands/assistant.rs` that serve, through a `mockito` server, a search that answers 422 with an error naming `graph_id`, and assert that verify reports the graph ID missing and exits with status 3, and a 422 naming another field, and assert that verify exits with status 1. + - An integration test of a file whose prompt tag the workspace lacks, which reports the prompt and exits with status 3. + - A unit test that a prompt reference with no owner and no tag, such as `support-system`, makes the prompt check request `GET /api/v1/commits/-/support-system/latest`, one that `my-org/support-system:prod` requests `GET /api/v1/commits/my-org/support-system/prod`, and one that `-/support-system:prod` requests `GET /api/v1/commits/-/support-system/prod`. + - An integration test that runs verify with `--workspace-id` set to the destination workspace, for an assistant whose referenced prompt tag exists only in that workspace, and asserts that verify exits with status 0 and that `checks` reports the prompt as found. This test carries `#[ignore = "needs task 2.1"]`, because until then the destination workspace is the test workspace. + - A CLI test that `langstar -f json assistant verify` prints one JSON object with `status`, `differences` and `checks` to stdout. +- [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written + +## 10. assistant import + +- [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`. Verify with these tests: + - An integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment, and checks the name, graph ID, configuration, context, metadata and prompt references through the Agent Server API. + - An integration test that `--record ` writes the destination record to that path. + - Integration tests that a file whose graph ID the deployment lacks, and a file with a missing prompt tag, each create no assistant and exit with status 3. + - A unit test in `cli/src/commands/assistant.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, an assistant read back that differs from the file, calls the import code twice, and asserts that the first call gives exit status 3 and leaves the new assistant ID in the destination record, and that the second call creates no assistant. + - A CLI test that `assistant import` of a prompt export exits with status 1, names `kind` and the value `prompt`, and creates no assistant. + - Unit tests in `cli/src/commands/assistant.rs` that serve the Agent Server through a `mockito` server, one for each scenario of "Send an assistant's settings in one field" in `specs/assistant/spec.md`. They assert that a file with the same settings in both fields sends `config.configurable` and no `context`, that a file with only `context` sends `context` alone, and that a file whose two fields differ sends no request, leaves the destination record as it was, names both paths without their values and exits with status 1. + - Unit tests, through a `mockito` server, of the read-back for a file that holds settings in one field only: a file with `config.configurable` alone against a read-back that holds the same values in both fields, and a file with `context` alone against the same read-back. Each asserts that import reports no difference and exits with status 0. + - An integration test that imports an assistant from a file holding settings in `config.configurable` alone onto the test deployment, runs `langstar assistant compare` and asserts status 0, then imports the same file again and asserts that it creates no second assistant and reports no difference. A file an older deployment produced therefore copies onto a deployment that answers with both fields, and stays matched on every import after the first. A second integration test does the same for a file holding settings in `context` alone. + - An integration test that creates an assistant with a chosen ID on the destination deployment, writes a destination record that holds that ID as pending, imports the file, and asserts that the deployment holds one assistant from the file, with that ID, and that the record marks the ID created. +- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`. Verify with these tests: + - An integration test that a repeated import creates no assistant. + - A unit test, through a `mockito` server, that `--update` of a file with the same settings in both fields sends `config.configurable` and no `context`. + - An integration test that `--update` changes the recorded assistant's name, configuration, context and metadata on the existing test deployment, and a second integration test, carrying `#[ignore = "needs task 2.4"]`, that `--update` moves the recorded assistant from `test_graph` to `test_graph_alt` on the destination deployment from 2.4. + - An integration test that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, and that `--assistant-id ` without `--update`, against an assistant that differs from the file, creates and changes no assistant and exits with status 3. + - An integration test that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}`, or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does. + - An integration test that `--update` of an assistant with a description and settings from a file that holds neither a description nor settings in either field clears both, or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does. + - A unit test, through a `mockito` server, that `--update` of a file holding settings in `config.configurable` alone, against an assistant whose `context` holds other settings, sends `config.configurable` and no `context`, so the update never clears the field it did not send. + - An integration test that a deleted recorded assistant exits with status 1. + - An integration test that `--dry-run` creates no assistant, changes none and leaves the destination record as it was. +- [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace, then imports an assistant that refers to them onto the destination deployment and runs `assistant verify`, passing `--workspace-id` set to the destination workspace to both assistant commands; verify that it passes, and that verify fails after deleting one prompt tag in the destination workspace through the tag delete call from 3.2 +- [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written + +## Workflow follow-up + +- Archive the change once 2.1, 2.3 and 2.4 have merged and a CI run shows the tests whose `#[ignore]` those tasks removed passing. If the control plane answered 2.3 with a 409, archive it once codekiln has decided what the assistant tests run against. +- Run `openspec validate export-import-prompts-and-assistants --strict`, then archive the change in the pull request for group 10.